Regional Healthcare Provider Recovers 400 Hours of Lost Productivity a Month
One of my clients, a regional not-for-profit healthcare provider, was suffering significant and ongoing problems with a specialized application used by their clinicians in their Sports Rehab division. This sports rehab specific application is used by American Physical Therapy Association (APTA) certified medical staff over the course of their patient care and clinical processes. Clinicians rely on this application for every aspect of their interaction with patients – looking up records, recording treatments plans, providing progress reports, etc.
Citrix centralized management also centralizes performance problems
The APTA application runs on traditional desktops and on virtualized Citrix servers. The majority of staff use the APTA application by accessing a Citrix virtual environment within the healthcare provider’s own centralized data center. This type of centralized setup is nothing new to healthcare. Healthcare IT shops have used Citrix-delivered clinical apps since the first Win32-based based EMRs. An approach that centralizes management, however, also centralizes application performance challenges.
For over a year, the sports rehab clinicians bore the brunt of a few common challenges associated with this approach. Three main problems impacted their poor user experience:
- Difficulty initially connecting to the Citrix hosted application
- Challenges in remaining connected to the application after initially logging in
- Significant performance challenges within the application—specifically for critical high volume business activities
After attempting to validate, quantify and troubleshoot the APTA application — both internally and with the application vendor — IT finally turned to Aternity in order to resolve these problems.
Three key pain points
Once they started to monitor end user experience of the APTA application with Aternity Workforce APM, three key pain points quickly became apparent.
1. Lost productivity – for both medical staff and IT
As shown in the Productivity Dashboard below, clinicians experienced excessive response times for key business activities, which led to 400 hours of lost productivity over a 30 day timeframe. No healthcare organization can afford that kind of impact to its staff.
In addition, because IT lacked the tools to properly investigate and resolve the root cause of these performance problems, Mean Time to Repair (MTTR) for these application problems was around 10 hours.
2. Poor patient care
IT and the clinicians weren’t the only ones impacted by this problem. Even more importantly, patient care suffered as well. As the Productivity Dashboard shows, two of the excessively long business activities, running the Initial Evaluation Report and the Plan of Care Report, took between 20 and 30 seconds on average.
3. Finger pointing between the application vendor and the network team
The third main problem had to do with the complexity of troubleshooting application performance on virtual environments. Since virtual environments disrupt the traditional relationship between applications, physical hardware, operating systems, and presentation layers, problem resolution is notoriously difficult. In this case, the APTA application vendor blamed the network for poor connectivity and app performance. The network team never found a network issue that could cause these problems, so they blamed right back. Caught in the middle were the clinicians and the patients.
Tracing the problem to an application memory leak
Because Aternity Workforce APM provides visibility into the VDI blind spots for virtual desktop monitoring, and Citrix XenApp monitoring, we were able to quickly resolve the problem. Throughout the day, we routinely saw drastic increases in the process memory utilization for each of the four servers hosting the APTA app, from around 200MB in the morning to 2-2.5 times that by the evening.
Looking at the process memory consumption of individual user’s devices over time confirmed a memory leak as the root cause. The chart below shows a single user’s memory consumption starting at <100MB and maxing out at nearly 900MB. This pattern of escalating memory consumption is especially problematic in virtualized environments, since a single user’s consumption impacts others on the shared environment.
In reviewing the APTA connectivity problems, we identified the issue by looking at application crashes—which can appear to end users as if application disconnects. As the dashboard below shows, 99 crashes occurred over a 30 day period, and 73 of them were due to the kernelbase.dll.
Once the app owner resolved the memory leak and received patches from the vendor for the application crashes, connectivity and performance issues were resolved. The medical staff recovered 400 hours of lost productivity per month. The IT organization lowered its MTTR metrics for this type of problem from 10 hours to minutes. And most importantly, clinicians had more time to spend with patients, delivering a better overall quality of care.
If you’re experiencing problems similar to the one I’ve described, you can register for a free product evaluation of Aternity Workforce APM today.