Diagnosis unclear: how a healthcare provider resolved elusive application performance and availability issues
Resolving intermittent issues is one of the most challenging problems that IT Operations teams face, since they are difficult to diagnose. Intermittent issues don’t provide IT teams with enough clues to help them quickly identify and resolve problems. The usual approach to analyzing patterns and trends don’t apply.
One of my clients, a national healthcare system, recently dealt with this type of challenge with CPR+, their specialized pharmacy application. Despite the fact that all users in every location accessed CPR+ running on a shared Citrix virtual environment in the company’s primary data center, certain users and locations experienced intermittent cases of poor performance, while others did not.
Poor Performance Impacts Patient Care and the Business
Users’ complaints centered on both performance and availability:
- Performance: Users experienced excessively long login times accessing CPR+, some in excess of over a minute
- Availability: Once logged in, users experienced difficulty staying connected. The application would just ‘disappear’. Then, when users attempted to reconnect, they would experience long login times again
This cycle of poor user experience impacted the provider’s business in a variety of ways
- Patient Care: Ability to create, track, and transfer pharmacy orders
- Patient Safety: Ability to electronically validate drug-drug interactions and produce patient documentation
- Compliance: Ability to report compliance with HIPAA and Medicare laws
- Revenue Management: Ability to submit timely claims for reimbursement
The company’s IT team spent significant time attempting to validate and quantify user complaints within CPR+ with limited success. Here’s how Aternity helped.
Step 1: Validate User Complaints
The first step in dealing with an intermittent problem is to validate user complaints. Monitoring the actual End Users’ Experience with CPR+ showed that users in certain locations experienced the performance and availability problems more frequently than others did, even though they were all utilizing shared Citrix servers and the same version of the application.
Some locations experience CPR+ problems while others do not
Step 2: Identify Possible Causes
As we began to drill into the performance and availability challenges, it became apparent we were facing a multi-faceted problem.
Items Impacting Excessive Login Times:
- In instances where it took 27-60+ seconds to launch CPR+ (23 seconds slower than average) infrastructure side processing delays were driving this. Excessively slow log-in times occurred at specific times of day, generally around shift change
- Users in one facility experienced Single Sign-On response times of 24-40 seconds, while users in a different facility saw 1.8-3.5 seconds
- One Wireless Access Point delivered low signal strength for connected devices, leading to poor response time when accessing CPR+
During the morning shift change around 7:30 AM, <1> the infrastructure becomes the major contributor to excessively slow CPR+ launch times of over 30 seconds, even though overall, <2> the client device is by far the larger contributor to delay
The Application Disappearing:
- Certain devices experienced 40-50 network disconnects per day, compared to an average between 0-1 per device
- Users with outdated versions of the Citrix Receiver experienced Citrix Client Crashes that caused the ‘Application Disappearing’ complaint
- Similarly, over 500 CPR+ crashes occurred in a 30-day period, producing the same ‘Application Disappearing’ result
Crashes in CPR+, the Citrix Receiver, and Single Sign-On resulted in complaints of the ‘Application Disappearing’
Step 3: Resolve the Issues
To address the performance challenges, IT took the following steps:
Spikes in CPR+ launch times: IT noted that excessive launch times were caused by infrastructure batch and other scheduled jobs occurring during busy work times. IT adjusted these jobs to occur during off hours.
Single Sign-On delays: The desktop support group identified unintended differences in user profiles and in Group Policy Objects (GPO) between users in different facilities. Once these differences were eliminated, each group experienced satisfactory Single Sign-On response times.
Wireless Access: Additional wireless coverage was easy to obtain for end-points impacted by the significant degradation in performance.
For the availability challenges, once IT upgraded the Citrix Receiver, patched the crashing modules of CPR+, and upgraded the network drivers on the machines experiencing network disconnects, they eliminated 95% of the ‘Application Disappearing’ complaints.
IMPROVING END USER EXPERIENCE IN DIGITAL HEALTHCARE DELIVERY
Healthcare organizations around the world use Aternity to ensure the productivity of their medical staff. Aternity is the industry’s leader in providing HIPAA-certified End User Experience Management solutions for any app running on physical, virtual, and mobile devices. Healthcare organizations leverage Aternity’s proactive problem detection, trending, and analysis to optimize the impact of IT on clinician productivity, validate SLAs of Cloud-delivered apps, make informed capacity planning decisions, and ensure success of BYOD/mHealth initiatives.
For more information on Aternity’s solutions for healthcare visit our website, or stop by our booth (#10328) at the HIMSS16 Conference & Exhibition scheduled from February 29th through March 4th in Las Vegas.