4 Steps: Hold Your SaaS Vendor Accountable to Application Performance SLAs
Many of my enterprise customers have turned to a SaaS model for apps such as office productivity, HR, and CRM. Rather than acquiring, implementing, and maintaining software and the associated infrastructure, they use a SaaS solution on a pay-as-you-go basis for the seats they need. The benefits of SaaS to IT are clear:
- Faster time to deploy
- Reduced operational and capital expenses
- Smooth software and hardware upgrades
- The ability to shift staff to more strategic IT projects
However, these IT benefits are outweighed if SaaS delivered applications don’t deliver a quality end user experience to the workforce relying on them. Although SaaS vendors offer Service Level Agreements (SLAs) governing availability goals, incident response time commitments, and penalties, these metrics are insufficient to guarantee excellent end user experience. Penalties paid at the end of the month don’t help the end user whose app is slow when they schedule a meeting, file an expense report, or look up a customer.
Professional services firm implements SaaS expense reporting and gets an earful
I recently helped one of my customers address these SLA short-comings during their implementation of Infor Expense Management, a SaaS-based timesheet and expense management application. The company, a professional services firm with over 10,000 staff located in 100 offices around the world, works with the largest enterprises in the world on projects which can last from several months up to a year. Staff assigned to projects must account for their time in order to ensure the project remains within budget, so tracking timesheets and expenses is fundamental to the top-line revenue of the company as well as workforce reimbursement.
Aside from the Aternity administrator, my key point of contact within the firm’s IT shop was the product owner responsible for the Infor application. Shortly after rolling out Infor, she was barraged by user complaints of poor performance, but her colleagues could provide no details other than ‘it takes forever to enter my time sheets and expense reports.’ The product owner had no visibility into which areas of the application were having performance issues, what users were doing within the application when they experienced the slow response, or who was affected other than those who reported the issues.
Step 1: Identify the critical business activities
It sounds obvious, but monitoring the overall availability of an application is one thing, but monitoring the application’s performance is another. An app doesn’t have to be unavailable to cause a major impact to workforce productivity. What counts is the end user’s experience as they use the app to execute the business activities for which they are responsible. Since the firm had been a long time user of Aternity, the firm’s Aternity administrator reached out to me for help in addressing the problem.
So, our first step was to identify business critical activities. Together with the Infor product owner, we mapped the user’s interaction with the Infor app to key workflows, such as ‘login,’ ‘edit report,’ ‘save report,’ ‘add attendees,’ ‘create new time sheet,’ etc. This is done using ‘Signatures,’ part of the functionality within Aternity’s Business Activity Analytics. All in all, we created 24 different business activities that represented the top workflows for which the Infor app was used.
Step 2: Determine what constitutes acceptable performance for the activities
This step involves another basic concept. No application vendor or SaaS provider can tell a business what constitutes acceptable performance. Only the business can determine the appropriate response time metrics for business transactions. Acceptable performance must be based on what the end users expect, which is often driven by customer expectations. Business-driven thresholds for business activities are the foundation of a meaningful, application performance-based SLA.
In this case, the Infor product owner identified the thresholds for each activity based on reasonable expected response times. For example, the threshold for the ‘login’ activity was set to 6 seconds.
Step 3: Monitor SLA compliance across the enterprise
With thresholds set for the 24 key Infor business activities, the firm could now proactively monitor the end user experience of the workforce.
As the SLA dashboard above shows, the application was only around 30-40 percent compliant, meaning that over 60 percent of transactions failed to meet the established performance thresholds. Outside of the U.S., performance was even worse.
Step 4: Isolate and resolve the cause of poor performance
Problem resolution for SaaS-delivered applications is a challenge because the enterprise doesn’t have access to the infrastructure on which the application runs. In this case, though, my customer used Aternity to identify a direct, positive correlation between poor response time for certain business activities and the amount of incoming network traffic.
After some investigation, Infor determined that the application was not caching requests properly. As the dashboard below shows, the change they made to the application over the weekend of October 18 improved performance by almost 40%.
The key to application performance SLAs for SaaS – The end user’s perspective
Going beyond the basic availability, incident response time, and penalty policies that many SaaS vendors offer as part of their SLA requires monitoring the application from the end user’s perspective. Even SaaS vendors which share infrastructure performance metrics fail to capture end user experience.
Without this complete picture, enterprises lack information critical to rapid problem resolution. As this customer story shows, using Aternity is one way to get the true understanding of end user experience on which application performance SLAs can be based.
To learn more about Aternity’s unique value and End User Experience Management advantages, read the Aternity whitepaper on the EUEM Landscape.