— By MARK DISCIULLO
The tepid economy is putting pressure on everyone from executives to User Experience (UX) teams to show direct, measurable results. So, I’m often surprised to hear of the many projects that include a UX component to them, yet there isn’t any true, quantifiable success criteria defined for UX. Even more rare, are efforts to baseline the current design experience of an interface or product prior to a relaunch so any newly “defined” success criteria has some context. This is critical information to know so you can quantify whether or not your new designs have truly made improvements compared to past designs. Anything that is done as an organization should have justification – otherwise, why do it?
UX is still being treated as though it’s a very subjective topic to measure. It’s unfortunate that in many cases, success is simply summed up with statements such as “it’s now easier to use”, “it’s better then before” or…my favorite…”we tested 5-7 users and they all said it was more user-friendly.”
“This is not acceptable. Not having a measurement strategy is what keeps UX as a “nice to have” as opposed to being an influential driving force within an organization.”
Without credible UX success measurements, we all risk not being able to quantify our success. Without credible UX success measurements, we are unable to align our efforts to an organization’s business objectives and desired outcomes. This often results in UX efforts becoming very unfocused, undefined and easily changed on a whim. Basically, you’re left having to tell a very subjective story of your UX success or failures, which unfortunately, could lead to you and your team being very exposed.
Why are we not measuring our UX efforts?
Often times stakeholders or clients don’t realize that UX can be quantified and measured. As UX professionals need to take the lead and show them how to connect the dots between how UX (or lack there of) is impacting the bottom-line and how it ties back to the organizations core mission.
We need to be backing our UX efforts with quantifiable stories such as, “We decreased shopping cart abandonment by 30% which lead to 10% higher sales”, “We increased intranet adoption by 55% .”, “We increased overall user satisfaction by 30%.” The list goes on. We need more answers to the “so what” question. The questions that quantify “why” we are conducting any UX efforts in the first place.
So how do we do this?
Imagine if you could clearly define measureable goals for your UX efforts? What if you could clearly connect the dots from your UX efforts to your project’s bottom line metrics or desired business outcomes? What if you could align UX to your organizations primary mission statement? What if along the way you could also glean actionable insights that could be used for continuously evolving the design of your product, website, or application? Talk about maximizing your UX investments! There is a simple methodology for doing this. It really only requires that you commit to it and ensure the results are made available for the stakeholders and team members to make informed UX decisions.
STEP 1 – Decide What to Measure
A sound measurement strategy includes a mix of both qualitative and quantitative metrics. These metrics are gathered over time so that they aren’t presented as snapshots of a single point in time but presented as trends. (see examples below!)
The measurements or KPI’s (key performance indicators) can be derived from a variety of sources such as user research (usability testing, surveys, structured interviews, heuristic evaluations, card sorting, etc) and/or analytics (i.e. data from your Webtrends, Google Analytics, Coremetrics, etc), But again, since this is not only limited to measuring user interfaces and websites, also consider sales data, lead generation stats, social media metrics, etc…basically any user touchpoint that can be considered a measurement of a “desired outcome that aligns to a business goal.”
Marko Hurst has modeled an excellent approach for aligning business goals to desired UX outcomes. In fact, he’s created a useful “Goals-to-Outcomes” worksheet that helps facilitate this. I’ve followed this approach with much success. It involves defining and mapping the following “Business Goals>User Goals>Tactics>Desired Outcomes.” This approach yields very focus and accurate results for defining specific Key Performance Indicators (KPIs) for UX efforts.
When specifically measuring the performance of a user interface, the process involves designing a test that defines representational tasks users are most likely performing, and/or tasks that the business or client would desire them to be performing. The foundational set of metrics (with examples below showing trending) to measure the “usability” of a product or website are typically:
Motion Failure Rate – I prefer to establish an ideal or “happy path” and compare the participants actual results against that. This is more specifically defined as the average amount of clicks, motions, or gestures (basically any deviations from a predefined “happy path” or process) that were errors. This is more insightful then the typical “# of Click” metric used by many UX practitioners. I emphasis this because high numbers in this area are not always a bad thing from a user perspective, as long as they are successfully following the most optimized path and achieving their goals.
Satisfaction – Measuring satisfaction is often overlooked, yet is very simple to measure, and extremely important in terms of UX. There are various methods available to gather this rating, but the one I most prefer is using a simple three (3) question post-task survey. The participant is asked to complete this immediately following each task being tested. It’s made up of three components (or questions) that gather user ratings for overall ease of use, satisfaction, and “perceived” amount of time. The three (3) ratings are then averaged to create a combined score.
When reporting our UX findings, I find it very useful to segment the behaviors and findings of the top most satisfied participants and comparing that to the bottom least satisfied.
In addition to establishing metrics that mainly measure “usability” consider inlcuding metrics that also measure “usefulness.” Jeff Sauro outline a straight-forward approach to consider for going beyond standard usability metrics in his blog titled “Measuring Usefulness” He mentions incorporating metrics that gather insights from users `that measure how useful this product is for accomplishing their goals. This is very important, since no matter how usable something is, if it’s not of any real value to the user…this is all a moot point!
Measuring Social Media
In regards to social media products and applications, this has brought whole new layer of things to measure such as amplification, awareness, conversation, engagement, reach, participation, etc. Before getting caught up in the fire hose of social media metrics, again, make sure you first established clear goals and desired outcomes. (As mentioned earlier using the “Goals to Outcomes” worksheet.) Map these to your business bottom-line. What you truly need to measure will reveal itself.
I have found the following blog post from the well respected web analytics guru, Avinash Kaushik, to be a great place to provide perspective on how you should approach measuring your Social Media efforts: Best Social Media Metrics: Conversation, Amplification, Applause, Economic Value
But again, at the core of any UX research, always plan to talk with to your intended audience to gain insights on the user satisfaction, usability and usefulness of your current or planned social media experience.
STEP 2 – Design your measurement test approach – Make it repeatable
Once you have defined what needs to be measured, you will need to design your test protocol. Often times called a “moderators script” or “approach brief,” this maps out the details of what you will be measuring, the participants you plan to recruit, the activities and the tasks you will have them perform (user tests) as well as establishes the duration and pace of the study. One hour sessions are most ideal, but I’ve conducted successful 1/2 and 90 minute sessions with equal amounts of success depending on the context of the study. Always make sure you secure a well-versed moderator who can run a successful “structured” study. A lot is riding on the moderator being able to conduct these sessions properly…and still keep the participants focused and engaged.
Also, ensure your test approach can be repeatable! Your test approach can not be considered a one time testing effort. You should always be focused on establishing a measurement “strategy” as opposed to a measurement “snapshot.”
STEP 3 – Baseline the Current Experience
Start measuring now! Measure what you can now to start gathering intel on your current experience. If you are performing an redesign and seeking baseline metrics to later compare your post launch efforts, ensure what you measure now will be available to measure again in your newly redesigned experience.
STEP 4 – Measure the New Experience
After the launch of your product, follow up with the same study and data gathering methods you employed during the baseline exercise. Pay special attention to having you or your team recreate the baseline tests in the same manner as originally implemented. New questions, tasks, and data often times need to be added into the mix to account for new features that didn’t exist in the baselined design. If possible, and only if you can create a meaningful story, map this new information to previous measures. Otherwise, this new information has to be called out in your report as “newly acquired” and these new metrics will be used to define new baselines for future measurement efforts.
STEP 5 – Plan How You Will Visualize Your Data – Consider the story you’ll be telling over time
I’ll admit, I am terrible with numbers, but has not hindered the success I’ve had employing measurement strategies. I’m a visual person, and this has helped more then I could imagine in this areas of UX and measurement. Executives have become numb to copious amounts of data. What actually provides meaning and drives change is the ability to share a story. Using metrics support a story and provides credibility. Thoughtful visualizations maximizes the impact of the data.
So, at the early parts of any measurement efforts, it’s also wise to consider how this information will be visually represented. It’s a key part of how the story will be told. If I’m collecting tons of data that can’t be distilled down to simple visual representations, this is usually a key indicator that there isn’t a story here worth reporting on.
Finally when it comes to reporting, don’t plan to report your metrics as a single set. Different metrics serve the needs of different audiences. Executives need a very different scorecard than the UX team. Typically there are three types of scorecards: the strategic, the operational, and one with daily task-based reporting.
STEP 6 – Establish a Continuous Improvement Program
Use the learning from this initial cycle to tweak the process and refine it so you can repeat it on an established regular basis (monthly, quarterly, per release, annually, etc) whatever is most feasibile for your website or application. Assign UX resources to manage this and report back findings and recommendations on a regular basis.
Why all this effort? User experience trends are constantly evolving. How will you ensure that your product, website or application is keeping up and providing the best experience? If you don’t design your approach to be a continuous process, all you are really doing is creating a measurement “snapshot” a one time view of your UX in a single point in time. Always think trends over time, that is were the real insights reveal themselves.
Too much to handle? – Then start with small tightly controlled experiments
If this is sounding very grandiose and requiring tons of effort, it doesn’t have to be. It can be very easy to start. If it’s not possible to fully dive in to an overall measurement strategy, It is highly recommended that you begin with small incremental “tightly controlled experiments” to get into the measurement mindset. Yo will be surprised how far you get with just a simple structured study, a few credible measurements, and a good story.
The Final Measure:
When the C-Level folks gather at the table to talk business, they aren’t judging success and making decisions because “it just feels like we made an improvement” (or at least the smart ones!) No…they’re discussing important bottom-line metrics, and credible facts to base their decisions and future strategies on. It’s their language. UX needs to be part of that discussion. UX needs to distill their story that speaks that same language and also resonates at that level.
Having a solid UX measurement strategy in place gives UX the credibility to sit at the “big table” and be recognized as a core part of any business.