6 Steps for Measuring Success on UX Projects

– By MARK DISCIULLO 

The tepid economy is putting pressure on everyone from executives to User Experience (UX) teams to show direct, measurable results. So, I’m often surprised to hear of the many projects that include a UX component to them, yet there isn’t any true, quantifiable success criteria defined for UX. Even more rare, are efforts to baseline the current design experience of an interface or product prior to a relaunch so any newly “defined” success criteria has some context. This is critical information to know so you can quantify whether or not your new designs have truly made improvements compared to past designs. Anything that is done as an organization should have justification – otherwise, why do it?

Measuring the User Experience

UX is still being treated as though it’s a very subjective topic to measure. It’s unfortunate that in many cases, success is simply summed up with statements such as “it’s now easier to use”, “it’s better then before” or…my favorite…”we tested 5-7 users and they all said it was more user-friendly.”

“This is not acceptable. Not having a measurement strategy is what keeps UX as a “nice to have” as opposed to being an influential driving force within an organization.”

Without credible UX success measurements, we all risk not being able to quantify our success.  Without credible UX success measurements, we are unable to align our efforts to an organization’s business objectives and desired outcomes. This often results in UX efforts becoming very unfocused, undefined and easily changed on a whim. Basically, you’re left having to tell a very subjective story of your UX success or failures, which unfortunately, could lead to you and your team being very exposed.

Why are we not measuring our UX efforts?

Often times stakeholders or clients don’t realize that UX can be quantified and measured. As UX professionals need to take the lead and show them how to connect the dots between how UX (or lack there of) is impacting the bottom-line and how it ties back to the organizations core mission.

We need to be backing our UX efforts with quantifiable stories such as, “We decreased shopping cart abandonment by 30% which lead to 10% higher sales”, “We increased intranet adoption by 55% .”, “We increased overall user satisfaction by 30%.” The list goes on. We need more answers to the “so what” question. The questions that quantify “why” we are conducting any UX efforts in the first place.

Measuring the User Experience (UX)

So how do we do this? 

Imagine if you could clearly define measureable goals for your UX efforts? What if you could clearly connect the dots from your UX efforts to your project’s bottom line metrics or desired business outcomes? What if you could align UX to your organizations primary mission statement? What if along the way you could also glean actionable insights that could be used for continuously evolving the design of your product, website, or application? Talk about maximizing your UX investments! There is a simple methodology for doing this. It really only requires that you commit to it and ensure the results are made available for the stakeholders and team members to make informed UX decisions.

STEP 1 – Decide What to Measure

A sound measurement strategy includes a mix of both qualitative and quantitative metrics. These metrics are gathered over time so that they aren’t presented as snapshots of a single point in time but presented as trends. (see examples below!)

The measurements or KPI’s (key performance indicators) can be derived from a variety of sources such as user research (usability testing, surveys, structured interviews, heuristic evaluations, card sorting, etc) and/or analytics (i.e. data from your Webtrends, Google Analytics, Coremetrics, etc), But again, since this is not only limited to measuring user interfaces and websites, also consider sales data, lead generation stats, social media metrics, etc…basically any user touchpoint that can be considered a measurement of a “desired outcome that aligns to a business goal.”

Marko Hurst has modeled an excellent approach for aligning business goals to desired UX outcomes.  In fact, he’s created a useful “Goals-to-Outcomes” worksheet that helps facilitate this. I’ve followed this approach with much success. It involves defining and mapping the following “Business Goals>User Goals>Tactics>Desired Outcomes.” This approach yields very focus and accurate results for defining specific Key Performance Indicators (KPIs) for UX efforts.

When specifically measuring the performance of a user interface, the process involves designing a test that defines representational tasks users are most likely performing, and/or tasks that the business or client would desire them to be performing. The foundational set of metrics (with examples below showing trending) to measure the “usability” of a product or website are typically:

Success Rate – The number of participants that successfully completed a task
UX Task Success Example

Time-on-Task – The average amount of time it took for participants to complete a task
UX Time on Task Example

# of Errors – The average number of times an error occurred, per participant, while performing a particular task
UX Errors Example

Motion Failure Rate –  I prefer to establish an ideal or “happy path” and compare the participants actual results against that. This is more specifically defined as the average amount of clicks, motions, or gestures (basically any deviations from a predefined “happy path” or process) that were errors. This is more insightful then the typical “# of Click” metric used by many UX practitioners. I emphasis this because high numbers in this area are not always a bad thing from a user perspective, as long as they are successfully following the most optimized path and achieving their goals.
UX Motion Failure Example

Satisfaction – Measuring satisfaction is often overlooked, yet is very simple to measure, and extremely important in terms of UX. There are various methods available to gather this rating, but the one I most prefer is using a simple three (3) question post-task survey. The participant is asked to complete this immediately following each task being tested. It’s made up of three components (or questions) that gather user ratings for overall ease of use, satisfaction, and “perceived” amount of time. The three (3) ratings are then averaged to create a combined score.
UX Satisfaction Rate Example

When reporting our UX findings, I find it very useful to segment the behaviors and findings of the top most satisfied participants and comparing that to the bottom least satisfied.

Measuring “Usefulness”

In addition to establishing metrics that mainly measure “usability” consider inlcuding metrics that also measure “usefulness.” Jeff Sauro outline a straight-forward approach to consider for going beyond standard usability metrics in his blog titled “Measuring Usefulness” He mentions incorporating metrics that gather insights from users `that measure how useful this product is for accomplishing their goals. This is very important, since no matter how usable something is, if it’s not of any real value to the user…this is all a moot point!


Measuring Social Media

In regards to social media products and applications, this has brought whole new layer of things to measure such as amplification, awareness, conversation, engagement, reach, participation, etc. Before getting caught up in the fire hose of social media metrics, again, make sure you first established clear goals and desired outcomes. (As mentioned earlier using the “Goals to  Outcomes” worksheet.) Map these to your business bottom-line. What you truly need to measure will reveal itself.

I have found the following blog post from the well respected web analytics guru, Avinash Kaushik, to be a great place to provide perspective on how you should approach measuring your Social Media efforts: Best Social Media Metrics: Conversation, Amplification, Applause, Economic Value

But again, at the core of any UX research, always plan to talk with to your intended audience to gain insights on the user satisfaction, usability and usefulness of your current or planned social media experience.


STEP 2 – Design your measurement test approach –
Make it repeatable

Once you have defined what needs to be measured, you will need to design your test protocol. Often times called a “moderators script” or “approach brief,” this maps out the details of what you will be measuring, the participants you plan to recruit, the activities and the tasks you will have them perform (user tests) as well as establishes the duration and pace of the study. One hour sessions are most ideal, but I’ve conducted successful 1/2 and 90 minute sessions with equal amounts of success depending on the context of the study. Always make sure you secure a well-versed moderator who can run a successful “structured” study. A lot is riding on the moderator being able to conduct these sessions properly…and still keep the participants focused and engaged.

Also, ensure your test approach can be repeatable! Your test approach can not be considered a one time testing effort. You should always be focused on establishing a measurement “strategy” as opposed to a measurement “snapshot.”

STEP 3 – Baseline the Current Experience

Start measuring now! Measure what you can now to start gathering intel on your current experience. If you are performing an redesign and seeking baseline metrics to later compare your post launch efforts, ensure what you measure now will be available to measure again in your newly redesigned experience.

STEP 4 – Measure the New Experience

After the launch of your product, follow up with the same study and data gathering methods you employed during the baseline exercise. Pay special attention to having you or your team recreate the baseline tests in the same manner as originally implemented. New questions, tasks, and data often times need to be added into the mix to account for new features that didn’t exist in the baselined design. If possible, and only if you can create a meaningful story, map this new information to previous measures. Otherwise, this new information has to be called out in your report as “newly acquired” and these new metrics will be used to define new baselines for future measurement efforts.

STEP 5 – Plan How You Will Visualize Your Data – Consider the story you’ll be telling over time

I’ll admit, I am terrible with numbers, but has not hindered the success I’ve had employing measurement strategies.  I’m a visual person, and this has helped more then I could imagine in this areas of UX and measurement. Executives have become numb to copious amounts of data. What actually provides meaning and drives change is the ability to share a story. Using metrics support a story and provides credibility.  Thoughtful visualizations maximizes the impact of the data.

User Experience KPI Dashboard

Example: User Experience KPI Dashboard

So, at the early parts of any measurement efforts, it’s also wise to consider how this information will be visually represented. It’s a key part of how the story will be told. If I’m collecting tons of data that can’t be distilled down to simple visual representations, this is usually a key indicator that there isn’t a story here worth reporting on.

Finally when it comes to reporting, don’t plan to report your metrics as a single set. Different metrics serve the needs of different audiences. Executives need a very different scorecard than the UX team. Typically there are three types of scorecards: the strategic, the operational, and one with daily task-based reporting.

STEP 6 – Establish a Continuous Improvement Program

Use the learning from this initial cycle to tweak the process and refine it so you can repeat it on an established  regular basis (monthly, quarterly, per release, annually, etc) whatever is most feasibile for your website or application. Assign UX resources to manage this and report back findings and recommendations on a regular basis.

Why all this effort? User experience trends are constantly evolving. How will you ensure that your product, website or application is keeping up and providing the best experience? If you don’t design your approach to be a continuous process, all you are really doing is creating a measurement “snapshot” a one time view of your UX in a single point in time. Always think trends over time, that is were the real insights reveal themselves.

User Experience Continuous Improvement Program

Example: User Experience Continuous Improvement Program

Too much to handle? – Then start with small tightly controlled experiments

If this is sounding very grandiose and requiring tons of effort, it doesn’t have to be. It can be very easy to start. If it’s not possible to fully dive in to an overall measurement strategy, It is highly recommended that you begin with small incremental “tightly controlled experiments” to get into the measurement mindset. Yo will be surprised how far you get with just a simple structured study, a few credible measurements, and a good story.

Again, Avinash Kaushik, provides valuable details on this topic in his blog post “Controlled Experiments – Measuring Incrementally

The Final Measure:

When the C-Level folks gather at the table to talk business, they aren’t judging success and making decisions because “it just feels like we made an improvement” (or at least the smart ones!) No…they’re discussing important bottom-line metrics, and credible facts to base their decisions and future strategies on. It’s their language. UX needs to be part of that discussion. UX needs to distill their story that speaks that same language and also resonates at that level.

Having a solid UX measurement strategy in place gives UX the credibility to sit at the “big table” and be recognized as a core part of any business.

About these ads

12 thoughts on “6 Steps for Measuring Success on UX Projects

  1. Pingback: Usability by rossm - Pearltrees

  2. Pingback: Friday links « w.illi.am/

    • It is a custom designed UI for a Financial Client of mine. Google analytics inspired many of the overall layout/navigation choices, but other then that it’s a custom design that was intended to be implemented via HTML5/CSS3

  3. Hi Mark, great post, really liked your mention of establishing goals and keeping them the central focus, also the fact that usability needs to be complemented by usefulness.

  4. Hi Mark, great post, liked how you mentioned sticking to the goals of the project, also agree that usability needs to be complemented by usefulness.

  5. Pingback: Success with User Experience | Vox

  6. Hi Mark, thanks for this post and all the useful information! Do you have any advice for new products that offer completely new system / functionality (not a redesign). Can you understand the baseline? If not, how do you show the success of a new product with metrics?

    • Thanks for your comments Evgenia. For a scenario like what you described where all/most functionality is changing drastically, I would recommend relying on satisfaction ratings/metrics for benchmarking. From a UX perspective, its definitely a constant you can always measure from phase to phase no matter how much things change. In cases such as this, satisfaction actually becomes a very powerful metric. Also, depending on the type of product you are redesigning you can always lean on some basic business metrics (total revenues/conversions, productivity, adoption) to get some barometer of UX success/failure early on when there isn’t much UX benchmarking to rely on from the previous design. With the business metrics can at least make some valid assumptions that if you are meeting/exceeding key business goals you (hopefully) haven’t diminished the user experience. …but still, in the world of your new design, you will want to start benchmarking right away so you can begin the process of gathering insights to continue evolving your design. Hope that helps, feel free to reach out if you have any additional questions.

  7. Pingback: A importância de definir KPIs para o projeto (e voltar a eles a cada nova decisão de design) | Arquitetura de Informação

  8. Pingback: Interview with Zeke Franco |

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s