A better user experience can lead to increased satisfaction, more loyal customers, and ultimately more revenue.
But what does it take to deliver a better user experience? Will applying one of the many methods, or hiring the right UX designer or researcher suffice?
There is a general sense in the UX community that a mature UX organization will lead to a better user experience.
This sentiment is well founded as there is a history of assessing maturity for software organizations. The Capability Maturity Model (CMM)— one of the earliest maturity models — was originally developed for the U.S. military as an objective tool to assess the ability of software contractors to get a job done.
A mature organization under the CMM was deemed qualified to deliver effective software (or at least deliver on the contract more successfully). The CMM is based on assessing processes and documentation, and is also the basis for other quality programs such as TQM and Six Sigma.
While UX is related to software development, it’s certainly not the same thing. To understand how much a mature organization may lead to better user experiences, UX needs its own model for measuring maturity.
History of UX Maturity Modeling
There is surprisingly little data about what factors are necessary for “mature” UX, how UX maturity (or lack thereof) affects a company and the success of its products, and even what the levels of maturity should be.
A number of models attempt to categorize and define UX maturity (e.g., Nielsen, 2006; Schaffer, 2004). While there are some differences in the names and breakdowns given to these models, in general they consist of 5 to 7 levels that progress in a pattern similar to Jonathan Earthy’s pioneering “Organisational Human Centeredness Scale” [pdf] (which goes from “unrecognized” to “institutionalized”) and similar to the CMM. The levels are:
- Unrecognized
- Recognized
- Considered
- Implemented
- Integrated
- Institutionalized
These categories have some consensus about global indicators of maturity, such as having sufficient and dedicated budgets or resources, employing the right research methods, processes for when and how user-centered processes are involved, and company culture and attitude toward UX.
While the models have broad similarities, they offer few details about what characterizes mature UX. For example, what are the right methods, what is the “best” ratio of UX designers to developers, should teams be centralized or distributed, and does any of this really matter?
Limitations of Existing Models
It’s not clear how these models were developed so practitioners are left to assume that authors have largely derived them through their own professional experiences rather than using empirical methods. Existing models have been useful in that they have inspired organizations to evolve and provided guidelines for increasing the focus on the user, but it’s unclear whether they reflect how a UX organization actually affects user experiences. Unsurprisingly, some practitioners are skeptical of the validity of UX maturity models or even worse, see them as simply a sales gimmick.
Validating & Refining UX Maturity Models
To understand the characteristics of and value derived by UX maturity we wanted to take a more empirical approach at MeasuringU. We started with the ideas put forth in the existing maturity models and looked to validate these ideas by surveying a broad sample of the UX industry. A visualization of this process can be seen in Figure 1.
Figure 1: Method for deriving an empirical “tool” for assessing UX maturity.
Building a Tool to Assess & Benchmark UX Maturity
The first phase of developing a UX maturity model consisted of the creation of survey items based on in-depth interviews with senior managers and seasoned UX professionals at several companies including IBM, Autodesk, Capital One, and United Healthcare. Several of these professionals had previously been involved in assessments of organizational UX maturity, although one admitted that the approach was “a little bit like snake oil” with little consistency in measurement and reporting.
The interviewees expressed excitement about our endeavor and provided valuable insights that contributed to the development of our model. In particular, their integration in organizations at various levels of UX helped us refine our terminology and question wording to be understandable to our respondent pool. Our interviews also highlighted the need to assess product and organizational success in addition to UX maturity, as the aspiration of most organizations is not to be “mature” but to be successful.
In addition to interviews with UX experts, we utilized domains and items from other surveys and theoretical models on the state of UX to build our assessment tool, as well as collaborative discussion with our research team.
Ultimately, we developed a survey measuring 11 specific aspects:
- Individual Characteristics
- Organization Characteristics
- UX Staffing
- UX Research Methods
- Leadership and Culture
- UX Integration
- UX Training and Skills
- Product Success Metrics
- Organization Success Metrics
- UX Budget/Resources
- UX Challenges and Future Direction
The validity and reliability of this assessment tool will be continually refined as we test and build our sample. Towards this objective, we released the survey to a non-probability sample of UX professionals using social media and snowball sampling.
We have collected data from over 100 organizations, including data from researchers, UX generalists, manager/directors, and interaction and visual designers.
We will present the preliminary findings at this year’s CHI conference in Denver and release more detailed findings later this year when our analysis is complete.
Summary
How do you measure UX maturity? The short answer is that while many models describe maturity, none of them have been effectively validated to determine what matters. The models generally consist of different stages, with maturity progressing from unrecognized or ad hoc to institutionalized. They also identify common components to assess, including the importance of budgets, methods, processes, and attitudes. We’ve embarked on an empirical validation of the existing models of maturity and built an industry-wide survey to benchmark UX practices and help move the practice of assessing maturity from snake oil to science.
