Driving Inclusion: Start with the User Research

• Annabel Weiner & Rachael Worley
• May 14, 2024
• 8 min read

There is no digital strategy without an accessibility strategy. At some point in almost every person’s life they will develop a type of disability. It could be visible, invisible, temporary or permanent. While there are federal requirements mandating compliance to support accessibility, having a strong strategy in place to ensure you are meeting every customer where they are is the right thing to do. It’s a key part of human-centered and inclusive design. That understanding and level of empathy is interwoven into our strategy and overall Accessibility Practice, through a People with Disabilities (PWD) research program. In honor of Global Accessibility Awareness Day (GAAD) we wanted to share our multi-year award-winning approach – including C Space’s Insights Inclusivity Award – and learnings to help others create meaningful, inclusive experiences.

Setting Up the Practice 

As we grew our accessibility program, we saw that a critical missing piece was conducting user research with people with disabilities (PWD) throughout the product lifecycle. Historically, our Customer Experience (CX) and Accessibility practices were distinctly different. We saw this as an opportunity to partner to expand our participant pool to include PWD in our research practices.  

We recognized there was a lot for these two teams to learn from each other. Members of our accessibility team shadowed experienced CX researchers to learn about creating test plans, test scripts and techniques for moderating research. CX researchers also learned from an accessibility expert about different types of disabilities, assistive technologies, accessibility design considerations and how to adapt test script language to be more inclusive. For each study that included participants with disabilities, we partnered an accessibility expert with a CX researcher to align our testing goals and practices. 

Maximizing Resources 

With limited resources, we need to find ways to prioritize our accessibility research efforts. One approach is to complete a test with all assistive technology users to focus on accessibility and the user experience of PWD. Another approach is to have a participant pool with some participants who use assistive technology and some who do not have disclosed disabilities. About 1 in 4 people in the United States have a disability, some of these are invisible and some are visible, and our goal is to reflect this in our research pool. Both approaches are valuable. The first puts accessibility at the forefront and demonstrates the importance of focusing on users who often navigate technologies in different ways than the people who designed it, reducing the risk of designing with “like-me” bias. The latter ties accessibility into the routine research that the CX team is familiar with running. We often prefer this approach, because it naturally builds accessibility research into the project and shows how testing with a diverse user pool can lead to more innovation.  

Adapting Tools and Testing Methods 

We focus on conducting accessibility research in the early design phase and have developed ways to adapt prototyping tools that aren’t inherently accessible for assistive technology users. These include - 

• Screen Reader Users - We adapted a “Wizard of Oz” method where someone on our team acts as the voice of a screen reader instead of using a real screen reader. 

• Alternative Navigation - We’ve made adaptions where the researcher can share their screen and ask the participant to verbally walk us through their feedback or what step they’d take next.   

Both methods are similar in that they are taking away the need to use a specific prototyping software, and instead relying on communication between the researcher and participant to navigate tasks. While these are not always perfect research methods, they allow us to get feedback from users who are usually excluded from this phase of research.  

A key part of this work is creating inclusive test plans and test scripts that recognize different abilities. We thoughtfully craft and tweak the language for each type of assistive technology user to account for unique differences in navigation methods. One example of this is prompting screen reader users with things they might hear instead of what they might see. While these are new considerations for researchers, testing with PWD is not too complicated or time consuming to add to a research plan. This is a myth we have been working to debunk, because with a little education and training researchers can easily add assistive technology users to their testing pool without much additional prep time. 

Sharing Findings 

While we conduct manual accessibility testing on every project, unfortunately we’re unable to perform user research with assistive technology users on every project. Therefore, we look for different opportunities to share these research insights with the larger UX and digital organization. For example, when we improve a component in our design system from research with PWD, this benefits the accessibility and usability of each instance that component is used across our products. We also, pick out pieces of feedback that could apply more generally to other projects to share with larger teams. Even sharing a quick video clip over Slack or Microsoft Teams has been an effective approach.  

Another strategy has been to host viewing parties and invite teammates to observe research sessions and have a discussion afterwards. These methods of information sharing create dialogue that drives empathy, awareness and improvements in our design and development process. After hearing about research, we observed designers commenting on design changes they decided to make from hearing about this research regardless of if the research was conducted on their specific project or another.  

Evaluating Progress 

To evaluate progress, we utilize various approaches. From a quantitative standpoint, we track the Accessibility Usability Scale (AUS) scores of each task-based test, which is a survey developed by Fable,  an accessibility testing platform powered by people with disabilities, to evaluate the accessibility and usability of a product. This 10-question survey is very similar to the System Usability Scale (SUS). However, while SUS focuses on usability, the AUS score has a focus on the usability for assistive technology users. It has been helpful to track our progress over time, and to use this data to create annual goals for what AUS scores we ask our teams to strive for. We also have a repository of design and development improvements teams have made from this research that help us when we are showing stakeholders the importance of testing with PWD.  

One of our biggest takeaways from researching and co-designing with PWD is that more diversity in our testing pool leads to more innovation and more creative solutions. While we are specifically focused on improving our products for PWD through this research, we have found that this research usually extends to others who do not use assistive technology and improves the experience for everyone. Some of the products this program has impacted include our mobile application, Ally.com, spending buckets tool and our debt management tool.  

Developing a PWD program as a key component of our overall accessibility practice has proven to be a smart investment and a pillar of our commitment to ensuring inclusive experiences designed for everyone. While we’re still learning and evolving our approaches, we’re encouraged by the creativity and collaboration of our teams and the positive impact it's having across our customers.