Qualitative and Quantitative Research: Balancing Data Types in UX Research

To achieve seamless and intuitive user experiences, we must heavily rely on research to make informed decisions. Using a combination of qualitative and quantitative research allows us to collect unique insights and triangulate our knowledge to draw patterns and trends around user needs, motivations, and goals. Each approach has several research methods, with their own strengths and weaknesses, and we must strike the right balance of each to draw evidence-based conclusions.

‍

Qualitative Research

Qualitative methods such as interviews, focus groups, and moderated user tests help us to understand the ‘why’ behind user behaviours. The recommended number of participants is around 5-6 to uncover 85% of usability issues. Often if we speak to more than this number of individuals, the research hits a point of saturation and there are diminishing returns, and we stop finding out any new information (Neilson Norman Group, 2021). Of course, if we are conducting a large study and have multiple persona types and we are still finding out new insights we can make the call to extend the number of research participants further. This type of research works best in the exploratory phase when we don’t know what to expect and require more understanding around the problem space before thinking about a solution.

‍

Strengths of Qualitative Research:

• With this type of research, we can dig deep, and build an understanding of what users are thinking and feeling when they use our product.
• It provides us with a lot of rich in-depth information about user needs, motivations, and goals.

‍

Weaknesses of Qualitative Research:

• It is a hands-on and time-consuming method of data collection, spanning across days or weeks of the researcher's time.
• Participants are often aware they are being researched, which could change their behaviour and how they interact with the product or what they tell us.
• It can often be fairly difficult to systematically analyse as the opinions collected are subjective and open to interpretation by the researcher, resulting in bias.

‍

Project example: On a recent project we aimed to understand the market need for a new cyber security training product. We had access to customers, internal stakeholders, and cyber security professionals and ran over 30 hours of interviews with these individuals. Some common themes were were identified during the interviews, but a lot of the insights we uncovered were subjective and based on how the individuals saw the problem space from their point of view. This made it difficult to tag the research into well-defined categories. To overcome this, it was important to generate research hypothesis and key questions at the start of the project. We mapped the insights across each interview back to the research questions which helped to streamline the analysis and informed our recommendations for the product.

‍

Quantitative Research

Quantitative research is much more focused on numbers and scale to understand patterns and trends in user behaviour. Often quantitative research uses tools such as surveys, behaviour analysis, unmoderated testing, and A/B testing. The number of participants usually ranges from around 20-40 as we want the information to be statistically significant, but also don’t want to waste time and resources (Neilson Norman Group, 2021). This type of research often works best if we already have a design concept we want to validate. For example what is the task success rate or did the user navigate through the system more efficiently using Option A or Option B.

‍

Strengths of Quantitative Research:

• It allows us to gather data and insights across a larger scale of participants and is often objective which helps to validate ideas and minimise bias.
• It often allows us to test products in a live environment, being as close to the natural way a user would be interacting with a product as can be.
• This type of research is usually less time-consuming, involving some initial investment in setting up the study, then we can let it run its course and analyse the findings.

‍

Weaknesses of Quantitative Research:

• As we are not talking to individuals we can’t dig any deeper into why a user may have found a certain area of the product difficult to use. Therefore we have to make assumptions based on what we think the user is trying to achieve which could be incorrect.
• Often using methods such as surveys, individuals tend to skip through these very quickly and the answers we collect aren’t as insightful as they could be.

‍

Project example: On a recent project I worked on we were using Hotjar to analyse an e-commerce checkout flow. We found that it was taking users a long time to complete their payment, and they were navigating back and forth throughout the check-out and back to the product screen looking for information that might help them proceed. Without speaking to these individuals, we had no way of knowing the exact cause of the confusion and had to make educated assumptions based on how many individuals were navigating in the same type of way and draw recommendations for improving the product based on these assumptions. Going forward, we will conduct usability tests of the updated functionality to ensure we have correctly identified user frustrations, and pivot accordingly.

‍

‍

To select the appropriate research methods, we must look at the bigger picture and the goals we are aiming to achieve. For example, if we are researching a new product idea this is high risk with many unknowns so we would want to use a combination of qualitative and quantitative research methods to understand the full picture before investing. However, for an existing product with an established customer base, we might only be researching an updated feature and a single qualitative research method might suffice. It is important to remember the number of research methods at our disposal and use them in a way where they will complement each other. As an example of using both, we might ask a large number of users what features they prefer in a survey and then bring these feature ideas to life through prototyping. We can put the prototype in front of a smaller group of users to understand if the features resonate and add value before going into development.

‍

Balancing qualitative and quantitative research is key to a holistic understanding of user needs. By leveraging both approaches, we can uncover deep insights, validate hypotheses, and make informed design decisions that lead to seamless user experiences. Selecting a research approach depends on the time, resources, and budget of our team. However, if we can, we should leverage the strengths of both to create a comprehensive user-centred research plan to gain a well-rounded understanding of user needs and expectations.