Usability test at The Royal Library

October 2023

The Royal Danish Library

UX Research Intern

How to improve usability testing based on experience

This research case aimed to reflect on and improve the planning and execution of usability tests based on practical experience from collecting data from the Roskilde Festival, History students at Copenhagen University and a game industry event named SpilBar56.

Problem

A problem during (and before) my internship was that the research studies were executed without careful planning. There were too many methods and questions to keep an overview. This resulted in poor data consistency, making it difficult to see patterns. When looking at the charts or findings that I wanted to present to the Department of Game Designers, many results could not be clearly explained regarding which optimised changes needed to be implemented in the game.

Roskilde Festival

My first day of the internship was at the Roskilde Festival (Jun. 2023), where I was not involved in the planning process. The team behind the planning phase was a former UX Researcher and my then-current colleague Danny, who lives in the UK, I was the only researcher to execute with three other colleagues, and I conducted user research based on “trial and error”. It resulted in a week of gathering data from 92 participants. However, it lacked consistency because we forgot to ask the same questions for each user because of the amount of 7 methods and 46 interview questions, making it difficult to recall each.

History students

GAMUCATEX's history-related game had not been focusing on participants interested in history. Therefore, we designed a survey and asked Copenhagen University students majoring in History (Sep. 2023). They did not try to play our game but were asked about their game preferences. Due to having to design the survey in a short time slot, the respondents misunderstood some questions.

Spilbar

We conducted user-based testing on the game (Sep. 2023) and a collection of representative users trying out a range of representative activities. Afterwards, the participants filled out a survey, but it took a long time, which gathered a queue of other users who wanted to fill it out but had to wait.

Spilbar

Roskilde Festival

Purpose

This project aimed to use these past user studies to reflect on and learn from. Then, I conducted a more well-planned usability test at The Royal Library at a game event on October 29th. Where the attendees could, for instance, see the TV show Troldspejlet live with the host Jakob Stegelmann.

Methodologies

Focused Interview

To be able to plan the usability test at the game event hosted by The Royal Library, I used a focused interview with my UX team members, Danny and Vasileia. The reason why the focused interview is fitting is because the topics of the interview are something they each, including me, the moderator, have been involved in (See Figure below). I wanted to discuss what we have learned from past studies and how we should tackle this future study.

Goal

The goal of this research was to get feedback on the needs of various departments. This has not been a common goal in the past, but since every department at GAMUCATEX has an influence on the game’s development like a supply chain, my team and I wanted to include them. We also assumed that it would not be surprising to us if there were some questions other departments need answers to since previous findings from data collection were difficult to pinpoint or elaborate on due to a lack of consistency and follow-up questions.

Interview guide

Fewer questions

When reflecting on the Roskilde Festival research, Danny suggested that to gain more precise insight, we need to ask follow-up questions. User test methods consisted of a semi-structured interview with 13 questions designed after the request of different team departments of the company. My UX team and I wanted to gather data based on a wish list from our colleagues among game designers, developers, artists, producers, historians etc. And we wanted to focus on fewer questions, so there was time for follow-up questions and further elaboration. They were asked before, during and after the usability test.

Excluding note-taking

Vasileia suggested that to make it easier to transcribe during the analysing process, we should be active as researchers by keeping notes. A valid point. However, what I then afterwards argued against is the overuse of methods. It would only be Vasielia and me conducting, so using gameplay tests, observation, maybe note-taking of pain points, and audio interviews would be too much.

Excluding survey

Interviews and surveys serve different purposes. The questionnaire and survey during the Roskilde Festival had demographic questions. Qualitative interviews help to understand why users behave a certain way and how some participants use the product. Still, they cannot precisely determine the frequency of certain user traits. The survey is the most effective way to learn about the demographics. Therefore, there were no demographic questions for this user study.

Formulation of questions

We could have formulated some questions differently regarding the history students' survey. We phrased a question: “How much did you spend in a year on history-related games? (Add currency)”. It was an exploratory goal to see if there was a relationship between their game preferences and spending habits. Many participants misunderstood how much time they spent, while the question was about how much money they spent. We aimed to formulate the questions to be shorter and more precise and add follow up questions for this study based on our reflections.

Participant sorting system

When it comes to sorting data collection, looking back at the Roskilde Festival, my team and I lost track of the participant structuring system. We often asked each other: “Which participant does this screen-recorded video belong to?”. When conducting a usability test, it is in the researcher's best interest to say in the microphone which participant is about to be interviewed. This could be done by saying, “This is Participant 4”. Vasileia and I decided to be divided up in a numbering and alphabetical system. So we planned for Vasileia to label her participants as, e.g. number “1”, and I did the same but with an alphabet, e.g. “A”.

This was so helpful to us that we implemented it for The Roskilde Festival 2024 study.

Considering all Stakeholders

Team departments

My team and I have planned the event with different stakeholders in mind. We have included the questions of the other team departments.

Participants

We assumed that our participants would be parents with children, and therefore, we did not want to incorporate a survey for this event. We did not want the chosen methods to be too time-consuming for them while they simultaneously have to keep an eye on their children.

Their children

We also wanted to include some playful activities for the children. We learned from a sprint meeting that the game designers were, at that time, incorporating ideas for the users to customise banners. That is why we printed out a banner template and brought coloured pencils for the kids to design while their parents tested out our game.

Data analysis

We sorted the data collected from 8 participants into folders and transcribed the interviews. Rather than looking for numerical data, our qualitative study sought to comprehend complicated circumstances by analyzing transcribed interviews. Every time we found an answer to our questions during the transcription, we placed a header above it in the Google Docs file to make it easier to find it. Once that task was done, we created a new and blank Docs file and placed the answers from each participant under each of the 13 questions. That made it easier to see if there was a pattern. Those patterns made it easier to analyse some findings. Some findings were based on assumptions due to, for instance, nonverbal cues, but we made it as clear as possible during the presentation when we made an assumption. *For ethical reasons, I will not present the findings.

Presenting insights

Gains:

The other departments saw value in insight within:

Participants understanding of deck categories
Participants feeling confused
Participant’s view on engagement
Participants wish for account creation

Pains:

Only received 15 minutes
Difficult to present qualitative data
Too much text on the slides

Reflections

The day started with Vasileia and I showing up a half hour earlier to set up the testing booth. We placed the two laptops on each table with a gap in between to reduce distractions and noise. However, once the team members from other departments arrived, they placed the tables together. There was a big TV screen next to the two tables, and they decided to connect one of the laptops so attendees passing by could watch the gameplay conducted by a participant. It is a great idea for gaining attention. The issue is that the first question we wanted to ask was regarding the participant's first impression of the game, which was only by looking at the Main Menu. In the future, it would be wise for me to articulate better the reasons behind my choices for the other team members. The positive side of this user study was the environment with less noise and distractions compared to previous projects, and the participants had more time and patience to play and test the game. The longer gameplay takes, the more cognitive flow will be interrupted during the participant's “think aloud”, so I believe it was positive that they had time to think and reach.

It was a pleasure to conduct user gameplay tests with my colleague UX Designer Vasileia Kakavou. We did it in style, dressed as Vikings!

I met some cool and inspiring game creators, namely Søren and Andreas from Triple Topping Games. It's always a pleasure to meet people with creative mindsets.

Quote

"She excels in planning meetings and fostering productive discussions about tasks and priorities. Together, we have conducted and designed numerous user testing sessions and analyses, where her insights and organizational skills were instrumental in our success."

- Vasileia Kakavou - UX Designer at GAMUCATEX