tl;dr: We did our first remote user test in our very own app. It worked, people think it is a better app for informal remote meetings than other existing ones (yay) and many new ideas and feature requests did pop up. And bugs. In this article, we share our findings and some notes regarding our research set-up.
🎯 People get the idea, they want to use it. There are so many possible use cases, that we need to prioritize.
🙀 Onboarding needs to improve significantly. Users are automatically located in one corner within the app once they join. So it might happen, that users find themselves outside the audible sound radius of others. This was perceived as a tech defect and users started to reboot, re-connect headphones, etc. instead of moving around and exploring the spatial sound.
🐣 For the next test, we need to recruit a new group of test users, who do not know the project yet.
Besides these guiding questions for our test of course we still wanted to get answers on a meta-level. As we did not really find a way to explicitly get answers, we wrote them down for us, but did not ask them directly:
The informal test of the test
As described, this was the first time we actually did show our app to others. 1 week before we started to use our app for our own meetings, but never with more than 5 people. Knowing, that more users equal potential bandwidth issues in jitsi, we decided to host a regular informal remote after-work-beer in our app. This session turned out to be a lot of fun and helped us to spot some issues to take into account during the formally planned user-test session.
The group situation
Six users were invited to join our app one after another. This set-up was designed to enable the observation of how users arrive, how they explore, and if they discover the interaction pattern Drag and Drop to move around.
This was obstructed by a technical issue, that caused some new users to NOT see or hear other users. The research design and assumption that new users could explore the room, driven by curiosity and the other participants they see, did therefore not work out as planned.
After a (supported) arrival, all users together did explore the app by themselves and quickly started conversations amongst each other or with our team members who were there to moderate and observe.
All test users were invited to join a debrief interview individually in which we wanted to understand the following topics in more detail: