Designing Metrix

Devising a way to understand audiences like never before

Project description: 

Metrix is a bespoke research tool I designed and built during my PhD to bridge a critical gap in audience studies. While researchers typically rely on either real-time data (such as body position or eye tracking) or post-hoc data (such as surveys or interviews), I found both methods incomplete on their own. Metrix was my answer — a system that captured in-the-moment audience sentiment and linked it directly to what participants reported after the fact.

This project describes the design of the Metrix interface and data collection system that I built. This system supported the core methodology of my doctoral research.

Role:

Designer and developer

Introduction

Studying audience response in live performance is tricky. Real-time data tells us when something happened, but not why. Post-hoc data, like interviews and surveys, tell us what people think happened but not what they actually noticed in the moment.

For my PhD research on audience perception of error, I decided to develop a novel methodology for audience studies that combined both real-time and post-hoc data to get new insights into how audiences think. To do this, I needed a tool that:

• Would let people indicate reactions while watching a live performance
• Would continuously collect time stamps of interactions for each participant
• Required almost no training
• Could function discreetly in a darkened theater and not be distracting
• Could persistently identify each participant across two phases of data collection
• Provided a control interface so I could start and stop data collection

Nothing like this existed, so I designed and built it from scratch. It was called Metrix.

Designing a tool for real-time data collection

For the real time half of this kind of audience study, I needed software that would do the following:

• Interface needs:
  • Provide a simple interface for study participants to provide real-time feedback during music performances
  • Be suitable for use in a darkened performance space without being distracting
  • Be as frictionless as possible for the audience to learn 
• Technical needs:
  • Operate on any smartphone with a browser
  • Continuously collect and store data with a unique identifier for each participant, so I could link their real-time and post-hoc responses later
  • Support up to 75 simultaneous connections (the capacity of the concert theatre)
• Operational needs:
  
  • Provide the investigator (me) a control interface for controlling when the data collection was activated

There were no tools on the market that could do this, which was surprising, considering how ubiquitous smartphones had already become and how little research existed in this real-time space. So, I built one from scratch. 

The public-facing interface

The public-facing interface was designed with the following in mind:

• Provide a simple interface for study participants to provide real-time feedback
  
• Be suitable for use in a darkened performance space without being distracting
  
• Be as frictionless as possible for the audience to learn 
  

A two-button solution

For simplicity I used a two-button interface that only required simple taps. Sliders or text or other complexity would risk distracting attention away from the performance. The two buttons provided a way to indicate to perceived states:

“I’m enjoying this” - indicated by a green side with a ":)" icon
“I heard an error" - indicated by a red side with an "X" icon

I chose colours that were dark enough to be visible on an individual phone, but not visible to others or the performers. The audience learned to use the app in a very short onboarding session with a practice round.

Other than the colours and symbols the interface was totally featureless so there was nothing to look at, except for a subtle colour change on tap to indicate that the tap had been recorded.

Generating identifiers

One of the most important parts of Metrix was that it had to provide a way of aligning real-time and post-hoc data, on a participant-by-participant basis. The way Metrix does this is by generating usernames.

When a participant connects to the site, they're asked to enter the study. Metrix then generates a username from two randomly-selected words. This username is displayed on the screen between performances when participants are asked to fill out written surveys, so they can write their usernames on their survey book.

Design iteration

I was concerned about the red/green interface, wondering if this was creating a binary relationship between the questions, but wanted to provide some easy-to-perceive reminder of which side was for which indication. I was also concerned that the ":)" and "X" indicators were too abstract, but again, removing them seemed wrong. 

The data from the first study indicated that participants often tapped both buttons at once, suggesting they understood that enjoyment and error weren't mutually exclusive. Participants were also asked for feedback on Metrix and there were no indications that it was confusing. However, this was a crucial thing to get right so I held a 20-person focus group and did some heuristic testing.

The focus group indicated that they didn't find the symbols to be confusing, but there was some not-insignificant feedback that the red/green colours were suggesting a binary relationship. Based on this feedback I designed a grey version for the following study.

In the following study I asked participants for feedback. They overwhelmingly indicated that the grey interface made it difficult to rememebr which side was associated with which question, and they found  this confusion distracting, so ultimately I used the red/green interface going forward. I did, however, add a segment to the 10-minute onboarding where I clarified that this interface is asking them two different questions, not asking them to indicate either side of a binary phenomenon.

Meeting technical requirements

The technical requirements were:

• Operate on any mobile phone 
• Continuously collect and store data with a unique identifier for each participant, so I could link their real-time and post-hoc responses later
• Suport up to 75 simultaneous client connections
  
  

Metrix is a web app built using Node.js that runs on a virtual private server (VPS). Metrix’s interface is designed and implemented using HTML, CSS and JavaScript. Metrix communicates with connected devices using web sockets, implemented using the JavaScript library Socket.io.

When the participant's interface is active, they can tap the two buttons. Button tap data is sent from the participant devices to Metrix through asynchronous HTTP POST requests, and Metrix in turn stores that data in a database built with MongoDB, a non-relational database that stores data in JSON-like documents. 

Metrix also has a control interface so the investigator can make participant interfaces active and inactive remotely. This allowed me to start collecting data at the beginning of the performance (and timestamping that start), and made all participant interfaces active. At the end of the performance I could end collection, and paticipant interfaces would display their username for filling out a short written survey. 

Designing the control interface

An important part of Metrix is the control interface, that allows the investigator to make participant devices active (when data is being collected) and inactive (when the participants' usernames are displayed on their devices and no data is collected). 

At the time this was just a web page with a few buttons, but since my doctoral research I've built out this idea significantly to visualise Metrix as a SaaS product that would support this kind of research.

After logging in the investigator is taken to their main dashboard, that lists their upcoming studies and their completed studies. For each study they can adjust settings. These have tooltips to avoid jargon and provide clarity:

The interface for any given study has a set of features, including:

• A live count of the number of connected devices, so the investigator can tell if something is going wrong
• Controls for starting and stopping each segment of a performance (when data is actually being collected)
  
  

There's also always a way to start a practice session. This is a state in which participant interfaces are active and they can tap the buttons but no data is collected. I found this to be really important when doing these studies; getting people comformatable with the interface, how it felt, and how to tap the buttons without looking at them was crucial for consistency across all comparitive cases. 

In the control interface, the rest of the interface is inactive when a segment (or a practice session) is active.

The outcomes

Metrix formed the basis of my PhD research, which included a novel methodology for studying audiences in real time. I was able to carry out multiple studies that not only examined audience perception of live performance, but also investigated the differences between what we see in the moment and what we say about it afterwards.

Publications

Want to read more about the insights that I gained from using Metrix? Here's some publications:

Bin S M A, Bryan-Kinns N, McPherson A. "Skip the pre-concert demo: How technical familiarity and musical style affect audience response." 

Bin S M A, Bryan-Kinns N, McPherson AP. "Hands where we can see them! investigating the impact of gesture size on audience perception."

Bin S M A, Bryan-Kinns N, McPherson AP. "Risky business: Disfluency as a design strategy."

Metrix makes it possible to capture both the moment, and the meaning.