User Evaluation & Testing

TacTalk Training Documentation Link

The link above will take you to our online training documentation which focuses primarily of the vocabulary used for our application. 

The document is split up into 3 parts:

1. The vocabulary page displaying all the key words and sentence structure.
   
2. An example video of the application being used to annotate a match.
3. A vocabulary quiz which is focused on sentence structure and using the key words.

User testing during COVID-19

Due to the current COVID-19 pandemic we had to adapt the way in which we could test our application. We were unable to attend a live match to evaluate our product working in the targeted environment.

Our approach to user testing during COVID-19 was to use a pre-recorded Gaelic Football match and have our test users annotate over it using the vocabulary learned from the documentation mentioned above.

We organised our user tests to focus on 2 areas of our project:

1. Vocabulary and structure.
2. Fatigue.

For both areas, we followed the test users over a video call going through the documentation. We gave them a brief introduction about the app concept and how to use and structure the vocabulary key words.

We then asked the test user to watch the example video of a match being annotated using the vocabulary structure.

After the example video, we asked the test user to go through the 10 question vocabulary quiz. At the end of the quiz we noted the score and whether or not they have played Gaelic Football before.

To test the vocabulary, we requested the test user to watch a pre-recorded match for 5 minutes while to test the user fatigue, we requested for them to watch the pre-recorded match for 35 minutes (until half-time) using the vocabulary that they had learned from the documentation. We took notes during the recording to see if they following the structure or making any mistakes. We then sent out a questionnaire to receive feedback on their experience.

You can find the questionnaire results and the User Evaluation Document for the vocabulary and fatigue tests below.

Coach & Analyst Interviews

As part of our user evaluation we also conducted two interviews with potential end users of our project. The two people that we interviewed were:

1. Mark Copeland - Sports lecturer in SRC Newry & GAA coach.
   
2. Kevin McGuigan - Leading Sport Scientists & GAA Performance Analysts in All Ireland.

We have linked the notes taken from both of the interviews below.

Voice Recognition Testing

We undertook different tests to measure the accuracy of the voice recognition in our project. The two tests that we trial were:

1. Simulated crowd noise test - Reading the same script increasing the volume each time.
2. Prepared script test - The same script was read by four individuals that differ in gender, ethnicity and background.

The results for both tests are displayed below.