Y4 Team Project Technical Journal

What has been done

• Optimized interactions between the parsing engine and the database
• Merge the data and functions of active_game documents to game documents on the database
• Additional game stats returned by the updateGame API endpoint

After a multitude of user testing, we realized our current parsing engine is making too much connection to the database, sometimes causing us to reach our 500 connection quota after more than 10 minutes of using. I resolved this problem by removing excessive connections to the database during the parsing process by the parsing engine, and reduce the number of database interactions to 2 times every 15 seconds. It is estimated that during a 60 minute game, the parsing engine will make at least 240 connections to the database. We can further reduce the amount of connections to 120 connections per game by making the android application to call updateGame every 30 seconds instead of 15 seconds.

 For the purpose of reducing database interactions with the parsing engine, and also simplify document structures, we decided to merge the functionality and structure of active_game document and game document together.

Following the suggestions of GAA personals after we showed our app demo, we also added in multiple statistic types which can be processed by an analyst:

• Poessesion: Number of Home team passes divided by total passes
• Pass completion: Number of completed passes by total passes
• Shot conversion Rate: Divide number of goals by number of shots then multiply by 100
• Kickouts won: Kickouts that did not resulted in a turnover
• Zone with most shots: the zone which had the most shots
• Zone with most kickouts:the zone which had the most kickouts

What has been done

• New Tutorial Page
• New Audio Recording Method

Finished user testing

• 5 minute background noise test
• 2 minute accent test
• 10  minute user vocab test
• 30 minute user fatigue test

Currently, our Android App uses a single WaveRecorder to record user audio, to keep recording for user input, the recorder stops the recording process every 15 second, uploads the audio to our speech to text service while also starting a new recording process, this is put on a loop until the user stops recording.

The main problem with this method is the potential for a single word to be split into two files, resulting in important keywords being incomprehensible by the speech to text service.

To counter this problem, I have created a new branch of our Android App that uses two recorder, which the application cycles between them during the recording process. Both of the recorder's start and stop recording process interval are set in a way that the audio will always have a 2 second padding of the previous audio file, this will minimizes the chance of individual text being separated in two files, save for really long words that will take more than 2 seconds to pronounce. 

What if the same text in the 2 second margin is output by two files?

An additional server function will be added to check using the text's timestamps that, if two text are overlapping and if they are the same text. This function will then remove the duplicate words to ensure the accuracy of the CommandParser. 

Addressing the user fatigue

To address the user fatigue, we have discussed several possibilities, and came to the conclusion that an optional "Buddy system" can be used by our user. The idea is to allow two users to annotate the match at the same time - however to not overlap inputs with users, only one user can annotate the match.

For the time being, our "Buddy system" is simply passing the phone device manually to another user when the current user wants to take a break, drink or doing other things. In the future, we would like to handle the "Buddy system" by setting up an extra timer to remind the user to switch with his "buddy".

What has been done

• Received feed back from GAA personnel on stats and application operation
•  Change of direction for the application development to focus more on user testing

After the last meeting, our team has decided to shift our project to focus on user testing, a few tests has been formulated to according to the question of our panel member.

Will users be able to learn these keywords?

A tutorial page and a quiz page has been implemented for the user to go through a short training on the use of the keywords. We have also prepared a 10 minute test after the training session to test if users are able to use the keywords to annotate the match. The criteria for assessing user training is the time spend on the tutorial page understanding the instructions, the score of the quiz, and the accuracy of the annotation result.

Will users be able to use the app for a long time, without compromising the accuracy of the annotation?

A 30 minute test has been formulated, where the user will use the application as long as possible. Criteria to assess user fatigues are the accuracy of their annotation on the application.

Will users be able to use the app in a noisy environment?

We decided to use Youtube to simulate stadium crowd noise while the user is using the app to determine the influence of background noise on the application.

What has been done

• New utility endpoint to collect voice recording of key words to identify common (here)
• In addition to the utility function, an endpoint to display the table of key words voice result, this is use to identify common voice to text mistake, and allow us to adjust the text comparison parameters to fine tune the text to event results
• The Command Parser now creates a new possession (an array of events) every time a different team gets hold onto the ball
• Game stats such as point, goals, shots and kick pass are now tallied and return as a response, allowing our android app to update game stats every 5 second

Quality of life changes

• Separation of server end points into multiple JS file for better source management and readability.
• An endpoint to clear game events and input list object, as well as resetting the game to it's initial stage, this is used by the developers to reset game to it's initial state so that they can test the game over and over again, before this implemented, we need to reset the game manually in the database.

What has been done

• Connecting Voice to Text Service to Server
• Complete Overhaul of the Command Parsing System
• API Endpoint for user login and registration

In this sprint, I have connected the Google voice to text services to the server. Every time an audio file is uploaded to Google Storage Service, a cloud function is called to process this audio file to text. The audio file submitted by the android device is recorded and renames in the format of [game_object_id]_[order_number], the game_object_id is the reference of the game which is being recorded in the database while the order_number is use to indicate the order of the audio file. By putting these information in the audio file name, we can transport additional information to Google Service without the need of adding another call to send the order number and game object id. After the audio file is processed, the audio file's name is also parsed so order number and game object id can be extracted, it is then in the cloud function the processed text, order number and game id is send to the server to be uploaded to the database.

For the parsing function, in V1 the function simply takes a full string and outputs a full event, I have reworked the function to take into account of  time delay in processing from voice to text and audio files being potentially being cut mid sentence.  This function is called every 5 second by the android application until all the audio file finished processing and all it's text is parsed.

Advantages of Parsing Function V2

• Will Wait for audio order to be corrected before parsing
• Tolerance for delay from Google Service
• Audio file being cut mid-sentence can still be processed as a whole event
• User can gradually add information to the event instead of saying the whole event in one go
• Will still run after end of match since audio file might still be converting to text after match ends

Command Parsing Engine Flowchart: V1 vs V2

Summary

• Transition to Express JS for API end point
• Created documentation html page for front end developers
• Optimization to several functions

This week, as per Francis's suggestion, I have transitioned the API end point of our node JS server to using Express JS. 

For the convenience of Thomas, Shikui and Josh, our front-end developers, I have also implemented an index page at the root of our API endpoint.

I have optimized several functions, creating a game will now return the created game's unique object id in the database, which can be stored during a session and further update game events for that game, using the unique id as a reference.

The same thing has also been done to the create game event functions, now instead of requiring to fill up all the values, they only required the text input, time stamp and game id to update the game events by utilizing the command parser.

Naming conventions and function naming have also been centralized to improve readability.

Summary

• Created create and delete function
• Added teams and users data structure to database
• Need to improve data security in the future once app functionalities are fully implemented
• Some solutions for the command parser

This week, I have considered other data structures required for the service. I have added teams and users into the database, the users are the representation of our users of the applications, they have a username, email and password.

Teams are the representation of the teams managed by the users, they have a list of users who is managing the team - that can read, alter or add data for teams and also the games they participated in. They also have a list of team members.

For now, everything is stored in plain text, needless to say this is a bad practice and a potential endangerment to user data security. After the minimum functionalities of the application is implemented, my first and foremost concern should be the data security of the users that are stored and transmitted.

As for the command parser, I have added more key words into the library for the command parser to read, as well as implemented some stop gap measures to ensure the overlapping key words does not affect one another when being read by the command parser.

Summary

• Prototyped command parsing function that converts text input to game events
• Identified potential problems in key words
• Considering options to further improve the command parsing function

This week, I have create the initial prototype for the command parsing function on node js. The purpose of this function is to take a line of text, the time stamp of when this text is recorded and optionally the previous event to generate a possession event json object using the schema suggested by Kevin, our instructor and adviser on this project.

The initial command parsing function utilizes three array of numeric event id, position id, and outcome id each indicating the respective aspect of the event, each id, is also associated with a range of keyword. For example, the event id 1 represents the kick out event, it is associated with keywords such as "kick out" and "kickout", the command parsing function identifies these keywords in the provided text and returns the associated id for each field.

The reason to use a range of keywords is minimize the chance of typo or text written in a different way causing the command parsing function to not recognize the keywords. 

The order of the keywords to be identified is also arrange in a way so that similar terms won't be identified when not supposed to, for example: for the keyword "free shot" and "shot", "free shot" is prioritized in the function to be check first. This is because if the command text provided is "free shot player 21", "free shot" will be identified first instead of "shot" in the provided text.

For extracting the player number information within the text provided to the function, a regex is used to to extract the player number using the keyword "player". Further more, I used a npm module call words-to-numbers to convert potential texts into their numeric form, such as from "player twenty-two" to "player 22".

The next problem is the fact that some of the keywords overlaps with other keywords while having different meaning and context, I will need to look into this problem more next week and try to come up with a solution.

Summary

• Talked to Instructor for data structures and user-to-application workflow
• Populated MongoDB with mock up data according to the data structure suggested by Kevin
• Continue development of the Node JS server

We have talked to Kevin, our instructor and advisor on this project about the data structure of our game event which records player performance and events during a GAA game.

We have decided to use a JSON structure, since it is the standard practice for transmitting information and also fast. The structure also works well in a MongoDB environment.

Kevin talked to us about using ids to represent events, position, outcome and the player number in an event, he provided us a document explaining what each number represents as well as sample data for us to look on.

I populated the MongoDB database with the sample data, I have also created insert functions in the node js server to upload new game event onto the database using the provided parameter.

The next task requires me to create a function that will generate the events based on the text provided to the server, which I can then upload to the database.

Summary

• Considering different hosting services, decided to use Heroku during development for it's speed in deployment and simplicity
• Tested REST API system using mock up functions and data
• Considering different authentication methods to control access

This week, I have decided on using Heroku as the hosting service for our node js server. The primary reason for this choice is due to the simplicity of the set up process, compared to google cloud and AWS cloud service, we have set up our server on Heroku simply by connecting it to our Github repo where our source code for the server is. 

A mock up function is also created to test if the node js server on Heroku is able to connect to our database hosted on MongoDB, then sends the data to our simple webpage client. The initial test was successful confirming the connectivity between the database, server and the client application using REST API. 

The only thing left to do on the server is to use API keys to authenticate users, but for now the server will be enough for the purpose of development.

Summary

• Decided project for our Y4 Team project
• Identified important aspect of the system and potential problems
• Considering database technology for our cloud service, selected MongoDB as our database system

We have decided to choose the Voice command tactical annotation system for sports team as our team project.

The idea is to produce a mobile application to allow user to input player performance statistics using voice command, the voice input is then processed either on the phone or sent to a server to be processed and converted into statistic format, allowing user to record and refine the data and analyze team performance for better strategic planning and training.

For this project, we have to consider 4 aspect of the system.

The main mobile application that records user input, the web application that allows user to view, edit and analyze the collected data, the database to store the information as well as the server that facilitates traffics among all the system.

The most important part of the project is the voice recognition system that will allow users to perform voice input. Thomas was able to put up a voice to text application demo on android Java.

For the database aspect of the service that I'm responsible of, I have compared MySQL and MongoDB, ultimately I choose to use MongoDB. The reason behind this is because MongoDB stores the document in a JSON format, this will allow us to easily transport data between different aspect of system while MySQL requires us to convert the pulled data into JSON format before transport. MongoDB is also NO-SQL database, this is faster than MySQL and more compatible with the cloud server, there is also MongoDB Atlas which allows us to quickly set up a MongoDB server online for free. MongoDB, being structure less also allows a more flexible database schema during development.