External Partner Link

Organisation

“ABB (ASEA Brown Boveri) is a Swedish-Swiss multinational corporation headquartered in Zürich, Switzerland, operating mainly in robotics, power, heavy electrical equipments, and automation technology areas. It is ranked 286th in The World's Most Admired companies in the Fortune 500 global list of 2016. ABB has been a Global Fortune 500 company for 23 years.

ABB is one of the largest engineering companies as well as one of the largest conglomerates in the world. ABB has operations in around 100 countries, with approximately 132,000 employees in December 2016” - Wikipedia. (https://en.wikipedia.org/wiki/ABB_Group).

We are working with a branch of ABB based in Dundalk. The dundalk branch is heading research and has took interest in the possibilities of augmented reality. Most of the training for engineers of ABB happens in Dundalk.

Key Personnel

Our key contact with ABB is Brian Whelan, a member of the Research and Development department in the Dundalk branch of ABB. Brian has given us access to ABB’s facilities to help give us a taste of what kind of work it is ABB does and how we could augment that workflow with this project. We have received a Microsoft HoloLens development kit from Brian for the development of “ABB Training Day”.

Project

ABB Training Day is an augmented reality project based on teaching employees on how to maintain and repair ABB machines. Engineers need to be trained and our project offers a new and fun way of teaching a field engineer by completing realistic scenarios. ABB will use our product as a learning tool that will help track employee’s ability to complete complex repair jobs. This may develop into more than just an experience and might eventually replace some of the textbook learning because of how informative and involving it can be. From our talks with Brian Whelan, it seems to have potential to replace some of the heavy manuals that usually need to be transported around with engineers when maintaining and repairing machines.  

Support

ABB has provided us with one Microsoft HoloLens head-mounted display (HMD). DkIT has also purchased a second HoloLens and our group will be sharing this device with two other groups. This is our target device for “ABB Training Day”. The HoloLens will allow us to create mixed-reality experiences that utilise both real and virtual objects and interactions.

Along with this cutting-edge development kit ABB has provided highly detailed CAD models for one of it’s ACC units and all of the components found inside it. This is sensitive information which we cannot get through any other source apart from directly through ABB.

A detailed storyboard for one repair job was also provided. This storyboard will be the basis for our training experience. A list of symptoms and linked problems were also provided which will allow us build a basic expert system to be used in “Abb Training Day” as a diagnosis tool.

ABB will be providing us with technical support and information on ABB hardware that will be incorporated into this game. Their feedback will be vital and so will support us by testing builds of our game and giving their personal thoughts and feedback on each build.

Game Description

ABB Training Day is a game based around simulating tasks commonly faced in ABB’s machine maintenance through the use of augmented reality in Microsoft’s Hololens. This game will be light hearted and a fun but educational experience. The game will include scoring systems to create competition and tasks around the maintenance and operation of ABB’s machines.

1.1 Genre

ABB Training Day is an educational game. It could be included as a puzzle game given how the user will be given tasks on how to fix machines. We have taken inspiration from both “Job Simulator” by Owlchemy Labs and “Keep Talking and Nobody Explodes” by Steel Crate Games.

Job Simulator, allows players to perform tasks from common jobs in a wacky humorous way with a robot helper that mostly mocks the player. We enjoy aspects of this but would like to take a similar but more professional and educational approach to our game.

Keep Talking and Nobody Explodes, puts players in time limited scenarios where they must defuse a bomb by completing multiple mechanical themed puzzles. We are taking inspiration from the style of puzzles they use that encourage players to pay attention to detail.

1.2 Audience

Our game is targeted to the employees involved in maintaining or repairing the machines at ABB. These employees are mostly engineers and are focused on working with complicated machinery.  

“Engineering is also an information-intensive field, with research finding that engineers spend 55.8% of their time engaged in various different information behaviours, including 14.2% actively seeking information from other people (7.8%) and information repositories such as documents and databases (6.4%).” - Wikipedia (https://en.wikipedia.org/wiki/Engineer).

Our game will appeal to ABB as it will make the training process more effective and efficient, and it should appeal to engineers as it is informative and fun. Currently our audience will be limited to being an english speaking audience only as we will have a considerable amount of narration from our characters in english.

1.3 User Examples

Our product is being made specifically for one end user, the ABB Field Engineer, however, higher-ups may also be interested in a demonstration to showcase the potential of AR technology applications in the industry. Both users are relatively the same. Both would not be into gaming on average, so it’s important for us to make this a fun experience for non gamers and also use simple controls with a slow introduction. Our ability to show off applications of AR will get engineers to push for the ability to use this technology, and for any potential CTOs to see the value in this technology.

The specialist requirements for these users are: that we train users in fixing machines and that we demonstrate the capability of AR technology in the industry.

ABB Engineer

ABB Chief Technology Officer 

1.4 Goals

ABB Training Day will improve users efficiency with diagnosing, repairing and maintaining ABB machines by giving them experience by guiding them on working with and maintaining said machines.

Goals within the game involve:

• Having an understanding of safety when working on machines by wearing appropriate safety equipment and watching out for various hazards such as electricity and fires.
• Diagnosing common problems with machines.
• Communicating to others what parts are needed.
• Proper procedures when replacing parts within machines.

1.5 Rules

1.5.1 Diagnose machines correctly

The player will use their own knowledge along with the services provided by GUS to diagnose machines. GUS will offer his assistance when asked and will monitor the players progress. There is no need in the trainee fixing something that is not broken. Warnings and other messages will be made apparent through GUS to the trainee in such cases. This can incur a minor penalty to the players final score.

1.5.2 Replace broken/malfunctioning components

The player will get a replacement part for a malfunctioning component. They will remove the broken component and place in the new and working component to fix the machine. The player may receive small penalties to their final score if they have replaced a part incorrectly.

1.5.3 Keep an eye out for hazards

The player will make sure not to handle any live wires and will have to follow correct procedures when replacing parts. Hazards in ABB Training day will be clearly marked to keep the trainee out of any harm. Markings, Signs and other indications will be put into place to make sure that the trainee will be placed in a safe learning environment. Failure to follow these warnings will result in the player having a minor or even potentially a major penalty added to their final score depending on the seriousness of the hazard.

1.6 Specialist I/O Devices

The sole specialist I/O Device we will be using is the Microsoft HoloLens. This cutting edge augmented reality head mounted display can spatially map the real world and project holograms into the real world. You can view and interact with these holograms. The potential user base for this device is vast as companies have started to build software for it to help with industry tasks.

1.7 Player Interaction Techniques/Mechanics

Within ABB Training Day there are two main mechanics, voice commanding GUS and using hand gestures to manipulate virtual objects. Both of these mechanics make use of the Microsoft HoloLens’ main sensors (Microsoft Cortana AI Voice Recognition API and the time of flight (ToF) depth camera).

1.7.1 AI Voice Recognition - Controlling GUS

When using the HoloLens head mounted display (HMD) in the context of our game and in the context of ABB and the potential use of the product in a factory environment we have to take into consideration that the trainee/field engineer will need to keep their hands free of any controlling device to access real world equipment, machinery and interfaces.

Microsoft have an AI Voice Recognition API that allows us to create Grammatical structures that our game can recognise. Using these grammatical structures, we can do away with the majority of potentially required inputs to be replaced with voice commands. Of course voice commands cannot offer the accuracy of tactile controls but the actions we plan to use voice commands for are not actions that the trainee/field engineer do physically themselves but commands given to GUS to augment and manipulate objects in the virtual scene.  

These commands would act like a gizmo tool in a 3D editor or a right click menu in a photo editing application. Examples of these commands would be :-

• “GUS, get another Infrared Moisture Sensor” - a new virtual Infrared Moisture Sensor object would appear in the scene
• “GUS, show me the Cover Panel” - GUS will move to the location of the requested machine part and a holographic arrow from the player to the object will appear to guide the trainee/field engineer
• “GUS, what is this?” - GUS will give an explanation of what whatever the trainee/field engineer is currently looking at.

The goal of this mechanic is to make all of these actions passive and secondary to whatever the trainee/field engineer is doing.

1.7.2 Hand Gestures

For actions that require more direct, accurate interactions we are using the HoloLens’ depth sensing cameras to track the trainee/field engineer’s hands. Using this, we can pick up, move and rotate virtual objects, click virtual buttons, turn dials and virtually draw 3d notes/diagrams onto the virtual and real world machines. These actions are incredibly powerful and they allow the trainee/field engineer interact with the game in an intuitive and accurate manner. 

An example of this mechanic is looking at GUS and using and pinch gesture with your thumb and forefinger to activate a radial menu around GUS. Then you can use these hand gestures to select and change options in the menu.

Another example of this mechanic would be the solving of the training puzzle. When repairing a machine the trainee/field engineer can pick up different components and place them in the correct location on the broken machine in the correct order. This mechanic brings will allow a wider audience who may not have experience with traditional games controls to participate in out game.

1.7.3 Using Both Mechanics in Tandem

Here is an example of how the trainee/field engineer would use both of these mechanics in a play through of ABB Training Day :-

• Starting the game, a broken machine appears before you
• The machine is still connected to the power so you ask GUS, “GUS turn off Machine”
• You ask GUS, “GUS, scan Machine”
• GUS scans the machine and responds that there seems to be a problem with a fuse
• You ask GUS, “GUS, show me the fuse”
• GUS flies to the rear of the machine leavin and arrow trail behind him pointing to a row of panels, GUS tells you the fuse is located under one of these panels
• You ask GUS, “GUS, x-ray the panels”
• GUS x-rays the panels allowing you to see inside them and you locate the fuse behind panel 2
• Picking up a virtual screwdriver you unscrew panel 2,
• Opening the panel you remove the broken component inside
• While holding and looking at the broken fuse you ask GUS, “GUS, get another one of these”
• A new fuse appears
• You pick up the fuse and place it inside panel 2
• You rescrew panel 2 and ask GUS, “GUS restart machine”
• The machine restarts and it works
• A report sheet appears in front of you giving you information and statistics of how you did

In this example, the trainee/field engineer utilises both of the main mechanics in a natural way to solve a basic training puzzle.

1.8 Choices

The player must make choices on what is the proper safety equipment to wear. They must also choose what tools are needed for getting into certain parts of machines and also what parts are needed to be replaced.

1.9 Narrative

The narrative for the game is that robots have taken over all jobs except one. Robot maintenance. Humans live in a happy utopia where the only workers are field engineers. You play as one of these engineers whose job is to fix malfunctioning machinery for ABB. As it’s your first day on the job you are being helped by your own helper robot called GUS, short for Guided User Systems. The game starts with the player going through a health and safety tutorial where they must correctly identify safety equipment. There will be funny visual representations of why safety equipment is needed by another bot which is significantly damaged from not wearing any safety equipment. Once the player has selected the correct equipment they move on to the next stage.

The player will then be set to find out what is wrong with the machine in front of them. There will be a series of small tasks and GUS will be able to help the player from time to time. The player interacts using voice commands and gestures to GUS for tools, parts and services such as x-ray mode which allows the player to inspect a machine making the interior visible. The player fixes the faulty component and is awarded a score based on their accuracy and time. They may then play another “level” or exit the game.

Graphic Design

2.1 Visual Style – realism vs. Stylised

The models in our game will be as realistic as possible, mirroring the real machine that ABB has in its facilities. The model will be mapped on top of the real machine using AR. When a certain part of the machine needs to be replaced i.e the abc  it will be highlighted with a red overlay. Modeled screws that are animated from the machine will also hold a realistic style in our game. The characters, tools or any other puzzles/ interactables our game use will be styled using cartoon graphic style (see example). Combining the realistic ABB machinery with our cartoon models and character will allow our our game to have a simulator feel. This will all be done using the Hololens with its capabilities.

2.2 Special Visual Effects

Our main effect will be a hologram that will be projected from our character GUS. The hologram effect will be a blue beam of light, centered in that beam will be the character model of GUS using the effect displayed below(See 2.2). The hologram effect will be a 3rd party unity shader found on Github

2.3 Designing Reusable Assets

There are many reusable assets in our game, many will be standard assets like screws or other machinery parts. Machinery parts themselves can be reused on other machinery. An adjustable spanner model, The actual ABC’s, wires/plugs and General items you might find in a training facility.

2.4  Character

Meet GUS. 

GUS will be the only character in our game, he will act as the trainees sidekick when it comes to solving the tasks. The reason our game only has the one character in it is because our game is immersed into the real world where the players of the game are themselves. GUS as seen in the image above in snot just an odd looking tennis ball. What you can see here is a robotic orb, this orb projects a hologram giving gus a friendly face trainees can bond with. Here is what GUS will look like Activated in our game. As you can see he displays a very friendly face.

2.5 Animation

There are two types of animations in our game, character and environment. We want to give our characters a robotic cartoony feel an example for this would be TARS from the movie interstellar. The animation on the characters is a crucial part as it will set the mood for the Game. GUS will be floating around the room projecting different holograms that will be animated as a beam of light. GUS’s holograms will also be animated. If GUS were to be celebrating he might project himself a little party hat with some confetti falling down. This could be played out after a trainee gets a high score.

The method of animating GUS will be to use 3Ds max. GUS will be rigged using the Character Animated Toolkit (CAT), this will allow us to animate the movement of GUS. We will also morph GUS’s eyes and eyebrows, this will allow GUS to express himself and display emotions.

Our game will also include standard environment animation. This type of animation isn't too complex as the game will be set in ABB’s training facility. When a machine in  placed in the room for fixing the player will see diels turing or screws being manipulated small stander animation for the machinery in our game.

GUS can also manipulate the environment within the game.Like break through a wall to get a tool that he might have forgotten in his inventory. This will create an animation where the wall will break open and leave a mes in the room. Of course GUS will have to be apologetic about this.  

Audio Design

3.1 Sound Design Overview

ABB training day will take advantage of the Spatial Audio capabilities of the HoloLens. This will allow us to route sounds in three-dimensional space. This is vital for immersion of the engineer.

Initially we have been recording dialogue lines using a voice actor and then editing the voice to sound robotic. Moving forward we will be using Lyrebird, a neural network that can synthesise speech.

Diegetic sounds will be implemented for the machines for stage one and two. These sounds will not be needed for stage three. Sound effects will be implemented for the user interactions. This is to give auditory feedback to the engineer and to keep immersion. Non-Diegetic sounds will consist of menu feedback and victory music.

GUS’ voice will be synthesised using a program called Lyrebird. He will also make noise as he moves around the world. It is important that we ensure he does not make too much noise though as it could annoy the engineer.

We will be using a zoom h2 handy recorder to record sounds. It has a spatial audio mode in which it records 3D audio. DkIT have a number of these that students can book out.

3.2 Emotion & Atmosphere

Due to the nature of augmented reality the atmosphere will be created by the atmosphere in which the simulation is run. This will be factory setting with multiple large machines. Due to the size of the room required to fit these machines, we can safely assume there will be some reverberation. This will be taken into account when implementing sounds into the game. 

Should the engineer make mistakes, there will be consequences in the atmosphere. GUS will scold the engineer and depending on what they have failed warning alarms may ring.

3.3 Non-diegetic Sound

ABB Training Day will make use of Non-Diegetic sounds in the radial menu and through the use of victory music. Both of these are to give auditory feedback to the engineer.

As the user cycles through the radial menu a click noise will play. When the focus changes from one option to the next the noise will play. This gives the engineer immediate feedback.

When the engineer completes his task a short triade will play. This ‘victory music’ is designed to give the a sense of achievement and to signify they have completed their current task. There will be two different victory triades, a standard one to signify the end of a task and a rarer one that will signify placement in on the high score list.

3.4 Diegetic Sound Effects & Foley

As ABB Training day is an Augmented Reality experience some sounds that would be expected in standard games will not be implemented. An example of this would be footsteps. Footstep sounds are in the vast majority of games, but will not be needed for this game. As the engineer moves with the HoloLens on their head, they will hear their own movement. If we were to implement movement sounds, they would hear both the game sounds and their own. This would break the immersion.

Diegetic sound will be implemented for engineer interactions in stage one and stage two of the experience. Example sounds:

• Screw tightening
• GUS movement
• Warning Alarm
• Plugging wires
• switching switches
• Removing/placing components

3.5 Dialog

Initially for the voice of GUS we have recorded a voice actor and then applied a Vocoder effect to give it a robotic sound. But now we plan to implement a neural network speech synthesiser. This will give GUS an even more robotic feel than our initial technique. It also means that we do not need to get a voice actor everytime we need to record additional dialogue. The software that we are going to use is called Lyrebird. It works by taking between 100 and 300 lines of dialogue. It can then build a voice from those lines are synthesis any sentences. The less lines it is fed the more robotic it sounds, which is what we want.

Major/Unique elements

4.1 Augmented Reality

Augmented reality will be a major element of our game. We will be using Vuforia and unity engine to create an augmented reality experience using a Microsoft HoloLens headset. By projecting augmented versions of ABB machinery, we will create a training simulation for engineers. The player will be able to perform maintenance tasks and emergency procedures on the augmented versions of the machines.

4.2 Voice Commands

The main interface will be through G.U.S using voice commands. Using an .xml file we will be defining our own grammar for the HoloLens voice recognition technology. Using this players will be able to inspect items, highlight specific areas and use an x-ray mode to see inside ABB machines. Inspecting items will give the player information regarding the given item. This information will either be relayed by GUS or be displayed to the user. Highlighting specific areas will allow the player to mark specific areas for easy identification. Using the X-Ray on machines will allow the player to see inside the inner workings of the machine, as opposed to disassembling it.

4.3 Gamification

Gamification is a key aspect of our game, any gamification features we add will be the implementation of our gamification strategy, which is simply that we aim for players to become better at certain tasks that engineers may perform, while providing an entertaining experience.

Specific game features that we’ll be implementing are:

A scoring system, this allows users to see how well they’ve performed a task, and allows them to track if they’ve gotten better or worse. A player will receive points based on how he carried out a task, so for example if he gets asks GUS for help and gets a task wrong, he will be penalised less than if he just outright got a task wrong. This has the effect of motivating the player to learn and to ask questions when he is unsure.

Achievements or badges may be awarded to players who perform particular tasks exceptionally well.

Leaderboards, following up on the scoring system, this will allow players that have worked hard or are better in certain aspects to be recognised. This will instill them with a sense of achievement and reward, and allows for competition between players as to who can get the best score.

Collaboration, will allow players to see from the perspective of the HoloLens user, so for example if the user with the headset is the best at a task, viewers can look through his perspective how he plays the game and learn by watching.

Appendix

• ABB - https://en.wikipedia.org/wiki/ABB_Group
• Microsoft Hololens:
  • https://www.microsoft.com/en-ie/hololens
  • https://en.wikipedia.org/wiki/Microsoft_HoloLens

• Microsoft Voice Input for Unity - https://developer.microsoft.com/en-us/windows/mixed-reality/voice_input_in_unity
• Engineer - https://en.wikipedia.org/wiki/Engineer
• Lyrebird - https://lyrebird.ai/
• Zoom H2 - https://www.zoom-na.com/products/field-video-recording/field-recording/zoom-h2n-handy-recorder
• Spatial Audio - https://www.microsoft.com/en-us/research/project/spatial-audio/