Unity Project Setup
One of the first things to do in Unity when working with the HoloLens is to make sure that the project is correctly setup for use with the HoloLens and that the necessary files are installed.
- Unity 2017.3 or greater (with.NET scripting backend), previous versions will not work with the HoloLens
- Visual Studio 2017 with the Universal Windows Platform Development and Game Development with Unity workloads.
- HoloLens Emulator (Optional), which allows you to run holographic applications on your desktop without a HoloLens. (Requires Hyper-V)
- Windows 10 Fall Creators Update.
If using the HoloLens Emulator, it's important that your system is running 64-bit Windows 10 and is compatible with Hyper-V.
Spectator Camera
One key need for us was the ability to demonstrate the application to a user. Unlike traditional games where others can easily see what a player is doing, only the person wearing the HoloLens can experience it. By using Microsoft's Spectator Camera setup, multiple users with HoloLens' can join in or spectate, and this also allows us to record in-game footage from various angles.
The setup for this is relatively complex, firstly it requires a number of hardware components:
|
|
After acquiring and assembling the hardware, a number of software components are necessary:
|
|
After this, Microsoft's spectator view project must be tweaked and built, then it is necessary to run the calibration app that comes with it; this synchronizes the DSLR camera with the HoloLens for an accurate video feed. After importing the necessary .dll files (from the software components) and the calibration details to the Unity project, we can now record footage of our application through the DSLR.
Vuforia
Vuforia is a Unity plugin which extends the capabilities of the HoloLens by allowing for the recognition of Markers. It's installed simply by importing the necessary Unity package file. When a marker is detected, a GameObject can be placed at its location in the world and allows for real-time tracking. For Training Day, this was used to augment a wooden cabinet with a detailed Actuator Cabinet, which a player has to fix.
There are a few things to consider when using Marker technology. The actual size of the marker must be correctly measured in centimeters and these dimensions then imported to Vuforia for more accurate tracking. For maximum tracking ability, the marker should contain as many distinct shapes as is possible (think QR code). The larger the marker, or if you use multiple markers, the real-time tracking will become more stable. The most accurate tracking would be achieved by a marker spanning the whole width of the object, and if the marker is discrete then it would give the appearance of being marker-less. Additionally, 3D models can be used as markers so modeling and texturing your object may be used for accurate tracking.
After setting the marker up and importing it to Unity by hooking up the marker database, drag a GameObject so that the marker is the parent of the GameObject. Make sure the units of the marker in the Unity scene match the units of the original measurements (Unity uses meters). Finally, scale the object you wish to appear in proportion to the marker.
Augmented Cabinet
The marker used for augmenting the cabinet
HoloLens Toolkit
The HoloLens Toolkit is an extremely useful plugin which provides various examples that can be used right off the bat with the HoloLens, such as color picking, menus, on-screen keyboards and other interesting functionality. To install the HoloToolkit, get the .unitypackage file from GitHub and import it.
A large part of Augmented Reality is the design, as some traditional design practices may not apply to an AR headset. This Toolkit allows a developer to focus less on programming functionality and more on designing an immersive and usable experience.
Features for Ease of Development
One thing I realized early on in the development of this project was the tediousness and extremely slow pace of developing for the HoloLens. In order to get a project from Unity to the HoloLens involved building two projects. Then, uploading the second build to the HoloLens would allow testing. This process was far too slow so I developed a system where we could use PC input to play the game, which mapped to the HoloLens input closely so that as a general rule of thumb, if it worked on the PC, it worked on the HoloLens.
This allowed for rapid development on the PC using the mouse as input. After features were completed they would then be tested extensively on the PC, bugs would be found and fixed in a fraction of the time it would take to debug on the HoloLens. Then, these features would be tested on the HoloLens and any further Holo specific bugs would be taken care of. Had a system like this not been implemented, the project would not have been anywhere near its current state.