First Responder AR

FRAR (First Responder Augmented Reality) is a HoloLens 2 app for trainee fire fighters to practice using a pump panel. This project was funded by a NIST grant and developed at RTI International in collaboration with White Cross Volunteer Fire Department. FRAR is not complete but is able to run well within Unity.

Check out the repo

Using FRAR

  • If this is someone’s first time using HoloLens, then select Tutorial from the floating menu. They can learn various hand motions and experiment with interacting with holograms before using the pump panel.
  • Explore the Pump Panel is where users can interact with the panel and learn about what each component does.
  • Quiz Mode should be launched after they’ve tried out the panel and read definitions.
  • The number in the top right of the UI in Quiz Mode is your score. There was some confusion about this throughout testing. The UI for Quiz Mode is not the most polished but it’s functional.
  • Quit at any time by pressing the quit button.
  • Pin the pump panel in place by either pressing or air tapping the pin icon in the upper right corner.

Explore mode controls

  • Point at panel components to show a UI popup with definitions for the currently selected component.
  • You can show/hide definitions by pressing the show/hide definitions button.
  • The engine on/off button will, as you might imagine, turn the engine on and off.
  • Levers can be pulled by hovering over one and either pinching or closing your fist. It’s quite finicky so YMMV. It’s oddly enough more reliable in Unity.
  • Interactable components glow orange when your hand hovers over one.
  • The engine throttle and Discharge Relief Valve can be grabbed and rotated like in the real world.
  • Finally, the panel itself can be grabbed, resized, and placed wherever the user finds it comfortable.

Attribution

Joost van Schaik’s blog was invaluable. I’m using his code from “Controlling a Unity animation using HoloLens 2 hand tracking” to enable lever pull and pushes.

Several CC licensed audio tracks are used within the quiz game. You can find an attribution file within the source code.