Virtual Reality (VR) provides an immersive user experience which makes them a cost-effective solution to employ for various training purposes. However, a major shortcoming of VR systems is their limitation when it comes to interacting with the environment. Typically, when users wear a head-mounted display their vision will be limited to the virtual world and their external vision will be blocked. They will not be able to see pertinent objects in their environment such as controllers, buttons or even their hands. This project demonstrates a design of a training system for the aerospace industry where real and virtual images blended. The real world images are obtained from a camera mounted on the head-mounted display. Some of the predefined objects, such as game controllers and user’s hands, are detected via deep learning algorithms and blended into the virtual reality images providing a more comfortable and immersive user experience. Furthermore, camera and object detection algorithms are employed to interact with VR headset making it a more convenient tool for training simulators.
Technology Features, Specifications and Advantages
Driving simulator with editing tool for creating training scenarios by experts. Developed using open source tools such as Unity, OpenCv for trainer and trainee interface development, YOLO and MobileNet-SSD for object classification.
The key technology features of this technology are:
- Simple: Authoring tool for virtual environment allows domain experts to build training scenarios without the knowledge of game programming.
- Flexible: Various applications can be built by choosing digital assets from drag and drop menu.
- Mix-reality: Blending virtual and real images of the object for easy interaction with game controllers.
- Handy: User able to switch between real and virtual images using hand gestures.
Key advantages over existing VR glasses:
- Superimposed real-world objects in the VR making training more realistic while not losing the immersive sense of the VR scenarios.
- Quick control / toggling using hand gestures.
Simulators are widely used for personnel training for learners who are operating machines or systems at critical and/or dangerous environments. This way, operators gain a risk-free and in-depth training experience before embarking on the actual machines or systems. There exists various driving simulators, however their high cost, limited field of view (due to the use of monitors), and lack of portability are major drawbacks. The proposed system can be used for training in:
- Driver training
- Heavy equipment use
- Immersive training experience at low cost
- Easy to build hundreds of training scenarios
- Able to interact with the real-world environment while wearing VR headset