Mixed reality, which blends the real world with digital experiences, allows technology professionals to develop a new genre of apps with multiple potential benefits and use cases. While the learning curve is inevitable, there are plenty of techniques, tools and training opportunities already available.
Mixed reality takes existing extended reality (XR) to its next stage. While VR is about being fully immersed in the digital world, and AR adds digital objects to the physical world, mixed reality anchors those objects to a physical space and then supports user interaction. An example might include a digital model of a toy that can be moved around. Mixed reality is also ideal for collaboration: for instance, a remotely-based colleague’s avatar might appear in a room during a meeting.
The Possibilities Are Vast
Across all kinds of industries, mixed reality applications are already emerging. For instance, in manufacturing, the focus has been on increasing efficiency for employees in technical roles. It can be used to walk technicians through complicated processes, for example, or contact an expert to provide visual instructions for a difficult task In healthcare, mixed reality could be used for practising surgical procedures, learning anatomy, or even playing a role in telemedicine in the future.
For designers and architects, previewing a project before it is finished, in its intended physical environment, can help spot problems and support decisions around changes before committing to costs.
Creating unique experiences for customers is another use for mixed reality, allowing concepts or products to be visualised in an exciting new way. Potential buyers can manipulate a digital product, even turn it around in their hands, or take it apart.
Essential Steps Towards Mixed Reality
Once the business case for a mixed reality app is determined, many organisations specialise in creating extended and mixed reality solutions for various industries. However, for any team wishing to try out their ideas in-house, there are already some well-established processes, a range of tools, and a growing volume of available education for mixed reality.
So, if going the in-house route, the next stage is the creation of a mock-up. Because 3D modelling is not something every organisation has easy access to, bodystorming has become popular in mixed reality development. Bodystorming uses traditional craft modelling materials (such as clay) to create a physical prototype. Then, like a regular brainstorm, a bodystorm session helps get input and feedback from the whole team and think spatially about the content of the eventual app.
The next step is to literally ‘act out’ how a user might move through the mixed reality experience, and it is helpful to include non-technical staff as actors or provide feedback. For instance, if the intended application is in healthcare, the insight of medical professionals will be valuable. Also, like any other design project, creating a storyboard of the proposed mixed reality app will make it easier to share with various stakeholders, should their feedback or buy-in be required.
Types of Tools
Many of the tools used in mixed reality are already well-known and tried and tested by many software engineers, but others may be new to some users. Games engines, in particular, have become integral to mixed reality development, with Unreal and Unity leading the pack. Both enable developers to create visually stunning content and render 3D visualisations in digital form in real-time, plus spin up and test multiple iterations fast.
Other tools used for mixed reality include version control software, which is also widely used in the games industry and different software environments, to manage changes to files over time and store these modifications. In this way, the team has a ‘single source of truth’, and if an issue crops up, it is possible to roll back to a previous version of a project and deal with any problems. In heavily regulated industries, version control is also essential for achieving compliance, especially when the company’s mixed reality applications collect user data.
One of the primary elements of mixed reality is head-mounted displays (HMDs). The good news is that many manufacturers offer kits – some of them at no cost – to assist aspiring mixed reality developers.
HMDs project images onto lenses to give the illusion that the mixed reality digital image appears to be in front of the wearer. In common with VR headsets, HMDs have handheld controllers that can interact with digital objects. Some HMDs also have eye-tracking capabilities. However, how they differ from VR headsets is that the viewer sees through the lenses to the world around them.
Other relevant equipment may include depth-sensing cameras designed for spatial requirements. Finally, a wealth of information is available, including online courses (such as Microsoft’s HoloLens online tutorials for Mixed Reality for Unreal Engine).
Taking the plunge into mixed reality development is, at the very least, a chance to learn something new and have fun. Still, equally, it could be something that makes a difference to all kinds of professionals and users, and opens up new market opportunities.
By Brad Hart, CTO at Perforce Software.