A look into the wonderful and exciting world of virtual, alternate and mixed reality.
Intro
The concept of virtual reality is not a new one, neither is the one of augmented reality. It has been around for quite a long time already. Perhaps mixed reality will be the next big thing? I’ll leave that up to you to decide.
Ever since we have been able to create visual representations of our own or other worlds, mankind has been fascinated by transferring the sensory perception into this world. It is in the recent years that technological advancements have made it possible to do this in ever increasing realistic and engaging ways.
A trip down memory lane…
In 1962 some crazy person built what is generally believed to be the first virtual reality experience. This machine pictured above is called a Sensorama. The user of the Sensorama sits on a tilted chair and watches a short film with stereoscopic 3D images in wide-angle view with stereo sound and even added effects such as wind and aroma. A predecessor of those 4D movies, available in cinemas and theme parks these days.
This, of course, cannot be compared with the more advanced implementations we have today.
In the 70’s and 80’s virtual reality really started blossoming in certain specialized areas. In 1968 Dr Ivan Sutherland created the Sword of Damocles at MIT. This head mounted display (or HMD) was so heavy it needed to be suspended from the ceiling. It was able to track head movements and show rudimentary 3D images.
During the 90’s and 00’s: more sophisticated military, flight and combat simulators started showing up. In the privatized industry: simulators for airline pilots, doctors and surgeons. Different ways of interacting with this virtual world also started appearing:
- Head tracking
- Touch screens
- Gloves
- Entire flight simulators
- …
This is also when video game companies first tried to capitalize on this exciting new technology and bring it to a wider audience. But the technology was still too rough and did not catch on. Some of these early attempts include:
- Nintendo Virtual Boy
- Virtuality (arcade system)
- iGlasses
- VFX-1
But then came 2012. The year in which the Oculus Rift was kickstarted. This sparked a renewed interest in virtual reality which, thanks to technological innovations, was looking much, much better it ever had!
All kinds of realities
Alright, so we have been talking about virtual reality for a while. But what exactly are virtual reality (VR), augmented reality (AR) and mixed reality (MR) and how do they relate.
Above you see the reality - virtuality continuum.
Let’s start on the left side. The left side depicts reality or the Real World. With the Real World we mean the world where you, as the reader of this article, are sitting in a chair looking at your computer screen or holding your smartphone. For the sake of this article we assume that the world is real and that there are no matrix-like shenanigans going on. Simply put, the world as you know it and experience it every day, the world you and I live in.
On the other end of the spectrum we see the Virtual Environment. Everything in the virtual environment is virtual, so ‘not real’. It is the imaginary world that you see on a screen. The user is locked out of the real world. An extreme example we all know of would be The Matrix, in which the person lives their life inside The Matrix not knowing the entire thing is not real.
Augmented Reality (AR) and Augmented Virtuality (AV) are situated in between these two ends of the spectrum. As their names suggest, AR is more closely aligned with the real world while AV is closer to the virtual world. In other words AR consists of a mostly real world with a few virtual elements augmenting the experience. AV exists mostly in the virtual world with a few elements of our real world visible.
Mixed or Merged reality (MR) is a term used to describe anything between reality and virtuality. This means it contains both elements of the real world and a virtual world. It can go all the way from a mostly real world with a bit of virtual world sprinkled on top to mostly virtual world with only trace elements of the real world remaining visible. With MR as opposed to AR and AV, the virtual world is aware of the real world and can interact accordingly.
We see the term X-realities being used to describe any of the VR, AR, MR and any other realities.
These terms are pretty vague and some have even changed meaning over the years. They outline the boundaries of the different areas in the spectrum, but they tend to bleed over into one another very easily, and get mixed up a lot!
Virtual Reality
We call something virtual reality when the user is emerged in a virtual world and is completely blocked out from the real one. The simplest example of this is a 360° video experience. More advanced implementations have the user take part in a fully virtual world where their movements and actions are tracked and translated into actions in the virtual world.
Some examples of virtual reality implementations are:
- Games, both recreational and educational
- Photo content: photo spheres
- Video content: video spheres, 360 videos
- Creative applications like 3D painting, sculpting, DJ applications and many others…
Currently the most popular application for VR is video games. There are a lot of games with VR support on the market already, and more are getting released almost every day! Shooting games and simulators seem to be an exceptional match for the platform. Although mostly used for gaming purposes, a VR headset can also be used for viewing 360 scenes and videos. Thanks to head tracking, the viewers perspective is adjusted automatically when the head is moved. The more advanced headsets like the Oculus Rift and the HTC Vive even have external sensor arrays to track user movement, and have hand based controllers for interaction. Although you have to deal with additional setup compared to a phone where you can just drag the viewport or use your phone’s motion sensors.
Augmented Reality
Last year, the world exploded with people young and old crowding the streets in search for Pokémon.
Pokémon Go is a fine example of an AR application. It is augmented reality because the application is still grounded mostly in the real world. The virtual elements are layered on top of the real world, but they do not interact with it.
There are 4 types of augmented reality applications.
- Marker
- Inverse Marker
- Markerless
- Projection
Marker based
Marker based AR apps use image recognition to recognize a specific pattern or marker. The marker is detected and replaced or covered with a virtual object. These types of applications are very simple and used for showcasing or displaying additional information.
Inverse marker based
Inverse marker based AR is very similar to regular marker based AR. These applications are used in conjunction with large screens with cameras where the user only has to control the marker.
Markerless
These markerless applications use positional tracking and GPS to determine where to show things by mapping the environment and creating a spatial awareness to track objects when moving. The popular Pokémon Go and Ikea apps are perfect examples of this.
Projection based
Projection based AR projects images rather than showing them on the screen. This requires hardware capable of projecting so is not as widespread. It is used more in manufacturing.
The great thing about AR is that any recent smartphone is capable of running AR applications.
Many AR apps only need a camera to work.
While specialized depth sensing sensors exist, they are not as widespread and are used more in specialized industries.
Mixed Reality
Mixed reality takes the best parts of both virtual and augmented reality. What makes mixed reality special is that it understands the environment, it can interact with and respond to changes or events happening in the real world. Hardware that supports MR takes the form of glasses and headsets with cameras and other sensors. These sensors are used to map the physical 3D space so the virtual objects know where they are situated and know what is in the real world. This is where you get the holographic experiences, even tough they aren’t real holograms, they look and feel exactly as you would expect them to.
Hardware and current pain points
Current devices are a big step up from what we had even a few years ago, but there are still many improvements to be made. The high-end VR devices require beefy computers equipped with high-end graphics cards. The headsets requiring a smartphone offer a more mobile experience but are still limited in the visual effects department. For a virtual reality experience to feel smooth and natural it needs to be rendered at a minimum of 90 frames per second. If the device can not keep up, the user might experience VR sickness, similar to motion sickness. Because our eyes are so close to the screen and the resolution our eyes can perceive is so high, pixels become visible, detracting from the experience. When looking closely, the screen looks like a raster. This can be solved with a higher resolution but that again requires more computational power. This much content at such a high pace also requires a lot of bandwidth and low latency.
Another deterrent for VR headsets is the extensive setup required to make them work. High-end devices require base stations or tracking cameras to accurately measure movement. There are devices with sensors built in so that they are not dependent on external base stations, this technique is called inside-out tracking.
Creating realities
Making software that is AR/VR/MR enabled can be quite different from traditional business application development. These type of applications are often heavily dependent on visual elements and can require extra artistic knowledge. For virtual and mixed reality game engines can be used to create “games” to create these experiences.
These are some of the most popular game engines that are free for personal user:
- Unity
- Unreal Engine
- CryEngine
Think of these as a sort of photoshop for software. We can leverage the editors to see what we’re working on and make changes immediately. These tools are ideal for virtual reality. Both Unity and Unreal Engine have plugins available for augmented reality.
The more business targeted applications are often augmented reality enabled applications. Both Apple and Google have made big efforts to support this on their respective OSes and basically every smartphone with a camera is now AR enabled. Apple’s ARKit and Google’s ARCore are the respective augmented reality APIs for their IOS and Android platforms. Since the inception of these two APIs the amount of AR apps has greatly increased, Google is even discontinuing its Tango project in favor of ARCore. There are many other SDKs available, many of which even have their own Unity plugin. Vuforia and ARToolkit seem to be the most popular ones.
Microsoft created the Windows Mixed Reality framework for mixed, virtual or augmented reality apps to run on windows 10 based computers.
Business cases & Examples
So, as software developers, what can we do with all this fancy tech?
Our Dutch colleagues already made an awesomely detailed VR application to help train personnel of the Royal Dutch Navy. Make sure to have a look since it is a truly impressive feat of software engineering.
Virtual reality is widely used for training purposes. Whether it is for a boat, an aircraft, a machine or a human body, a virtual space can prepare someone for situations that are otherwise very costly or difficult to simulate.
Augmented and mixed reality can show information when and where you need it. On a mobile phone screen, through glasses or projected on a surface, having the correct information at hand is always useful.
Visualizing a product before it is manufactured can enhance the design process by discovering points of improvement much earlier on. Visualization like Ikea’s app for trying furniture or a tour through historic Bruges or even business cards with an AR marker for increased memorability.
For us at Ordina, we see most potential in AR and MR. Mobile based AR applications that can be used for showcases, in the IoT world, or even just augmented digital signage with inverse marker based AR.
Conclusion
The years to come will bring even more advancements to the wonderful world of virtual, augmented and mixed realities. A lot of things are still shaping up and being developed, ever improving on earlier versions. A lot of solutions and standards are still being figured out. Which means that now is a great time for us to try stuff out, and get a feel for the technology, whilst preparing and working on proof of concept applications.
