Where is VR?
“It’s a machine. But through this machine we become more compassionate, we become more empathetic, and we become more connected. And ultimately, we become more human.”―Chris Milk, TED: Clouds Over Sidra
Today, you can plug in some headphones and strap on a virtual reality (VR) headset ― really just a specialized pair of goggles attached to a smart phone ― and suddenly be transported to a different place, a different time, a different perspective.
Samsung Gear VR
After decades of false promise, virtual reality might be coming soon to a home near you.
VR suddenly began to feel very real in 2012, when Palmer Luckey raised over $2-million in a Kickstarter campaign for Oculus VR, “the first truly immersive virtual reality headset for video games.” Two years later, the startup was acquired by Facebook for $2-billion. The social media giant sees VR as the next common communication platform: people in different places will play games together in the same virtual space, as they currently do in networked gaming environments. They’ll chat, shop, watch movies in a shared virtual living room, and instead of video calls, they’ll drop in virtually on special occasions to catch the big event. VR might be more than the future of entertainment or social media. VR might be its own thing entirely.
VR has the power to transform our lives, for better and worse.
The dream of a cinematic experience that could duplicate reality has existed for more than a half century. In the 1950s, Cinerama took audiences on a roller-coaster ride with 146-degree vision. Then, IMAX made the experience bigger. Planetariums made it rounder. Yet each medium frames an image projected onto a screen: the viewer watches the screen rather than feeling immersed inside of a three-dimensional, digital environment. With VR, which requires a computer to draw the virtual world in real-time (at 90 hertz in stereo) and track every minor head-movement ― without which we have nausea, for the first time we have a medium that breaks the fourth wall. VR connects the viewer and the viewed in a way that has never happened before in the history of media.
Oculus Rift “Crescent Bay”
In communicating the capacity of VR, there’s a terrible tendency to sound like a carnival barker. In early 2016, Oculus showed their first consumer version of Rift: the “Crescent Bay” headset was made available in 20 countries, and other hardware, software and content for VR has made headlines ― the news has been a funfair.
“There are two groups when it comes to VR, there are people who are absolutely convinced that VR is the future ― not only for computer gaming, but of all entertainment and online social interaction. And there’s another group who haven’t tried it yet. It is that powerful.”―Bernard Roehl, Immersed.
Microsoft, Samsung, Sony, HTC are some of many companies invested in taking on the challenge of VR. There seems to be a gold-rush-like frenzy gripping the industry. Last September marked the first time a VR experience won a Creative Arts Emmy Award (in the best user experience and visual design category). In January, more than a handful of VR experiences made headlines at the Sundance Film Festival. VR experiences have also given extra spotlight to some of the latest blockbusters: The Hobbit: A Thief in the Shadows by EPIC games and Weta Digital, The Martian by FOX Innovation Labs and Jurassic World: Apatosaurus by Felix & Paul Studios. VR companies were front and center at the manic Consumer Electronics Show (CES) in Las Vegas. And in March, the Game Developer Conference (GDC) in San Francisco is prepared to launch the first Virtual Reality Developer Conference (VRDC) ― a two-day summit on the state of the art of producing VR (and AR) experiences, alongside GDC 2016. With VR markets estimated to boom into an industry worth between $70 to $150-billion by 2020, many creators are wary of the hype: the unrealistic expectations and a flood of opportunistic experiences that don’t make the effort to convey an artistic voice or vision for VR. This might turn-off potential users before the medium has had a chance to mature.
The Year of VR
For the early adopters, the trail-blazers, the one per cent ― the answer is undeniably, yes: VR has arrived in 2016. Google could make the case that its Cardboard is more widely available. Facebook would have you see Oculus Rift and believe that “next-generation virtual reality” is already here, for US$599 (plus a computer). That’s an important add-on ― the price of VR headgear invariably accounts for the computer and any peripheral devices, and none counts the cost of a room to accommodate the virtual space. So, for most of us, most of the other 99 per cent ― the answer is, still: a couple years away. It might just be two. 2018 might be the year of VR, but before we get there ― one more thing about the aforementioned brand names: where is Apple VR? If, when Apple takes its bite of the VR market ― the question really is: where will VR be when a definitive platform is mirrored by an innovative, open source or low-cost alternative that truly makes the experience available to the masses.
The month after Disney announced that Star Wars: Episode VIII was delayed (from May to December of 2017); the year after The Force Awakens generated $2-billion in the global box office ― Spielberg’s upcoming rendition of Ready Player One, which is being produced by Warner Bros., and which is also billed as “an event film”, was pushed back from that same December to March of the following year. 2018. No one can fault the decision. (Though no one can speak for Tarantino: Hateful Eight went up the same weekend as Star Wars in 2015.)
Ready Player One is the Ernest Cline book about a mostly virtual future and a VR treasure hunt that is infused with nostalgia for 1980s pop culture ― not coincidently ― with more than a couple references to the works of Steven Spielberg. Warner Bros., which has not worked alongside Spielberg since A.I. Artificial Intelligence was released in 2001, has attached Spielberg to Ready Player One and, in doing so, likely secured rights to reference his early films produced through Amblin Entertainment. However, Spielberg did not become attached to the project to comment on his past; instead, he’s talking to us about our future ― a future that has a lot of its building blocks falling into place right now.
The story behind A.I. ― originally conceived by Stanley Kubrick ― was about a future when artificial life is capable of experiencing love: “If a robot could genuinely love a person, what responsibility does that person hold toward that mecha in return?” In a documentary on the film, Spielberg focused on the extent to which we have a moral responsibility to the intelligent robots that we will someday create. In taking on Ready Player One, Spielberg is looking to balance the morality and the more fantastic aspects of VR, “to show why it’s interesting not living in the real world, but what we’re missing by not. It’s a cautionary tale, but it’s also a big rockin’ adventure movie, too.”
Art of Ready Player One
The hype surrounding VR in this first-quarter of 2016 marks this year as significant in the development of the medium, but the hype is not going away anytime soon. In the next couple years Samsung Gear VR, Sony Playstation VR, HTC Vive will become more sophisticated, more accessible, and more companies will find ways to offer more affordable VR systems. Microsoft might find the right space for its HoloLens. Apple VR might become a thing. A yearly press cycle would see the second or third consumer version of Oculus Rift released March of 2018 ― weeks before the theatrical release of Ready Player One.
Spielberg has a history with cinematic adventures that explore the direction of human-computer technology. His films have a way of addressing the present times: A.I. was released in a year when Stephen Hawking encouraged the development of genetic engineering, essentially, to compensate for Moore’s Law. Spielberg’s sci-fi stories have a special relation to computerized imagery, which is the most spectacular digital technology that has changed the manufacture and look of contemporary cinema. It’s not inconceivable that Spielberg, with Warner Brothers, with their rendition of Ernest Cline’s Ready Player One in IMAX in 2018, have more impact on the early and late-majority consumer markets for VR than this first marked year of consumer feedback. It is ironic that the most anticipated experience in relation to VR may still be a cinematic one. That may change in 2018.
“VR ― it feels like it is 1993 again ― it is like ILM has just done Jurassic Park and the whole world is ready for a new way of seeing things, and I think VR and interactivity is bringing that new way to story tellers and everyone is again trying to work out the new problems and seeing what can be done.”―Kim Libreri, Epic Games CTO
Digital Domain, Framestore, Weta and other visual effects studios are extending their expertise to the virtual space.
Tupac Hologram (2012) by Digital Domain
VFX in VR
Digital Domain produces visual effects for complex narrative scenes in feature films such as Her, Curious case of Benjamin Button, and the same production pipeline tools helped them create the groundbreaking and award winning work on the Tupac hologram at Coachella, in 2012. Digital Domain was behind the CGI-rendered VR film Evolution of Verse, directed by Chris Milk and showcased at last year’s Sundance Film Festival.
Evolution of Verse (2015) by Chris Milk and Digital Domain
Framestore, which was awarded an Oscar for the visual effects of Gravity in 2014, has since become a leader in VR experiences as well. They now have a dedicated VR team: Framestore VR Studio that takes advantage of their existing digital worlds to develop interactive cinematic experiences, like one set inside the Endurance space ship from Interstellar or one set 700 feet up on top of the wall in Game of Thrones. These projects, which were built for Oculus, and which were built with Unreal Engine and with Unity Game Engine, respectively, were only made available in a showcase format at special events, because they could then include wind machines and rumble packs to create a more immersive experience; also, because the graphics hardware needed to run them was more intensive than could be expected of the average consumer’s personal computer.
Notwithstanding an early impressive VR portfolio, which included interactive product ads for example for Coca-Cola, Framestore will be first to admit that the VR industry is still in its infancy. However, these early examples may also indicate a high quality market for VR: immersive trailers. There is clearly money in promotional trailers, which are already short format, which is important for VR trailers, or VR environments, to download, or stream, and to run on the average consumer’s VR kit. There is virtually infinite room to explore a new capacity for engagement. Earlier, in the context of the future of VR technology, a certain ‘fruit’ company merited honorable mention; now, in the context of the future of VR content, it’s easy to imagine a company like Disney coming in to lead by example.
One of the most impressive showcases of a VR environment was at GDC in March 2015, when Weta Digital and Epic Games showed a one-off experience: A Thief in the Shadows was a face-to-face with Smaug in his treasure chamber full of gold. The experience was designed to take full advantage of the new Oculus Rift “Crescent Bay” headset, the latest Oculus Audio SDK, as well as all of the digital assets from The Hobbit, including its complex sound design, with dialogue that was re-cut and re-imagined from the film’s original recordings. The character is very much with you ― just in front of you as with a cinema screen, but its roar emanates from a wide area. To realize the virtual scale of a giant dragon mouth, the sound design team positioned various frequencies of that roar in different audio-spatial regions relative to the user. To handle Smaug’s animation, Weta re-engineered a lot of its post-production tools, which handle large numbers of real-time assets on its motion capture stage, for their VR project. This was an exploration of the capabilities of their real-time technology. Like Digital Domain and Framestore, the production pipeline developed for crowd simulations (at Weta) was applied to Smaug’s facial animation in a way that compressed it down to a hero ‘bake’ that would run on the GPUs.
The Hobbit (2014) by Weta Digital
Similar to the two by Framestore, this experience was a one-off because the Unreal 4 Engine, which was modificied to support the quality of animation (and scaled to handle the large data streams and complex rendering), was running on a PC with a previously unannounced NVIDIA Titan X graphics card. NVIDIA’s next generation graphics-processing architecture enabled the team to encode the creature deformations in a way that would stream the simulation live in their game engine, which was punching well above the weight of a AAA console game to achieve this demo. Epic Games had only 11 milli-seconds to render a high-resolution stereo pair of 400,000 skin verts performing through a scene with lights, shadows, particles, coins, fire and from the most sophisticated visual effects company in the world. Capturing the moody lighting of the film, efficiently in a game engine, took on a theatrical lighting approach, with a cascading set of point lights timed to Smaug’s movement. The fire was rendered using some deep image compositing tricks, combined with live action fire. The fire was at the most 6 layers deep to the viewer in the demo space.
“Theater is not a bad analogy for VR ― as we don’t have cuts and it all has to work for the audience, it all has to be one continuous performance, and everything had to be re-done as one big thing that you could look at from any angle.”―Kim Libreri, Epic Games CTO
For the many working in this space already, the key is to not just reuse the trailer but to extend the narrative, to allow for an experience where the VR informs the fan in ways beyond the traditional preview. VR comes with a new set of rules and narrative tools, and just showing the same footage in the round is not using this new medium to its fullest. There is hope, however, that VR will be used in more than just advertising or trailers. It is fundamentally a new cinematic language and we are really yet to see the definitive VR experience. The question that remains to be explored: what’s the point of the immersion?
Strangers (2014) by Felix & Paul Studios
Back in 2014, VR buzz surrounded a strangely intimate encounter with musician Patrick Watson at his Montreal studio, created by ― also Montreal-based ― Felix & Paul Studios. Felix Lajeunesse noted thatStrangers focused on the intimacy of the medium, “it’s really about you as a viewer and him as an artist sharing a space, and being together in a moment of what you would call vulnerability, because he’s not performing, he’s writing music.” Another VR experience in the same year was focused on the stunning natural scenery in the film The Wild Within (2014), and showcased British Columbia’s raw wilderness to the Oculus Rift, to create an interactive travel experience as well as to promote the film.
We can now have a virtual adventure.
Documentary and narrative short film are two of the early popular genres in VR. New Deal Studios have produced a war adventure called The Mission. It was filmed in partnership with Jaunt VR, who developed a working production pipeline from camera to VR projection for live action production. This genre of VR has received particular criticism: that it has potential for manipulation, that the cumbersome VR filming gear can require someone to direct subjects ― taboo in documentary film-making. VR documentaries have also been criticized for simply placing viewers in a scene, banking on the novelty of virtual presence in that scene to create a sense of empathy in the viewer. In an earlier era, photographs themselves had this effect, giving people a glimpse of the manufactured reality of another person’s existence. In any medium that becomes pervasive, though the theshold may be higher for more interactive experiences, we become desensitized. Just as with the best photographs that make it on National Geographic covers, we can still find that deep sense of empathy. It’s just not every photograph anymore. We still have a need for story. We still have to make it meaningful.
We still have to think about its design.
“Our modern world is determined by them; the architects, the designers. They design how you experience he physical world, but the ubiquity of smart phones with gps and compass and network and more computing power than we know what to do with, has created this space where a new where a new reality can lie on top of the physical, real world. It’s an augmented reality, and it adds a whole new level of creativity and design to the everyday world.”―Roman Mars, 99 Percent Invisible (e/03)
The Future: Virtually Designed Now
Every new medium comes with its own set of rules, but this one comes with virtually limitless space. The VR experience that is designed with simplicity in mind ― with everyday life in mind, that balances information and interaction, and that draws from the lessons of mobile design from the past ten years (and of the Internet of the past twenty) will show the way for the virtual world for the next ten-to-twenty years. Three key design elements that will shape the way we experience VR: the displays, the peripherals and the interactions.
Oculus VR Controllers
In the short-term, so particularly for the early adopters, VR displays of today have an effect on our vision that is yet to be determined. Companies have to think about this, too. VR companies also have to think long-term about their versions, controllers, sensors. Multiple and peripheral devices may fragment a user base. Then, developers are less assured of who has what. (Design with simplicity in mind.) In the next few years, with millions of consumers try VR, developers have an opportunity to evolve the ergonomics of devices and gestures that will shape the way we interact with VR. This can even be designed to encourage good posture and real-world activity. (Design with health and help in mind.)
Microsoft has made HoloLens only available as a developer kit, despite that a consumer version from Samsung, Sony and HTC have all hit the market in early 2016 ― at a time when VR has captured the imagination of many consumers. The problem is not the hardware. Microsoft has been the focus of the tech buzz as much as any company that has invested in VR. They also own Minecraft ― a game with real VR potential on any platform. Still, so far, they have presented the most sophisticated VR demos, and The HoloLens developer kit is arguably the best of the available options ― that is if you ignore the price tag of $3,000 (plus a computer). The logic is that developers, unlike consumers, will put HoloLens to use creating some compelling VR (and AR) titles. At this stage, there are very few things that prospective consumers can do with the headset. For this reason, perhaps also because of the virtual minefields bubbling in mainstream buzz, Microsoft believes the consumer world is not ready for VR.
For others, the more: the merrier.
Facebook is forging ahead with Oculus: VR as the social network. Welcome to the future with virtually no privacy. If you think Facebook (also Google, Microsoft, etc.) knows a lot about us now, think ahead to when we interact with the world through VR, which will soon be able to analyze our every subtle eye movement, lip quiver, finger twitch ― to read physical cues to discern when we are at our most susceptible to commercial persuasion. Some of these massive software companies, after all, make money by transforming our behaviour into data for marketers. The inconvenience of design research that is good is that it can be applied to a pursuit that is lesser.
“The internet has become an econimic engine devoted to converting big data from billions of people into large profits for a small handfull of players.”―Ana Serrano, TEDx: Life Inside the Bubble of a Virtual Reality World.
Samsung is making the most of their mobile market with Gear VR, a companion headset for select Samsung phones ― select powerful ‘pocket’ computers (in select pockets). It is the highest resolution VR display early to market, and is powered by Oculus VR technology. Samsung’s Gear VR is a very literal stepstone from the mobile to the virtual space, designed for films or pre-rendered pictures in the round. It’s one of the least expensive VR solutions: US$99 (plus a Samsung phone). For action and movement though, dedicated headgear has better head-tracking results.
Samsung Gear VR
A high frame rate of at least 90 frames per second, and low latency plus a pixel persistence lower than 3 milliseconds to avoid nausea when moving the head around, is imperative for immersion in VR. This is where Sony’s Playstation VR comes into play, with a 5.7 inch, 1080p OLED screens that can refresh at 120 frames per second and with very low persistence. This is an improvement over the motion blurring that often occurs with older LCD screens. By comparison, the Samsung Gear VR has a 2560 by 1440 display that can only update as fast as 60 frames per second. The Sony headset design also includes LED trackers, but the advantage to developers for the Playstation VR is the dedicated console. The VR gamer solution: US$399 (plus a PS4). For the time being, Oculus Rift is the high-end PC tool and Samsung Gear VR is the go to mobile solution.
Sony Playstation VR
Alternatively, the Vive combines HTC’s hardware with Valve’s Steam VR technology. Valve is a powerful force in the non-console gaming industry, so while the device looks similar to other VR headsets, it is aimed at PC gaming. The Vive has two 1200 by 1080 displays that run at 90 frames per second, two tracked wireless VR controllers, and an array of sensors to allow for full room tracking or what HTC calls a “room-scale 360-degree solution.” HTC Vive is at the highest price-point of the early consumer versions: US$799 (plus a computer).
While games for Valve’s Steam VR, and also projects such as A Thief in the Shadows, are created in a computer graphics environment, there is also the option of filming, which requires a 360-degree camera rig and all of the post-production software for the tricky task of stitching together the footage, and adding binaural sound. (‘360-degree’ is fast becoming the descriptor for video in the round ― on a spherical surface, so, for the nerds reading along: ‘two-Pi-steradian’ is the more accurate naming.) Early camera rigs for VR range from basic six-camera GoPro rigs to multiple Red EPIC cameras and Light-field technology. Filming panoramic environments is not new, however, until recently, viewers did not have the same power to move through a space in ‘360-degree’ video. At GDC, in 2015, NextVR showed off Light-field camera technology using a multiple-EPIC rig that shoots 6K resolution at 80 frames per second and, critically, in a stereoscopic format that captures not just the visuals, but the three-dimensional depth, shape, size of an environment and of all the objects in that captured environment. This technological breakthrough promises the viewer a heightened sense of immersion never before possible in VR. Viewers can now look around a live-action scene.
“By incorporating dynamically-generated 3D geometry with ultra-high resolution stereoscopic video, we’ve created most the vivid and life-like VR experience currently possible.”―Dave Cole, GDC: Where Will Immersive Tech be in 2 Years?
There is almost no major visual effects studio who is not at least exploring cinematic games or VR. Side Effects Software has become a leader in the technology of VR having made its highly procedural software, Houdini FX, available as a plugin to other development software, namely Unity Game Engine and Unreal Engine, to drive the visual effects simulations.
VR in 2018
This article has delved into the current state of VR, and hinted at the direction of its industry. Perhaps the more meaningful technology, and the next thing after VR, is Augmented Reality (AR) ― the more transparent, more integrated, more social headset of the future (discussed in the earlier article: What is AR?). AR is also here, now, and its definitive experience is also yet to come. It will no doubt build on lessons learned in the development of VR, and will continue to develop out of experiences based in the mobile space. Unlike VR, AR allows us to see the room we’re in, and friends can react and see each other while also seeing virtual information such as characters and graphics. VR isolates the viewer in a virtual world, while AR holds the promise of a virtual environment plus a visible connection to the real world (with less risk of running into things).
“We’re trying to make it cool and mainstream and something everybody wants to use. Now, what we have today isn’t that, but it’s a good step on the path. [As] the cost goes down, as the performance goes up, as it becomes something that’s light, comfy and easy to wear every day, I think virtual reality will get to the point where it doesn’t have this stigma of being this crazy, cyberpunk, science-fiction technology, and it actually becomes something that people use everyday.”―Palmer Luckey, Oculus
Think back to Virtuality and Sega VR ― or look them up if you don’t remember or weren’t around in the early 90s, when VR was thought to be what would save the arcade. Since then, VR (and the arcade) faded from the spotlight. Is it fair to question whether VR ― this time ― is here to stay? Consider that, also since the 90s, we have seen the Matrix movies and we have imagined being in a virtual environment and moving around, and VR ― this time ― gives a glimpse into the look and feel of the future that popular culture has pictured. Have you seen VR lately?
We are still in its early days, and it’s okay to let early adopters work out the kinks for a few years, until Ready Player One promotes its theatrical release with a companion VR trailer ― then maybe more will see.