The US Military Is Building Its Own Metaverse

On May 10, two fighter pilots performed a high-altitude proto-metaverse experiment. A few thousand feet above the desert of California, in a pair of Berkut 540 jets, they donned custom AR headsets to connect to a system that overlaid a ghostly, glowing image of a refueling aircraft flying alongside them in the sky. One of the pilots then performed a refueling maneuver with the virtual tanker while the other looked on. Welcome to the fledgling military metaverse.

It isn’t only Silicon Valley that’s gripped by metaverse mania these days. Just as tech companies and corporations are scrambling to develop strategies for virtual worlds, many defense startups, contractors, and funders are increasingly talking up the metaverse, even if its definition and utility aren’t always clear.

The key technologies needed for the metaverse—augmented and virtual reality, headmounted displays, 3D simulations and virtual environments built by artificial intelligence—are already found in the defense world. The result is a lot less polished, cutesy, and spacious than Mark Zuckerberg’s virtual world vision, but that’s partly the point. And there’s a good chance that the underlying tech could take off, even if it stutters in the civilian realm.

A mix of augmented reality, artificial intelligence, and video game graphics, for instance, have enabled fighter pilots to practice dogfighting against virtual opponents, including Chinese and Russian warplanes, while pulling several Gs. Red 6, the company that’s developing the technology, says this delivers a far more realistic test of a pilot’s abilities than a conventional flight simulator. “We can fly against whatever threat we want,” says Daniel Robinson, founder and CEO of Red 6. “And that threat could be controlled either by an individual remotely or by artificial intelligence.”

Red6’s AR technology has to work in more extreme conditions, with lower latency and higher reliability than consumer AR or VR headsets. Robinson adds that the company is now working on a platform that will allow many different scenarios to be represented in augmented or virtual reality. “What we’re building is really a military metaverse,” he says. “It’s like a multiplayer video game in the sky.”

Metaverse-related ideas are already part of some of the latest military systems. The high-tech helmet for the new F-35 fighter jet, for instance, includes an augmented reality display that shows telemetry data and target information on top of video footage from around the aircraft. In 2018, the US Army announced that it would pay Microsoft up to $22 billion to develop a version of its HoloLens augmented reality system for warfighters, known as the Integrated Visual Augmentation System (IVAS).

VR and AR have become routine aspects of military training in recent years. In 2014, the Office of Naval Research and the Institute for Creative Technologies at the University of Southern California developed Project BlueShark, a system that allowed sailors to drive vessels and collaborate in a virtual environment. Another effort, called Project Avenger, is now used to help train US Navy pilots. The US Air Force is using VR to teach pilots how to manage aircraft and missions. VR is also used to help treat veterans for chronic pain and post-traumatic stress. And Boeing has created an AR environment that lets mechanics practice working on planes before stepping aboard a real one.

Recently, the US military has begun exploring more complex virtual worlds. There is also growing interest in connecting and combining virtual worlds in a way that resembles metaverse thinking. In December 2021, the US Air Force held a high-level conference involving over 250 people in locations stretching from the US to Japan, via a virtual environment. “The promise is integrating these technologies,” says Caitlin Dohrman, general manager of the defense division of Improbable, a company that develops virtual world technologies, has created sprawling virtual battlefields featuring over 10,000 individually controlled characters for the UK’s military wargames, and also works with the US Department of Defense (DOD). “It is an extremely complex type of simulation, especially given the fidelity that the military demands,” Dohrman says. “You can either have live players who are participating in the simulation or [characters] can be AI-enabled, which is often what the military does.”

Palmer Luckey, the founder of Oculus, a VR company Facebook acquired in 2014, says Zuckerberg’s decision to go all-in on VR and the metaverse created a massive amount of expectation in the commercial world. “Everyone on their quarterly corporate calls, like a week or two later, they’re being asked by investors, ‘What’s your metaverse play?’,” he says.