ArtCenter College of Design
IF: Journal of Interactive + Inclusive Futures

2035 Metaverse Gestural Interfaces Experiments

Carina Geng, Haoran Xu, Zoe Zhang, Duoning Zheng

The Film

2035 Metaverse Gestural Interfaces Experiments

A simulated digital environment.

Meta · verse

Metaverse originated in the 1992 science fiction novel Snow Crash as a portmanteau of "meta" and "universe", metaverse means a simulated digital environment. As time has developed, the imagination of the metaverse has become more and more diverse and people are debating its definition. Some metaverse iterations involve integration between virtual and physical spaces and virtual economies.

Let's start with...

What does METAVERSE mean in our eyes?

It is...

A bridge that integrates virtual and physical spaces.
A future space that goes beyond mimicking the real world to help people achieve things beyond the laws of physics.
A system that combines multiple senses and interactions to provide a more intuitive, immersive and productive experience.

Why will Metaverse come true?


Virtual goods are profitable because they can be replicated and distributed at no cost. (University of Virginia)
New job opportunities: With virtual options: freelancing and remote work will expand. (All Work)
DAO and blockchain will offer metaverse a decentralized economy solution. (Ether)


Making it possible to genetically modify humans to reduce cell defects, fight deadly diseases, and extend people's lives. (Future Science)


5G / VR / AR
Materializing dreams with the internet of senses, allow us to control our sound environment and experience smell, taste, textures, and temperature digitally. (Ericsson)
Digital touch or the haptic feedback of current VR systems. (Ericsson)


Reducing the human need for travel. Eliminating commutes will reduce air pollution and the warming of the planet. (CNBC)
Saving lives by reducing accidents that occur during training to perform dangerous tasks. (CNBC)


Fully in control of their data. (Ericsson)
A more immersive and diverse experience and communication. (iNews)
More possibilities for building long-distance relationships. (iNews)
The public wants to escape from the reality. (US News)


The metaverse is already a full-on political push, with the goal to position the company “far from the controversies of social media”, such as privacy, antitrust, content moderation, and political extremism. (BROOKINGS)
Will be a new playing field for politicians as the voters can now buy digital merchandise to show support for their candidate. (the quint)

2035 Metaverse Gestural Interfaces Experiments

More Insights During Research

2035 Metaverse Gestural Interfaces Experiments

All in all, the development of technology does not necessarily bring good aspects, it depends on how humans balance the 6 aspects just mentioned.

In short, we believe that no matter how perfect the metaverse is, people will more value real-life interactions and experiences. Mixed reality will be the key component of the metaverse

2035 Metaverse Gestural Interfaces Experiments

If people don't balance well...

2035 Metaverse Gestural Interfaces Experiments

If people do it ok...

2035 Metaverse Gestural Interfaces Experiments

The best scenario we can imagine will start with a Comprehensive development of VR and ar technology. Because that’s the basic allows people to enter the metaverse era.

However, experience in Metaverse is NOT PERFECT YET.

Before we go deep into the further explanation, let us introduce our persona - Jack.

2035 Metaverse Gestural Interfaces Experiments

Scenario - Part 1

“Try new interactions and find inspiration to make my own game.”

Jack was first exposed to the concept of Metaverse in the year 2020. He is now a long-time user of the metaverse world. Every morning after waking up, he brushes his teeth and prepares breakfast by playing interactive games of his choice.

In the morning, he meets his friends to play board games together. The friends all join in with their favorite avatars. His room temporarily turns into a virtual game room where they simulate the real plot of the board game.

He is undoubtedly a devoted gamer. After lunch, he usually chooses to spend time on the game to relax.

The advanced technology encourages each player to explore and create their own game world together, and Jack really likes the current model because he has always wished he could share his imaginary world with other players.

However, Jack was well aware that as a philosophy student, he had no technical knowledge or any talent for design. All that could help him was his years of experience playing games. Although today's codeless era can help him develop without the burden. He still wants anyone to help him understand the possibilities of interaction in the virtual world.

Current Problems

What are the barriers to metaverse?

Based on our research of the metaverse, we found that the user experience is only good when these interaction opportunities are addressed.


High learning costs due to the lack of natural and intuitive gestures

Gesture interface design for AR and VR is still emerging and new to users. It requires significant learning costs and therefore hinders the expansion of the metaverse.


Gesture interface & interactions in VR is lacking creativity

Interactions in VR are limited by the classic way we interact with the world. What would we have if we could go beyond the traditional ways of interacting and break the limits of physics?


The virtual world and the real world still have boundaries.

The Metaverse is supposed to be the perfect combination of virtual and real worlds. Users can use the same set of interaction logic (Gestures, commands…) to travel between the real and virtual worlds at will. Any element of the real world should be accessible in the virtual world.

As a result

We want to explore the possibilities of gestural interfaces to make them more usable, intuitive, and efficient.

And users like Jack can apply interactions everywhere as just simple as sending emojis.

Scenario - Part 2

“Try new interactions and find inspiration to make my own game.”

The 2035 Technology Sharing Conference is happening this evening. Jack guessed that maybe this conference would help him find his way around.

He joined the virtual conference and found that the conference was sharing this year's interactive technologies.

The conference had announced a new game - a library of gestures that could be experienced! By purchasing this product, users will not only be able to experience real-life examples of how brand new gestures can be used in immersive gaming practices. They will also be able to use it in any other scenario and space, just like a design system or even a set of emojis.

He is very excited because it means that in the future he can learn and collect any interaction he wants and easily use them in his own game.

He can't wait to open a live stream to experience the game with his friends.

2035 Metaverse Gestural Interfaces Experiments

Our Game

What is our game?

To help gamers explore the different possibilities of gesture interfaces in the future, we have researched current gestures and created a library of creative and intuitive gestures for brand new usage scenarios.

In this game, we created a storyline where the player is in the process of a creepy mental recovery, which requires them to use various methods to escape from two scenarios: the room and the factory.

By scanning the physical manual they purchased, the gamer enters our game to explore and escape. After they finish the game, they can see all the gestures they used in the game displayed in the library.

These gestures are not only applicable to this game but can also be used in any mixed reality. In addition, gamers can pull gestures from the library at any time and use them in their games.

The process

Develop Our Gesture Library

Natural gestures

Drag in space

Gestures for Modification

Move multi-selected items
Slurp different Attributes

Special Gesture

Selection area
2D to 3D

2035 Metaverse Gestural Interfaces Experiments

The process

Gesture Interface Principles

According to Don Norman's article: Gestural Interfaces: A Step Backwards In Usability, we set design principles.


Perceptible & Obvious signifiers.


Whether it is a game progress change or showing the result of object interaction.

Non-destructive operations

hence the importance of "undo"


All operations can be discovered by systematic exploration of menus or Guidance


Operations should work. Period. And events should not happen randomly.


Not only logically, but also systematically. A good set of gestures should be applied across systems to reduce user memory costs.

The process

Possible Signifier & Feedback in Metaverse

According to Don Norman's article: Gestural Interfaces: A Step Backwards In Usability, we set design principles.

2035 Metaverse Gestural Interfaces Experiments

The process

Exploration Insights & Learning Outcomes

2035 Metaverse Gestural Interfaces Experiments

Mimicking the logic of the real world can help users align their mental models of VR interactions with their real-world experiences.

Metaphors can extend the affordance of gestural interactions. There can even be magical interactions in the virtual world.

Object selection in 3D space will always have a higher error rate than in 2D space, as well as being much more complex. Scaling down the interactive elements to the space reachable by gesture interaction can help users to retrieve and select more easily.