How might we create an experience that can bridge the digital and physicals world together using augmented reality?

In this mini-project, I used augmented reality based on a smartphone to create a functional prototype of a pet app. Through out the process, I built, test, and iterated on the features of the application by myself.

However, after testing with other users, I realized that my design did not fully cover the potential use cases of the AR experiences. This was because my testing setup was based on me using the app at my desk and placing the AR pet right on my desktop. But when other users were playing with the app, they tried placing the pet on the floor, and ceiling both near and far, which I didn't account for in my design, therefore breaking the experience all together.

From this project, I learned a lot about the tools used to design and build smartphone-based AR applications. I also got a better understanding of how people interact with AR objects through their phones and how to design for the context of the user.

Learning Unity and AR Development

With no prior experience in working with AR or Unity, I set out my journey by planning out what are some of the features that I wanted to include in my application. This included: placing an AR object in the real world, interacting with said object, and some kind of gameplay mechanic that makes the application not boring.

Next, I went through a large amount of YouTube tutorials, learning the development process one feature at a time.

Developing the Prototype

As I'm learning my way around Unity, I was also building very simple prototypes of the game as well. This included using blocks and spheres as placeholders for 3D models and figuring out how I want the interaction to be.

Goals

• Leverage the interaction between the real world and the digital
• Emphasize the experience of combining real world interaction with digital on-screen interaction
• Creating interactions that involve the user moving within the real world, not just interacting through the phone.
• Make sure the user interface does not impede into the AR experience.

Asset Creation

After I was satisfied with the interactions that I designed, I used Blender to create some 3D assets as well as mocked up the UI in Figma. Here's the progression of my development journey.

Key Features

Placing "Bao-zi" on a surface

• As soon as the app launches, the user's camera will turn on and tries to detect surfaces that the user can place "Bao-zi".
• The user will have to tap on the surface to place "Bao-zi" into the real world.

Feeding "Bao-zi"

• The user can feed "Bao-zi" with a cookie.
• To feed "Bao-zi" a cookie will appear in front of the camera. The user will then need to move the phone close to "Bao-zi" so that it can reach the cookie.
• Once the cookie touches "Bao-zi", it will eat the cookie jump for joy from the delicious snack.
• Eating the cookie will increase "Bao-zi's" health as well.

Playing the asteroid minigame

• Being a super energetic pet, "Bao-zi" can get bored easily. To have some fun, the user can play a minigame with "Bao-zi".
• The goal of the game is to avoid a set number of asteroids falling from the sky.
• If "Bao-zi" is hit by an asteroid, it will lose a chunk of it's health.
• The user will need to feed "Bao-zi" before the health decreases to zero.

User Testing

After finishing the prototype, I tested the app with my friends and family. Here's what I found:

• Users were confused what they're suppose to do to place the Pet in the real world since the indicators were not clear.
• One user tried to place the pet on the floor, which resulted in a weird scaling where the pet was too small to interact with.
• The interaction for feeding the pet with a virtual cookie was not clear, users tried to swipe up to feed, similar to throwing a Pokéball in Pokémon Go. This functionality was not designed.
• Users did not understand that the joystick is used to move the pet when in the interactive game.

Demo Video

What I learned

Firstly, I learned that it is very important to make sure that the users understand the interactions that I design through uses of signifiers or instructions. Since AR is such a new platforms, users may not have the same expectations on how things could be interacted with.

Secondly, I will need to make sure I understand the context of how the application is going to be used. While I was developing and testing the application myself, I only tested by placing "Bao-zi" on the table in front of me. This resulted in me designing all of the other interactions around a tabletop experience. However this experience instantly fell apart when a user placed "Bao-zi" on the floor, resulting in the models being too small to see. This taught me that thinking about all the possible context for the experience is extremely important for an AR experience since the experience is not tied to a 2D screen, but the 3D world around us.

Lastly, I also learned that it's important to keep interactions consistent. Users will always be bringing in their own past experiences with other applications/games. Therefore, when designing a feature, it's important to keep the interactions consistent to avoid confusing the users.

‍