麻豆村

Physical Presence Pet (PPP), a project developed by a team of students, began with a simple idea: making a VR pet that could substitute for the real ones they鈥檇 left behind to come to 麻豆村. What they ended up creating was much bigger: an unprecedented experiment in virtual companionship that cut across disciplines and anticipates new ways of integrating technology into our world.听

For students at the ETC, technology is integrated into their work from the start. Students are introduced to cutting-edge immersive technology their very first week: virtual reality (VR) headsets are standard, and immersive extended reality (XR) experiences that combine the digital world with our physical one are a constant topic of discussion.听

The members of PPP wondered what they could do with them. 鈥淲e started thinking about the idea of a virtual pet ... one that would be a companion for you while you do work or go about everyday life,鈥� said Jerry Zheng, a first-year student at the ETC. 鈥淏ecause when we came to the ETC, many of us moved here from another country or into an apartment that didn鈥檛 allow pets 鈥� it was hard to take them with us.鈥�

They pitched it to a committee of ETC faculty members, including . When he heard their initial pitch, he was surprised. He鈥檇 proposed something similar himself, and had even begun prototyping it. 鈥淗e took the head off a teddy bear, and put a Meta Quest controller in its neck,鈥� said first-year student and PPP technical producer William Zhang. 鈥淭hen, he made a VR teddy bear head that moved when you touched the controller.鈥�

鈥淚 wanted to explore what happens when your eyes see one thing, and your hands feel another. It was about hacking your senses,鈥� Ham said.听

With the team鈥檚 agreement, they merged their ideas with Ham鈥檚. 鈥淭he tactile plush element became a big part, and that really made it different from any other VR project,鈥� Zhang said.

听

Soft Technologies, Hard Challenges

What followed was months of experimentation, troubleshooting and collaboration.听

Early testing revealed a major issue: when embedded in a plush, the Meta Quest controller wouldn't track properly. That challenge sparked an unexpected pivot to a new technology: the Apple Vision Pro.

The Vision Pro鈥檚 gesture and eye tracking eliminated the need for traditional controllers, and its ability to overlay digital onto physical spaces opened new possibilities. 鈥淣o previous project has used the Vision Pro,鈥� Zheng said. 鈥淲e wanted to find out what it could do.鈥�

They named the creature they planned to bring to life Luceal and crafted a plush version of her out of soft, white fur. Inside, she was packed with handmade sensors the team learned to build with a little help from .

鈥淚 showed them how to create custom soft sensors and textile circuits using conductive fabrics, yarns and threads. E-textiles were new to the team, and they were excited to create custom soft sensors that could be subtly and smoothly integrated into their stuffed animal form,鈥� Robinson said.

By embedding them in Luceal, users could pet and squeeze the plush as much as they wanted without damaging it. 鈥淲e created the exact same function as commercial variable resistors, but with the flexibility of cutting them in any shape we want,鈥� said first-year student and PPP programmer Brian He.

But the most complex challenge the team faced was aligning the digital and physical pets in space 鈥� their self-described 鈥渉oly grail.鈥� 鈥淚f people see a virtual thing floating in space 鈥� and the physical plush is somewhere else 鈥� they鈥檒l want to touch the virtual one,鈥� said Zhang.

For much of the semester, the digital and physical Luceals were separated due to the Vision Pro鈥檚 restrictions on spatial data and the imprecise nature of tracking hands in motion. A last-minute idea from turned out to be the key.

鈥淭he students' ideas were way ahead of what the technology could handle,鈥� Schell said. 鈥淭hey couldn't find a way to reliably track the position and orientation of a stuffed animal. I'd suggested to the team that they should look at tracking what they could track, and in this case that ended up being the player's hands. They embraced that idea, and they found a clever way to overlay the virtual pet onto the real one.鈥�

With only a single week left, the team made their holy grail a reality.听

A Team Effort, In All Senses

Physical Presence Pet required every team member to work across multiple disciplines 鈥� from user experience (UX) research to game development to spatial rendering. 鈥淚t combined art, code, physical manufacturing, embedded systems, electronics,鈥� said He. 鈥淎ll of them had to be present for the final product to work.鈥�

Each team member brought their unique specializations and faced unique hurdles. First-year student and PPP technical artist Paige Li had to work around the inherent limitations of new technology. 鈥淲hat I saw when developing on my computer wasn鈥檛 the same on the Vision Pro 鈥� colors and scale looked completely different. I had to find a lot of workarounds to get the visual effects to work.鈥�

For Joy Lim, sourcing the fabric posed a huge challenge. 鈥淎fter we decided on one, we found out it was out of stock almost everywhere,鈥� Lim said. 鈥淚 ran to three different fabric stores and luckily I found the last eight and just grabbed all of them. And then I was terrified every time I cut the fabric, because we had a limited amount.鈥�

There were also challenges no one could have expected. Halfway through the semester, Zheng was rushed to the hospital by fellow team member Brian He. After emergency surgery, it was unclear if he could continue.听

鈥淲e scheduled many, many meetings with Derek and other staff about how we should handle it,鈥� said second-year student and PPP assistant producer and animator Sophie Huang. 鈥淏ut ultimately, we still figured things out in time for our midterm presentation that week.鈥澨�

Zheng recovered, and he came back to work ready to contribute 鈥� still managing to design a special Luceal mobile app they showed off in their final presentation.

Luceal Lives On

Though the semester is over, Physical Presence Pet continues on.

The mobile app Zheng designed allows users to customize Luceal鈥檚 appearance and take selfies with her. 鈥淲ith XR, it鈥檚 so hard to share what you see with other people,鈥� he said. 鈥淲hat if you just want to show off your Luceal?鈥�

And while no formal post-semester plans are set, the team is optimistic and discussions about next steps have already begun. 鈥淚 think there鈥檚 some interesting potential to extend it during the summer,鈥� said He. 鈥淭here might be companies out there who鈥檇 be interested in this hybrid of physical and virtual pets.鈥�

鈥淭his is a perfect entrepreneurial product in my mind,鈥� Ham said. 鈥淎nd now they鈥檙e at the point where they鈥檙e asking 鈥業s this a product? Is this a company? Is this something you develop and try to sell to a toy company?鈥� They can talk about it 鈥� quite literally 鈥� as a project with multiple touch points.鈥�

鈥淭he project was a wonderful exploration, and I feel sure that five or 10 years from now, stuffed animals with augmented reality overlays will be a popular toy,鈥� Schell said. 鈥淢y favorite thing about the ETC is how the students are always showing us concrete visions of the future.鈥�