In this episode, we sit down with Lucas Rizzotto, a mad scientist, creative technologist and the mind behind Pillow: one of the most innovative MR experience I have ever tried. We will look at how he merges technology and storytelling in innovative ways, the challenges he faced during development and how AI made the impossible possible.
Interview with Lucas Rizzotto
Which project are you most proud of and why?
Lucas Rizzotto: I'm proud of all of them for different reasons, but I'm usually the proudest of the latest thing I made because making anything is hard and making things consistently is even harder. I have a soft spot for the microwave documentary I did. It was really a film, like a single-location thriller, and making that video felt like I became a filmmaker.
Can you explain the microwave project for those who haven't seen it?
Lucas Rizzotto: Sure, back in the days of GPT-3, before it was big and had many restrictions, I looked at AI as an imaginary friend. As a kid, I talked to my microwave every day, so I thought it would be cool to bring that imaginary friend back to life. I took a microwave, put GPT-3 in it, and replaced Alexa with a custom interface. Things got weird when I told the microwave I was inside, and it turned on by itself. The video was about the surreal experience of interacting with AI as if it were real.
I gave my microwave a soul with AI and it tried to kill me
How do you combine emerging tech and storytelling in your projects?
Lucas Rizzotto: For me, stories are how you make people care about things. I create technology that tells a really good story so people can connect with it. In the early days of VR and AR, no one understood what I was doing. But if you create something cool and tell a compelling story around it, people will be interested. I always wanted to be a filmmaker, so combining technology and storytelling comes naturally to me. It’s not just about technical skill; it’s about having a unique perspective.
Can you elaborate on the importance of having a unique perspective?
Lucas Rizzotto: Having a unique perspective means approaching the medium from an angle others aren't. Even if the project is technically simple, a unique take can make it impactful. Early in my career, I built projects to impress hackathon judges and won, but those projects were generic. When I started building things I was passionate about, I made some of my best work. A good metric is asking if your project wouldn't exist without you. Those are the projects with a unique perspective.
What was the most technically challenging project you've worked on?
Lucas Rizzotto: Definitely Pillow, the mixed reality app I made. The difference between making a prototype for a video and creating a full-fledged product is huge. The last 20% takes 80% of the time. Pillow was a major challenge because shipping a product requires a lot more work than just making something for a hackathon. It's about consistently putting in effort and making sure it works reliably for users.
What was the inspiration behind your project Pillow?
Lucas Rizzotto: Pillow was inspired by the desire to create a truly relaxing and immersive experience for users while they lie in bed. It started with the idea that most VR and AR applications required users to be active or in a specific physical position, which I found limiting. By focusing on a lying-down experience, I could design a variety of relaxing activities like stargazing, fishing, and interactive bedtime stories. It’s about making technology comfortable and accessible in everyday life.
Pillow - Mixed Reality in Bed (Trailer)
Can you share a technical challenge you faced while developing Pillow and how you overcame it?
Lucas Rizzotto: One major challenge was making Pillow compatible across different physical environments. For instance, users would start the app in their living room and then move to their bedroom, causing the app to seemingly stop working. We had to account for these transitions and design the app to be resilient in various real-world settings. It involved extensive testing and iterating to ensure a seamless user experience regardless of their physical space.
How was the design process of a laying down experience and what have you learned?
Lucas Rizzotto: Designing for lying down in XR involved addressing issues like arm fatigue and head positioning. We discovered the best angle for lying down is 68 degrees, not 90. We tested extensively to find solutions. Mixed reality is challenging because you have to account for various play spaces, from bunk beds to warehouses. Designing experiences that work in all these scenarios requires a lot of iteration. Despite the difficulties, it's gratifying to see Pillow having a real impact.
Can you explain the asynchronous social interactions in your fishing game within Pillow?
Lucas Rizzotto: The fishing game in Pillow allows users to fish for voice messages left by other users. Each fish represents a message, and every day brings a new prompt, creating an evolving pond of conversations. This asynchronous interaction mirrors how we use email or social media, where you can engage at your own pace. It adds a layer of convenience and intimacy that’s often missing in real-time interactions.
How do you handle content moderation in Pillow’s social features, like the fishing game you just mentioned?
Lucas Rizzotto: We use AI, specifically GPT-4, for content moderation. When users leave voice messages in the fishing game, we transcribe these messages and run them through the AI to screen for inappropriate content. GPT-4's strictness in language use makes it an effective moderation tool, ensuring that the messages are safe and suitable for all users. This automated system allows us to maintain a positive and respectful community without extensive manual oversight.
How has AI influenced your development process, especially with Pillow?
Lucas Rizzotto: AI has been transformative. For instance, in Pillow, we use AI for content moderation, generating interactive bedtime stories, and even synthesizing voice content. These applications of AI allow us to do things that would be impossible or highly impractical for humans, like writing detailed information about every star for our stargazing experience. AI enables us to expand the scope and quality of our offerings significantly.
What is next for you and Pillow?
Lucas Rizzotto: Pillow is my main focus, but I also enjoy side projects and hobbies. I'm documenting the creation process of Pillow, sharing the interesting and funny parts. I'm also working on a stand-up comedy act because I love interacting with live audiences. I'm exploring different ways to communicate and share my journey with Pillow, and I'm also starting to do advisory gigs to help companies avoid development mistakes. Mixed reality is cool but challenging, and I'm excited to keep pushing its boundaries.
That’s it for today,
See you next week
