Today we will dive into the topic of spatial memories!
- During the interview with Wist’s co-founder Andrew McHugh we will discover how Wist allows to capture and experience memories in 3D unlike anything we have seen so far.
- We will look at a less known tool that allows you to turn any 2D video into a spatial and more immersive experience
Interview with Andrew McHugh
What is Wist, and how do you actually capture a spatial memory?
Andrew McHugh: Wist lets you capture any memory just like you capture them right now. You take out your phone, record what feels like a video, but we capture 3D information about the world. We package that together, and then you can step back inside that memory again, either in Mobile AR on your phone or a headset like the Quest or Vision Pro. It feels like you're there, and that's the really magical thing about it. The ability to relive moments in a way that feels truly immersive is what sets Wist apart. It's not just about seeing a video, it's about being able to walk around and experience that memory as if you're really there again.
Do you need any specific device to capture memories?
Andrew McHugh: Right now, we use the LiDAR sensors on the pro model iPhones and iPad. The LiDAR sensor is crucial because it helps us capture detailed depth information, which is essential for reprojecting the image back into 3D. Without it, we wouldn't be able to create the immersive experiences that Wist is capable of.
Is the processing of captured memories done locally or in the cloud?
Andrew McHugh: Right now, it's all processed locally, which is nice because you don't always have the ability to connect to the cloud. It also helps with storage later. Soon, we will have some amount of cloud processing, but we're still deciding which parts will roll out and when. Processing locally ensures that even if you're in a place with poor connectivity, you can still capture and relive your memories.
How do you experience a memory with Wist?
Andrew McHugh: On the phone, it's kind of like a mobile AR experience. You see the real world and then the memory overlaid on it. On Quest and Vision Pro, it’s even more immersive. It feels more like you were there because you just feel completely surrounded by the memory. It's hard to describe other than just saying “you are there again”. The ability to re-experience a memory in such an immersive way is incredibly powerful. It can be quite emotional, bringing you right back to the moment as if it’s happening again.
How does a spatial memory capture in Wist differ from a spatial video on Vision Pro?
Andrew McHugh: On Apple's side, they capture directly from two cameras and feed that directly back into your left eye and right eye. It's powerful but has limitations when you move your head. On Wist, we capture the camera, the depth and the pose, making it feel more like there’s “this thing” back in your world. You can move around and wander, experiencing the memory from different angles. This means if you want to follow your kid as they move around, you can do that in Wist, whereas with Vision Pro, you're more fixed in one position, watching the action unfold in front of you but not moving through it.
How do you handle gaps and holes in the memory capture?
Andrew McHugh: We focus on making it feel like you're just taking a video. We use 3D models and we are experimenting with diffusion models to try to fill in some of the holes as you move around. It's a work in progress, and we're iterating and prototyping to get it right. We want to ensure that it works well every time for our users. Filling in those gaps is essential for maintaining the illusion of being there, and while it's challenging, it's also a big part of what makes the technology so compelling.
How do you envision re-experiencing these memories becoming a part of our lives?
Andrew McHugh: Right now, Wist is a single-player experience, but this year we’re rolling out social and multiplayer features. We’re focusing on sharing with close friends and family because capturing a spatial memory feels intimate. I think the intimacy of these moments requires a close-knit sharing model. We’re also exploring ways to surface these memories, like notifications or home screen widgets, to remind users of their captured moments. This is about making these memories a more integrated part of your life, not just something you look at occasionally but something that can pop up and surprise you, bringing joy and nostalgia.
What led you to leave Samsung and pursue capturing spatial memories?
Andrew McHugh: Memories have always been special to me. Life events, like the passing of my grandmother, made me realize how important it is to capture and relive these moments. I saw the technical capabilities on the iPhone and Quest reaching a point where something could exist in this space. It felt riskier to stay at Samsung and not pursue this passion for creating something that should exist. I felt I had the skills and the vision to make it happen, and that drove me to take the leap.
What’s next for Wist, and how do you plan to use AI in your app?
Andrew McHugh: We launched the TestFlight on Vision Pro and are continuing to build out features to match those on Quest and iPhone. We’re also using machine learning models to enhance captures, improve depth accuracy, and refine segmentation. Social and multiplayer features are big goals for this year. AI will help us upscale images and fill in gaps, making the captured memories even more immersive and accurate. The potential of AI to improve the fidelity and realism of our captures is huge, and we’re excited to see how it can help us create even more compelling experiences.
Check out the full interview right here 👇
https://www.podpage.com/xr-ai-spotlight/how-to-capture-and-relive-memories-in-3d-without-a-vision-pro/
Product Spotlight: Turn any 2D video to 3D with Owl3D
Owl3D is an AI-powered 2D to 3D video converter. The tool goes beyond the SBS effect you might be familiar with and creates depth maps that add an extra layer of realism to the experience. Unlike many other tools, it is a software you need to install on your machine and runs 100% locally.
Download here: https://www.owl3d.com/
You have various settings you can play with including the “pop-out effect” and the strength of the 3D.
You can also pick various exporting options (even just the depth maps) and to experience in VR you can just drag the output in the video folder of your Meta Quest and use the built-in player.
That’s it for today!
See you next week
