The Talk Show in 3D!
Like he does every year, John Gruber hosted a version of his The Talk Show podcast live after WWDC.1 The past many years he's been able to pull guests from within Apple's executive ranks, and this year was no different with Craig Federighi and Greg "Joz" Joswiak joining – but also for the first time, John "JG" Giannandrea, Apple's SVP of Machine learning and AI Strategy. Yes, Apple uses "AI" on their very own website! Someone should tell them!
Anyway, the addition of Giannandrea was solid as he seemed pretty direct and candid when responding to Gruber's questions about "Apple Intelligence" – yes, including Siri.2 He could also go deeper on some of the new tech being used and developed here, with some nice assists from Federighi. The AI content creation/moderation/augmentation discussion was especially interesting, I thought.
So yes, the content is good and you can watch it on YouTube here. But what was really amazing this year was the fact that they streamed it live in 3D if you happened to own a Vision Pro. While I do own one, I also live in London, so the live show was at 3am my time. Instead, I watched it a couple days later in the Theater app, a Vision Pro app created by the Sandwich team. Here's Gruber:
Not just shooting this event in 3D, but also streaming it live, was entirely the initiative of my dear friend, Mr. Sandwich himself, Adam Lisagor. I was asked a few times this week whether it was Apple who wanted to stream this live in 3D for viewing on Vision Pro. Nope. It was Adam who pitched me on the idea, only about eight weeks ago. I was like, “Well, sure, that sounds awesome, but how in the world would we do that? What camera could we use to shoot with? How would we stream it? What app would people be able to view it in? It’s a great idea but none of this seems possible.” Adam was like “I think we can do it.”
And, son of a bitch, they did it.
I watched about half on the Vision Pro and half on YouTube – it's just so much more compelling within the Theater app (though yes, far more convenient on YouTube!). After the novelty of such an experience wears off, your brain almost makes you believe you're actually there (beyond the ability to look left or right and see yourself alone in a virtual theater even though there are crowd reactions).
Once I started talking with Apple about arranging for guests, we did let them know our plans to shoot and livestream in 3D for viewing in Theater on Vision Pro—an app that, at the time, was in early beta. Maybe even alpha. Apple’s reaction echoed my own: sounds great but how?
It's both incredible and a bit concerning that Apple didn't know how the Sandwich team might be able to do this. Yes, Apple has their own rigs for shooting Spatial Video, but they haven't live-streamed anything yet – presumably the bandwidth would be far too high. At the same time, the fact that a small team can pull this off – might I suggest they acquire the Theater and SpatialGen products and teams that did this in order to populate the Vision Pro with some actual content?
We’re describing it as “3D video with spatial audio”, not “spatial video”, because that’s a more precise description of the effect. True spatial video would be more immersive—“Look left to see Joz, look right to see Gruber”—and the effect we achieved wasn’t quite like that. But what we did get is immersive, and very compelling. To the best of my knowledge, this was the first event livestreamed in 3D for viewing in VisionOS, and while my role was simply as the host of the show, that’s pretty damn cool. I’m just amazed at what Lisagor and his team at Sandwich (and SpatialGen) were able to pull off in a matter of weeks.
This is exactly the type of content that makes Vision Pro shine. Even in this first-ever stab at doing it, it's an incredible way to experience this type of talk. The fact that a group outside of Apple pulled it off is awesome but again, a bit worrisome. I appreciated Joz's history lesson during the show about how most Apple products are written off at first, but the reality here remains that Apple is moving far too slowly with the Vision Pro from a content perspective, if nothing else. It's obviously hard, but obviously it can be done – these teams just pulled it off!
Screenshot these within the Theater app from different seating points. They obviously fail to convey the 3D element...
1 I've been a guest many times on The Talk Show over the years, most recently episode 400 a few weeks ago.
2 Federighi is always good on stage, but I also feel like he filibusters a bit with jokes. Joz is better in pre-recorded, scripted videos, IMO. He's too media trained and says mainly exactly what you expect him to say -- and almost seems to be there mainly to keep Federighi and Giannandrea in line and on the talking points.