The end goal of Apple’s augmented reality program is currently a mystery, particularly in light of a recent DigiTimes rumor claiming that Apple disbanded the team for responsible for its AR head-mounted display. This rumor flies in the face of a slightly earlier report from analyst Ming-Chi Kuo claiming we’d see some kind of Apple AR headset as early as next year.
There’s little doubt, though, that Apple is still championing augmented reality with a zeal we find in few other competitors, and the progress in the new ARKit 3 (and so in iOS 13 and iPadOS 13) feels like a great leap forward. That leap is so great, in fact, that the many of the features only work with iPhones or iPads running A12 chips or better. (And for that matter, that progress strongly suggests that Apple is designing a headset or glasses that will allow these features to shine, but anyway.)
The slight downside is that current phones can only use this technology with machine learning technology tied to the iPhone’s rear camera, so you won’t always get the fluid captures you’d expect from the front-facing camera’s TrueDepth sensor. As we’d hoped, though, Apple is rumored to be including VCSEL time-of-flight sensors in an upcoming iPhone’s rear camera, but we likely won’t see them until 2020. So maybe we won’t have glasses anytime soon, but at least the iPhones and iPads may be better.
On the more immediate horizon, here’s what you can expect in Apple’s latest devices once iOS 13 and iPadOS 13 drop later this fall.
People occlusion
An easy way to understand how occlusion works is to think of an eclipse. When the moon passes in front of the sun, it’s occluding it. In AR terms, that means a Pokémon you find in Pokémon Go’s AR view might appear to stand behind chairs and other furniture. As iOS currently works with ARKit 2, the illusion of “reality” only works if the Pokémon is “sitting” on a flat space.
ARKit 3, though, can “read” the floor and figure how where the digital objects should stand in relation to any real-world people. Apple showed how this could work (with much applause) with footage of a woman walking through a living room littered with large digital chess pieces and a toy soldier. After the keynote, Apple let visitors see it in action with a game featuring two iPad-wielding players who tried to paddle a giant digital ball toward huge bowling pins at the end of the “court.”
Occlusion also allows for “green-screen” effects much like those used for special effects in modern motion pictures.
Motion capture
ARKit 3 supports real-time body tracking, which means it’s able to study the movements of a person’s joints and mimic them with an animated avatar. It’s a fun-looking feature, and I like to think that refined future versions could allow for something like full-body Memoji. Apple only gave this feature a few seconds, but it could end up being the most influential.
But it needs some refinement. Even Apple’s presentation at WWDC showed an avatar with slightly choppy animations. I’ve little doubt that iOS motion tracking will improve once the iPhone’s rear camera gets proper time-of-flight sensors.
Collaborative sessions
Augmented reality as we know it is largely a solitary experience (at least on iPhones), but ARKit 3 changes that by introducing collaborative sessions that let people play (or work) within the same AR spaces.
At WWDC, two representatives from Mojang showed how this would work in the upcoming Minecraft Earth by building a castle together. The demo was also a great way to show how ARKit 3’s other features work in practice, as when the developers illustrated occlusion by walking behind digital flowers.
Simultaneous use of front and rear cameras
ARKit 3 now lets developers track your environs with both the front and rear cameras at once. I haven’t had a chance to see this feature in action yet, but it sounds as though it’ll be absolutely devastating for your iPhone’s battery.
In practical terms, this feature would allow iOS users to use their faces to interact with digital objects that have been mapped out with the rear camera, which could lead to games involving other characters reaction to your facial expressions. (It also sounds like it’ll make a great Accessibility feature.)
Tracking for multiple faces
Apple’s TrueDepth sensors are already impressive when they’re used with one face, but ARKit 3 allows your iPhone or iPad to track up to three faces at once. It’s a cool feature to have, even if it sounds as though using it will require all three people to scrunch in close. Maybe it will be good for some Memoji-based singing sessions?
This feature will only work with devices with at least an A12 chip and a TrueDepth sensor, and right now that means the iPhone X, iPhone XS, iPhone XS Max, iPhone XR, and iPad Pro.
Smaller technical improvements
ARKit 3 also benefits from several less headline-grabby features that nevertheless enhance the AR experience on iOS. It now lets your device detect up to 100 images at a time, and in line with the introduction of occlusion, it’s also better at recognizing and guessing the sizes of physical objects. AR experiences should be faster in general, and iPhones and iPads can now detect planes in the environment more swiftly.
RealityKit and Reality Composer
Apple wants developers to use cool the new features in ARKit 3, and so it’s made the process easier with AR tools that work with Xcode on Mac and a custom iOS app. RealityKit greatly simplifies the task of incorporating camera effects, animation, physics, and audio into AR projects. Reality Composer aims to help developers who have little experience with 3D modeling by giving them access to a library of AR-optimized 3D assets.
Eye contact correction in FaceTime
Apple didn’t mention its Attention Correct feature on stage, but it was discovered in the iOS 13 developer beta by developer Mike Rundle (who apparently predicted it in 2017). With it’s on during a FaceTime call—and you can toggle it off through Settings if you wish—your eyes always appear to be looking directly at the other person. Normally, you won’t be making direct eye contact since you’re looking at the display, not the camera. As Observant AI CEO Dan Shukin pointed out on Twitter, iOS uses ARKit to achieve this magic. Unfortunately, as of the time of writing, it only appears to work for the iPhone XS and XS Max.