Quest 2 Passthrough API

It there any plan to support passthrough for Quest 2 via the newly released API?

I am still very unsure that the passthrough API gives a really useful result. We could at some point experiment with it anyway. A priori I believe that it might work for “plain old AR” rendering, by which I mean have a virtual model floating in front of a blurry grayscale image of the real world. This assumes you’re OK with your virtual model having no physical connection to the real world, and don’t really look too closely at the grayscale stuff, because its actual 3D position does not exactly match what you’d see with your eyes—this is a problem I notice quite well with the grayscale image when I set up the guardian boundaries now.

For anything more complex than this “plain old AR” rendering, I suspect that the video and tracking quality is insufficient, and that matching real-world positions to virtual-world ones needs to be carefully redone all the time and so isn’t really useable for more than a demo. Again, I may be wrong about that, but that’s what stopped me from considering it seriously so far.

1 Like

For an example of what I mean: I strongly suspect that the image from the Occlusions paragraph of is hiding the fact that you can’t get that result in a way that convinces the user. The hand-tracking adds a delay which is mostly likely different from the one of the passthrough image (which needs to be lower, or the user will feel nauseated when turning the head). So the virtual hand won’t be at precisely the correct position unless you keep your hand very still, which was probably exactly what they did here to get that picture.

More generally, I do not know how well the big switch on the wall really appears to be attached to the wall; I would suspect that it wobbles around whenever you move or turn your head…

IOW I don’t think the Quest or Quest 2 have the necessary technology built into them to get really good results for this. It is based on a feature that was meant for positional tracking—which the Quest is, as far as I know, the best headset at doing—but not so much more. Same as with the hand-tracking, actually: it was added later on top of the positional tracking cameras, and it roughly works, but it does not work well enough to be really useful.

I think what you call “plain old AR” is exactly what I was thinking of. Just using it as the ambient environment to work in. Give a bit better idea of scale & maybe help anchor people new to VR who I am showing off models to.

Although there might be some interesting workflows to explore where we combine VR with using keyboard & mouse.

I’m so glad someone asked this question - I was going to ask it myself if I didn’t find anything in the forum search.
I would really like to see passthrough implemented as a feature so that I can overlay renovation plans in VR Sketch on top of existing geometry in the real world. I would imagine it like a slider that changes the opacity of the passthrough in VR.
Currently, I can sort of cheat this by using the fade to passthrough feature on the Quest 2 when I am on the edge of the room scale boundary line while in the VR Sketch model. It just takes lining up a datum point to existing geometry and recentering your ‘home’ position to align with the datum point, but definitely helps validate scale and position of objects in the CAD model to the real world.

I hope the devs see this and respond! Thanks for creating such a great product and supporting STEM education with free licences! I’m using VR Sketch to enhance the Graphics Technology class I teach (high school - 15 year old students in that particular class). Happy to share with you one of my student group’s projects from this academic year that capitalises on VR Sketch’s functionality:

Implementing Passthrough API would take this unit of work to the next level!

1 Like

Hi! It seems there is a lot of interest in at least trying out the AR mode of the passthrough. Hearing about going on the edge of the room scale boundary line, it makes us a little bit more confident that it can really be useful. Our own experience is that the scale of the real world that you see in Passthrough is slightly wrong, which is at first quite disorienting too; that’s why we don’t really buy the argument that this slightly-off, grainy, grayscale AR mode gives first-time users a better grip on VR. However, you are giving a different use case. We will try to implement it and see. (So far we’re testing with a Quest 1 to make sure VR Sketch still works there, but I think this new API is only available on Quest 2. It will have to wait until the end of the holidays.)