It there any plan to support passthrough for Quest 2 via the newly released API?
Quest 2 Passthrough API
I am still very unsure that the passthrough API gives a really useful result. We could at some point experiment with it anyway. A priori I believe that it might work for âplain old ARâ rendering, by which I mean have a virtual model floating in front of a blurry grayscale image of the real world. This assumes youâre OK with your virtual model having no physical connection to the real world, and donât really look too closely at the grayscale stuff, because its actual 3D position does not exactly match what youâd see with your eyesâthis is a problem I notice quite well with the grayscale image when I set up the guardian boundaries now.
For anything more complex than this âplain old ARâ rendering, I suspect that the video and tracking quality is insufficient, and that matching real-world positions to virtual-world ones needs to be carefully redone all the time and so isnât really useable for more than a demo. Again, I may be wrong about that, but thatâs what stopped me from considering it seriously so far.
1 Like
For an example of what I mean: I strongly suspect that the image from the Occlusions paragraph of https://developer.oculus.com/documentation/unity/unity-customize-passthrough/#occlusions is hiding the fact that you canât get that result in a way that convinces the user. The hand-tracking adds a delay which is mostly likely different from the one of the passthrough image (which needs to be lower, or the user will feel nauseated when turning the head). So the virtual hand wonât be at precisely the correct position unless you keep your hand very still, which was probably exactly what they did here to get that picture.
More generally, I do not know how well the big switch on the wall really appears to be attached to the wall; I would suspect that it wobbles around whenever you move or turn your headâŚ
IOW I donât think the Quest or Quest 2 have the necessary technology built into them to get really good results for this. It is based on a feature that was meant for positional trackingâwhich the Quest is, as far as I know, the best headset at doingâbut not so much more. Same as with the hand-tracking, actually: it was added later on top of the positional tracking cameras, and it roughly works, but it does not work well enough to be really useful.
I think what you call âplain old ARâ is exactly what I was thinking of. Just using it as the ambient environment to work in. Give a bit better idea of scale & maybe help anchor people new to VR who I am showing off models to.
Although there might be some interesting workflows to explore where we combine VR with using keyboard & mouse.
Iâm so glad someone asked this question - I was going to ask it myself if I didnât find anything in the forum search.
I would really like to see passthrough implemented as a feature so that I can overlay renovation plans in VR Sketch on top of existing geometry in the real world. I would imagine it like a slider that changes the opacity of the passthrough in VR.
Currently, I can sort of cheat this by using the fade to passthrough feature on the Quest 2 when I am on the edge of the room scale boundary line while in the VR Sketch model. It just takes lining up a datum point to existing geometry and recentering your âhomeâ position to align with the datum point, but definitely helps validate scale and position of objects in the CAD model to the real world.
I hope the devs see this and respond! Thanks for creating such a great product and supporting STEM education with free licences! Iâm using VR Sketch to enhance the Graphics Technology class I teach (high school - 15 year old students in that particular class). Happy to share with you one of my student groupâs projects from this academic year that capitalises on VR Sketchâs functionality:
Implementing Passthrough API would take this unit of work to the next level!
1 Like
Hi! It seems there is a lot of interest in at least trying out the AR mode of the passthrough. Hearing about going on the edge of the room scale boundary line, it makes us a little bit more confident that it can really be useful. Our own experience is that the scale of the real world that you see in Passthrough is slightly wrong, which is at first quite disorienting too; thatâs why we donât really buy the argument that this slightly-off, grainy, grayscale AR mode gives first-time users a better grip on VR. However, you are giving a different use case. We will try to implement it and see. (So far weâre testing with a Quest 1 to make sure VR Sketch still works there, but I think this new API is only available on Quest 2. It will have to wait until the end of the holidays.)
2 Likes
Hi!!! I am trying to develop an app that lets me take measures of rooms using the passthrough api. There is an app that already does this, and you should have a look at it as it does quite a good job. The only thing that lacks is better looking assets and textures, but the idea is really cool.
I think that if vrsketch includes passthrough api features it would be ground breaking as no one else does it yet (apart from this guy, but his approach is more on the VR games side rather than for real architecture use).
In my case, I managed to grab the measures of my apartment, and with the help of the developer I got the 3d file (FBX), and then modified it in sketchup. After that, I took the model into unity and developed a mixed reality app that allows me walking in my own house and see the modifications we have planned for our living room and our kitchen. Have a look at my video demo-casting my approach:
I finally managed to get the passthrough working in the Quest 2 version. I donât know if it would work in the PC version with Oculus Link, but maybe. I can make a development version of VR Sketch which displays the passthrough in the background, for the people reading this forum topic. Would you be interested in the standalone Quest, or in the PC version with Oculus Link?
(Note: as stated previously, I remain convinced that the passthrough quality is by far not enough for a proper experience. But if it doesnât look difficult to make a checkbox in VR Sketch to use it instead of the skybox, then why not.)
2 Likes
Hello Arigo! For me it makes sense to have it as a stand alone application runing on the Quest⌠that would allow an architect going to a clients property and sketch the modifications the customer wants. Please have a look at the app I mentioned in my post (Custom home mapper) and give it a go. It costs only 8$. Check this video to know better what it is capable of. If you could bring that funcionality to SketchVR it would be GOLD
Check this tutorial video on how it works: https://www.youtube.com/watch?v=Ww8jUW_UfHY
IMO now that airlink is a think, standalone is somewhat less interesting.
Anyone who is into VR enough to use this app to model in VR will also probably have a VR capable PC would be my assumption.
Standalone Quest would be useful. Model in table top mode (warehouse) would be great
Here is a first version of VR Sketch for Quest which is able to display the passthrough:
There are two modes: the standard grayscale image with the Sketchup model fully opaque on top of it; and a variant where an edge-enhancement algorithm is run and you get black lines, which are displayed even through the Sketchup model. To enable these two modes, go to the settings dialog, âBackgroundâŚâ button on the first page. The two versions are basically two new built-in âbackground imagesâ.
2 Likes
Reminder: installation instructions at https://vrsketch.eu/docs-getting-started-quest.html section âInstallation without using the App Labâ. Also, once you are set up as a âdeveloperâ, you can disable the guardian in the Quest (settings, in the Developer page). This works well particularly if you use the âPassthrough #2â mode because you should see the roomâs outlines even through your own model in this mode.
Hello Arigo! I have been testing the app and it works great! Amazing job! There is something I want to do but I don´t know if it is possible right now with the app or is something that doesn´t exist⌠If you check the video of the custom home mapper app, you can set the boundaries of a house with the help of the joystick. Example, imagine you have a big furniture in a room and you can´t move it, if you are able to move the drawing line with the joystick you can approximately draw that wall.
Let me know what you think.
You are trying to draw the walls, but canât reach a particular point because there is an obstacle - did I understand you correctly?
At the moment you can find various workarounds using the general flexibility of SketchUp and VRSketch. For example, if there is a table, you can still draw the shape of the walls but higher, and fix it later. Or you can drag the whole model with the grip button, sketch the missing wall, and drag the model back as closely as possible. (What might help here is positioning the model by putting both controllers at known physical points, like in the video.)
But I also see the point of being able to reach places where the controller cannot physically go, either behind an obstacle or to draw the correct height of a wall too high to reach, say. I think a special option could be added: the tool usually takes action just in front of the controller, but we could imagine configuring it to be farther away, much like drawing with a pencil held at the end of a stick instead of directly in your hand.
Hi Arigo,
Thanks so much for looking into this. For our use case in an educational context, I would prefer the PC version with Oculus Link. We have had issues getting the Quest 2 connected and working on enterprise Wi-Fi (which schools use), but have managed to get it working more reliably by pushing the SketchUp model from a PC/laptop directly to a wired headset via Oculus Link.
I will test out the the APK at school when Iâve got some time and report back soon.
I will look into it. Right now Oculus seems to consider passthrough with Oculus Link as a âdeveloper-onlyâ feature. I think you should be good if you already registered as developer to install an older or non-official version of VRSketch on the Quest (without the Link).
1 Like
I think it will be very difficult to support the Passthrough API âofficiallyâ on the Oculus Link version.
May I suggest an alternative: at one point I had troubles with my wireless network, but managed to work around it in the following way. On Windows 10, âSettingsâ, âMobile wireless access pointâ, âActivateâ it (top button). This enables âShare my Internet connection via⌠Wi-Fiâ. You can do that whether your Windows machine is itself connected to Internet via Wi-Fi, or a network cable, or even nothing at all, I believe. Note the network name and password here (or change it by clicking âModifyâ). Then configure your Quest to connect to this new Wi-Fi network. As long as the Quest remains in range, then it will get a very-good-quality Wi-Fi network connection directly to your PC. The VRSketch extension in SketchUp should be able to find this Quest on the local network. (Make sure it does: only click the big âGOâ button after it says âlocal wifi connectionâ, not âencrypted connection via the cloudâ.)
2 Likes
Hi Arigo,
I finally found some time to experiment with the Passthrough API during the school holidays.
I documented my test demo with a video:
Thanks again for implementing passthrough! For the record, I prefer the passthrough 2 mode over passthrough 1 but appreciate you including the option for both styles.
Btw, I got around the enterprise Wi-Fi issue by hotspotting both my laptop and Quest 2 headset to my mobile phone as a temporary measure.
I forgot to test your alternative method - will give that a go next time.
Hi Arigo,
The mobile hotspot from my laptop worked well! Thanks again for this tip.