Quest 2 Passthrough API

[Trevor_Jones]

Cool, I did something similar myself. Now that Quest Pro is out imagine the possibilities!

Imagine this with full color passthrough. With fake objects overlaid onto the real world

[arigo]

Thanks Dat!

Quest Pro: we are testing it, and the passthrough works out of the box already in existing versions of VR Sketch. You get a colorful and less-grainy passthrough that is definitely an improvement over the Quest 2. Not a perfect image, but good enough to convince our senses that it is the real world, maybe seen through some kind of slightly uneven glass which induces some minor deformations when you look around. There are no noticeable delays nor other stuff that could create sickness.

The next version will just move the VR Sketch setting to a less hidden place.

[filipejrc]

Hi. I’m loving VR Sketch and the Passthrough.
Is the option of drawing the height of a wall that we can’t reach already available? As in, be able to draw shames at a distance, maybe with the laser pointer, or similar to how the Quest Sytem room setup works.

Thank you.

[arigo]

Hi!

Drawing in VR Sketch works a little bit differently than drawing in Sketchup. In Sketchup, your are moving the mouse over a screen. A screen position can mean any 3D point that is behind that screen’s pixel. So Sketchup contains logic to know which 3D point you mean, i.e. at which “depth” inside the screen you are really pointing. For example, it’s difficult to draw in Sketchup a line that ends at a free-standing 3D position (you have to make use of alignments).

In VR Sketch, we took the opposite approach. Almost always, you specify a real 3D position, and that’s where the controller is. There are only a few exceptions, like laser-selection and painting of faces, where you select an existing face by pointing a laser.

When you draw a room, the way you are supposed to make the wall is to scale and move the model until the target points become easily reachable. That’s the way VR Sketch is designed. Drawing using the passthrough as the basis invalidates that approach…

It’s difficult at this point to merge a completely different approach into VR Sketch, but maybe we could tweak existing approaches. For example, right now you can “lock” one or two axes when you draw a line or a rectangle; this allows you to start drawing a wall (a rectangle) and lock it to be vertical. But you still need to physically move the controller as high as the top of the wall. What I’m thinking now—which would be more useful in your case—is that the locked rectangle should go not up to the same height as the controller, but instead up to a point defined by laser. It might be useful in general to make this change if it allows some positioning-at-distance to be done. I’ll try it out.

Armin Rigo

[filipejrc]

Thank you for your answer.
Something like your suggesting would be incredibly useful, especially for interior designers who could go to someone’s house, put the Quest Pro on, and quickly point the laser at the walls and draw the room layout in seconds. Exactly how the Quest Pro “Room Setup” works when you draw your walls and palying aread.
Thank you.

[arigo]

I’m coming back to this now that VR Sketch 18 has been released, sorry for the delay.

It’s unclear to me how what you have in mind could integrate with the rest of VR Sketch. You basically want something to draw walls on the floor like in the “Room Setup” (maybe as wavy lines? or by placing and connecting points on the floor, without needing to physically bend down?); and then something else to specify how high these walls should go until they reach the ceiling (with a uniform-height ceiling? or something more complex?).

All this is in theory possible to implement, and it would be a quick way to start making a model in your situation. But this looks very custom-made; it wouldn’t have any other more general purpose. If we were talking about Sketchup instead of VR Sketch, it would be the kind of thing you would find in a custom extension made by a 3rd party, and not in the core of Sketchup itself.

Needs more thinking from our side… Maybe at some point we should provide a way to write such 3rd party extensions for VR Sketch?

[smart_glasses_hub]

I’ve been investigating how to use and troubleshoot the Oculus Quest 2’s Passthrough feature for the average user. Here’s a summary of what I found:

• Understanding Passthrough:
  • Passthrough allows you to see your real surroundings through the headset’s cameras.
  • It’s primarily a supportive function with black and white, low-resolution imagery.
• Activating Passthrough:
  • Passthrough Home Environment: Activate this to interact with the real world while in VR, useful for 2D apps.
  • Double Tap Passthrough: A handy feature to quickly switch to Passthrough by tapping the side of the headset.
• Passthrough for Augmented Reality (AR):
  • Passthrough technology has the potential to be used for AR applications, like playing the piano or interactive games.
• Troubleshooting Passthrough Issues:
  • Problems usually relate to tracking and play area mapping.
  • Solutions include cleaning cameras, adjusting lighting, removing reflective items, and restarting the headset.
• Specific Passthrough Problems:
  • Distorted or Glitching View: Clean cameras, modify lighting, remove bright lights, clear clutter, and restart the headset.
  • Double Tap Passthrough Not Working: Ensure the feature is enabled in Guardian settings, delete guardian history, or try a hard reboot.
  • Stuck in Passthrough Mode: Disable double tap, modify lighting, restart the headset, or as a last resort, factory reset.

These insights should help in understanding and effectively using the Quest 2’s Passthrough feature, as well as addressing common issues. Feel free to check out my post for more details on any of these points!

[arigo]

Thank you! Let me add the new features of the passthrough in the Quest 3:

• better quality image than in the Quest 2, and in color

• can be combined with the Room Depth sensors (only in VR Sketch, not elsewhere, so far, I believe). This avoids the visual issue when the VR model is partially outside the real room or just bigger than it: the parts of the VR model that are behind real walls are made almost invisible, as if your real walls were only slightly transparent. Activation: in Settings, Rendering, Passthrough, choose the option “Room depth”. It probably requires allowing “spatial data” for VR Sketch: if you chose “no” when you originally started VR Sketch, you need to figure out where to re-enable that permission in the main menus of the Quest (in “Apps”, “Permissions”).