Quest 2 Passthrough API

[Trevor_Jones]

Cool, I did something similar myself. Now that Quest Pro is out imagine the possibilities!

Imagine this with full color passthrough. With fake objects overlaid onto the real world

[arigo]

Thanks Dat!

Quest Pro: we are testing it, and the passthrough works out of the box already in existing versions of VR Sketch. You get a colorful and less-grainy passthrough that is definitely an improvement over the Quest 2. Not a perfect image, but good enough to convince our senses that it is the real world, maybe seen through some kind of slightly uneven glass which induces some minor deformations when you look around. There are no noticeable delays nor other stuff that could create sickness.

The next version will just move the VR Sketch setting to a less hidden place.

[filipejrc]

Hi. I’m loving VR Sketch and the Passthrough.
Is the option of drawing the height of a wall that we can’t reach already available? As in, be able to draw shames at a distance, maybe with the laser pointer, or similar to how the Quest Sytem room setup works.

Thank you.

[arigo]

Hi!

Drawing in VR Sketch works a little bit differently than drawing in Sketchup. In Sketchup, your are moving the mouse over a screen. A screen position can mean any 3D point that is behind that screen’s pixel. So Sketchup contains logic to know which 3D point you mean, i.e. at which “depth” inside the screen you are really pointing. For example, it’s difficult to draw in Sketchup a line that ends at a free-standing 3D position (you have to make use of alignments).

In VR Sketch, we took the opposite approach. Almost always, you specify a real 3D position, and that’s where the controller is. There are only a few exceptions, like laser-selection and painting of faces, where you select an existing face by pointing a laser.

When you draw a room, the way you are supposed to make the wall is to scale and move the model until the target points become easily reachable. That’s the way VR Sketch is designed. Drawing using the passthrough as the basis invalidates that approach…

It’s difficult at this point to merge a completely different approach into VR Sketch, but maybe we could tweak existing approaches. For example, right now you can “lock” one or two axes when you draw a line or a rectangle; this allows you to start drawing a wall (a rectangle) and lock it to be vertical. But you still need to physically move the controller as high as the top of the wall. What I’m thinking now—which would be more useful in your case—is that the locked rectangle should go not up to the same height as the controller, but instead up to a point defined by laser. It might be useful in general to make this change if it allows some positioning-at-distance to be done. I’ll try it out.

Armin Rigo

[filipejrc]

Thank you for your answer.
Something like your suggesting would be incredibly useful, especially for interior designers who could go to someone’s house, put the Quest Pro on, and quickly point the laser at the walls and draw the room layout in seconds. Exactly how the Quest Pro “Room Setup” works when you draw your walls and palying aread.
Thank you.

[arigo]

I’m coming back to this now that VR Sketch 18 has been released, sorry for the delay.

It’s unclear to me how what you have in mind could integrate with the rest of VR Sketch. You basically want something to draw walls on the floor like in the “Room Setup” (maybe as wavy lines? or by placing and connecting points on the floor, without needing to physically bend down?); and then something else to specify how high these walls should go until they reach the ceiling (with a uniform-height ceiling? or something more complex?).

All this is in theory possible to implement, and it would be a quick way to start making a model in your situation. But this looks very custom-made; it wouldn’t have any other more general purpose. If we were talking about Sketchup instead of VR Sketch, it would be the kind of thing you would find in a custom extension made by a 3rd party, and not in the core of Sketchup itself.

Needs more thinking from our side… Maybe at some point we should provide a way to write such 3rd party extensions for VR Sketch?

[smart_glasses_hub]

I’ve been investigating how to use and troubleshoot the Oculus Quest 2’s Passthrough feature for the average user. Here’s a summary of what I found:

• Understanding Passthrough:
  • Passthrough allows you to see your real surroundings through the headset’s cameras.
  • It’s primarily a supportive function with black and white, low-resolution imagery.
• Activating Passthrough:
  • Passthrough Home Environment: Activate this to interact with the real world while in VR, useful for 2D apps.
  • Double Tap Passthrough: A handy feature to quickly switch to Passthrough by tapping the side of the headset.
• Passthrough for Augmented Reality (AR):
  • Passthrough technology has the potential to be used for AR applications, like playing the piano or interactive games.
• Troubleshooting Passthrough Issues:
  • Problems usually relate to tracking and play area mapping.
  • Solutions include cleaning cameras, adjusting lighting, removing reflective items, and restarting the headset.
• Specific Passthrough Problems:
  • Distorted or Glitching View: Clean cameras, modify lighting, remove bright lights, clear clutter, and restart the headset.
  • Double Tap Passthrough Not Working: Ensure the feature is enabled in Guardian settings, delete guardian history, or try a hard reboot.
  • Stuck in Passthrough Mode: Disable double tap, modify lighting, restart the headset, or as a last resort, factory reset.

These insights should help in understanding and effectively using the Quest 2’s Passthrough feature, as well as addressing common issues. Feel free to check out my post for more details on any of these points!

[arigo]

Thank you! Let me add the new features of the passthrough in the Quest 3:

• better quality image than in the Quest 2, and in color

• can be combined with the Room Depth sensors (only in VR Sketch, not elsewhere, so far, I believe). This avoids the visual issue when the VR model is partially outside the real room or just bigger than it: the parts of the VR model that are behind real walls are made almost invisible, as if your real walls were only slightly transparent. Activation: in Settings, Rendering, Passthrough, choose the option “Room depth”. It probably requires allowing “spatial data” for VR Sketch: if you chose “no” when you originally started VR Sketch, you need to figure out where to re-enable that permission in the main menus of the Quest (in “Apps”, “Permissions”).

[arigo]

Hi @filipejrc,

I’ve done a first-draft version of “positioning-at-distance”, which allow some limited “at-distance” pointing when drawing rectangles, lines, and a few others tools (Move, Extrude, Arc drawing, and some variants of Tape Measure). This new feature is available in the Quest release 23.2.0, available on the App Lab store.

This works as an extension of these tools’ existing “lock to plane” or “lock to line” menu command. For example, when you draw a vertical wall, you start in the usual way with a rectangle; then make sure the wall is vertical, but don’t press the trigger again yet; instead, press the B button to bring the context menu, and pick “lock to plane”. Then you can continue to draw the wall but it will be locked to that vertical plane. The new thing added in 23.2.0 is that you can also point “at-distance” to where in that plane the end of that wall should go.

If you want to give it a try and have some feedback, it would be welcome!

[Christopher_Overbeck]

v71 Quest has new keyboard cutout support for any keyboard.
I was thinking of a sort of desk mode. Where you can edit with keyboard & mouse while seated. Thinking be default your virtual “cursor” is centered on your look direction, then you can offset that with mouse movement. And if it “locks” on a snap point, then it stays even as you look around until you jiggle your mouse. Maybe middle mouse button to recenter cursor on curren vision.

just spitballing. Feels like it might not take to much effort to prototype & see if its promising.

[arigo]

@Christopher_Overbeck There are many issues with what you have in mind. I don’t think it’s viable a priori. You wouldn’t have a proper way to enter 3D positions, and most user interactions in VR Sketch are based on that. You’re suggesting an alternative way, but it’s not at all as natural as the human brain being very good at knowing exactly where his hands are in 3D (the controllers are there to make that information available to the software with precision). Without that, many things in VR Sketch become a lot more convoluted. For example, Sketchup has got some ways to map the position of the 2D mouse to a 3D point, which works nicely, but is full of logic that has no equivalent in VR Sketch. And moving/scaling/rotating the model is very different too.

It may be interesting to try out something where you are sitting in front of a keyboard and a mouse (NB. if you see the keyboard but not the mouse, that’s another issue, but let’s ignore that for now). You’d end up with something much closer to Sketchup, with your view of the model being in 3D but the user interactions being Sketchup-like rather than VR Sketch-like. Not saying this would be a bad thing, but very different from VR Sketch. It feels that it’s closer to using the keyboard and mouse in the desktop view with Sketchup on the screen. The only difference would be that the model is rendered in stereo 3D instead of flat on the virtual screen.

[Christopher_Overbeck]

It’s certainly a niche I don’t think anyone’s tried yet. Might be a smoother use case for professionals who aren’t as much of VR enthusiasts.
I know there are CAD products like the space mouse & extra button pads for assigning custom shortcuts.

Maybe mouse can navigate normal menus & select tools while left hand serves as mix between ability to type keyboard shortcuts / numbers, and can lift off and be used as the pointer device with hand tracking for the model hovering above keyboard / in front of screen.
So right hand would handle selecting tools with mouse or let go and type dims into numpad, while left can move from keyboard for shortcuts/typing to being used to select points.

VR gives us so many options for user interface combinations that have not been tried before.