Feature request - backfaces wipeout

#1

Hello there!
I would like to suggest a new feature for VR sketch.
Let’s assume I want to add a double height in a portion of my room ceiling. What happenes right know is that you can place the new space up above your head, but you will always see the hidden sides. You will always see the sides that should be hidden by your ceiling. I hope this description makes sense :sweat_smile: In order to avoid that I have to build a fake ceiling all around the double height… this is not what I want, cause I want to see my real passthrough ceiling.

So I suggest making a specific material/property(?) to apply to the ceiling, which acts as an invisible wipeout for our 3d model. The point is to be able to choose from which point of view the new 3d model is visible or not. Or maybe this option could be something that works with backfaces?

What do you think?

Thank you everyone!
Have a nice day

edit: here two snapshots of what I mean, an easy example of a hole in the ground.

00
001262×729 846 KB
01
011262×729 858 KB

Thank you again :slight_smile:

VR Sketch 22.0
#2

Hi!

To see if I understood you correctly: you’d like a special material that can be applied to faces, which is drawn in VR as a special “color” which lets the passthrough image appear. The point is then: if you have such a face hiding other parts of the model because that face is in front, then the other parts are hidden and the passthrough image is visible instead. Is that correct?

Yes, that can be added easily. It’s already kinda present with the Quest 3’s depth sensor passthrough integration, which is implemented by adding to the VR scene a shape approximating the real world, as computed by the depth sensor. That shape is drawn with a material that works like described above. You could get roughly what you want if you use that feature—but you need a Quest 3, and moreover the effect always come from the scanned depth, and it’s also not always very precise. What you’re asking for (if I understood correctly) is something similar, but explicitly in your Sketchup model.

There are also other use cases where it would help, like starting from a fully virtual model and then making the real keyboard of the computer appear in passthrough on the virtual table. You’d do it with just a face (or maybe a flat box) with this special material inside an otherwise normal model.

#3

Thank you for your quick response!
I almost forgot the most important part. I am talking about AR with Quest 3.
A couple of days ago I tried VR Sketch with my colleagues using the app background in passthrough mode and they were all VERY impressed. I’ve shown them a 3d model of a product design we are making, placing it on the top of our desks. VR Sketch worked flawlessly as AR.
As an Architect, the next step is to try to add rooms to an existing building and look at them.
And here is the problem…

I am not sure about the tech behind the passthrough…

Yes, exactly. But I don’t think it has something to do with depth sensor (I guess, not sure).
I am simply looking for a way to let my 3d model be visible in VR sketch ONLY if your point of view doesn’t cross this “wipeout” material/object.

I make another example. Let’s say you want to add a window to your bedroom wall.
You model the window frames, the glass, and a simple 3d garden outside, just to have something to look at through your brand-new window. When you start VR sketch you can anchor your window perfectly to your real wall, which you SEE because of the passtrough. But you will ALSO see the garden outside in its entirety regardless of your only window.
I attach some screenshots, it’s easier :sweat_smile:

Thank you again for your support!
Filippo

01_full
01_full2174×1648 933 KB

02_no wall
02_no wall2174×1648 834 KB

03_ok
03_ok2174×1648 950 KB

VR Sketch 22.0
#4

OK, thanks for the confirmation! I can post here a prototype version of VR Sketch with that feature enabled. Do you use the PC VR version (with the Oculus/Meta Link application on your PC) or the standalone version running directly on the Quest?

#5

I use the stand alone app in the Quest 3, uploading the 3d model via Rhinoceros.

Thank you again, can’t wait to try it out! :100:

#6

Here is a pre-version:

https://vrsketch.eu/download/baroquesoftware_vrsketch_v21.1.PASSTHROUGHMAT_quest2.apk

Installation instructions: see docs section “Installation without using the App Lab”.

This gives a flag to materials, which you need to set in VR inside the materials panel window (invoked by using the icon above the paint bucket tool). Pick a material and click on the menu icon in the corner, and choose “special passthrough”. This is not saved so far, so you’ll have to do it again the next time you open the model.

Is that what you had in mind?

#7

Hello there!
I tried it today. I managed to install Sidequest and the beta version but the special passthrough flag is not clickable. I recorded a small video to show you.
Sorry for the poor lighting and the mess in studio, I was in a hurry :sweat_smile:
You will see a hole in the ceiling, with two materials orange and blue. The idea is that the orange material should be flagged as “special passthrough” but I can’t click it.
Video here: https://drive.google.com/file/d/1NnI8hDwCrIh7M0IK_SsPwBq8zhPpc6IR/view?usp=drive_link
Am I missing something?
Thank you again! :100:

#8

Oops, forgot a detail! The menu command was not clickable if you’re running from Rhino. It should be fixed (same URL, download again).

#9

Dear Arigo,
it works and it is insane! Thank you so much!
This opens so many new possibilities. I can’t wait to share this update with my colleagues.
This special passthrough can be used in so many creative ways to show our design to clients.

Thank you again, great work! :100: :100: :100:

#10

Great to hear!

I realized that in the last image above, the “virtual sky” is not rendered. I’m talking about this surface here, which I assume should be the sky above the outdoor garden:

image
image851×998 193 KB

If you don’t do anything, you’ll see it as passthrough and see the real wall. I guess you already worked around that by enclosing the whole garden inside “walls” and a “ceiling” in your model. It would be cooler if you could get the standard sky showing up there (it’s currently impossible). There are several solutions I could think of to implement this, but none seems very straightforward. You could add a transparent face for the window and give it a special material that says “nothing behind that should use the passthrough, instead draw the standard sky”. Or, you really need to make a box around the garden, but give it a material that says “draw this as the sky”. Third solution: the passthrough is ONLY drawn by the new “special passthrough” material; you’d need to ensure that you model a complete box around the whole play area, with a hole for the window, and give it this “special passthrough” material (i.e. what you’re doing but with a complete box instead of just the yellow wall). The virtual sky would appear because it is not behind a “special passthrough” face.

#11

Ah, maybe we could obtain the 3rd option by NOT activating passthrough in the settings, so the virtual sky is still visible, but still making use of a “special passthrough” material (right now this material is rendered as black because passthrough is completely disabled, but we could detect that case and render it as the passthrough image even though the rest of the model doesn’t use passthrough).

#12

This option looks very interesting. By implementing this option you can actually apply “windows” to the VR world (a VR world which is otherwise hidden by, let’s say, a box encapsulating everything with the “special pass through” enabled). Like when you play First Encounters, the showcase app for the Quest 3, and you shot the walls (they break, creating a “portal” hole to another planet).

I am wondering which case scenario could take advantage of this. Adding rooms and windows to existing architecture for sure!

#13

First Encounters looks like a cool demo, but sadly it appears that Meta removed it from the store with no explanation.

For what we’re trying to do here, I’m thinking about other use cases. You might want to see half of your real room in passthrough, but across the middle of it, there is a virtual wall with a door opening towards a purely virtual garden with the virtual sky—and you can walk there too, by walking inside the hidden half of the real room.

The easiest solution that seems to work and would be somewhat clear to use: if your model makes use of the “special passthrough” material, then the unobstructed sky always gets drawn as the virtual sky, and passthrough is only visible on “special passthrough” faces.

Maybe in that case the passthrough mode should be enabled automatically; right now if you didn’t enable passthrough in the VR Sketch settings then the “special passthrough” material is fully black.

#14

Here is it, if you want to play with it:

https://vrsketch.eu/download/baroquesoftware_vrsketch_v21.1.PASSTHROUGHMAT.2_quest2.apk

If there is a “special passthrough” material, then the virtual sky is drawn. You have to manually place a big box around the area where you want the passthrough to appear. This lets you control exactly where the passthrough should be displayed, and where the virtual sky should appear.

#15

I will try it in the next few days! Thank you again

#16

Hello again!

I’ve tried the new passthrough and it looks ok!
Initially, it is a bit counter-intuitive, but it makes sense in the end and works quite well.

I’ve shown this feature to my colleagues and, for the first time, they were genuinely impressed and finally got the potential of VR/AR.

They asked me a couple of questions that may be useful for future improvements… I summarize them here:

  • it will be interesting to add dynamic occlusion in the passthru. Is it the API function?
  • while in AR, with this new feature of passthrough, we noticed that it would be great if a “shadow” is projected onto the “passthrough object”. In rendering software there is the occlusion, let’s say a “fake shadow” around the corners of objects. This would be great because it will remove the impression of floating objects when they are actually stick to something (a wall, the ground… etc)
  • how to save your model from rhino to VRsketch? I see there is the function for Sketchup, but I couldn’t find it in Rhino… Am I missing something?

Thank you again, good job as always! :smiley:

Again

#17

Hi!

Great, thanks! I agree that the approach is somewhat counter-intuitive; we’ll keep thinking about it… About your questions:

  • dynamic occlusion: yes, there is actually a different API to get a dynamic depth map. However, that looks quite harder to use and there might be compatibility issues for VR Sketch. That’s why we didn’t look into that in more details so far. We’re not sure about the precision, too. In your “hole-in-the-ground” screenshot (the first one in this thread), it might result in a constant jiggle of the exact boundary between passthrough image and VR faces. As far as I can tell, right now the boundary isn’t very precise but at least it shouldn’t jiggle all the time.

  • “shadows”: such shadows are called “ambient occlusion”, and I see the point. It’s not completely obvious because we would need to add them to the passthrough only near the VR faces, but not everywhere else. A naive approach to add the passthrough’s model to the ambient occlusion computation would make all inner angles of the real room darker—but they are already darker, because they are a real world image! The resulting image might look strange. To do it correctly, we’d need to limit it to VR faces near the passthrough. It requires digging inside the details of the ambient occlusion logic and seeing how we can tweak it for that case.

  • saving the model from Rhino to VR Sketch: do you mean upload the model to our cloud, so that you can view it from VR Sketch later without needing to run Rhino at the same time? If so, then yes, that’s not supported at the moment. Actually, for Sketchup we only offer this feature for our subscribed customers; but you can’t subscribe with the Rhino “demo” version… so not for now, sorry. It’s unclear if we will continue to develop our Rhino offer. If we do, then it’s clearly something that we’ll add (together with “multiplayer” support).

Thanks for all your comments, it’s appreciated!

Armin Rigo

#18

Hi again,

About dynamic occlusion: sorry, I think I confused matters a bit. We do have support for the Quest 3’s depth scan, but only the “static” one, not the “dynamic” one; this is the passthrough mode called “room depth”. There is also another mode which additionally tries to figure out where things are in the world dynamically, which we don’t support. But I just realized that this may not be what you meant: that “room depth” mode is already something you can’t use so far. This is because doing so is equivalent to making the shape of the real room in the model (approximately, with ~10-centimeter-triangles), and giving that whole shape the material “special passthrough”. But then you can’t dig a window or a hole in that shape.

So another way to approach the problem would be to figure out a solution that would work with “room depth” as well (and maybe be more intuitive too) . If you had a special material that you would use on the window or hole itself, instead of the “special passthrough” material, then you could use that “room depth” mode. Any passthrough pixels that are found by the room scanning to be farther than the “special window” material would not be displayed. Or maybe the condition should be “farther, or a few centimeters nearer, too, to account for imprecision”.

Armin

#19

We’ve been working on other things, but here’s an update on this.

We need a solution that works in all cases: with the basic, non-depth passthrough OR with the room-depth-configured passthrough; and in models with “gardens” like we’re talking about here AND with “regular” virtual stuff inside the room. The only way that seems to always work would be to ask you, the model’s designer, to enclose not the real room but the “garden” inside a big box and paint the outside of that box with the “special passthrough” material.

For example, in the first picture you posted above, the material from the backfaces (yellow-ish) would be flagged “special passthrough”. In the second picture sets, with the virtual garden outside, you would similarly enclose the outside garden in a box and paint its outside with a material which is flagged “special passthrough”. In this case, the garden is bigger than the window, so the box would be as big as needed, with a rectangular hole for the window. The parts outside the window, like all the rest of the box, must be painted with the “special passthrough” material (but only outside).

This would tell VR Sketch which parts of the garden are hidden (they are the parts where we see the special-material box from the outside); which parts to draw as the garden’s objects (they are the parts that contain objects in front of the inside of the special-material box); and it gives us a way to add, for future work, support for picking a custom sky, too (drawn on the part that see directly the special-material box from the inside). And this can all be done in a way that works with both passthrough modes (even though the room-depth mode tries to draw the whole room including the virtual window, the virtual window part could be explicitly ignored because it is in front of the garden’s sky).

I’ll keep you updated on the progress of that. Feedback welcome!