[HN Gopher] Text2Room: Extracting Textured 3D Meshes from 2D Tex...
___________________________________________________________________
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image
Models
Author : amichail
Score : 118 points
Date : 2023-03-22 16:50 UTC (6 hours ago)
(HTM) web link (lukashoel.github.io)
(TXT) w3m dump (lukashoel.github.io)
| bilsbie wrote:
| When is this stuff making it into games! This would be amazing on
| the quest.
| worldsayshi wrote:
| Shouldn't be too hard to integrate. Just need to load the
| result into a Unity scene.
|
| If nobody has tried it in a week or two I might give it a go.
| nineteen999 wrote:
| It's cool and all but all your lighting is going to be pre-
| baked ...
| MayeulC wrote:
| Pretty cool. Now, I wonder, can't you label a certain region of
| space with a prompt, and let the diffuser do its job? Maybe with
| some mathematical function to bend into another area.
|
| The idea would be to roughly place the elements in a 3D scene,
| and adjust the prompt as the camera is moved around the scene.
|
| Here, it's obvious that the "fireplace" prompt causes the model
| to place a new fireplace as the previous one comes out of view.
|
| Even if you can't precisely label portions of an image, changing
| the prompt as the camera moves (or changing the weight
| coefficients for a prompt describing multiple orientations) would
| avoid that kind of "unnatural" result.
|
| Regardless, impressive results! I wonder if it would perform
| better if it was re-trained to output a depth channel as well.
|
| It could be useful for (artistically) filling gaps in
| photogrammetry projects.
|
| I can't wait for painting or drawing styles to be applied to the
| output!
| gs17 wrote:
| The example trajectories (https://github.com/lukasHoel/text2roo
| m/tree/main/model/traje...) seem to have different prompts for
| different angles, so you can definitely give a vague layout of
| the room.
| jimmySixDOF wrote:
| A different project, perhaps, although with the speed they are
| popping up it's not easy to keep track, but I was just playing
| around in a live multiplayer 3D worldspace [1] where a text
| prompt to instant 360 Skybox is a really cool feature to see
| working as it forms all around you in realtime (cool on PC
| amazing in VR). It extends the pipeline of whatever Blockade Labs
| are using under the hood [2].
|
| [1] https://hyperfy.io/ai-sky
|
| [2] skybox.blockadelabs.com
| avaer wrote:
| Definitely hard to keep up with the tech, even if you're deep
| in it.
|
| I presented a 3D gameplay hack of this at the recent Blockade
| meetup: https://youtu.be/TfRJeedTeOs
|
| The metric depth model I used (ZoeDepth) is quite new -- most
| previous models were inverse relative depth, with poor scaling
| properties, especially for artistic worlds.
|
| But now there is a much better depth model coming from Intel
| called Depth Fusion which they are adding to the Blockade API
| and also open sourcing (!)...
|
| Also worth checking out what's possible with SD ControlNet:
| https://twitter.com/BlockadeLabs/status/1634578058287132674
| wsgeorge wrote:
| Reminds me of this project submitted yesterday [0]. I'm trying
| hard to keep up with the pace of projects and papers being
| announced. This is all very exciting!
|
| [0] https://zero123.cs.columbia.edu/
| smaddox wrote:
| Cool. Stereoscopic diffusion images coming soon.
| jasonjamerson wrote:
| Brilliant, been waiting for / working toward this. Is there a way
| to try it out?
___________________________________________________________________
(page generated 2023-03-22 23:00 UTC)