[HN Gopher] Show HN: Scribble Diffusion - Turn your sketch into ...
___________________________________________________________________
Show HN: Scribble Diffusion - Turn your sketch into a refined image
using AI
Author : zsikelianos
Score : 171 points
Date : 2023-02-28 16:52 UTC (6 hours ago)
(HTM) web link (scribblediffusion.com)
(TXT) w3m dump (scribblediffusion.com)
| dougdonohoe wrote:
| "squirrel with acetylene torch attacking a tank"
|
| Didn't get the squirrel part quite right, but maybe it's my
| drawing :-)
|
| https://scribblediffusion.com/scribbles/bxc3jaofkzdh5nyjlff5...
| dwringer wrote:
| If you expanded the prompt to describe the squirrel in
| redundantly verbose detail it would probably pick it up just
| fine. The more synonymous descriptors you can work in, and the
| more naturally you can phrase it, the better it will work.
|
| But IMHO the interesting thing about controlnet is being able
| to use pre-rendered production art as a basis and allowing SD
| to respect the original proportions/model/etc. For rough
| sketches I prefer img2img without controlnet as it gives the
| algorithm leeway to fix, reinterpret, or "be inspired by" my
| input image without being too attached to it (since it's full
| of imperfections anyway).
| dragonwriter wrote:
| This looks like its just a quick front-end to the ControlNet
| scribble model (perhaps with a customized underlying model
| instead of base Stable Diffusion), with canned settings,
| presumable a canned negative prompt, and maybe some common stuff
| added beyond the user input to the positive prompt. Which is not
| to be dismissive, its a very nice demo of what SD+ControlNet
| scribble can do.
|
| But for people who like it, the ControlNet scribble model (and
| the other ControlNet models, depth-map based, pose control, edge
| detection, etc.) [0] are supported in the ControlNet extension
| [1] to the A1111 Stable Diffusion Web UI [2], and probably
| similar extensions for other popular stable diffusion UIs. Should
| work in any current browser, and at least the A1111 UI, with
| ControlNet models, works on machines with as little as 4GB VRAM.
|
| [0] home repo: https://huggingface.co/lllyasviel/ControlNet but
| for WebUI you probably want the ones linked from the readme of
| the WebUI ControlNet extensions [1]
|
| [1] https://github.com/Mikubill/sd-webui-controlnet (EDIT: even
| if you aren't using the A1111 WebUI, this repo has a nice set of
| examples of what each of the ControlNet models does, so it may be
| worth checking out.)
|
| [2] https://github.com/AUTOMATIC1111/stable-diffusion-webui
| liuliu wrote:
| You probably don't need canned positive prompt. ControlNet has
| a "guess mode" and in reality, basically just send the control
| to positive part with empty prompt while not sending control to
| negative part with empty prompt.
|
| Edit: nvm, this particular demo does require you to type in
| positive prompt.
| dragonwriter wrote:
| > You probably don't need canned positive prompt.
|
| With more playing, I'd say this probably doesn't have a
| canned positive prompt, just the user input.
| pamelafox wrote:
| I'm surprised, it actually did pretty well with a prompt I've
| been trying on all the generative AIs: "princess volcano kitty
| cat"
|
| https://imgur.com/hVfCIwU
|
| I use that prompt because it's something my preschooler draws on
| a daily basis with no qualms at all, but can often be hard for
| generative AI to imagine. So it's getting close to the
| imaginative abilities of a 3 year old! Halfway there.
| 1minusp wrote:
| Will be nice to be able to upload a picture/sketch instead of
| having to sketch something out in the scratchpad. I tend to
| sketch block diagrams from time to time and hate having to use
| some tool to draw/drag/align things. Very cool idea though!
| zsikelianos wrote:
| Hey there. Scribble Diffusion author here.
|
| If you already have your own images, you can use the Replicate
| model directly: https://replicate.com/jagilley/controlnet-
| scribble -- you can upload your image using the Replicate web
| UI or do it programmatically using the model's HTTP API.
| slhomme wrote:
| Uploading my own images was one of the reasons I created my own
| version of this. It works great with kids pictures! This is so
| fun to see my kids smile on their face every time they see the
| AI version of their drawings!
|
| Uploaded photos Examples: https://imgur.com/a/jeWgRvH Website:
| https://crappydrawings.ai
| ted_bunny wrote:
| Get ready to see many images with memes (Loss, Dickbutt)
| hidden in them. Might be good or bad, depending on your
| stance on wasted time.
| 1minusp wrote:
| this is neat! I'll have to try this out.
| jszymborski wrote:
| Very cool.
|
| Naturally does better with what I imagine are in-distribution[0]
| doodles/words than out-of-distribution[1], but still very cool
| (and fun!).
|
| [0]
| https://scribblediffusion.com/scribbles/h373dd42xbduzerzhlos...
|
| [1]
| https://scribblediffusion.com/scribbles/23xdfz5mtffwfgib32t7...
| latexr wrote:
| I opened the page and did not change anything of the given
| example. After multiple tries, I continued to fail to produce the
| prompted image:
|
| https://i.imgur.com/08o9zkG.jpg
|
| https://i.imgur.com/KkBtTyd.jpg
|
| Eventually I did get something which looked like a partial
| success, but with low resolution and not something I'd consider
| appealing:
|
| https://i.imgur.com/SJWkwBp.jpg
|
| https://i.imgur.com/BpSUD4j.jpg
|
| This is not a dig on the author. I have yet to see a simple
| prompt give a good result with Stable Diffusion. Are there
| examples of it?
| jihadjihad wrote:
| _" ceci n'est pas un arbre"_:
| https://scribblediffusion.com/scribbles/tywfxjjrffbaljs5syn4...
| amflare wrote:
| This is cool. At first I was skeptical that the sketch had
| nothing to do with anything and it was just feeding the prompt
| into an AI. But using the example "sun behind mountains"[0] it
| seemed to work alright. What really blew my mind was when I
| paired the sketch with a prompt that made no sense at all[1].
| Somehow the AI cobbled together a picture that used the prompt,
| but also has all the same geometry of the sketch.
|
| [0]:
| https://scribblediffusion.com/scribbles/elun6gwkxrcr7eqy5jao...
| [1]:
| https://scribblediffusion.com/scribbles/tpsty6qcxjfrxbxz3n6b...
| Waterluvian wrote:
| I would love to be able to paste an image in and resize it and
| other basic stuff.
|
| Ie. Let me stage a mock up. Because I can't draw a cat.
| bombcar wrote:
| This older one does pretty good with cats
| https://affinelayer.com/pixsrv/
| crazysim wrote:
| Is it possible to do the opposite?
| dragonwriter wrote:
| Yes, scribble annotator (included with the A1111 WebUI
| ControlNet extension) does image -> scribble.
| moswlks wrote:
| Wow, that's great, look at the level of detail:
| https://scribblediffusion.com/scribbles/pnj5qy3libhjvp3ybjaf...
| skykooler wrote:
| This seems to be broken now - As soon as I've drawn a line, it
| vanishes, and I can't draw anymore. Using Firefox on Linux.
| EGreg wrote:
| If we are using words then how is the scribble being used?
| mdp2021 wrote:
| Technically or practically?
|
| Practically, they are combined. (Triangle) + 'ice cream' = cone
| ; (Triangle) + 'mountains' = peak ; (Circle) + 'ice cream' =
| scoop ; (Circle) + 'mountains' = sun
| pjgalbraith wrote:
| Looks like the ControlNet scribble model which is super fun to
| play with. I've done some examples of what you can get with more
| detailed sketches
| https://twitter.com/P_Galbraith/status/1625842298914471938?c...
| andybak wrote:
| You went to the trouble of registering a domain but you didn't go
| to the trouble of even testing on Firefox? Even if you didn't
| have the time to fix it, then a message acknowledging that it's
| broken on some browsers is a nice gesture.
| dragonwriter wrote:
| > You went to the trouble of registering a domain but you
| didn't go to the trouble of even testing on Firefox?
|
| How much trouble do you imagine registering a domain is?
| andybak wrote:
| How much trouble is firing up a browser and pasting in a URL?
| dragonwriter wrote:
| _Installing_ a browser is more trouble than buying a domain
| (done each several times.)
| siddarthd2919 wrote:
| I think it is biased to the text and not the scribble. I
| scribbled a car and tried one with a the text car and bar. The
| first one resulted in a SUV and second resulted in home/Kitchen
| Bar. I don't even think the scribble matters.
| dragonwriter wrote:
| The scribble matters. Witness:
|
| "a photo of a cat":
| https://scribblediffusion.com/scribbles/5ymr7r66yjgx3jaibns2...
|
| vs:
|
| "a photo of a cat":
| https://scribblediffusion.com/scribbles/2noqq6amlzcpvpfsyweo...
| janosdebugs wrote:
| The drawing doesn't work for me on Firefox.
| rnosov wrote:
| Can you place more than two objects in the sketch? I'm trying to
| sketch a house with a tree side-by-side, but it always drawing
| trees in front of the house.
| hammock wrote:
| Yes I drew a gecko on a rock with a palm tree and it was
| perfect
| LanceJones wrote:
| I just tried to draw exactly the same thing. I'm no artist, so
| I thought a house, a tree, the sun and some clouds were my best
| bet (like I drew when I was 5; now 55). :-)
| whalesalad wrote:
| lol, pretty cool -
| https://scribblediffusion.com/scribbles/nnabaloz3fb3jl67ktje...
| sinistersnare wrote:
| Looks awesome, I think this is a great mix with prompt
| engineering.
|
| It seems this does not work on Firefox? I could only draw on
| about half the canvas and it was pretty buggy. Dont support the
| chrome monoculture!
| gowld wrote:
| There's a solution linked at the bottom of the page:
| https://github.com/replicate/scribble-diffusion
| notpushkin wrote:
| Is there?
| dragonwriter wrote:
| I think he is referring to "fork the repo and fix the bug
| yourself" as the solution, but given the existence of far
| more featureful publicly available Web UIs for Stable
| Diffusion + ControlNet, if you just want to get something
| that works on Firefox, you don't need to expend that much
| work if you are wiling to host it yourself rather than just
| using something someone already has up on the web.
| zsikelianos wrote:
| Hey, Scribble Diffusion author here.
|
| Sorry about the trouble. The Firefox incompatibility is the
| result of a bug in the underlying npm package we're using to
| render the drawing tool and canvas.
|
| The issue is being tracked here:
| https://github.com/replicate/scribble-diffusion/issues/17#is...
|
| We may need to wait for a fix to that, or consider swapping out
| the package we use for scribbling on a canvas.
| [deleted]
| pjgalbraith wrote:
| You can run it locally with a different (much better) WebUI see
| https://github.com/Mikubill/sd-webui-controlnet for example.
| dylan604 wrote:
| wow, yeah, very frustrating experience
| aidenn0 wrote:
| Per github seems to be related to:
|
| https://github.com/vinothpandian/react-sketch-canvas/pull/11...
| citizenkeen wrote:
| Agreed. I nearly just closed the tab when it didn't work on
| Firefox.
| dragonwriter wrote:
| If you haven't played with Stable Diffusion based stuff before,
| note that as well as telling it _what_ (in terms of subject) you
| want in the prompt, you can also tell it _style_ things. Compare:
|
| "a goofy owl":
| https://scribblediffusion.com/scribbles/oymg4kadgvezxppvwkf5...
|
| "a goofy owl, realistic photograph, depth of field, 4k, HDR":
| https://scribblediffusion.com/scribbles/va5l24amjnb55g62renz...
|
| "a goody owl, pointillism":
| https://scribblediffusion.com/scribbles/5dfnru4f6zguphjvvdrl...
| phonescreen_man wrote:
| Love this, my fave sd webUIs..
| https://scribblediffusion.com/scribbles/r7vdh5aw4ja5dnpidpqz...
| aidenn0 wrote:
| I firefox there's something wonky going on with the scribble
| area; the bottom of the image doesn't seem to show up until after
| you've drawn a lot more in the top of the image.
| zsikelianos wrote:
| Yeah sorry about that. Tracking the issue here:
| https://github.com/replicate/scribble-diffusion/issues/17#is...
| iamgopal wrote:
| https://scribblediffusion.com/scribbles/x56jepp2p5bxhc3yrwzu...
| It works!!
___________________________________________________________________
(page generated 2023-02-28 23:00 UTC)