[HN Gopher] Redbean 2.0 turned into more than a hobby project
___________________________________________________________________
Redbean 2.0 turned into more than a hobby project
Author : jart
Score : 393 points
Date : 2022-06-16 10:47 UTC (12 hours ago)
(HTM) web link (justine.lol)
(TXT) w3m dump (justine.lol)
| lxe wrote:
| I love how this is a huge hack and it works so well. Nicely done.
| sscarduzio wrote:
| No ARM64 love?
| DrBazza wrote:
| https://justine.lol/ape.html
|
| One of best technical posts I've read in a while.
| rank0 wrote:
| The project depends on x86 and AFAIK, the voodoo that enables
| the cross-platform executable is simply not available on other
| ISAs.
| panarky wrote:
| Maybe this is common knowledge, but this is a quick and dirty way
| to reduce bots, spam and abuse. if
| geo:get('location', 'accuracy_radius') >= 100 then
| SetStatus(403) Write('you can only post comments from
| your home internet connection') return end
|
| Bad actors often use cheap cloud instances, and the IP addresses
| of their data centers typically have an accuracy radius of 1000
| km.
| _wolfie_ wrote:
| Well, I guess I was a bot all along with 200 on my home IP.
| smt88 wrote:
| This is a really bad idea. Bot detection is an incredibly
| complex problem, and simple rules like this are more likely to
| catch humans than (motivated) bad actors.
| bigiain wrote:
| "(Motivated) bad actors" are a different problem to drive by
| bot spammers.
|
| While I agree with all the people saying "this'll block real
| people at significant rates", for some things (like personal
| blog replies or small forums) that can be a perfectly
| sensible trade off for some sites.
|
| Sure, if you are Netflix, or a government department website
| that people need to use, it's a terrible idea.
|
| For somebody who's sick of deleting spam replies on their
| personal blog posts? Simple rules with known and acceptable
| unintended consequences might be a really good idea.
| cmeacham98 wrote:
| You'd be better off banning known "cloud" IP ranges than this
| (and I think even that protection is questionable).
| heavyset_go wrote:
| It's questionable, and it blocks a lot of legitimate traffic.
| bigiain wrote:
| "A lot"?
|
| What sort of "legitimate traffic" comes off
| AWS/Azure/DigitalOcean et al?
|
| And how does the volume of that (and relevance of that
| traffic) compare to home add and mobile CGNAT ipaddresses?
|
| Unless I'm selling something, dropping all cloud instance
| IP ranges doesn't em like it'd block a lot of "legitimate
| traffic" to me, at least not enough to care about. If my
| choices were "spend time implementing a more sophisticated
| locking technique to reduce blogspam" or "write more
| blogposts", I'd be triggerhappy blocking everybody using
| AWS o browse from and writing more content.
|
| I use FSecure as a VPN, and I see it "do weird shit" at
| about the same rate as I see when I'm using a Hertzner box
| as a wire guard endpoint. (Most common thing there is I
| can't order via menulog when I've got a vpn endpoint out of
| I am getting it delivered to, and I've just got used to
| switching off my phone's vpn while ordering..)
| m00dy wrote:
| I wouldn't prefer that rule.
| heavyset_go wrote:
| I'd imagine this might hurt those behind CGNAT or anyone that
| chooses to use VPN services.
| [deleted]
| detaro wrote:
| Have a normal connection at one of Germanys biggest ISPs, am a
| bot according to that rule.
| 8organicbits wrote:
| The same is true for some large US based ISPs. Lot's of home
| users would be considered bots using this rule.
| andrewmcwatters wrote:
| Thanks for sharing this, but how are accuracy radii defined by
| MaxMind?
| achillean wrote:
| For a more specific check, you could query internetdb.shodan.io
| and see if the client IP has port 22 open, has the "vpn" tag or
| is any other services that you wouldn't expect to see from a
| visitor. We have a bunch of enterprise users that created rules
| to filter traffic out from IPs that are compromised or
| otherwise look malicious.
| gazby wrote:
| This strikes me as a terrible idea, relying on an idiosyncrasy
| of an arbitrary database to deny access to users.
|
| For anyone doing something like this with a service that's
| useful to people, consider MaxMind's minFraud service. It's
| PAYG and you can choose three different levels of information
| at three different price points. It's targeted at fraud
| minimization, but you can hand it as little as an IP address
| and it'll give you what you need to know.
| panarky wrote:
| Depends on the use case.
|
| MaxMind's minFraud is $0.015 per request and requires a call
| to an external endpoint. Justified for finance and shopping
| apps, not great for a high volume free service.
|
| And you don't have to deny based on this one signal alone,
| you could fall back to recaptcha or minFraud to reduce the
| false positive rate.
| capableweb wrote:
| > MaxMind's minFraud is $0.015 per request and requires a
| call to an external endpoint. Justified for finance and
| shopping apps, not great for a high volume free service.
|
| What fraud are you trying to prevent that gets abused in a
| free service? Most of the abuse I see for free services I
| host is DDoS, brute-forcing paths/authentication and port
| scanning, but all of those are trivial to deal with rate-
| limiting.
| zdragnar wrote:
| Any and all public form submissions will get spammed.
|
| My wife ran a small business for awhile that catered to
| local customers only (i.e. not an online business,
| products didn't ship through the mail. She added a
| contact us type form which submitted to google forms as a
| quick alternate to phone calls for prospective clients.
| It was used almost exclusively by people trying to sell
| her things from overseas.
|
| She wasn't even running ads or anything online, so how
| they ever found the website in the first place, I have no
| idea.
| capableweb wrote:
| That's also relatively easy to protect. First defense is
| rate-limited, second is captchas, third is having a
| dynamic "secret" key fetching from some backend via JS
| that gets injected into the form and works as a
| "password" to successfully trigger the email to get sent
| (but displaying success messages even if it's not).
| fire wrote:
| I find it more than a little amusing that we go from decrying
| an arbitrary database solution[1] to recommending one
|
| 1: ( which is likely to be maxmind's geoip DB in many cases
| anyway )
| gazby wrote:
| We go from decrying use of an arbitrary implementation
| detail of a database not fit for purpose, to recommending
| using a separate database for its intended purpose that
| just happens to be from the same vendor.
| [deleted]
| mister_goo wrote:
| This is a single executable web server with Lua and other libs,
| with a zip file appended. I have done something similar with
| Python+tcltk.
| ok_dad wrote:
| Yea but this one runs on 6 operating systems with the same
| binary artifact.
| ramesh31 wrote:
| But can it read email?
| capableweb wrote:
| Why not?
| https://github.com/vrld/imap4.lua/blob/master/example.lua
| sergiotapia wrote:
| Is this related to the ORM? https://redbeanphp.com/index.php
| xmonkee wrote:
| no
| quickthrower2 wrote:
| The second hardest problem in CS
| JodieBenitez wrote:
| Great ORM by the way.
| TimTheTinker wrote:
| This is _incredible_. Justine 's work continues to amaze.
|
| Work like this proves that assumption-questioning research on its
| own can produce incredible results if done by the right people.
|
| I think a lot of companies have stopped doing this kind of
| research, in favor of chasing more immediate profits, to their
| long-term detriment.
| m00dy wrote:
| I heard that version 3.0 will be on baremetal, is that true ?
| jart wrote:
| We're working on improving our bare metal support. Once we
| embed a TCP/IP stack your redbean will optionally be able to
| run as its own autonomous operating system with the absolute
| lowest network latency and minimal attack surface area. It's a
| idea whose time has come now that everything runs under a
| hypervisor in the cloud. If the hypervisor is the new operating
| system, then it'd be nice to not need a second operating system
| too.
| rank0 wrote:
| I love your work! How do you find the time to work on
| readbean/ape/etc? I sponsor you on github, but surely
| donations pale in comparison to what you could earn at an
| engineering day job.
|
| Do you plan on writing your own tcp/ip stack with
| cosmopolitan? Why not pull in the networking stack and
| syscall libraries from MirageOS?
| jart wrote:
| It's not so much could but have. I used to work at Google
| Brain. I decided I'd rather be doing this instead. I make
| it work by living modestly, eating ramen, and avoiding
| doctors. Thank you for sponsoring me! It helps a lot.
| eigenvalue wrote:
| I have really enjoyed your submissions in the past year.
| Just sponsored you on Github. I feel like your work is of
| sufficient excellence and utility that you shouldn't need
| to eat unhealthy food to get by. Have you considered
| applying for grants from the non-profit arms of big tech
| companies? I do think that if you can combine Python in
| redbean in a seamless way, it would really drive adoption
| and lead to more sponsors for your work. Anyway, thanks
| for many hours of entertainment watching blinking lights.
| c-linkage wrote:
| I must be getting old because I can't tell if this is sarcasm
| or not.
| kitd wrote:
| Why should it be? It's basically a unikernel in a zip file.
| jart wrote:
| That's correct. Redbean is already capable of booting on
| bare metal. We do what we must because we can. The
| executable file format redbean uses is a DOS Master Boot
| Record with a partition table, so it's technically a
| valid disk image you can `dd` too. If you're willing to
| trade away Windows support, then it can be an EFI image.
| The main blocker is figuring out how to get an e1000
| and/or VirtIO driver in there with a TCP/IP stack. Right
| now Cosmopolitan bare metal support is only adequate for
| stdio applications, which use the serial port and read
| from the zip fs.
| unsafecast wrote:
| That sounds amazing, I love your work! Is ARM support
| planned? A raspi seems like the perfect target for this.
| capableweb wrote:
| The page for APE (https://justine.lol/ape.html) says the
| following about ARM:
|
| > It'll be nice to know that any normal PC program we
| write will "just work" on Raspberry Pi and Apple ARM. All
| we have to do embed an ARM build of the emulator above
| within our x86 executables, and have them morph and re-
| exec appropriately, similar to how Cosmopolitan is
| already doing doing with qemu-x86_64, except that this
| wouldn't need to be installed beforehand. The tradeoff is
| that, if we do this, binaries will only be 10x smaller
| than Go's Hello World, instead of 100x smaller. The other
| tradeoff is the GCC Runtime Exception forbids code
| morphing, but I already took care of that for you, by
| rewriting the GNU runtimes.
|
| Also this, from a GitHub issue (https://github.com/jart/c
| osmopolitan/issues/354#issuecomment...):
|
| > Probably related to #399. The recommended approach
| would be to use a full emulator like Bochs. It's not
| something we use at the moment so we can't provide
| support on this. Although we do intend to have APE
| support ARM at some point in the future.
| csdvrx wrote:
| > That's correct. Redbean is already capable of booting
| on bare metal. We do what we must because we can.
|
| I love the spirit :)
|
| > The main blocker is figuring out how to get an e1000
| and/or VirtIO driver in there with a TCP/IP stack.
|
| Why? Is it for performance reasons or security reasons?
| (or both)
|
| > Right now Cosmopolitan bare metal support is only
| adequate for stdio applications, which use the serial
| port and read from the zip fs.
|
| I'd suggest you "think different", and use instead
| something like ppp to create a TCP/IP stack over a serial
| link.
|
| Modern btuart implementations already routinely achieve
| >1Mbps on commercial devices. The GSI as seen on the
| Intel Serial IO devices support bitrates over 20Mbps.
|
| This could buy you time until you find a better solution,
| if it's ever needed (which I doubt as back of the
| envelope estimations make me believe you'll hit other
| limitations before)
| branon wrote:
| For the good of all of us! Except the ones who are dead.
| egeozcan wrote:
| This, and "we do what we must because we can" from the GP
| are references to the ending credits in Portal, the game,
| if anyone is wondering.
| csdvrx wrote:
| Recently, I took this approach to get proper ACPI S01x
| sleep on my Lenovo X1 Fold (a wonderful device BTW, once
| you take the time to work around Intel drivers bugs)
|
| A battery discharge rate of 0.5%/h in sleep is just
| great... but I think I can do better: I'm now trying for
| 0.25%/h.
|
| Imagine if you could immediately resume your foldable
| oled tablet, and it'd have only lost like 6% of the
| battery. With a 20% hibernate trigger, it would remain
| immediately available for over 3 days straight!
| moonchild wrote:
| > minimal attack surface area
|
| Just double-checking--it is still written in c?
| db65edfc7996 wrote:
| So many great features in this release. Love the .args and
| subprocess support. I have been toying with redbean on and off,
| and I have been wanting to see if I could kludge together a poor
| man's long running job (scheduled) runner (something like
| Celery).
|
| Also, why is redbean.dev not ported to 2.0?
| jart wrote:
| Try hitting ctrl+shift+r in your browser. Sometimes browsers
| don't respect the cache headers which say 1 hour max and I
| haven't figured out why. This new release includes functions
| like setitimer() to help the primitives be there for anyone
| wanting to use Lua to build something like Celery.
| unsafecast wrote:
| Firefox uses Ctrl-R instead.
| capableweb wrote:
| Maybe that depends on the OS? Firefox on Linux does a
| normal refresh for me when doing CTRL+R but a hard refresh
| (without any cache) when I do CTRL+SHIFT+R.
| twicetwice wrote:
| This is standard at least across Chrome and Firefox on
| Linux and MacOS-- (ctrl|cmd)+R does a normal page
| refresh, (ctrl|cmd)+shift+R does a "hard refresh" where
| it reloads every resource on the page without caching.
|
| Also, in Chrome, if and only if you have the dev tools
| open, right clicking the reload button will give you a
| menu with the options "Normal Reload", "Hard Reload", and
| "Empty Cache and Hard Reload". The third option will
| ensure that requests initiated by JS or that otherwise
| weren't part of the page load also won't be served from
| cache.
| igravious wrote:
| How long has this been a thing, and (more worryingly) how
| have I never heard about it until now? Is this common
| knowledge?
| capableweb wrote:
| I think I've been using that shortcut for as long as I
| can remember. In web development circles it's certainly
| common knowledge, as you always end up in a situation or
| two where you need to reload without cache, but you don't
| want to clear the entire browser's cache.
| daxterspeed wrote:
| HTTP has some strange rules about using stale caches so
| usually you want to add must-revalidate to your cache-control
| header. That ensures that the browser must revalidate once
| the cache goes stale.
|
| https://developer.mozilla.org/en-
| US/docs/Web/HTTP/Headers/Ca...
|
| Personally I've opted for "stale only" caching, so everything
| is served with Cache-Control: max-age=0,must-revalidate and a
| Last-Modified header and the browser will always make
| corresponding If-Modified-Since requests. This means
| significantly more requests per page, even if the responses
| are mostly 304 Not Modified, but getting to avoid all forms
| of cache busting makes developing a lot nicer.
| mhd wrote:
| Lua REPL based on new minimalistic readline lib? Interesting...
| Severian wrote:
| Redbean is amazing. I was able to run some TLS stress tests
| against it (where you setup a handshake and just disconnect). I
| was able to pump out about 1200 connections per second between my
| desktop and laptop on 1gb ethernet. I only had 5 handshake
| errors.
|
| I look forward to playing with this!
| jart wrote:
| Half a percent is small but there should ideally be zero
| errors. Could you file a report so we can fix that?
|
| Part of what makes the redbean TLS stack fast, is that I spent
| a lot of time hacking on MbedTLS to improve its performance.
| One of things that's counter intuitive about crypto code is
| assembly can be safer in additional to being faster, since it
| helps guarantee the compiler doesn't add branches, for things
| like bit overflow carrying in arithmetic operations. One
| example is
| https://github.com/jart/cosmopolitan/blob/master/third_party...
| which I wrote to make the NSA curve much faster. I've been
| meaning to upstream into MbedTLS. Another thing I added is
| https://github.com/jart/cosmopolitan/blob/master/libc/nexgen...
| which greatly improves the performance of RSA multiplication
| using Intel's ADX ISA. But C crypto code can be beautiful too!
| For example, I found Everest (curve25519) particularly
| impressive, since they solve overflow in a different way, and
| as such, it's quite possibly the only large piece of C/C++ code
| I've ever seen that contains _zero_ conditional branches and
| _zero_ pointers.
| https://github.com/jart/cosmopolitan/blob/master/third_party...
| The closest thing I've found to measuring the performance
| impact of my changes is by recording how long the MbedTLS test
| suite takes to run. The speedup for suite_rsa was 1.91x faster,
| ECP is 1.86x, and ECDSA is 2.84x.
| mysterydip wrote:
| Really impressive feature additions since the first time I heard
| of it. Is Lua backend-able? I've been looking for a langage since
| PHP, the node ecosystem seems too volatile to me.
| oliwary wrote:
| Lua also powers itch.io! [0] Leafo, the creator, has also built
| Moonscript [1], a language that compiles to lua, and Lapis [2],
| a web-framework for Lua and Moonscript. I am always fascinated
| by his productivity.
|
| [0] https://leafo.itch.io/ [1] https://moonscript.org/ [2]
| https://leafo.net/lapis/
| paulclinger wrote:
| > Is Lua backend-able?
|
| I'd say yes; I've been using it to develop
| https://github.com/pkulchenko/fullmoon, which is a redbean-
| based webframework written in Lua.
| jay_kyburz wrote:
| Thank you for ZeroBrane! I am a Love2D user!
| ifyouhate wrote:
| tomcam wrote:
| Fullmoon looks impressive and your README.md is very well
| written.
| VWWHFSfQ wrote:
| > Is Lua backend-able?
|
| Yes and some of the largest web infrastructures in the world
| run on it! I'm not sure if it's still true, but it's my
| understanding that CloudFlare runs it at the edge globally. I
| personally know of several other _very large services_ running
| it at massive scale.
|
| In my experience Lua in nginx is such a pleasure to work with.
| It's all transparently async using nginx's event loop. You just
| write regular procedural code in Lua and the runtime handles
| yielding/resuming for you automatically. There's no special
| async/await stuff. Just write your Lua code and the runtime
| figures out the yield points internally. It's a breath of fresh
| air.
|
| [0] https://openresty.org
|
| [1] https://en.wikipedia.org/wiki/OpenResty
|
| Edit:
|
| This was from several years ago, but CloudFlare built their WAF
| product on nginx/Lua:
|
| https://blog.cloudflare.com/cloudflares-new-waf-compiling-to...
| adolph wrote:
| Is Kong still OpenResty based?
| VWWHFSfQ wrote:
| Yes Kong is a custom distribution of OpenResty
| formerly_proven wrote:
| Likely more because of Mike Pall's LuaJIT instead of the
| traditional reference implementation.
| [deleted]
| turnsout wrote:
| Just when you're feeling confident in your own coding skills,
| Justine comes along and releases an update. Just incredible
| stuff!
| heavyset_go wrote:
| Seriously, APE was just impressive and they just keep on going.
| [deleted]
| vsroy wrote:
| How does Readbean beat Nginx? My assumption was that forking is
| very slow.
| jart wrote:
| One thing that helps is the binary size is tiny and the memory
| usage patterns are tight. It's bloat that usually makes fork()
| go slow, since if you have a lot of little pieces updating
| disparate memory locations, then you're going to have a lot of
| page faults. Redbean was built with a singular first-principles
| vision. When I started working on this, all I had was an empty
| file and an assembler. As such, I spent a lot of time using
| things like linker scripts to organize exactly where in the
| binary code gets relocated.
| nielsole wrote:
| Just guessing here, but redbeam likely does not have to access
| the filesystem as the files are in the binary and thus saves on
| syscalls and context switches.
| eismcc wrote:
| It'd be fun to add raft support so that multiple readbeans could
| share state such as monkey patching updates and other useful
| distributed features. On bare metal this could be a whole stack
| in a tiny bin.
| pstuart wrote:
| yep. It's being done today with dqlite -- perhaps there's an
| opportunity for synergy?
| silisili wrote:
| Just curious about the name - what made you pick redbean? Not
| hating on it, it's just unusual enough for me to ask about the
| backstory there.
| jart wrote:
| In my GitHub profile photo https://github.com/jart you'll see
| me holding a stuffed toy I call greenbean which has been cute
| little work desk companion of mine for some time. Later on I
| came into possession of a stuffed red lobster. A Chinese friend
| of mine suggested the name redbean as a companion for
| greenbean. Later in 2020 I was writing demo programs to
| showcase the capabilities of Actually Portable Executable. When
| I decided I wanted to write an HTTP server, redbean was looking
| up at me on my desk. That's when the idea clicked and it's been
| redbean ever since.
| languageserver wrote:
| > Thanks to Linus Torvalds, we not only have a consensus on
| architecture, but we've come pretty close to having a consensus
| on the input output mechanism by which programs communicate with
| their host machines, via the SYSCALL instruction.
|
| Could someone explain this? I cannot find any information online
| justinsaccount wrote:
| https://en.wikipedia.org/wiki/Linux_kernel_interfaces
| rcarmo wrote:
| Only thing it's missing is Fennel, although I suspect it should
| be trivial to add :)
| capableweb wrote:
| That was my first thought as well. If it can support Fennel as
| a first-class citizen, then I might just switch wholesale to
| using Redbean for any application development.
| masukomi wrote:
| yup. I was thinking the same thing.
| sydthrowaway wrote:
| Dead link
| jart wrote:
| Do you use Comcast Xfinity? Try using 8.8.8.8 for DNS.
| sydthrowaway wrote:
| True, but why this workaround?
| twicetwice wrote:
| I ran into the same issue on Xfinity, why is it broken there?
| Some nonsense Comcast gets up to?
| zX41ZdbW wrote:
| Maybe Cosmopolitan Libc can be developed to become a full-
| featured glibc replacement like Musl and what LLVM libc is going
| to target? To be able to link with large C++ codebases it is
| still missing a ton of small cruft...
| newaccount2021 wrote:
| csdvrx wrote:
| I love it!
|
| If I may suggest, supporting more languages than lua could help
| bring more people: say php or python, maybe with something like
| vscodium?
|
| IIRC python2 was a WIP (issue #141 for cosmopolitan)
|
| Personally I'd prefer perl for performance, and vscodevim or
| similar to edit code, but I have weird tastes :)
| giraffe_lady wrote:
| Lua has similar uses to those but has a uniquely simple
| implementation and very close relationship with C that probably
| makes it a far, far better choice for this. I'm not really a
| lua lover in general but this is basically exactly what its
| made for and it shines in this role in a way those won't.
| jart wrote:
| We ported Fabrice Bellard's QuickJS engine to Cosmopolitan last
| year. We ported Python3 too. (You can build Python3 as a 5mb
| single file Actually Portable Executable!) So integrating them
| into redbean is on the table. The big question is whether or
| not redbean should be multitenant with languages, or if we
| should ship separate binaries where you choose the redbean
| "flavor" that has your preferred one. I'm leaning towards the
| latter, since there can only be one REPL language. What do you
| think?
| eigenvalue wrote:
| I also think the latter is cleaner. I might also suggest that
| if you are going to integrate with Python, you should make it
| as "batteries included" as possible with the Python data
| stack (numpy/pandas/scipy/sklearn) so that you can easily add
| a redbean web app to an existing Python project without
| worrying about how those "heavy duty" libraries will work.
| bitexploder wrote:
| Would love to see Python support. This project and your libc
| sort of break my brain and I reverse engineered weird stuff
| for a decade professionally. I understand how it all works,
| but it is easy to understand something once someone built it.
| Some of the design choices and constraints were just not
| things I would have thought reasonable for a project like
| this.
| foxhop wrote:
| that seems sane to me with my novice understanding of the
| project. im most confortable with Python3 & have existing
| projects which I would try to port over as a test.
|
| a python3 flavor could be experimental at first, I bet it
| would boost overall adoption & be an unfair advantage over
| nginx approach.
|
| having multiple repl in one binary would remind me of
| containers in a very strange but cool way, that would be more
| a fun experimental feature a rabbit hole for others to
| explore.
| csdvrx wrote:
| > having multiple repl in one binary would remind me of
| containers in a very strange but cool way, that would be
| more a fun experimental feature a rabbit hole for others to
| explore.
|
| It would also let the flavors of redbean compete for
| donations, which could be directed towards your language of
| choice (say python3 for you) while limiting the size and
| complexity.
| thesuperbigfrog wrote:
| >> The big question is whether or not redbean should be
| multitenant with languages, or if we should ship separate
| binaries where you choose the redbean "flavor" that has your
| preferred one. I'm leaning towards the latter, since there
| can only be one REPL language. What do you think?
|
| Different flavors of redbean would be awesome!
|
| Thank you for building redbean.
| db65edfc7996 wrote:
| It is a resource strapped project. I doubt they have the
| bandwidth to throw in additional languages. Plus, Lua is
| perfect for this niche: small, easily embeddable, fast, and
| simple enough to pick up in an hour. There are even a variety
| of languages which can transpile down to Lua.
| gavinray wrote:
| TypescriptToLua is excellent and actively maintained
___________________________________________________________________
(page generated 2022-06-16 23:00 UTC)