[HN Gopher] Server-Sent Events: an alternative to WebSockets
___________________________________________________________________
Server-Sent Events: an alternative to WebSockets
Author : tyrion
Score : 381 points
Date : 2022-02-12 14:05 UTC (8 hours ago)
(HTM) web link (germano.dev)
(TXT) w3m dump (germano.dev)
| lima wrote:
| One issue with SSE is that dumb enterprise middleboxes and
| Windows antivirus software break them :(
|
| They'll try to read the entire stream to completion and will hang
| forever.
| bullen wrote:
| I managed to get through almost all middle men by using 2
| tricks:
|
| 1) Push a large amount of data on the pull (the comet-stream
| SSE never ending request) response to trigger the middle thing
| to flush the data.
|
| 2) Using SSE instead of just Comet-Stream since they will see
| the header and realize this is going to be real-time data.
|
| We had 99.6% succes rate on the connection from 350.000 players
| from all over the world (even satellite connections in the
| Pacific and modems in Siberia) which is a world record for any
| service.
| Matheus28 wrote:
| While 350k simultaneous connections is nice, I'd be extremely
| skeptical of that being any kind of world record
| [deleted]
| bullen wrote:
| The world record is not the 1.100 concurrent users per
| machine (T2 small then medium on AWS) we had at peak, but
| the 99.6% connections we managed. All other multiplayer
| games have ~80% if they are lucky!
|
| 350.000 was the total number of players during 6 years.
| szastamasta wrote:
| My experience with sse is pretty bad. They are unreliable, don't
| support headers and require keep-alive hackery. In my experience
| WebSockets are so much better.
|
| Also ease of use doesn't really convince me. It's like 5 lines of
| code with socket.io to have working websockets, without all the
| downsides of sse.
| mikojan wrote:
| What? How do they not support headers?
|
| You have to send "Content-Type: text/event-stream" just to make
| them work.
|
| And you keep the connection alive by sending "Connection: keep-
| alive" as well.
|
| I've never had any issues using SSEs.
| szastamasta wrote:
| I mean you cannot send stuff from client. If you're using
| tokens for auth and don't want to use session cookies, you
| end with ugly polyfils.
| coder543 wrote:
| > If you're using tokens for auth and don't want to use
| session cookies
|
| That sounds like a self-inflicted problem. Even if you're
| using tokens, why not store them in a session cookie marked
| with SameSite=strict, httpOnly, and secure? Seems like it
| would make everything simpler, unless you're trying to
| build some kind of cross-site widget, I guess.
| szastamasta wrote:
| I need to work with more than 1 backend :)
| coder543 wrote:
| This is such an opaque response, I don't know what else
| could be said. If you're sending the same token to
| multiple websites, something feels very wrong with that
| situation. If it's all the same website, you can have
| multiple backends "mounted" on different paths, and that
| won't cause any problems with a SameSite cookie.
| szastamasta wrote:
| Then you need a single point of failure that is handling
| session validation. Without it part of your app might
| work even without your sessions storage.
| coder543 wrote:
| You can store a JWT in a session cookie. You don't need a
| SPoF for session validation, if that's not what you want.
| Kyro38 wrote:
| SSE won't work with tokens, see
| https://stackoverflow.com/questions/28176933/http-
| authorizat...
| ricardobeat wrote:
| Mind expanding on your experience and how are websockets more
| reliable than SSE? one of the main benefits of SSE is
| reliability from running on plain HTTP.
| dnautics wrote:
| I've done both. One big one is Sse connections will
| eventually time out, and you WILL have to renegotiate, so
| there will be a huge latency spike on those events. They are
| easier in elixir than most pls, but honestly if you're using
| elixir, you might as well use phoenix's builtin we socket
| support.
| odonnellryan wrote:
| How is this different from websockets? They will eventually
| close for various reasons, sometimes in not obvious ways.
| bullen wrote:
| Not if you send "noop" messages.
| dnautics wrote:
| In my experience, sse times out way more than ws, even if
| you are always sending (I was streaming jpegs using sse).
| jFriedensreich wrote:
| sounds like you did not really evaluate both technologies at
| the heart but only some libraries on top?
| szastamasta wrote:
| Yeah, sorry. In socket.io it's 2 lines. You need 5 lines with
| browser APIs :).
|
| You simply get stuff like auto-reconnect and graceful
| failover to long polling for free when using socket.io
| coder543 wrote:
| SSE EventSource also has built-in auto-reconnect, and it
| doesn't even need to support failover to long polling.
|
| Neither of those being built into a third party websocket
| library are actually _advantages_ for websocket... they
| just speak to the additional complexity of websocket. Plus,
| long polling as a fallback mechanism can only be possible
| with server side support for both long polling and
| websocket. Might as well just use SSE at that point.
| bullen wrote:
| They don't support headers in javascript, that is more a
| problem with javascript than SSE.
|
| Read my comment below about that.
| 88913527 wrote:
| HTTP headers must be written before the body; so once you start
| writing the body, you can't switch back to writing headers.
|
| Server-sent events appears to me to just be chunked transfer
| encoding [0], with the data structured in a particular way (at
| least from the perspective of the server) in this reference
| implementation (tl,dr it's a stream):
|
| https://gist.github.com/jareware/aae9748a1873ef8a91e5#file-s...
|
| [0]: https://en.wikipedia.org/wiki/Chunked_transfer_encoding
| patrickthebold wrote:
| Maybe I misunderstood your claim, but there is:
| https://developer.mozilla.org/en-
| US/docs/Web/HTTP/Headers/Tr...
|
| Which seems to be what you need to send 'headers' after a
| chunked response.
| 88913527 wrote:
| You understood correctly; I was mis-informed. Today I
| learned about the "Trailer" header. I'm curious how HTTP
| clients handle that. A client like window.fetch will
| resolve with a headers object-- does it include the
| trailers or not? I'd have to test it out.
| bullen wrote:
| I made the backend for this MMO on SSE over HTTP/1.1:
|
| https://store.steampowered.com/app/486310/Meadow/
|
| We have had a total of 350.000 players over 6 years and the
| backend out-scales all other multiplayer servers that exist and
| it's open source:
|
| https://github.com/tinspin/fuse
|
| You don't need HTTP/2 to make SSE work well. Actually the HTTP/2
| TCP head-of-line issue and all the workarounds for that probably
| make it harder to scale without technical debt.
| bastawhiz wrote:
| Can you explain how H2 would make it harder to scale SSE?
| bullen wrote:
| The mistake they did was to assume only one TCP socket should
| be used; the TCP has it's own head-of-line limitations just
| like HTTP/1.1 has if you limit the number of sockets
| (HTTP/1.1 had 2 sockets allowed per client, but Chrome
| doesn't care...) it's easily solvable by using more sockets
| but then you get into concurrency problems between the
| sockets.
|
| That said if you, like SSE on HTTP/1.1; use 2 sockets per
| client (breaking the RFC, one for upstream and one for
| downstream) you are golden but then why use HTTP/2 in the
| first place?
|
| HTTP/2 creates more problems than solutions and so does
| HTTP/3 unfortunately until their protocol fossilizes which is
| the real feature of a protocol, to become stable so everyone
| can rely on things working.
|
| In that sense HTTP/1.1 is THE protocol of human civilization
| until the end of times; together with SMTP (the oldest
| protocol of the bunch) and DNS (which is centralized and
| should be replaced btw).
| jupp0r wrote:
| The issues with TCP head-of-line blocking are resolved in
| HTTP/3 (QUIC).
| bullen wrote:
| Sure but then HTTP/3 is still binary and it's in flux
| meaning most routers don't play nice with it yet and
| since HTTP/1.1 works great for 99.9% of the usecases I
| would say it's a complete waste of time, unless you have
| some new agenda to push.
|
| Really people should try and build great things on the
| protocols we have instead of always trying to re-discover
| the wheel, note: NOT the same as re-inventing the wheel:
| http://move.rupy.se/file/wheel.jpg
| bushbaba wrote:
| For FANG scale, a 1% performance improvement for certain
| services has measurable business results.
|
| Take Snap. Say they reduced time to view a snap by 10ms.
| After 100 snaps that's an additional 1 second of
| engagement. This could equate to an additional ad
| impression every week per user. Which is many millions of
| additional revenue.
| vlovich123 wrote:
| HTTP/3 is E2E encrypted and built on UDP. What does "most
| routers don't play nice with it yet" mean in that
| context? Do you mean middleware boxes/routers rather than
| end user routers?
| ikiris wrote:
| It means they don't actually understand networking, but
| think they do.
| klabb3 wrote:
| > HTTP/3 is E2E encrypted
|
| Please elaborate.
| homarp wrote:
| https://www.youtube.com/watch?v=J4fR5aztSwQ - Securing
| the Next Version of HTTP: How QUIC and HTTP/3 Compare to
| HTTP/2
|
| "QUIC is a new always-encrypted general-purpose transport
| protocol being standardized at the IETF designed for
| multiplexing multiple streams of data on a single
| connection. HTTP/3 runs over QUIC and roughly replaces
| HTTP/2 over TLS and TCP. QUIC combines the cryptographic
| and transport handshakes in a way to allow connecting to
| a new server in a single round trip and to allow
| establishing a resumed connection in zero round trips,
| with the client sending encrypted application data in its
| first flight. QUIC uses TLS 1.3 as the basis for its
| cryptographic handshake.
|
| This talk will provide an overview of what the QUIC
| protocol does and how it works, and then will dive deep
| into some of the technical details. The deep dive will
| focus on security-related aspects of the protocol,
| including how QUIC combines the transport and
| cryptographic handshakes, and how resumption, including
| zero-round-trip resumption works. This will also cover
| how QUIC's notion of a connection differs from the
| 5-tuple sometimes used to identify connections, and what
| QUIC looks like on the wire.
|
| In addition to covering details of how QUIC works, this
| talk will also address implementation and deployment
| considerations. This will include how a load balancer can
| be used with cooperating servers to route connections to
| a fleet of servers while still maintaining necessary
| privacy and security properties. It will also look back
| at some of the issues with HTTP/2 and discuss which ones
| may need to be addressed in QUIC implementations as well
| or are solved by the design of QUIC and HTTP/3."
| anderspitman wrote:
| For me the question is not so much "yet" as "maybe
| never", since some networks block UDP altogether, and
| HTTP/3 has a robust fallback mechanism.
| fwsgonzo wrote:
| While I agree, we shouldn't discount one less RTT for
| encrypted connections. Latency is a problem that never
| really goes away, and we can only try to reduce RTTs.
| y4mi wrote:
| > _and DNS (which is centralized and should be replaced
| btw_
|
| So much nonsense in a single paragraph, amazing.
|
| If anything DNS is _less_ centralized then http and SMTP.
| Its a surprisingly complicated system for what it does
| because of all the caching etc, but calling it more
| centralized then http is just is just ignorant to a silly
| degree
| dlsa wrote:
| Couldn't find a license file in the root folder of that github.
| I found a license in a cpp file buried in the sec folder. You
| should consider putting the licensing for this kind of project
| in a straightforward and locatable place.
| smashah wrote:
| Love your hybrid model via gumroad! I do something similar for
| my own open-source project
|
| https://github.com/open-wa/wa-automate-nodejs
|
| There should be some sort of support group for those of us
| trying to monetize (sans donations) our open source projects!
| jayd16 wrote:
| >backend out-scales all other multiplayer servers
|
| Can you explain what you mean here? What was your peak active
| user count, what was peak per server instance, and why you
| think that beats anything else?
| rlabrecque wrote:
| Agreed, I'm curious as well. We load tested with real-clients
| faux-users, up to 1 million concurrent. And only stopped at 1
| million because the test was becoming cost prohibitive.
| shams93 wrote:
| Its a lot easier to scale than websockets where you need a pub
| sub solution and a controller to published shared state
| changes. See is really simply incomparision
| herodoturtle wrote:
| Nice work, thanks for sharing.
| stavros wrote:
| Probably not the same person, but did you ever play on RoD by
| any chance?
| bullen wrote:
| Probably not, since I dont know what RoD is.
| stavros wrote:
| I thought so, thanks!
| HWR_14 wrote:
| Your license makes some sense, but it seems to include a
| variable perpetual subscription cost via gumroad. Without an
| account (assuming I found the right site), I have no idea what
| you would be asking for. I recommend making it a little clearer
| on the landing page.
|
| That's said, it's very cool. Do you have a development blog for
| Meadow?
| Aeolun wrote:
| > MIT but [bunch of stuff]
|
| Not MIT then. The beauty of MIT is that there _is_ no stuff.
| bullen wrote:
| We already discussed this in an earlier thread, and however
| bad this looks it's better than my own license.
|
| Here it's clear, you can either use the code without money
| involved and then you have MIT (+ show logo and some
| example code is still mine).
|
| If you want money then you have to share some of it.
| dkersten wrote:
| While I get and support the intent, I don't like this
| usage of the name of the MIT license. I personally like
| the license because it tells me at a glance that I can
| use it for any purpose, commercial or otherwise, as long
| as the copyright and license is included and that there
| is no warranty. That's it, no complications, no other
| demands, no "if it makes X money or not", just I include
| the copyright and license terms and that's it, I can use
| the software whichever way I like.
|
| Your license is not that. You have extra conditions that
| add complexity. I can no longer go "oh like MIT" and
| immediately use it for any purpose, because you require
| extras especially if I were to make money. That seems
| completely against the spirit of the simplicity of the
| MIT license which says you can do whatever you like,
| commercial or otherwise, as long as the copyright and
| license are included.
|
| I think you should make your own license that includes
| the text of the MIT license, except removing the
| irrelevant parts (ie the commercial aspects include a
| caveat about requiring payment). You can still have a
| separate line of text explaining that the license is like
| the MIT license but with XYZ changes (basically the text
| you have now). But the license is not the MIT license and
| you should therefore have a separate license text that
| spells it out exactly. Not "its this, except scratch half
| of it because these additional terms override a good
| chunk of it".
| smw wrote:
| I respect your right to license you product however you
| want, but please don't call that open source.
| bullen wrote:
| Open-source also means the source is open, what you are
| looking for is free and honestly nothing is free... if
| you have a better term I'm open for suggestions.
|
| But really open-source (as in free) is the misnomer here,
| it should be called free open-source, or FOSS as some
| correctly name it.
| e12e wrote:
| That battle has been fought already, and the accepted
| term is "source available", not "open source". (And gnu
| adds Free or "libre" software, which is software licence
| in a way that tries to ensure the "four freedoms" for
| _all downstream users_ of software - such as freedom zero
| - the right to run software (no need for eg:
| cryptographic signature /trusted software - without a way
| for the _user_ to define trust).
| bullen wrote:
| Ok, fixed it elsewhere and in my brain... :/ Thx! Can't
| edit the comment though.
| dragonwriter wrote:
| > FOSS
|
| FOSS or F/OSS is a combination of Free (as defined by the
| FSF) and Open Source (as defined by the OSI) (the last S
| is Software), which recognizes that the two terms, while
| they come from groups with different ideological
| motivations, refer to approximately the same substantive
| licensing features and almost without exception the same
| set of licenses.
| hamburglar wrote:
| Personally, while I appreciate the difference from a
| promotion-of-FOSS point of view, I find it obnoxious that
| FOSS idealists think they can dictate the usage of the
| generic phrase "open source" and start these kinds of
| arguments in threads where non-completely-free software
| whose source is open comes up. We haven't all agreed on
| your terminology, and the argument is not "settled"
| except in the minds of the folks who think everyone
| should be on board with making this purity distinction.
| Some people find the distinction uninteresting and don't
| need to bother themselves with the ideological argument
| or agree to its terminology. And trying to be the
| arbiters of language is not a good look for the
| "information wants to be free" crowd.
| Plasmoid wrote:
| I've seen these referred to as "Source-available
| licenses". This would cover things like Mongo's SSPL.
|
| The bare reality is that it's just a commercial license.
| Spivak wrote:
| Requiring attribution doesn't make something not open
| source. At best this means that the example code isn't
| open source.
| bullen wrote:
| Added link in the readme! Thx.
|
| No, no dev log but I'll tell you some things that where
| incredible during that project:
|
| - I started the fuse project 4 months before I set foot in
| the Meadow project office (we had like 3 meetings during
| those 4 months just to touch base on the vision)! This is a
| VERY good way of making things smooth, you need to give
| tech/backend/foundation people a head start of atleast 6
| months in ANY project.
|
| - We spent ONLY 6 weeks (!!!) implementing the entire games
| multiplayer features because I was 100% ready for the job
| after 4 months. Not a single hickup...
|
| - Then for 7 months they finished the game client without me
| and released without ANY problems (I came back to the office
| that week and that's when I solved the anti-virus/proxy
| cacheing/buffering problem!).
|
| I think Meadow is the only MMO in history so far to have ZERO
| breaking bugs on release (we just had one UTF-8 client bug
| that we patched after 15 minutes and nobody noticed except
| the poor person that put a strange character in their name).
| llacb47 wrote:
| Google uses SSE for hangouts/gchat.
| KaoruAoiShiho wrote:
| I have investigated SSE for https://fiction.live a few years back
| but stayed with websockets. Maybe it's time for another look. I
| pay around $300 a month for the websocket server, it's probably
| not worth it yet to try to optimize that but if we keep growing
| at this rate it may soon be.
| oneweekwonder wrote:
| Personally i use mqtt over websockets, paho[0] is a good js
| library. It support last will for dc's and the message queue
| design makes it easy to think of and debug. There also a lot of
| mq brokers that will scale well.
|
| [0]:
| https://www.eclipse.org/paho/index.php?page=clients/js/index...
| wedn3sday wrote:
| This seems fairly cool, and I appreciate the write up, but god I
| hate it so much when people write code samples that try and be
| fancy and use non-code-characters in their code samples. Clarity
| is much more important then aesthetics when it comes to code
| examples, if Im trying to understand something I've never seen
| before, having a bunch of extra non-existant symbols does not
| help.
| asiachick wrote:
| Agreed. I use them in my editor but I ban them from my blog
| posts. They aren't helpful to others
| Rebelgecko wrote:
| Which characters, the funky '[?]'? I've seen those pop up a few
| other times recently, which makes me wonder if there's some
| editor extension that just came out that maps != and !==
| roywiggins wrote:
| They're ligatures.
|
| https://github.com/tonsky/FiraCode
| DHowett wrote:
| I'm guessing that you are referring to the "coding ligatures"
| in the author's font selection for code blocks?
|
| You can likely configure your user agent to ignore site-
| specified fonts.
| loh wrote:
| Are you referring to the `!==` and `=>` in their code being
| converted to what appears to be a single symbol?
|
| Upon further inspection, it looks like the actual code on the
| page is `!==` and `=>` but the font ("Fira Code") seems to be
| somehow converting those sequences of characters into a single
| symbol, which is actually still the same number of characters
| but joined to appear as a single one. I had no idea fonts could
| do that.
| red_trumpet wrote:
| That's called a ligature[1], and clasically used for joining
| for example ff or fi into more readable symbols.
|
| [1] https://en.wikipedia.org/wiki/Ligature_(writing)
| dpweb wrote:
| Very easy to implement - still using code I wrote 8 years ago,
| which is like 20 lines client and server, choosing it at the time
| over ws.
|
| Essentially just new EventSource(), text/event-stream header, and
| keep conn open. Zero dependencies in browser and nodejs. Needs no
| separate auth.
| nickjj wrote:
| This is why I really really like Hotwire Turbo[0] which is a
| back-end agnostic way to do fast and partial HTML based page
| updates over HTTP and it optionally supports broadcasting events
| with WebSockets (or SSE[1]) only when it makes sense.
|
| So many alternatives to Hotwire want to use WebSockets for
| everything, even for serving HTML from a page transition that's
| not broadcast to anyone. I share the same sentiment as the author
| in that WebSockets have real pitfalls and I'd go even further and
| say unless used tastefully and sparingly they break the whole
| ethos of the web.
|
| HTTP is a rock solid protocol and super optimized / well known
| and easy to scale since it's stateless. I hate the idea of going
| to a site where after it loads, every little component of the
| page is updated live under my feet. The web is about giving users
| control. I think the idea of push based updates like showing
| notifications and other minor updates are great when used in
| moderation but SSE can do this. I don't like the direction of
| some frameworks around wanting to broadcast everything and use
| WebSockets to serve HTML to 1 client.
|
| I hope in the future Hotwire Turbo alternatives seriously
| consider using HTTP and SSE as an official transport layer.
|
| [0]: https://hotwired.dev/
|
| [1]: https://twitter.com/dhh/status/1346095619597889536?lang=en
| gibsonf1 wrote:
| Solid has a great solution for this:
| https://solid.github.io/notifications/protocol
| mmcclimon wrote:
| SSEs are one of the standard push mechanisms in JMAP [1], and
| they're part of what make the Fastmail UI so fast. They're
| straightforward to implement, for both server and client, and the
| only thing I don't like about them is that Firefox dev tools make
| them totally impossible to debug.
|
| 1. https://jmap.io/spec-core.html#event-source
| ok_dad wrote:
| > the only thing I don't like about them is that Firefox dev
| tools make them totally impossible to debug
|
| You can't say that and not say more about it, haha. Please
| expand on this?
|
| Also, I'm a Fastmail customer and appreciate the nimble UI,
| thanks!
| coder543 wrote:
| I think their information could be outdated. Since Firefox
| 82, you can supposedly inspect the content of SSE streams:
| https://developer.mozilla.org/en-
| US/docs/Tools/Network_Monit...
|
| Before that... yeah, the Firefox dev tools were not very
| helpful for SSE.
| mmcclimon wrote:
| Hmm! You're right that I hadn't looked it a while, so I
| checked before making the comment above. I'm still seeing
| the same thing I always have, which is "No response data
| available for this request". Possibly something is slightly
| wrong somewhere (though Chrome dev tools seem fine on the
| same), but you've given me something to look into, thanks!
| coder543 wrote:
| That is interesting. I just tested it myself, and at
| least for my setup (Firefox on Mac on ARM), the events
| only showed up in the dev tools if the server closed the
| SSE connection... so, maybe Firefox still hasn't fully
| fixed this problem.
| mmcclimon wrote:
| Yeah, that seems to be the case (confirmed with their
| little example at https://github.com/mdn/dom-
| examples/tree/master/server-sent-...). Once the
| connection is closed you can see things, but that's not
| particularly useful for debugging!
| dnr wrote:
| The Fastmail UI is indeed snappy, except when it suddenly
| decides it has to reload the page, which seems to be multiple
| times a day these days (and always when I need to search for a
| specific email). Can you make it do what one of my other
| favorite apps does: when there's a new version available, make
| a small pop up with a reload button, but don't force a reload
| (until maybe weeks later)?
| alin23 wrote:
| ESPHome (an easy to use firmware for ESP32 chips) uses SSE to
| send sensor data to subscribers.
|
| I made use of that in Lunar (https://lunar.fyi/#sensor) to be
| able to adjust monitor brightness based on ambient light readings
| from an external wireless sensor.
|
| At first it felt weird that I have to wait for responses instead
| of polling with requests myself, but the ESP is not a very
| powerful chip and making one HTTP request every second would have
| been too much.
|
| SSE also allows the sensor to compare previous readings and only
| send data when something changed, which removes some of the
| complexity with debouncing in the app code.
| rcarmo wrote:
| I have always preferred SSE to WebSockets. You can do a _lot_
| with a minuscule amount of code, and it is great for updating
| charts and status UIs on the fly without hacking extra ports,
| server daemons and whatnot.
| njx wrote:
| My theory why SSE did not take off is because WordPress does not
| support it.
| havkom wrote:
| The most compatible technique is long polling (with a re-
| established connection after X seconds if no event). Works
| suprisingly well in many cases and is not blocket by any proxies.
| bullen wrote:
| long-polling are blocked to almost exactly the same extent as
| comet-stream and SSE. The only thing you have to do is to push
| more data on the response so that the proxy is forced to flush
| the response!
|
| Since IE7 is no longer used we can bury long-polling for good.
| notreallyserio wrote:
| How much more data do you have to send? Is it small enough
| you aren't concerned about impacting user traffic quotas?
| bullen wrote:
| Just enough to trigger the buffer... 1024-8192 bytes or
| something like that... a fart in space since it's just once
| per session!
| U1F984 wrote:
| The extra setup step for websocket should not be required:
| https://caddyserver.com/docs/v2-upgrade#proxy
|
| I also had no problems with HAProxy, it worked with websockets
| without any issues or extra handling.
| laerus wrote:
| With WebTransport around the corner I don't think is worth the
| time investing in learning a, what seems to me, obsolete
| technology. I can understand it for already big projects working
| with SSE that don't want to pay the cost of upgrading/changing
| but for anything new I cannot be bothered since Websockets work
| good enough for my use cases.
|
| What worries me though is the trend of dismissal of newer
| technologies as being useless or bad and the resistance to
| change.
| slimsag wrote:
| I'm confused, you believe that web developers have a trend of
| dismissing newer technologies and resistance to change? Have I
| missed something or..?
| jessaustin wrote:
| Around the corner? There seems to be nothing about this in any
| browser. [0] That would put this what, five years out before it
| could be used in straightforward fashion? Please be practical.
|
| [0] https://caniuse.com/?search=webtransport
| 0xbkt wrote:
| See https://github.com/Fyrd/caniuse/issues/5707 and
| https://chromestatus.com/feature/4854144902889472#consensus.
| jessaustin wrote:
| Thanks for pointing that out. I suppose caniuse missing out
| on the first experimental version of the first browser to
| support this isn't terribly misleading. Maybe when they get
| the basics of the API figured out we can start deprecating
| other things...
| coder543 wrote:
| WebTransport seems like it will be significantly lower level
| and more complex to use than SSE, both on the server and the
| client. To say that this "obsoletes" SSE seems like a serious
| stretch.
|
| SSE runs over HTTP/3 just as well as any other HTTP feature,
| and WebTransport is built on HTTP/3 to give you much finer
| grained control of the HTTP/3 streams. If your application
| doesn't benefit significantly from that control, then you're
| just adding needless complexity.
| anderspitman wrote:
| My personal browser streaming TL;DR goes something like this:
|
| * Start with SSE
|
| * If you need to send binary data, use long polling or WebSockets
|
| * If you need fast bidi streaming, use WebSockets
|
| * If you need backpressure and multiplexing for WebSockets, use
| RSocket or omnistreams[1] (one of my projects).
|
| * Make sure you account for SSE browser connection limits,
| preferably by minimizing the number of streams needed, or by
| using HTTP/2 (mind head-of-line blocking) or splitting your
| HTTP/1.1 backend across multiple domains and doing round-robin on
| the frontend.
|
| [0]: https://rsocket.io/
|
| [1]: https://github.com/omnistreams/omnistreams-spec
| whazor wrote:
| I tried out server side events, but they are still quite
| troubling with the lack of headers and cookies. I remember I
| needed some polyfill version which gave more issues.
| bullen wrote:
| How do you mean lack of headers and cookies?
|
| That is wrong. Edit: Actually it seems correct (a javascript
| problem, not SSE problem) but it's a non-problem if you use a
| parameter for that data instead and read it on the server.
| tytho wrote:
| You cannot send custom headers when using the built-in
| EventSource[1] constructor, however you can pass the
| 'include' value to the credentials option. Many polyfills
| allow custom headers.
|
| However you are correct that if you're not using JavaScript
| and connecting directly to the SSE endpoint via something
| else besides a browser client, nothing is preventing anyone
| from using custom headers.
|
| [1] https://developer.mozilla.org/en-
| US/docs/Web/API/EventSource...
| withinboredom wrote:
| I'm pretty sure I saw him sending headers in the talk. Did
| you watch the talk?
| tytho wrote:
| He was likely using a polyfill. It's definitely not in
| the spec and there's an open discussion about trying to
| get it added: https://github.com/whatwg/html/issues/2177
| bullen wrote:
| Aha, well why do you need to send a header when you can
| just put the data on the GET URL like so
| "blabla?cookie=erWR32" for example?
|
| In my example I use this code: var
| source = new EventSource('pull?name=one');
| source.onmessage = function (event) {
| document.getElementById('events').innerHTML += event.data;
| };
| tytho wrote:
| I think that works great! The complaint I've heard is
| that you may need to support multiple ways to
| authenticate opening up more attack surface.
| kreetx wrote:
| What if you use http-only cookies?
| tytho wrote:
| You can pass a 'withCredentials' option.
| samwillis wrote:
| I have used SSEs extensively, I think they are brilliant and
| massively underused.
|
| The one thing I wish they supported was a binary event data type
| (mixed in with text events), effectively being able to send in my
| case image data as an event. The only way to do it currently is
| as a Base64 string.
| jtwebman wrote:
| Send an event that tells the browser to request the binary
| image.
| samwillis wrote:
| In my case I was aiming for low latency with a dynamically
| generated image. To send a url to a saved image, I would have
| to save it first to a location for the browser to download it
| form. That would add at least 400ms, probably more.
|
| Ultimately what I did was run an SSE request and long polling
| image request in parallel, but that wasn't ideal as I had to
| coordinate that on the backend.
| bckr wrote:
| I'm curious if you could have kept the image in memory (or
| in Redis) and served it that way
| samwillis wrote:
| That's actually not too far from what we do. The image is
| created by a backend service with communication (queue
| and responses) to the front end servers via Redis.
| However rather than saving the image in its entirety to
| Redis, it's streamed via it in chunks using LPUSH and
| BLPOP.
|
| This lets us then stream the image as a steaming http
| response from the front end, potentially before the jpg
| has finished being generated on the backend.
|
| So from the SSE we know the url the image is going to be
| at before it's ready, and effectively long poll with a
| 'new Image()'.
| keredson wrote:
| SSE supports gzip compression, and a gzip-ed base64 is almost
| as small as the original jpg:
|
| $ ls -l PXL_20210926_231226615.*
|
| -rw-rw-r-- 1 derek derek 8322217 Feb 12 09:20
| PXL_20210926_231226615.base64
|
| -rw-rw-r-- 1 derek derek 6296892 Feb 12 09:21
| PXL_20210926_231226615.base64.gz
|
| -rw-rw-r-- 1 derek derek 6160600 Oct 3 15:31
| PXL_20210926_231226615.jpg
| samwillis wrote:
| Quite true, however from memory Django doesn't (or didn't)
| support gzip on streaming responses and as we host on Heroku
| we didn't want to introduce another http server such as Nginx
| into the Heroku Dyno.
|
| As an aside, Django with Gevent/Gunicorn does SSE well from
| our experience.
| goodpoint wrote:
| --- WebSockets cannot benefit from any HTTP feature. That is:
| No support for compression No support for HTTP/2
| multiplexing Potential issues with proxies No
| protection from Cross-Site Hijacking
|
| ---
|
| Is that true? The web never cease to amaze.
| __s wrote:
| WebSockets support compression (ofc, the article goes on to
| detail this & point out flaws. I'd argue that compression is
| not generally useful in web sockets in the context of many
| small messages, so it makes sense to be default-off for servers
| as it's something which should be enabled explicitly when
| necessary, but the client should be default-on since the server
| is where the resource usage decision matters)
|
| I don't see why WebSockets should benefit from HTTP. Besides
| the handshake to setup the bidirectional channel, they're a
| separate protocol. I'll agree that servers should think twice
| about using them: they necessitate a lack of statelessness &
| HTTP has plenty of benefits for most web usecases
|
| Still, this is a good article. SSE looks interesting. I host an
| online card game openEtG, which is far enough from real time
| that SSE could potentially be a way to reduce having a
| connection to every user on the site
| bullen wrote:
| The problem with WebSockets is that hey are:
|
| 1) More complex and binary so you cannot debug them as easily,
| specially on live and specially if you use HTTPS.
|
| 2) The implementations don't parallelize the processing, with
| Comet-Stream + SSE you just need to find a application server
| that has concurrency and you are set to scale on the entire
| machines cores.
|
| 3) WebSockets still have more problems with Firewalls.
| sb8244 wrote:
| I can't find any downsides of SSE presented. My experience is
| that they're nice in theory but the devils in the details. The
| biggest issue being that you basically need http/2 to make them
| practical.
| anderspitman wrote:
| In some cases you might actually be better served sticking with
| HTTP/1.1 and serving SSE over several domains, to avoid HTTP/2
| head-of-line blocking.
| bullen wrote:
| Absolutely not, HTTP/1.1 is the way to make SSE fly:
|
| https://github.com/tinspin/rupy/wiki/Comet-Stream
|
| Old page, search for "event-stream"... Comet-stream is a
| collection of techniques of which SSE is one.
|
| My experience is that SSE goes through anti-viruses better!
| mwcampbell wrote:
| > My experience is that SSE goes through anti-viruses better!
|
| Hmm, another commenter says the opposite:
|
| https://news.ycombinator.com/item?id=30313692
| bullen wrote:
| He just needs to push more data on the reply to force the
| anti-virus to flush the data. Easy peasy.
| anderspitman wrote:
| Take this for what it's worth, but I see you share rupy on
| pretty much every thread that mentions WebSockets, and I
| click on the link pretty much every time, and I still have
| basically no idea what it is. Documentation probably isn't
| your priority at the moment, but even just a couple
| paragraphs could go a long way.
| ByThyGrace wrote:
| I had the same impression as you. I want to learn more
| about fuse but even their "sales pitch" page is in the same
| tone of "fuse can do a lot" (and that's fine, I'm sold!)
| except there is very little documentation at the moment.
| kreetx wrote:
| SSEs had a severe connection limit, something like 4 connections
| per domain per browser (IIRC), so if you had four tabs open then
| opening new ones would fail.
| oplav wrote:
| 6 connections per domain per browser:
| https://bugs.chromium.org/p/chromium/issues/detail?id=275955
|
| There are some hacks to work around it though.
| coder543 wrote:
| Browsers also limit the number of websocket connections. But,
| if you're using HTTP/2, as you should be, then the multiplexing
| means that you can have effectively unlimited SSE connections
| through a limited number of TCP connections, and those TCP
| connections will be shared across tabs.
|
| (There's one person in this thread who is just ridiculously
| opposed to HTTP/2, but... HTTP/2 has serious benefits. It
| wasn't developed in a vacuum by people who had no idea what
| they were doing, and it wasn't developed aimlessly or without
| real world testing. It is used by pretty much all major
| websites, and they _absolutely_ wouldn 't use it if HTTP/1.1
| was better... those major websites exist to serve their
| customers, not to conspiratorially push an agenda of broken
| technologies that make the customer experience worse.)
| anderspitman wrote:
| You can also make your HTTP/1.1 SSE endpoints available on
| multiple domains and have the client round-robin them.
| Obviously adds some complexity, but sometimes it's a tradeoff
| worth making for example if you're on lossy networks and
| trying to avoid HTTP/2 head-of-line blocking.
| jcheng wrote:
| > Browsers also limit the number of websocket connections
|
| True but the limit for websockets these days is in the
| hundreds, as opposed to 6 for regular HTTP requests.
| coder543 wrote:
| https://stackoverflow.com/questions/26003756/is-there-a-
| limi...
|
| It appears to be 30 per domain, not "hundreds", at least as
| of the time this answer was written. I didn't see anything
| more recent that contradicted this.
|
| In practice, this is unlikely to be problematic unless
| you're using multiple websockets per page, but the limit of
| 6 TCP connections is _even less likely_ to be a problem if
| you're using HTTP /2, since those will be shared across
| tabs, which isn't the case for the dedicated connection
| used for each websocket.
| jcheng wrote:
| It's 255 for Chrome and has been since 2015, 200 for
| Firefox since longer than that.
|
| https://chromium.googlesource.com/chromium/src/net/+/259a
| 070...
|
| Agree that it should be _much_ less of a problem with
| HTTP /2 than HTTP/1.1.
| jshen wrote:
| Question for those of you who build features on web using things
| like SSE or web sockets, how do you build those features in
| native mobile apps?
| johnny22 wrote:
| isn't that just an event dispatcher?
| mythz wrote:
| We use SSE for our APIs Server Events feature
| https://docs.servicestack.net/server-events with C#,
| JS/TypeScript and Java high-level clients.
|
| It's a beautifully simple & elegant lightweight push events
| option that works over standard HTTP, the main gotcha for
| maintaining long-lived connections is that server/clients should
| implement their own heartbeat to be able to detect & auto
| reconnect failed connections which was the only reliable way
| we've found to detect & resolve broken connections.
| dabeeeenster wrote:
| "the main gotcha for maintaining long-lived connections is that
| server/clients should implement their own heartbeat to be able
| to detect & auto reconnect failed connections"
|
| That sounds like a total nightmare!
| ec109685 wrote:
| Definitely needed. That would be true for all long lived
| connection protocols in order to detect connection
| interruptions in a timely fashion.
| ravenstine wrote:
| I usually use SSEs for personal projects because they are way
| more simple than WebSockets (not that those aren't also simple)
| and most of the time my web apps just need to listen for
| something coming from the server and not bidirectional
| communication.
| Too wrote:
| Can someone give a brief summary of how this differs from long
| polling. It looks very similar except it has a small layer of
| formalized event/data/id structure on top? Are there any
| differences in the lower connection layers, or any added support
| by browsers and proxies given some new headers?
|
| What are the benefits of SSE vs long polling?
| anderspitman wrote:
| SSE doesn't support binary data. Text only.
| TimWolla wrote:
| > What are the benefits of SSE vs long polling?
|
| The underlying mechanism effectively is the same: A long
| running HTTP response stream. However long-polling commonly is
| implemented by "silence" until an event comes in and then
| performing another request to wait for the next event, whereas
| SSE sends you multiple events per request.
| waylandsmithers wrote:
| I had the pleasure of being forced to use in SSE due to working
| with a proxy that didn't support websockets.
|
| Personally I think it's a great solution for longer running tasks
| like "Export your data to CSV" when the client just needs to get
| an update that it's done and here's the url to download it.
| axiosgunnar wrote:
| So do I understand correctly that when using SSE, the login
| cookie of the user is not automatically sent with the SSE request
| like it is with all normal HTTP requests? And I have to redo auth
| somehow?
| bastawhiz wrote:
| It should automatically send first party cookies, though you
| may need to specify withCredentials.
| ponytech wrote:
| One problem I had with WebSockets is you can not set custom HTTP
| headers when opening the connection. I wanted to implement a JWT
| based authentication in my backend and had to pass the token
| either as a query parameter or in a cookie.
|
| Anyone knows the rationale behind this limitation?
| charlietran wrote:
| The workaround/hack is to send your token via the "Sec-
| WebSocket-Protocol" header, which is the one header you're
| allowed to set in browser when opening a connection. The catch
| is that your WebSocket server needs to echo this back on a
| successful connection.
| TimWolla wrote:
| > RFC 8441, released on September 2018, tries to fix this
| limitation by adding support for "Bootstrapping WebSockets with
| HTTP/2". It has been implemented in Firefox and Chrome. However,
| as far as I know, no major reverse-proxy implements it.
|
| HAProxy supports RFC 8441 automatically. It's possible to disable
| it, because support in clients tends to be buggy-ish:
| https://cbonte.github.io/haproxy-dconv/2.4/configuration.htm...
|
| Generally I can second recommendation of using SSE / long running
| response streams over WebSockets for the same reasons as the
| article.
| The_rationalist wrote:
| for bidi Rsocket is much better than wevsocket, in fact its
| official support is the best feature of spring boot
| julianlam wrote:
| This is really interesting! I wonder why it never really took
| off, whereas websockets via Socket.IO/Engine.io did.
|
| At NodeBB, we ended up relying on websockets for almost
| everything, which was a mistake. We were using it for simple
| call-and-response actions, where a proper RESTful API would've
| been a better (more scalable, better supported, etc.) solution.
|
| In the end, we migrated a large part of our existing socket.io
| implementation to use plain REST. SSE sounds like the second part
| of that solution, so we can ditch socket.io completely if we
| really wanted to.
|
| Very cool!
| shahinghasemi wrote:
| > At NodeBB, we ended up relying on websockets for almost
| everything, which was a mistake.
|
| Would you please elaborate on the challenges/disadvantages
| you've encountered in comparison to REST/HTTP?
| foxbarrington wrote:
| I'm a huge fan of SSE. In the first chapter of my book Fullstack
| Node.js I use it for the real-time chat example because it
| requires almost zero setup. I've also been using SSE on
| https://rambly.app to handle all the WebRTC signaling so that
| clients can find new peers. Works great.
| viiralvx wrote:
| Rambly looks sick, thanks for sharing!
| rough-sea wrote:
| A complete SSE example in 25 lines on Deno Deploy:
| https://dash.deno.com/playground/server-sent-events
| pictur wrote:
| Does SSE offer support for capturing connect/disconnect
| situations?
| bullen wrote:
| The TCP stack can give you that info if you are lucky in your
| topography but generally you cannot rely on this working 100%.
|
| The way I solve it is to send "noop" messages at regular
| intervals so that the socket write will return -1 and then I
| know something is off and reconnect.
| rawoke083600 wrote:
| I like them, they surprisingly easy to use..
|
| One example where i found it to be not the _perfect_ solution was
| with a web turn-based game.
|
| The SSE was perfect to update gamestate to all clients, but to
| have great latency from the players point of view whenever the
| player had to do something, it was via a normal ajax-http call.
|
| Eventually I had to switch to _uglier_ websockets and keep
| connection open.
|
| Http-keep-alive was that reliable.
| coder543 wrote:
| With HTTP/2, the browser holds a TCP connection open that has
| various streams multiplexed on top. One of those streams would
| be your SSE stream. When the client makes an AJAX call to the
| server, it would be sent through the already-open HTTP/2
| connection, so the latency is very comparable to websocket --
| no new connection is needed, no costly handshakes.
|
| With the downsides of HTTP/1.1 being used with SSE, websockets
| actually made a lot of sense, but in many ways they were a
| kludge that was only needed until HTTP/2 came along. As you
| said, communicating back to the server in response to SSE
| wasn't great with HTTP/1.1. That's before mentioning the
| limited number of TCP connections that a browser will allow for
| any site, so you couldn't use SSE on too many tabs without
| running out of connections altogether, breaking things.
| johnny22 wrote:
| I think it comes down to whether your your communication is
| more oriented towards sending than receiving. If the clients
| receive way more than they send, then SSE is probably fine, but
| if it's truly bidirectional then it might not work as well.
| bullen wrote:
| You just needed to send a "noop" (no operation) message at
| regular intervals.
| jcelerier wrote:
| that puts it instantly in the "fired if you ever use it" bin
| Vosporos wrote:
| Fired for using a keep-alive message???
| notreallyserio wrote:
| Why's that?
| hishamp wrote:
| We moved away from WebSockets to SSE, realised it wasn't makings
| thing any better. In fact, it made things worse, so we switched
| back to WebSockets again and worked on scaling WebSockets. SSE
| will work much better for other cases, just didn't work out for
| our case.
|
| First reason was that it was an array of connections you loop
| through to broadcast some data. We had around 2000 active
| connections and needed a less than 1000ms latency, with
| WebSocket, even though we faced connections drops, client
| received data on time. But in SSE, it took many seconds to reach
| some clients, since the data was time critical, WebSocket seemed
| much easier to scale for our purposes. Another issue was that SSE
| is like an idea you get done with HTTP APIs, so it doesn't have
| much support around it like WS. Things like rooms, clientIds etc
| needed to be managed manually, which was also a quite big task by
| itself. And a few other minor reasons too combined made us switch
| back to WS.
|
| I think SSE will suit much better for connections where bulk
| broadcast is less, like in shared docs editing, showing stuff
| like "1234 users is watching this product" etc. And keep in mind
| that all this is coming from a mediocre full stack developer with
| 3 YOE only, so take it with a grain of salt.
| DaiPlusPlus wrote:
| Your write-up sounds like your issues with SSE stemmed from the
| framework/platform/server-stack you're using rather than of any
| problems inherent in SSE.
|
| I haven't observed any latency or scaling issues with SSE - on
| the contrary: in my ASP.NET Core projects, running behind IIS
| (with QUIC enabled), I get better scaling and throughput with
| SSE compared to raw WebSockets (and still-better when compared
| to SignalR), though latency is already minimal so I don't think
| that can be improved upon.
|
| That said, I do prefer using the existing pre-built SignalR
| libraries (both server-side and client-side: browser and native
| executables) because the library's design takes away all the
| drudgery.
| nly wrote:
| Checkout nchan.io
| pbowyer wrote:
| There's also the Mercure protocol, built on top of Server-Sent
| Events: https://mercure.rocks/
| captn3m0 wrote:
| I think SSE might make a lot of sense for Serverless workloads?
| You don't have to worry about running a websocket server, any
| serverless host with HTTP support will do. Long-polling might be
| costlier though?
| tgv wrote:
| But SSE is a oneway street, isn't it? The client gets one chance
| to send days, and that's it? Or is there some way around it?
| jessaustin wrote:
| Clients can always send normal http messages to the server.
| Probably not ideal for "bi-directional" traffic, but it's an
| option in a pinch.
| leeoniya wrote:
| the biggest drawback with SSE, even when unidirectional comm is
| sufficient is
|
| > SSE is subject to limitation with regards to the maximum number
| of open connections. This can be especially painful when opening
| various tabs as the limit is per browser and set to a very low
| number (6).
|
| https://ably.com/blog/websockets-vs-sse
|
| SharedWorker could be one way to solve this, but lack of Safari
| support is a blocker, as usual. https://developer.mozilla.org/en-
| US/docs/Web/API/SharedWorke...
|
| also, for websockets, there are various libs that handle auto-
| reconnnects
|
| https://github.com/github/stable-socket
|
| https://github.com/joewalnes/reconnecting-websocket
|
| https://dev.to/jeroendk/how-to-implement-a-random-exponentia...
| coder543 wrote:
| This isn't a problem with HTTP/2. You can have as many SSE
| connections as you want across as many tabs as the user wants
| to use. Browsers multiplex the streams over a handful of shared
| HTTP/2 connections.
|
| If you're still using HTTP/1.1, then yes, this would be a
| problem.
| leeoniya wrote:
| hmmm, you might be right. i wonder what steered me away.
| maybe each SSE response re-sends headers, which can be larger
| than the message itself?
|
| maybe it was inability to do broadcast to multiple open sse
| sockets from nodejs.
|
| i should revisit.
|
| https://medium.com/blogging-greymatter-io/server-sent-
| events...
| bullen wrote:
| It used to be 2 sockets per client, so now it's 6?
|
| Well it's a non-problem, if you need more bandwith than one
| socket in each direction can provide you have much bigger
| problems than the connection limit; which you can just ignore.
| leeoniya wrote:
| the problem is multiple tabs. if you have, e.g. a bunch of
| Grafana dashboards open on multiple screens in different tabs
| (on same domain), you will exhaust your HTTP connection limit
| very quickly with SSE.
|
| in most cases this is not a concern, but in _some_ cases it
| is.
| bullen wrote:
| Aha, ok yes then you would need to have many subdomains?
|
| Or make your own tab system inside one browser tab.
|
| I can see why that is a problem for some.
| beebeepka wrote:
| So, what are the downsides to using websockets? They are my go-to
| solution when I am doing a game, chat, or something else that
| needs interactivity.
| bullen wrote:
| See my comment below:
| https://news.ycombinator.com/item?id=30313403
| herodoturtle wrote:
| Been reading all your comments on this thread (thank you)
| with interest.
|
| Can you recommend some resources for learning SSE in depth?
| bullen wrote:
| I would look at my own app-server:
| https://github.com/tinspin/rupy
|
| It's not the most well documented but it's the smallest
| implementation while still being one of the most performant
| so you can learn more than just SSE.
| mceachen wrote:
| https://developer.mozilla.org/en-US/docs/Web/API/Server-
| sent...
| mmzeeman wrote:
| Did research on SSE a short while ago. Found out that the
| mimetype "text/event-stream" was blocked by a couple of anti-
| virus products. So that was a no-go for us.
| bastawhiz wrote:
| How did you find that out?
| mmzeeman wrote:
| https://github.com/mmzeeman/mod_sse
| pornel wrote:
| It's not blocked. It's just that some very badly written
| proxies can try to buffer the "whole" response, and SSE is
| technically a never-ending file.
|
| It's possible to detect that, and fall back to long polling.
| Send an event immediately after opening a new connection, and
| see if it arrives at the client within a short timeout. If it
| doesn't, make your server close the connection after every
| message sent (connection close will make AV let the response
| through). The client will reconnect automatically.
|
| Or run: while(true) alert("antivirus software
| is worse than malware")
| ronsor wrote:
| These days I feel like the only way to win against poorly
| designed antiviruses and firewalls is to--ironically enough--
| behave like malware and obfuscate what's going on.
| captn3m0 wrote:
| I was using SSE when they'd just launched (almost a decade ago
| now) and never faced any AV issues.
| azinman2 wrote:
| Is that still the case now? How big and broad an audience do
| you have?
|
| My experience, now a bit dated, is that long polling is the
| only thing that will work 100% of the time.
| bullen wrote:
| They don't block it, they cache the response until there is
| enough data in the buffer... just push more garbage data on the
| first chunks...
| quickthrower2 wrote:
| Is it worth upgrading a long polling solution to SSE? Would I see
| much benefit?
|
| What I mean by that is client sends request, server responds in
| up to 2 minutes with result or a try again flag. Either way
| client resends request and then uses response data if provided.
| bullen wrote:
| Yes, since IE7 is out of the game long-polling is no longer
| needed.
|
| Comet-stream and SSE will save you alot of bandwidth and CPU!!!
| layer8 wrote:
| What is particular about IE7? According to
| https://caniuse.com/eventsource, SSE is unsupported through
| IE11.
| jFriedensreich wrote:
| this is what i have been telling people for years, but its hard
| to get the word out there. usually every dev just reflexes
| without thinking to websockets when anything realtime or push
| related comes up.
| steve76 wrote:
___________________________________________________________________
(page generated 2022-02-12 23:00 UTC)