[HN Gopher] Processing billions of events in real time
___________________________________________________________________
Processing billions of events in real time
Author : 1cvmask
Score : 35 points
Date : 2021-11-15 22:06 UTC (53 minutes ago)
(HTM) web link (blog.twitter.com)
(TXT) w3m dump (blog.twitter.com)
| javajosh wrote:
| Philosophically, isn't it _wrong_ to process billions of events
| in real time? I mean, the magnitude of the moral hazard is
| astounding. The alternative isn 't to stop sending messages, but
| rather to not put them through a bottleneck like Twitter. This
| would have the dual benefit of a) not giving central control over
| messages and b) not requiring exotic solutions.
|
| I fear that technologists (including myself) are fascinated by
| the exotic solutions required by extreme centralization, and are
| more than happy to solve those rather than question the need for
| them in the first place.
| LindyTalker wrote:
| Huh? People are using Twitter to create these events? Who else
| would you want to process it? It's not a bottleneck to put
| things through Twitter if people are literally using Twitter
| staticassertion wrote:
| I assume they're suggesting some sort of peer to peer system.
| endisneigh wrote:
| This is an interesting blog post, but one thing I really wish
| more blog posts would do is compare the actual user experience
| from the simpler architecture.
|
| For example this new architecture, from the end-users
| perspective, compared to 2010 Twitter would be interesting. I'm
| sure much of the technology is needed for monetization, but it
| would be a fascinating look anyway.
| blakesterz wrote:
| There's an HN post from a few years ago with a bunch of
| interesting reading from the "old days" at Twitter.
|
| https://news.ycombinator.com/item?id=17147404
|
| I think somewhere in there is a link to a story about how only
| one popular user (Was it Ashton Kutcher?) could tweet at a time.
| I seem to recall it ran on a singly MySQL server for quite a
| while too.
| Supermancho wrote:
| Where I work, they use more and more google services every day,
| to my chagrin (since I've spent years learning the quirks of
| AWS).
|
| Amazon's solutions can be fit into various architectures, but are
| more generalist tools than the BigQuery, DataFlow and BigTable
| application.
|
| Google's solutions are also cheaper and/or easier to work with,
| for very large data processing.
| arecurrence wrote:
| Cloud Service Usage Price matters a lot. I actively move
| systems off of AWS to save costs. Take Cloudfront as an
| example. Bunny is half a cent per GB and has no per-request
| cost. I've seen Cloudfront cost over 20x more. A $10,000 bill
| instead of $200,000 per month is well worth the engineering
| work.
|
| Btw, Bunny is also $10 flat rate for unlimited image
| processing... and they only bill the post-processed bandwidth
| cost. The price of my AWS equivalent for that pipeline was
| quite the comparison. :)
| tedmiston wrote:
| I misread your comment as "Bunny" being a GCP service
| competing with AWS CloudFront.
|
| Is Bunny Optimizer (bunny.net) [1] what you are referring to?
|
| [1]: https://bunny.net/optimizer/#pricing-details
| fuddle wrote:
| Is it just me or are most of the links broken in the blog post?
| arthurcolle wrote:
| what DNS server are you using, old sport?
| [deleted]
| fuddle wrote:
| Oops I meant to comment on this HN post: "Handling five billion
| sessions a day in real time"
| https://news.ycombinator.com/item?id=29231932
| zdwolfe wrote:
| Works for me
___________________________________________________________________
(page generated 2021-11-15 23:00 UTC)