[HN Gopher] Anything can be a message queue if you use it wrongl...
___________________________________________________________________
Anything can be a message queue if you use it wrongly enough
Author : cendyne
Score : 336 points
Date : 2023-06-04 15:06 UTC (7 hours ago)
(HTM) web link (xeiaso.net)
(TXT) w3m dump (xeiaso.net)
| NotYourLawyer wrote:
| [flagged]
| xena wrote:
| It's unfortunate that you were offended by the cartoon shark in
| the programming article. You are entitled to your own
| preferences and expectations, but so am I. I have the right to
| express myself creatively and share my content with the world
| for free. I don't have to please you or anyone else with my
| personal choices. Your feedback is noted, but please be more
| mindful and respectful of other readers who may like my work
| and the artists who were contracted to create the art in
| question. Thank you.
|
| One path to find peace is to kill the part of yourself that
| cringes rather than try to smother out others who are "cringe".
| Otherwise, there are plenty of cookie-cutter medium.com blogs
| and LinkedIn posts if you want something more to your
| standards.
| Lammy wrote:
| [flagged]
| retrocryptid wrote:
| Sadly, they fundamentally misunderstood the "everything is a
| file" paradigm of *nix. It's not that everything is an extent of
| octets, it's that everything has a directory entry so C programs
| can use the open call to create a file handle. It might be more
| appropriate to say "Everything looks like a file in the file
| system and most every operation on a thing represented by a
| directory entry goes through a filehandle."
| h2odragon wrote:
| What could be worse than IPv6? _this_
|
| Someone submit it as "IPv8" immediately.
| ranger_danger wrote:
| IPv9 is already a thing
| h2odragon wrote:
| Should this be IPv13 then?
| Veserv wrote:
| Very funny.
|
| Also, you can actually make it cost competitive if the object you
| store is the last n milliseconds of packets instead of one packet
| each. So, instead of incurring two API calls per packet, you
| incur two API calls per minimum buffering time. If S3 is zero-
| rated for any ingress/egress then you get "infinite" bandwidth
| for 4.22$ * 3 (for the active case) = 12.66$ a day if you are
| willing to accept 500 ms minimum latency, or ~600$ a day at a
| more reasonable 10 ms. If you are saturating even just a 1 Gb
| link for a whole day that is ~10,000 GB which would be ~700$ via
| the blessed channel, so you could very well come out ahead.
|
| You could do even better if you out-of-band signal the readiness
| so you do not need to poll while idle. Then you only incur a cost
| while actively transmitting so as long as you average 1 Gb/s on
| the channel you should be coming out even or ahead with minimal
| latency impact.
| derefr wrote:
| This isn't theoretical; many companies do PostgreSQL async 1:N
| physical replication, by using e.g. https://pgbackrest.org/ to
| have the primary push WAL segment files (a.k.a. "the last n
| milliseconds of packets" in the write-ahead log) as objects to
| S3. All the read-replicas then independently discover the new
| objects in S3 as they become available; fetch them; and replay
| them.
|
| > You could do even better if you out-of-band signal the
| readiness so you do not need to poll while idle.
|
| S3 and its clones have "object lifecycle notifications", where
| you can be informed by a push-based mechanism whenever a new
| object is put into the bucket.
|
| But -- what do you have to do, to get these notifications?
|
| Subscribe to a real message queue, that S3 puts these
| notifications into.
|
| So using it here would be somewhat cheating ;)
| rootw0rm wrote:
| The title of this article is what malware authors say every day
| =)
| Codesleuth wrote:
| > What if there was a way you could reduce that cost for your own
| services by up to 700%?
|
| How can something be reduced over 100% What is it that they
| actually mean here?
| Kye wrote:
| Like any good extended comedy bit, you have to read to the
| punchline to get it.
| Codesleuth wrote:
| Ah, good call. I'll continue...
| hdjjhhvvhga wrote:
| I believe they meant "7 times", so around 85%.
| [deleted]
| andrewstuart wrote:
| Could someone explain this please?
| xena wrote:
| Here's the key paragraph in the article:
|
| > In Linux, you can create a TUN/TAP device to let applications
| control how network or datagram links work. In essence, it lets
| you create a file descriptor that you can read packets from and
| write packets to. As long as you get the packets to their
| intended destination somehow and get any other packets that
| come back to the same file descriptor, the implementation isn't
| relevant. This is how OpenVPN, ZeroTier, FreeLAN, Tinc,
| Hamachi, WireGuard and Tailscale work: they read packets from
| the kernel, encrypt them, send them to the destination, decrypt
| incoming packets, and then write them back into the kernel.
| wolfgang42 wrote:
| This is one of those articles where the journey is far more
| interesting than the destination.
|
| (Edit: this comment made more sense when it was replying to a
| different complaint about the article; the parent comment seems
| to have been edited in the interim.)
| ignoramous wrote:
| Setup a TUN/TAP device, which is a file one can read (egress
| packets) / write to (ingress packets). Setup appropriate _ip
| route_ s.
|
| Read (egress) packets from the TUN/TAP device.
|
| Serialize them, assign sequence numbers, and upload them to S3.
|
| On the recieve side, poll for newer objects (packets) in S3.
|
| Write them to your TUN/TAP device.
|
| ---
|
| Here, the TUN device is a _router_ using S3 as _gateway_.
| cratermoon wrote:
| This is astoundingly well-written and informative. It's the sort
| of thing I come here for. It's also engaging and charming in a
| way rarely seen in tech writing.
|
| I like that they started out by talking about $0.07/G cost and
| went through the whole exercise before pointing out what
| immediately came to mind for me when they started pushing bytes
| in and out of S3.
| devdiary wrote:
| Been there, done that
| cosmolev wrote:
| I think we can safely expand it even further: Anything can be a
| anything if you use it wrongly enough.
| [deleted]
| XorNot wrote:
| This is the best worst thing in software that I have ever seen. I
| love it so much.
| ranger_danger wrote:
| [flagged]
| ruffman wrote:
| I think there are more things worth complaining about, rather
| than about this content specifically. Hopefully you enjoyed the
| article though.
| proto_lambda wrote:
| Which exact aspect of the cartoony animal drawings is causing
| you distress?
| _dain_ wrote:
| [flagged]
| KaiserPro wrote:
| Don't fear the furry. Embrace your inner 6ft anthropomorphic
| wolf thing, its not going to kill you. (well not unless you get
| heat stroke, furries die in hot convention centres people. )
| Kye wrote:
| Adding another sparkle to the dog every time someone
| complains.
| duskwuff wrote:
| "I will fucking increase the fucking thing."
|
| https://floccinaucinihilipilificationa.tumblr.com/post/9604
| 0...
| throw-4e451c8 wrote:
| [flagged]
| [deleted]
| NoraCodes wrote:
| Counterexample: I would downvote this if it wasn't already in
| downvote hell, and I think BSGSP is very good and funny.
|
| Why are people so committed to hating the concept of
| anthropomorphic animal mascots?
|
| Edit because apparently that's what we're doing here: I did
| not flag your post.
| GLaNIK wrote:
| [flagged]
| atmavatar wrote:
| > The bytes are stored in the cloud, which is slightly slower to
| read from than it would be to read data out of the heap.
|
| Given latency and bandwidth differences, that's like saying that
| it's slightly slower to transport water by driving standard,
| 1-gallon jugs between the US east and west coasts than it is to
| transport it a few miles using a tanker truck.
| AceJohnny2 wrote:
| that's the joke.
| jamest wrote:
| The satire in the title is reminiscent of how Firebase was born.
|
| We were previously working on a chat system called Envolve
| (https://www.envolve.com), that was 'Facebook Chat for any
| website'. A game that was using us for in-game chat created
| channels, used display: none on them, and passed game state
| through the chat.
|
| We scratched our head, asked them why, and learned they wanted to
| focus on the frontend, not to deal with realtime message passing.
|
| This led us to create a 'headless version' of our chat infra (re-
| written in Scala) that became the Firebase Realtime Database.
| bombcar wrote:
| This is an important lesson - if someone is using your tool in
| unexpected ways don't just shut them down; there's likely a
| business case that could be identified and specialized in.
| agumonkey wrote:
| I saw the opposite happen so often and every time it's very
| painful to see people have knee jerk reaction instead of
| thinking outside the box.
| dylan604 wrote:
| Nobody wants the product you want to make. They all want you
| to make the product they need, and will do things the devs of
| your product could never imagine with your product.
| b33j0r wrote:
| Indeed. Tangentially, there is another side to the coin.
|
| The antithesis of this is the contract model, when a client
| comes to you and asks you to make something specific,
| rather than describing what they need. In this case they
| are almost always wrong. They'll rarely admit it unless you
| show them some amazing alternative, though.
|
| Requirements = a set of problems to be solved. More often
| you get handed a diagram of a 12" tall Stonehenge sketched
| on a napkin with exceeding confidence that it will be 12
| feet tall.
|
| Then they yell at you for doing what they said.
| tshaddox wrote:
| I.e. "If I had asked people what they wanted, they would
| have said faster horses."
|
| (Probably not a real quote, and probably not even
| accurate.)
| quintes wrote:
| It was Ford as I recall
| james-revisoai wrote:
| Did you detect this MixPanel or something?
| cosmolev wrote:
| At some point you can also discover it is Turing complete.
| xena wrote:
| Wait, are you saying that AWS S3 is turing complete? How the
| hell if so.
| peter_d_sherman wrote:
| >" _Access to S3 is zero-rated in many cases with S3, however the
| real advantage comes when you are using this cross-region._ "
|
| Or cross-Country... as in two Countries that are geoblocked from
| one another...
|
| >"This lets you have a worker in us-east-1 communicate with
| another worker in us-west-1 without having to incur the high
| bandwidth cost per gigabyte when using Managed NAT Gateway."
|
| Simplified algorithm for bypassing geoblock via the above method:
|
| 1) Select a cloud storage provider (could be any cloud storage
| provider or shared persistent storage platform; doesn't
| necessarily need to be Amazon/S3) that works or does business in
| two countries, Country A and Country B, where normal IP traffic
| is geoblocked between Country A and Country B.
|
| 2) Use shared cloud storage objects/buckets/rows (call them
| whatever you will, "keyed persistent storage discrete thingies"
| for lack of a better term!) as the article suggests, to _emulate_
| IP traffic between user A in Country A and user B in blocked
| country B...
|
| 3) Combined with a P2P or other front end app that knows how to
| use this method of communication (along with code stubs, such
| that people could customize it to their own cloud storage
| provider or platform) if/when normal country-to-country IP
| communication is blocked for whatever reason (zombie apocalypse?
| <g>) could make a powerful future P2P communications tool, for
| lawful purposes...
|
| Anyway, great article!
| JoelMcCracken wrote:
| This is the first hn headline that I can remember actually making
| me laugh out loud. Bravo.
| whartung wrote:
| I know of a Famous Large Company that used email as their message
| queue to synchronize the data across two of their large systems.
|
| It's a perfectly apt message queue, just a bit heavyweight. But
| if it's "light enough", it comes "for free" with many OSes.
| lucianbr wrote:
| Using SMTP to transfer messages seems just plain common sense.
| Far different than IPv6-over-S3.
| anotherevan wrote:
| One of my customers used email (a gmail account, no less) as a
| message queue between their front end site and the back-office
| processor. This worked quite well for close to a decade I
| think.
|
| It basically evolved from when applications from their original
| customer facing site were emailed and manually entered into the
| back-office system by a human. They were looking to automate
| this with a minimum of changes to too many of the moving parts
| at once, so I reformatted the sent email to contain an XML
| payload so the new back-office automation could read and
| process it (and in a pinch, a human could still review any
| problem applications) using Java's mail APIs.
|
| Things evolved, the front-end web site got replaced with a
| Wordpress site, but the email message queue kept working for a
| long time. In the last year or so it was getting more and more
| onerous though. Reconciling information between the front-end
| and the mail box showed not all emails were being delivered,
| and authentication to gmail was becoming more and more of a
| burdensome moving target.
|
| I just recently replaced the whole thing with an API call made
| from the back-office to the Wordpress site to access the stored
| data. (The original site didn't store, just emailed, which was
| why this was not an option historically.)
| c54 wrote:
| Ha! I think i know what company you're referring to
| SteveNuts wrote:
| Oracle came to my mind but I'm sure it's in common use in
| many places
| simonjgreen wrote:
| Before there was REST and other http based standards ebXML was
| a hotness and tools like bizspark etc. You could see the
| logical progression through:
|
| I fill in the order form and post it. They mail back an
| invoice.
|
| I fax the order form, they fax back an invoice.
|
| My computer sends the order form in a structured way over
| email.
|
| My computer sends the order form in a structured way over http.
|
| And that's why many large telcos and banks still use ebXML in
| their B2B transactions. It's fundamentally the same business
| process and logic it's always been, with glacially slow
| improvement in performance over time.
| ahefner wrote:
| I always thought using e-mail in this would be good for systems
| that occasionally need a human in the loop. Normally one
| service processes and mails results to the next, occasionally
| something is forwarded to a human, who can make some decision,
| edit the mail, then forward it along back into the automated
| path.
| vidarh wrote:
| I co-founded a webmail provider in '99, and when we needed a
| message queue reaching for our heavily customized Qmail setup
| was a relatively natural choice. I've mentioned it here before.
| Provides all the routing and retries we needed, and made
| debugging trivial (e-mail from our desktop clients to the
| queues worked; cc:'ing a real mailbox with copies worked;
| checking out queues with POP3 worked...)
| doubled112 wrote:
| Email is a message queue in the most literal of senses, isn't
| it?
| xena wrote:
| Yes, it's actually somewhat of a decent (if cursed) message
| queue for many usecases too. Not to mention the debuggability
| (you already know how to use an email client).
| marcosdumay wrote:
| > if cursed
|
| As long as you keep it internal-only, get a reliable server
| that doesn't do any anti-spam shit (AKA, nothing from MS or
| any other large company), doesn't use some unreliable
| proprietary database (again no MS), and have a reliable
| journaled disk, your only important failure mode will be
| losing disks. That's an easy failure mode to deal with.
| Ozzie_osman wrote:
| You could use Slack, too.
| flir wrote:
| Email keeps everything on-prem.
| EvanAnderson wrote:
| Active Directory is (was?) capable of running replication
| traffic (for very particular use cases) over SMTP. I always
| thought that was ingenious.
| KaiserPro wrote:
| A famous financial news paper used an FTP server as its kafka
| "fan in" node.
|
| It was so old that the 1u case started to droop in the middle.
| (it was an sparc something or other with real SCSI ultra 320
| drives in them)
| mikece wrote:
| As a junior dev I used folder pairs (to_process and processed) as
| a way to move messages between loosely coupled systems with file
| system watchers picking up new files in the to_process folder.
| Very light weight, got the job done, and was told it was in
| production for over a decade (even after better best practices
| came into use).
| derefr wrote:
| FYI, this is an old Unixism called a "spool directory" :)
| WeAddValue wrote:
| FYI, which was probably derived from the old mainframe-ism
| which had spooling before Unix was written. See
| https://en.wikipedia.org/wiki/Spooling
| derefr wrote:
| Literally spooling message-filled tape onto a tape reel! I
| never even thought about that.
|
| On the topic of message queues, I wonder if anyone ever
| designed a _mechanical, asynchronous hardware IPC_
| mechanism based on tape. Imagine: an endless-loop tape that
| stretches between a "sender" tape-drive on one machine,
| and a "receiver" tape-drive on another machine; with each
| tape drive locally tensioning the tape as it passes
| through, but letting it hang slack outside the drive; with
| two tape-width buffer boxes in between the two drives, for
| taking up the slack; and where each tape-drive has a strain
| gauge on its input-end capstan.
|
| With this design, you'd never need to seek the tape
| (backward, at least) -- the receive head would just read
| the tape forward at whatever speed it could until it ran
| out of slack in the tape (i.e. blocked on read); and the
| send head would write the tape forward at whatever speed
| _it_ could until _it_ ran out of slack (i.e. blocked on
| write.) The tape itself would be a literal, physical ring
| buffer.
| wolfgang42 wrote:
| Western Union built a store-and-forward message switching
| system in pretty much exactly the way you describe (but
| with paper tape rather than magnetic):
| https://en.wikipedia.org/wiki/Plan_55-A
| florbo wrote:
| Sometimes a filesystem-based queue like that is all you really
| need. Recently I used S3 API compatible storage for something
| similar, as introducing AMQP or pubsub would have been
| immensely overkill for this low volume component.
| wolfgang42 wrote:
| This is a very sensible design for small-scale queueing
| systems. A few implementation notes I've learned along the way:
|
| - You'll want writers to create a temp file and then rename it
| into place; otherwise your reader might pick up a file that's
| only half-written. (POSIX rename() is atomic within a
| filesystem.)
|
| - It's also helpful to have an `in_process` folder, and a
| cronjob to send alerts if a task sits in that place too long.
| That way you can quickly catch crashes and other failures.
|
| - You can have multiple readers; they cooperate by rename()ing
| the input file into in_process/ before they start working on
| it, and ignoring ENOENT (which indicates some other process got
| to it first).
|
| - This pattern is great for loosely-coupled systems. In my case
| I was importing orders into a system that ran very slowly; once
| a reliable import queue was available new use cases kept
| presenting themselves, and it was very easy to add them; since
| the API was just "write a file in the correct format",
| individual scripts could be written in whatever way was easiest
| for the particular task at hand. (It helped that the input
| format hadn't changed in a couple decades; if it had needed
| changing a lot you'd want a more structured system. But this is
| the same as any distributed systems interface design.)
| fukawi2 wrote:
| I'm doing the same right now. The incoming messages are coming
| from a radio network at 1200 baud, so I don't foresee capacity
| problems :D
| ipython wrote:
| Given that Corey Quinn is involved, I'm surprised that route 53
| isn't included somewhere (or maybe it is, I'm still reading
| through it)
| xena wrote:
| I was gonna add Route 53 (to have each node set its own DNS
| record), but I'm saving that for part 2. Be afraid, part 2 is
| coming.
| sam1r wrote:
| >> Read time in minutes: 40
|
| I wonder if this part is satire, if so, how come? I definitely
| feel it should be much less.
| jerf wrote:
| Estimate includes the additional latency you will experience
| while downloading the post through the mechanism described in
| the post, which is the only _true_ way to read it.
| xena wrote:
| My read time estimate code is here:
| https://github.com/Xe/site/blob/aa3608afa6c62695ca0ab139f823...
|
| I've been trying to play with the constants over the years to
| make the read time estimate more "accurate", but it's a tough
| nut to crack in general. So I can go over my numbers more
| accurately, how long did it take you to read it?
| ignoramous wrote:
| fwiw, it took 5m to read. Nb: I was already familiar with a
| lot of the terms in the post (partly because I've already
| experimented relaying IP over Cloudflare's DurableObjects
| instead of S3), and skipped the dialectics.
| xena wrote:
| you what
| Kye wrote:
| I don't even know what that is and I can tell it's
| cursed.
| KaiserPro wrote:
| This, this is what the internet used to be like.
|
| ok, what the good parts of the internet used to be like.
|
| I also love that its highlights an AWS dark pattern. more, more I
| say.
| jheriko wrote:
| meanwhile in the minds of ZeroMQ devs...
| d0gsg0w00f wrote:
| I always thought it would be fun to host a Rube Goldberg
| competition for systems engineering. Whoever could accomplish a
| simple task with the most ridiculous system would win.
|
| Something like: read the 54th line of this file hosted at xx
| address.
|
| Then a submission could look like: 1. FTP a
| file to a server with the address. 2. the server reads
| the file and spins up a VM. 3. The VM polls an endpoint
| to download code to pull the address. 4. The code
| downloads the file and splits it into a single file per line.
| 5. A script loads all the files into an array and accesses
| array[53] to get the answer.
|
| ...you get the idea
| Nextgrid wrote:
| > Whoever could accomplish a simple task with the most
| ridiculous system would win.
|
| See also:
|
| * Microservices
|
| * "Web scale"
|
| * Kubernetes
|
| * Cloud computing
| sitkack wrote:
| I initially expected a horrible amount of flamewar tinder, but it
| is an Overton Window expanding Art Project and I love it!
|
| The Cardio rate control library is cool,
| https://pkg.go.dev/within.website/x/cardio this library would be
| a great problem for using a neural net to model a PID loop.
|
| https://www.semanticscholar.org/paper/A-Design-of-FPGA-Based...
|
| https://www.semanticscholar.org/paper/Self-Tuning-Neural-Net...
| mtlmtlmtlmtl wrote:
| I love this! Reminds me of Harder Drive by Tom 7:
|
| https://youtube.com/watch?v=JcJSW7Rprio
|
| It's better to go in unspoiled so I won't reveal any details.
| KineticLensman wrote:
| Spoiler: "we can of course ignore air resistance because
| chainsaws cut through the air like butter"
| Sniffnoy wrote:
| I'm going to have to spoil some of it to point out that part of
| what he's done is reinventing the idea of delay-line memory,
| although he makes no mention of this:
| https://en.wikipedia.org/wiki/Delay-line_memory
|
| ...of course if that's all there were to it it wouldn't be
| interesting, but I won't reveal what the rest is as per your
| suggestions. :)
| mtlmtlmtlmtl wrote:
| True, I suppose. Though the implementation details are
| certainly refreshing :)
| loeg wrote:
| Tl;dr: S3-based tun/tap virtual network device.
| KaiserPro wrote:
| my dear child, the fun is not the destination, the fun is the
| slow decent into madness to get there. For other people: don't
| read the summary, read the whole thing. its a work of art.
| loeg wrote:
| You don't need to condescend. Knowing the subject matter in
| advance doesn't lessen the experience.
| revskill wrote:
| Wait, to understand all of the SRE things, one might need
| another 6 months.
| xena wrote:
| Is this a challenge.
| revskill wrote:
| I'd love to learn from you more if you don't mind.
| xg15 wrote:
| Too late, already deployed to prod :)
| artem_dev wrote:
| DigitalOcean had a nice blog post on this "From 15,000 database
| connections to under 100: DigitalOcean's tale of tech debt"
| https://www.digitalocean.com/blog/from-15-000-database-conne...
| they used MySQL as a message queue and migrated from it.
| Kye wrote:
| Such dark works from such adorable creatures. The nature of
| furry.
| disruptiveink wrote:
| I'm not part of this subculture, but I honesty enjoy this type
| of idonsyncratic presentation which always was, and hopefully
| always will be part of hacker culture.
|
| The content is not only relevant and hilarious, but I'd much
| rather personal blogs like this keep being updated instead of
| seeing the the sterile Medium or LinkedIn Pulse versions,
| usually written to impress future employers.
|
| Zany visuals, doing things with your friends just because you
| can and writing up detailed accounts of it to share with
| everyone is the quintessential hacker culture. If Hackers'
| Cyberdelia was set in the 2020s, no doubt Mara would fit in.
| xena wrote:
| Author here. Thank you for liking this. One of the most
| amusing things is that doing this zany visuals/detailed
| writeups/unabashed character in my writing actually makes me
| more impressive to future employers than any sterile LinkedIn
| posts ever will. I estimate that at this point if I wanted to
| get hired I could literally post a banger like this, add a
| banner to it halfway in that says "by the way, I'm looking
| for gainful employment, if you want someone doing DevRel that
| writes like this and gives talks like [link], please get in
| contact" and I could probably get a job in a week or two.
|
| I want to keep hacker culture alive by not accepting the
| gentrification of it. Hacker culture is queer, neurodiverse,
| furry, weaboo, and more. As a philosopher of the arts, that
| is the kind of culture I want to create more of. Not
| platitudes on LinkedIn. I want to create the kind of culture
| that celebrates art for the sake of art.
|
| I may be "cringe", but I am free.
| sneak wrote:
| [flagged]
| xena wrote:
| I'm sorry to hear that you felt offended by the cartoon
| shark in the article about programming. I understand that
| you may have different preferences and expectations for
| the content you consume. However, I hope you can also
| respect my creative choices and freedom of expression. I
| am not obligated to cater to your personal tastes or
| opinions, especially when I am providing my content to
| the world for free. I appreciate your feedback, but I
| also ask you to be more considerate and respectful of
| other readers who may enjoy my work. Thank you.
|
| I would ask you to perform self-reflection to ascertain
| as to why the cartoon shark discomforts you. You may want
| to sit with that discomfort to understand why you feel
| that way so you can hopefully move past that and towards
| a more peaceful future.
|
| There are a number of logical fallacies being presented
| in your assertions, however I don't feel like I need to
| spend the time to pick them apart.
|
| Be well,
| chrsig wrote:
| > I is not obligated to cater to your personal tastes or
| opinions, especially when I am providing my content to
| the world for free.
|
| I don't believe OP asserted anything to the contrary,
| could you highlight where you inferred that?
|
| > I appreciate your feedback, but I also ask you to be
| more considerate and respectful of other readers who may
| enjoy my work.
|
| It would be helpful if you would be willing to expand on
| how you found the OPs statements disrespectful or
| inconsiderate.
| soft_dev_person wrote:
| sneak wanted OP to be less openly queer in their personal
| blog for their own comfort, due to their own inability to
| imagine that the subculture OP is part of is more than
| their own narrow definition of it.
|
| I guess the disrespect is clearer when one has grown up
| with its many forms, but assuming your request for more
| context comes out of a sincere desire to increase your
| understanding, I hope this helped.
| chrsig wrote:
| > assuming your request for more context comes out of a
| sincere desire to increase your understanding, I hope
| this helped.
|
| I appreciate the benefit of the doubt -- I was trying to
| approach it with the assumption that there's probably
| some perspective that I wasn't aware of.
|
| > sneak wanted OP to be less openly queer in their
| personal blog for their own comfort
|
| I can see how it may be interpreted that way. To me it
| read more like "hey, great content, presentation of it
| wasn't for me" in a way to convey feedback in the event
| the author wanted to better capture their attention. I
| didn't get the sense that there was an expectation that
| the author change their writing style to accommodate
| their preferences.
|
| re-reading this, I don't think I appreciated that it was
| really in defense of the readers rather than the content
|
| > I also ask you to be more considerate and respectful of
| other readers who may enjoy my work
|
| I'm surmising that the real problem statement is:
|
| > I'm sure there are non-sexual furries but the
| subculture is primarily a sexual one (same as with BDSM)
|
| which I can appreciate as a generalization, I suppose I
| wouldn't have personally considered furries as under the
| queer umbrella, but I'm certainly no voice of authority
| on the matter. I suppose there's also the possibility
| that 'furries' was a dog whistle for 'queer', and being
| less charitable to sneak, I could better understand
| finding disrespect in the post.
| duskwuff wrote:
| > I suppose I wouldn't have personally considered furries
| as under the queer umbrella
|
| The furry subculture is _deeply_ queer, both conceptually
| and in terms of its members. But "queer" doesn't mean
| "sexual".
| Kye wrote:
| So queer the fandom got booted out of the anime cons it
| spawned out of for being so. Unchained from the narrow
| views of the anime community, our power only grew.
|
| Essential viewing:
| https://www.youtube.com/watch?v=iv0QaTW3kEY
| sneak wrote:
| [flagged]
| xena wrote:
| Thank you for your feedback. I will increase the thing.
| DonHopkins wrote:
| I'm sure there are non-sexual straight people but the
| dominant culture is primarily a sexual one.
| sneak wrote:
| > _sneak wanted OP to be less openly queer in their
| personal blog for their own comfort, due to their own
| inability to imagine that the subculture OP is part of is
| more than their own narrow definition of it._
|
| This is entirely false.
|
| soft_dev_person, what I do want is for you to constrain
| your assertions about me to facts, and refrain from
| inaccurate or baseless speculation. It does neither you
| nor I any favors.
| soft_dev_person wrote:
| I understand that is what you want, but you cannot
| reserve yourself from being interpreted when posting on a
| public forum.
|
| The language you chose to use and the details you chose
| to focus on is what triggered the response. If your point
| was that the cartoons was distracting, you could have
| made that clear in a less disrespectful way (or let the
| content speak for itself), or perhaps consider if it was
| worth contributing to the discussion.
|
| Again, you double down on one definition of this
| particular subculture in another reply. It is reminiscent
| of how others have tried to define what certain
| subcultures related to sexual orientation (some now more
| mainstream) entail, while not being part of them, and I
| hope you understand how this can be problematic. I
| appreciate that this may not have been your intention.
| proto_lambda wrote:
| Leaving aside the "furry == sexual" connotations:
|
| > It overshadows the content
|
| Only if you make it. Relax. What someone is wearing or
| what innocent pictures they put alongside their
| presentation does not necessarily have to concern you at
| all. Just ignore it and focus on the actual subject
| matter.
| isoprophlex wrote:
| > I may be "cringe", but I am free.
|
| Words to live by, words to die for.
|
| Please keep posting bangers!
| KaiserPro wrote:
| I am totally going to start putting cartoon section
| summaries in my technical docs.
| xena wrote:
| Here's the secret for why my cartoon dialogues work so
| well: I use the Socratic method.
| https://news.ycombinator.com/item?id=34461459
| loeg wrote:
| FWIW, I don't think they work well.
| tschumacher wrote:
| I once used a MySQL database as a replacement for a message
| queue. This was the easiest solution to implement since all the
| servers were already connected to the database anyways. A server
| would write a new row to the table and all the servers would
| remember the last row they had already seen. Occasionally the
| table is cleared. I'm sure there are some race conditions in the
| system but its only purpose is to send Discord notifications when
| someone breaks a highscore in a video game, so its not really
| critical. It's still working that way today.
| Ozzie_osman wrote:
| This is actually pretty common, and usually a "good enough"
| solution. You can also add things like scheduling (add a run_at
| column), at least once execution (mark a row when it is being
| processed, delete it only when successful), topics, etc with
| minor modifications to your table.
|
| If you want something that works "well enough" I'd say it's a
| reasonable choice.
| larperdoodle wrote:
| Yeah, I'm using it as a transactional outbox to ensure at
| least once delivery to SNS.
|
| Can't really think of a better way to ensure that a message
| is always sent if the DB transactions succeeds and is never
| sent if the DB transaction fails
| Ozzie_osman wrote:
| You can get so far by ensuring at least once and making
| everything idempotent (will get you as close to "exactly
| once" as you can). With a database, the most common pattern
| is: insert the row for the job, when a worker starts
| working on it, mark it as in progress so it doesn't get
| started again, if the task fails, or after some reasonable
| time-out period, another worker can pick up the task again,
| ultimately the row for the task is only ever deleted when a
| worker successfully completes it.
| scarface74 wrote:
| I've run into all sorts of database locking issues and
| concurrency issues when using a database as a queue. I saw that
| mistake made a long time ago and I would never do it myself.
| wolfgang42 wrote:
| Database engines are getting features like SELECT FOR UPDATE
| SKIP LOCKED, so what were once serious blockers on this idea
| may no longer be as much of a problem.
| [deleted]
| scarface74 wrote:
| And if you skip records that you are depending on for your
| poor man's queue, aren't you just hiding bugs?
| Spivak wrote:
| No? Unless there's some edge case with that statement I
| don't know about. That statement is basically tailor made
| for queues so you can select jobs that aren't currently
| being worked on by other workers.
|
| Inasmuch as you trust your db's locking correctness it
| eliminates the concurrency issues. You can very naively
| have n workers pulling jobs from a queue not stepping on
| each-other.
| moron4hire wrote:
| I briefly worked for a major corporation 15 years ago that did
| this with SQL Server to create distributed worker processes to
| handle all the AI-generated used car listings and photo
| recolorings [0] for almost all of the used car lots in the
| country.
|
| [0] Why take hundreds of photos of Honda Civics in red, green,
| blue, and black when you already have a dozen in white?
| dylan604 wrote:
| Why even take the dozen in white when they have a model you
| can render in any manner? Most car commercials do not have
| real cars in them. Maybe the shots of a car actually in
| motion, but most of the static shots are 3D models placed
| onto backgrounds. I don't know why, but I was surprised by
| this when I worked in a post house that did a lot of car
| commercials. One of the roles for a coworker was to get flown
| around to locations to take the images for the background
| plates using photogrammetry. "Can't fly an Alexa through the
| back glass to zoom in on the dash now can we" was one
| comment.
| moron4hire wrote:
| Again, 15 years ago, this was a web app company, and Honda
| ain't giving any models to no used care sales anything.
| [deleted]
| UniverseHacker wrote:
| Why not sqlite with a lockfile? /s
| hardwaresofton wrote:
| See also: postgres with SKIP LOCKED.
|
| Great talk on it from Citus Con 2022:
|
| Queues in PostgreSQL | Citus Con: An Event for Postgres 2022 --
| https://www.youtube.com/watch?v=WIRy1Ws47ic&list=PLlrxD0Htie...
| )
|
| Citus is a horizontal scale out extension for postgres --
| imagine the power of those two things together!
|
| JK you don't have to imagine:
|
| https://www.citusdata.com/blog/2018/01/24/citus-and-pg-partm...
| andrewstuart wrote:
| I wrote an implementation in Python.
|
| https://github.com/starqueue/starqueue
|
| The code is in there for Postgres, MS SQL and MySQL (which
| all support SKIP LOCKED) though at some point I abandoned all
| but Postgres.
|
| If I was to write another message queue then I wouldn't use a
| database, I'd use the file system based around Linux file
| moves, which are atomic. What I really want is a message
| queue that is fast and requires zero config, file based
| message queues are both.... better than a database.
| gchq-7703 wrote:
| Interesting. What makes you want to switch to the file
| system? I wrote one for a project[0] a while back (for
| MongoDB) and it didn't seem like the database introduced
| too much complexity. I didn't write the implementation from
| scratch, but the couple hundred lines of code were easy to
| reason about.
|
| [0] https://github.com/gchq/Bailo/tree/main/lib/p-mongo-
| queue
| [deleted]
| andrewstuart wrote:
| The filesystem means zero configuration.
|
| I found almost all message queues to be horribly complex
| to configure, debug and run. Even database queues require
| a lot of config, relative to using the file system.
|
| I did actually write a file system based message queue in
| Rust and it instantly maxed out the disk at about 30,000
| messages a second. It did about 7 million messages a
| second when run as a purely RAM message queue but that
| didn't use file system at all.
|
| It depends what you're doing of course... running a bank
| on a file system queue wouldn't make sense.
|
| A fast message queue should be a tiny executable that you
| run and you're in business in seconds, no faffing around
| with even a minute of config.
|
| I just hate configuration in general.
| orangepurple wrote:
| > running a bank on a file system queue wouldn't make
| sense
|
| Bank transfers are done by a text file sent over FTP
| (file transfer protocol; circa 1971)
|
| https://orientalbank.com/assets/Pdfs/ach/ACH_ORIGINATION_
| AGR...
| Izkata wrote:
| > I did actually write a file system based message queue
| in Rust and it instantly maxed out the disk at about
| 30,000 messages a second. It did about 7 million messages
| a second when run as a purely RAM message queue but that
| didn't use file system at all.
|
| Did you try an in-memory filesystem through tmpfs?
| moron4hire wrote:
| I really feel like file systems aren't used for enough
| things. File systems come with so much useful metadata.
|
| I've experimented with using the file system for storing
| configuration data where each config value is a single
| file. Nested structures just use directories. The name of
| the file is the field name. The file extension is a hint
| about the data type contained within (.txt is obvious, but
| I also liked .bool). Parsing data is trivial. I don't need
| any special viewing or editing tools, just a file manager
| and text editor. You can see when a specific config value
| was changed by checking the file update time. You don't
| have to load the whole configuration just to access one
| field. And you could conceivably TAR up the whole thing if
| you wanted to transmit it somewhere.
|
| I use it to configure little sub-projects in my personal
| website. I really like it, but I shudder to think of the
| complaining I'd hear from other developers if I ever used
| it in a work project, just because it's not whatever
| they've ever seen before and would require a moment of two
| of thinking on their behalf to get over ingrained habits.
| djbusby wrote:
| I've done and do this too. Its worked awesome for 20+
| years.
| moron4hire wrote:
| Oh man, symlinks to share common configuration.
| bboygravity wrote:
| Sounds very much like how .ini files work and/or how
| regedit works on windows? :p
|
| Not a bad idea at all, but perhaps re-inventing a very
| old wheel?
| moron4hire wrote:
| It's not at all like INI files. INI files have field
| names and section delimiters in them. An INI files might
| be: [UI] DarkMode=1
| Theme=solarized [Options] Indent=4
| InventMode=tabs
|
| Whereas this file system config would have individual
| files per value. <user-
| home>/.config/<app-name>/UI/DarkMode.bool --> 1
|
| It's a single byte file. You read the entire file
| contents and if it's not zero, it's true. The existence
| of the file tells you whether or not to use a default
| value. An INI file would have to be fully parsed before
| we know whether it contains a value for that config
| value. <user-home>/.config/<app-
| name>/UI/Theme.txt --> solarized
|
| You read the entire file contents, trim leading and
| trailing whitespace and toLower for good measure if you
| want, then validate against your list of installed
| themes. Done. No goofy JSON or YAML parser in site.
|
| And what reinvention is there? If you're just using a
| system that already exists, you're not reinventing
| anything.
| binwiederhier wrote:
| A company I used to work for extensively used this
| method. It's incredibly useful to be able to read a
| config or state value from any language and even bash
| scripts quickly.
|
| However, and this is a big drawback, once you have too
| many config files and you start reading and writing from
| different processes, you get into bottleneck situations
| quickly.
| aa-jv wrote:
| This has been a thing since before databases were relational.
| 4G languages (Progress, etc.) were especially nice for their
| ability to wrap a queue table around a series of reversible
| transactions, if you coded things right .. meaning a lot of
| modules written for app infrastructure were based on an 'inbox
| table' methodology ..
| renewiltord wrote:
| Segment did so quite successfully
| https://segment.com/blog/introducing-centrifuge/
| hu3 wrote:
| > We decided to store Centrifuge data inside Amazon's RDS
| instances running on MySQL. RDS gives us managed datastores,
| and MySQL provides us with the ability to re-order our jobs.
|
| Interesting. Thanks for sharing
| pinkcan wrote:
| every half year or so I remember centrifuge, and get a little
| sad they didn't write more about it
| rollcat wrote:
| I've built a hybrid task queue/process supervisor on top of
| SQL. Classical task queues like Celery didn't exactly fit our
| use case: a single process could run for hours or days, but in
| case of a node failing, it must be resurrected elsewhere as
| soon as possible (within seconds). I didn't have the time to
| re-architect everything for Kubernetes, or rewrite half the
| product in Erlang; so I built that weird thing. It's been super
| stable, running mission critical code, and making us money -
| for several years now.
| bob1029 wrote:
| I am doing the same with SQL Server. The messages table is more
| of a bus than a queue in our case (columns like ReplyToId,
| etc). Using it for RPC communication between cloud bits. Much
| cheaper than Azure Service Bus and friends.
| [deleted]
| Digit-Al wrote:
| Just out of interest: any reason you're not using the SQL
| server service broker?
| bob1029 wrote:
| It's only available in managed instances or on-prem. We are
| trying to use the "serverless" flavor of Azure SQL
| throughout:
|
| https://learn.microsoft.com/en-us/azure/azure-
| sql/database/f...
| nickpeterson wrote:
| I don't know the OPs answer, but I'd hazard to guess
| because ssmb is a completely neglected feature with very
| little in the way of community or UI. In theory it would be
| great, but MS basically never invested in it after its
| release and now it's just a random, "who knows when we'll
| drop support for this" sql server feature.
| headPoet wrote:
| "anything stored in that pointer to memory you got back from
| malloc() is stored in an area of ram called "the heap", which is
| moderately slower to access than it is to access the stack." Is
| this true, or a myth? Ignoring the allocation cost and access
| patterns making cache misses more likely, surely memory is just
| memory
| twoodfin wrote:
| By definition, everything on the heap got where it is
| dynamically, so the minimum number of pointers to chase to find
| it is 1.
|
| In contrast, what's where on the stack can often be (and often
| must be) known statically by the compiler and accessed
| directly, even moved entirely out of memory and into a
| register. (It's possible to do this with suitably constrained
| dynamic allocations, but the optimization is much harder.)
| xena wrote:
| I'm pretty sure this is still the case. I'm not sure how cross-
| platform that assumption is (it won't work in Go where it has
| heapstacks I don't think), but classically yeah the stack is
| put is slightly faster memory with fewer access barriers.
| pavlov wrote:
| The only thing worse would be this "on the blockchain."
|
| On second thought it seems likely that some charlatan has already
| years ago raised a couple of millions in an ICO for such a
| project. "We're making the entire Internet web3-compatible!"
| xg15 wrote:
| I mean. Chain, queue, it's not that different if you squint
| hard enough...
| Izkata wrote:
| 2018: Modeling of Blockchain Based Systems Using Queuing
| Theory Simulation
|
| https://ieeexplore.ieee.org/document/8632560
| jheriko wrote:
| i liked this:
|
| > <Mara> As an example of this, consider the tar command. The
| name tar stands for "Tape ARchive". It was a format that was
| created for writing backups to actual magnetic tape drives. Most
| commonly, it's used to download source code from GitHub or as an
| interchange format for downloading software packages (or other
| things that need to put multiple files in one distributable
| unit).
|
| a very cute description for a barrier to entry enforced by moron
| level nerds insistent that we cargo cult instead of think.
| morelisp wrote:
| Great point, let's just rename it ar.
| notamy wrote:
| ar already exists and is a different format!
| https://en.wikipedia.org/wiki/Ar_(Unix)
| wolfgang42 wrote:
| It's not clear to me which bit you're complaining about--I
| assume it's either that the format is still used and/or that
| name describes an obsolete use. But the former seems perfectly
| reasonable if it still serves a useful purpose, and the latter
| largely irrelevant (these days a "tarball" is generally as an
| arbitrary word, like "zip file"); so I'm curious to hear your
| perspective on what the problems are.
___________________________________________________________________
(page generated 2023-06-04 23:00 UTC)