[HN Gopher] Gluon, a high-performance IMAP library
___________________________________________________________________
Gluon, a high-performance IMAP library
Author : henrybear327
Score : 214 points
Date : 2023-02-23 11:25 UTC (1 days ago)
(HTM) web link (proton.me)
(TXT) w3m dump (proton.me)
| cuuupid wrote:
| I spent the better part of 5 years playing with IMAP. This is a
| huge step in the right direction but there's a lot with email
| this is missing:
|
| - Nobody reads or interacts with emails anymore, they use
| threads. Threading is one of the most impossibly hard challenges
| to solve and requires a thread-first view of the inbox.
|
| - IMAP represents now a small minority of servers. ProtonMail may
| be embracing it but for actually interacting with mailservers,
| don't expect good interoperability or fast interactions when
| working with the two big dogs (Exchange and GSuite)
|
| - Similarly most mailservers don't act in good faith, their rate
| limits for non-internal clients are ridiculously low for modern
| email, and the failure rate is surprisingly high for commands
|
| - There is no real spec for IMAP (the article quotes 3501, but
| 4549 and 6531 are also "standards"), it's a suggestion at best
| and every mailserver goes its own way. For example deleting a
| message does different things in Exchange and GSuite. GSuite has
| "labels" which are represented as IMAP folders but don't always
| act like them.
|
| - It's unclear what the mailserver will accept. Valid MIME isn't
| always going to get through
|
| - Most email that's important is business email nowadays. These
| are 80% Exchange. Exchange/365 tenants for the most part block
| IMAP (found this out the hard way) and Microsoft really wants to
| push Graph (just like they tried to push EWS). This manifests in
| weird ways like big angry red warnings on the dashboard if you
| try to enable IMAP, and automatically disabling it on new
| tenants, so most IT admins will refuse to turn it on even if you
| explain it's just as secure.
|
| - Remaining 20% are mostly GSuite. Google, as usual, is trying
| its hardest to create an email client monopoly for its users, and
| so there's a $75000 annual fee for "security audits" by "vetted
| firms" regardless of how many real compliance certifications you
| have. They say this is for privacy and security but it's just a
| gatekeeping fee and this is obvious due to the large number of
| email data abusers out there (looking at Unroll). There are even
| companies (Nylas) that run their whole business by speeding up
| the certification process and providing an API that works better
| than Google's poorly documented own SDK. If you get through all
| of this your users are shown a consent screen where everything is
| unchecked by default, which sounds cool but basically means many
| of your users are going to login without knowing how to give
| consent to your email client.
|
| Overall, the space is characterized by Google trying its hardest
| to kill any startups in the space and prevent innovation that
| could threaten their consumer monopoly, and Microsoft trying to
| keep people in their ecosystem by just changing their external
| API frequently to force competitors to rewrite their codebases
| constantly.
| einpoklum wrote:
| > Nobody reads or interacts with emails anymore, they use
| threads.
|
| Not remotely true. Perhaps you're thinking about Gmail webmail
| users?
|
| > and requires a thread-first view of the inbox.
|
| Seeing how Sent messages, which aren't in the inbox, are
| typically part of threads - even that would not be enough...
|
| > IMAP represents now a small minority of servers
|
| Not sure what you're counting. If you're counting email users
| on the servers - Google and Outlook365 both support IMAP. If
| you're counting kinds of server _software_, then maybe, I have
| no idea.
|
| > their rate limits for non-internal clients are ridiculously
| low for modern email
|
| You mean, bandwidth rates, or rates of commands/second etc.?
|
| Also, if by "modern email" you mean "email where people send
| large files" - that IMHO is a problem, and people just
| shouldn't do that. Same for other social media BTW. The amount
| of smartphone space wasted by dumb video clips your friends
| shared with you by WhatsApp or whatever is stupefying :-(
|
| Anyway, people who try to use email to send and receive large
| files are kind of abusing the service. IMHO.
|
| > - Most email that's important is business email nowadays.
|
| I don't think so. Or rather - maybe in the country you live in,
| and even then I doubt it.
|
| > Valid MIME isn't always going to get through
|
| ... and it doesn't help that so few clients can properly
| present, let alone generate, MIME messages other than with the
| most simple of structure. :-(
|
| > Exchange/365 tenants for the most part block IMAP
|
| AFAICT, most people who use MS services can use their publicly-
| available IMAP servers.
|
| > the space is characterized by Google trying its hardest to
| kill etc.
|
| (sigh) not surprising unfortunately. Monopolistic corporations
| gonna monopolize-coroprately I suppose.
| basilgohar wrote:
| I was hoping to find at least some mention of JMAP [0] in this,
| but sadly, this is IMAP-only. While I an happy for this product,
| I'm disappointed at no convergence toward something more modern
| than IMAP which I've only heard derision towards by implementors.
|
| [0]
| https://en.m.wikipedia.org/wiki/JSON_Meta_Application_Protoc...
| mxuribe wrote:
| With all this recent attention on HN around imap, smtp, mail
| servers and the like....i also expected something more modern
| to be announced - either JMAP or somrthing else altogether. I'm
| not disappointed because the more, the merrier...but nowadays
| with so much of my time in the matrix world, i somewhat xpected
| to start hearing anouncements of alternatives to email.
| ThePhysicist wrote:
| My prediction is that e-mail will still be around in 10
| years, and Matrix will be mostly abandoned, because e-mail is
| mostly centered around protocols, whereas Matrix is mostly
| centered around specific libraries and clients (in my
| opinion). You can already see Matrix crumble under its own
| complexity, in a few years they will have added so much
| functionality that it will become impossible for anyone but
| Element to provide fully-featured client software. IMAP &
| POP3, by comparison, are so simple and haven't changed in
| decades, which is why they are so useful.
| mxuribe wrote:
| > ...e-mail is mostly centered around protocols, whereas
| Matrix is mostly centered around specific libraries and
| clients (in my opinion)...
|
| I disagree on this, but that's ok that we differ in
| opinion. Also, i would disagree about the crumbling of
| matrix - either in the protocol sense or app/client or even
| implementation sense. Could some areas like clients, UX,
| etc. improve? Sure, of course! Evolution of software need
| not end if there's room to improve. I'm an admitted fanboy
| of matrix, but even i know that its not perfect and i hope
| for continued improvement of matrix. As great as email has
| been for decades - warts and all - i simply have been
| expecting (hoping?) for something else by now. It need not
| be matrix - though again, i'm a fan - but i just meant i
| hope for open, decentralized mechanism, protocols that
| serve the world, and become as solid and de facto as email
| has been (again warts and all). In 10 years, i also
| beleieve email will still be around...Now, whether it will
| be considered as standard and universally-accepted as a
| means for communications, we'll see i guess.
| lucideer wrote:
| My hope is that they're both around (& actively developed &
| used) in 10 years, but yes: email is much less likely to
| die than Matrix.
| Arathorn wrote:
| I'm not sure you could kill Matrix now even if you wanted
| to? Heck, given people still run gopher & WAIS & fidonet
| servers today, so even if everyone runs away from Matrix,
| it'll still keep burbling away somewhere :)
| ThePhysicist wrote:
| Certainly not kill it, I just don't think it will be
| nearly as ubiquitous as e-mail.
| ThePhysicist wrote:
| My prediction is that with further advances in
| networking, devices will be permanently online and
| asynchronous communication will become less relevant
| (because you can always directly reach your peer), so all
| these fancy E2EE crypto protocols will be replaced with
| TLS or a similar protocol.
| lucideer wrote:
| > _with so much of my time in the matrix world, i somewhat
| xpected to start hearing anouncements of alternatives to
| email._
|
| I presume you're suggesting a _new_ alternative to email
| rather than Matrix being itself an alternative (which isn 't
| its intent).
|
| The thing about email is it's a set of related protocols - if
| you want a modern alternative you need to look at those
| protocols & address legacy problems with them, replacing the
| ones that need replacing. That's what JMAP is. What you're
| asking for is JMAP.
|
| JMAP probably isn't perfect, but nothing is & there's no
| better alternatives I know of. And one major provider using
| it = more adoption than any other modern alternatives.
| catiopatio wrote:
| > no better alternatives
|
| IMAP. IMAP is the better alternative.
|
| It's universally supported and battle-tested. We have great
| servers and clients for it.
|
| It's not perfect, but it's also not fundamentally flawed in
| any way that might put the future of e-mail at risk.
|
| Bolting HTTP and JSON onto e-mail via JMAP -- and requiring
| that everyone implement an entirely new, larger protocol
| stack -- is not an obvious value proposition.
| lucideer wrote:
| > _IMAP is the better alternative._
|
| So the definition of the word "alternative" in this
| instance is "something other than IMAP" - the argument
| for JMAP is not competing with IMAP, there's a pre-
| determined assumption that we're already unhappy with
| IMAP, hence the search for an "alternative".
|
| It sounds like you disagree fundamentally with moving on
| from IMAP in the first place, which is fair enough.
| That's not a debate I'm getting into here: all I'm saying
| is _IFF_ we are moving on from IMAP, JMAP looks like the
| best alternative.
| einpoklum wrote:
| > We have great servers and clients for it.
|
| I don't know about servers, but I'm still waiting to see
| a great email client. (For now I get by with Thunderbird,
| and it's actually going downhill.)
| tjoff wrote:
| Disagree, the dominance of IMAP makes email much worse
| than it needs to be. To the very point of putting email
| at risk in the long run.
| mxuribe wrote:
| > I presume you're suggesting a new alternative to email
| rather than Matrix being itself an alternative (which isn't
| its intent)...
|
| Yes, the intent of my statement was hoping for an
| alternative to email, and i did not mean to imply that
| matrix would be the thing that would replace email. Both
| things seek to achieve different things. and even in some
| rare overlappiny use-cases, i would say that at this time
| matrix does not universally replace email.
|
| What i should have stated was that with such attention and
| (at least in some circles) excitement around newer areas
| like matrix...That maybe traditional email software devs
| might have - by now - looked into newer alternatives to
| email. With apologies to email software devs.
|
| And, yep, I had heard of JMAP some time ago....and i
| figured by now that it would be far more popular than it
| is, and that it would have been implemented by many more
| entities by now. Oh well.
| lucideer wrote:
| > _i figured by now that it would be far more popular
| than it is, and that it would have been implemented by
| many more entities by now. Oh well._
|
| Well IMAP is 37 years old and Google still haven't gotten
| around to implementing it properly. Email as a technology
| moves pretty slowly.
| mxuribe wrote:
| Lol! You are so right on all points there!!! :-D
| Avamander wrote:
| To be fair here, IMAP has some terrible ideas that have
| only been remedied with later standards.
|
| For example the (in)famous label thing, _why_ should a
| letter belong in a single folder. Pointless restriction.
| lucideer wrote:
| What's the infamous label thing? Are you talking about
| IMAP or Gmail's broken implementation?
|
| I was under the impression the first public IMAP spec had
| support for "flags"... did it need later fixing?
| mikl wrote:
| Yeah, it's a shame. IMAP is not a great protocol (to say the
| least), but jmap is nowhere near critical mass. At present, its
| not worth the effort to implement for pretty much anyone, so
| it's only idealists like Fastmail who've done so anyway.
| the_duke wrote:
| Wasn't JMAP initiated by Fastmail and initially mostly based
| on their internal implementation.
| Mailtemi wrote:
| Initially, the proposition relied on Fastmail protocol,
| which was/is superior to IMAP. It was pleasing that
| standardization began, as MS Activesync charged a few
| dollars per user just for the protocol. Even the Apache
| James mail server supported it as a draft. Despite becoming
| a standard make it better , but lost momentum.
|
| Still hope at least both will add push notification to 3th
| party apps. And a BIG hope for https://stalw.art/
| NetOpWibby wrote:
| +1 for Stalwart
| lucideer wrote:
| > _its not worth the effort to implement for pretty much
| anyone, so it's only idealists like Fastmail who've done so
| anyway._
|
| Worth mentioning that Fastmail use an open source backend:
| Cyrus[0] - which started adding JMAP support in 2015 (though
| I suspect that was Fastmail devs contributing the feature).
|
| [0] http://www.cyrusimap.org/
| elric wrote:
| At this point, JMAP is like Duke Nukem Forever or Winds of
| Winter. Why does email need JSON over HTTP? IMAP might not be
| the sexiest protocol, but it serves hundreds of millions of
| users just fine.
|
| JMAP, on the other hand, is hardly used, and it doesn't seem to
| solve any issues that users face, which means demand is never
| likely to reach critical mass.
| tracker1 wrote:
| Well, if _using_ the system... it makes serialization far
| less complex in terms of normalization. You can serialize
| /deserialize through what have become relatively standard
| libraries for pretty much every language under the sun. This
| can be utilized in/out of class/record based representations
| of the data in your language of choice, and acted upon
| directly. The use of HTTP is as sensible as anything else, as
| the HTTP actions and paths align relatively well to resource
| endpoints as well as a text based protocol.
|
| The abstractions and libraries available make this easier...
| if you look at IMAP it's byzantine by comparison.
| epilys wrote:
| Because it was designed for webmail frontends, which can
| communicate with servers from the client side.
| kitsunesoba wrote:
| The biggest issue with IMAP is that writing a quality client
| for it that works on all possible permutations of the spec
| (especially with the botched Gmail IMAP implementation in the
| mix) is a huge, huge pain in the rear which is why there's a
| dearth of high quality FOSS mail clients... you've got the
| old standbys like Thunderbird and Claws and that's about it.
| Newer clients like Geary either die on the vine or get stuck
| in a somewhat spartan state because making it more full-
| featured takes knowhow, time, and manpower that's not
| generally available.
|
| On the other hand lots of devs know how to work with JSON-
| over-HTTP APIs and a lot of modern languages even come with
| JSON serialization/deserialization out of the box, with makes
| writing quality clients a lot more accessible to a lot more
| people.
| Thaxll wrote:
| Who is actually using JMAP? I never heard of that, what's the
| point of implementing something that is not used at all.
| aloisdg wrote:
| This is an chicken-egg problem.
| StalwartLabs wrote:
| Mostly Fastmail but there are also a few open-source projects
| that support it https://jmap.io/software.html
| erk__ wrote:
| It is used by Fastmail (who was also developers of the
| standard)
|
| I think it is not a great idea to say that it is not a good
| idea to not implement things that no one uses, because that
| is a certain way to not make more people use it.
|
| There are some servers that support it such as Stalwart [0],
| though I don't know about any older servers and/or clients
| that support it well.
|
| [0]: https://github.com/stalwartlabs/jmap-server
| tankerkiller125 wrote:
| Until Dovecot and systems like it support JMAP, I don't see
| it gaining the critical mass it needs to succeed.
| lucideer wrote:
| Well I can't see Dovecot devs deciding to implement it if
| they keep reading comments like this dissuading the
| effort...
| catiopatio wrote:
| What advantage would they gain by implementing it?
| lucideer wrote:
| There's three considerations
|
| (a) is IMAP good / sufficient / fit for purpose
|
| (b) if not, is there a better alternative available
|
| (c) if there is, is it worth the development effort to
| support it.
|
| If we assume the answers to (a) & (b) are NO & JMAP, then
| the benefits are included in those answers & only
| question remaining is (c). The person I was replying to
| seemed to be only arguing that the answer to (c) is NO.
| tracker1 wrote:
| (a) No (b) JMAP (c) Long term, probably.
| lucideer wrote:
| > _what 's the point of implementing something that is not
| used at all_
|
| If this question made any sense we would never make anything
| new at all. How do you think new things get started?
| catiopatio wrote:
| There's no clear value proposition for a JSON-over-HTTP
| replacement for IMAP-over-TCP, especially for clients and
| servers that have already implemented IMAP.
|
| Why would dovecot want to add an entire HTTP server just to
| reproduce what they already have, with more layers of poorly
| fit abstraction, despite there being no significant mail
| clients that require it?
| Mailtemi wrote:
| Each JMAP server should include IMAP QRESYNC like
| functionality at least. Plus the ability to batch JMAP calls.
| For instance, if you have five folders with IMAP, it might
| take at least five hops to access them. With JMAP, it could
| be accomplished in just one call, returning the location of
| each new email in each folder.
| alerighi wrote:
| This really benefits when you first connect a client to a
| server that has to sync all the mail locally, and partially
| each time the client starts that has to check all the
| folders for new messages. But realistically when a client
| is opened (that is when you turn on the computer till you
| turn it off, if you turn it off and don't put it to sleep).
| Also most people has only a few folders, say less than 10.
| calvinmorrison wrote:
| JSON happens to be over JSON and HTTP, IMAP is over TCP and
| the IMAP protocol. both of those pieces are a relatively
| small amount of code to parse messages from rather than what
| to do with said messages. You can transport jmap over
| whatever you want theoretically.
| didip wrote:
| Written in Go? Awesome!
| elashri wrote:
| Proton, Gloun, "<Proton Mail Bridge> is 1000% faster, far more
| reliable". A typical usage of language for people in High energy
| physics. Not surprise that the ProtonMail was founded at CERN
| [1].
|
| [1] https://en.wikipedia.org/wiki/Proton_AG
| rdevsrex wrote:
| This is pretty cool. Just the other day there was a thread about
| https://poste.io.
|
| I definitely want to play around with it.
| sagangpuluh wrote:
| Pragmatick
| beepbooptheory wrote:
| Really hoping the name is inspired by the Graham Priest book...
|
| https://global.oup.com/academic/product/one-9780199688258?cc...
| ushakov wrote:
| Their website and branding looks so different, I had to double
| check that it's the Proton I was thinking about
| jawadch93 wrote:
| [dead]
| photochemsyn wrote:
| See this recent HN discussion on an email overview with a good
| IMAP explainer:
|
| https://explained-from-first-principles.com/email/#internet-...
|
| https://news.ycombinator.com/item?id=34846606
|
| Proton Mail Bridge (for use with something like a desktop
| Thunderbird on Linux setup) does require the paid tier of Proton,
| which isn't immediately clear. Unlike Google they're not making
| money of harvesting and selling personal data, so I don't really
| mind.
|
| addendum: The IMAP state diagram is a finite state machine,
| interestingly enough:
|
| https://www.researchgate.net/figure/Finite-state-machine-of-...
| hummus_bae wrote:
| [dead]
| superkuh wrote:
| Too bad that most megacorp email providers no longer allow you to
| use just IMAP. Instead there's a bastardized OAuth2.0 (not good
| like Oauth1 was) toolkit they use for requiring HTTP(s) mediated
| authorization. Just IMAP will not work. And every megacorp's
| implementation of this is different so they have to be handled by
| slightly different plugins/etc.
|
| The megacorps are trying real hard to kill off open protocols and
| switch everyone over to their in house "versions" of a public
| protocol for making proprietary protocols.
| merb wrote:
| actually that has nothing to do with imap. Imap supports even
| OAuth2 Authentication trough sasl and the oauth2 authentication
| for imap will be more and more of an standard, it just sucks
| that most providers don't support a few things that makes it
| easier, like dynamic clients, etc.
| Tijdreiziger wrote:
| I don't even want to use 'plain' IMAP authentication, because
| it doesn't support 2FA.
| Avamander wrote:
| This amongst other similar reasons is why large providers
| also don't want to allow plain IMAP authentication. Too
| easy to abuse and too disruptive to curtail.
| sir wrote:
| If you use this proxy of mine then any IMAP (or POP/SMTP)
| client can be used with a "modern" email provider, regardless
| of whether it supports OAuth 2.0 natively:
| https://github.com/simonrob/email-oauth2-proxy. No need for
| your client to know about OAuth at all.
| tpoacher wrote:
| This looks great! Does it work in a way that would make it
| need to be "authorized" by the organization?
|
| Recently I wanted to use the alpine terminal email client,
| which does support OAuth2, but I was thwarted by a notice
| saying "application alpine requesting access; write to your
| admin to authorise". Several months later, apparently whether
| to add alpine or not to the list of approved software is
| still "being discussed". Not exactly holding my breath for it
| ...
| sir wrote:
| You do still need details for an authorised client, but
| it's possible to use those of one that's already approved.
| See the readme section that explains this aspect:
| https://github.com/simonrob/email-
| oauth2-proxy#oauth-20-clie...
| meghan_rain wrote:
| > IMAP clients typically refer to messages by their "sequence
| number", the message's position in a mailbox. The first message
| has the sequence number "1", the second "2", and so on. If a
| client wants to mark a message as "read", it will send a command
| to the server such as "mark message 5 as read". But what if
| another client deleted the fourth message in the mailbox? The
| sequence numbers of all messages after the deleted message will
| be shifted down by one; the client that sent the "mark message 5
| as read" command now refers to a different message than it
| intended.
|
| > IMAP servers (which include applications such as the Proton
| Mail Bridge) need to be able to handle this situation. When one
| client moves messages into or out of a mailbox, the server needs
| to notify all other clients of the changes so that they can
| update their own view of the mailbox. And until the clients have
| received the update, the server needs to remember what each
| client thinks the mailbox looks like to correctly interpret the
| client's commands.
|
| Holy F what a clusterfuck
| throw0101c wrote:
| >> _IMAP clients typically refer to messages by their "sequence
| number"_ [...]
|
| > _Holy F what a clusterfuck_
|
| Or you could use the unique identifier of the message (UID)
| instead of the sequence number:
|
| > _Messages in IMAP4rev1 are accessed by one of two numbers;
| the unique identifier or the message sequence number._
|
| * https://www.rfc-editor.org/rfc/rfc3501#section-2.3.1
|
| > _A 32-bit value assigned to each message, which when used
| with the unique identifier validity value (see below) forms a
| 64-bit value that MUST NOT refer to any other message in the
| mailbox or any subsequent mailbox with the same name forever._
|
| * https://www.rfc-editor.org/rfc/rfc3501#section-2.3.1.1
|
| The designers of the IMAP protocol aren't as dumb as the above
| quotation would have you to believe and multi-client operations
| were thought of.
| meghan_rain wrote:
| The UID capability is a newer, hail-mary attempt to fix the
| issue, and thus it's optional, not all IMAP servers support
| it
| tedivm wrote:
| I wrote a popular open source IMAP library in 2009 and even
| then the UID capability was there and widely adopted. My
| library relied on it 100%, completely ignoring sequence
| numbers, and I haven't received a single issue related to
| this.
|
| It's not that new, it's not a hail mary, and it is very
| well supported.
|
| * https://packagist.org/packages/tedivm/fetch
|
| * https://github.com/tedious/Fetch
| vidarh wrote:
| UID was introduced in IMAPv4 in RFC 1730 in _1994_ , and
| it's not optional for IMAPv4 or IMAPv4.1
|
| I last wrote IMAP server-side software in 2000. Even then,
| I was not aware of any IMAP servers that didn't support
| UID. It was essential, as most clients expected it to be
| there.
|
| There _might_ still have been some at that point, but I 'd
| be shocked if anyone is still running any IMAP2 or IMAP3
| servers.
| [deleted]
| citrin_ru wrote:
| Newer as in only ~30 years old? It is likely some obscure
| IMAP servers doesn't support them but AFAIK all popular
| ones do.
| globular-toast wrote:
| It's easy to take things like REST for granted. Until you learn
| about the stuff that came before it, like this!
| rglover wrote:
| Stuff like this makes the idea of an "Email 2" protocol being
| worthwhile. Shocking how over-complicated it is to send
| text/html between two parties.
| Avamander wrote:
| People have called things email killers several times. An
| intelligent approach would be to gradually rebuild parts on
| top of better foundations and push towards that.
|
| The amount of deadlocks in adopting things like JMAP, IMAP
| NOTIFY, OAuth2 (+ Dynamic Client Registration) and so on and
| on is immense. Won't because others don't, too few do, we see
| no benefit yet etc. it's really hindering.
| evmar wrote:
| A long time ago I wrote a toy IMAP client, and ran into
| similarly confusing issues about how the protocol is supposed
| to work:
|
| https://github.com/evmar/go-imap/blob/master/imap/doc.go
| calvinmorrison wrote:
| > Holy F what a clusterfuck
|
| And that's why when people say "JMAP is pointless", just go
| back and read some of the email specs!
| Avamander wrote:
| Until things like Thunderbird support it, its adoption is
| unfortunately going to remain at the level it is currently
| at.
| tracker1 wrote:
| Which sucks as it's a massive chicken/egg problem... more
| clients and hosts need to support it.
|
| Of course, from what I understand is there are JMAP/IMAP
| gateway options, that should ease availability... assuming
| these things actually gain any traction ever.
| ketralnis wrote:
| Pretty much no client or server uses it that way though. The
| UID extension is more or less required these days.
|
| But IMAP is pretty twisted in other ways too. IMAP is _not_ a
| request-response protocol like you might think from other
| protocols. It looks like it at first because you can say things
| like (from memory, not the real protocol): >
| cmd1 LIST (here cmd1 is a request/response identifier chosen by
| the client; the server sends responses prefixed by this
| identifier) < cmd1 LIST RESPONSE...
|
| But you can also multiplex many simultaneous commands like
| > cmd1 LIST folder1 > cmd2 LIST folder2 > cmd3
| LIST folder3 < cmd3 LIST folder3 RESPONSE...
| < cmd1 LIST folder1 RESPONSE... < cmd2 LIST folder2
| RESPONSE...
|
| (note that they can come out of order)
|
| But it gets weirder because you can receive _unsolicited_
| responses. Sometimes you want that like >
| cmd1 WATCH folder1 (starts a watch which may have responses but
| not immediately) > (do other stuff) < *
| NEW MESSAGE folder1 DATA... (some time later) < * FLAGS
| CHANGE folder1 DATA... (much more time later)
|
| (the * means that you didn't ask for this so it's not prefixed
| by an ID that you gave) Here it's sort of solicited or at least
| something you did on purpose, but there are other more
| complicated cases where you'll receive data that you never
| asked for. Again from loose memory but an example might be
| > cmd1 OPEN folder1 > (do stuff with folder1)
| (much later) < * CLOSING folder1 WAS DELETED
|
| That means that your client needs to always be listening for
| data on the socket and reading it out even if the client is
| sitting in the background, and that if you send a command you
| might receive responses to other previous or unsolicited
| commands before the response to the command you just sent.
|
| It's a cool way to do what it wants to do for GUI clients in
| particular because IMAP and GUIs have the same event-oriented
| model. But it makes implementing IMAP as a library a little
| more difficult because you need bidirectional hooks into the
| containing application. And you can't easily just have a `if
| POP3 then... else if IMAP4 then...` abstraction layer without
| some work to make it happen. Heck even just having a function
| that returns the list of messages requires a background thread
| and a little bit of lying about what's happening underneath.
| Leaving data on the socket unconsumed can get you disconnected
| by keepalive checks so the background task is required if you
| want to avoid that (which by observing mail.app it seems to
| just accept the disconnection and reconnect when it happens).
| Python has an imap library in the stdlib which "avoids" this by
| being so low-level that it sidesteps some of the issues by
| making you implement request/response on top of it, and its
| model also makes perfect IDLE implementation difficult (maybe
| impossible? but I didn't put enough time into it to prove
| this).
|
| Because of the complication many GUI clients don't actually
| support the liveness built in to the protocol. They just ignore
| any reponses they didn't ask for, don't use WATCH/IDLE, and do
| the 15 minute polling cycle as if they were POP3. The protocol
| has all of the tools to have the liveness of Google Docs but
| I'm not aware of any GUIs that use it right.
| alerighi wrote:
| What is difficult about that? Every protocol works this way,
| nothing new.
|
| The fact that the server may send you messages even they are
| not responses... of course, otherwise how do you think
| notifications can work? The only other options is the client
| to poll the server periodically (how it was done by older
| clients, and iPhone client...), that is inefficient, consumes
| resources and you don't get immediate notifications. With an
| open socket the software stays in the background and gets
| woken up by the OS when a byte arrives, how is it complex?
|
| > The protocol has all of the tools to have the liveness of
| Google Docs but I'm not aware of any GUIs that use it right
|
| Never had any problem with Thunderbird. If you have
| Thunderbird open in the background and the GMail web
| interface both open at the same time most of the time you see
| first the notification from Thunderbird than the email in the
| GMail web interface. It's really fast, to the point that if I
| send myself a mail from my phone it seems to arrive
| immediately, and it's quite impressive.
| ketralnis wrote:
| > What is difficult about that?
|
| It's not independently difficult per se, it's just
| mismatched to the way that other protocols often work in a
| way that, in practise, few clients implement well. Most
| clients and libraries implement it _as if it were_ request-
| response and as a result either produce wrong answers or
| are notably less efficient as you note.
|
| > Every protocol works this way, nothing new.
|
| No? POP3 and HTTP as easy examples are simple blocking
| request-response. Some do either pipelining or multiplexing
| with the command-tagging, but don't have unsolicited
| messages. I can think of only a few other comparable
| protocols with this paradigm but "every" is just not true.
|
| > of course, otherwise how do you think notifications can
| work?
|
| Yes? This is indeed _why_ it works this way. I don't know
| what you're correcting me about. It does this to implement
| notifications and liveness. We're agreeing.
|
| > Never had any problem with Thunderbird
|
| Thunderbird is one of the better implementers, yes. There
| are messages that it ignores like "folder deleted" but
| according to Dovecot https://www.dovecot.org/clients/ it
| implements IDLE correctly which is uncommon among clients.
| See https://wiki2.dovecot.org/Clients for more client
| quirks. The dovecot config file
| https://dovecot.org/doc/dovecot-example.conf has a
| imap_client_workarounds setting for many of these common
| bugs including "delay-newmail" which disables several of
| the unsolicited message types until the server is confident
| that the client is actually expecting to receive something.
|
| In practise, in real life, clients get this wrong. That's
| all I'm trying to say.
|
| Hacker News always has the well-actuallies but I'm pretty
| familiar with this and have implemented both an IMAP server
| and client. I'm not sure what you're trying to "correct" me
| about.
| alerighi wrote:
| > HTTP as easy examples are simple blocking request-
| response
|
| Well HTTP/1.1, HTTP/2 (and HTTP/3) surely not. You can
| make more request on a single channel and also the server
| can send you data you didn't requested (PUSH). Otherwise
| a modern site with a ton of resources would either take a
| minute to load or have to open 100 connections to the
| server.
|
| MQTT is another example of protocol that works this way.
|
| > "every" is just not true
|
| Every protocol that was born after the 90s, let's
| rephrase it like that.
| vidarh wrote:
| The issue was not making multiple requests on a single
| connection, but that IMAP can interleave and reorder
| responses. It's still not _that_ hard, but it 's a
| change.
|
| HTTP/1.1 does _not_ allow data you didn 't request in
| random places. You need to specifically request a feed of
| server sent events. In every other instance you know that
| the response that come back will be replies to a request
| (in the case of SSE's, the reply will just keep coming as
| long as the server sends events), and in the same order
| as the requests. That means that a "dumb" client can just
| synchronously send requests and read the reply in a
| string request/response manner without worrying about
| getting something else back. Neither of those constraints
| hold for IMAP.
| vidarh wrote:
| No, every protocol does not work that way. A whole lot of
| protocols has no concept of multiplexing a connection or
| server initiated events. Since we're talking about e-mail,
| neither SMTP or POP3 has either of these complexities (SMTP
| allows client-initiated _pipelining_ if the client sends
| the right greeting), but doesn 't allow the _server_ to
| multiplex responses or initiate anything, POP3 allows none
| of it at all). It 's not new now, but it _is_ moderately
| more complex if you 're implementing a client.
|
| I don't think it's quite as complex as suggested above,
| though. You just want to provide access to the socket in
| case the client wants to select() etc., and offer an API to
| send a request, a queue to read responses from that allows
| you to filter by request, and maybe a wrapper that sends a
| request and waits for a reply matching that request (while
| putting everything else onto the queue). Then the client
| can decide if it wants to pretend the protocol is mostly
| request/response or take proper advantage.
| amelius wrote:
| I'm wondering why they don't give every mail a UUID. Probably
| because someone wanted to save a few bytes.
| c0l0 wrote:
| They do, on a per-session basis - but it's an optional
| feature (that's supported by ~100% of all imap server
| implementations).
| KRAKRISMOTT wrote:
| What's the name of this extension?
| epilys wrote:
| It's actually in the original IMAP4rev1 RFC:
| https://www.rfc-editor.org/rfc/rfc3501#section-2.3.1.1
|
| Using UIDs is opt-in, by prefixing `UID ` in front of
| some commands. There are indeed extensions that add more
| `UID ` commands to the protocol. I've implemented IMAP
| clients and never used sequence numbers at all.
| vidarh wrote:
| UID was present already in RFC 1730 (IMAPv4):
|
| https://www.rfc-editor.org/rfc/rfc1730#section-6.4.9
| Mailtemi wrote:
| Yep, just a few bites :) Especially if you're on vacation.
| IMAP IDs can change, requiring the email client to re-
| download all emails.
| jcranmer wrote:
| One of the extensions to IMAP is the UID extension, which
| gives every email something akin to a UUID. It's technically
| not mandatory, but virtually every email client is going to
| try to use it if possible, and likely some email clients
| don't even attempt to use message sequence numbers.
|
| So you don't need to maintain message sequence numbers,
| except for the few places in the protocol where only the
| message sequence number is reported (e.g., the unsolicited
| message deleted alerts). In practice, I would actually expect
| that trying to use message sequence numbers would be buggier
| than not using them. But servers still have to support them!
| ihateolives wrote:
| So basically no one does Google search anymore before naming
| their product? https://gluonhq.com/
| codetrotter wrote:
| All the good names are taken.
|
| As long as the product is different it's fine. IMAP library vs
| some Java thing.
| zitterbewegung wrote:
| Moving to emojis is the next step. See huggingface.co
| magicalhippo wrote:
| Oh, it's the emoji? I've just seen the domain name out and
| about so assumed it was Alien reference.
| layer8 wrote:
| Same here, though now that I think about it, a facehugger
| is kinda the opposite.
| codetrotter wrote:
| Maybe. But if a hugging face hugs you, it probably hugs
| your face. This would make the hugging face a face
| hugger. (But it would not necessarily make it a
| facehugger.)
| codetrotter wrote:
| Same, never made the connection to the emoji myself
| either.
|
| Then again, I don't use the "hugging face" emoji almost
| ever. But even if I did use it, I would probably think of
| it visually or just as the word "hug". That's how I think
| of and use other emojis -- either visually or as a simple
| word.
|
| Either way, I will intentionally continue to think of
| huggingface.co as being a reference to Alien facehuggers.
| zitterbewegung wrote:
| The joke would be facehugger but maybe it could be added
| to Unicode.
| maxnoe wrote:
| Gluons are what holds protons together
| HeckFeck wrote:
| Someone might confuse it with the Gluon Gun as well:
| https://half-life.fandom.com/wiki/Gluon_Gun
| tylersmith wrote:
| It's not very easy to confuse a software library with a
| fictional gun. You have to try pretty hard.
| HeckFeck wrote:
| I was being a little sarcastic with my comment's parent,
| but maybe this isn't the forum for it.
| sporedro wrote:
| Somewhat related but every time I see the AWS lambda I
| immediately think half life.
| cozzyd wrote:
| More like https://pdg.lbl.gov
| robin_reala wrote:
| We live in a world where Cisco had both an iPhone[1] and iOS[2]
| before Apple did.
|
| [1] https://en.wikipedia.org/wiki/Linksys_iPhone
|
| [2] https://en.wikipedia.org/wiki/Cisco_IOS
| chomp wrote:
| If people wanted unique project names, they wouldn't choose
| dictionary nouns/verbs. By your rules, gluonhq shoulda chose a
| different name because The Real Gluon is an OpenWRT project.
| Come on, just let people name stuff.
| cozzyd wrote:
| gluon is a terrible name for an e-mail library due to asymptotic
| freedom.
|
| As soon as your e-mail leaves your server, it will spawn pions
| and be screened.
___________________________________________________________________
(page generated 2023-02-24 23:00 UTC)