[HN Gopher] Hijacking Email with Cloudflare Email Routing
___________________________________________________________________
Hijacking Email with Cloudflare Email Routing
Author : albertpedersen
Score : 201 points
Date : 2022-08-03 14:02 UTC (8 hours ago)
(HTM) web link (albertpedersen.com)
(TXT) w3m dump (albertpedersen.com)
| albertpedersen wrote:
| Here's the write-up of a simple but severe exploit I found in
| Cloudflare's email forwarding service.
| LinuxBender wrote:
| That is a good write-up and a good find. Thankyou for
| publishing it and for the responsible disclosure timeline.
| DanAtC wrote:
| Over 7 months after it was fixed? Why not disclose
| immediately after?
| LinuxBender wrote:
| Disclosing a vulnerability immediately after it is
| discovered has a few problems. One is a risk to the
| customers, as script kiddies will create git repos full of
| tools to automation exploitation of the vulnerability.
| Another risk is that people will jump to conclusions
| without a proper root cause analysis being performed that
| determines how this happened, what is required to prevent
| it from happening and if there may be more aspects to this
| vulnerability than was were originally thought to exist.
| Another reason to not disclose immediately would be that in
| most cases it will violate the agreement the penetration
| tester or security researcher has with the bug bounty
| program. Disclosing immediately would mean they do not get
| paid for their discovery. This payment for bugs concept
| provides an incentive for people to help a company fix
| their bugs that their own developers and QA teams may have
| overlooked.
| lizardactivist wrote:
| Why would anyone even allow the biggest man-in-the-middle on the
| Internet to route their e-mail in the first place? It's a
| situation that will never be private and secure.
| JimWestergren wrote:
| A smaller man-in-the-middle would be more secure? I mean if
| email forward is really necessary (in my case yes) and
| successful delivery of email is really important (so I don't
| want to use my own VPS and the headache it brings) are there
| any better options for me than CloudFlare? My domains already
| use CloudFlare for their CDN etc.
| fjni wrote:
| Elephant in the room.
| ejcx wrote:
| I lead Product Security at Cloudflare, thanks for the writeup
| Albert and the fantastic security research throughout the past
| year.
|
| Once this issue was fixed we investigated all prior email routing
| configurations to ensure that this had only been found as part of
| Albert's responsible disclosure to us.
|
| Since some comments are addressing that this happened 7 months
| ago. Our disclosure policy is to allow researchers to write about
| us once the issue is fixed, but give us a week heads up before
| they publish so we aren't surprised, can coordinate any public
| comms we want to make, FAQs that need to be written for inbound
| questions from customers, and can tailor our response to the
| issue at hand. Can answer other questions if you have any.
|
| We disclosed the issue here earlier this week once Albert told us
| he was writing a blog: https://hackerone.com/reports/1419341
| dreadlordbone wrote:
| Any info about your devs doing only client side verification of
| the beta flag? Seems like a pretty bad practice.
| ejcx wrote:
| What was missing was the server side ownership check. We
| decide which customer owns the real "example.com" which is
| very battle tested logic, but had missed the check in this
| new service. The client side validation is expected too,
| though
| Operyl wrote:
| I'm curious, if you'll disclose it, what the exclusionary
| policy was. Enterprise zones?
| dreadlordbone wrote:
| Tracking, thank you.
| OJFord wrote:
| It's obviously not good in general, it's 'obscurity' (as in
| 'is not security') really, but it seems pretty harmless for a
| gradual roll-out feature toggle? If someone cares enough and
| knows enough to find they can get past it, let them play with
| the beta feature?
|
| After all, it got them this responsible disclosure.
|
| It wasn't part of the vulnerability, it just allowed OP (who
| happened not to be legitimately in the beta) to find it.
| systemvoltage wrote:
| Well, the whole point of Beta program is to limit the
| participation and thus the exposure of attack vectors, and
| reduce the impact during the testing phase.
| nmjohn wrote:
| I've built numerous systems that could be called "beta
| programs" - and the goal of none of them was security
| related - but rather product feedback related.
| OJFord wrote:
| I suppose I wouldn't personally think of it as limiting
| exposure like that - prod is prod, beta feature or not -
| but if one does for a given product then yes sure it's a
| bad bug.
|
| Do we really think Cloudflare Email Routing private beta
| was private to somehow trusted parties only though?
| Presumably 'N-mutual trusted parties' too, for regulatory
| compliance. I assume not; not least because the product
| security lead is here in the comments saying they vetted
| logs etc. after the fact to ensure that only OP took
| advantage of this.
| shyn3 wrote:
| I wouldn't rely on any companies logs or audit trail.
| OJFord wrote:
| Fine, not the point, they did it - implies the invitees
| weren't trusted 'they won't/doesn't matter if they do pwn
| each other' customers.
| Operyl wrote:
| Usually rollouts like that are slow to prevent a flood of
| people using it at once, and to slowly open the flood gates.
| Not many people will manually change the flag, either. I
| don't think the fact that the feature flag was just client
| side is a "smoking gun".
| brightball wrote:
| Agreed. Feature flags can be complicated to implement both
| client side and server side.
|
| Ideally you want both but it doesn't always work out that
| way.
| edm0nd wrote:
| Thank you for awarding this a bounty of $6,000.
| nop_slide wrote:
| Only a $6000 reward for being able to intercept someone's email?!
| jtokoph wrote:
| Seems to be double their max payout of $3,000:
| https://hackerone.com/cloudflare?type=team#user-content-rewa...
|
| It was also comprised of two separate $3,000 rewards. Maybe
| they treated it as two vulnerabilities?
| albertpedersen wrote:
| In this case the second $3000 bounty was due to a 2x
| promotion at the time. The guideline for a critical is $3000,
| but Cloudflare does occasionally award bonuses for severe
| vulnerabilities (e.g. https://hackerone.com/reports/1478633).
| riedel wrote:
| 6k seems to be a really low payout given the potential impact
| (particularly with respect to personal data), the work needed
| to discover such vulnerabilities, the revenue of cloudflare
| and the potential money that could be made by a blackhead. Or
| am I naive?
| happyopossum wrote:
| Likely impacted by the fact that this was in a private beta and
| not a publicly available product. Betas are "supposed to" have
| bugs.
| r1ch wrote:
| A well-run bug bounty should encourage early reporting of
| issues. Should he have waited until public release so the
| impact was bigger?
| rsstack wrote:
| It's a moot point because this hasn't happened. Cloudflare
| didn't lower the payout because it was in beta.
| r1ch wrote:
| Yeah I didn't realize CF's payouts were so low.
| Bytewave81 wrote:
| A "private" beta enforced solely by a clientside check,
| notably.
| rsstack wrote:
| Sure, but the client-side check completely limits the
| exposure Cloudflare have because big enterprise corporates
| don't get themselves into a private beta by playing with
| client-side checks. Only a few companies were using this
| feature at the time.
| jmull3n wrote:
| Good find. I can't believe it was that easy, tests should have
| caught this.
| dustinmoris wrote:
| Wow did Cloudflare outsource all their development work to some
| cheap agency or how come they develop features like this with
| zero security and pretty damn naive, simplistic and childish bugs
| you wouldn't expect from a company that routes half of the
| internet's traffic through their servers.
| megraf wrote:
| Hey Albert, awesome first post. You made a great deal to include
| the _right_ amount of detail, and your writing style is
| sufficient enough to keep me engaged through the entire post.
| Keep going!
| wizofaus wrote:
| So now email is unsafe for MFA/password reset messages too?
| (actually I've long argued it's not really any safer than SMS, at
| least in countries where there are decent regulations around
| porting numbers).
| theunixbeard wrote:
| Awesome work, Albert! Looks like you are crushing it on
| HackerOne, over $37K in bounties?
|
| https://hackerone.com/albertspedersen?type=user
|
| You've obviously got a strong career in Security in the future.
| Have you looked at any Crypto projects? Seems like there are some
| massive bounties on https://immunefi.com and similar sites.
| sammy2244 wrote:
| upupandup wrote:
| wow this is insane. who knows how many emails were skimmed on
| cloudflare? definitely will not be trusting this service because
| how many other vulnerabilities are not yet discovered?
| ctippett wrote:
| I place greater trust in organisations that are proactive about
| their security posture, versus an organisation where this type
| of vulnerability would never have been publicly disclosed.
| upupandup wrote:
| Proactive but not preventive. You know only after an incident
| occurs. While I appreciate it, the risk isn't mitigated at
| all unless you don't use the said service. ex. Heroku
| ejcx wrote:
| I lead Product Security at Cloudflare (and I'm one of Albert's
| biggest fans, he's contributed a lot to our bug bounty, thank
| you Albert).
|
| Once he reported the issue we investigated all prior email
| routing configurations to ensure that this had only been found
| as part of Albert's responsible disclosure to us.
|
| We disclosed the issue here earlier this week:
| https://hackerone.com/reports/1419341
| boredpudding wrote:
| Is the 'Burp' mentioned in this, the 'Burp Suite' that can be
| downloaded here? https://portswigger.net/burp/communitydownload
|
| Am new to this kinda stuff but would love to play with it. Always
| love reading up on responsible disclosures.
| albertpedersen wrote:
| Yup, 'Burp' refers to the free version of 'Burp Suite'. I don't
| use Burp Suite anymore though. Some months ago I started using
| mitmproxy (https://github.com/mitmproxy/mitmproxy) due to it's
| Python scripting API. I have never looked back since then.
| zegerius wrote:
| This is also my goto.
|
| I work with it to do reverse engineering of APIs for apps on
| Android icw Frida.
| js2 wrote:
| > The restrictions were all client-side.
|
| :-(
|
| I've been in this industry since 1996. Must every programmer make
| this mistake for themselves before they learn? I see this over,
| and over, and over again, at company after company after company.
| I'm really surprised a company as seemingly competent as
| Cloudflare would make this mistake, to say nothing of the larger
| error of allowing forwarding to be setup on an unverified domain.
|
| I weep.
|
| Edit: I'd appreciate a response from Cloudflare on how this
| slipped through the cracks and what changes they are making to
| prevent such mistakes in the future. Not trusting user input is
| among the most basic thing I'd expect a programmer, especially
| one working at CF, to know in 2022, so I'm assuming this was some
| sort of miscommunication between teams.
| jgrahamc wrote:
| It wasn't mean to be client-side only, it was meant to be
| client-side and server-side. Unfortunately, the server-side
| check wasn't happening in the way intended allowing Albert to
| find this vulnerability. We have a special document that
| describes bug classes that need special attention and this
| particular incident has been added to it.
| ffhhj wrote:
| >> We have a special document that describes bug classes that
| need special attention
|
| Sounds like manual testing that should be automated.
| js2 wrote:
| How was this feature able to ship to production w/o the
| server-side check being tested?
|
| Does CF have a launch process for new features? Does that
| process include reviewing the special document and testing
| the new feature against the various bug classes?
| OJFord wrote:
| I think you're giving too much weight to this angle - if OP
| had happened to be in the beta, that part wouldn't have been
| found or mentioned at all.
|
| Bypassing a beta feature toggle isn't really a vulnerability
| IMO, it just allowed OP to find one in this case.
| mox1 wrote:
| What!?
|
| So nobody here can think of any possible scenario where
| bypassing a server-side check for 'Account X can access
| Feature Y' could directly lead to a security issue?
|
| This is absolutely 100% a vulnerability - insomuch as that
| CloudFlare should have an explicit policy that ALL account
| features are enabled / verified server-side, not client
| side.
|
| Think of all the spam that would have happened, had this
| been discovered on underground black-hat forums.
| OJFord wrote:
| Do you want to offer some? It's not clear this even
| bypassed payment that should have been due. That would be
| worse, and still not really a _vulnerability_.
|
| > Think of all the spam that would have happened, had
| this been discovered on underground black-hat forums.
|
| What spam would have happened as a result of early access
| to a new Cloudflare feature, that's independent of any
| (other) bugs/security flaws in that feature?
|
| (Also, even with the _actual_ vulnerability here, what
| 'spam' would have happened? This hijacks recieving.
| Worse, yes, but I don't see how it helps spammers.)
| systemvoltage wrote:
| jgc is just responding to the OP, don't strawman him. OP is
| the person focusing on the client-side beta check, so jgc
| is responding to _that_ particular thing. Has zero bearing
| on the overall issue.
| OJFord wrote:
| I'm not 'strawmanning' him, I'm saying responding to that
| line of argument (or at least in the way that he did,
| accepting the claim it's bad and defending on the back
| foot) lends it undeserved credence.
|
| I agree it 'has zero bearing on the overall issue'. That
| is pretty much my entire point.
| freedomben wrote:
| Sadly, yes :-(
|
| At this point I'm nearly an old man yelling at a cloud, but
| anytime a junior engineer starts I try to go over a handful of
| basic security things. Seems that most people coming out of
| school are aware of SQL injection and XSS, but the concept of
| client-side security is no security is not sinking in, even
| after it's been explained.
|
| I've spent a lot of time trying to figure out why, and the best
| I can come up with is that they just think something like the
| following myths:
|
| * Nobody would ever do that.
|
| * Nobody would ever hit the API directly.
|
| * It would be too hard to do that for somebody who isn't the
| developer.
|
| * The need for an auth token would make it impossible to hit it
| with curl or another client.
|
| * Nobody can change the client code because it's obfuscated and
| unreadable for humans.
|
| Doing it on the server side is harder and a pain in the ass, so
| there's a lot of incentive to do it on the client and not worry
| about the server. It particularly doesn't help that as an
| industry now we're obsessed with speed and quality be damned.
| We can always fix it/do it right later (spoiler alert: you
| never do and you never have time because there will always be
| important new features that are high priority).
| deanc wrote:
| Because we are humans and we make mistakes. Not everyone is an
| expert at every one thing and whilst this is a pretty bad bug
| and should have come up in the planning phase when considering
| security risks, or a security review before hitting production,
| shit happens. Get off your high horse.
| dcow wrote:
| This type of response makes me think you don't actually
| understand how important security is during the product
| development process and to customers. And that GP is on the
| mark being critical. Someone messed up and humans are humans,
| everyone gets that. But that doesn't absolve CF of some due
| explanation to people who are rightfully very concerned. CF
| got really lucky that a bad actor didn't find this. I don't
| like high horses either but you could also use a dose of
| humility.
| deanc wrote:
| I very much do understand the important of security in
| product development. I've been part of numerous security
| audits amongst other things I shan't get too deep into here
| as it's not the point.
|
| The point is that software is developed by humans. Humans
| make mistakes. Find me a single Fortune 500 company that
| hasn't had an embarrassing security issue due to human
| error. As I said, shit happens.
|
| The response time for fixing this and everything else
| entailed in a big bounty process looks good to me. I'm sure
| whatever team worked on this feature will learn from it.
| js2 wrote:
| We put processes in place because not everyone is an expert
| and because humans make mistakes. Shipping new features at a
| company of CloudFlare's maturity and scope should have a
| launch checklist. That checklist should include verifying the
| security of new API endpoints. This was a big goof up and I
| think my criticism is warranted, though I could have worded
| it more charitably.
| groffee wrote:
| It's literally the most basic shit imaginable, never trust
| user input.
| vxNsr wrote:
| Yea this was a surprise.
| powerhour wrote:
| > The bug has since been fixed and Cloudflare has kindly allowed
| me to publish this write-up.
|
| Seems wild that they can deny you the opportunity to publish your
| findings for 7 months after the fix for a beta product went live.
| What is that about? Was that a requirement to get the bug bounty?
| Seems like a great way to encourage finding another buyer
| instead.
| [deleted]
___________________________________________________________________
(page generated 2022-08-03 23:00 UTC)