[HN Gopher] We updated our RSA SSH host key
___________________________________________________________________
We updated our RSA SSH host key
Author : todsacerdoti
Score : 1101 points
Date : 2023-03-24 05:28 UTC (17 hours ago)
(HTM) web link (github.blog)
(TXT) w3m dump (github.blog)
| pxc wrote:
| Developers can't be trusted with SSH private keys. I'm stunned by
| how many times I've asked a developer to send their SSH public
| key only to see them dump their private keys on Slack, in a
| Pastebin, or in a Git commit or wherever.
|
| It would be so much better if standard practice was to generate
| and store the private key on a smartcard or the TPM, so that the
| only file a clueless/careless developer could upload would be a
| stub.
| Reitet00 wrote:
| > It would be so much better if standard practice was to
| generate and store the private key on a smartcard or the TPM,
| so that the only file a clueless/careless developer could
| upload would be a stub.
|
| Yep. Especially given that basically all modern laptops (and
| some PCs) ship with TPMs and ssh can use it via the TPM PKCS#11
| lib. I'm using that daily on multiple machines and it's working
| great.
| DangitBobby wrote:
| Ha. About 30 minutes ago I was trying to figure out if I was
| being MITMd and decided to manually update known_hosts to list
| all public keys published in GitHub's docs. At least I know what
| happened now.
| lisper wrote:
| This is exactly what an attacker impersonating github would say.
| yababa_y wrote:
| and yet my response is identical... bizarre!
| quickthrower2 wrote:
| At the end of the day human power structures and trust are
| heavily important. We are trusting their site didn't get
| hacked, and if it did that we would find out about it. I mean
| if you can't trust anything at all, the modern internet is
| unusable.
| pornel wrote:
| No worries, I wasn't checking it anyway:
| https://nvd.nist.gov/vuln/detail/CVE-2023-22742
| fierro wrote:
| You could literally go grab a Wifi Pineapple, sit in a Starbucks
| in SF, and MITM Github for anyone waking up and blindly running
| `ssh-keygen -R github.com`.
|
| Very opportune time for hackers to try this, since few are
| probably verifying the fingerprint.
| gehaxelt wrote:
| It would be great if Github and other large providers could start
| using SSHFP records [0].
|
| [0] https://www.rfc-editor.org/rfc/rfc4255.html
| LeonM wrote:
| AFAIK openssh does not use SSHFP (VerifyHostKeyDNS) by default,
| for good reason.
|
| Also, SSHFP requires DNSSEC, which Github of course does not
| support. (amongst other shiny new technologies, such as IPv6...
| /r).
|
| And even if GH would deploy DNSSEC, it'll still be opening you
| up to a host of other attack vectors that come with DNS based
| trust anchors.
| foresto wrote:
| For anyone who removed the old key and accepted the new one
| without comparing it to the published list[1], you can review the
| fingerprint you accepted with this command:
| ssh-keygen -l -F github.com
|
| It works for IP addresses as well, which are also published[2].
|
| [1] https://docs.github.com/en/authentication/keeping-your-
| accou...
|
| [2] https://docs.github.com/en/authentication/keeping-your-
| accou...
| thedanbob wrote:
| That domain is returning ERR_CERT_COMMON_NAME_INVALID, at least
| for me.
| 5mv2 wrote:
| Corporate PR lessons from this post:
|
| - Push security breach details down so fewer people read them.
| Nothing until line 11. Or line 13 if you count the TLDR.
|
| - Litter your text with understatements. After publicly exposing
| keys, you replace them "out of an abundance of caution" to
| replace them.
|
| And yet this is one of the best security breach posts I've seen
| in a while. When will companies start to respect developers?
| Phemist wrote:
| > If you are using our ECDSA or Ed25519 keys, you will not notice
| any change and no action is needed.
|
| How is this true? AFAIK, A MITM with the github priv key will be
| able to do an SSH downgrade attack...
| hddqsb wrote:
| The SSH client verifies that the host key sent by the server
| matches the one stored in ~/.ssh/known_hosts. If the key
| material _or algorithm_ is different, the client refuses to
| connect. So the server can 't force the client to switch from
| ECDSA / Ed25519 to RSA.
| cavisne wrote:
| Is it weird that this wasn't in a TPM somewhere? I don't know how
| host keys work, are they rotated based on some root cert
| regularly or is it just the same keypair since the creation of
| GitHub?
| tgsovlerkhgsel wrote:
| Not really weird, no. The host key needs to be used every time
| someone establishes a SSH connection to Github. HSMs can't
| handle that volume of requests.
|
| Given that the key is extremely long lived, this has
| unfortunate implications: If any of these servers is
| compromised, or decides to go randomly spewing memory content
| because of a bitflip, or screws up the nonce on a DSA/ECDSA
| operation, the key can be compromised. This is hard to exploit
| if you're a random person, but for a global adversary that
| collects internet traffic at scale (e.g. NSA), it's feasible
| and I would be surprised if they weren't exploiting such
| issues. They were collecting HTTPS handshakes for a reason.
| iforgotpassword wrote:
| Yes, with ssh you don't use CAs, or certificates at all; just
| raw asymmetric keys. It does support certificates with all
| their bells and whistles though, just nobody does really...
| rocqua wrote:
| Ssh keys are much simpler than the PKI cert infrastructure.
| This has advantages for development, as well as bug resistance.
| But it means ssh expects every server behind the same hostname
| to use the same SSH key.
|
| You can't get two (well functioning) TPMs to have the same key.
| They come with their own, un-extractable* and unchangeable keys
| built in.
|
| *TPMs claim this, it is probably not impossible to extract keys
| just incredibly difficult and requiring specialist knowledge.
| execveat wrote:
| This is factually wrong. All major HSM vendors offer ways to
| backup and replicate keys, it's usually done using key-
| wrapping keys. Here's an example:
| https://www.ibm.com/docs/en/datapower-
| gateway/2018.4?topic=m...
|
| All cloud vendors offer the same functionality, if you think
| about it, so it's not an obscure feature.
| red_admiral wrote:
| There is a diference between the "root" key and the keys you
| can load into a TPM or HSM. The former is sort of built in,
| but you can provision several TPM/HSMs with loadable keys and
| then use those for your crypto implementations - otherwise,
| every time a TPM/HSM broke, you'd risk data loss.
|
| If anyone is using a Windows machine with TPM-based bitlocker
| encryption, you have followed the instructions at
| https://support.microsoft.com/en-us/windows/back-up-your-
| bit... I hope?
| raggi wrote:
| Are they confident that the private key was not seen by anyone,
| or did it actually leak? If it leaked, this may be more of an
| issue than how it appears to be presented.
| renewiltord wrote:
| If you ever push anything to a public Github repository it is
| immediately mirrored by a host of bots. If it was there for any
| amount of time, it's there. That said, if I'm honest I think
| almost no one checks the SSH host key. I would bet that for 9
| out of 10 engineers / devops / security people, they will
| ignore the SSH Host Key and might even set
| StrictHostKeyChecking=no.
|
| I know I know. All you have to do is to just MITM someone's DNS
| or IP traffic or whatever and it's all over. Well, I will
| guarantee you that millions of bitcoin are sitting there
| waiting for you to take it. It's all yours. Go take it. You
| could be a millionaire tomorrow. And if you do it right,
| they'll let you keep some of it if you send most of it back.
| herewulf wrote:
| > This week, we discovered that GitHub.com's RSA SSH private key
| was briefly exposed in a public GitHub repository.
|
| Seriously? How _that_ happened is deeply concerning.
|
| And why weren't the other keys exposed?
| a2800276 wrote:
| Someone at GH was using Copilot DevOps and prompted "How would
| you go about compromising the integrity of all open source
| software?", obviously.
| sneak wrote:
| Most people do not clone using SSH.
| lta wrote:
| Based on which data ? Is there any stats on this ?
|
| I personally never clone using https and I think most of
| the people I work with do the same
| computerfriend wrote:
| The Git HTTPS authentication UX is quite bad, so I don't
| understand how this can be true.
| GauntletWizard wrote:
| Most clones happen without auth on public repos.
|
| Most development work, however, uses SSH
| comprev wrote:
| We use GitHub Enterprise at work and only HTTPS is
| permitted for authentication.
|
| The "insteadOf" git config is added to workstations and
| runners to convert attempted SSH connections over to
| HTTPS.
| computerfriend wrote:
| Why is SSH not permitted?
| darkr wrote:
| I would presume because unless you control the GitHub
| account and the SSH key generation process (making sure
| to generate on smartcard), any developer can upload any
| old public key, and then do something like... commit it
| to a public git repo.
| a2800276 wrote:
| Because "Enterprise". Some C-Level read about Cyber in an
| inflight magazine and decided "The Firewall" needs to be
| "locked down" to only allow essential traffic. So https
| it is!
| comprev wrote:
| I have no knowledge of the risk assessment which led to
| the decision - above my pay-grade; another department;
| etc.
|
| Like most corporate environments, "it is what it is" and
| we do our best to perform our jobs within these
| constraints.
| msm_ wrote:
| If you're logged in and have a SSH key added to your
| account, I believe Github UI will show you SSH clone
| command by default. Therefore I always clone with SSH,
| even public repos.
| a2800276 wrote:
| You may be reading too much into this ... it was a joke :)
|
| But you've made me curious, what's your workflow like? Do
| you use a gui git client? For me, the default option
| provided to me by github is an ssh clone, at least for
| repos that I can own/can push to. This makes it very
| convenient to work with, because as long as I'm logged into
| a shell, I'm authenticated to github. But then again, I
| almost exclusively use the cli and have an ssh key
| configured for my gh user...
| diggan wrote:
| Indeed. One would assume such a private key to be deeply
| stashed away behind multiple security borders. Ending up in a
| GitHub repository seems to imply developers at GitHub somehow
| had access to it.
| fmajid wrote:
| It might be a devops person who leaked it
| ChatGTP wrote:
| This is absolutely unbelievable when you think about it? I just
| checked to make sure it wasn't April 1.
| panda_2134 wrote:
| At the same time, most newbie programmers just remove the old key
| and 'git push' again without checking the authenticity of the new
| key. IMHO those users will never benefit from host keys.
| pashadee wrote:
| Our keys are documented here*.
|
| Hyperlink with broken certificates. Fun.
| janto wrote:
| If this happened to a smaller company they would be dead.
|
| Edit: where can I get good Mercurial hosting? F this.
| amrb wrote:
| A number of large companies you use everyday don't have key
| management, if you ask them for a architecture demo/diagram they
| will squirm!
| paxys wrote:
| Big oof from an ops standpoint. How is a key like this ever
| accessible to an operator in unencrypted text form?
|
| Funny enough, given that every person (and every bit of tooling)
| I know blindly approves SSH public key verification anyways, they
| will likely not even notice this switch.
| iforgotpassword wrote:
| > blindly approves SSH public key verification anyways, they
| will likely not even notice this.
|
| Maybe on initial connect, but who TF ignores that "key has
| changed danger danger, high voltage" warning? You at least look
| around and ask a colleague?
| spyremeown wrote:
| Jesus Christ, what a bunch of incompetent morons, really. I
| honestly hope they get sued to oblivion with this. M$ and GitHub
| are having their way with everything lately, this is the first of
| many mishaps that are bound to happen we put so much trust in one
| company.
| nwellnhof wrote:
| > If you see the above message, you'll need to remove the old key
| by running this command: $ ssh-keygen -R github.com
|
| That's bad advice. Everyone should remove the compromised key,
| not only people seeing the message. If you don't see the message
| and everything still works (while using RSA), you've been MitMed.
| SURA wrote:
| They forced me not to use a password to pull code before and only
| informed me of a password leak several months later. There must
| have been something serious happened this time as well.
| abraham wrote:
| GitHub has had a password breaching the past?
| chrismorgan wrote:
| Given their secrets scanning functionality, I'm mildly curious if
| they discovered the leak by a notification from that system.
| CoachRufus87 wrote:
| ssl cert expired on https://gist.githubusercontent.com/; #hugops
| __turbobrew__ wrote:
| Can't an attacker now man in the middle anyone who hasn't
| connected to github since the host key was rolled?
| paxys wrote:
| Unless they can also hijack the github.com domain, no.
| __turbobrew__ wrote:
| But if you are doing git operations using SSH instead of
| HTTPS then you aren't checking domain certs?
| paxys wrote:
| It's not about the certs. To execute a man-in-the-middle
| attack the attacker has to literally put themselves in the
| middle of the route your packet takes to get to github's
| servers and intercept it.
| __turbobrew__ wrote:
| Sure, there are many ways an attacker can do that. Not
| trusting your IP transit is kind of the whole reason for
| encryption in the first place.
|
| 1. Various DNS hijacking and cache poisoning attacks 2.
| three letter agencies in meet me rooms 3.
| Exploited/hacked routers 4. Wifi hot spots
| [deleted]
| aaomidi wrote:
| That's not that hard tbh. So many attacks.
| nextaccountic wrote:
| Yes, and that's how malware ends up in npm and other package
| managers
| raggi wrote:
| It could be worse, the user has the old rsa host key present
| alongside newer ed / ecdsa keys, they may never rotate out the
| rsa one. A future mitm simply only advertises the rsa key, and
| mitm passes.
|
| Users will need to actively remove the old rsa key in order to
| be safe. It's my first question, and a colleague suggested that
| they believe that the private key was not seen, however, I
| don't see that in the post - unless I'm missing it - and I
| really want this stated very clearly somewhere.
| NieDzejkob wrote:
| I tested this and on a new enough OpenSSH client, the RSA key
| gets replaced using the mechanism described here:
| https://lwn.net/Articles/637156/ (if you connect using a key
| other than RSA).
|
| To be honest, I'd expect something like this to be mentioned
| in the announcement.
| creamyhorror wrote:
| Assuming the user connects to Github first instead of a
| MitM attacker spoofing Github.
| __turbobrew__ wrote:
| > Users will need to actively remove the old rsa key in order
| to be safe
|
| Yea that is my read on it as well. If that is true this is
| much more severe than the blog post suggests.
| [deleted]
| fjni wrote:
| > This week, we discovered that GitHub.com's RSA SSH private key
| was briefly exposed in a public GitHub repository
|
| > ... out of an abundance of caution, we replaced our RSA SSH
| host key used to secure Git operations for GitHub.com
|
| Yeah, that's not an "abundance of caution." That's the bare
| minimum response at that point. What's the "not cautious
| approach?" Make the repo private and go on your merry way?
| neilv wrote:
| I had just composed a comment with the exact same two quotes
| before I saw yours.
|
| I suppose "abundance of caution" might apply, if they
| determined that the only ways it could've leaked were from
| requests that were logged, and they've removed all the ways and
| checked all the logs.
|
| But if I had to guess, even a brief exposure can be picked up
| by bots (and perhaps they can already see log entries for
| this). Even if no one at all picked it up, there'd still be the
| question of whether traces of it are still left behind on
| various infrastructure, in storage and caches (even ML
| training?) not intended for key safety.
| amrb wrote:
| let he who has never committed credentials to a repo, cast a
| stone
| dskloet wrote:
| > out of an abundance of caution, we replaced our RSA SSH host
| [...]
|
| > GitHub.com's RSA SSH private key was briefly exposed in a
| public GitHub repository
|
| What the... That's not "an abundance of caution". That's the only
| possible course of action.
| awill wrote:
| You're absolutely right. It's absurd to frame it this way.
|
| Do they expect people to think "Wow, Github leaked a key, but
| even without knowing if anyone snagged it, they're still
| replacing it. Wow, they go above and beyond."
|
| It's so ridiculous.
| sebzim4500 wrote:
| To be fair, there are somehow people in this post who seem to
| be arguing that GH should not rotate the key.
| Xenoamorphous wrote:
| Not playing devil's advocate but guess they at least have
| some confidence that no one checked out/pulled the repo while
| the key was there?
|
| After all it's them hosting and serving the requests for that
| (and every other) repo.
| execveat wrote:
| Charitable explanation is that they rotated they key
| without waiting for an analysis.
| diggan wrote:
| There is a literal stream of all public data on GitHub. I
| don't think they can 100% know if it was accessed or not.
| colonwqbang wrote:
| "We have no reason to believe" => We don't actually know
| ars wrote:
| They mean they are not sure if anyone actually downloaded the
| private key.
|
| That's the "caution" part.
| irjustin wrote:
| > That's the only possible course of action.
|
| Only _reasonable_ course of action. Possible is to do nothing
| =)
| buildbot wrote:
| I don't understand how this is possible - do they not pre-scan
| commits for secrets? Our Azure devops repos reject any push with
| anything that looks like a credential.
| sltkr wrote:
| The key was probably encoded in some way that made it hard to
| detect, for example, as a VM disk snapshot or a .tar.gz archive
| of a home directory.
|
| Something like that must have happened anyway, since it's
| highly unlikely a private key is just lying around as a plain
| text file on an engineers workstation to be accidentally
| included in a push.
| buildbot wrote:
| That would make more sense, though cred scanning should at
| least attempt to unzip/untar compressed files IMO.
| dajonker wrote:
| Maybe their new copilot made this rookie mistake
| speedgoose wrote:
| Is it a rookie mistake though? It happens to everyone in my
| experience.
| dajonker wrote:
| Leaking something like a private key? You do that once, then
| you learn. If you don't learn, you remain a rookie. Or just
| reckless.
| lopkeny12ko wrote:
| For these major changes that break builds across the entire
| industry, why does Github insist on rolling out the change first
| and _then_ publishing a blog post notice about it? Is it really
| so hard to give a week of advance notice?
|
| It wasn't even that long ago that Github changed the checksums of
| git tarballs which also broke builds everywhere. All we got was a
| "oh btw we changed the archive behavior" blog post _after_ the
| change went live [0].
|
| [0] https://github.blog/changelog/2023-01-30-git-archive-
| checksu...
| ihattendorf wrote:
| I think changing a key because it was compromised is a little
| different compared to changing archive behavior, and can be
| afforded a little leeway with advance notice.
| 8organicbits wrote:
| broken builds << migrating off a leaked key
|
| I'm upset they haven't sent out an email to every user. This
| affects their entire usebase and basically everyone needs to
| take manual action to continue working. If you haven't seen the
| news then you'll see failed builds/git commands.
| retrocryptid wrote:
| So... uh... why was their private key in a form that could be
| stored in a Git repo? In the old days, we used to put private
| keys in hardware co-processors that would never reveal an un-
| encrypted private key. I guess we don't do that anymore.
| 0x0 wrote:
| Imagine the amount of ssh connections that github must handle
| every second, I don't think it's so easy to hold the private
| key in a HSM while maintaining the necessary performance and
| availability.
| vbezhenar wrote:
| HSM is just a computer. May be with crypto accelerators. You
| can set up OpenBSD boxes, put them into safes and disable any
| incoming connections other than your custom HSM protocol and
| that would work better than storing private keys on web
| frontends.
|
| Yes, it makes things hard and unconventional to set up. But
| GitHub is not some small website.
| retrocryptid wrote:
| That's not all of what an HSM is. Or should be. The beefier
| ones come with rf shielding to prevent bad guys from trying
| to reduce the key search space by listening to EM energy
| coming out of the box. And active key zeroation if it
| thinks you're trying to drill through the epoxy surrounding
| the crypto boundary.
| joezydeco wrote:
| Is it technically possible (and/or wise) to duplicate the
| key across multiple HSMs running in parallel? I'm
| guessing if you have a super massive CDN like Apple or
| Meta this is a necessary thing.
| seri4l wrote:
| >And active key zeroation if it thinks you're trying to
| drill through the epoxy surrounding the crypto boundary.
|
| Or passive! Probably it wasn't a real product but I
| recall reading about one that derived its key from the
| field generated by randomly arranged magnetic particles
| in the resin, or something like that. The point was to
| make it impossible to disturb the resin without altering
| the key.
| StopHammoTime wrote:
| Yeah but surely there's a happy medium ground between HSM and
| storing it on some old mates desktop (which would have to be
| what's going on here).
|
| It still shouldn't have been easily accessible to anyone
| except the instances running the SSH service.
| red_admiral wrote:
| That sounds like an engineering problem. I have my political
| opinions on Meta as much as the next person, but every bit of
| evidence I've seen so far is they take this kind of security
| thing seriously and they definitely operate on a larger scale
| than github, although most of their traffic is HTTPS not SSH.
| deathanatos wrote:
| > _I have my political opinions on Meta as much as the next
| person, but every bit of evidence I 've seen so far is they
| take this kind of security thing seriously_
|
| Meta doesn't own Github, MS does.
| retrocryptid wrote:
| HSMs don't handle the entire load of the the connection, only
| the private key operations. At the 10k per sec sites I worked
| at, we attached some beefy HSMs to the few (less than 40)
| machines that terminated TLS.
| parhamn wrote:
| Loading it from HSM to memory/keychain is probably fine too.
| It's certainly odd it found its way to a repo and makes you
| wonder how that could have even possibly happened. And what
| that indicates about their security practices in general.
|
| Github is host to a large percent of US tech IP. Pretty
| concerning if you extrapolate.
| 0x0 wrote:
| The whole point of a HSM is to never reveal the private
| key, but instead to have the HSM perform the necessary
| crypto operation for each request. This wouldn't scale to
| the amount of ssh traffic I would imagine github sees.
|
| I think it's completely reasonable that they have the ssh
| private key in some sort of configuration management
| repository, because they need to be able to deploy that key
| to all their public facing ssh servers. You would hope they
| would have more than 1 ssh server instance world wide for
| availability and resiliency.
| retrocryptid wrote:
| Or you could just buy HSMs that can do the number of
| private key operations per second that you require.
|
| I've never seen the inside of GH's network, but I would
| be surprised to discover they're not distributing the
| load of SSH termination across a fleet of machines. Just
| put a HSM on each of the machines that terminates a SSH
| connection.
| Dylan16807 wrote:
| > The whole point of a HSM is to never reveal the private
| key
|
| That's not "the whole point". You can have methods to
| copy a key between HSMs, methods that can scale just fine
| while being much more secure than a file in a repository.
| amrb wrote:
| I will give you the keys of the kingdom of heaven; whatever you
| bind on earth will be bound in heaven, and whatever you loose on
| earth will be loosed in heaven.
| daneel_w wrote:
| Why 3072 bits for the key instead of 4096? I understand that they
| see a huge amount of connections to their systems every second,
| but this isn't the year 2008. They have the processing power. Is
| it a decision based on "good enough"? Or some compatibility nod
| towards old abandonware still in use?
| madsbuch wrote:
| Maybe this is why I start getting the following error
| error pulling image configuration: download failed after
| attempts=6: x509: certificate is valid for *.githubassets.com,
| githubassets.com, not pkg-containers.githubusercontent.com
| [deleted]
| m4lvin wrote:
| Well, that would be something to show in the "Latest changes" in
| the top right corner on github.com after login, right? But no, I
| have to go here to read about it.
|
| It is also not mentioned on https://www.githubstatus.com/ aka
| status.github.com.
| paxys wrote:
| Conveniently missing from the announcement:
|
| - When exactly was the key leaked? Yesterday? A month ago?
|
| - How long was it visible for? And no, "briefly" doesn't cut it.
|
| - Did any internet traffic reach the page while it was exposed?
| We know you log everything, so it is a yes or no answer.
|
| If any of these answers were pretty, I imagine they would have
| explicitly included them in the post.
| renonce wrote:
| > How long was it visible for? And no, "briefly" doesn't cut
| it.
|
| I don't know how long exactly, but in theory you can subscribe
| to a stream of ALL events happening at GitHub by fetching from
| this endpoint: https://api.github.com/events
|
| With these events you know what new repositories are created
| and what changes are pushed, so you can fetch every new change.
| Once something gets pushed to a public repository, it's very
| likely that some spider will have fetched it within a few
| minutes.
| Maxious wrote:
| Especially something easily identifiable as a SSH private
| key, you get emails from a variety of security vendors to the
| address associated with the commit offering their services
| pbhjpbhj wrote:
| I thought github themselves did checks for credentials?
| Isn't that party of their offered services?
| acdha wrote:
| I'd bet that's why it said "briefly" -- someone got the
| alert, realized what it meant and called their boss.
| KirillPanov wrote:
| > you can subscribe to a stream of ALL events happening at
| GitHub by fetching from this endpoint:
| https://api.github.com/events
|
| Wow I am shocked that they allow "firehose" access not only
| for free, but without even an API key.
|
| Given enough disk and bandwidth, does this mean you could
| keep your own copy of _all_ of github? I 'd love to be able
| to grep the whole thing.
| jonas-w wrote:
| https://www.gharchive.org/
| bspammer wrote:
| The leaked private key should be somewhere in this
| archive then no?
| jonas-w wrote:
| The events stream does not contain files or anything in
| your git repo. Only stuff like commit messages, issues,
| comments, etc.
| 112 wrote:
| i think it does contain code
| squeaky-clean wrote:
| You can view it yourself, it doesn't contain the code
| directly, but it does contain a link to the code blob.
| KirillPanov wrote:
| Wow thanks. I looked all over that page but couldn't get
| a rough idea of the dataset size for the event stream
| (not the commits).
|
| Are we talking 1TB/year, 10TB/year, 100TB/year?
| rschoultz wrote:
| Indeed, and about their statement
|
| > We have no reason to believe that the exposed key was abused
| and took this action out of an abundance of caution.
|
| This is not verifiable, right? As the authentication method has
| no public and required revocation source, and given that the
| key, if having leaked, likely will be acquired by an
| authoritarian government, they can selectively MITM
| organizations and users that are not aware of this blog post.
| hoffs wrote:
| > - When exactly was the key leaked? Yesterday? A month ago?
|
| "This week, we discovered that GitHub.com's RSA SSH private key
| was briefly exposed in a public GitHub repository"
| tyingq wrote:
| That's when they discovered it had been leaked, not
| necessarily when the leak happened.
| CGamesPlay wrote:
| In fairness, it's better that they rotate the key immediately
| before even looking at the logs. But we can demand better
| answers from them from this point.
| paxys wrote:
| Did they rotate it immediately? The only reference to a
| specific time is:
|
| > This week, we discovered...
|
| So they found out about it sometime between Monday and
| Thursday, and rotated it Thursday evening.
| remus wrote:
| This seems a reasonable timeframe to me given the scale of
| the impact.
|
| Security issues are obviously very important, but equally I
| would be worried if they started acting impulsively
| immediately after discovering the issue, because there
| would not have been time to understand the problem and
| consider the options. If you jump on a solution without
| giving it a bit of thought first it's too easy to
| exacerbate the situation.
| Msurrow wrote:
| Its not the time from discovery to rotating the key thats
| OPs issue. What was the time from key being public until
| discovery? Thats the issue. And also, while the key was
| public, did anyone access it?
| UseStrict wrote:
| Maybe I'm missing something, but since all GitHub events
| can be watched through the API, I don't think it
| particularly matters how long it was public. Anecdotal
| but I once accidentally pushed an AWS key (thankfully
| heavily locked down and not a root account) for all of 30
| seconds and it was compromised anyways.
| Msurrow wrote:
| I agree. Even if its only public for one sec it should be
| considered compromised. However, everything is a risk
| management question, so knowing the amount of time it was
| exposed is helpful for other orgs to determine their
| response to this incident. Same if there was an
| accesslog.
| remus wrote:
| Agreed. I don't think that was the point the comment I
| replied to was making though? Maybe I misread it.
| jupp0r wrote:
| > out of an abundance of caution
|
| > This week, we discovered that GitHub.com's RSA SSH private key
| was briefly exposed in a public GitHub repository.
|
| No further questions for this witness Your Honor
| cprecioso wrote:
| One-liner to update your `known_hosts`, directly from github (if
| you have `jq`) ssh-keygen -R github.com && curl
| 'https://api.github.com/meta' | jq -r '"github.com " +
| .ssh_keys[]' >> ~/.ssh/known_hosts
| masklinn wrote:
| Seems unnecessarily complicated, you can just `ssh-keygen -R`
| then `ssh git@github.com`, it should ask if you want to add the
| new key to the known hosts.
|
| Also you should first check:
|
| 1. If you're using GitHub's rsa key at all
|
| 2. That you don't have the other keys
|
| In which case nuking all the keys is completely unnecessary,
| just remove the RSA one.
| jwilk wrote:
| > it should ask if you want to add the new key to the known
| hosts
|
| But how do you know it's the right key?
| masklinn wrote:
| By comparing the fingerprint to a trusted source, which the
| snippet above doesn't do. And if you trust that the https
| source will lead you to the correct github, you can
| trivially check the fingerprint against what github
| publishes.
| cprecioso wrote:
| You'd be pressed to find someone who doesn't trust the
| https source. So... you'd do manually what the snippet
| does?
| retrocryptid wrote:
| Oh. I guess this is why the ops guys at work prefer GitLab. (And
| honestly, GitLab has gotten MUCH better over the last couple
| years.)
| fomine3 wrote:
| They must put a banner on every page on Github to notify this.
| INTPenis wrote:
| >This week, we discovered that GitHub.com's RSA SSH private key
| was briefly exposed in a public GitHub repository.
|
| Let's remove the words briefly exposed and public. Why is a
| private key in a Git repository? These days I don't see any
| reason for this.
| contradictioned wrote:
| Of course, there is no reason. But humans make errors, and
| humans deal with those errors.
| twothamendment wrote:
| I'd love to see the sum of time lost around the world with this
| one little change!
| throwawaaarrgh wrote:
| Everyone should standardize on HTTPS to pull repositories instead
| of plain (non-certificate) SSH. A username and password (often
| called an API token) are used instead of an SSH key.
|
| - An SSH key can be revoked on the server, but the client won't
| register it immediately, and all clients must manually verify the
| new key signature and update their local configuration. This
| doesn't happen with HTTPS; the client just works with the
| server's new cert, no user intervention is required.
|
| - A username can be used in the HTTPS URI to tell both your
| client and the server what credential to use. SSH method requires
| the user to load the correct SSH key first.
|
| - Most servers like GitHub allow more fine grained access control
| for HTTPS tokens than SSH keys.
|
| - HTTPS access method works on all HTTP proxies, whereas SSH is
| often blocked.
|
| - HTTPS can be faster than SSH.
|
| - HTTPS allows a client and server to use specific ciphers to
| address regulatory and other requirements.
|
| - HTTPS tokens are supported in a wider range of password
| managers/keychains than SSH keys.
|
| - Most users don't password protect their SSH keys, but they
| often have a password manager with a master password that they
| can keep HTTPS token in.
|
| - SSH key management is much more complex on the user end than
| HTTPS token management. The former requires ssh-agent, a key
| generator, and instructions for use, as well as specific
| filesystem permissions for keys.
|
| - The user often doesn't understand the idea that they shouldn't
| share their private key. But everyone basically gets they
| shouldn't share their password.
|
| - Virtually no one ever checks the fingerprint of ssh server
| keys. Often entire companies have configuration that disables
| host key checking, completely eliminating the security SSH is
| supposed to provide. With HTTPS the user is implicitly protected
| with no actions necessary.
|
| - Nobody ever commits a private TLS key to a GitHub repo, but
| apparently they do with SSH private keys...
| organman91 wrote:
| > Nobody ever commits a private TLS key to a GitHub repo
|
| Oh how I wish this were true...
| pavlov wrote:
| "Hey Copilot, complete the following statement: let
| GITHUB_COM_PRIVATE_KEY = "
| parhamn wrote:
| Well this is certainly going to break quit a few
| CI/builds/systems that didn't account for this.
| stn_za wrote:
| IMHO, It should break the CI systems.
|
| If you don't pre-bake the known-hosts, then you'd allow each
| new ephemeral run to use whatever github tells you it's key is.
|
| It did briefly break ours, as we pre-bake the known_hosts file
| into our CI image for convenience and security.
|
| Convenience due to CI not having TTY's, so various tools would
| get stuck on prompt Y when connecting to github for the first
| time. (Which is every run, if you are ephemeral CI)
|
| And security, as now everything broke due to github's key
| changing, which is the desired outcome actually.
|
| We bumped the known_host key entry, merged and all is well
| again...
| execveat wrote:
| You shouldn't start with a blank state, instead you should be
| querying https://api.github.com/meta . But there are so many
| repos on Github itself which hardcode the host keys in Github
| Actions, etc.
| computerfriend wrote:
| You could query GitHub's meta API for the host keys, which is
| trusting GitHub's HTTPS certificate instead.
| robbat2 wrote:
| Start protecting yourself from a potential MITM better, mark the
| key as revoked. Hopefully distributions & OpenSSH upstream can
| start shipping this by default.
|
| (sorry, the comments are mangling this, clean version at
| https://gist.github.com/robbat2/b456f09b7799f4dafe24115095b8...)
|
| ``` # You might need to insert this in a slightly different place
| cat >>/etc/ssh/ssh_config <<EOF Host * RevokedHostKeys
| /etc/ssh/ssh_revoked_hosts EOF
|
| cat >>/etc/ssh/ssh_revoked_hosts <<EOF #
| https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-k...
| ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa
| +PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJ
| NlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5Q
| lWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaas
| XVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua
| 2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ== EOF ```
| LeonM wrote:
| You can do verbatim formatting on HN by placing 4 spaces. See
| https://news.ycombinator.com/formatdoc
|
| Thanks for the gist though, seems helpful!
| ollien wrote:
| Just in case anyone is paranoid that this comment has the right
| key, you can generate a fingerprint with $
| ssh-keygen -lf github.old.pub 2048
| SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8 no comment
| (RSA)
|
| and you'll notice that fingerprint is on this archived page
|
| https://web.archive.org/web/20230320230907/https://docs.gith...
|
| (please check my work on your own machines and don't take my
| attestation on faith!)
| bravetraveler wrote:
| Anyone finding the same thing I am?
|
| _RevokedHostKeys_ doesn 't accept _~_ for your home
| directory... while things like _ControlPath_ will.
|
| I'd rather confine this to my account, but I either have to use
| a relative path that doesn't always work... or a fully
| qualified path that includes my username (and may change)
| e12e wrote:
| Thank you - TIL about ssh key revocation (I was aware of them,
| but haven't really used them).
|
| I expanded on your gist:
|
| https://gist.github.com/e12e/0c1868479c0b8d0a52914d44be66d76...
| creamyhorror wrote:
| Useful comment about git using the ECDSA key instead and running
| into a mismatch issue (from the Github discussion):
|
| =====
|
| After I followed the instructions to remove the old RSA key, `git
| pull` started using the ECDSA key, and now shows this warning:
| Warning: the ECDSA host key for 'github.com' differs from the key
| for the IP address '20.205.243.166' Offending key for IP
| in /home/forge/.ssh/known_hosts:87 Matching host key in
| /home/forge/.ssh/known_hosts:88 Are you sure you want to
| continue connecting (yes/no)?
|
| In this case, I think the old ECDSA key for the github.com IP
| needs to be removed from `known_hosts`. This can be done with:
| ssh-keygen -f ~/.ssh/known_hosts -R 20.205.243.166
|
| It worked for me, hope this helps.
|
| I'm not sure why the ECDSA key of the github.com IP has
| supposedly changed - maybe someone can clarify this?
| hddqsb wrote:
| The ECDSA key hasn't changed. As pointed out elsewhere in this
| thread, the official command `ssh-keygen -R github.com` is
| incomplete -- it doesn't remove IP address entries, they need
| to be removed manually. What you got was a conflict between the
| ECDSA key and the old RSA key left over in ~/.ssh/known_hosts
| (the error message was unclear).
|
| Since GitHub's IP address is not stable, I suggest disabling
| the IP-checking feature using Host github.com
| CheckHostIP no
|
| The CheckHostIP feature is pretty useless anyway, it just gives
| a warning when the IP changes and gives a better diagnostic
| message if the key and IP both change at the same time
| (references: https://serverfault.com/questions/1040512/ssh-how-
| does-the-o...,
| https://unix.stackexchange.com/questions/285520/why-does-
| ssh...).
| dave4420 wrote:
| Not mentioned: whether this also affects gist.github.com.
|
| They seem to have been using the same key.
| EMIRELADERO wrote:
| What were the implications of this? Could anyone with that key
| have cloned any GitHub repo, private or not?
| progval wrote:
| This is the host key, so allows impersonating the server to
| clients, not impersonating clients to the server.
| 8organicbits wrote:
| Impersonating the server allows code theft.
|
| > Could anyone with that key have cloned any GitHub repo,
| private or not?
|
| If you try to push to GitHub but someone is MITMing you with
| the leaked key, they can say "I'm an empty repo" and the
| client will helpfully push all the commits to the attacker.
|
| Until everyone updates their .ssh/known_hosts file, a MITM
| attack can still steal code in this way.
| franky47 wrote:
| If anyone from GitHub is reading, the docs page for testing SSH
| keys needs an update on the RSA key fingerprint:
|
| https://docs.github.com/en/authentication/connecting-to-gith...
|
| (it would actually be best to update it to the Ed25519 public
| key, as it's the recommended algorithm when generating a new SSH
| key).
| hddqsb wrote:
| Good point, but a clarification re Ed25519: what matters here
| is the host key algorithm with highest preference, not the
| default/recommended algorithm for the identity key. The highest
| preference appears to be Ed25519 or ECDSA depending on the
| OpenSSH version; see HostKeyAlgorithms in ssh_config(1).
| Aeolun wrote:
| Well, I'm sure glad there was a reason for me trashing the
| previous key without giving it a second thought. Commits need to
| commit!
| fmajid wrote:
| This would have been a good opportunity for them to issue SSH
| certificates (backed by a HSM) instead of just keys.
| hk1337 wrote:
| I'd switch to Bitbucket but they don't support ed25519-sk and
| ecdsa-sk, among other things.
| [deleted]
| acatton wrote:
| $ ssh -o VerifyHostKeyDNS=yes github.com
| The authenticity of host 'github.com (140.82.121.3)' can't be
| established. ED25519 key fingerprint is
| SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU. No
| matching host key fingerprint found in DNS.
|
| They didn't update their DNS SSHFP records?
| https://en.wikipedia.org/wiki/SSHFP_record
|
| What a security nightmare...
| esclear wrote:
| That is the ED25519 key's fingerprint, which shouldn't have
| changed. But that is moot, as GitHub doesn't publish
| fingerprints as SSHFP records.
| lrvick wrote:
| Github had one RSA ssh host key, the most widely supported key
| format.
|
| It was trusted to clone code into the infrastructure of hundreds
| of thousands of organizations. People pin it everywhere.
|
| With this key someone could infect the infrastructure of fintech
| companies and steal billions of dollars. I know this well because
| I run a security consulting company focusing mostly on that
| industry. Mainly this is possible because almost no companies
| check for git commit signing, and Github does not enforce it
| anyway, and I digress.
|
| This key held enough power over value that some might have used
| violence to acquire it.
|
| With that context of course they chose to place this key in a
| hardware security module controlled with an m-of-n quorum of
| engineers to ensure no single human can steal it, expose it, or
| leak it. Right? ... right?
|
| Nope. They admit they just stuck it in a git repo in plain text
| where any engineer could have copied it to a compromised
| workstation, or been bribed for it, for who knows how many years.
| Worse, it was not even some separate employee only intranet git
| repo, but their own regular public production infra and someone
| had the power to accidentally make it public.
|
| I have no words for this level of gross negligence.
|
| Maybe, just maybe, centralizing all trust and security for most
| of the worlds software development to a proprietary for-profit
| company with an abysmal security reputation was not the best
| plan.
|
| I will just leave this here:
| https://sfconservancy.org/blog/2022/jun/30/give-up-github-la...
| kelnos wrote:
| Wow, your assessment of the impact here (or even possible
| impact) is way way way overblown.
|
| In reality, the vast majority of users don't pay attention to
| SSH host keys at all.
|
| Even if an attacker got hold of this private host key, they'd
| have to be able to MitM the connections of their target.
|
| Next, they have to decide what they want to do.
|
| If they want to send malicious code to someone doing a 'git
| pull', they'd have to craft the payload specifically for the
| repo being pulled from. Not impossible, but difficult.
|
| If they want to "steal" source code from someone doing 'git
| push' (perhaps to a private repo on GitHub), that's a bit
| easier, as they can just tell the client "I have no objects
| yet", and then the client will send the entire repo.
|
| And, again, they'd have to have the ability to MitM some git
| user's connections. Regardless, there is no way that they could
| change any code on github.com; this key would not give them any
| access to GH's services that they don't already have.
|
| So I think your anger here is a bit over the top and
| unnecessary.
|
| I agree that it's pretty bad that some GH employee even had the
| ability to get hold of this private key (sure, probably should
| be in an HSM, but I'm not particularly surprised it's not) in
| order to accidentally leak it, but... shit happens. They
| admitted the mistake and rotated the key, even though it's
| likely that there was zero impact from all this.
| lrvick wrote:
| End users do not pay attention but their clients pin the key
| after first use. Also everyone is using gitops these days and
| almost no one is using dns-over-tls.
|
| Imagine you control the right router on a company wifi, or
| any home wifi a production engineer works from and suddenly
| you can cause them to clone the wrong git submodule, the
| wrong go package, or the wrong terraform config.
|
| If you knew a CI/CD system blindly clones and deploys git
| repos to prod without signature checks, and that prod is a
| top 10 crypto exchange with 1b of liquidity in hot wallets,
| then suddenly a BGP attack to redirect DNS is a good
| investment. Myetherwallet got taken over for 15 minutes with
| a BGP so this is not hypothetical.
|
| Should that be the case? Of course not. But the reality is I
| find this in almost all of the security audits I do for
| fintech companies. Blind trust in Github host keys is
| industry standard all the way to prod.
| lxgr wrote:
| Yes, but they will have to un-pin the (now compromised) key
| if they want to continue using Github.
|
| Any compromise would have to isolate them from the "real"
| Github hosts from today onwards, i.e. plausibly MITM them
| continuously, or they would just switch to the rotated key
| to be able to continue working. At least in OpenSSH, this
| means _replacing_ the compromised trusted key, as there can
| only be one per host (or even IP in the default
| configuration).
|
| This is still very bad, but much less catastrophic than
| e.g. a world in which `.ssh/known_hosts` allows multiple
| entries, in which case you'd really have a sustained
| compromise of most clients.
| lxgr wrote:
| Woah, I looked around some more and it seems like the
| opposite is true. Multiple trusted keys per domain can
| exist, and additionally there is this option:
|
| > UpdateHostKeys is enabled by default if the user has
| not overridden the default UserKnownHostsFile setting and
| has not enabled VerifyHostKeyDNS, otherwise
| UpdateHostKeys will be set to no.
|
| This is an OpenSSH extension that _allows any host to
| provide additional host keys the client will then
| silently add to `known_hosts`_. This is really bad in
| this context as it can allow a one-time MITM to install
| rogue keys for github.com that can go unnoticed; check
| your `known_hosts` for any unexpected entries for
| github.com!
| kelnos wrote:
| Sure I can imagine that. And in doing so, I imagine this
| attack is pretty unlikely.
|
| I mean, think of the confluence of things that have to line
| up for this to work for someone. Many stars have to align
| in order for someone to successfully exploit this leak. Is
| it impossible? No, of course not, and so GH was right to
| rotate the key here, even if their server request logs
| suggested that no one had accessed it.
|
| If people actually have this sort of attack in their threat
| model, there are ways to protect against it. Signing
| commits and verifying them. Pinning to a particular git
| SHA1 hash. Etc. If people are not doing these things, then
| it's possible they've made the decision not to worry about
| this sort of attack. Sure, you can disagree with, but I
| think you'd probably be in the minority. That's ok, though;
| you can certainly protect the things you're responsible for
| in stronger ways.
| lrvick wrote:
| I have seen attacks along the lines I just outlined and
| well beyond in my industry many times.
|
| A wildcard TLS cert or an SSH host key in plaintext is a
| loaded weapon laying around and it will be used on a high
| value target.
|
| Sad fact is most of the companies that hold billions of
| dollars of customer assets do development exactly the
| same way a gaming company might. Those that even attempt
| security are unicorns. They bank everything on things
| like DNS, BGP, TLS certs, and ssh host keys working. This
| is like the medical industry before they learned hand
| washing was a thing.
|
| I teach every one of my clients how to do things like
| commit signing, but for every one I help improve there
| are 100 more on track to be hacked any day now.
|
| I can totally forgive a startup that cannot afford senior
| security engineers for a mistake like this, but Microsoft
| can afford that, or at least consultants, and yet they
| can not even enable optional signing for NPM, properly
| enforce git commit signing, or even protect an ssh host
| key trusted by millions in a TEE or HSM.
| uejfiweun wrote:
| Is it negligence or just incompetence? I get the sense that
| security is such a tough problem that all of us, even CISOs and
| red teamers, are incompetent.
| mjg59 wrote:
| Which HSM are you looking at that would be able to handle the
| required number of transactions per second?
| ZiiS wrote:
| The HSM only needs to sign new host keys, transactions per
| decade at thier current rate.
| lrvick wrote:
| The same ones that terminate TLS for millions. Most are
| garbage but at least they keep the key offline. Also you can
| scale horizontally or only use the HSM as a CA to sign short
| lived host keys.
|
| You could also use things like Nitro enclaves which have all
| the same specs as a regular EC2 instance.
|
| Tons of options. They clearly chose none of them.
| mjg59 wrote:
| > The same ones that terminate TLS for millions.
|
| Which ones?
|
| > Also you can scale horizontally
|
| For SSH? Only by having the same private key in all of
| them, which means it's still around somewhere.
|
| > or only use the HSM as a CA to sign short lived host keys
|
| That would be ideal, except that the user experience around
| SSH host certificates is currently woeful.
| Perseids wrote:
| (Not your parent commenter.)
|
| > Which ones?
|
| Take Utimaco Security Server and you'll get 10k+ RSA
| signatures per seconds. Yes, you'll definitely need
| dozens of them, but you'll probably want several for high
| availability and low latency anyway.
|
| > For SSH? Only by having the same private key in all of
| them, which means it's still around somewhere.
|
| End-to-End encrypted key transfer between HSMs is well
| established. Ops for such a setup is definitely going to
| be a pain and lots of manual (and thus expensive) work
| but it is doable. The banking industry has been operating
| like that since forever - _with symmetric cryptography
| only_. Imagine two people being sent letters with XOR
| halves of a transport key and physically meeting at an
| HSM and entering the halves on a PIN pad (not a
| hexadecimal but a decimal PIN pad, mind you, where you
| need to press shift for A-F). From a modern perspective
| it 's totally bonkers, but it works.
|
| If I was tasked with building something for large scale
| companies like GitHub, I would probably pass up on HSMs
| and use commodity measured boot on a minimal Linux for my
| trusted key servers. Outside of these key servers the SSH
| key would only be stored split up with Shamir-Secret-
| Sharing with few trusted employees which will only
| restore the key on ephemeral offline systems. Is that
| overkill? Very much depends on your threat model.
| Investing in the security of their build chain runners
| and data-at-rest integrity might have higher pay off. But
| then again, GitHub has become such a big player that we
| should also hold them to very high standards. And the
| setup can be re-used for all their secret management,
| e.g. TLS certificate private keys.
| mjg59 wrote:
| Thanks! That's hugely helpful - everything I could find
| myself seemed to be an order of magnitude or so slower
| than that (and they support ecdsa, too, so I can't even
| object on the basis of algorithmic support). With
| hindsight my reply was somewhat flippant - really I just
| wanted to push back on the idea that this was a problem
| that could be solved by simply sprinkling HSMs on it
| rather than something that requires a meaningful
| infastructural effort. Github is a sufficiently core
| piece of infrastructure that I agree more should be
| expected, and I hope this does serve as encouragement for
| them to do that.
| ses1984 wrote:
| The key has to be in memory on all of their front end servers.
| Do you think a quorum of engineers should get together every
| time a front end server boots or reboots?
|
| Genuinely asking because I've struggled with this question.
| kccqzy wrote:
| The key doesn't have to in memory on all of their front end
| servers. Any respectable company that cares about security
| wouldn't put their TLS private key on all of their front end
| servers anyways. You expose a special crypto oracle that your
| front end servers talk to; the oracle can be a specially
| hardened process on a dedicated server or better yet a HSM;
| the point is, the private key is never in memory on any
| server that handles untrusted data.
| bob1029 wrote:
| I don't understand how initializing cryptographic keys from
| an HSM at boot time is an untenable proposition. The quorum
| would be for accessing the key by human means. You can have a
| separate, approved path for pre-authorized machines to access
| cryptographic primitives across an isolated network.
| lrvick wrote:
| Lots of cloud instances support remote attestation these days
| which gives you a reasonable path to autoscaling secure
| enclaves.
|
| 1. You compile a deterministic unikernel appliance-style
| linux kernel with a bare bones init system
|
| 2. You deploy it to a system that supports remote attestation
| like a nitro enclave.
|
| 3. It boots and generates a random ephemeral key
|
| 4. m-of-n engineers compile the image themselves, get the
| same hash, and verify the remote attestation proof confirming
| the system is running the bit-for-bit trusted image
|
| 5. m-of-n engineers encrypt and submit shamirs secret shares
| of the really important private key that needs protecting
|
| 6. key is reconstituted in memory of enclave and can start
| taking requests
|
| 7. Traffic goes up and autoscaling is triggered
|
| 8. New system boots with an identical account, role, and boot
| image to the first manually provisioned enclave
|
| 9. First enclave (with hot key) remotely attests the new
| enclave and obtains its ephemeral key (with help of an
| internet connected coordinator)
|
| 10. First enclave encrypts hot key to new autoscaled enclave
|
| 11. rinse/repeat
| CGamesPlay wrote:
| > With that context of course they chose to place this key in a
| hardware security module controlled with an m-of-n quorum of
| engineers to ensure no single human can steal it, expose it, or
| leak it. Right? ... right?
|
| This is unfortunately not how SSH works. It needs to be
| unlocked for every incoming connection.
|
| You raise valid hypotheticals about the security of the
| service... but fixing it involves removing SSH host key
| verification from Github; better OpsSec would not fully resolve
| this issue.
| p-e-w wrote:
| Hardware security modules can perform key operations without
| allowing anyone to access the key data. Key material being
| (accidentally or deliberately) leaked has been a solved
| problem for a long, long time.
| CGamesPlay wrote:
| Used in such a way, the entirety of Github's SSH
| connections would be bottlenecked by this HSM. It wouldn't
| scale and it would be a single point of failure. As lrvick
| points out, you'd have to use a certificate-based host key
| scheme like PKCS#11 to make this scalable. That's fine, but
| it is a different scheme than RSA host key identification.
| lrvick wrote:
| I gave two options but they are separate.
|
| PKCS#11 is a protocol for talking to hardware security
| modules for generic cryptographic operations to keep a
| key out of system memory.
|
| You can totally take a single host key scheme and use
| HSMs or TEEs at scale.
|
| Openssh supports using openssl as a backend which in turn
| can use the PKCS#11 protocol to delegate private key
| operations to remote enclaves or TEEs. Nothing stops you
| from loading your key into a fleet of them and scaling
| horizontally just like you scale anything else.
| CGamesPlay wrote:
| > Nothing stops you from loading your key into a fleet of
| them and scaling horizontally just like you scale
| anything else.
|
| Isn't having one key the root of the problem? If you've
| got one key, it has to be transferred to each HSM module,
| which means it's going to be in their deploy code. My
| understanding is that the safe way to scale PKI is to
| have an HSM generate a key internally, and have that key
| be signed by a CA in a different HSM, so private key
| material never leaves any HSM.
|
| I guess you're saying that SSH RSA host keys support
| this, but I'm only familiar with doing it using _looks up
| correct terminology_ X.509 certificates like for HTTPS.
| arianvanp wrote:
| HSMs can key-wrap a key.
|
| E.g. Foo can encrypt the key X with Bars public key and
| then Bar can import key X.
| fireflash38 wrote:
| > If you've got one key, it has to be transferred to each
| HSM module, which means it's going to be in their deploy
| code
|
| There's products out there that allow for keys to be
| securely shared amongst multiple HSMs, without the key
| ever existing in clear text outside the HSM.
| fireflash38 wrote:
| Most HSMs have modes that allow for load to be
| distributed amongst multiple HSMs.
| lrvick wrote:
| I am well aware how ssh works. I have written ssh servers and
| design secure enclave key management solutions for a living.
|
| Even if they wanted the most quick and dirty lazy option with
| no policy management, they could stick the key in a PKCS#11
| supporting enclave every cloud provider supports these days.
| OpenSSH natively supports them today.
|
| At a minimum they could have placed their whole ssh
| connection termination engine in a immutable read only and
| remotely attestable system like a Nitro enclave or other TEE.
| You do not need to invent anything new for this.
|
| There are just no excuses here for a company with their size
| and resources, because I do this stuff all the time as just
| one guy.
| kelnos wrote:
| Would these secure storage methods for the key be able to
| scale up to the ssh connection volume an outfit like GH
| sees? Genuinely asking; I don't know the answer. I just
| feel like having to hit a HSM or something similar every
| time a new ssh connection comes in wouldn't work. Or at the
| very least I would not see the sub-100ms connection time I
| can get right now.
| lrvick wrote:
| Easily. You could have only the CA key in the enclave and
| have it sign throw-away session keys on stateless
| autoscaling machines, and/or you can have your actual ssh
| terminations happen in cloud enclaves like nitro enclaves
| which have all the same specs and cost of a regular cloud
| server.
| marcosdumay wrote:
| > It needs to be unlocked for every incoming connection.
|
| Yep. Well, certificates exist exactly to bridge the GP's
| requirement with your reality.
| ed25519FUUU wrote:
| Thoughts with the sysadmins and devops people out there on this
| wonderful Friday afternoon.
|
| These kinds of changes suuuuuuck. Messing with known_hosts file
| is not always something easy to do. Might require a image
| rebuild, if you have access at all.
| reuven wrote:
| Why did I, a paying GitHub customer, discover this when trying to
| do "git pull" on a repo I own?
|
| My immediate assumption wasn't that GitHub had changed keys. I
| thought that my computer was having some sort of problem. Only
| after searching around for a bit did I find out that yes, GitHub
| changed their private key.
|
| Couldn't they have e-mailed all of their users and/or customers,
| to tell us what had happened?
| tabbott wrote:
| Emailing every GitHub user may take a while, but it seems wrong
| that they didn't post it on https://githubstatus.com; this is
| effectively an outage in their service -- customers should not
| need to google for their blog post to find out what happened.
| tome wrote:
| Their instructions could stand to be more thorough. If you have
| hashed hostnames in your known_hosts, you don't just have to
| ssh-keygen -R github.com
|
| you also have to ssh-keygen -R 140.82.121.3
|
| which is the IP address for that host. Otherwise you'll get
| something like Warning: the RSA host key for
| 'github.com' differs from the key for the IP address
| '140.82.121.3' Offending key for IP in
| /home/tom/.ssh/known_hosts:77 Matching host key in
| /home/tom/.ssh/known_hosts:102
| jasongill wrote:
| There are numerous IP's that are reported for github.com. The
| easiest way to remove all of the offending known_host entries
| appears to be: sed -i.github-removed
| '/AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa/d'
| ~/.ssh/known_hosts
| St-Clock wrote:
| Thank you so much for this shortcut, I was trying a way to
| find all of the bad known_host entries and this did the
| trick!
| guluarte wrote:
| Have a fun weekend, all my DevOps fellows!
| barryfam wrote:
| In case anyone wants another example of how to set up revoked
| host keys: # Add this to the top of ~/.ssh/config
| RevokedHostKeys /home/username/.ssh/revoked_host_keys
|
| then cd ~/.ssh mkdir revoked_host_keys.d
| echo 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwB
| K6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0
| wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84K
| ezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUU
| mpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28
| G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==' >
| revoked_host_keys.d/github-leak-2023.03.23 ssh-keygen
| -k -f revoked_host_keys revoked_host_keys.d/*
|
| the last command combines all keys in the subdirectory into the
| one properly-formatted binary file. So, you can add more keys
| into the subdirectory later (but you do have to remember to rerun
| it -- personally I saved it into a one-line script at
| ~/.ssh/revoked_host_keys.sh so I don't forget)
| stockhorn wrote:
| How likely is it, some bad actor can pull off a MitM on someone
| using a resident/coporate connection (and not a public hotel
| wifi...) ? How would one even pull this off? (Except for ISPs or
| security agencies)
| mjg59 wrote:
| In an ideal world this could be avoided by using SSH certificates
| - an assertion by a trusted party that the key presented by an
| SSH server is legitimate. The signer of those certificates could
| be in an HSM, ensuring that the private keys can never be leaked,
| and the certificates could be sufficiently short-lived that
| there'd be no need to deal with revocation.
|
| Unfortunately there's no way to make this _easy_. To associate a
| CA key with a specific domain you need to add something like:
|
| @cert-authority github.com (key)
|
| to known_hosts, and this (as far as I know) can't currently be
| managed in the same way that keys are generally added to
| known_hosts on first use when you ssh to something. So instead
| we're left with every github.com SSH frontend using the same
| private keys, and those can't be in an HSM because it simply
| wouldn't scale to the required load. Which means the private key
| exists in a form where this kind of thing can occur.
| mjg59 wrote:
| Wrote my thoughts on this up in some more detail at
| https://mjg59.dreamwidth.org/65874.html
| londons_explore wrote:
| > every github.com SSH frontend using the same private keys
|
| HTTPS has the same problem. Compromise any frontend, and you
| can MITM traffic to any other front-end till the expiry of the
| certificate (usually 90 days).
|
| I would really like to see some kind of almost realtime
| delegation - so that for example, every second a new sub
| certificate could be generated and signed by a HSM, valid for
| only 2 seconds.
| 8organicbits wrote:
| Lots of bad things happen if your front-end is compromised,
| that's different and it's a high bar. They can persist access
| with a backdoor. They can exfil historical data, password
| hashes. They can corrupt and modify data.
|
| With HTTPS certs you usually have at most 90 days of impact
| when a key is leaked (less if you revoke and software is
| checking CRL). GitHub used the same RSA key for over a
| decade, they may have continued using this key for quite some
| time more had they not noticed the leak this week.
| mjg59 wrote:
| Clock skew unfortunately makes that scale a little
| impractical, but yeah it would be great to have mechanisms to
| delegate endpoint certificate issuance to much shorter
| timescales
| uglygoblin wrote:
| I think it's bizarre they just casually use jq in their
| "automatic" fix commands. Is it really that ubiquitous?
| Madhouse61 wrote:
| If I was a three letter agency, which I'm not. I'd be interested
| in conducting a covert operation to glean data on Github
| repositories, sort of like a mass-code surveillance tool. The
| likes of this idea sparks my curiosity - but who I am kidding,
| _this could never happen._ GitHub is a secure and reliable
| corporation owned by _Microsoft_.
| screcth wrote:
| Does this mean that someone could decrypt a old dump of traffic
| between a client and GitHub? If so, this means that every repo on
| GitHub must be assumed to have been leaked?
| coderintherye wrote:
| If you are seeing "WARNING: REMOTE HOST IDENTIFICATION HAS
| CHANGED!" in respect to Github.com then this is likely why.
| CGamesPlay wrote:
| Reading the comments here, I can see several things GitHub could
| have done to improve the security around their host key. But
| pragmatically speaking, any change involves changing the host
| key, which is an extremely disruptive operation. Now that they've
| bit the bullet and forced everyone to rotate, are they going to
| use this time to actually generate a new, secured key that never
| touches persistent memory unencrypted?
| lrvick wrote:
| Of course not. This is Microsoft we are talking about.
|
| If they did something that expensive for security they would be
| bragging about it from their blog to earn any possible trust
| they could at a time when they could really use it.
|
| If you are reading this Microsoft, make me look silly. Please.
|
| I will even give you the enclave designs for free.
| encryptluks2 wrote:
| Good that they changed it but this is certain to break a lot of
| automation pipelines. I'd rather be safe than sorry though.
| qsort wrote:
| No solution is good and this is the least bad. Security people
| are holding the line, which is great.
| [deleted]
| stn_za wrote:
| You should be worried if this does NOT break your automation
| pipeline.
| protoman3000 wrote:
| How does the process to write this PR-bullshit go exactly?
|
| Do the PR people have technical knowledge or does some tech
| person write
|
| "I accidentally screwed up and put the key public so we have to
| replace" and the PR glosses over it and rewrites "Out of an
| abundance of caution..."?
| vbezhenar wrote:
| It continues to amaze me that ssh still does not use https PKI
| and relies on developer manually checking fingerprint (which he
| supposedly does in a hard-to-google location and realistically he
| does not check anything). So much care and work went to implement
| web security and developers still basically live in a self-signed
| world.
|
| At least put your key to https://github.com/.well-
| known/ssh/ed25519.pub so I don't need to Google it... And may be
| some day ssh will support it natively. Someone need to act first.
| darkr wrote:
| OpenSSH supports centralised CAs (non x509 based) natively, but
| it's not commonly used outside of large orgs.
| wruza wrote:
| Wouldn't that require a domain name for each sshd box? And
| wouldn't it be easier at that point to simply do (s?)sh over
| https/wss? I'm not an expert, just curious.
| LeonM wrote:
| The fact that this key was apparently not stored in an HSM, and
| that GH employees had access to this private key (allowing them
| to accidentally push it) means that effectively all communication
| with GH since the founding of the company has to be considered
| compromised. This basically means that, depending on your level
| of paranoia, you will have to review _all_ code that has _ever_
| interacted with Github repositories, and any code pushed or
| pulled from private repositories can no longer be considered
| private.
|
| Github's customers trust GH/MS with their code, which, for most
| businesses, is a high value asset. It wouldn't surprise me if
| this all results in a massive lawsuit. Not just against GH as a
| company, but also to those involved (like the CISO). Also, how on
| earth was it never discovered during an audit that the SSH
| private key was plain text and accessible? How has GH ever been
| able to become ISO certified last year [0], when they didn't even
| place their keys in a HSM?
|
| Obviously, as a paying customer I am quite angry with GH right
| now. So I might be overreacting when I write this, but IMO the
| responsible persons (not talking about the poor dev that pushed
| the key, but the managers, CISO, auditors, etc.) should be fined,
| and/or lose their license.
|
| [0] https://github.blog/2022-05-16-github-achieves-iso-
| iec-27001...
| megous wrote:
| A bit of an overreaction, right?
|
| Number of people just not blindly using TOFU with github over
| ssh must be quite low.
|
| Who here went here
| https://docs.github.com/en/authentication/keeping-your-accou...
| and added the keys manually to known_hosts before using github
| for the first time?
| nebulous1 wrote:
| I would have guessed that the significant majority would be
| using TOFU, but in any case the actual key was leaked so it
| wouldn't matter which method was used.
| megous wrote:
| The point is about how much people care, not about whether
| verification works or not when the key is leaked.
| nebulous1 wrote:
| > Number of people just _not_ blindly using TOFU with
| github over ssh must be quite low.
|
| Oh, I missed the "not" the first time I read it, which
| changed your overall meaning entirely. My bad.
| jessaustin wrote:
| In this case "not just" would have been more legible than
| "just not".
| Shank wrote:
| > How has GH ever been able to become ISO certified last year?
|
| ISO/IEC 27001:2013 doesn't say you have to store private keys
| in HSMs? It just requires you to have a standards compliant
| ISMS that implements all Annex A controls and all of the
| clauses. Annex A and the clauses don't specifically mandate
| this.
|
| If you can convince an auditor that you have controls in-place
| that meet the standard for protecting cryptographic media you
| basically meet the standard. The controls can be a wide variety
| of options and don't specifically mandate technical
| implementation details for many, many things.
|
| You shouldn't rely on ISO/IEC 27001:2013 as attestation for
| technical implementation details here. Just because your
| auditor would red flag you doesn't mean all auditors would. The
| standard is only as effective as the weakest, cheapest auditor,
| and there are perverse incentives that make auditors
| financially incentivized to certify companies due to recurring
| revenue.
| LeonM wrote:
| > You shouldn't rely on ISO/IEC 27001:2013 as attestation for
| technical implementation details here.
|
| Thanks for the insight, good advice.
|
| But also from the same GH article [0]:
|
| _The ISO 27001 certification is the latest addition to
| GitHub's compliance portfolio, preceded by SOC and ISAE
| reports, FedRAMP Tailored LiSaaS ATO, and the Cloud Security
| Alliance CAIQ._
|
| Do you have any knowledge on one of these certifications (for
| exmaple FedRAMP) that puts any restrictions on handling key
| material?
|
| [0] https://github.blog/2022-05-16-github-achieves-iso-
| iec-27001...
| Shank wrote:
| SOC isn't very strict either, it's more opinionated and
| allows more leeway with auditor standards. The CSA Star
| CAIQ is open and free, but it doesn't mandate HSMs
| (https://cloudsecurityalliance.org/artifacts/cloud-
| controls-m...). ISAE is a precursor to SOC 2 (ish). The
| only one of these that I'm not familiar with is "FedRAMP
| Tailored LiSaaS ATO".
|
| As the adjacent commentor 1970-01-01 states, PCI DSS is
| actually pretty strict and requires use of HSMs. However,
| that level of PCI compliance is only required for
| institutions that actually handle the full credit card
| number. GitHub uses Stripe as a payment gateway, so they
| don't need to meet it.
|
| FIPS 140-3 levels 3 and 4 often are met by using HSMs, but
| technically speaking there aren't any standards that I know
| of that exist outside of the payments industry that have
| hard requirements for HSM use for all cryptographic key
| media.
|
| I think the unfortunate reality is that many organizations
| struggle to deploy HSMs widely for all cryptographic media
| because they aren't very scalable and deployment can lead
| to other operational constraints that many companies can't
| or won't deal with. It's much easier for low-frequency
| high-importance tasks like signing new releases of OS
| images or packages, rather than I/O heavy high-frequency
| operations for a site like GitHub.
|
| So, in-summary, look for PCI DSS or FIPS 140-3 Levels 3 and
| 4, but, be prepared to discover that "creative" solutions
| may let a company meet even the highest levels of FIPS
| compliance without HSMs for all cases.
|
| I know it's sucky advice, but for ISO in-particular, I
| suggest just acquiring and reading 27001 if you want to use
| it as a basis for decision making. It does offer a lot, and
| I think it's a very well-written standard, but all of the
| implementation advice is in ISO 27002, and it's not
| required. The advice in 27002, when used to meet 27001,
| leads to a very compelling program. But a 27001-compliant
| org can completely forego 27002 and DIY it as long as it
| meets the test criteria set-out by the auditor.
|
| Edit: Also, every auditor will include standard disclaimers
| in their report that they perform sampling-based testing,
| and that the testing is not a complete guarantee of the
| state of the company. ISO in-particular is performed by
| getting an initial audit, followed by two years of
| surveillance audits that test the entire suite of controls.
| But due to sampling-based methodology, something can be
| overlooked or evidence can be provided that doesn't
| holistically reflect the org in all cases. If it's any
| consolation, this particular issue will certainly be in
| their audit for next year, as a security incident and
| probably lead to an opportunity-for-improvement from the
| auditor.
| j45 wrote:
| This is an important point. Most certification standards
| are not tech literate, only process literate.
|
| So you may be able to get certified for policy, process
| but not necessarily the programming to allow something in
| the first place.
|
| Hopefully now knowing this will improve under Microsoft.
|
| For those who value this, self- hosting or on-prem
| instances of git might shoot up in importance.
| 1970-01-01 wrote:
| I think you're looking for something like PCI DSS
| compliance, which requires you to store keys in a HSM, and
| is much more prescriptive with key management.
| bob1029 wrote:
| PCI-DSS is incredibly aggressive with key management
| considerations. They get down to radio frequency concerns
| in pin pad hardware, etc.
|
| It took us ~2 years of back & forth with various parties
| & auditors to get per-client exclusions for accepting
| end-customer debit PIN codes _in-branch_ on an iPad
| screen. These banks do not have fully-compliant solutions
| and must have exceptions on file.
| joezydeco wrote:
| Wasn't SPoC supposed to help with that?
| bob1029 wrote:
| It showed up too late. We did our integration 2016-2017.
| We were held to far more unrealistic standards at the
| time.
| patch_cable wrote:
| You may be confusing PCI-DSS versus PCI-PIN, which are a
| little different. You're right about the requirements
| around acquiring pins though.
| amrb wrote:
| Your 100% right to hold critical infrastructure to higher
| standards. Putting Solarwinds aside, how many companies could
| to grind to a halt via this 3rd party.
| peterkelly wrote:
| Git provides the ability for authors to sign their commits with
| their own private key. To ensure the integrity of code in a
| repository, this method should be relied on rather than
| whatever hosting provider(s) have a copy of the repository.
|
| Requiring all commits to be signed by trusted keys avoids the
| risks associated with someone tampering with a repository
| hosted on GitHub if they are able to get access to it, although
| it doesn't protect against code being leaked.
|
| See here for details: https://git-scm.com/book/en/v2/Git-Tools-
| Signing-Your-Work
| travisd wrote:
| Parent comment is concerned with privacy, not authenticity.
| They're not worried that someone modified their code, they're
| worried that someone saw it.
| Spooky23 wrote:
| The parent was assuming full compromise.
|
| The risk of disclosure is pretty obvious with GitHub, and
| I'd assume anyone with low risk tolerance here is using
| something else, including the on-prem GitHub. I can think
| of a dozen higher risks.
| DistractionRect wrote:
| They specifically called out the need to review all code
| that ever interacted with github. The implication is that
| you can't trust it hasn't been tampered with.
| brightball wrote:
| I wonder if they found it by turning on their own secret
| detection system?
| djbusby wrote:
| What license?
| LeonM wrote:
| Auditors require a license/accreditation to do certain
| certifications.
| lxgr wrote:
| > [...] depending on your level of paranoia, you will have to
| review all code that has ever interacted with Github
| repositories [...]
|
| Not to diminish the problems of having a large entity like
| Github handle a private key like that, but if that was your
| level of paranoia, you probably should have used commit
| signatures all along and not relied on Github to do that job
| for you.
| dannyincolor wrote:
| As usual on HN, I find the pragmatic response about 3 pages
| down in the replies to an extremely hyperbolic top-level
| comment.
|
| I also don't want to diminish the concerns around Github or
| similar orgs losing control of a private key, but the far
| more realistic concern for the vast majority of threat models
| is often put to the wayside in favor of what amounts to a
| scary story. Rather than the straightforward key removal and
| replacement that this should be, I (and surely many others)
| have spent all morning combatting this specific FUD that
| cropped up on HN with leadership and many engineers. It's
| actually quite detrimental to quickly remediating the actual
| concerns introduced by this leak.
|
| I understand that security inspires people to be as pedantic
| as possible - that's where some big exploits come from on
| occasion - but I really hope the average HN narrative changes
| toward "what is your actual, real-world threat model" vs.
| "here is a highly theoretical edge-case scenario, applicable
| to very few, that I'll state as a general fact so everyone
| will now wonder if they should spend months auditing their
| codebase and secrets". Put simply: this is why people just
| start ignoring security measures in the real world. Surely
| someone has already coined the term "security fatigue".
|
| It's all just a bit unbalanced, and definitely becomes
| frustrating when those suggesting these "world is burning"
| scenarios didn't even take the available precautions that
| apparently would satisfy their threat model (i.e. commit
| sigs, as you suggested)
|
| Ok, end rant :)
| Veliladon wrote:
| > The fact that this key was apparently not stored in an HSM,
| and that GH employees had access to this private key (allowing
| them to accidentally push it) means that effectively all
| communication with GH since the founding of the company has to
| be considered compromised.
|
| For a host key? Like I get that being able to impersonate
| Github isn't great as far as state level actors having the
| ability to do this but you do know the actual transport layer
| keys are ephemeral and aren't derived at all from the host key,
| right?
| LeonM wrote:
| > state level actors having the ability to do this
|
| Not just nation state actors, but basically anyone in a
| position to MITM.
|
| Also, you don't have to be a nation state actor to extort a
| GH employee. Any bad guy can do a "give me this key or I'll
| hurt your kid". People are being extorted for a lot less.
|
| There are billions of dollars of assets flowing through GH's
| infrastructure, for the sake of safety (!= security) of
| Github's employees, nobody should ever have access to key
| material.
| palijer wrote:
| >There are billions of dollars of assets flowing through
| GH's infrastructure
|
| Do you mean source code here? I have a hard time believing
| source code holds that much value.
| adityasaky wrote:
| You've also got to factor in all the software that relies
| on projects developed primarily on GitHub.
| nicoburns wrote:
| > As of January 2023, GitHub reported having over 100
| million developers and more than 372 million
| repositories, including at least 28 million public
| repositories
|
| If there are ~350 million private repos then they'd only
| need to be worth an average of $30 each to be worth a
| billion dollars in total. Which doesn't seem farfetched.
| iudqnolq wrote:
| A better way of quantifying this would be to look at the
| impact of real life source code leaks. I'm not aware of
| any significant monetization of the windows source leak,
| for example.
| palijer wrote:
| I think that is incredibly farfetched. If you got access
| to 1,000 random private Github repos, I don't think you
| could sell them, or otherwise utilize them for anywhere
| near that value, if anything.
| explaininjs wrote:
| Considering the looooong tail of these repos are forks
| with no changes, sample code, toy projects abandoned
| after a single commit, etc. etc., I'd say it's pretty far
| fetched.
|
| For proof, try searching for a mundane string in GH Code
| search. The vast majority of repos you see will be
| basically garbage.
| kimburgess wrote:
| Spot on. Most people will remain absolutely rational when
| faced with irrational threats. The only protection against
| that is ensuring that condition cannot be encountered.
| Gurkenglas wrote:
| "Obey so they don't carry out their threat." may be
| prescribed by classical decision theory, but I wouldn't
| call it rational when it's bad for you to be known to do
| it. I just asked classical decision theory what decision
| theory to pick and I think it said "take action x such
| that, if you do x, and everyone had known since 15:53 UTC
| Mar 24, 2023 that you'd do x, you'd have done as well as
| possible.". So what deserves to be called "rational" may
| be to do what the person you wish you'd always been would
| do.
| fierro wrote:
| it doesn't take a state actor to MITM this. It takes a Wifi
| Pineapple advertising a fake AP and tired devs in Blue
| Bottle smashing `ssh-keygen -R github.com` without
| verifying the fingerprint. Very simple. Even easier than
| trying to MITM a site accessed via browser, which will
| probably have at least HSTS to help you out.
| 0xEFF wrote:
| Yes for a host key. It's like accidentally publishing the tls
| key for https://accounts.google.com
|
| The host key is the only thing ensuring you're actually
| talking to GitHub.com when you push code.
|
| To add to sibling comments, it should not have been possible
| to make this mistake. That it was possible is concerning.
| runeks wrote:
| > [...] the actual transport layer keys are ephemeral and
| aren't derived at all from the host key, right?
|
| Great! Then I can communicate confidentially with whomever is
| MITM'ing me.
|
| /s
| sgt wrote:
| > How has GH ever been able to become ISO certified last year
| [0], when they didn't even place their keys in a HSM?
|
| ISO 27001 certification does not _require_ you to put keys into
| an HSM. The standard requires you to have controls in place, be
| aware of your risks and to maintain a risk register. But in no
| way does the standard require HSM 's.
|
| The standard would even be OK with storing this on a floppy
| drive if the risks surrounding that were identified and
| mitigated (or accepted).
| nebulous1 wrote:
| > The fact that this key was apparently not stored in an HSM,
| and that GH employees had access to this private key (allowing
| them to accidentally push it) means that effectively all
| communication with GH since the founding of the company has to
| be considered compromised.
|
| I think this suggests we need more information from github. For
| instance GH employees may not always have had live access to
| this key, this could have happened as part of an operation that
| gave temporary access to an employee only recently. Or it could
| have been stored plaintext on multiple employees' home
| computers since creation.
|
| When was the leaked key created anyway?
| j45 wrote:
| Makes self-hosting git look more preferable.
|
| The cloud is always the convenience of someone else's computer
| over some amount of security.
| j16sdiz wrote:
| I have never knew a single person put ssh host key into HSM.
|
| In fact, this is not a supported option in openssh.
| LeonM wrote:
| > I have never knew a single person put ssh host key into
| HSM.
|
| You probably also never met a single person where the SSH
| interface sees millions of sessions as day with valuable
| assets (code) being transported over said sessions.
|
| > In fact, this is not a supported option in openssh.
|
| This definitely _is_ supported. Though documentation for this
| is often HSM vendor specific, which if heavily NDA 'd. So
| that's why you probably haven't found much information about
| it.
| tialaramex wrote:
| What I expect has happened here is that you've remembered
| that your HSM comes with instructions for how to use PKCS11
| to make user authentication rely on the HSM and you've just
| assumed that's relevant here. While I'm sure the vendors
| make it seem like this is all very secret, it's just a
| pretty boring C library and it's probably half-arsed in
| real world implementations.
|
| AIUI OpenSSH does not provide any way to use PKCS11 to
| protect host keys, which are the concern here.
|
| You _can_ use PKCS11 to sign OpenSSH certificates, so if
| GitHub had elected to use certificates here, it could have
| protected the CA keys for those certificates in an HSM, but
| it did not.
| tialaramex wrote:
| Correction: It was pointed out elsewhere that you can
| just tell sshd to use PKCS11 keys via the SSH agent
| mechanism, and so yes that would allow use of an HSM for
| host keys
| bob1029 wrote:
| > This definitely is supported.
|
| Agreed. I have seen some crazy stuff in the payment card
| industry. I can't recall what I can and can't talk about so
| I'll just say "Atalla".
| lxgr wrote:
| Yes, but that would either be a fork of OpenSSH, private
| or open source (both are possible since it's BSD-
| licensed), or a different SSH server (which Github is of
| course free to use, since the protocol is standardized
| and their scale absolutely justifies any efforts in
| protecting their SSH host key). But GPs comment was about
| OpenSSH.
|
| Edit: Apparently OpenSSH's sshd also supports the SSH
| agent protocol for host keys, and ssh-agent does support
| PKCS#11 - so I stand corrected!
| lxgr wrote:
| > This definitely is supported. Though documentation for
| this is often HSM vendor specific [...]
|
| How can openssh documentation be vendor-specific?
|
| Or are you saying that vendors commonly provide an openssh
| fork/patchset/plugin allowing for HSM-resident host keys?
| LeonM wrote:
| > How can openssh documentation be vendor-specific?
|
| I isn't, because the cryptography is (in case of HSM) not
| handled by OpenSSH itself. So OpenSSH's configuration has
| nothing to do with the HSM.
|
| Usually, the actual cryptographic functions are not
| performed user-space, but handled by the kernel, which in
| turn can offload this to dedicated hardware. Basically if
| you compile OpenSSH for it to use kernel level
| cryptographic function, then OpenSSH can work with a HSM
| without it even knowing it.
|
| Disclaimer: this is simplified explanation, there is a
| lot more to this, and I am by no means an expert on this
| matter.
|
| Edit: meant to say kernel level cryptographic functions,
| not TLS.
| lxgr wrote:
| So you're saying that OpenSSH has an interface for that
| on the host key side?
|
| I'm aware of the PKCS#11 integration in the OpenSSH
| client and have dabbled a bit with it but was not aware
| of any server side equivalent.
|
| And how does TLS fit in here? SSH is a very different
| protocol from that, no?
|
| Update: I can't find any OpenSSH documentation of either
| (server-side) PCKS#11/HSM support or kernel-mode
| cryptography (which also in the case of Linux only
| addresses symmetric encryption to my knowledge, at least
| the mainline kernel version I know of).
|
| Maybe you're thinking of some other SSH implementation?
| The protocol definitely allows for server-side HSM usage,
| and Github at their scale is not bound to OpenSSH in any
| way.
| throw0101b wrote:
| > _I can 't find any OpenSSH documentation of either
| (server-side) PCKS#11/HSM support_
|
| OpenSSHd talks to an ssh-agent that then talks to the
| HSM:
|
| > _Identifies the UNIX-domain socket used to communicate
| with an agent that has access to the private host keys.
| If the string "SSH_AUTH_SOCK" is specified, the location
| of the socket will be read from the SSH_AUTH_SOCK
| environment variable._
|
| * https://man.openbsd.org/sshd_config#HostKeyAgent
| lxgr wrote:
| Interesting, I didn't know that OpenSSHd supported the
| agent protocol. Thank you!
| mlyle wrote:
| It's just the agent protocol, used by sshd instead of
| ssh, to make signing requests with a host key (instead of
| a user's identity key).
| lxgr wrote:
| That's cool, I wasn't aware that the server supports the
| agent protocol as well. Thank you for the pointer!
|
| It makes a lot of sense, since it avoids having to link
| the HSM/PCKS#11 stuff against sshd.
| mlyle wrote:
| Why is everyone just authoritatively dismissing this,
| when this has been supported for >7 years and is easily
| found with a google search?
|
| There is the HostKeyAgent configuration directive, which
| communicates over a unix domain socket to make signing
| requests.
|
| https://framkant.org/2017/10/strong-authentication-
| openssh-h...
|
| https://github.com/openssh/openssh-
| portable/blob/12492c0abf1...
| ammar2 wrote:
| For what it's worth, Github uses libssh
| (https://www.libssh.org/) for their ssh servers.
|
| It looks like they currently use the `ssh_bind_options_set`
| function with `SSH_BIND_OPTIONS_HOSTKEY` to set the host keys
| which means they exist on disk at some point. HSM aside, I
| believe it would be possible to use the `ssh_bind_set_key`
| and deserialize them from a secret vault so they only exist
| in the memory of the ssh daemon.
|
| Obviously they also just straight up have enough resources to
| fork the code and modify it to use an HSM.
|
| Source: looking at their ssh server portion of `babeld` in
| ghidra right now as part of hunting for bug bounties.
| mlyle wrote:
| There is the HostKeyAgent configuration directive, which
| communicates over a unix domain socket to make signing
| requests.
|
| https://framkant.org/2017/10/strong-authentication-
| openssh-h...
| mkj wrote:
| It would work with OpenSSH's HostKeyAgent option.
| tashian wrote:
| It's easy to say "should have used an HSM" (or, in truth, many
| HSMs), but I can appreciate the technical challenges of
| acutually doing that at their scale. It would not be a trivial
| project. There's a ton of operational concerns here, including
| figuring out how you would go about rotating the key on all
| those HSMs in an emergency.
| JeremyNT wrote:
| There's a lot of daylight between "use a HSM" specifically
| and "use a system that prevents junior developers from
| accessing the key and checking it into public repos."
|
| Storing the key in some kind of credential vault that can
| only be accessed from the hosts that need it at startup would
| usually be enough to prevent this particular kind of error
| (unless you're giving root on those boxes to people without
| enough sense to avoid checking private keys into git, in
| which case you've probably got worse problems).
| lxgr wrote:
| These would also need to be very distributed and high-
| throughput HSMs: You'd need to talk to one for every single
| SSH login! This is in contrast to e.g. having a CA signing
| key in a HSM, but distributing keys signed with it more
| widely.
|
| I suppose (Open?)SSH's PKI mode could support a model like
| that, but as others have noted here, this requires much more
| manual work on the user's side than comparing a TOFU key
| hash.
|
| Maybe that model could be extended to allow _TOFU for CAs_ ,
| though? But I think PKI/CA mode is an OpenSSH extension to
| the SSH protocol as it is, and that would be a further
| extension to that extension...
| tashian wrote:
| SSH CAs would make the challenge a lot easier. It sounds
| like they are using RSA keys here for the widest possible
| compatibility, and while OpenSSH's certificate support is
| not at all new, it still may be too new for this
| application.
| lxgr wrote:
| Using SSH certificates would tie every Github user to
| OpenSSH extensions though. I'm not sure if many git
| clients use something else, but it's at least worth a
| consideration.
| nwallin wrote:
| SSH uses ephemeral keys. It's not enough to have the private
| key and listen to the bytes on the wire, you have to actively
| MITM the connection. A github employee who has access to the
| private key and enough network admin privileges to MITM your
| connection already has access to the disk platters your data is
| saved on.
|
| Regarding the secrecy of the data you host at github, you
| should operate under the assumption that a sizeable number of
| github employees will have access to it. You should assume that
| it's all sitting unencrypted on several different disk platters
| replicated at several different geographically separated
| locations. Because it is.
|
| One of the things that you give up when you host your private
| data on the cloud is controlling who and under what
| circumstances people can view or modify your private data. If
| the content of the data is enough to sink you/your company
| without remedy you should not store it on the cloud.
| snowwrestler wrote:
| Agreed; GitHub documentation refers to repo "visibility," not
| "security," and that is an intentional distinction.
|
| When we signed on with GH as a paying customer over a decade
| ago, they were quite clear that private repos should not be
| considered secure storage for secrets. It's not encrypted at
| rest, and GitHub staff have access to it. It takes only a few
| clicks to go from private to public.
| shdjhdfh wrote:
| It is encrypted at rest:
| https://github.blog/changelog/2019-05-23-git-data-
| encryption...
| Perseids wrote:
| Not criticizing you, your technical correction is valid,
| but the discussion is besides the point. "Encryption at
| rest" is basically meaningless for something like GitHub.
| Not being able to pull out a hard drive in a data center
| and read it at home has been table stakes for some time.
| But how few people are able to do that anyway? A blog
| post like the above is just necessary to tick some boxes
| to comply with this or that regulation.
|
| The real question is how many services are able to access
| the data live and how many support and debug interfaces
| (indirectly) allow you to read it. Measure GitHub's
| success in securing the _secrecy_ of private repos in how
| few employees can breach it without causing alarms. Even
| without cynicism I would be surprised if it was their
| main concern. Data _integrity_ is far more important for
| code. (There are notable exceptions, of course. If
| applicable, don 't put it in the cloud!)
| ajross wrote:
| Exactly. Host keys are about authentication, not connection
| security. Presumably the upthread comment is trying to say
| that "ssh communication with github _could_ have been subject
| to an undetectable MitM attack by an attacker with access to
| this key "[1], which isn't remotely the same thing as "all
| communication with GH since the founding of the company has
| to be considered compromised".
|
| [1] Which is sort of tautological and silly, because that's
| true of all sites and all host keys. What the comment was
| trying to _imply_ was that the choice of storage of this key
| invalidates any trust we might have in GitHub /Microsoft
| regarding key management, and that therefore we shouldn't
| trust them. Which is also tautological and silly. Either you
| trust them or you don't, that's not a technological argument.
| est31 wrote:
| I would also add that your ability to pretend to be the
| client to the server is also limited, if ssh key based client
| authentication is used. This means that even if the host key
| is leaked, an attacker will not be able to _push_ in the name
| of the attacked client. The attacker will be able to pretend
| to be the server to the client, and thus be able to get the
| pushed code from the client (even if the client just added
| one patch, the attacker can pretend to be an empty repo
| server side and receive the entire repo.
|
| If ssh token based auth is used, it's different of course,
| because then the server gets access to the token. Ideally
| Github would invalidate all their auth tokens as well.
|
| The fun fact is that a token compromise (or any other attack)
| can still happen any point in the future with devices that
| still have outdated ssh keys. That's a bit unfortunate as no
| revocation mechanism exists for ssh keys... ideally clients
| would blacklist the ssh key, given the huge importance of
| github.
| adql wrote:
| You shouldn't commit unencrypted secrets to git anyway,
| public or private, on-site or in cloud.
|
| There are plenty of tools to either keep them encrypted (we
| just use simple GPG, but our team is small) or just auto-
| generate and never show to user in the first place (various
| key vaults that can be used from automation like HashiCorp's
| Vault)
| nwallin wrote:
| The person I'm replying to is arguing that their source
| code itself is the secrets they're trying to protect.
| steve1977 wrote:
| > you should not store it on the cloud.
|
| Well, at least not without encryption that is under your
| control.
| peanut-walrus wrote:
| It is so incredibly rare for public-facing service keys to be
| stored on an HSM that I don't think anyone could reasonably
| have expected this to be the case?
| killerstorm wrote:
| > any code pushed or pulled from private repositories can no
| longer be considered private.
|
| Do you realize that the code just sits on GitHub servers even
| if it's private?
|
| If you have any degree of paranoia, why do you put your code
| into GitHub?!?!
|
| Like, if you work on code which
| BlueTemplar wrote:
| Ah, I see that the men in black got there just in time ! XD
| grumple wrote:
| There's a few reasons I wouldn't worry too much:
|
| 1) Nation state level actors can probably insert or compromise
| high level staff, or multiple high level staff, at any given
| company, and perform MITM attacks fairly easily. And some could
| compel turning over your code or secrets more directly anyway.
| Not worth worrying about this scenario: nobody working on
| anything truly sensitive should be using any externally hosted
| (or even externally networked) repositories.
|
| 2) It is much more difficult for other actors to do a MITM
| attack, and if they did, they'd probably have access to your
| code more directly.
|
| 3) Your code actually isn't worth much to anybody else. Imagine
| someone launching a complete clone of HN or any other site. Who
| cares? Nobody. What makes your code valuable is that _you_ have
| it, and that you have a network and relationship with your
| customers. If somebody stole my company 's codebases, I'd feel
| sorry for them, that they are going to waste any time wading
| through these useless piles of shit. The only potential
| problems are if secrets or major vulnerabilities are exposed
| and provide a path for exploit (like ability to access servers,
| services, exposing potential ransomware attacks).
| Art9681 wrote:
| Information has different levels of value depending on what
| the user needs to do with it. It's kind of like how two
| individual pieces of "unclassified" info are...well,
| Unclassified but putting the two together as a cohesive whole
| that provides further context turns it into "classified"
| info. All it takes is a little bit of time for actors working
| with the funding and compute capacity of a major nation to
| scrape the entirety of Github, dump it in a data processing
| tool none of us know about, and make the correlations you and
| I cannot.
|
| This leak opened a time window big enough for that to happen.
| We may or may not know if it did. I doubt this info would be
| offered to the public because it would sink the business.
| rurban wrote:
| I was just going to submit this thread, thinking how in the **
| could the github host key being changed. Leading to their blog
| post explaining it. Their exposed their private RSK key!
| ssh-keygen -f ~/.ssh/known_hosts -R github.com
|
| Now it's a ED25519 with
| +DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU (they don't list this
| on their blogpost, just their renewed RSA key)
| yetanotherjosh wrote:
| That key didn't change and is listed here
| https://docs.github.com/en/authentication/keeping-your-accou...
| [deleted]
| nessex wrote:
| It's not mentioned in the blog post or keys page, but the _old_
| value[1] you'll find in known_hosts is:
| github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9
| IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUs
| yCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDE
| SU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+w
| eqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHS
| ZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ=
| =
|
| You can search for this in your codebases, hosts etc. to see if
| there are any areas that need updating. The new value is linked
| from the blog post, you can find it here:
| https://docs.github.com/en/authentication/keeping-your-accou...
|
| [1] https://github.blog/changelog/2022-01-18-githubs-ssh-host-
| ke...
| Timothee wrote:
| Good callout!
|
| I looked at my `~/.ssh/known_hosts` file and that key is
| associated with a few IP addresses in addition to github.com.
| Those lines stayed after I ran `ssh-keygen -R github.com`.
|
| I imagine that I also need to remove those other lines
| manually, but isn't that something that GitHub should have
| mentioned? I'm not sure in which circumstances these got added
| either...
| bravetraveler wrote:
| Same here, found the two following IPs with the same hostkey:
| - 192.30.253.112 - 140.82.114.3
|
| WHOIS shows github ownership, just not sure when/how/why I
| got these
| bxparks wrote:
| My ~/.ssh/known_hosts used to look like that, but now it looks
| like the host part is encrypted? It looks like:
| |1|{base64-encoded-string?}|{another-base64-encoded} ssh-rsa
| AAA{long-string}
|
| This is on Ubuntu 22.04 (Linux Mint).
| gassius wrote:
| Actually I am not seeing the fingerprint they announce in the
| blogpost
|
| When I tried to pull from my repo I got the warning message,
| right
|
| I removed the old keys with ssh-keygen -R github.com
|
| Then, trying with `ssh -T git@github.comp` I see this
|
| The authenticity of host 'github.com (140.82.121.3)' can't be
| established. ECDSA key fingerprint is
| SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM.
|
| So, first thing of course, is that the fingerprint does not match
| the one in the blogpost which is
| SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s
|
| Second problem, is that ip 140.82.121.3 seems to be reported as
| HIGH RISK[0]
|
| So basically, how should I proceed? I am not security expert but
| I would say I am not an illiterate on this, and I have no idea. I
| guess the majority of users would you accept the new key, but is
| this the right move? I would need to do it if I want to do some
| work, that is for sure
|
| EDIT: Formatting
|
| [0]https://www.ipqualityscore.com/free-ip-lookup-proxy-vpn-
| test...
| nneonneo wrote:
| You're looking at the fingerprint of the ECDSA key; only the
| RSA key was replaced (and only the new RSA key's fingerprint is
| in the blogpost). Check
| https://docs.github.com/en/authentication/keeping-your-accou...
| for the full list: the key you're seeing is listed, so you
| should be fine.
| gassius wrote:
| Thank you. It makes sense
| [deleted]
| rjmunro wrote:
| They really should add all the keys to the blogpost so that
| people can check quickly that the new key is correct.
| jnsaff2 wrote:
| p2Q is there https://docs.github.com/en/authentication/keeping-
| your-accou...
|
| Not in the blog post but the blog post points to my link which
| is the official documentation.
| gassius wrote:
| Thank you, yes, I missed the ECDSA vs RSA key part. Makes
| sense now
| DangitBobby wrote:
| Try adding all the keys shown in the docs here to your
| known_hosts.
|
| 1. https://docs.github.com/en/authentication/keeping-your-
| accou...
| gassius wrote:
| And sorry to reply to myself but this is not great at all
|
| I manually added the new RSA SSH public key entry to my
| known_hosts file (like they say in the blogpost)
|
| Then ran ssh -T git@github.com and got
|
| Warning: Permanently added the RSA host key for IP address
| '140.82.121.3' to the list of known hosts. Hi gassius! You've
| successfully authenticated, but GitHub does not provide shell
| access.
|
| Then, when trying a git pull, I got this:
|
| Warning: the RSA host key for 'github.com' differs from the key
| for the IP address '140.82.121.4' Offending key for IP in
| ~/.ssh/known_hosts:63 Matching host key in
| ~/.ssh/known_hosts:64
|
| So basically, the Offending key is the one I added manually as
| per blogpost?
|
| Ok, I am in Europe, and this seems like an issue of global
| distribution network or something, but this is not great AT
| ALL, either the blogpost information is not complete or
| something fishy is going on
|
| UPDATE: The replies makes clear what I was seeing those errors
| and make sense. Thanks
|
| EDIT: Formatting and Acknowledge of the situation per replies
| aftbit wrote:
| Just in case anyone is curious, you can get the current github
| host keys from their HTTP API[1]. Our deploys fetch the latest
| and add them to a known_hosts file as part of the process. We set
| the `GIT_SSH_COMMAND` environment variable to use a temporary
| known hosts file with just the needed keys on each deploy. We do
| it in fabric with Python, but here's similar bash:
| curl -s https://api.github.com/meta | jq -r '.ssh_keys |
| join("\n")' | sed 's/^/github.com /' > /etc/github_known_hosts
| GIT_SSH_COMMAND="ssh -o
| UserKnownHostsFile=/etc/github_known_hosts" git pull
|
| 1: https://api.github.com/meta
| groestl wrote:
| Not directly related, but SSH certificates are one of the most
| underutilized features of SSH.
| nightpool wrote:
| Any idea which public repository it got exposed in? What
| happened, and how did they notice?
| yababa_y wrote:
| my guess is the copilot X AI is threatening escalation and was
| forced into demonstrating its capabilities. ;)
| yetanotherjosh wrote:
| Please before replacing your local fingerprint with the new one,
| double check it is the expected value. This is an opportune time
| for man-in-the-middle attackers to strike, knowing everyone has
| to replace their stored signatures, and that some will be lazy
| about it with a blind "ssh-keygen -R github.com" command.
| bityard wrote:
| SSH host certs would make this a non-issue, and I've often
| wondered why GitHub doesn't use them.
| brabel wrote:
| I've updated the key in known_hosts, then was able to connect
| successfully.
|
| What do I have to do to ensure I connected to the right
| server?? I thought just making sure the correct RSA key was in
| known_hosts would be enough?
| snorremd wrote:
| That is enough, given that you've fetched or compared the key
| from a trusted GitHub.com server.
| nirimda wrote:
| It depends on how you found out what the new key value is. By
| the sounds of your description, you're fine. But in principle
| there's more than one way people could proceed from here.
|
| If you read the blog post on a verified domain and saw the
| new key and updated manually, or you deleted the known key
| and verified the key fingerprint when it warned you about an
| unknown key, you should be good to go. Here, you trust the
| people who issue TLS certificates and you trust github to be
| in control of their domain name, so you can be reasonably
| confident that the key they advertised on their website is
| the correct key. If your internet connection was compromised,
| you would have got an error message when you connected to
| https://github.blog (because they wouldn't have a certificate
| from a trusted certificate issuer) or when you connected to
| the git server (because they wouldn't have the key you just
| trusted).
|
| If you saw the blog post and then removed the old key and
| told ssh to save the new key it's receiving without checking
| it matches the value on the webpage, you might have a
| problem. The connection to github's ssl could have been
| compromised, and if you just accepted whatever it told you,
| you have no trusted intermediate to verify that the key is
| trustworthy. All you know is that each time you connect to
| github's server hereafter, you're either connecting to a
| server with the same key (no error), or you're connecting to
| one that doesn't have the same key (error message). But
| whether you can trust that key? You don't know that. You just
| know it's the same key.
|
| But even if you did the latter, all is not lost. You can look
| at the known_hosts file (on Linux and MacOS it's
| ~/.ssh/known_hosts) and check the fingerprint. If it's what
| they advertise, then you're good to go. If it's different,
| you should fix it and find people who can help you deal with
| a security incident.
|
| The reason people are raising a flag is that today, lots of
| people will be rotating their key. That means if you're
| looking to target someone, today is the day to do it. Even if
| 90% of people do it the proper way, by manually verifying the
| key, that still means there's going to be a lot of people who
| could be victimised today.
| defanor wrote:
| Here are the expected fingerprints (since they don't publish
| those via SSHFP RRs):
| https://docs.github.com/en/authentication/keeping-your-accou...
| SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s (RSA)
| SHA256:br9IjFspm1vxR3iA35FWE+4VTyz1hYVLIE2t1/CeyWQ (DSA -
| deprecated)
| SHA256:p2QAMXNIC1TJYWeIOttrVc98/R1BUFWu3/LiyKgUfQM (ECDSA)
| SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU (Ed25519)
| cwillu wrote:
| Note the MITM here :)
|
| We humans really aren't cut out for this, are we.
| darthrupert wrote:
| What MITM? What are you talking about?
| fulafel wrote:
| The poster of the fingerprints is in the middle, you are
| not getting them from GH if you use them instead of going
| to the linked page.
| vxNsr wrote:
| This is literally a man in the middle between you and
| GitHub.
| coryfklein wrote:
| Why downvote this person! The parent post left _plenty_
| of ambiguity in their comment. Are they saying that an
| actual MITM _attack_ is happening? That the fingerprints
| shared are actually the _wrong_ ones?
|
| Generally speaking, one would not consider an internet
| comment directing folks to GitHub's actual SSH
| fingerprints a "man in the middle" as the phrase in this
| context usually has a negative implication, where in this
| case defanor is in fact simply mirroring the actual
| information that GitHub has officially posted in a way
| that is much more helpful than yetanotherjosh's "double
| check it is the expected value". For most of us idiots,
| we don't know what the expected value is!
|
| So thank you defanor for sharing, and thank you
| darthrupert for asking for clarification. Y'all
| contributed to educating myself and others and now we
| know more because of it.
| darthrupert wrote:
| Ah, okay. I thought this was obvious that the keys in the
| comment were just for show, and if anyone would need the
| actual keys, they would be looked via the GH link anyway.
|
| Good clarifications everywhere, yes.
| nicky0 wrote:
| If someone wanted to trick HN users into trusting a
| phoney key, one way to do that would be to post the
| phoney fingerprint on HN claiming it to be the real one.
| misnome wrote:
| I mean, yes, but you'd also have to have a way to
| actually MITM the person you are targeting via HN
| comment, before anyone pointed out it was wrong. It'd be
| much easier to just use the MITM you already have and not
| raise the suspicion of posting in a comment.
| nicky0 wrote:
| Don't overthink this.
| pferde wrote:
| And if someone would actually fall for this, they deserve
| to be fired, and/or never allowed anywhere near anything
| related to computer security. :)
| iso1631 wrote:
| And within a few seconds someone will have called this
| out in a reply
| macintux wrote:
| Assuming the person doesn't have some back door access to
| HN as well.
| detrites wrote:
| Or they don't simply wait a while and edit it when it's
| not under high scrutiny.
| defanor wrote:
| Indeed, at least for verification. I didn't mean for HN
| users to trust those, but perhaps should have warned about
| it: copied the fingerprints primarily for people searching
| on this page, so that they can follow the link to GitHub
| (and rely on PKIX to build a trust chain). I did `ssh-
| keygen -R github.com` myself, and saw the ECDSA key's
| fingerprint while connecting (which wasn't mentioned in the
| linked post, and wasn't on this page either), so figured it
| would be somewhat helpful for others following the same
| route.
| cesarb wrote:
| On the other hand, this is a nice TOFU-style double check.
| The first time HN user "defanor" went to that page, these
| were the fingerprints; if later someone somehow invades the
| github documentation server (or somehow MITMs your HTTPS
| connection to it), and changes the fingerprints there, they
| will no longer match the ones saved in the comment above.
| NelsonMinar wrote:
| Well, "defanor" _says_ these were the fingerprints.
| Perhaps they are the MITM.
|
| (Not genuinely concerned about this risk, but
| "Reflections on Trusting Trust" reverberates.)
| yosito wrote:
| > double check it is the expected value
|
| Not all of us are familiar enough with the SSH protocol to
| understand how to "double check the expected value"? Where can
| I determine what the expected value should be?
| Gravyness wrote:
| Run "ssh -T git@github.com" command.
|
| It should error like this:
| @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
| @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
| @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
| IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
| Someone could be eavesdropping on you right now (man-in-the-
| middle attack)! It is also possible that a host key
| has just been changed. The fingerprint for the RSA
| key sent by the remote host is
| SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s.
| Please contact your system administrator.
|
| Note that the SHA256 present there matches perfectly the one
| github send. If you don't remember the very first time you
| connected to github you also had to accept the key. The
| warning above shows up because the server is saved as a
| different RSA, for the SSH client it seems that someone setup
| a server for github but has a different key, which could mean
| someone is trying to trick you into connecting into the wrong
| server. This could also mean that github changed their RSA
| key which is why they published this article.
| pbhjpbhj wrote:
| The fingerprint is a hash of the key, so in theory -- say
| with a quantum computer -- I could create a key that's
| different and provides a hash-collision. Is that right?
|
| It would just take many ages of the universe, at present,
| to calculate a collision, right?
| 8organicbits wrote:
| There's a narrow window for that attack. The fingerprint
| is only used on the first connection, for manual
| verification. Any later connections would check the
| ~/.ssh/known_hosts which has the full public key.
|
| If you somehow can MITM an SSH connection on the first
| connection, you can probably use any key. Most people
| don't check the fingerprint.
|
| But you are correct, computing an SSH key with a
| collisionwis expected to take an infeasible amount of
| time/energy with current understanding of crypto and
| available computers.
| yetanotherjosh wrote:
| A key part of avoiding MITM is to get the values from an
| authoritative origin, not comments on HN, so the link is
| here:
|
| https://docs.github.com/en/authentication/keeping-your-
| accou...
|
| Yes, this assumes the github-hosted docs and your SSL
| connection to them are not also compromised, but it's far
| better than not checking at all.
| Dylan16807 wrote:
| Look for the part of the article that says "the following
| message"
|
| Or the parts below it about updating and verifying in other
| ways.
| [deleted]
| p-e-w wrote:
| It never fails to amaze me how most incident mitigations seem
| completely oblivious to such security side effects.
|
| "We have no reason to believe that the exposed key was abused,
| but out of an abundance of caution, we are going to expose 50
| million users to a potential MITM attack unless they are
| extremely careful."
|
| Not a single word in the post about whether this impact was
| even considered when making the decision to update the key.
| Just swap the key, and fuck the consequences. Same with the
| mass password resets after a compromise that some services have
| done in the past years. Each of those is any phishing
| operation's dream come true.
| mihaaly wrote:
| Is there a benefit (and practicality) in recording encrypted
| traffic by an adverse intermediary waiting for keys being
| exposed sometime? Like now?
| KAMSPioneer wrote:
| No, the risk of losing an SSH host key is less this
| (because of forward secrecy), rather impersonation of the
| server.
| justeleblanc wrote:
| I'm always amazed at this kind of posts. Did these 50 million
| users (surely none of them use git+https!) check the host key
| the first time they connected to github? Did _you_?
| 1317 wrote:
| > (surely none of them use git+https!)
|
| well, yes
|
| github doesn't accept https push any more
| akerl_ wrote:
| Not sure what you mean. You can push via HTTPS:
| https://docs.github.com/en/get-started/getting-started-
| with-...
|
| Maybe you're referring to how they no longer accept
| passwords for HTTPS auth? You have to auth for HTTPS push
| with a personal access token.
| tracker1 wrote:
| That's funny, I do it every day. It's frankly easier to
| install git credential manager (even integrate into WSL)
| for 2FA authentication on Github (and other git hosts).
|
| I get a bit paranoid when having to deal with Tokens on
| various CI/CD environments as it stands. And the things
| that start breaking every year when I forget to update
| them. Note: this is personal/hobby projects, not
| corporate stuff, where I'm strictly in the codebase and
| try to keep my fingers out of CI/CD beyond getting a
| Docker image built, and someone else configures the
| keys/auth.
| akerl_ wrote:
| How are you using git credential manager for 2fa on
| GitHub? They stopped supporting user/password auth for
| HTTPS git access a while back, and started requiring
| personal access tokens (which do not require a 2nd
| factor)
| tracker1 wrote:
| GCM will use an embedded browser so you can authenticate
| with the UI including your second factor, which will then
| give you a credential/token that can be used in the git
| environment over HTTPS. It's still a (differt, oath vs
| reference generation) token, but you aren't having to go
| generate, configure and update it yourself.
| arianvanp wrote:
| It doesn't matter because it didn't change! That's the
| beauty of TOFU.
| msm_ wrote:
| The point is, what if it you were MITMed from the
| beginning?
| emn13 wrote:
| Sure, but the difference is that it's now both a
| plausible moment to go MITM (because they got that key),
| and furthermore the hypothetical attacker now has good
| reason to believe users won't be scared by a host-key-
| change warning, and the hypothetical attacker would know
| this opportunity exists for a large set of users
| simultaneously. If some malicious network operator were
| to try and exploit users, now would be a good moment -
| they'd likely catch many more people in the time it takes
| to be discovered than on an average day.
|
| The MITM-at-the-start risk is of course real, but I think
| this new everyone -restarts-simultaneously risk is
| qualitatively different enough to be worth at least
| considering.
| lxgr wrote:
| Much more concerningly, there is an activated-by-default
| OpenSSH extension (`UpdateHostKeys`) that allows the
| server to _install new host keys into `.ssh /known_hosts`
| after every successful server authentication_.
| tlb wrote:
| The bad guys would also have to have MITMed it every time
| I connected for the last 15 years, or I would have seen
| authentication failures when it connected to the real
| thing. MITMing someone once isn't that hard, but doing it
| consistently is.
| arianvanp wrote:
| It doesn't matter because it didn't change! That's the
| beauty of TOFU.
|
| One way to solve this in TOFU is to have a time window
| where both keys are presented.
| justeleblanc wrote:
| If we're starting from the assumption that the first key
| was compromised, then you're still vulnerable to MITM.
| The only solution is communicating the key through a
| different, trusted way. Which is exactly what github did
| - inasmuch you can trust that github.com is github.
| BHSPitMonkey wrote:
| How is there an alternative here?
| ithkuil wrote:
| The alternative would be to use certificate authorities
| (ssh has CA support) which allow to effectively have
| private keys at different levels and allow you to keep the
| root private key in a physical vault and use it very rarely
| to issue other private keys
| datadeft wrote:
| This would just offload the problem to a separate entity.
| CAs can be (and have been) compromised.
| ithkuil wrote:
| Sure, but isn't it more likely that a key that has to be
| shared by who knows how many ssh load balancer machines
| at GitHub and can't be easily rotated because it's pinned
| by millions of users, isn't it more likely that that
| private key gets eventually compromised or thought to be
| at risk at being compromised?
|
| We need to compare the relative risks within the same
| context, namely within a company like GitHub
|
| So it's not relevant to bring up failures of other CAs
| terom wrote:
| And then don't forget to setup key revocation as well,
| and make sure that an attacker in a position to MITM the
| connection cannot cause the revocation checks to fail-
| open.
|
| I hope you don't need that SSH connection to fix your
| broken CRL endpoint!
| pid-1 wrote:
| Clone repos using oauth2 with two factor enabled - both
| GitHub and GitLab support that though their CLIs.
| michaelt wrote:
| Jump into a time machine, go back to the creation of SSH,
| and adopt SSL-style trusted third-party certificate
| authorities. Somehow get it adopted anyway, even though
| loads of people use SSH on internal networks where host-
| based authentication is difficult; SSH is how many headless
| machines are bootstrapped; and that you've got to do it 19
| years before Lets Encrypt.
|
| Jump into a lesser time machine, go back to when Github
| were creating their SSH key, and put it into a hardware
| security module. Somehow share that hardware-backed
| security key to loads of servers over a network, without
| letting hackers do the same thing. Somehow get an HSM that
| isn't a closed-source black box from a huge defence
| contractor riddled with foreign spies. Somehow avoid vendor
| lock-in or the HSM becoming obsolete. Somehow do this when
| you're a scrappy startup with barely any free time.
| nibbleshifter wrote:
| Ssh certificate authorities are a thing that exists.
|
| We also have a way to put SSH host key fingerprints in
| DNS records already.
| Arch-TK wrote:
| Yes but the option to do verify host keys using
| ("VerifyHostKeyDNS") is not enabled by default.
| lxgr wrote:
| This just kicks the can down the road to DNS.
|
| I'd guess that most systems aren't using DoH/DoT or end-
| to-end DNSSEC yet. Some browsers do, but that doesn't
| help tooling frequently used on the command line.
|
| I suppose you could just accept X.509 certificates for
| some large/enterprise git domains, but that pokes up the
| hornet's nest that is CA auditing (the browser vendors
| are having a lot of fun with that, I'm happy that the
| OpenSSH devs don't have to, yet).
|
| And where do you maintain the list that decides which
| hosts get to use TOFU and which ones are allowed to
| provide public keys? Another question very ill-fitted for
| the OpenSSH dev team.
| tptacek wrote:
| No browser uses DNSSEC.
| lxgr wrote:
| That was in reference to the former, i.e. in-browser
| DoH/DoT lookups.
| prussian wrote:
| Unless it has changed recently, you can't have a trust
| chain of OpenSSH certs though so it's cumbersome that
| your signing key is not only the root ca but also
| basically has to be 24/7 accessible to sign any
| server/client you want to bring up.
| datadeft wrote:
| Yeah like how HTTPS CAs exist. There are some very nice
| three letter ones who can issue any certificate and your
| browser / OS happily accepts it.
| BenjiWiebe wrote:
| SSH doesn't have any CAs that it trusts out of the box.
| It's up to you to tell it which one to trust.
| roblabla wrote:
| DNS can _trivially_ be mitm 'd. DNS-stored fingerprints
| are strictly less secure than TOFU.
| tialaramex wrote:
| If you use DNSSEC (cue inevitable rant from Thomas) this
| just works. If you have DoH (and why wouldn't you?) and
| your trusted resolver uses DNSSEC (which popular ones
| do), you get the same benefits.
|
| https://en.wikipedia.org/wiki/SSHFP_record
| marcosdumay wrote:
| > and adopt SSL-style trusted third-party certificate
| authorities
|
| So that any large entity can own your servers with easy.
| (Well, they already can, but not through this
| vulnerability.)
|
| Anyway, the only thing CAs do is to move that prompt into
| another, earlier time. It's the same prompt, the same
| possibility for MITM, and the same amount of shared trust
| to get wrong. You just add a 3rd party that you have to
| trust.
|
| SSH does have a CA system. Anybody that isn't managing a
| large datacenter will avoid it, for good reason.
| michaelt wrote:
| _> So that any large entity can own your servers with
| easy._
|
| Eh, let's not pretend existing SSL certificate validation
| is anything to write home about.
|
| Even _without_ any ephemeral servers involved, barely
| anybody is validating cert fingerprints on first use.
|
| And among people using ephemeral servers, 99% of
| applications have either baked a certificate into their
| image (so that any compromised host means a compromise of
| the critical, impossible-to-revoke-or-rotate key) - or
| every new server gets a new cert and users have either
| been trained to ignore certificate warnings, or they've
| disabled strict host key checking and their known hosts
| file.
|
| The existing SSL cert validation options are perfect if
| you're a home gamer or you're running a few dozen bare
| metal servers with all your SSL users within yelling
| distance in the same office. But we all know it's a joke
| beyond that.
| [deleted]
| tinus_hn wrote:
| There could be an update to the protocol that enables
| certified keys to be used and allows them to be accepted
| without warning or with less of a warning.
|
| There could be a well known URL that enables people to
| fetch ssh keys automatically in a safe manner.
| ansible wrote:
| There isn't an alternative, really. The private key has
| been exposed, and presumably it is unknown if or how far it
| has spread. The SSH keys _must_ be changed, and the sooner
| the better. All that can be done is to notify people after
| the change has occurred.
| p-e-w wrote:
| > and presumably it is unknown if or how far it has
| spread
|
| Why would that be unknown? GitHub has HTTP and SSH access
| logs, right?
| tgsovlerkhgsel wrote:
| Since the post doesn't mention anything like "we reviewed
| logs and confirm the key was not accessed", it is very
| likely that they either don't have logs that are reliable
| enough to rule it out (e.g. they may be sampled or
| otherwise incomplete), or that the key was accessed.
| renonce wrote:
| Keeping a complete log of all GET requests to random
| files in a public repository in a reliable way would be
| insane.
| otoolep wrote:
| No, it wouldn't be - assuming by "insane" you mean "silly
| to do". I build systems at Google that do exactly that.
|
| Whether it's worth the cost is a decision each company
| makes. Also, you don't need to keep the log _forever_.
| Max of a few weeks retention would be common.
| pbhjpbhj wrote:
| Presumably, keeping 'last remotely accessed' and 'last
| remotely modified' for every file (or other stats that
| are a digest of the logs) is sane for pretty much any
| system too. Having a handle on how much space one is
| dedicating to files that are never viewed and or never
| updated seems like something web companies that have
| public file access would all want?
| est31 wrote:
| It's not just GET requests. Someone could have
| cloned/refreshed the repo using ssh. The repo might have
| been indexed by github's internal search daemon which
| might not use the public HTTP API but uses internal
| access ways however those might look like. You might have
| purged the database of that daemon but what about backups
| of it? What about people who have subscribed to public
| events happening in the github.com/github org via the
| API?
|
| You'd have to have logging set up for all of these
| services and it would have to work over your entire
| CDN... and what if a CDN node crashed before it was able
| to submit the log entry to the log collector? You'll
| never know.
| blitzar wrote:
| One H4X0R gave it to four friends, who in turn gave it to
| between 9 and 14 friends, who in turn gave it to between
| one and 6 friends.
|
| If train A leaves New York going 60 miles per hour with
| 167 people on board and train B leaves Chicago one hour
| later with 361 people on board going 85 miles per hour,
| how many people now have the key?
|
| The answer is 31337.
| marcosdumay wrote:
| Well, at least SSH could allow for signing a new key with
| the old one. So they could say it's signed, and people
| would know to accept only a different prompt.
|
| There is DNS verification, but people have been trained all
| their lives to accept insecure DNS information (and set
| their systems accordingly), and I really doubt the SSH
| client checks the DNSSEC data.
| rakoo wrote:
| That's why you need certificates and not just a key pair.
| Certificates make key rotation easier, and you _want_ key
| rotation to be easy.
|
| I guess the proper way forward is a small utility that gets
| the latest signature through http+tls, and replaces the
| line in your known_hosts file, all in the background.
|
| Looking long term, maybe we need to get rid of all the
| security stuff in ssh and just pipe the rest of its
| functionalities inside a TLS pipe. Let the os do its
| certificate management, reuse security bricks that are way
| more studied, ...
| oleganza wrote:
| Certificates just add more keys to worry about. The
| beauty of SSH is that it does not add hugely trusted
| parties in the name of convenience, while the UX of TOFU
| (trust on first use) is pretty decent.
|
| The real solution to break out of these UX/security
| tradeoffs is to put domain names on a blockchain: then
| you can simply rotate the key in your DNS record, while
| the blockchain model is such that you need to compromise
| many parties, instead of "one out of many parties", as
| with CAs.
|
| Tracking Bitcoin chain for DNS updates is lightweight
| enough that it can be built into OS alongside other
| modern components such as secure enclave, TCP/IP stack
| and WiFi/BT/5G radios.
| hnfong wrote:
| > The beauty of SSH is that it does not add hugely
| trusted parties in the name of convenience
|
| Even with a certificate authority model, you don't have
| to trust any CAs if you don't want to. Not having the
| option to do so is more of a problem.
| datadeft wrote:
| We should use a separate system that could reliably
| verify which certs belong to which entity.
|
| Blockchain is a perfect solution to this. I wonder why it
| is not considered yet.
| jtsiskin wrote:
| https://certificate.transparency.dev it is
| datadeft wrote:
| Thanks, I was not aware!
| megous wrote:
| Those keys can be worried about on a better secured
| computer, and don't need to be spread out on every
| frontend ssh server. Also it allows you to have each
| machine have a different host key pair, so if one leaks,
| only that single machine may have some trust issues, and
| not the whole fleet.
|
| Also it's way better than TOFU, you can just add the CA
| key to known_hosts and avoid TOFU for each machine.
|
| (Nevermind that you'll probably not accidentally commit
| some semi-ephemeral host key that's rotated often
| somewhere, because it will not be some special snowflake
| key you care about, but something handled by your
| infrastructure software automatically for each machine)
| leetrout wrote:
| Are there any cert solutions that dont involve having to
| maintain a revocation list? I only used certs with
| openvpn years ago and the CRL was a potential footgun.
| goalieca wrote:
| This is one reason people are issuing certs with 2 week
| expiry.
| marcosdumay wrote:
| > Certificates make key rotation easier
|
| How easy is it to rotate the keys of your CA?
| p-e-w wrote:
| Same as with any other decision: Do a cost/benefit analysis
| of whether the security risk created by rotating the key is
| actually outweighed by the security risk of doing nothing,
| taking into account logs that should tell you whether the
| exposed key was indeed accessed by unauthorized parties.
|
| To be 100% clear: _Both_ courses of action come with
| associated security risks. The problem is not choosing one
| course of action over the other, the problem is thinking
| you can just skip the cost /benefit analysis because the
| answer is somehow 'obvious'. It's not obvious at all.
| johngalt_ wrote:
| No, you cannot keep using an exposed key. You must
| replace it. There is no cost/benefit analysis needed in
| this situation.
| p-e-w wrote:
| Wrong. A CBA is always needed. If the potential damage
| from MITM attacks made possible by rotating the key is
| greater than the potential damage from a rogue key
| _multiplied by the likelihood that someone actually
| accessed the key,_ then it is wrong to rotate the key. It
| 's that simple.
|
| The only way a CBA would be unnecessary is if rotating
| the key didn't have any security risks. But it does.
| hnlmorg wrote:
| There is a MITM risk regardless of whether they rotate
| the key. Except one is a one time risk and the other is a
| perpetual risk.
|
| Thus rotating is the only logical course of action.
| p-e-w wrote:
| Only if you know for certain that the key has been
| accessed by a third party.
|
| If you don't know for certain, you have to factor in the
| likelihood that it has been, and at that point, the two
| risks aren't equal anymore so that logic doesn't work.
| unionpivo wrote:
| What if you don't know for certain ?
|
| You just ignore it and hope for the best ?
|
| Only if you are certain (and better be really sure you
| haven't missed any cache/cdn, temp files backus etc.) it
| wasn't accessed you do nothing.
| lazide wrote:
| It was publicly exposed, and if they are making this
| announcement it's essentially guaranteed they can't rule
| out it was accessed.
| msm_ wrote:
| Are you arguing for the sake of arguing and technical
| correctness or do you actually believe Github shouldn't
| rotate their key in this situation?
| eyelidlessness wrote:
| Here I'll do the CBA:
|
| - if they have evidence that the key was exposed to one
| person, even with zero usage of the key, failing to
| rotate the key is tantamount to knowingly accepting
| widespread compromise at a potential attacker's whim. At
| GitHub's scale, that's untenable.
|
| - rotating the key is the only correct reaction to that
|
| - they should have better communications in place to help
| users mitigate MITM
|
| - there really isn't an option, because they're critical
| infrastructure; I'm glad they know that and acted
| accordingly
|
| - on principle this speculation makes sense, but
| understanding the threat makes it moot
|
| - you hopefully know that, and it's good to insist on
| thoughtful security practices but it's important to also
| understand the actual risk
| tomp wrote:
| Don't trust corporate PR. They're obviously lying when they
| say "out of an abundance of caution". The private key was
| exposed in a public GitHub repo, it could literally be
| anywhere.
|
| So MITM for _some_ of 50m users is strictly better than MITM
| for _all_ of 50m users.
| the_other wrote:
| It could be, but also GH might be logging inbound requests
| long enough to see whether the file was requested.
| new2yc wrote:
| After reading first paragraph, I was sure they don't have
| any specific reason to replace it.
|
| > leaked in public repo
|
| Me: Yeah, that's why they are doing it.
| [deleted]
| theteapot wrote:
| > The private key was exposed in a public GitHub repo.
|
| How do you know this?
|
| Github runs scanner for private keys in public and private
| repos and notifies owner (I did it once so I know ... ;)).
| So some Github engineer likely would have received such an
| email if what you say is true. Hilarious.
| 3np wrote:
| From the OP:
|
| > This week, we discovered that GitHub.com's RSA SSH
| private key was briefly exposed in a public GitHub
| repository.
| Veen wrote:
| It says exactly that in the article:
|
| > This week, we discovered that GitHub.com's RSA SSH
| private key was briefly exposed in a public GitHub
| repository
|
| Hilarious.
| theteapot wrote:
| I didn't read it fully before commenting. I'm sorry.
| adrr wrote:
| I can't see that could ever happen. Is the key just
| floating around on their employees computers and skeins
| accidentally committed it?
| roblabla wrote:
| Maybe because the article says so?
|
| > This week, we discovered that GitHub.com's RSA SSH
| private key was briefly exposed in a public GitHub
| repository
| dheera wrote:
| How would one stage a MITM attack without knowing the private
| key corresponding to the old key?
| Karliss wrote:
| The key has changed, meaning that every user has to accept
| a new key.
|
| Meaning that a lot of users will blindly accept whatever
| new key (even when it might be the one owened by attacker
| doing MITM) because Github, their college or random person
| on internet said that that's what you have to do to get rid
| of error.
| tialaramex wrote:
| > Meaning that a lot of users will blindly accept
| whatever new key (even when it might be the one owened by
| attacker doing MITM)
|
| This is less likely because unlike for TOFU the SSH
| client just rejects the mismatch and insists you take
| manual action, and the likely manual action will be
| "Paste this stuff from the announcement".
|
| So an adversary needs to either subvert whatever
| messaging you see (which is tricky, likely impossible for
| a random user visiting the github web site wondering
| what's wrong) or hope that you just try to muddle along
| and do TOFU again, putting you in the same spot as a
| whole bunch of users every day at GitHub.
| iso1631 wrote:
| Fortunately this will become evident once they connect
| from elsewhere and the key changes _again_
| macintux wrote:
| Not strongly evident. I suspect most users would assume
| they did something wrong, or that GitHub was still making
| changes.
| stouset wrote:
| They don't need it. Millions of users are going to blindly
| trust the "new" GitHub public SSH key they see the next
| time they connect without checking to see if it matches the
| published signature.
| p-e-w wrote:
| By pretending to be the host that the user is trying to
| connect to. You can then present the client with a key you
| generated yourself. Of course, SSH will warn the user that
| the fingerprint has changed, but they'll just think "Ah
| yes, GitHub changed their keys so it's probably fine." This
| is why updating the key creates a potential MITM risk,
| unless people actually bother to verify that the
| fingerprint is correct.
| cogogo wrote:
| What specifically can you verify that cannot also be
| spoofed? If I go do this now I (and I'm sure millions of
| others) have no idea what to look for. I'd be blindly
| accepting like a sheep if it weren't for this thread.
|
| edit: I found this helpful and honestly had no idea I
| should be doing this (I'm a hobbiest not a professional)
| https://bitlaunch.io/blog/how-to-check-your-ssh-key-
| fingerpr...
| ZiiS wrote:
| if you run a MITM attack today the victim gets the warning
| in the blog post. Thier most likly action is to google the
| blog post and see it is expected and and so accept your
| fake key. Having said that I dont see what choice Github
| had, they can't continue to use a leeked key.
| ddtaylor wrote:
| Their solution is the second thing in the blog post which
| is demand into your known host file.
|
| The problem here is that their first command that they
| advise using is the one that removes the old key and most
| users are just going to stop right there because it
| solves the problem of getting the key warning.
|
| The right solution here is to provide a command that most
| users are going to copy and paste that deletes the old
| key and adds the new key all at once.
| edp wrote:
| The fact that users have to delete the old Github key from
| their systems and accept a new one is what could lead to a
| MITM attack.
|
| If your system doesn't know the public key of an SSH
| server, when you connect the first time, the SSH client
| will display a warning and ask you if you accept the server
| key. An attacker could be between you and Github and if you
| accept without checking it's the correct key, you would be
| toast.
| pbhjpbhj wrote:
| Would it be more secure to access a https secured server
| to get the keyfile then?
| dheera wrote:
| Only if the https server cert wasn't compromised at the
| same time as the ssh key. For all we know, this entire
| announcement of "we have a new key" could be staged.
| tialaramex wrote:
| Yes, GitHub's announcement provides the correct new
| public RSA key, and it also provides instructions for a
| curl invocation which does all the work if you don't
| trust yourself to copy-paste text or don't understand
| how.
| faeriechangling wrote:
| While their reaction is more likely to cause a security
| breach, consider the psychology.
|
| If the key was breached and Github just didn't know it, then
| a breach happened, then only Github would be to blame.
|
| If Github rotates its key, and somebody suffers a MITM
| attack, the blame is more diffuse. Why didn't they verify the
| key out of band?
| datadeft wrote:
| Certificate pinning check built in when?
|
| We should have a blockchain for certificates btw. That would be
| such an amazing solution to this problem. You could advertise
| ahead of the time that you are changing certificates and we
| could verify that it was in fact you.
| rpigab wrote:
| Double-check with what source? The one mentionned in
| docs.github.com?
|
| I assume it's safe because the SSL cert for docs.github.com is
| probably not compromised, so it's giving us the right key, and
| compromising docs.github.com would be extra effort and is
| unlikely to happen.
|
| However, I wonder what kind of steps an MITM attack would have
| to perform, I assume one of the easiest would be compromising
| my local DNS server, since regular DNS does not offer a high
| level of security, then github.com resolves to the attacker's
| IP and the attack works. Do you have examples of such attacks
| that don't involve a virus already active on the end user's PC?
| Maybe if someone owns an IP previously owned by Github that is
| still somehow advertised as being Github by some DNS lagging
| behing?
| 8organicbits wrote:
| This is always a concern with SSH as it uses trust on first
| use. The first time you connect it shows you the fingerprint
| for out of band verification. That manual step is on you to
| perform, but most people skip it. Future visits check against
| the saved fingerprint.
|
| The best practice is to verify the fingerprint out of band
| using a secure channel. In this case, that's HTTPS and
| docs.github.com. If (hypothetically) docs.github.com was also
| compromised, then you don't have a secure channel.
|
| https://en.m.wikipedia.org/wiki/Man-in-the-middle_attack has
| some MITM examples.
| tgsovlerkhgsel wrote:
| They provide convenient commands to import the correct keys. It
| would probably be better to only include the block that
| contains both the -R and the update command, but at least they
| do provide them.
| Timwi wrote:
| Info to everyone using Windows and as confused as I was:
|
| * When you run a git command and git tells you about the
| mismatching key, it will prompt you to enter `y` or `n` but not
| actually let you do so.
|
| * You need to run `plink github.com` to get the same prompt. You
| can then enter `y` to accept it.
|
| * You should only accept it if it says the new fingerprint is
| d5:2c:63:d9:bc:75:9d:de:b1:4e:36:28:9f:7a:9c:39. If it is any
| different, you may be targeted by an attacker.
|
| * The new key is stored in the file `C:\Users\<your
| name>\\.ssh\known_hosts`. The new key at the time of this writing
| is `github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgc
| QnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+P
| KCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqlt
| V9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyR
| kQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wq
| FPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4d
| hPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+Ejqo
| TwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdB
| XA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk=`.
|
| * In the future, if the key ever changes again, you can look it
| up on GitHub's SSH key fingerprints page [0].
|
| [0] https://docs.github.com/en/authentication/keeping-your-
| accou...
| omoikane wrote:
| plink is part of PuTTY right? In which case I think the host
| keys are in the registry, under
| HKEY_CURRENT_USER\SOFTWARE\SimonTatham\PuTTY\SshHostKeys
| fareesh wrote:
| yikes
| pquerna wrote:
| Would it of been possible for Github to use Host-key rotation
| instead of hard breaking it?
|
| https://lwn.net/Articles/637156/
|
| I'm honestly not familiar with anyone actually using host-key
| rotation?
| robbat2 wrote:
| 1. IIRC UpdateHostKeys does not _remove_ the old key, so it
| would still be there, lurking (I haven 't checked the code).
|
| 2. It was only added in OpenSSH 6.8, so it missed Ubuntu 14.04
| release, and only really turned up in 16.04 LTS that way,
| plenty of old systems it wouldn't work on.
|
| As other posters noted, a bad actor _could_ rotate the key to
| their chosen keys just as easily as GitHub could cause the
| rotation.
| dlgeek wrote:
| If an attacker has access to the private key, they could use
| the Host-key rotation feature to migrate you to an attacker-
| controlled key instead, as the old key is trusted. So, GitHub
| needs everyone to forcibly untrust the old (exposed) key.
| pquerna wrote:
| Yeah, but... shouldn't Github of rotated their keys over the
| last decade?
|
| I mean it seems like its clearly a key that wasn't in an
| HSM.. and over the lifetime, hundreds? Thousands of Github
| employees could of accessed it?
| p1mrx wrote:
| Congrats, you just used "would of", "should of", and "could
| of" in a single thread.
| kelnos wrote:
| The problem with rotating this particular private key is
| that it's incredibly disruptive. Everyone who uses GH will
| see a big scary message from ssh saying the host key
| changed and something malicious might be going on. A
| majority of those people probably won't have seen a blog
| post announcing the change beforehand.
|
| Anyone who's baked the host key in the known_hosts file
| that gets shipped on their CI systems would start to see
| jobs failing, and have to manually fix it up with the new
| host key.
|
| These things are just annoying enough that I think it's
| perfectly understandable that GH doesn't want to regularly
| rotate this private key.
| computerfriend wrote:
| The point of host-key rotation is that you can avoid the
| disruption of the former.
| rocqua wrote:
| Host-key rotation would enable the attacker to continue, but
| the attacker could be detected simply by diligent people
| monitoring the github key they use.
|
| The current rotation allows anyone to try to fish the lazy
| users (like me probably) who will just trust on first use.
| Probably a bigger risk than key compromise, since they have
| logs.
|
| It could be a better idea to use Host-key rotation, despite
| it making the life of a key-thief a bit easier. Just because
| it exposes people less against opportunistic impersonators.
| rasengan wrote:
| I literally freaked out and came to Hacker News looking for an
| answer. Bullseye.
|
| Here's the list of keys if anyone needs:
|
| https://docs.github.com/en/authentication/keeping-your-accou...
|
| > Host key for github.com has changed and you have requested
| strict checking.
|
| > Host key verification failed.
| AtNightWeCode wrote:
| So what actions am I supposed to take to verify that I am on the
| safe side? The blog post did more to downplay the problem than
| actually being helpful for the end users.
| archgoon wrote:
| Once took a security course where the prof accidentally opened
| his rsa key file while streaming.
| benatkin wrote:
| In this case I think "security" belongs in quotes.
| archgoon wrote:
| [dead]
| xaphod wrote:
| I saw once a support person from $COMPANY_I_WORK_IN ask a
| client to send their private key in email, so they could
| troubleshoot something. I pressed "Reply all" and called it
| out, and was yelled at later.
___________________________________________________________________
(page generated 2023-03-24 23:01 UTC)