[HN Gopher] The Story of the SolarWinds Hack
___________________________________________________________________
The Story of the SolarWinds Hack
Author : bokchoi
Score : 212 points
Date : 2021-04-17 09:42 UTC (13 hours ago)
(HTM) web link (www.npr.org)
(TXT) w3m dump (www.npr.org)
| netfortius wrote:
| This article is a case of [a lot of] Monday morning
| quarterback[s]. Except for Mandia, I wouldn't allow any other
| exec(s) to speak about this, publicly, as to the why and how.
| Side note: I bet Bejtlich wishes he was still in that team ;-)
| 1cvmask wrote:
| It's nice how they equivocate over the ease of entry and their
| security policies:
|
| There was another unsettling report about passwords. A security
| researcher in Bangalore, India, named Vinoth Kumar told NPR that
| he had found the password to a server with SolarWinds apps and
| tools on a public message board and the password was:
| "solarwinds123." Kumar said he sent a message to SolarWinds in
| November and got an automated response back thanking him for his
| help and saying the problem had been fixed.
|
| When NPR asked SolarWinds' vice president of security, Brown,
| about this, he said that the password "had nothing to do with
| this event at all, it was a password to a FTP site." An FTP site
| is what you use to transfer files over the Internet. He said the
| password was shared by an intern and it was "not an account that
| was linked to our active directory."
| frombody wrote:
| The password was in a file that was committed to github IIRC.
|
| I think both the intern that posted the file and the person
| making that statement both did not realize that someone had
| given the user write access.. which in my opinion was the
| actual mistake, not the ftp or the password.
| ta9999 wrote:
| Something to remember is that these sorts of things happen in
| many organizations.
|
| You should always verify the data you get, be careful with
| complex supply chains, and avoid binaries you didn't build
| yourself. Don't skip these things just because "that's the
| way it's always been done." 10 years ago very little internet
| traffic was encrypted, security still means progress not the
| status quo.
| dogman144 wrote:
| How a Vp security can ignore a privesc risk like that is pretty
| inexcusable. Ever vuln falls on a risk mgmt spectrum but that's
| a really nonsense answer to give. Weak PW mgmt on a FTP server
| that you let interns set should raise some areas of interest.
| CyberRage wrote:
| You don't get this kind of attack because you had an exposed
| FTP server.
|
| The attack implanted malicious code into their code, learning
| the tooling, process and responsibilities of the personal.
|
| They then reversed engineered the protocol and used it in
| their backdoor to look basically the same as regular
| communications.
|
| The issue is that we blindly trust 3rd party software that is
| used by hundreds of companies. this makes SolarWinds a prime
| target, one that is worth the efforts taken in this case.
| boomboomsubban wrote:
| >You don't get this kind of attack because you had an
| exposed FTP server
|
| This kind of attack needs an entry point, and an exposed
| FTP server provides the potential for one. Whether it
| actually was the entry point is a separate matter,
| willfully ignoring one unlocked door means there's likely
| to be others.
| CyberRage wrote:
| Initial access is part of the day-to-day these days.
|
| you can't cover all entry points, it's a matter of time
| for someone to make a mistake. the fact that the
| adversary showed these extreme levels of proficiency and
| dedication tells me that the vast majority of companies
| would have fallen for that. In fact, the backdoor was
| running for months on targets like Microsoft, gov
| agencies, security companies like Malwarebytes.
|
| These companies know a thing or two about security.
|
| Today we work with "assume breach" mentality that assumes
| you are already compromised.
| ridaj wrote:
| I don't think they ignored it? Says it was addressed. I think
| it was OK to dispel the implication that the elaborate supply
| chain attack was allowed in the first place by sloppy pw
| practices. You don't want that to be the takeaway and then
| other companies thinking, well our pw management practice is
| really good so we don't have to worry about being a victim so
| much
| rjzzleep wrote:
| They did ignore it. I think it was over a year actually. I
| don't feel like looking it up right now but it was also
| discussed in another submission. Whatever the actual number
| was it was VERY long.
| de6u99er wrote:
| Anyone know how the software update was actually compromised in
| the first place?
| lambdasquirrel wrote:
| It's kinda-sorta in the article. The hackers managed to put
| some hack into the Solarwinds build scripts.
|
| Nice writing I guess, but, what allowed them to get into the
| build scripts?
| matwood wrote:
| The wiki page on the attack speculates an Office360 account was
| hacked. Presumably it was an account from an admin, and from
| there I could see them probing until finding credentials for
| the build system.
| trashcan wrote:
| No idea if it is related, but a SAML implementation security
| issue was disclosed the same (or very close) day that the
| SolarWinds attack became public knowledge. Maybe that gave
| them access to the admin account?
| trulyme wrote:
| So they hacked MS through SW which was hacked through MS?
| Ironic if true.
| tptacek wrote:
| _Network monitoring software is a key part of the backroom
| operations we never see. [...] By its very nature, it touches
| everything -- which is why hacking it was genius._
|
| This is frustrating to read, since plenty of people did in fact
| warn that these kinds of systems were easy targets.
| bostonsre wrote:
| Yea.. not sure I'd call it genius. Think if you ask anyone that
| is a little knowledgeable about what would be the juiciest
| target for a nation state to hack, a large portion of people
| would have said something like SolarWinds.
|
| It seems like SolarWinds should have known better themselves as
| well. There is no way that their upper management didn't know
| that they would be an amazing target for a hack. Supply chain
| attacks are not that new. Their lax security seems extremely
| negligent.
| tptacek wrote:
| Ninety-nine times out of a hundred, defenders call attacks
| "genius" as a way of subverting accountability. What makes
| this particular incident pernicious is that it already had a
| built-in deflection of accountability --- the responsibility
| for ensuring that SolarWinds was fit for purpose was diffuse;
| hundreds of giant companies with large security teams all
| believed it was someone else's job to verify that SolarWinds
| could safely deliver its functionality.
|
| I've worked with people who don't operate this way, and who
| take continuous flack from CIOs for spending resources on
| verification for COTS IT management tools. But those teams
| are, in my experience, very rare --- and the SolarWinds hack
| provides further evidence of that view.
|
| It's not a perfect predictor, but a reasonable rule of thumb:
| if you've never heard of a vendor's security team, chances
| are they barely have one. That's obviously true of... most
| vendors! So you should be careful when you select one for a
| role as sensitive as fleetwide agent-based monitoring, where
| a vulnerability or a software supply chain fuckup is going to
| create mass compromise. This seems so clear to me that it
| barely counts as insight.
| comboy wrote:
| I was just playing with Grafana a few days ago, their cloud
| version. When you install agent, it opens up ports with
| unprotected metrics on public IP. Opens up ports on your
| production without info about it whatsoever, because it is in
| theory a push agent. Why would you do that?
|
| Support response:
|
| "Regarding your second message about port 12345 on the Grafana
| Agent -- the HTTP server only exposes the Agent's internal
| metrics and provides an API for the Agent's status. The gRPC
| server is used for agents to communicate with one another, if
| the scraping service is used. It does not allow anyone to get
| access to their metrics.
|
| That said there's we understand the concern and upon review
| will look at documenting that agents listen on 0.0.0.0 by
| default, and that you can change it by setting
| http_listen_address and grpc_listen_address in the Agent's
| server config to 127.0.0.1: (...)"
|
| Also, sometimes I feel like I'm the only person in the world
| who is not comfortable in running tons of untrusted docker
| containers. If I put a link to a binary here and tell you to
| run it, I doubt any HN reader would. But 400MB of binaries? No
| worries, I have just the right tools to run them as root.
| adolph wrote:
| How fortuitous is it that a months long investigation can be
| published right when the US announces sanctions? Great job
| National Radio!
|
| _Like razor blades in peanut butter cups_ , says CrowdStrike.
| covidthrow wrote:
| I fear that you're being downvoted for pointing out the even
| bigger threat of our national media's exceedingly more
| dangerous acts of propaganda, presumably because readers either
| don't recognize it or are wilfully blind to it because it
| reflects their own biases.
|
| The SolarWinds hack was a devastating attack on our
| sovereignty.
|
| And so is the other.
| encryptluks2 wrote:
| The SolarWinds hack in addition to election interference
| should easily be seen as an attack worthy of taking out Putin
| IMO.
| askl56 wrote:
| Yes, let's assassinate a world leader of the country with
| the most nukes in the world based on unfounded claims of
| election interference and a hack.
|
| Do you have any idea what America did in Russia in the
| immediate aftermath of the fall of the Soviet Union? The
| 1996 election in Russia which was majorly "interfered" with
| by Clinton: https://archive.is/R7i5u, not to mention the
| fact that most Russian hacking activities are done using
| NSA backdoors which were leaked?
|
| I'm surprised and disappointed to see such flagrant ivory
| tower imperialism on HN.
| SV_BubbleTime wrote:
| > By design, the hack appeared to work only under very specific
| circumstances. Its victims had to download the tainted update
| and then actually deploy it. That was the first condition. The
| second was that their compromised networks needed to be
| connected to the Internet, so the hackers could communicate
| with their servers.
|
| Yea, wow, thanks NPR. Hard hitting stuff right there. Those are
| "very specific circumstances" that just happen to generally
| apply to a huge percentage of hacks.
|
| I normally appreciate some stories on NPR, but you're right.
| This is a narrative piece.
| meowface wrote:
| That's a perfectly valid paragraph. In a decent environment,
| outbound internet access should be restricted to only the
| hosts / networks / ports that require it. Especially for
| server environments. Many servers running the backdoored
| Orion probably tried to beacon but failed for that reason.
| (And I'd assume the backdoor would probably first verify
| outbound internet access so that the failed beacon doesn't
| generate a firewall/ACL deny event that a security team might
| detect.)
|
| Plus, the article is written to condense technical
| information into something that's as layman-friendly as
| possible. The specific malicious update has to be downloaded,
| and also installed, and also running on a server which can
| reach out to anything on the internet. Their point is that
| there are only going to be so many servers that both use this
| software and meet those conditions, and that's in part why
| the backdoor took so long to identify. This is maybe a little
| obvious to people with infosec knowledge, but definitely not
| obvious to their target audience.
|
| The article timing is interesting, but I don't think a
| coincidence is that unlikely. If you read the whole article,
| it covers enough that I could see it taking months to make.
|
| I don't think coordination with the government is that
| unlikely, either, or perhaps just a pragmatic editorial
| decision ("everyone knows sanctions are likely going to be
| placed sometime in the next few months, and maybe we should
| wait until then so we can include those details in the
| story"). Both of those scenarios are more likely than a
| coincidence, probably - but, either way, I think your post
| seems overly cynical in general.
| lanstin wrote:
| I whitelist my networks (by port and host name, At least
| until TLS 1.3 removes the visible SNI) and my top denies
| list is very interesting. They have access to bits and
| pieces of the internet (especially PyPi damn you runtime
| downloads) and all connections to an allowed port will
| succeed (they will just be closed after the SNI or IP check
| fails).
|
| Nonetheless the article was correct the hack did not need
| bizarre or rare circumstances to take effect.
| trulyme wrote:
| Wait... What do you mean by PyPi runtime downloads???
| meowface wrote:
| Not exactly sure what they meant, but maybe just that
| it's often hard to install a single Python package if the
| machine can't reach PyPI. If you download a tarball or
| wheel of some Python package on a different computer and
| transfer it to the isolated computer, there's a pretty
| good chance the package is going to have at least one
| third-party sub-dependency, in which case trying to
| install it with pip or setup.py will cause it to try to
| install more packages from PyPI.
|
| Docker's a decent way to address this, since you install
| the dependencies when building the image rather than when
| running the container, so you can build the image
| elsewhere or through CI and run the container on any
| server without having to allow access to package
| management repo servers.
| ridaj wrote:
| I agree with the parallels with aviation regulation, there needs
| to be something forcing a supplier's hand to solve this. The way
| to protect against supply chain attacks is to invest in a
| security-hardened build system (eg don't build releases on dev
| workstations, do them on build farms by build software that is
| the only thing able to access the release signing keys). This
| costs too much for most companies, so if they don't have the
| obligation to build it, they'll do features instead.
| Godel_unicode wrote:
| > do them on build farms by build software that is the only
| thing able to access the release signing keys
|
| You're aware that this is exactly what solarwinds did, right?
| ridaj wrote:
| Well and then harden _that_ obv
| Godel_unicode wrote:
| Which they did. The problem of hardening your build
| infrastructure against someone who has admin access for
| months is... non-trivial.
|
| This boils down to the question of should average companies
| be including the Russian intelligence services in their
| threat model? To paraphrase James Mickens great USENIX
| paper, if your threat model includes the SVR, you're going
| to be SVR'd upon.
| Veserv wrote:
| First, you do not need to be the Russian intelligence
| services to pull off this attack. Given prevailing trends
| in the vulnerabilities market this sort of attack would
| cost at most $1M to pull of which puts it within the
| capabilities of maybe ~50,000,000 _individuals_ worldwide
| let alone organizations. If the SVR is anything like the
| CIA they are probably running at least 1,000 programs of
| similar scale simultaneously, so it is not as if the
| attack was supported by the full weight of the Russian
| intelligence services.
|
| Second, a company's threat model should include entities
| that want to attack them. Given that they are claiming
| the SVR wanted to and did attack them, it would be
| ridiculous to not include them since that would be
| empirical evidence that they are an actual threat actor.
| Even if we were to ignore empirical evidence any company
| like SolarWinds that sells to wide swaths of government
| agencies in critical capacities should absolutely be
| including foreign intelligence services in their threat
| models and should probably be required to demonstrate
| effectiveness against attacks funded to at least the
| $100M level since only at that level does it start to
| actually get problematic for state actors to run
| operations.
| kerenua wrote:
| how are you smart
| SuchAnonMuchWow wrote:
| pro tip: don't get hacked in the first place, it will avoid
| trouble down the line ! /s
| amaccuish wrote:
| > But as CrowdStrike's decryption program chewed its way through
| the zeroes and ones, Meyers' heart sank. The crime scene was a
| bust. It had been wiped down
|
| That's a lot of words to say, we don't know who did it. I had a
| quick look but couldn't find anything, why are the fingers being
| pointed at Russia?
| ipsin wrote:
| Tool use, essentially. Of course that could be spoofed, but I
| think that's the origin of the claim.
|
| https://www.reuters.com/article/us-global-cyber-solarwinds/s...
| joe_the_user wrote:
| Tools can be stolen. In fact, if we're claiming whoever did
| this was a super-genius, they would have stolen or spoofed
| the tools they used to point at someone else. Unless they
| were Russia being so clever they were pretending to be
| someone pretending to be Russia!
|
| Edit: Your link show Kaspersky labs making the claim that
| this was the FSB. Yet the West also claims Kaspersky is
| controlled _by_ FSB! Well, you could say "they should know".
| Or maybe they want to humor the West so their ban will be
| lifted. Or maybe they aren't controlled by the FSB at all.
| But if the West can't figure that out, how do they expect to
| figure out the true origin of the hack.
|
| "A riddle wrapped in mystery, inside an enigma"
| sorokod wrote:
| _" The tradecraft was phenomenal"_
|
| Indeed, consider Figure 5 here [1]. A truly diabolical
| mastermind.
|
| But seriously, the article looks like window dressing for common
| incompetence.
|
| [1]
| https://www.microsoft.com/security/blog/2020/12/18/analyzing...
| andix wrote:
| I'm quite sure there are a lot of attacks like that. Most of them
| just never get noticed.
|
| The best backdoors are those, which are never found.
| boomboomsubban wrote:
| Here's an excellent article on the topic, this is a huge uproar
| on an everyday occurrence.
|
| https://blog.thinkst.com/p/if-nsa-has-been-hacking-everythin...
| makomk wrote:
| Like, say, that backdoor someone wrote an article about
| recently which ran from RAM and had a sophisticated self-
| destruct mechanism that erased all traces if anyone tried to
| dump its memory? I wonder how many companies had exploits like
| that which they either didn't notice or didn't have the
| sophistication to actually catch and dump.
| tkinom wrote:
| There are defences for this: If one controls/monitors for
| every app in system for network access, as soon as any
| unusually network access are triggered, it is investigated
| and block.
|
| In my home windows setup, only windows defender, firefox and
| chrome are allowed out going internet access in regular base.
| Everything else are blocked.
|
| Windows update are only allowed when I in the mood for it
| (~once a year). Anyone can do this easily by control
| srvhost.exe 's internet access with windows firewall app.
| CyberRage wrote:
| Well, usually highly targeted attacks are orchestrated with a
| reason.
|
| You want to leverage your access to perform actions because you
| got get revealed\blocked even not intentionally.
|
| Once you act, let's deploy some ransom, wipe some data, shut
| down power plants, you will be shown.
|
| Keeping a perfectly stealth backdoor for years as environments,
| software and personal change is extremely difficult.
| cyberlab wrote:
| > The routine update, it turns out, is no longer so routine
|
| Is there the rare case that we shouldn't update because the
| update could contain a malicious payload? If the update gets
| served over plaintext HTTP I would treat it as suspicious and may
| even block it from connecting at all. I run the risk of having
| outdated software, but that can be addressed by storing the
| software in a machine that's not connected to The Internet in any
| way, so it can't really do anything/talk to a C2 server (if
| someone does decide to execute an 0day with the software or
| inject malicious code via a rogue update).
| [deleted]
| covidthrow wrote:
| Plaintext transport doesn't matter if at least _one_ part of
| the payload chain is cryptographically protected /verified.
|
| If you have a machine that's air-gapped and its only IO is
| strictly humans (read: keyboard/screen, _not_ USB or other
| electronic means) then your weak point is the human, so center
| your security around that. You can look at security of lottery
| machines to get a good idea how that 's handled.
|
| But if you're updating the machine with updates, then it
| doesn't really fit that criteria, soooo....
| trynton wrote:
| @cyberlab: "Is there the rare case that we shouldn't update
| because the update could contain a malicious payload?"
|
| You don't ever update your security "computers" from some
| third-party outsourcer. What you do is have your own people
| constantly probing their own systems for potential
| vulnerability and patching it themselves.
|
| * jeez ..SolarWinds run their stuff on FTP and "active
| directory". It's got to be a joke.
|
| * "computers" .. not allowed to use the 'W' word ;]
| trynton wrote:
| Don't put remote control software* on your security
| infrastructure such that when a third party gets hacked, your
| whole network is exposed.
|
| * The presence of such software making it that much more easy to
| hijack your "computers"
|
| * Please "Hacker News", no more anti-Russian neocon propaganda.
| Who's really to blame is the idiot that put that configuration
| in, in the first place.
| arminiusreturns wrote:
| Let me introduce you to PTECH and see if you still think
| Solarwinds was worse. Warning, a conspiracy rabbit hole lies this
| way, proceed with caution, lest your view of the world be
| challenged.
|
| A start: https://www.youtube.com/watch?v=UuZRMpt_Tas&t=195
| matkoniecz wrote:
| Is there something about that available in a serious form? (not
| a movie, especially Youtube movie)
| EMM_386 wrote:
| > Is there something about that available in a serious form?
| (not a movie, especially Youtube movie)
|
| Wall Street Journal:
|
| https://www.wsj.com/articles/SB1039184086357188113
|
| Wikipedia:
|
| https://en.wikipedia.org/wiki/Ptech
| arminiusreturns wrote:
| edit: The topic level link in the youtube description sends
| you straight to the list of documentation...
|
| 1. http://en.wikipedia.org/wiki/Ptech
|
| 2. https://archive.org/details/GunsNButterIndiraSinghPtechAnd
| Th...
|
| 3. http://masshightech.bizjournals.com/masshightech/stories/2
| 00...
|
| 4. http://archive.is/RuleH
|
| 5. http://web.archive.org/web/20080905224929/http://www.theam
| er...
|
| 6. http://web.archive.org/web/20080820045652/http://www.total
| 91...
|
| 7. http://web.archive.org/web/20080515202659/http://www.judic
| ia...
|
| 8. http://web.archive.org/web/20081015113454/http://911citize
| ns...
|
| 9. http://web.archive.org/web/20080915185702/http://counterte
| rr...
|
| 10. http://rememberingmichael.wordpress.com/2008/03/20/in-
| memory...
|
| 11. http://www.abovetopsecret.com/forum/thread183761/pg1
| matkoniecz wrote:
| > hand-hold on an important topic
|
| I assume that things available only as youtube video,
| unavailable in a text form are unimportant.
|
| (not all things in text form are important, but it is a
| nice filter that basically always works)
| arminiusreturns wrote:
| I think that is a very limited and closed mindset that
| inherently precludes a lot of first hand source
| information.
|
| something about leading a horse to water I suppose
| lanstin wrote:
| The first and most important skill needed for the
| internet to be a net gain for a persons understanding is
| the ability to quickly develop and modify heuristics to
| filter out bad data. Filtering good data is fine, the
| truth repeats itself in many formulations, but believing
| bad data or even spending too long noticing it is bad and
| you are worse off than just using a library and books.
| thanks4linx wrote:
| thx for the linx haters gonna hate
| tomComb wrote:
| Given that audio and video are much less likely than written
| works to include references and are harder to reference
| themselves, I think they are poor evidence of anything
| contentions such as a conspiracy theory.
| miohtama wrote:
| How much value SolarWinds shareholders have lost because of this?
| If it is not number one incentive for investors to fix, then
| there won't be change in business practices. This is why GDPR in
| the EU has (some) teeth.
| seereadhack wrote:
| How might you compartmentalize admin access the whole way down
| the stack at the enterprise level? What if you were to start from
| scratch?
| williesleg wrote:
| Yay china and russia!
| airstrike wrote:
| To me the "worst nightmare" was a story in the NYT about a
| hypothetical concerted attack against healthcare infrastructure,
| transit and more. Sadly I can't seem to find the link but it was
| a few years ago...
| vmh1928 wrote:
| Not so hypothetical.
|
| https://www.nytimes.com/2020/12/03/us/politics/vaccine-cyber...
|
| That's from December. Still going on.
|
| https://www.reuters.com/article/us-health-coronavirus-vaccin...
| germinalphrase wrote:
| "A 'Worst Nightmare' cyberattack" that we all... just take in
| stride? Either the consequences are themselves clandestine, or
| cyberattacks aren't as meaningful as our headlines would
| indicate.
| exporectomy wrote:
| The worst nightmare of the vice president of security at
| SolarWinds, not of the average person.
| zaphar wrote:
| An attack directed at your own govt is potentially a
| nightmare for the average individual. If the wrong
| information is stolen it could be used much farther down the
| road. Your govt may find itself at a disadvantage at a
| critical moment.
|
| I'm a sense it's only not a nightmare if you aren't paying
| attention.
| dogman144 wrote:
| Clandestine I think. Immediate reaction steps to this from CISA
| were pretty unprecedented; a govt-wide unpluggening on a Sunday
| night of a specific vendor doesn't happen a lot.
| PeterisP wrote:
| We can compare and contrast with the effects of NotPetya,
| which caused widespread obvious economic damage (e.g. Maersk
| shipping and Merck losses) - due to the number of affected
| companies, Solarwinds had the potential to be worse, but I'm
| not sure if you can be more destructive than that without it
| being obviously visible.
| germinalphrase wrote:
| Perhaps the damage is just not visible yet. The sand has
| not been tossed in the gear box. The blue prints have not
| been built. Maybe it's a precursor event to a longer
| decline.
| chevill wrote:
| >I'm not sure if you can be more destructive than that
| without it being obviously visible.
|
| I don't know if it the damage was greater than NotPetya but
| you definitely can have something more destructive without
| it being immediately apparent. If you lose credit card
| numbers and PII from your customers you HAVE to report it
| to the public but there are different rules for the loss of
| incredibly valuable intellectual property.
| rst wrote:
| The Biden administration just announced sanctions against the
| Russian government for their (presumed) responsibility for the
| SolarWinds attack. They're not shrugging this one off.
|
| https://www.theverge.com/2021/4/15/22385371/russia-sanctions...
|
| (Which is itself a bit odd. The US has argued in other contexts
| for "cyber norms" which would allow pure espionage operations,
| but put more restrictions on attacks. And so far as anyone can
| tell so far, SolarWinds was a pure espionage operation -- using
| tools that could be repurposed to do something else, but you
| could say that about a lot of operations in this sphere,
| including US operations that our government wants everyone
| _else_ to shrug off. Yet here are the sanctions. I expect the
| "norms" push, to the extent that the current administration
| still wants to pursue it, will take some kind of a hit...)
___________________________________________________________________
(page generated 2021-04-17 23:00 UTC)