[HN Gopher] Don't Wanna Pay Ransom Gangs? Test Your Backups
___________________________________________________________________
Don't Wanna Pay Ransom Gangs? Test Your Backups
Author : parsecs
Score : 84 points
Date : 2021-07-19 21:14 UTC (1 hours ago)
(HTM) web link (krebsonsecurity.com)
(TXT) w3m dump (krebsonsecurity.com)
| whartung wrote:
| "Test your backups" is so easy to say, but quite difficult for
| many to do. There are a lot of shops that probably don't know how
| to recreate a machine from scratch. How many systems are
| developed as balls of clay. Little bits added and smeared in over
| time until the ball just gets bigger, but each piece lost in the
| process. How many folks can go through their local config files
| and explain all of entries, how many can even tell which ones
| they have changed, or why? Especially when they were changed by
| Frank, but he left 2 years ago.
|
| You'd like to think you can just restore the system from backup
| and it'll just light back up. But how do you test this without
| cratering your existing system? Like a boat in a basement, many
| system are built in-situ and can be very rigid.
|
| Modern environments like cloud computing and creation scripts can
| mitigate this a bit organically, but how many of these systems
| are just a tower running Windows w/SQL Server and who knows what
| else? Plus whatever client software is on the client machines.
|
| How do you test that in isolation?
|
| At least read the media to see if it can be read (who doesn't
| love watching a backup tape fail halfway through the restore).
|
| Simply, it takes a lot of engineering to make a system that can
| be reliably restored, much less on a frequent basis. And this is
| all engineering that doesn't impact the actual project -- getting
| features to users and empowering the business. Which can make the
| task even more difficult.
| xwdv wrote:
| This is like a sales pitch for just paying the ransom.
| MattGaiser wrote:
| It probably works more often than not.
| shyn3 wrote:
| It works until it doesn't, such as in a hardware failure.
|
| [1] https://www.anandtech.com/show/15673/dell-hpe-updates-
| for-40...
| tw04 wrote:
| This is all solved, it just takes money and typically bringing
| in outside experts. Occasionally it will require changes to
| apps but most of the time it can be retrofit.
|
| No it isn't easy, but it's also not an impossible task.
| LinuxBender wrote:
| Don't just test your backups. Make sure your automation can't
| clobber or tamper with your backups. This includes both local and
| disaster recovery sites. Give your pen-test team super-user privs
| on your automation and give them Amazon gift cards if they can
| tamper with your backups. If they can't mess with the backups,
| give the gift cards to whoever designed and hardened your
| infrastructure.
| renewiltord wrote:
| Which organizations currently do this?
| wizzwizz4 wrote:
| Why not actual money? Amazon gift cards leak metadata to
| Amazon, and can only be used to buy stuff from Amazon.
| jagged-chisel wrote:
| And they support Amazon.
| wizzwizz4 wrote:
| Oh, of course. Can't believe I forgot the biggest reason.
| LinuxBender wrote:
| Good point. Cash bonus and maybe RSU's if they company is
| public.
| jonas21 wrote:
| It used to be that you could give employees gift cards up to
| a certain amount as awards and it would not be considered
| taxable income (but I believe that's no longer the case).
| jagged-chisel wrote:
| Any gift(s) up to a total value of ... $13k? -ish? I don't
| know what the limit is now. Google's cafeteria is (was?
| depending on that limit...) an example of how to benefit
| employees without causing the employee additional tax.
| dmoy wrote:
| Setting aside the gift card bit (addressed in above
| comment), $13k sounds way too high. Like two orders of
| magnitude too high.
|
| From irs.gov
|
| > Whether an item or service is de minimis depends on all
| the facts and circumstances. In addition, if a benefit is
| too large to be considered de minimis, the entire value
| of the benefit is taxable to the employee, not just the
| excess over a designated de minimis amount. The IRS has
| ruled previously in a particular case that items with a
| value exceeding $100 could not be considered de minimis,
| even under unusual circumstances.
|
| Which about matches with what I've seen at BigCo.
|
| $40 box of tools as a gift? Did not show up on my
| paycheck.
|
| $150 electronic device as a gift? Showed up on my
| paycheck.
| mikeyouse wrote:
| That's a different issue - IRS clamped down on gift cards
| and non-cash compensation that used to be considered de
| minimis. Now most employers gross up and report any gift
| card type gift over ~$5.
|
| https://www.irs.gov/government-entities/federal-state-
| local-...
| bentcorner wrote:
| I think logistically its easier for a team within an org to
| spend "their" money on gift cards for intermittent activities
| and hand them out as necessary. Getting stuff onto the actual
| payroll is probably more complicated.
| edoceo wrote:
| Hey Payroll, edoceo needs an off-cycle bonus of $$$.
|
| Your manager should be able to write a similar email.
| Volundr wrote:
| At least at the company I recently left, this kicks off
| an approval process within both the HR and accounting
| departments. Meanwhile an Amazon purchase (and thus an
| Amazon gift card) is something I could put on my card and
| expense, or approve someone else doing myself.
|
| I get it doesn't make sense, but that's corporate America
| for you.
|
| That said, be careful of the gift card route. Depending
| on the amount you can find yourself in the wrong side of
| the IRS that way.
| decebalus1 wrote:
| If your disaster recovery process isn't tested, you actually
| don't have any disaster recovery. It's not only about 'how long
| it takes' it's also about whether or not it works at all. Can you
| rebuild from scratch? What happens if your entire infrastructure
| goes down at the same time? What happens if a datacenter you rely
| on just disappears? What happens if you lose access to your
| systems? Can you lose access to your systems? IMHO one of the
| only silver lining of these attacks is that organizations are
| starting to ask these questions more often.
| slownews45 wrote:
| There is another approach. Scrub old data you don't need.
|
| 2-3 year email retention on corp email.
|
| Paper files for sensitive client info (or don't keep it).
|
| We can reinstall office / windows / active director etc.
|
| Mandatory 2FA on google suite?
|
| Git codebases on github etc for LOB apps (we can rebuild and
| redeploy).
|
| We use the lock features in S3 for copies of data that must be
| kept. Not sure I can even unlock to delete as account owner
| without waiting for timeouts.
| PhantomGremlin wrote:
| There's been a lot of good advice here about backups and disaster
| recovery.
|
| But there's also a lot of other stuff to consider:
|
| Compartmentalization. Finance and Engineering and Sales only need
| to interact in limited ways. How about some firewalls between
| them, limiting types of access?
|
| Location isolation. Why does something that happens in Peoria
| affect Tuscaloosa? Once a ransomware gang breaches a perimeter,
| why is it allowed countrywide (or worldwide) access to a company?
|
| Monitoring. Aren't there tools that can alert on various
| anomalous patterns? All of a sudden, gigabytes of data start
| being exfiltrated? All of a sudden, processes fire up on a
| multitude of servers? Monitoring these things is hard to do at
| scale, but surely possible?
|
| Microsoft. In 2002, Bill Gates "Finally Discovers Security". How
| much longer will Microsoft be given a free pass? How many more
| "critical" vulnerabilities will their software have?
| https://www.wired.com/2002/01/gates-finally-discovers-securi...
|
| I could go on and on. But why should I? Why can't MBA-type CEOs
| take IT seriously? Why can't they hire competent people and fund
| them and listen to them?
| blooalien wrote:
| > ... "and listen to them?"
|
| That's the part I've always had trouble gettin' out of most
| "management" types. They hire you for your expertise, and then
| undermine it at every opportunity to "save money" or to exert
| their "authoritah".
| jerhewet wrote:
| Isn't this more of a "We don't want our client / customer
| information released to The World At Large" question? I would
| think most business entities have backups of some kind (Scripps
| being the only exception I can think of), and will pay the ransom
| to keep any sensitive information off the market.
|
| Edit: Should have added that I find it hard to believe that
| companies have PB of data backed up. I could believe GB, and
| maybe even TB, but PB is pretty hard to swallow. The past three
| companies I've worked for (25 year span) had, at most, a couple
| of gigs of sensitive information that couldn't be easily
| replicated.
| nelgaard wrote:
| I also find it hard to believe that a ransomware gang could
| encrypt 50 Petabytes without anyone noticing it. It would also
| take some time to decrypt 50 petabytes if you paid the
| criminals and got the key.
|
| And would you trust you data after criminals had access to it?
| stan_rogers wrote:
| Ransomware attacks rarely indicate any data leakage; all they
| usually do is prevent _you_ from accessing your own data (by
| encrypting your drive with a key you don 't have access to).
| intothev01d wrote:
| O rly? https://www.trendmicro.com/vinfo/us/security/news/cybe
| rcrime...
| runnerup wrote:
| These days attacks labeled "ransomware" in the news seem to
| be hybrid attacks. There usually is sensitive data
| exfiltration in addition to encrypt-in-place.
| djrogers wrote:
| The current trend is double-extortion ransomware attacks -
| encrypt your copy of your data, and threaten to release it
| publicly as well.
|
| [1] https://www.cybereason.com/blog/rise-of-double-extortion-
| shi...
| throwawaysleep wrote:
| How does one learn how to do proper backups? Using my throwaway
| as I suspect my company doesn't do them (and even if they do, I
| don't know where they are or what to do with them as the main
| engineer left on my piece of software).
| Severian wrote:
| 3-2-1 Backup Rule:
|
| Three copies of your data. Two "local" but on different mediums
| (disk/tape, disk/object storage), and at least one copy offsite.
|
| Then yes, absolutely perform a recovery and see how long it
| takes. RTOs need to be really low. Recovering from object storage
| is going to take at least a magnitude more time than on-prem.
|
| Also, storage snapshots/replications are not backups, stop using
| them as such. Replicating is good for instant failover, but if
| your environment is hacked they are probably going to be
| destroyed as well.
| Waterluvian wrote:
| I'm a novice and am dealing with data that isn't too complicated,
| large, or important. My approach is to build restore directly
| into the normal workflow. I test my backups by using them each
| week.
|
| A stack is spawned from a database backup and once it passes
| tests, replaces the previous one.
|
| Not sure how smart this all is but my goal is to learn through
| application.
| TheDong wrote:
| The main reason I think this normally isn't done is that it
| requires downtime to do safely most of the time.
|
| In order to not lose data, you can't have any writes between
| the time when the backup was taken and the present, or you need
| code which reconciles additional state and adds it onto the
| backup before switching over.
|
| Normally, backup restoration is done during a maintenance
| window where the site is disabled so no writes can happen, and
| then usually a window of writes are lost anyway (i.e. 'last X
| hours, since the backup was taken')
|
| For your use-case, do you just have very few writes? Do you
| lose writes? Do you have some other clever strategy to deal
| with it?
| dragontamer wrote:
| It should be noted that not everyone is a global company.
|
| A typical bank / credit union may only serve one town. As
| such, it would be socially acceptable to designate 3am to 4am
| as a regular maintenance window where services are shutdown.
| Waterluvian wrote:
| Good point. The 5 minutes of downtime is simply tolerated. My
| captive audience are dozens of humans and thousands of robots
| all willing to try again.
| nickdothutton wrote:
| Not really just a backup and restore. You need to be able to
| rebuild from zero. I think of it more as a disaster recovery
| exercise, and for those... you are only as good as your last
| _real_ rehearsal. That may mean a suitcase of tapes, a sheet of
| paper, and a rack of blank servers. Then you have the problem of
| release of confidential information. For this reason, the
| sweetest target for ransomware is the company who can neither
| recover their data, nor can they afford to have it publicly
| posted or monetised by the gang. Oh and you do store those
| backups offline dont you? Ransomware gangs have been known to
| loiter and observe their target for weeks to learn how to
| sabotage backups when the time comes.
| user3939382 wrote:
| What sucks for HIPAA is that you can get fined for the breach
| itself, regardless of your backup management.
| MattGaiser wrote:
| Seems appropriate.
| edoceo wrote:
| Not really a problem with HIPPA is it?
___________________________________________________________________
(page generated 2021-07-19 23:00 UTC)