[HN Gopher] Backblaze restore for Personal Backup is awful
___________________________________________________________________
Backblaze restore for Personal Backup is awful
I've been using Backblaze for a few years for my home computer. You
know how everyone keeps telling you, a backup is only a true backup
once you've done at least one restore? Now I know why (silly me).
I just got a "Safety Freeze" error [1] - essentially some
inconsistency with my backup. Backblaze will not tell me the actual
cause of this inconsistency. It's possible that some data might be
missing - Backblaze doesn't tell me though. The only official
solution is to _manually check all files_ (millions in my case). I
also can 't download a full backup since Backblaze only allows
downloads of up to 500GB at once. So my only option is to do a full
hard drive restore, costing $189 + customs in Europe, so at the end
probably closer to 300EUR. But even then I won't know if/which of
my data was corrupted. What bugs me is that the Backblaze desktop
software should be able to resolve this - it should be possible to
do a hash of all the files that are in the most recent backup, and
cross-check it with the hashes of the files on my machine. Not
sure what I should do now. [1]
https://www.backblaze.com/safety_frozen.html
Author : cloogshicer
Score : 78 points
Date : 2021-12-12 21:23 UTC (1 hours ago)
| rgovostes wrote:
| On macOS, Backblaze ships with 21 identical copies of the same
| executable, nearly 200 MB in all, presumably because they don't
| realize you can just execute one binary _n_ times (with different
| argv[0] if they 'd like) and they haven't written their code to
| be thread safe.
|
| Needless to say it doesn't instill confidence in the quality of
| software engineering that I rely on for disaster recovery.
| % openssl md5 /Library/Backblaze.bzpkg/bztrans*
| MD5(/Library/Backblaze.bzpkg/bztrans_thread00)=
| 772308dbd9b8083f4dc1c31bfe6a28da
| MD5(/Library/Backblaze.bzpkg/bztrans_thread01)=
| 772308dbd9b8083f4dc1c31bfe6a28da ...
| MD5(/Library/Backblaze.bzpkg/bztrans_thread19)=
| 772308dbd9b8083f4dc1c31bfe6a28da
| MD5(/Library/Backblaze.bzpkg/bztransmit)=
| 772308dbd9b8083f4dc1c31bfe6a28da
| dataflow wrote:
| Eh, even Clang does this. Install it on Windows and you'll get
| clang-cl.exe, clang++.exe, clang-cpp.exe, and clang.exe, all 90
| MB executables, and _almost_ entirely the same executables, but
| not bit-for-bit identical, so you can 't hardlink them. I
| actually hate this too, but my point is it doesn't necessarily
| say much about software quality.
| dn3500 wrote:
| Even if they're not very smart couldn't they hard link these?
| chomp wrote:
| > presumably because they don't realize you can just execute
| one binary n times
|
| This seems needlessly dismissive. I feel like they definitely
| know that you can execute the same binary multiple times.
| cloogshicer wrote:
| To actually resolve this issue, here's what I think I'll do
| (alternative suggestions much appreciated!):
|
| - Buy a new hard drive that's big enough to fit all my data
|
| - Download all the data from Backblaze to the new drive (will
| probably take a few days/weeks)
|
| - Write some tool that does the hashing/matching for me to see if
| anything's missing/corrupted (does something like this exist
| already?)
|
| - Switch to a better service or just local backups
| pieterhg wrote:
| I've used Beyond Compare, worked really well.
| cloogshicer wrote:
| Thanks for the suggestion!
| tailspin2019 wrote:
| Another vote here for Beyond Compare. Awesome software on
| both PC and Mac.
| DenseComet wrote:
| At the very least, you don't need to do the first two steps.
| Backblaze can ship you your backup on a hard drive, which you
| can then return for a full refund.
|
| https://www.backblaze.com/restore.html
| cloogshicer wrote:
| Thank you for the suggestion!
|
| The problem is that this is very expensive if you're not in
| the US. I'd have to pay for customs + return shipping, which
| would add up to much more than the cost of a new drive.
| juancn wrote:
| I was unable to recover backed up files on backblaze. It just
| doesn't work. They don't seem to do any periodic integrity checks
| on the cold data. I lost most of my files, they were able to
| recover maybe 20% of the data.
|
| Use anything else, but not backblaze.
| mabbo wrote:
| This reminds me of the xkcd comic "TornadoGuard"[0]. Whatever
| else the company is doing, if their core functionality doesn't
| fundamentally work when it needs to, then what is everyone
| paying for?
|
| [0]https://m.xkcd.com/937/
| mthoms wrote:
| I use Backblaze and am quite happy with the service. I wouldn't
| use it as my sole backup method however.
| juancn wrote:
| Try a full recovery and check it works. They lost most of my
| files.
| MichaelBurge wrote:
| Personally I bought a used tape machine and some $20 2.5TB tapes.
|
| It supports hardware encryption and the backup process is pretty
| much "tar cvf /dev/st0 /some/files". Downside is you don't
| actually save money doing this, because the drive costs ~$800.
| Upside is you're not relying on anybody else for worst-case
| scenario recovery of your important data.
|
| The actual read/write speed are about as fast as HDD, but if
| you're compressing it's easy to get blocked on CPU and get a
| fraction of that.
| quaffapint wrote:
| That would be very frustrating. I've been using B2 to store my
| encrypted backups (via Duplicati in my case) and that has been
| solid (and cheap) in both backup and restores. For those leery of
| their personal backup solution, maybe go that route.
| brandon272 wrote:
| Contact Backblaze support and see what they suggest. They may be
| able to point you to log files that offer more information on the
| reason for the safety freeze.
| cloogshicer wrote:
| I already did. They literally told me to manually cross-
| reference my hundreds of millions of files (edit: seems like
| it's "only" about half a million. I was mistaken).
|
| I did follow up, but haven't gotten a response yet. Will update
| as I get it.
| omg_ponies wrote:
| I find it very hard to believe that you were told to do
| something that equates to eyeballing hundreds of millions of
| files.
|
| Would you mind posting the exact information you gave them,
| and their response - redacting sensitive information of
| course.
| cloogshicer wrote:
| It sounds ridiculous, doesn't it?
|
| This was my message to them:
|
| > Hi, I was safety frozen. How can I find out the cause? I
| haven't made any significant changes to my system lately.
| In the support document it says that I should check if any
| data is missing - I have a lot of data backed up in
| backblaze, how can I know for sure nothing is missing or
| corrupted? How should I proceed? I've checked with
| CrystalDiskInfo and it seems that all SMART data is fine.
| Computer behaves normally. I've attached bzlogs and
| bzreports folders, just in case they're important.
|
| And this was their response:
|
| > Unfortunately there is simply no way to confirm what
| caused a safety freeze from our end. We can only provide
| the most common causes in this case, however we wouldn't be
| able to pinpoint the exact cause. The only way to verify
| any missing/deleted data would be to access your
| View/Restore Files page and cross reference what is found
| on our servers and what is found locally on your system.
| There wouldn't be any direct or automatic method of
| checking what data is missing, if any.
| oceanghost wrote:
| "Hundreds of millions of files" sounds more like a database
| to me...
| cloogshicer wrote:
| I was mistaken actually. It's "only" about 600k files.
| Sorry, and thanks for mentioning it! I'll correct it above
| too if I can.
| mlyle wrote:
| So, let me tell you about how I was getting safety frozen,
| and why.
|
| I had a Mac which crashed and that had a whole bunch of files
| go missing. Backblaze didn't want to do backups with a huge
| subset of my disk files gone without having gotten
| notification messages from the file changes API that this
| stuff _should_ be missing /was deleted. So they safety froze
| my machine.
|
| It's cool that Backblaze notices this. But yes, the question
| of what to do next isn't clear. Does one just start a new
| backup? Does one scour the backup looking for anything that
| may be missing on your machine, now? Etc.
| cloogshicer wrote:
| Yes, that's exactly the issue - that I don't know if/which
| files are missing/corrupted.
| brudgers wrote:
| The cloud is a great availability tool for personal work.
|
| It is a poor backup for personal work because the terms and
| conditions are for B2B.
|
| The terms and conditions are suited for contractual obligations
| under due diligence. They allow a business to avoid negligence
| claims when something goes wrong in a third party contract.
|
| The service is not designed around the sentimental value of
| baby's first steps. Stop paying for storage and it goes from
| viewable on everyone's iPad to gone.
|
| Personal work should be backed up on physical media. Multiple
| copies in multiple locations. If there's a copy in the cloud,
| that's convenient. But it is not durable.
|
| Good luck.
| anamexis wrote:
| I don't know, I would say B2 (or S3, etc.) are perfectly
| suitable as a secondary backup location. They are as durable or
| more durable than a NAS that I have in a friend's basement or
| something.
| azalemeth wrote:
| I'm looking for a new cloud backup service, one that works on
| linux/MacOS and with multiple machines. I've heard many mixed
| things about Backblaze -- ranging from "It's amazing!" to "Use
| their [pay per GB] buckets unless you are using a single Windows
| or MacOS computer", to "Other providers are cheaper". Certainly
| the best in terms of $/GB seems to be OpenDrive's "unlimited"
| option, which only includes 'no NASes' as a rather nebulous
| condition.
|
| I'd love to know -- what do other HN users for this? The story
| author's point about backups only being backups once you've used
| them to restore _really* rings true to me._
| CamperBob2 wrote:
| 1. Sign up for Dropbox
|
| 2. Sign up for the optional 'PackRat' service, or whatever they
| call it nowadays
|
| 3. xcopy c:\\*.* d:\dropbox\backup /s /e /d
|
| Somewhat tongue-in-cheek, but that's basically what I do, using
| the Windows equivalent of a cron job to refresh the backup once
| a day. Not on the whole c: drive, just on my development
| directory tree.
|
| The ability to dig through older versions of _all_ files has
| been incredibly valuable, given the large number of build
| targets and other assorted binaries that aren 't under source
| control.
| S04dKHzrKT wrote:
| rsync.net has a special pricing tier for Borg users that might
| be worth looking into.
|
| https://github.com/borgbackup/borg
|
| https://rsync.net/products/attic.html
| deadbunny wrote:
| I use rsync.net and borgmatic[1] backing up about a terrabyte.
| It's about the same price as S3 (with no egress charges,
| cheaper if you use just their Borg plan[2]) and you can backup
| a multitude of ways from rsync to zfs snapshots.
|
| It's not as user friendly as something with a GUI but IMO
| anyone on HN should be able to get it working in about 30 mins.
|
| 1. https://torsion.org/borgmatic/
|
| 2. https://rsync.net/products/attic.html
| neilv wrote:
| Borgmatic (or plain Borg) to rsync.net is appealing.
|
| One thing to notice about the super-affordable Borg plan is
| that it doesn't include free ZFS snapshots. My understanding
| is that you can have the SSH key used by the host to push its
| backup restricted to only Borg, and only append-only, within
| its repo... but if there's another way to access the ZFS
| (e.g., with an unrestricted SSH key), the Borg repo could be
| deleted. And then you might really want automatic ZFS
| snapshots as an additional layer of protection.
| gonewest wrote:
| I use the Backblaze client on laptops and desktops, and B2 for
| NAS backups. No complaints.
| akeck wrote:
| If I were in your situation with my personal data, I'd spend the
| money to get the restore hard drive shipped plus get hard
| drives/arrays for working space to do hash comparisons with the
| BB data. At this point, for some files, you may only have one
| good copy.
| cloogshicer wrote:
| Thanks for the suggestion, I think this is close to what I'll
| do (see below). Really sucks though, this is exactly what BB
| was supposed to prevent in the first place, all this manual
| labor.
| andrepew wrote:
| I don't like Backblaze because they require you to hand over your
| encryption key to their website to restore which kills any hope
| of it being a 0-knowledge solution.
|
| Right now I'm using Arq Backup + S3 and have been happy.
| 0xCMP wrote:
| I do the same, but with B2 which is nice cause it can be very
| cheap.
| anamexis wrote:
| I use Arq with B2, which seems like a happy compromise.
| deadbunny wrote:
| Has it ever been marketed as zero knowledge?
| summm wrote:
| How do they think this could be acceptable?
| tzs wrote:
| I was about to use Backblaze Personal and then noticed that.
| Was then about to use Arq + Backblaze B2 when I realized that
| (1) Arq supports OneDrive, (2) I have a TB of OneDrive as part
| of my Microsoft 365 subscription, and (3) I only use cloud
| storage for file transfer between mobile devices and desktop,
| so my OneDrive space was almost all unused, so went with Arq +
| OneDrive. 3 years of weekly backups later and I've still only
| used about half of my 1 TB.
|
| The only thing I'm not happy about with Arq is that a "verify"
| downloads all the backup data to checksum it. That takes 3 or 4
| days on my connection.
|
| I thought I read that many cloud storage provider APIs provide
| a way to ask the server for a checksum of a stored blob. I'd
| have expected Arq to make use of that, but maybe it is not
| reliable (the server might just report what the checksum is
| supposed to be, not actually read the data and checksum it?).
|
| Arq documents their storage format. I wonder if it would be
| possible to use a VM on Azure to access my OneDrive storage and
| do the checksumming on the VM?
| Trias11 wrote:
| Same.
|
| Arq + Wasabi
| 1123581321 wrote:
| I'm a fan of Arq and DIY storage, but how do you handle
| versioning?
| andrepew wrote:
| Arq has baked in file versioning unless you mean something
| else?
| 1123581321 wrote:
| Ah, of course. Sorry. :)
| galonk wrote:
| A lot of people sing BB's praises but I never had a good
| experience with them. The client was always slow, buggy, and
| resource hungry, and its UI is terrible. They got shirty with me
| for reporting bugs when I was using a macOS beta. And finally, at
| some point even though nothing about my computer changed (it was
| a Mac Mini, what was going to change), I got a message saying
| some security/copy protection system had detected that my
| computer was "different", and I had to un-install and re-install
| the entire app to fix it (there apparently being no easier way to
| unset a flag). I uninstalled and skipped the second part.
|
| Instead of using BB, get a Synology/Qnap/FreeNAS box to backup
| all your stuff locally, and back that up to another service (e.g.
| Glacier or Synology's own C2).
| willis936 wrote:
| I caution the casual reader against glacier. It's not what it
| appears at a glance. Your files should be put into a single
| archive before upload otherwise you'll spend weeks waiting for
| AWS scripts to manage old files.
|
| B2/S3 is what most people want.
| k8sToGo wrote:
| You need to differentiate between BB Personal Backup and BB B2
| service which is more like something you suggested. But these
| days I just use rsync.net + Wasabi + Kopia + rclone.
| felixforfun wrote:
| Maybe I'm missing something, but as far as I understand the
| simplest course of action is to reinstall Backblaze and inherit
| the existing backup. Or is there something that's preventing you
| from doing that?
| wanderingmind wrote:
| I don't use Backblaze but I use rclone which can be connected to
| a Backblaze backend. Rclone is opensource and has a subcommand
| check[0] that can compare files between remote/local or
| remote1/remote2. I suggest using it to find the missing files.
|
| [0] https://rclone.org/commands/rclone_check/
| k8sToGo wrote:
| Pretty sure what you mean is Backblaze B2, but the author is
| talking about Backblaze Personal Backup.
| wanderingmind wrote:
| Yes sorry didn't know they were different. Well atleast I
| learnt something today.
| mritzmann wrote:
| More information about "safety freez" from an backblaze engineer:
| https://old.reddit.com/r/backblaze/comments/hvcbpw/safety_fr...
| CamperBob2 wrote:
| "The log files that list what Backblaze has backed up are
| called "bz_done" files. They list 'what has been done'. Here is
| where they are located on _your computer_ : ... WARNING: don't
| edit those files -> you are guaranteed to corrupt your backup.
| You'll lose everything."
|
| Why does the integrity of the backup rely on files stored on
| the computer being backed up? This seems so stupid that I'm
| sure I'm missing a clue.
| gruez wrote:
| >Why does the integrity of the backup rely on files stored on
| the computer being backed up? This seems so stupid that I'm
| sure I'm missing a clue.
|
| Reading the explanation in the reddit thread, that's not the
| impression I got at all.
|
| 1. If your computer exploded, your backup integrity would not
| be compromised
|
| 2. If gremlins in your computer did mess with the file, your
| backups could be compromised. That sounds bad, until you
| realize that gremlins in your computer could also compromise
| the executable to do other things that could compromise the
| backup, (eg. telling the server to delete existing backup
| data because the retention period has passed or whatever, or
| simply uploading bad data and waiting for the retention
| period of 30 days to pass). Moral of the story: if the
| computer doing the backup can't be trusted to operate
| correctly, all bets are off.
| cloogshicer wrote:
| Yup. Sounds pretty terrifying.
| CamperBob2 wrote:
| That can't possibly be how it works. No way.
| cloogshicer wrote:
| Well, their suggested fix basically says that I should
| "unlink" my computer from the online backup, and relink
| it (by re-installing the Backblaze app). But before I do
| that, I should download any missing files, since they
| will be deleted from the backup upon re-link (if they
| can't be found on my computer).
|
| But I can't do that without knowing which files are
| missing.
| cloogshicer wrote:
| Thanks! I already found that, unfortunately, they also just
| say: Do a full restore via shipped hard drive (expensive for
| non-USA).
| [deleted]
| exhibitapp wrote:
| Tried to use backblaze to save personal files on a laptop I was
| going to lose access to, at first it said it was incorrectly
| permissioned. I changed a few setting and the message went away.
| My fun surprise when I went to restore from the backup and 90% of
| my files were missing and my former computer was bricked.
| 1123581321 wrote:
| Can you acknowledge you've been hit by a rare bug? The service
| isn't awful.
|
| My own experience with Backblaze personal backup was negative
| because I had so many directories and files that it took them
| longer than they anticipated to prepare a recovery drive, but I
| know that I don't have a typical use case and I recommend it
| without reservation to people who want that kind of whole
| computer backup.
| JadeNB wrote:
| > Can you acknowledge you've been hit by a rare bug? The
| service isn't awful.
|
| It seems to me that service that can't deal with the bug I'm
| actually having is awful, whether that bug is rare or common.
| As an individual user, I care about my experience, not the
| statistical aggregate of user experiences. (Generalised 'I'
| here--this is not my bug.)
| 1123581321 wrote:
| Well, sure, but in the context of a discussion forum it's
| less helpful, and usually the goal of writing it publicly is
| to deter potential happy users. We see this often in software
| support forums where someone has a rare bug and is offended
| that others won't stop their usage because of it.
| cloogshicer wrote:
| This doesn't seem to be a rare bug, but intended behavior as
| hardware components fail:
| https://www.backblaze.com/safety_frozen.html
|
| Also, the official solution (checking all backed up files
| manually) is genuinely awful, even if this were actually a rare
| bug.
| 1123581321 wrote:
| I agree about that line in the documentation being too simple
| (though it shouldn't deter the technical users here.) They
| should advise to order a backup drive and compare all the
| files, or provide CLI instructions to check the safety log
| file against the one-liner. They seem to provide much better,
| candid support for hardware issues/safety freezes on reddit,
| based on what others have posted here, which is cool of them.
| Contacting support should yield similarly complete answers.
| cloogshicer wrote:
| I already contacted support and this is the only thing I
| got so far. They literally told me to manually match all
| those files.
|
| Ordering a drive from them is very expensive if you're not
| in the US like me.
| 1123581321 wrote:
| Too bad. Yes, the location does make it rough.
|
| There are some funny stories about Backblaze personally
| delivering an emergency drive to someone in a remote part
| of the world, but they don't help the median user.
___________________________________________________________________
(page generated 2021-12-12 23:00 UTC)