[HN Gopher] Backing up data like the adult I supposedly am
___________________________________________________________________
Backing up data like the adult I supposedly am
Author : miked85
Score : 73 points
Date : 2020-09-19 11:32 UTC (1 days ago)
(HTM) web link (magnusson.io)
(TXT) w3m dump (magnusson.io)
| walterbell wrote:
| With many people using mobile devices, a plug for PhotoSync
| enabling seamless photo backup and sync between iOS, Android,
| Windows, Mac, Linux, local NAS, iXpand local flash and cloud
| services. They have both subscription/rental and
| lifetime/ownership licenses, https://www.photosync-app.com
|
| iOS storage management has improved with a user-visible
| "filesystem" and storage providers allowing edit-in-place, but
| there's still virtually no support for backup or rsync. The
| native iOS Files app is not a robust client for NAS storage. So
| far, the best option has been GoodReader (Russian devs) which
| implements robust sync (SMB, SFTP & more) within the app, along
| with optional in-app strong encryption that goes beyond iOS data
| protection. Unencrypted files are visible to other apps,
| https://www.goodreader.com/
|
| Samsung's iXpand has built an ecosystem of iOS apps that support
| their custom protocol for iXpand flash drives via Lightning. Now
| that iPad enables access to local storage via USB-C, we need a
| similar ability to mount a ZFS drive, even if Apple won't provide
| this natively in iOS.
|
| With a low-cost x86 SBC like Odroid H2+, an entry-level NAS can
| be constructed with Ubuntu ZFS and dual 3.5" drives.
| rsync wrote:
| "OS storage management has improved with a user-visible
| "filesystem" and storage providers allowing edit-in-place, but
| there's still virtually no support for backup or rsync. The
| native iOS Files app is not a robust client for NAS storage. So
| far, the best option has been GoodReader (Russian devs) which
| implements robust sync (SMB, SFTP & more) within the app, along
| with optional in-app strong encryption that goes beyond iOS
| data protection."
|
| Thank you - this is very interesting.
|
| Although I am a (casual) iphone user (my iphone has never seen
| my real name or my real phone number and has _never_ touched
| rsync.net) I was not aware of the user-visible filesystem nor
| was I aware of "Goodreader".
|
| Does this user-visible filesystem allow me to just copy over my
| entire music library (which is files and directories, and no
| knowledge of apple/itunes/ios) and then let itunes browse it,
| locally on the phone ? Or do I still need to do complicated
| import tasks ?
| DelightOne wrote:
| > So far, the best option has been GoodReader
|
| Dropbox?
| walterbell wrote:
| It's about flexibility. GoodReader supports both public
| clouds (Dropbox, OneDrive, Google Drive, SugarSync, Box)
| _and_ open protocols (WebDAV, FTP, SFTP, AFP, SMB) for
| private storage. Then, after remote files have been synced to
| /from the iDevice, editing them in-place from other apps.
| cosmie wrote:
| I don't use GoodReader, so can't say if it's exactly the
| same. But as of iOS 13, the Files app now supports directly
| connecting to external servers. SMB for sure, but can't
| find a definitive list of supported protocols.
|
| You can also use Files to directly browse and access files
| from third party cloud services, although it requires you
| to download the service's app first[1].
|
| [1] https://support.apple.com/guide/iphone/connect-
| external-devi...
| graton wrote:
| Like the author I have also been very happy with Borg backup
| software ( https://www.borgbackup.org/ ).
|
| The compression and de-duplication is very useful. A little bit
| of a learning curve to get everything up and running, but not too
| bad.
| linsomniac wrote:
| Of course, as with all backups, you should make sure to run a
| test recovery. I ran into a situation with Borg where the
| encoding of one of my filenames caused that file and anything
| else behind it in the backup to be unrecoverable. Or at least I
| was never able to find a way to recover it.
|
| I gave it the old college try to recover, using the different
| tools to try to access it (the fuse mount, the CLI), I tried
| all sorts of different settings for my locale. At the time I
| had at least 2 other backups of that so eventually I recovered
| from my primary backup. I was testing out Borg at the time.
|
| I've ended up using Restic more recently, and it seems to be
| fine. Uses kind of a lot of memory in some situations though.
| Small AWS instances have issues. My primary backups still go
| via rsync though.
| aborsy wrote:
| I also like Borg back up.
|
| I wonder if I am missing something compared to restic?
| magicalhippo wrote:
| I found restic did not scale well with a large amount of
| files when I tried it a few years ago. Has this changed? Is
| Borg better?
|
| I used to run Crashplan with near-continuous backup for my
| important files, and I'm still missing this.
| stevesimmons wrote:
| I too used to use Crashplan until they discontinued their
| personal subscription and with it the option to backup to
| local drives.
|
| Around the same time I tried Restic. That ran into
| difficulties (don't recall what anymore) so I switched to
| Borg.
|
| Borg has been 100% reliable, including a full restore of
| /home after my laptop was stolen.
| more-coffee wrote:
| Borg is supposedly a bit faster, restic supports a ton of
| backends https://github.com/restic/restic/issues/1875 Also
| restic does not allow to backup unencrypted.
|
| I haven't used borg. Only done some maintenance on restic
| backup jobs at work. Restic's command design is intuitive and
| the documentation is good. But Borg looks just fine in that
| regard as well.
| aborsy wrote:
| Thank you !
|
| I posted another question on Borg, in case you know the
| answer!
| rsync wrote:
| "I wonder if I am missing something compared to restic?"
|
| The biggest difference for us is that borg really requires a
| server side 'borg' binary to talk to, which we have built
| into rsync.net. restic, on the other hand, can just connect
| to any old SFTP endpoint.
|
| This means we need to preserve some amount of backwards-
| compat and so we maintain borg0.x _and_ borg1.x binaries in
| our environment (and eventually, borg2.x).
| aborsy wrote:
| I have a question about Borg's encryption. It's stated [0] that
| if multiple clients update same repository, the server might be
| able to decrypt data.
|
| Why is that the case and wouldn't that make the encryption very
| weak? Simultaneous updates happen quite often.
|
| Would restic have the same problem?
|
| -------------
|
| Update: The issue happens because Borg uses AES in the CTR mode
| (not AES GCM) and two clients could provide the same nonce. The
| server could then recover the plaintext from two cipher texts.
| This is the famous nonce reuse problem.
|
| So Borg developers are not using established primitives for
| this use case. Also, I am not comfortable with the OpenSSL even
| though it's got better since 2015. The libssl code base is a
| mess and buggy. On the other hand using the low level libcrypto
| library would expose developers to the crypto primitives with
| possibilities for errors for people not expert in cryptography.
|
| Borg should consider ChaCha-Poly135 as in rclone (or at least
| AES-GCM).
|
| [0]https://borgbackup.readthedocs.io/en/stable/internals/securi
| ...
| TimWolla wrote:
| > Why is that the case [...]?
|
| This is explained in the "Encryption" section: https://borgba
| ckup.readthedocs.io/en/stable/internals/securi...
|
| The important part is the part about avoiding re-use of the
| AES CTR value.
|
| > Simultaneous updates happen quite often.
|
| Personally I created a dedicated borg repository per machine
| I want to backup, because that avoids sharing passphrases
| across machines. This comes with the drawback that I cannot
| deduplicate across machines, but that is acceptable to me,
| because the data is mostly unique-ish anyway. I only backup
| the user data, not everything (e.g. /bin/).
| aborsy wrote:
| Yes, thanks!
|
| I meanwhile read about it and updated my comment.
|
| Would it be practical to rclone the output of the Borg into
| a cloud service using an rclone crypt remote? In my
| experience, rclone's crypt remote is sluggish, even
| locally. I am not sure how the mount would work.
|
| It's unfortunate that we have to get the dedup from Borg
| and the encryption from rclone!
| TimWolla wrote:
| I never used rclone, but I can tell you that a borg
| repository basically is a number of encrypted blobs of up
| to 500 MB size that are never going to be modified again
| (only created and deleted) + a few small metadata files.
| It _rsync_ s quite well.
| jng wrote:
| I use TineMachine to three separate identica disks: one at home,
| another one of the office, and and a third one at my parents
| place 1000 km away. Pretty anythingproof. I also have three other
| disks with three identical copies of my old archived stuff, in
| the same locations. Also all code repos are online (svn, git and
| hg), and I have most non-code stuff on Dropbox too. Restored
| entire machine from TimeMachine once when I upgraded the laptop,
| ideal experience. I'm not happy that Covid made me have the
| office disks at home now too, but otherwise, I feel pretty safe.
| remote_phone wrote:
| Is there a way to test if TimeMachine backups are uncorrupted?
| I backup using TimeMachine as well but as far as I can tell
| there is no way to verify a backup. I'm concerned that at some
| point my backup will get corrupted and I won't know why. This
| happened to my iPhone backup to iTunes, luckily I had a iCloud
| backup.
| rubatuga wrote:
| If you want to find out how to backup your iPhone on Linux, I've
| also made a guide! It's actually kind of complicated, but it can
| be fully automated. I connect my iPhone to my Linux server in my
| room to do an incremental backup every night at 5 AM (it also
| fast charges at 1.8 amps over USB C). I then create ZFS snapshots
| every week, since the iPhone backup is an overwrite type.
|
| https://www.naut.ca/blog/2020/03/20/self-hosting-series-part...
| walterbell wrote:
| Thanks for your self-hosting tutorials for iOS services! The
| next question is how to extract individual files from a backup,
| without needed an iDevice for a full restore. There are several
| commercial products sold for this purpose, but I've not yet
| seen OSS tools to parse iOS backups.
| RealStickman_ wrote:
| I should maybe have a closer look at borg. Just to learn what
| alternatives there are to my current restic + Backblaze B2 setup.
| toast0 wrote:
| What software do people like for backing up Windows desktops?
|
| I really want something that ends with a full disk image that's
| easy to restore to a new device, runs backups on a schedule (and
| will run a while after the next boot if the computer is off at
| the scheduled time), writes the images to a unix system on the
| LAN (either directly, or by writing to SMB), and doesn't cost an
| arm and a leg.
| theshrike79 wrote:
| I just went with Backblaze Personal. It's pretty much fire and
| forget.
|
| Doesn't provide a perfect full disk image, but it does store
| everything I need. I've done one full restore from them (fried
| motherboard from a power surge) and it went as smoothly as I
| could expect.
| jaden wrote:
| Last I checked BackBlaze only kept deleted files for a max of
| 30 days, making it a non-starter for my needs. I'm not sure
| if that's still the case.
| 4d66ba06 wrote:
| You can pay more for year long retention now
| chinathrow wrote:
| That's what they say in their offerings/docs, but in fact I
| had data kept far longer than the 30 days (after the laptop
| registered for Backblaze was already scrapped and not been
| running for months).
| csnover wrote:
| This is not a great choice. The Backblaze client is extremely
| insecure--like, arbitrary remote root code execution insecure
| --and they seem to me to either not care or are too
| incompetent (or both) to be trusted.[0]
|
| [0] https://twitter.com/zetafleet/status/1304664097989054464
| encom wrote:
| I like BackBlaze, but their client has always been
| absolutely terrible. Pretty shocked to hear that it's
| _this_ bad, I and just paid for another two years in
| advance.
| aborsy wrote:
| Unencrypted?
| philjohn wrote:
| As someone else commented - you can set a personal key.
| It's encrypted at rest (and in flight), but obviously
| Backblaze have that key.
| ls612 wrote:
| There's the option to set a password to encrypt it.
| aborsy wrote:
| Well I should perhaps elaborate: does it offer end to end
| authenticated encryption with keys that never leave
| user's device in an open source program?
|
| Another point, I suppose that backblaze comes with dedup
| and compression?
| jszymborski wrote:
| Re: encryption, long-story short, the keys used to
| decrypt your data are stored in their data centers, but
| you can also encrypt those keys with a symmetric key
| which only you know. [0]
|
| Re dedup/compression, it's a bit irrelevant because their
| plans are unmetered.
|
| [0] https://help.backblaze.com/hc/en-
| us/articles/217664688-Can-y...
| bosie wrote:
| regarding unmetered, i gave up on backblaze as their
| network connection seems incredibly slow. i think asking
| for compression and dedup is very relevant with them
| voltagex_ wrote:
| Yeah, their stock answer is "use more threads" but I
| could never use more than 30-50% of my upstream
| bandwidth. It doesn't help that the client is slow itself
| and seems to sometimes just stop backing up.
| tzs wrote:
| > [...] but you can also encrypt those keys with a
| symmetric key which only you know
|
| ...until you need to restore from backup. You then have
| to sign in on the Backblaze website and enter that key,
| the files you are trying to restore are then decrypted on
| their end, and bundled up and sent to you.
|
| They say that the key is only ever in RAM, and only then
| briefly.
| rzzzt wrote:
| I guess none of my suggestions will help :) but I sometimes run
| "Create a system image" from the Windows 7 backup and restore
| page that is still hanging around. It has an option to save to
| a network location.
|
| I think that even though some pop-up messages tell you that the
| previous backup will be blown away, it actually is incremental
| to a certain extent, and the recovery tool in the installer
| sometimes does list multiple dates to restore from -- although
| I'm not sure if and how data retention can be controlled. Also
| disk encryption is removed on restore, and I think the backup
| is not encrypted at rest either; you need to keep it in an
| encrypted location to begin with.
|
| For file-level backups, I'm using an rsync frontend, QtdSync,
| but I also had success with Borg running under Msys2's Python
| interpreter.
| EvanAnderson wrote:
| Windows system image backups with "physical" disks backing
| the storage (either locally-attached disks or via iSCSI) is
| actually reasonably nice. On later versions of Windows you
| can encrypt the backup with Bitlocker. Mounting prior backup
| generations via command line tools isn't too hateful. Bare
| metal restores of the entire system are very straightforward,
| too.
|
| Using a network location is somewhat less useful. You lose
| Volume Shadow Copy so it becomes a single generation full-
| backup-every-time solution. It's still easy to mount and to
| restore from, but marginally more useless.
|
| It would figure that Microsoft announced (last year, I
| believe) that the feature is no longer being developed.
| pgrote wrote:
| Acronis True Image works well for me. Scheduling with
| notifications of success/failure. You can backup locally to
| whatever windows can attach to or to a cloud.
|
| I've used to restore twice: same machine and new machine.
| Worked without an issue once the USB boot is created.
|
| I think the cost is reasonable for 5 workstations.
| magicalhippo wrote:
| Same here. I've set it to back up each night to the NAS
| locally and to their cloud.
|
| Had a SSD die on me a few years ago, the primary disk. With
| no warning it just bricked itself. Thanks to Acronis my
| computer was running again less than an hour later.
|
| Have also used it to restore documents and similar I
| accidentally deleted.
|
| Another nice feature they have is their malware protection
| service. It detects programs modifying a large number of
| files in a relatively short amount of time, blocks them until
| you say if it's ok or not.
| jhoechtl wrote:
| restic. It's fantastic and more importantly did'nt let me down
| even on faulty hardware.
| intricatedetail wrote:
| +1 for restic. Incredible tool.
| huhtenberg wrote:
| Yep. One of the most thoughtfully designed backup tools in
| existence... and actively developed at that!
| paxswill wrote:
| For my personal machines I've been using Veeam's free version.
| It's not as full featured as what I'd like (I have it set for
| nightly backups), but it seems to do the job alright. It offers
| to make a bootable flash drive for you at installation to make
| full restoration easier. I have it backing up over SMB to a
| FreeNAS box, but it doesn't look like the backup images are
| easily readable (the look to be some Veeam-specific format, but
| I didn't look to hard at them).
| MikusR wrote:
| Macrium Reflect free to a Samba (Raspberry pi) share.
| anonymousse1234 wrote:
| Veeam Agent for Microsoft Windows Free Edition
| remote_phone wrote:
| Windows already comes with a Backup and Restore. And it does
| both incremental backups and full disk images. I do both to my
| Synology nas every week. Maybe you're not using the
| "Professional" version?
| huhtenberg wrote:
| There are two go-to options, at least as far as /r/sysadmin and
| /r/datahoarder people are concerned - Veeam Endpoint Backup [1]
| or Macrium Reflect [2].
|
| However, another option is to back up just the data and
| reinstall the OS + programs in case of a disaster. I've been
| set up this way for nearly a decade, now using Bvckup 2 [3] as
| a replicator. This is faster and lighter on the system and it
| creates backups that are readily accessible.
|
| [1] https://www.veeam.com/windows-endpoint-server-backup-
| free.ht...
|
| [2] https://www.macrium.com
|
| [3] https://bvckup2.com
| voltagex_ wrote:
| For home use, I use Macrium and OneDrive. A good pattern I've
| found is to have a "clean" Windows 10 image (maybe with a few
| utilities), my personal data on OneDrive or a NAS and then
| something like PatchMyPC [1] to reinstall apps quickly.
|
| I also have bvckup2 (worth buying almost for the amazing UI
| alone) but I use it more for syncing some folders to and from
| a NAS.
|
| [1]: https://patchmypc.com/home-updater
| miked85 wrote:
| Arq: https://www.arqbackup.com/
| gruez wrote:
| Seems to be file-based, which isn't what OP wants.
| [deleted]
| huhtenberg wrote:
| Oh, man. No, just no.
|
| Arq 5 was OK.
|
| Arq 6 was shipped in a state that wasn't suitable even for
| beta. It corrupted and destroyed backups created with
| previous versions, couldn't complete new backups, wasn't
| working in fresh installs, had no documentation, no
| development plan and very poor communication from the dev
| addressing all these issues. The lash back was so bad that
| they closed their Twitter account and locked up Arq subreddit
| (only to claim later that it wasn't them, but Reddit itself
| that did that).
|
| A lot of people, me included, were expecting Arq 6 with a
| great deal of excitement only to witness one of the greatest
| dumpster fires in the recent history of ISVs. The news now is
| that they decided to just bury Arq 6 without trying to fix it
| and move on to Arq 7 - https://www.arqbackup.com/blog/next-
| up-arq-7/
| miked85 wrote:
| Interesting, thank you. I have been using Arq 5 without
| issues and was not aware of this.
| kneckebrot wrote:
| > half-assed rsync and shell script abomination
|
| I don't understand the author's difficulties with a minimalist
| bash-wrapped rsync-based backup. You can even hardlink to
| unchanged files from a previous backup to save space.
|
| This is how I wrap rsync: https://github.com/kaumanns/snapshot
|
| And regarding file permissions: why not simply use an EXT4 backup
| drive instead of an FAT32 one? Non-rhetorical question.
|
| My home network Raspberry has an HDD attached which gets fired up
| every couple days for a fresh snapshot of $HOME. The only thing I
| am missing is redundancy. And possibly encryption.
| linsomniac wrote:
| Getting an rsync wrapper to be robust takes some work. The
| wrapper script I use evolved over things I found while running
| it across ~200 hosts nightly for a couple years. It started as
| one of those hardlink scripts, but evolved into using zfs
| snapshots. My goal was to have it be the ultimate in
| reliability though, I wanted it to just work as much as
| possible, but be quiet unless the backup failed, at which point
| it should let me know.
|
| 15 years later, nightly backups across maybe 300 machines, this
| is what I have:
|
| https://github.com/tummy-dot-com/tummy-backup/blob/master/sb...
| TimWolla wrote:
| The author uses a systemd timer to schedule their backups. For
| backups going to a remote host I prefer adding a little bit of
| variance to the execution time to avoid consistently hitting some
| hotspot.
|
| From the timer I use to backup my server using Borg to rsync.net:
| [Timer] OnUnitActiveSec=24h RandomizedDelaySec=1h
|
| This will run the backup script every 24 hours with a random
| delay of up to 1 hour, so every 24.5 hours on average. This
| causes the job to nicely rotate around the day.
| corytheboyd wrote:
| That's really such a nice solution to the problem, nice.
|
| Can you imagine not reading the docs to discover those options.
| So you spin up a database to save state about runs to implement
| the delay. And you need a dashboard to monitor the various
| parts of the system for debugging.
|
| Or you read the docs
| gerdesj wrote:
| Whenever I use a scheduler I always use prime numbers wherever
| possible.
___________________________________________________________________
(page generated 2020-09-20 23:00 UTC)