You might not even like rsync. Yeah it's old. Yeah it's slow. But if you're working with Linux you're going to need to know it. In this video I walk through my favorite everyday flags for rsync. Su...

You might not even like rsync. Yeah it’s old. Yeah it’s slow. But if you’re working with Linux you’re going to need to know it.

In this video I walk through my favorite everyday flags for rsync.

Support the channel:
https://patreon.com/VeronicaExplains
https://ko-fi.com/VeronicaExplains
https://thestopbits.bandcamp.com/

Here’s a companion blog post, where I cover a bit more detail: https://vkc.sh/everyday-rsync

Also, @BreadOnPenguins made an awesome rsync video and you should check it out: https://www.youtube.com/watch?v=eifQI5uD6VQ

Lastly, I left out all of the ssh setup stuff because I made a video about that and the blog post goes into a smidge more detail. If you want to see a video covering the basics of using SSH, I made one a few years ago and it’s still pretty good: https://www.youtube.com/watch?v=3FKsdbjzBcc

Chapters:
1:18 Invoking rsync
4:05 The --delete flag for rsync
5:30 Compression flag: -z
6:02 Using tmux and rsync together
6:30 but Veronica… why not use (insert shiny object here)

  • NuXCOM_90Percent@lemmy.zipEnglish
    231·
    10 days ago

    I would generally argue that rsync is not a backup solution. But it is one of the best transfer/archiving solutions.

    Yes, it is INCREDIBLY powerful and is often 90% of what people actually want/need. But to be an actual backup solution you still need infrastructure around that. Bare minimum is a crontab. But if you are actually backing something up (not just copying it to a local directory) then you need some logging/retry logic on top of that.

    At which point you are building your own borg, as it were. Which, to be clear, is a great thing to do. But… backups are incredibly important and it is very much important to understand what a backup actually needs to be.

    • tal@olio.cafeEnglish
      8·
      10 days ago

      I would generally argue that rsync is not a backup solution.

      Yeah, if you want to use rsync specifically for backups, you’re probably better-off using something like rdiff-backup, which makes use of rsync to generate backups and store them efficiently, and drive it from something like backupninja, which will run the task periodically and notify you if it fails.

      rsync: one-way synchronization

      unison: bidirectional synchronization

      git: synchronization of text files with good interactive merging.

      rdiff-backup: rsync-based backups. I used to use this and moved to restic, as the backupninja target for rdiff-backup has kind of fallen into disrepair.

      That doesn’t mean “don’t use rsync”. I mean, rsync’s a fine tool. It’s just…not really a backup program on its own.

      • melfie@lemy.lolEnglish
        31·
        10 days ago

        Having a synced copy elsewhere is not an adequate backup and snapshots are pretty important. I recently had RAM go bad and my most recent backups had corrupt data, but having previous snapshots saved the day.

        • melfie@lemy.lolEnglish
          1·
          9 days ago

          Don’t understand the downvotes. This is the type of lesson people have learned from losing data and no sense in learning it the hard way yourself.

          • tomenzgg@midwest.socialEnglish
            1·
            9 days ago

            How would you pin down something like this? If it happened to me, I expect I just wouldn’t understand what’s going on.

      • koala@programming.devEnglish
        0·
        9 days ago

        Beware rdiff-backup. It certainly does turn rsync (not a backup program) into a backup program.

        However, I used rdiff-backup in the past and it can be a bit problematic. If I remember correctly, every “snapshot” you keep in rdiff-backup uses as many inodes as the thing you are backing up. (Because every “file” in the snapshot is either a file or a hard link to an identical version of that file in another snapshot.) So this can be a problem if you store many snapshots of many files.

        But it does make rsync a backup solution; a snapshot or a redundant copy is very useful, but it’s not a backup.

        (OTOH, rsync is still wonderful for large transfers.)

    • non_burglar@lemmy.worldEnglish
      0·
      10 days ago

      I use rsync and a pruning script in crontab on my NFS mounts. I’ve tested it numerous times breaking containers and restoring them from backup. It works great for me at home because I don’t need anything older than 4 monthly, 4 weekly, and 7 daily backups.

      However, in my job I prefer something like bacula. The extra features and granularity of restore options makes a world of difference when someone calls because they deleted prod files.

  • mesa@piefed.socialOPEnglish
    13·
    10 days ago

    Ive personally used rsync for backups for about…15 years or so? Its worked out great. An awesome video going over all the basics and what you can do with it.

    • Eager Eagle@lemmy.worldEnglish
      2·
      10 days ago

      It works fine if all you need is transfer, my issue with it it’s just not efficient. If you want a “time travel” feature, your only option is to duplicate data. Differential backups, compression, and encryption for off-site ones is where other tools shine.

      • bandwidthcrisis@lemmy.worldEnglish
        1·
        10 days ago

        I have it add a backup suffix based on the date. It moves changed and deleted files to another directory adding the date to the filename.

        It can also do hard-link copied so that you can have multiple full directory trees to avoid all that duplication.

        No file deltas or compression, but it does mean that you can access the backups directly.

    • okamiueru@lemmy.worldEnglish
      6·
      10 days ago

      That part threw me off. Last time i used it, I did incremental backups of a 500 gig disk once a week or so, and it took 20 seconds max.

    • HereIAm@lemmy.worldEnglish
      2·
      9 days ago

      Compared to something multi threaded, yes. But there are obviously a number of bottlenecks that might diminish the gains of a multi threaded program.

  • atk007@lemmy.worldEnglish
    7·
    9 days ago

    Rsnapshot. It uses rsync, but provides snapshot management and multiple backup versioning.

  • surph_ninja@lemmy.worldEnglish
    5·
    10 days ago

    Use borg/borgmatic for your backups. Use rsync to send your differentials to your secondary & offsite backup storage.

  • clif@lemmy.worldEnglish
    5·
    9 days ago

    I’ll never not upvote Veronica Explains. Excellent creator and excellent info on everything I’ve seen.

  • state_electrician@discuss.tchncs.deEnglish
    4·
    9 days ago

    Why videos? I feel like an old man yelling at clouds every time something that sounds interesting is presented in a fucking video. Videos are so damn awful. They take time, I need audio and I can’t copy&paste. Why have they become the default for things that should’ve been a blog post?

    • Encrypt-Keeper@lemmy.worldEnglish
      4·
      9 days ago

      I’m not super familiar with Syncthing, but judging by the name I’d say Syncthing is not at all meant for backups.

    • conartistpanda@lemmy.worldEnglish
      2·
      9 days ago

      Syncthing is technically to synchronize data across different devices in real time (which I do with my phone), but I also use it to transfer data weekly via wi-fi to my old 2013 laptop with a 500GB HDD and Linux Mint (I only boot it to transfer data, and even then I pause the transfers to this device when its done transferring stuff) so I can have larger data backups that wouldn’t fit in my phone, since LocalSend is unreliable for large amounts of data while Synchting can resume the transfer if anything goes wrong. On top of that Syncthing also works in Windows and Android out of the box.

  • Mio@feddit.nuEnglish
    3·
    10 days ago

    I think the there are better alternatives for backup like kopia and restic. Even seafile. Want protection against ransomware, storage compression, encryption, versioning, sync upon write and block deduplication.

  • ryper@lemmy.caEnglish
    3·
    10 days ago

    I was planning to use rsync to ship several TB of stuff from my old NAS to my new one soon. Since we’re already talking about rsync, I guess I may as well ask if this is right way to go?

    • Suburbanl3g3nd@lemmings.worldEnglish
      3·
      10 days ago

      I couldn’t tell you if it’s the right way but I used it on my Rpi4 to sync 4tb of stuff from my Plex drive to a backup and set a script up to have it check/mirror daily. Took a day and a half to copy and now it syncs in minutes tops when there’s new data

    • SayCyberOnceMore@feddit.ukEnglish
      2·
      10 days ago

      It depends

      rsync is fine, but to clarify a little further…

      If you think you’ll stop the transfer and want it to resume (and some data might have changed), then yep, rsync is best.

      But, if you’re just doing a 1-off bulk transfer in a single run, then you could use other tools like xcopy / scp or - if you’ve mounted the remote NAS at a local mount point - just plain old cp

      The reason for that is that rsync has to work out what’s at the other end for each file, so it’s doing some back & forwards communications each time which as someone else pointed out can load the CPU and reduce throughput.

      (From memory, I think Raspberry Pi don’t handle large transfers over scp well… I seem to recall a buffer gets saturated and the throughput drops off after a minute or so)

      Also, on a local network, there’s probably no point in using encryption or compression options - esp. for photos / videos / music… you’re just loading the CPU again to work out that it can’t compress any further.

    • GreenKnight23@lemmy.worldEnglish
      1·
      10 days ago

      yes, it’s the right way to go.

      rsync over ssh is the best, and works as long as rsync is installed on both systems.

      • qjkxbmwvz@startrek.websiteEnglish
        2·
        10 days ago

        On low end CPUs you can max out the CPU before maxing out network—if you want to get fancy, you can use rsync over an unencrypted remote shell like rsh, but I would only do this if the computers were directly connected to each other by one Ethernet cable.

  • Appoxo@lemmy.dbzer0.comEnglish
    2·
    9 days ago

    Veeam for image/block based backups of Windows, Linux and VMs.
    syncthing for syncing smaller files across devices.

    Thank you very much.

  • 1984@lemmy.todayEnglish
    2·
    9 days ago

    I never thought of it as slow. More like very reliable. I dont need my data to move fast, I need it to be copied with 100% reliability.

  • Xylight@lemdro.idEnglish
    2·
    10 days ago

    rsync for backups? I guess it depends on what kind of backup

    for redundant backups of my data and configs that I still have a live copy of, I use restic, it compresses extremely well

    I have used rsync to permanently move something to another drive though