20221101-smoke_me_a_kipper.md (8961B)
1 ## i'll be back(up) for breakfast 2 3 Earlier this year I wrote about my [backup setup](20220414-speak_of_the_dedup.html) and recently I had to put it to the test. 4 5 My PC is a tower that I have on a small stand next to my desk. In the past I had kept the case on my desk but it is rather large and dominates the space a bit too much. The other day my 1 year old toddled into the study and started pushing the power button on my PC, power cycling the machine a few times in quick succession. This was unknown to me until the next morning when I booted up my PC and noticed it was very sluggish and it crashed trying to open my browser. After it happened again I started digging through the logs and noticed some filesystem corruption. 6 7 As I described in my backup setup post, I have a 3 disk RAID 5 array as my $HOME. Because of the size I only nightly backup important documents, etc. A full backup is done periodically to an external drive I keep in my bug out bag. Unfortunately I had not done a full back in a while, but I knew my nightly backups were good so nothing too important was lost. 8 9 I had used [xfs](https://en.wikipedia.org/wiki/XFS){target="_blank" rel="noreferrer"} on my $HOME, so I unmounted the device and started an `xfs_repair`. The repair tool very quickly got to Phase 3, showing the output 10 ``` 11 Phase 3 - for each AG... 12 - scan and clear agi unlinked lists 13 - 09:50:01: scanning agi unlinked lists - 0 of 32 allocation groups done 14 ``` 15 16 The last line was repeated every 15 minutes, for over 36 hours, never changing from "0 allocation groups done". I don't think it was doing anything. Eventually I stopped it and ran the repair in check mode. This caused a segmentation fault at Phase 3. I tried again but got the same segfault. 17 18 After a few days of digging around and trying different things I decided the effort wasn't worth it. Reluctantly I accepted my losses and started the recovery. 19 20 Once the RAID array was reformatted I began the data copy from my external drive. This put me back to when it was last backed up. Then I could `rclone` my nightly backups from the last time it ran (before the corruption) and bring that data up to date. 21 22 This got me to a relatively good position. Okay I had lost some random downloads and a little bit of code that hadn't been pushed to my git server, but nothing serious. It is a little disappointing though, my backup setup is not good enough. 23 24 I would like to give [zfs](https://en.wikipedia.org/wiki/ZFS){target="_blank" rel="noreferrer"} a try, or even attempt a mini [ceph](https://ceph.io/en/){target="_blank" rel="noreferrer"} setup, but that would need some planning and some equipment purchases. I need something in the interim. 25 26 An external drive was purchased, which now sits permanently plugged into my PC. Instead of using `dedup` again I opted for an alternative tool, deciding on [BorgBackup](https://www.borgbackup.org/){target="_blank" rel="noreferrer"}. 27 28 After installing `borg` I initialised a new repo and kicked off a full backup 29 ``` 30 ──── ─ borg init -e repokey /media/backup/borg-kinakuta 31 ──── ─ borg create -v --stats /media/backup/borg-kinakuta::$(date +%Y%m%d) $HOME 32 ``` 33 34 The first time this ran it froze my system after about an hour and a half. 35 36 On the second attempt it did the same thing. This was frustrating, and looking at some help online there was indication of a bad disk. 37 38 When I rebooted after the second freeze my system dropped into maintenance mode, unable to mount /home. I checked the RAID array and sure enough one of the disks was missing. I reassembled the array ignoring the faulty disk and got back up and running, then ordered another drive.. 39 40 I decided to give `borg` another shot so kicked of the backup again. This time it succeeded. 41 ``` 42 ──── ─ borg create -v --stats /media/backup/borg-kinakuta::$(date +%Y%m%d) $HOME 43 Enter passphrase for key /media/backup/borg-kinakuta: 44 Creating archive at "/media/backup/borg-kinakuta::20221023" 45 ------------------------------------------------------------------------------ 46 Repository: /media/backup/borg-kinakuta 47 Archive name: 20221023 48 Archive fingerprint: 125d8f26a952dadb0053e17c8c73bb70852f509c3db4b340021c99f5f8daa8ff 49 Time (start): Sun, 2022-10-23 10:58:05 50 Time (end): Sun, 2022-10-23 20:34:51 51 Duration: 9 hours 36 minutes 46.09 seconds 52 Number of files: 1343775 53 Utilization of max. archive size: 0% 54 ------------------------------------------------------------------------------ 55 Original size Compressed size Deduplicated size 56 This archive: 2.36 TB 2.18 TB 1.98 TB 57 All archives: 2.36 TB 2.18 TB 1.98 TB 58 59 Unique chunks Total chunks 60 Chunk index: 1620226 2214299 61 ------------------------------------------------------------------------------ 62 ``` 63 64 Nine and a half hours was quicker than I was expecting. Over the next couple of days I ran backups each evening after I finished work. 65 66 ``` 67 ──── ─ borg create -v --stats /media/backup/borg-kinakuta::$(date +%Y%m%d) $HOME 68 Enter passphrase for key /media/backup/borg-kinakuta: 69 Creating archive at "/media/backup/borg-kinakuta::20221024" 70 ------------------------------------------------------------------------------ 71 Repository: /media/backup/borg-kinakuta 72 Archive name: 20221024 73 Archive fingerprint: ec9b9744bcc1baf896eb2b68c14278128688b2e4100172051bc839e57d02fd03 74 Time (start): Mon, 2022-10-24 17:35:58 75 Time (end): Mon, 2022-10-24 17:50:05 76 Duration: 14 minutes 7.59 seconds 77 Number of files: 1348078 78 Utilization of max. archive size: 0% 79 ------------------------------------------------------------------------------ 80 Original size Compressed size Deduplicated size 81 This archive: 2.36 TB 2.18 TB 325.62 MB 82 All archives: 4.72 TB 4.37 TB 1.98 TB 83 84 Unique chunks Total chunks 85 Chunk index: 1627368 4419526 86 ------------------------------------------------------------------------------ 87 ``` 88 89 ``` 90 ──── ─ borg create -x -v --stats /media/backup/borg-kinakuta::$(date +%Y%m%d) $HOME 91 Enter passphrase for key /media/backup/borg-kinakuta: 92 Creating archive at "/media/backup/borg-kinakuta::20221025" 93 ------------------------------------------------------------------------------ 94 Repository: /media/backup/borg-kinakuta 95 Archive name: 20221025 96 Archive fingerprint: 2ff6f558ca5e93f1c0dbfce317deada9d522e5a43538c05f38dea560e110bf9f 97 Time (start): Tue, 2022-10-25 17:32:16 98 Time (end): Tue, 2022-10-25 17:44:37 99 Duration: 12 minutes 20.92 seconds 100 Number of files: 1202775 101 Utilization of max. archive size: 0% 102 ------------------------------------------------------------------------------ 103 Original size Compressed size Deduplicated size 104 This archive: 2.35 TB 2.18 TB 277.59 MB 105 All archives: 7.08 TB 6.55 TB 1.98 TB 106 107 Unique chunks Total chunks 108 Chunk index: 1634630 6477362 109 ------------------------------------------------------------------------------ 110 ``` 111 112 Fourteen and twelve minutes to backup changes is great. I decided to leave it for two days and observe the time again. 113 ``` 114 ------------------------------------------------------------------------------ 115 Repository: /media/backup/borg-kinakuta 116 Archive name: 20221027 117 Archive fingerprint: a3cdca656ee4ee62a31cc135f794ef7de71b62ce7246496908ae529356092b97 118 Time (start): Thu, 2022-10-27 17:41:28 119 Time (end): Thu, 2022-10-27 17:54:23 120 Duration: 12 minutes 54.20 seconds 121 Number of files: 1208594 122 Utilization of max. archive size: 0% 123 ------------------------------------------------------------------------------ 124 Original size Compressed size Deduplicated size 125 This archive: 2.35 TB 2.18 TB 643.01 MB 126 All archives: 9.43 TB 8.72 TB 1.99 TB 127 128 Unique chunks Total chunks 129 Chunk index: 1647966 8540968 130 ------------------------------------------------------------------------------ 131 ``` 132 133 Almost thirteen minutes for two days of changes, pretty good. 134 135 I am really happy with these results from `borg`. When I get chance the next step is to play around with [borgmatic](https://torsion.org/borgmatic/){target="_blank" rel="noreferrer"} to automate the backups. 136 137 Another full backup will still be done to the drive in my bug out bag, I just have to be better at doing it more regularly. At least now if I need to restore I will be able to recover all of $HOME and not only the important things. 138 139 My PC has also been moved back onto the desk, away from little button pushers.