commit 6e4550a842db49dc9d099df2a0c3dffe032d5a35
parent c2fdefcc9f91bcf08fc0ebd832a3e98f0b68e376
Author: pyratebeard <root@pyratebeard.net>
Date: Sun, 23 Oct 2022 22:16:43 +0100
smoke_me_a_kipper
Diffstat:
1 file changed, 26 insertions(+), 8 deletions(-)
diff --git a/entry/smoke_me_a_kipper.md b/entry/smoke_me_a_kipper.md
@@ -2,9 +2,9 @@
Earlier this year I wrote about my [backup setup](20220414-speak_of_the_dedup.html) and recently I had to put it to the test.
-My PC is a tower that I have on a small stand next to my desk. In the past I had kept the case (an Antec 1200) on my desk but it is rather large and dominates the space a bit too much, I don't have a very big desk. The other day my 1 year old toddled into the study and started pushing the power button on my PC. This power cycled the machine a few times in quick succession. At the time I wasn't aware of this. The next morning I booted up my PC but noticed it was very sluggish. It crashed trying to open my browser. After it happened again I started digging through the logs and noticed some filesystem corruption.
+My PC is a tower that I have on a small stand next to my desk. In the past I had kept the case (an Antec 1200) on my desk but it is rather large and dominates the space a bit too much, I don't have a very big desk. The other day my 1 year old toddled into the study and started pushing the power button on my PC. This power cycled the machine a few times in quick succession. This was unknown to me until the next morning when I booted up my PC but noticed it was very sluggish. It crashed trying to open my browser. After it happened again I started digging through the logs and noticed some filesystem corruption.
-As I described in my "speak_of_the_dedup" post I have a 3 disk RAID array as my $HOME. Because of the size I only nightly backup important documents, etc. A full backup is done periodically to an external drive I keep in my bug out bag. Unfortunately I had not done a full back in a while, but I knew my nightly backups were good so nothing too important was lost.
+As I described in my backup setup post, I have a 3 disk RAID 5 array as my $HOME. Because of the size I only nightly backup important documents, etc. A full backup is done periodically to an external drive I keep in my bug out bag. Unfortunately I had not done a full back in a while, but I knew my nightly backups were good so nothing too important was lost.
I had used xfs on my $HOME, so I unmounted the device and started an `xfs_repair`. The repair tool very quickly got to Phase 3, showing the output
```
@@ -19,18 +19,36 @@ After a few days of digging around and trying different things I decided the eff
Once the RAID array was reformatted I began the data copy from my external drive. This put me back to when it was last backed up. Then I could `rclone` my nightly backups from the last time it ran (before the corruption) and bring that data up to date.
-This got me to a relatively good position. Okay I had lost some random downloads, and a little bit of code that hadn't been pushed to my git server, but nothing serious. It is a little disappointing though, my backup setup is not good enough.
+This got me to a relatively good position. Okay I had lost some random downloads and a little bit of code that hadn't been pushed to my git server, but nothing serious. It is a little disappointing though, my backup setup is not good enough.
-I decided I needed more regular backups of my $HOME, so I needed some more storage. I purchased another external drive which now sits permanently plugged into my PC. I was going to use `dedup` again but decided it would be better to use an alternative so I am not relying on only one tool. I opted for [borg](TK){target="_blank" rel="noreferrer"}.
+Rebuilding my whole setup to use [zfs](TK) is on my todo list, but right now I wanted a quick(er) fix without too much change.
+
+I decided I needed more regular backups of my $HOME, so I needed some more storage. I purchased another external drive which now sits permanently plugged into my PC. Instead of using `dedup` again I opted for an alternative tool, in this case I went with [borg](TK){target="_blank" rel="noreferrer"}.
After installing `borg` I initialised a new repo and kicked off a full backup
```
- ──── ─ borg init -e repokey /media/backup/new_repo
- ──── ─ borg create -v --stats /media/backup/new_repo::backup1 $HOME
+ ──── ─ borg init -e repokey /media/backup/kinakuta
+ ──── ─ borg create -v --stats /media/backup/kinakuta::$(date +%Y%m%d) $HOME
+```
+
+The first time this ran it froze my system after about an hour and a half.
+
+On the second attempt it did the same thing. This was frustrating, and looking at some help online there was indication of a bad disk.
+
+When I rebooted after the second freeze my system dropped into maintenance mode, unable to mount /home. I checked the RAID array and sure enough one of the disks was missing. I reassembled the array ignoring the faulty disk and got back up and running.
+
+I decided to give `borg` another shot so kicked of the backup again. This time it succeeded.
+```
+```
+
+Nine and a half hours was quicker than I was expecting. Over the next few days I ran backups after I finished work.
+```
+```
+
+```
```
-The first backup did take a long time, but now each evening I can run `borg` to backup my $HOME to the external drive.
-and `rclone` that latest archive to the cloud storage.
+I am really happy with these results from `borg`. The next step is to play around with [borgmatic](TK){target="_blank" rel="noreferrer"} when I get a chance.
Another full backup will still be done to the drive in my bug out bag, I just have to be better at doing it more regularly. At least now if I need to restore I will be able to recover all of $HOME and not only the important things.
