pyratelog

personal blog
git clone git://git.pyratebeard.net/pyratelog.git
Log | Files | Refs | README

commit 6e4550a842db49dc9d099df2a0c3dffe032d5a35
parent c2fdefcc9f91bcf08fc0ebd832a3e98f0b68e376
Author: pyratebeard <root@pyratebeard.net>
Date:   Sun, 23 Oct 2022 22:16:43 +0100

smoke_me_a_kipper

Diffstat:
Mentry/smoke_me_a_kipper.md | 34++++++++++++++++++++++++++--------
1 file changed, 26 insertions(+), 8 deletions(-)

diff --git a/entry/smoke_me_a_kipper.md b/entry/smoke_me_a_kipper.md @@ -2,9 +2,9 @@ Earlier this year I wrote about my [backup setup](20220414-speak_of_the_dedup.html) and recently I had to put it to the test. -My PC is a tower that I have on a small stand next to my desk. In the past I had kept the case (an Antec 1200) on my desk but it is rather large and dominates the space a bit too much, I don't have a very big desk. The other day my 1 year old toddled into the study and started pushing the power button on my PC. This power cycled the machine a few times in quick succession. At the time I wasn't aware of this. The next morning I booted up my PC but noticed it was very sluggish. It crashed trying to open my browser. After it happened again I started digging through the logs and noticed some filesystem corruption. +My PC is a tower that I have on a small stand next to my desk. In the past I had kept the case (an Antec 1200) on my desk but it is rather large and dominates the space a bit too much, I don't have a very big desk. The other day my 1 year old toddled into the study and started pushing the power button on my PC. This power cycled the machine a few times in quick succession. This was unknown to me until the next morning when I booted up my PC but noticed it was very sluggish. It crashed trying to open my browser. After it happened again I started digging through the logs and noticed some filesystem corruption. -As I described in my "speak_of_the_dedup" post I have a 3 disk RAID array as my $HOME. Because of the size I only nightly backup important documents, etc. A full backup is done periodically to an external drive I keep in my bug out bag. Unfortunately I had not done a full back in a while, but I knew my nightly backups were good so nothing too important was lost. +As I described in my backup setup post, I have a 3 disk RAID 5 array as my $HOME. Because of the size I only nightly backup important documents, etc. A full backup is done periodically to an external drive I keep in my bug out bag. Unfortunately I had not done a full back in a while, but I knew my nightly backups were good so nothing too important was lost. I had used xfs on my $HOME, so I unmounted the device and started an `xfs_repair`. The repair tool very quickly got to Phase 3, showing the output ``` @@ -19,18 +19,36 @@ After a few days of digging around and trying different things I decided the eff Once the RAID array was reformatted I began the data copy from my external drive. This put me back to when it was last backed up. Then I could `rclone` my nightly backups from the last time it ran (before the corruption) and bring that data up to date. -This got me to a relatively good position. Okay I had lost some random downloads, and a little bit of code that hadn't been pushed to my git server, but nothing serious. It is a little disappointing though, my backup setup is not good enough. +This got me to a relatively good position. Okay I had lost some random downloads and a little bit of code that hadn't been pushed to my git server, but nothing serious. It is a little disappointing though, my backup setup is not good enough. -I decided I needed more regular backups of my $HOME, so I needed some more storage. I purchased another external drive which now sits permanently plugged into my PC. I was going to use `dedup` again but decided it would be better to use an alternative so I am not relying on only one tool. I opted for [borg](TK){target="_blank" rel="noreferrer"}. +Rebuilding my whole setup to use [zfs](TK) is on my todo list, but right now I wanted a quick(er) fix without too much change. + +I decided I needed more regular backups of my $HOME, so I needed some more storage. I purchased another external drive which now sits permanently plugged into my PC. Instead of using `dedup` again I opted for an alternative tool, in this case I went with [borg](TK){target="_blank" rel="noreferrer"}. After installing `borg` I initialised a new repo and kicked off a full backup ``` - ──── ─ borg init -e repokey /media/backup/new_repo - ──── ─ borg create -v --stats /media/backup/new_repo::backup1 $HOME + ──── ─ borg init -e repokey /media/backup/kinakuta + ──── ─ borg create -v --stats /media/backup/kinakuta::$(date +%Y%m%d) $HOME +``` + +The first time this ran it froze my system after about an hour and a half. + +On the second attempt it did the same thing. This was frustrating, and looking at some help online there was indication of a bad disk. + +When I rebooted after the second freeze my system dropped into maintenance mode, unable to mount /home. I checked the RAID array and sure enough one of the disks was missing. I reassembled the array ignoring the faulty disk and got back up and running. + +I decided to give `borg` another shot so kicked of the backup again. This time it succeeded. +``` +``` + +Nine and a half hours was quicker than I was expecting. Over the next few days I ran backups after I finished work. +``` +``` + +``` ``` -The first backup did take a long time, but now each evening I can run `borg` to backup my $HOME to the external drive. -and `rclone` that latest archive to the cloud storage. +I am really happy with these results from `borg`. The next step is to play around with [borgmatic](TK){target="_blank" rel="noreferrer"} when I get a chance. Another full backup will still be done to the drive in my bug out bag, I just have to be better at doing it more regularly. At least now if I need to restore I will be able to recover all of $HOME and not only the important things.