enira.net

We are all living in a technological wasteland.

RSS
people
Dec 07 2015

NAS – Next-gen filesystems – BTRFS RAID 1

Setting up BTRFS and upgrading the kernel

My test setup is based on Ubuntu 14.04.3 LTS, this version is still using 3.19 kernel, for BTRFS it’s better to use a newer stable version so we will update the kernel. In my case I will be updating to kernel 4.1.13. (At the moment of testing this is thle latest stable: https://www.kernel.org/)
Download the header files from Ubuntu

wget kernel.ubuntu.com/~kernel-ppa/mainline/v4.1.13-wily/linux-headers-4.1.13-040113_4.1.13-040113.201511092325_all.deb 
wget kernel.ubuntu.com/~kernel-ppa/mainline/v4.1.13-wily/linux-headers-4.1.13-040113-generic_4.1.13-040113.201511092325_amd64.deb 
wget kernel.ubuntu.com/~kernel-ppa/mainline/v4.1.13-wily/linux-image-4.1.13-040113-generic_4.1.13-040113.201511092325_amd64.deb

And let’s install

sudo dpkg -i linux-headers-4.1*.deb linux-image-4.1*.deb

Once completed reboot, to see if the kernel has been applied.

sudo reboot

Checking the kernel can be done 

uname -r

Next install the ‘btrfs-tools’ package.

sudo apt-get install btrfs-tools

For convenience I am installing ‘samba’ too. 

sudo apt-get install samba

Create the password.

sudo smbpasswd -a btrfs

Disk setup

Let’s add three disks to our virtual machine. In this example I’ve added three 5GB disks.

lsblk

NAME                        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                           8:0    0     6G  0 disk
├─sda1                        8:1    0   243M  0 part /boot
├─sda2                        8:2    0     1K  0 part
└─sda5                        8:5    0   5.8G  0 part
  ├─btrfs--vg-root (dm-0)   252:0    0   4.8G  0 lvm  /
  └─btrfs--vg-swap_1 (dm-1) 252:1    0     1G  0 lvm  [SWAP]
sdb                           8:16   0     5G  0 disk
sdc                           8:32   0     5G  0 disk
sdd                           8:48   0     5G  0 disk

Test One: creating a RAID1

Let’s create a RAID1 disk spanning sdb & sdc. If you are using disks larger than ~16GB, please don’t use ‘–mixed’: See Intermezzo 1

sudo mkfs.btrfs -d raid1 -m raid1 -L disk-raid1 --mixed /dev/sdb /dev/sdc

Intermezzo 1: why ‘–mixed’?

To test the option ‘–mixed’ I ve created a setup without using this option. I’ve filled it up with a lot of data. And we’ll see what happens.

df -h

Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/btrfs--vg-root  4.6G  1.9G  2.5G  44% /
none                        4.0K     0  4.0K   0% /sys/fs/cgroup
udev                        477M  8.0K  477M   1% /dev
tmpfs                        98M  1.3M   97M   2% /run
none                        5.0M     0  5.0M   0% /run/lock
none                        488M     0  488M   0% /run/shm
none                        100M     0  100M   0% /run/user
/dev/sda1                   236M  100M  124M  45% /boot
/dev/sdc                    5.0G  3.8G  203M  96% /media/btrfs-raid1

So after 3.8GB the disk already lists quite full. Quite odd no? The disk size should somewhere around 5GB.
Let’s see what btrfs shows.

sudo btrfs filesystem df /media/btrfs-raid1

Data, RAID1: total=3.97GiB, used=3.77GiB
Data, single: total=8.00MiB, used=0.00
System, RAID1: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, RAID1: total=1.00GiB, used=4.23MiB
Metadata, single: total=8.00MiB, used=0.00
unknown, single: total=16.00MiB, used=0.00

So what you will see is that the Metadata is taking up 1GB (Metadata, RAID1: total=1.00GiB). So this means that 1GB of the RAID is unusable. Hence, 3.8GB + 1GB ~= 5GB.
Let’s try to balance.

sudo btrfs balance start -v /media/btrfs-raid1

Dumping filters: flags 0x7, state 0x0, force is off
  DATA (flags 0x0): balancing
  METADATA (flags 0x0): balancing
  SYSTEM (flags 0x0): balancing
ERROR: error during balancing '/media/btrfs-raid1' - No space left on device

Apparently it doesn’t work :(.

More info: https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_I_ran_out_of_disk_space.21

/end intermezzo

Let’s continue.
Our new device is now working on sdb & sdc. The ‘show’ command should show a little bit more information.

sudo btrfs fi show

Label: 'disk-raid1'  uuid: 07759bba-2b6b-4d9a-b09f-605acbd6da0b
        Total devices 2 FS bytes used 28.00KiB
        devid    1 size 5.00GiB used 1.02GiB path /dev/sdb
        devid    2 size 5.00GiB used 1.01GiB path /dev/sdc

Now let’s add this to our fstab file.

sudo nano /etc/fstab

Fstab allows to use ‘btrfs’ as mount type. So we will use this.

UUID=07759bba-2b6b-4d9a-b09f-605acbd6da0b /media/btrfs-raid1          btrfs defaults 0       0

Create our mountpoint.

sudo mkdir -p /media/btrfs-raid1

Now reboot and see if our fstab works.

sudo reboot

df -h

Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/btrfs--vg-root  4.6G  1.9G  2.5G  44% /
none                        4.0K     0  4.0K   0% /sys/fs/cgroup
udev                        477M  4.0K  477M   1% /dev
tmpfs                        98M  1.2M   97M   2% /run
none                        5.0M     0  5.0M   0% /run/lock
none                        488M     0  488M   0% /run/shm
none                        100M     0  100M   0% /run/user
/dev/sda1                   236M  100M  124M  45% /boot
/dev/sdc                    5.0G   17M  4.0G   1% /media/btrfs-raid1

Configuring Samba

Next create a samba share to add some test data.

sudo nano /etc/samba/smb.conf

[btrfs-raid1]
   comment = Test BTRFS RAID 1
   browseable = yes
   path = /media/btrfs-raid1
   valid users = btrfs
   writable = yes

Change ownership.

sudo chown -R btrfs:btrfs /media/btrfs-raid1/

sudo service smbd restart

Let’s create an MD5 sum of my copied files to verify the file integrity later on.

md5sum /media/btrfs-raid1/*

03486548bc7b0f1a3881dc00c0f8c5f8  /media/btrfs-raid1/S01E01 HDTV x264.mp4
a9390aed84a6be8c145046772296db26  /media/btrfs-raid1/S01E02 HDTV x264.mp4
2e37ed514579ac282986efd78ac3bb76  /media/btrfs-raid1/S01E03 HDTV x264.mp4
1596a5e56f14c843b5c27e2d3ff27ebd  /media/btrfs-raid1/S01E04 HDTV x264.mp4
f7d494d6858391ac5c312d141d9ee0e5  /media/btrfs-raid1/S01E05 HDTV x264.mp4
fe6f097ff136428bfc3e2a1b8e420e4e  /media/btrfs-raid1/S01E06 HDTV x264.mp4
43c5314079f08570f6bb24b5d6fde101  /media/btrfs-raid1/S01E07 HDTV x264.mp4
3b5ea952b632bbc58f608d64667cd2a1  /media/btrfs-raid1/S01E08 HDTV x264.mp4
db6b8bf608de2008455b462e76b0c1dd  /media/btrfs-raid1/S01E09 HDTV x264.mp4
0d5775373e1168feeef99889a1d8fe0a  /media/btrfs-raid1/S01E10 HDTV x264.mp4
8dd4b25c249778f197fdb33604fdb998  /media/btrfs-raid1/S01E11 HDTV x264.mp4

Test 2 Replacing a disk

Let’s replace sdb with sdd in our setup.

sdd is also a 5GB disk so this should not pose any problems.

lsblk

NAME                        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                           8:0    0     6G  0 disk
├─sda1                        8:1    0   243M  0 part /boot
├─sda2                        8:2    0     1K  0 part
└─sda5                        8:5    0   5.8G  0 part
  ├─btrfs--vg-root (dm-0)   252:0    0   4.8G  0 lvm  /
  └─btrfs--vg-swap_1 (dm-1) 252:1    0     1G  0 lvm  [SWAP]
sdb                           8:16   0     5G  0 disk
sdc                           8:32   0     5G  0 disk
sdd                           8:48   0     5G  0 disk

sudo btrfs fi show

Label: disk-raid1  uuid: 07759bba-2b6b-4d9a-b09f-605acbd6da0b
        Total devices 2 FS bytes used 4.68GiB
        devid    1 size 5.00GiB used 5.00GiB path /dev/sdb
        devid    2 size 5.00GiB used 4.99GiB path /dev/sdc

So lets start replacing. The command to replace a disk is ‘replace’.

sudo btrfs replace start /dev/sdb /dev/sdd /media/btrfs-raid1

During the replace you will see an attached ‘devid’. This will be there during the replace.

sudo btrfs fi show

Label: disk-raid1  uuid: 07759bba-2b6b-4d9a-b09f-605acbd6da0b
        Total devices 3 FS bytes used 4.68GiB
        devid    1 size 5.00GiB used 5.00GiB path /dev/sdb
        devid    2 size 5.00GiB used 4.99GiB path /dev/sdc
        devid    0 size 0.00 used 0.00 path

To check the replace status run ‘btrfs replace status’.

sudo btrfs replace status /media/btrfs-raid1

31.2% done, 0 write errs, 0 uncorr. read errs

Once everything is done the filesystem should reflect our desired new configuration.

sudo btrfs fi show

Label: disk-raid1  uuid: 07759bba-2b6b-4d9a-b09f-605acbd6da0b
        Total devices 2 FS bytes used 4.68GiB
        devid    1 size 5.00GiB used 5.00GiB path /dev/sdd
        devid    2 size 5.00GiB used 4.99GiB path /dev/sdc

df -h

Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/btrfs--vg-root  4.6G  1.9G  2.5G  44% /
none                        4.0K     0  4.0K   0% /sys/fs/cgroup
udev                        477M   12K  477M   1% /dev
tmpfs                        98M  1.3M   97M   2% /run
none                        5.0M     0  5.0M   0% /run/lock
none                        488M     0  488M   0% /run/shm
none                        100M     0  100M   0% /run/user
/dev/sda1                   236M  100M  124M  45% /boot
/dev/sdc                    5.0G  4.8G  307M  95% /media/btrfs-raid1

Let’s verify our MD5 sums again to see if everything is still consistent.

md5sum /media/btrfs-raid1/*

03486548bc7b0f1a3881dc00c0f8c5f8  /media/btrfs-raid1/S01E01 HDTV x264.mp4
a9390aed84a6be8c145046772296db26  /media/btrfs-raid1/S01E02 HDTV x264.mp4
2e37ed514579ac282986efd78ac3bb76  /media/btrfs-raid1/S01E03 HDTV x264.mp4
1596a5e56f14c843b5c27e2d3ff27ebd  /media/btrfs-raid1/S01E04 HDTV x264.mp4
f7d494d6858391ac5c312d141d9ee0e5  /media/btrfs-raid1/S01E05 HDTV x264.mp4
fe6f097ff136428bfc3e2a1b8e420e4e  /media/btrfs-raid1/S01E06 HDTV x264.mp4
43c5314079f08570f6bb24b5d6fde101  /media/btrfs-raid1/S01E07 HDTV x264.mp4
3b5ea952b632bbc58f608d64667cd2a1  /media/btrfs-raid1/S01E08 HDTV x264.mp4
db6b8bf608de2008455b462e76b0c1dd  /media/btrfs-raid1/S01E09 HDTV x264.mp4
0d5775373e1168feeef99889a1d8fe0a  /media/btrfs-raid1/S01E10 HDTV x264.mp4
8dd4b25c249778f197fdb33604fdb998  /media/btrfs-raid1/S01E11 HDTV x264.mp4

Test 3: Killing one off

In this case whe physically (or virtually disconnect a disk and see what happens and repair it)
So when booting you will see the ‘An error occured while mounting /media/btrfs-raid1’. Press s to skip.
The program ‘lsblk’ will list sdd missing. This is normal.

lsblk
NAME                        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                           8:0    0     6G  0 disk
├─sda1                        8:1    0   243M  0 part /boot
├─sda2                        8:2    0     1K  0 part
└─sda5                        8:5    0   5.8G  0 part
  ├─btrfs--vg-root (dm-0)   252:0    0   4.8G  0 lvm  /
  └─btrfs--vg-swap_1 (dm-1) 252:1    0     1G  0 lvm  [SWAP]
sdb                           8:16   0     5G  0 disk
sdc                           8:32   0     5G  0 disk

So let’s check the status of our BTRFS filesystem again.

sudo btrfs fi show

Label: 'disk-raid1'  uuid: 07759bba-2b6b-4d9a-b09f-605acbd6da0b
        Total devices 2 FS bytes used 4.68GiB
        devid    1 size 5.00GiB used 5.00GiB path /dev/sdc
        *** Some devices missing

Interesting, we see some are missing, but not which one.

So let’s try to mount our filesystem.

sudo mount -v -t btrfs LABEL=disk-raid1 /media/btrfs-raid1/

This won’t work because there is one drive missing. We will need to mount our filesystem in degraded mode.

sudo mount -v -t btrfs -o degraded LABEL=disk-raid1 /media/btrfs-raid1/

So once we are in degraded mode we will see a little bit more. We can see the devid, but we won’t see the drive. (As these is gone)

sudo btrfs fi show

Label: disk-raid1  uuid: 07759bba-2b6b-4d9a-b09f-605acbd6da0b
        Total devices 2 FS bytes used 4.68GiB
        devid    1 size 5.00GiB used 5.00GiB path /dev/sdc
        devid    2 size 5.00GiB used 4.99GiB path

So let’s verify our content first (in degraded mode).

md5sum /media/btrfs-raid1/*

03486548bc7b0f1a3881dc00c0f8c5f8  /media/btrfs-raid1/S01E01 HDTV x264.mp4
a9390aed84a6be8c145046772296db26  /media/btrfs-raid1/S01E02 HDTV x264.mp4
2e37ed514579ac282986efd78ac3bb76  /media/btrfs-raid1/S01E03 HDTV x264.mp4
1596a5e56f14c843b5c27e2d3ff27ebd  /media/btrfs-raid1/S01E04 HDTV x264.mp4
f7d494d6858391ac5c312d141d9ee0e5  /media/btrfs-raid1/S01E05 HDTV x264.mp4
fe6f097ff136428bfc3e2a1b8e420e4e  /media/btrfs-raid1/S01E06 HDTV x264.mp4
43c5314079f08570f6bb24b5d6fde101  /media/btrfs-raid1/S01E07 HDTV x264.mp4
3b5ea952b632bbc58f608d64667cd2a1  /media/btrfs-raid1/S01E08 HDTV x264.mp4
db6b8bf608de2008455b462e76b0c1dd  /media/btrfs-raid1/S01E09 HDTV x264.mp4
0d5775373e1168feeef99889a1d8fe0a  /media/btrfs-raid1/S01E10 HDTV x264.mp4
8dd4b25c249778f197fdb33604fdb998  /media/btrfs-raid1/S01E11 HDTV x264.mp4

Our filesystem still shows that all our data is still intact. So let’s continue with replacing.

So how do we replace? Add and remove? Think again… This will destroy your raid. Unless you have less than 50% used.

Luckily replace works with devids too. (As there is no /dev/ mapping anymore.) In our case the missing devid is ‘2’.

sudo btrfs replace start 2 /dev/sdb /media/btrfs-raid1

So let’s check if everything is working. Whilst replacing devid 2 will still be listed.

sudo btrfs fi show

Label: disk-raid1  uuid: 07759bba-2b6b-4d9a-b09f-605acbd6da0b
        Total devices 3 FS bytes used 4.68GiB
        devid    1 size 5.00GiB used 5.00GiB path /dev/sdc
        devid    2 size 5.00GiB used 4.99GiB path
        devid    0 size 0.00 used 0.00 path

To check the replace you can also ask the status.

sudo btrfs replace status /media/btrfs-raid1

Once done a show will show the repaired filesystem.

sudo btrfs fi show

Label: disk-raid1  uuid: 07759bba-2b6b-4d9a-b09f-605acbd6da0b
        Total devices 2 FS bytes used 4.68GiB
        devid    1 size 5.00GiB used 5.00GiB path /dev/sdc
        devid    2 size 5.00GiB used 4.99GiB path /dev/sdb

Let’s verify our file integrity once more.

md5sum /media/btrfs-raid1/*

03486548bc7b0f1a3881dc00c0f8c5f8  /media/btrfs-raid1/S01E01 HDTV x264.mp4
a9390aed84a6be8c145046772296db26  /media/btrfs-raid1/S01E02 HDTV x264.mp4
2e37ed514579ac282986efd78ac3bb76  /media/btrfs-raid1/S01E03 HDTV x264.mp4
1596a5e56f14c843b5c27e2d3ff27ebd  /media/btrfs-raid1/S01E04 HDTV x264.mp4
f7d494d6858391ac5c312d141d9ee0e5  /media/btrfs-raid1/S01E05 HDTV x264.mp4
fe6f097ff136428bfc3e2a1b8e420e4e  /media/btrfs-raid1/S01E06 HDTV x264.mp4
43c5314079f08570f6bb24b5d6fde101  /media/btrfs-raid1/S01E07 HDTV x264.mp4
3b5ea952b632bbc58f608d64667cd2a1  /media/btrfs-raid1/S01E08 HDTV x264.mp4
db6b8bf608de2008455b462e76b0c1dd  /media/btrfs-raid1/S01E09 HDTV x264.mp4
0d5775373e1168feeef99889a1d8fe0a  /media/btrfs-raid1/S01E10 HDTV x264.mp4
8dd4b25c249778f197fdb33604fdb998  /media/btrfs-raid1/S01E11 HDTV x264.mp4

Test 4: Redundancy check

For our last test we shall damage one of the vmdk files of the virtual machine. This can be done with wxHexEditor. Just open and replace with zeroes.

Once damaged let’s start scrubbing to see if we can detect some errors.

sudo btrfs scrub start /media/btrfs-raid1/

You can also request a status.

sudo watch btrfs scrub status /media/btrfs-raid1/

scrub status for 07759bba-2b6b-4d9a-b09f-605acbd6da0b
        scrub started at Fri Dec  4 15:28:30 2015 and finished after 29 seconds
        total bytes scrubbed: 9.36GiB with 7 errors
        error details: csum=7
        corrected errors: 7, uncorrectable errors: 0, unverified errors: 0

Or grep the results from syslog.

cat /var/log/syslog | grep BTRFS

Dec  4 15:28:52 btrfs kernel: [   70.751292] BTRFS: bdev /dev/sdb errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
Dec  4 15:28:52 btrfs kernel: [   70.799491] BTRFS: fixed up error at logical 3785674752 on dev /dev/sdb
Dec  4 15:28:55 btrfs kernel: [   73.465940] BTRFS: checksum error at logical 4282617856 on dev /dev/sdb, sector 8341960, root 5, inode 266, offset 512442368, length 4096, links 1 (path: S01E10 HDTV x264.mp4)
Dec  4 15:28:55 btrfs kernel: [   73.465946] BTRFS: bdev /dev/sdb errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
Dec  4 15:28:55 btrfs kernel: [   73.480159] BTRFS: fixed up error at logical 4282617856 on dev /dev/sdb
Dec  4 15:28:55 btrfs kernel: [   73.585207] BTRFS: checksum error at logical 4327469056 on dev /dev/sdb, sector 8429560, root 5, inode 266, offset 158330880, length 4096, links 1 (path: S01E10 HDTV x264.mp4)
Dec  4 15:28:55 btrfs kernel: [   73.585213] BTRFS: bdev /dev/sdb errs: wr 0, rd 0, flush 0, corrupt 5, gen 0
Dec  4 15:28:55 btrfs kernel: [   73.660404] BTRFS: fixed up error at logical 4327469056 on dev /dev/sdb
Dec  4 15:28:58 btrfs kernel: [   76.693662] BTRFS: checksum error at logical 4892540928 on dev /dev/sdb, sector 9533216, root 5, inode 267, offset 246824960, length 4096, links 1 (path: S01E11 HDTV x264.mp4)
Dec  4 15:28:58 btrfs kernel: [   76.693667] BTRFS: bdev /dev/sdb errs: wr 0, rd 0, flush 0, corrupt 6, gen 0
Dec  4 15:28:58 btrfs kernel: [   76.761209] BTRFS: fixed up error at logical 4892540928 on dev /dev/sdb
Dec  4 15:28:59 btrfs kernel: [   77.698926] BTRFS: checksum error at logical 5076742144 on dev /dev/sdb, sector 9892984, root 5, inode 267, offset 432074752, length 4096, links 1 (path: S01E11 HDTV x264.mp4)
Dec  4 15:28:59 btrfs kernel: [   77.698932] BTRFS: bdev /dev/sdb errs: wr 0, rd 0, flush 0, corrupt 7, gen 0
Dec  4 15:28:59 btrfs kernel: [   77.719481] BTRFS: fixed up error at logical 5076742144 on dev /dev/sdb

So this was every test I wanted to do for RAID1. Up next will be converting existing filesystems.

This entry was written by enira and posted on December 7, 2015 at 21:51 and filed under Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post.

Leave a Reply

You must be logged in to post a comment.