Slitaz project – Part 2 – Cluster storage nodes (GlusterFS)

–This article continues from the base.7z made in ‘Slitaz project – Part 1 – Building a base system‘.

Anyway, now that our base is made, it’s time to do something with it.

In this part I will create four GlusterFS nodes that will stripe the files across each server. GlusterFS defines a minimum hardware of 1 GB of RAM and 8 GB of disk space. Soooo… about the requirements: I’ll add a new disk of 8GB, but the memory limitations. I like a challenge, 64 Megs ought to be enough (for now).

So just copy the base system. And let’s rename it to ‘glusternode1’. Add a new 8 GB disk (or more) to it. Again IDE! and boot the machine.

Remember the ip script? Time to put it to some use. The first thing I like to do is assign a static IP so I can use putty to connect to it. After boot just type:

/home/base/ip.sh glusternode1 192.168.1 221 1

Reboot the machine and voila. For the gluster nodes I intend to use the following ip’s:

glusternode1       192.168.1.221
glusternode2       192.168.1.222
glusternode3       192.168.1.223
glusternode4       192.168.1.224

Ok so, time to format the disk that we’ll use for our storage and assign it.

fdisk /dev/hdb

For convenience: press o, n, p, 1, enter, enter, w
Format it using ext2:

mkfs.ext2 -b 4096 /dev/hdb1

Create the mount point:

mkdir /mnt/data

And add the disk to fstab.

nano /etc/fstab
dev/hdb1	/mnt/data	ext2	defaults	0	0

Once the server is rebooted, the disk should just pop up.

Now that our disk is ready, we need to install GlusterFS. Before we can build GlusterFS from source, we need some dependencies:

tazpkg get-install flex 
tazpkg get-install python
tazpkg get-install readline-dev
tazpkg get-install mpc-library
tazpkg get-install elfutils
tazpkg get-install openssl-dev
tazpkg get-install slitaz-toolchain

Just press ‘y’ to all additional dependencies (there are a lot of these).

Also, I like things clean.

tazpkg clean-cache

As of writing, the latest version of GlusterFS is 3.3.0 so I will be using this verison. Feel free to use a newer version and let me know if it still works.

I’ll be working in the base folder (for the convenience). Just download and extract GlusterFS.

cd /home/base
wget http://download.gluster.org/pub/gluster/glusterfs/3.3/3.3.0/glusterfs-3.3.0.tar.gz
tar xzvf /home/base/glusterfs-3.3.0.tar.gz
cd /home/base/glusterfs-3.3.0

Let’s configure our source:

./configure

After configure you should get a result like this:

GlusterFS configure summary
===========================
FUSE client        : yes
Infiniband verbs   : no
epoll IO multiplex : yes
argp-standalone    : no
fusermount         : no
readline           : yes
georeplication     : yes

Now let’s make GlusterFS and install it.

make
make install

Cleaning up:

cd /
rm -rf /home/base/glusterfs-3.3.0
rm /home/base/glusterfs-3.3.0.tar.gz

To follow the same conventions like lighttpd and samba I created a deamon script. You can download or create it yourself. This script can be used to start GlusterFS as a deamon.

Option 1: download

wget -O /etc/init.d/glusterd http://www.enira.net/wp-content/uploads/2012/06/glusterd.txt

Option 2: create

nano /etc/init.d/glusterd
#!/bin/sh
# /etc/init.d/glusterd: Start, stop and restart web server on SliTaz,
# at boot time or with the command line. Daemons options are configured
# with /etc/daemons.conf
#
. /etc/init.d/rc.functions
. /etc/daemons.conf
 
NAME=GlusterFS
DESC="gluster deamon"
DAEMON=/usr/local/sbin/glusterd
OPTIONS=$GLUSTERFS_OPTIONS
PIDFILE=/var/run/glusterd.pid
 
case "$1" in
  start)
    if active_pidfile $PIDFILE glusterd ; then
      echo "$NAME already running."
      exit 1
    fi
    echo -n "Starting $DESC: $NAME... "
    $DAEMON $OPTIONS
    status
    ;;
  stop)
    if ! active_pidfile $PIDFILE glusterd ; then
      echo "$NAME is not running."
      exit 1
    fi
    echo -n "Stopping $DESC: $NAME... "
    kill `cat $PIDFILE`
    rm $PIDFILE
    status
    ;;
  restart)
    if ! active_pidfile $PIDFILE glusterd ; then
      echo "$NAME is not running."
      exit 1
    fi
    echo -n "Restarting $DESC: $NAME... "
    kill `cat $PIDFILE`
    rm $PIDFILE
    sleep 2
    $DAEMON $OPTIONS
    status
    ;;
  *)
    echo ""
    echo -e "\033[1mUsage:\033[0m /etc/init.d/`basename $0` [start|stop|restart]"
    echo ""
    exit 1
    ;;
esac
 
exit 0

So add execute rights to the newly created script.

chmod +x /etc/init.d/glusterd

I noticed that it seems impossible to generate a PID for this executable. Luckily GlusterFS can make it’s own with the ‘–pid-file’ option. So before it can be used as a startup deamon you need to pass this as an optional paramater. These are found in daemons.conf.

nano /etc/daemons.conf

Just add these lines at the bottom:

# GlusterFS
GLUSTERFS_OPTIONS="--pid-file=/var/run/glusterd.pid"

And let’s add it as a deamon service to the file rcS.conf:

nano /etc/rcS.conf

Just search the line ‘RUN_DEAMONS’ and add ‘glusterd’ to it.

RUN_DAEMONS="dbus hald slim firewall dropbear lighttpd glusterd"

Now reboot your server and you should see a glusterd process.

root@glusternode1:~# ps aux | grep gluster
 1441 root       0:00 /usr/local/sbin/glusterd --pid-file=/var/run/glusterd.pid
 1496 root       0:00 grep gluster

Also this first start generates the ‘/var/lib/glusterd’ which we need.

To identify each gluster node they have to have their own unique uuid. Because if we copy the glusternode1 3 times, each server will list the same uuid, causing gluster to think all four machines are localhost!
This uuid can be found in the file ‘/var/lib/glusterd/glusterd.info’. Let’s assign a random one for ‘glusternode1’.

echo UUID=`uuidgen -r` > /var/lib/glusterd/glusterd.info

Finished with glusternode1! Phew. 3 to go. Shutdown the glusternode1 (tip:halt) and copy it three times(glusternode2,3,4).

Now let’s start assigning the IP’s and random uuid’s beginning with the last cluster. (To avoid ip conflicts.)

/home/base/ip.sh glusternode4 192.168.1 224 1
echo UUID=`uuidgen -r` > /var/lib/glusterd/glusterd.info
reboot
/home/base/ip.sh glusternode3 192.168.1 223 1
echo UUID=`uuidgen -r` > /var/lib/glusterd/glusterd.info
reboot
/home/base/ip.sh glusternode2 192.168.1 222 1
echo UUID=`uuidgen -r` > /var/lib/glusterd/glusterd.info
reboot

Now we have four nodes who know nothing about each other. To connect em all I’ll be working on glusternode1.

Just probe each node and they will synch configuration.

gluster peer probe 192.168.1.222
gluster peer probe 192.168.1.223
gluster peer probe 192.168.1.224

You can verify the connection with the ‘peer status’ command.

gluster peer status

Your output on glusternode1 should sort of look like this (uuid’s will vary):

root@glusternode1:~# gluster peer status
Number of Peers: 3
 
Hostname: 192.168.1.222
Uuid: efabc10f-5830-408d-aa07-2c05d4f074ef
State: Peer in Cluster (Connected)
 
Hostname: 192.168.1.223
Uuid: 5241b592-477f-4c5f-bb15-4b13fc48903b
State: Peer in Cluster (Connected)
 
Hostname: 192.168.1.224
Uuid: a54bfa2c-c1b7-4e9f-aaeb-18f45f7d66bc
State: Peer in Cluster (Connected)

If the output is similar like above, we can start creating a volume. I’ll be using the default striping. This means that all files will be spread on the cluster. There are various other possibilities but I won’t handle these.
Following command will create the storage cluster:

gluster volume create slitaz-volume 192.168.1.221:/mnt/data 192.168.1.222:/mnt/data 192.168.1.223:/mnt/data 192.168.1.224:/mnt/data

Offcourse one of the drawbacks of using such a small amount of RAM (64 MB) is that gluster will shit himself trying to load. Default glusterfs tries to use 64MB, wich is too much for the machine to handle. So we’ll need to tweak the cache-size. I tested a few values and 16MB seems to work best. (4MB and 8MB are too small.)

gluster volume set slitaz-volume cache-size 16MB

So normally everything should be ok now and a ‘volume info’ command should succeed.

gluster volume info
root@glusternode1:/mnt/data# gluster volume info
 
Volume Name: slitaz-volume
Type: Distribute
Volume ID: c3262294-869b-48b9-a9fb-1d95167fe182
Status: Created
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: 192.168.1.221:/mnt/data
Brick2: 192.168.1.222:/mnt/data
Brick3: 192.168.1.223:/mnt/data
Brick4: 192.168.1.224:/mnt/data
Options Reconfigured:
performance.cache-size: 16MB

Notice the ‘created’ status, this means our volume isn’t started yet. Starting can be done with:

gluster volume start slitaz-volume

Voila, your server nodes are finished and ready to accept files.

And like previous post. You can download the vmwares (all 4) here: glusternodes.7z (156.0 MB)

Leave a Reply