Setting up a network distributed file system, SSL/TLS encrypted, with GlusterFS on the server and clients.
GlusterFS is a network-attached storage file system like NFS but better and perfect for i.e. Webserver data or any other data storage for cloud services. The basic installation and setup is pretty much straight forward using our package manager. By default GlusterFS clients and server will communicate without any encryption. This short tutorial will show you how to quickly setup a basic client-server model with SSL/TLS enabled. Please note that further information to GlusterFS SSL/TLS can be found here.
You need to have or install at least two machines with Debian 8.x. If you don’t have two test servers yet, just login to your AWS account and create two instances with Debian 8.7 (https://wiki.debian.org/Cloud/AmazonEC2Image/Jessie).
For this example I use ap-southeast-2 (SYDNEY) ami-881317eb, t2.micro with the options “Auto-assign Public IP = enabled” and 8GB HDD. Please ensure that the security group/firewall rules you assign to the servers has nothing blocked for internal traffic (debug only).
Debian GlusterFS Master Server (gfs-master)
External IP: 18.104.22.168 Internal IP: 10.0.0.1 Connect command: ssh [email protected] -i mykey.pem SuperUser command: sudo su
Debian GlusterFS Client (gfs-client)
External IP: 22.214.171.124 Internal IP: 10.0.0.2 Connect command: ssh [email protected] -i mykey.pem SuperUser command: sudo su
First we update our machine, and the install GlusterFS client and server software.
ssh [email protected] -i mykey.pem sudo su apt-get update && apt-get upgrade apt-get install glusterfs-client glusterfs-server
Now we create our storage path on the server and a local mount directory
mkdir /srv/glusterfs/ mkdir /mnt/gfs-vol-test-1
Next we actually create a test volume called “gfs-vol-test-1”, start it and mount it.
gluster volume create gfs-vol-test-1 10.0.0.1:/srv/glusterfs/gfs-vol-test-1 force gluster volume start gfs-vol-test-1 mount.glusterfs 127.0.0.1:/gfs-vol-test-1 /mnt/gfs-vol-test-1
At last we create a test file in our new volume
echo 'hello gluster' > /mnt/gfs-vol-test-1/testglust.txt ls -l /mnt/gfs-vol-test-1/
First we update our machine, and the install the GlusterFS client software.
ssh [email protected] -i mykey.pem sudo su apt-get update && apt-get upgrade apt-get install glusterfs-client
First we create a local mount path
Now we can mount the volume and view our previous created text file from the master server
mount.glusterfs 10.0.0.1:/gfs-vol-test-1 /mnt/gfs-vol-test-1 ls -l /mnt/gfs-vol-test-1/ cat /mnt/gfs-vol-test-1/testglust.txt
We start again with the master server and have at first a look into the volume information
gluster volume info gfs-vol-test-1 > Volume Name: gfs-vol-test-1 > Type: Distribute > Volume ID: f3b5e3a2-a9a4-4099-b6cc-958c7d9572e0 > Status: Started > Number of Bricks: 1 > Transport-type: tcp > Bricks: > Brick1: 10.0.0.1:/srv/glusterfs/gfs-vol-test-1
This tells us that SSL/TLS is not yet setup.
For our example we create us our own local certificates:
openssl genrsa -out /etc/ssl/glusterfs.key 2048 openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj /CN=Anyone -out /etc/ssl/glusterfs.pem cp /etc/ssl/glusterfs.pem /etc/ssl/glusterfs.ca ls -l /etc/ssl/glusterfs* > /etc/ssl/glusterfs.ca > /etc/ssl/glusterfs.key > /etc/ssl/glusterfs.pem
Next we are actually enabling SSL/TLS support for clients and server:
gluster volume set gfs-vol-test-1 client.ssl on gluster volume set gfs-vol-test-1 server.ssl on touch /var/lib/glusterd/secure-access
At last we need to copy our certificates over to the client. In this demo setup I will use our GlusterFS test volume. DO NOT DO THIS IF THIS IS A LIVE SYSTEM! In production service please securely transfer the keys!
cp -ar /etc/ssl/glusterfs* /mnt/gfs-vol-test-1/
On our client side we now have to copy the keys into our SSL folder, create a config file for our client and then we unmount the volume.
cp /mnt/gfs-vol-test-1/glusterfs* /etc/ssl/ touch /var/lib/glusterd/secure-access umount /mnt/gfs-vol-test-1
Unmount the volume
Reboot server and not just restart the service, unless you stop the service and use kill to remove remaining not stopped daemons as the command “service glusterfs-server restart” is faulty or may takes up to 15 minutes to kill old connections!
Once rebooted mount the local volume
mount.glusterfs 127.0.0.1:/gfs-vol-test-1 /mnt/gfs-vol-test-1
On the client side we only need to execute the standard mount command
mount.glusterfs 10.0.0.1:/gfs-vol-test-1 /mnt/gfs-vol-test-1
I hope that this short tutorial will help you setting up your own distributed file system. Please note that the above is absolutely simplified. I have written this after been asked a few times for to setup the SSL part so I thought I give everyone a very easy tutorial which actually works out of the box (and yes I tested it).