-- Feb 2017 - See new post, updated for iRODS 4.2
I was asked to set up a reference implementation of iRODS using the B2STAGE-GridFTP data storage interface (DSI). This blog entry explains the steps used to set up and test this implementation. This was set up using the instructions at https://github.com/EUDAT-B2STAGE/B2STAGE-GridFTP.
I set up two Ubuntu 14.04 virtual machines on an internal network. These are:
Each of these servers could resolve the fully qualified domain name of the other.
I also installed iRODS 4.1.7 on server.example.org. This included the following packages:
Refer to https://docs.irods.org/master/manual/installation for iRODS installation instructions.
Run the following commands on server.example.org to install the packages that we will be using:
sudo apt-get install cmake
sudo apt-get install globus-gridftp-server-progs globus-simple-ca globus-gass-copy-progs
sudo apt-get install libglobus-common-dev libglobus-gridftp-server-dev libglobus-gridmap-callout-error-dev
sudo apt-get install libcurl4-openssl-dev
sudo apt-get install git
sudo apt-get install g++
sudo apt-get install dpkg-dev
sudo apt-get install cdbs
Run the following on client.example.org:
sudo apt-get install globus-gass-copy-progs
The following instructions should be performed on server.example.org.
First we need to clone the B2STAGE-GridFTP repository.
cd ~
git clone https://github.com/EUDAT-B2STAGE/B2STAGE-GridFTP
Next we will get ready to build the iRODS DSI. We will put the output files in /iRODS_DSI.
sudo mkdir /iRODS_DSI
sudo chmod 777 /iRODS_DSI
cd ~/B2STAGE-GridFTP
cp setup.sh.template setup.sh
Edit setup.sh and change the contents to:
export GLOBUS_LOCATION="/usr"
export IRODS_PATH="/usr"
export DEST_LIB_DIR="/iRODS_DSI"
export DEST_BIN_DIR="/iRODS_DSI"
export DEST_ETC_DIR="/iRODS_DSI"
Now build the iRODS DSI:
source setup.sh
cmake CMakeLists.txt
C_INCLUDE_PATH=/usr/include/x86_64-linux-gnu/globus make install
Next we need to edit /etc/gridftp.conf (as root) and add the following lines to the end.
$LD_LIBRARY_PATH "$LD_LIBRARY_PATH:/iRODS_DSI"
$irodsConnectAsAdmin "rods"
load_dsi_module iRODS
auth_level 4
Now we need to preload the GridFTP server library alongside the DSI library. Edit (as root) /etc/init.d/globus-gridftp-server and add the following lines to the start:
LD_PRELOAD="$LD_PRELOAD:/usr/lib/x86_64-linux-gnu/libglobus_gridftp_server.so:/iRODS_DSI/libglobus_gridftp_server_iRODS.so"
export LD_PRELOAD
We also need to change the iRODS default hash scheme. As the user irods, update /etc/irods/server_config.json and change the following line:
Old:
"default_hash_scheme": "SHA256",
New:
"default_hash_scheme": "MD5",
We will be running the GridFTP server in the root account. Add the file /root/.irods/irods_environment.json with the following contents:
{
"irods_host": "localhost",
"irods_zone_name": "tempZone",
"irods_port": 1247,
"irods_user_name": "rods",
"irods_default_resource": "demoResc"
}
Run `iinit` and enter the password for the rods user.
Note: If you run iinit without first creating the irods_environment.json file, iRODS will not ask you for the default resource and this variable will not be set. This will cause unexpected failures. If this is done, edit irods_environment.json and add in the irods_default_resource.
We are finished with the setup of the iRODS DSI. However, to test this configuration we need to create certificates on the client and server.
We need to create a certificate authority to sign are certificates. We will use SimpleCA for this purpose.
On server.example.org, create the CA by running:
grid-ca-create
Next create an deb package for this certificate authority.
grid-ca-package -d -cadir ~/.globus/simpleCA
Install the package on server.example.org:
sudo dpkg -i globus-simple-ca-ffffffff_0.0_all.deb
Note: Replace the ffffffff with the hexadecimal digits specific to your .deb file.
This will install the certificates into /etc/grid-security/certificates.
Update both /etc/grid-security/certificates/globus-host-ssl.conf.ffffffff (as root) and \~/.globus/simpleCA/grid-ca-ssl.conf and set the policy to "policy_anything".
Old:
policy = policy_match
New:
policy = policy_anything
Edit /etc/grid-security/certificates/ffffffff.signing_policy cond_subjects to '"*"':
New:
cond_subjects globus '"*"'
Copy the globus-simple-ca-ffffffff_0.0_all.deb file to the client, install this package on the client, and update the /etc/grid-security/certificates/ffffffff.signing_policy cond_subjects to '"*"''.
Copy the certificate authority files to \~/.globus:
cp /etc/grid-security/certificates/ffffffff.0 ~/.globus/
cp /etc/grid-security/certificates/ffffffff.signing_policy ~/.globus/certificate
Perform the following commands to create the private key and generate a certificate signing request:
# create key
openssl genrsa -des3 -out hostkey.pem 1024
# create certificate
openssl req -new -key hostkey.pem -out cert.csr
# Remove password from key
openssl rsa -in hostkey.pem -out temp.pem
mv temp.pem hostkey.pem
Sign the newly created certificate:
grid-ca-sign -in cert.csr -out hostcert.pem
*Note: Use the password you used when creating the certificate authority.*
Install the certificates on server.example.org. The certificates should be owned by root since the GridFTP server is run by root.
sudo cp hostkey.pem /etc/grid-security
sudo cp hostcert.pem /etc/grid-security
sudo chmod 600 /etc/grid-security/hostkey.pem
sudo chmod 644 /etc/grid-security/hostcert.pem
Copy hostcert.pem and hostkey.pem into the user's home directory on client.example.org. Run the following commands to install these certificates and set the permissions:
mkdir ~/.globus
cp ~/hostkey.pem ~/.globus/hostkey.pem
cp ~/hostcert.pem ~/.globus/hostcert.pem
chmod 600 ~/.globus/hostkey.pem
chmod 644 ~/.globus/hostcert.pem
Run the following command to get the subject name from the certificate:
openssl x509 -noout -in hostcert.pem -subject
On server.example.org, create the file /etc/grid-security/grid-mapfile and add the subject mapping to the user irods. The following is an example of the contents of this file. Replace the part inside the quotes with the subject name returned from the previous command.
"/C=US/ST=NC/L=CH/O=irods/CN=server.example.org" rods
Run the GridFTP server using the following command:
sudo /etc/init.d/globus-gridftp-server restart
On the server, create a 1GB random file:
dd if=/dev/urandom of=file.dat bs=1000 count=1000000
Put this file into iRODS.
iput file.dat
Now let's test retrieving this file from client.example.org.
globus-url-copy gsiftp://server.example.org:2811/tempZone/home/rods/file.dat file.dat
Remove the file from iRODS and let's try to put it from client.example.org to iRODS:
globus-url-copy file.dat gsiftp://server.example.org:2811/tempZone/home/rods/file.dat
The following table shows a comparison of getting a putting the 10 MB file between the client and server using native iRODS transport and GridFTP.
Protocol Get Time Put Time GridFTP 18.5 s 16 s iRODS CLI 23.5 s 16 s