Globus on the Rocks

This document describes installing the Globus Package on top of a NPACI Rocks Cluster. In particular, it discusses how to:

  1. Download the required software source
  2. Build from source
  3. Add paths to your .cshrc or .profile file
  4. Perform Initial security setup
  5. Get the various required "certificates"
    1. User
    2. Machine specific
    3. MDS/LDAP
  6. Put things where they need to be
  7. Modify various system configuration files
  8. Run some simple tests
  9. Deal with some strangeness
  10. Purge Globus build

The items shown in red require root privilege on your machine. The "root" things are simple to do. If you don't have root, do all of the other things and then ask someone with privilege to finish it up. Information on Rocks can be found at rocks.sdsc.edu. The Globus home page is www.globus.org.

This page borrows extensively from the Globus home page and the Install Guide for Globus 2.0 beta by Scott Gose of ANL We break with the Rocks tradition in that it we do not make use of the RPM system. An RPM version of this page will be written after these instructions are well tested.

Most of the tasks in deploying Globus can be done with simple scripts. Fill in the blank Web pages are provided that generate the scripts. A list of these scripts is provided here for reference.

Quick list of scripts

  1. Build Script Generator
  2. .cshrc and .profile Generator
  3. Configuring GSI security
  4. Gatekeeper Certificate and MDS/LDAP/ Certificate
  5. Add Certificate Granting Authorities
  6. Creating the grid-mapfile
  7. System Configuration Script Generator
  8. Purge Globus build


Download the required software source

We assume that the user installing this software is "globus". This is not a major assumption and should not effect usability. For others to use the software it is required that there be access to the users directory. The script given below will create a new directory and set permissions as needed.

We will get 5 required packages and 1 optional. Please download these packages from the links on this web page directly. The packages given here might not be the most current but they have been tested with these instructions. Put the "gz" files in "globus" home directory. Do not uncompress them.

all_gz.tar
A single tar file containing all of the files shown below
[an error occurred while processing this directive]

gpt-0.2.tar.gz
Globus Packaging Tools
[an error occurred while processing this directive]
globus_tools_bundle.tar.gz
Essential Grid Tools
[an error occurred while processing this directive]
globus_api_bundle.tar.gz
Grid APIs
[an error occurred while processing this directive]
globus_services_1_bundle.tar.gz
Essential Grid Services 1
[an error occurred while processing this directive]
globus_services_2_bundle.tar.gz
Essential Grid Services 2
[an error occurred while processing this directive]
globus_gsincftp-0.1.tar.gz
GSI-NCFTP, GSI-enabled version of ncftp FTP
[an error occurred while processing this directive]

For a description of these packages please see
www.globus.org/gt2/install/index.html
and
www.globus.org/gt2/install/beta-source.html

Build from source

Next we generate a script for building the distribution. This script is dependent on user name and build directory. Please click here to build the script. This script should be saved in a file, say, script1, in your home directory. Do a

chmod 700 script1
./script1

The script optionally creates a new directory, sets the protection for it and for the home directory, moves the "gz" files to the directory and does the build. A sample screen dump from this script can be seen here



Add paths to your .cshrc or .profile file

There are some important environmental variables that need to be set to run Globus software. The most important are GLOBUS_INSTALL_PATH and GLOBUS_LOCATION. These would normally be set for you when you login in your .cshrc or .profile file. There is also a script that should be run that sets some other variables, MANPATH, LD_LIBRARY_PATH, SASL_PATH and GLOBUS_PATH. The script comes in two flavors, for csh and Bourne shells. They are in in $GLOBUS_INSTALL_PATH/etc/globus-user-env.csh and *.sh. It is a good idea to invoke this within .cshrc or .profile.

There is an optional variable GLOBUS_HOSTNAME. This is needed only if your machines does not know its external name. If you do a

printenv HOST

and get something like "localhost" then you need to set GLOBUS_HOSTNAME to the "true" name of your machine.

A typical simple .cshrc file would look like:

set path = ( .:~/bin $path )
setenv GLOBUS_HOSTNAME slic15.sdsc.edu
setenv GLOBUS_INSTALL_PATH /home/globus/beta
setenv GLOBUS_LOCATION $GLOBUS_INSTALL_PATH
if ( ! $?prompt ) exit 0
if(-r $GLOBUS_INSTALL_PATH/etc/globus-user-env.csh)then
echo "source $GLOBUS_INSTALL_PATH/etc/globus-user-env.csh"
source $GLOBUS_INSTALL_PATH/etc/globus-user-env.csh
endif

The paths set in this file are based on where Globus was built. For this example, a user name of "globus" and a directory of "beta" gives GLOBUS_INSTALL_PATH = /home/globus/beta.

A web page that creates the .cshrc or .profile files based on username and directory can be accessed here. The "stuff" generated by this page sould be added to both your file .cshrc or .profile and the "globus" account files. Then login again.



Perform Initial security setup

This is the first thing that needs to be done as root. The Globus security information is held in the directory /etc/grid-security. A script is run to create this directory and to add three files to it. The files are globus-host-ssl.conf, globus-user-ssl.conf, and grid-security.conf.

The script, setup-gsi, will ask for DN (distinguished name) information for users certificates and host certificates. See Get certificates below. The DN is like the user name for Globus. It consists of of several parts
C
Country
O
Organization granting a certificate
OU
Users organization
CN
comman name or the persons real name
USERID
login name for a system

The information you provide determines what certificate granting organizations you will recognize and which users organizations are valid. Only people who have had certificates granted by the specified organizations and users from specified organizations will be allowed to use Globus on your system.

For the user certificates you specify OU and O. For host certificates you specify O. Multiple values can be given seperated my commas. The default OU is based on. Below we have a example run of setup-gsi. The text entered by "root" is shown in red. We assume that the Globus was build under the globus username and in the directory beta. For the user certificate we set os=sdsc.edu and o=NPACI, o=Grid, o=Globus.

Although this is pretty simple, a fill in the blank script generator for this task is provided below. Note: the script generated by the web page must be run as root.

root@slic15-> cd ~globus/beta/setup/globus
root@slic15-> setup-gsi
setup-gsi: Configuring GSI security
Installing /etc/grid-security/grid-security.conf...
Running grid-security-config...

G S I : C O N F I G U R A T I O N P R O C E D U R E


Before you use the Grid Security Infrastructure, you should first
define the DN (distinguished name) that should be used for your
organization's X509 certificates. If you do not define a DN,
a default DN will be assigned to you.

This script will ask some questions about site specific
information. This information is used to configure
the Grid Security Infrastructure for your site.

For some questions, a default response is given in [].
Pressing RETURN in response to such a question will enable the default.
This script will overwrite the file --

/etc/grid-security/grid-security.conf


Do you wish to continue (y/n) [y] : y
========================================================================

(1) Base DN for user certificates
[ ou=sdsc.edu, o=Globus, o=Grid ]
(2) Base DN for host certificates
[ o=Globus, o=Grid ]

========================================================================
(q) save, configure the GSI and Quit
(c) Cancel (exit without saving or configuring)
(h) Help
========================================================================

1
Enter the Base Distinguish Name (DN) for user certificates [ ou=sdsc.edu, o=Globus, o=Grid ] :
ou=sdsc.edu, o=NPACI, o=Grid, o=Globus
========================================================================

(1) Base DN for user certificates
[ ou=sdsc.edu, o=NPACI, o=Grid, o=Globus ]
(2) Base DN for host certificates
[ o=Globus, o=Grid ]

========================================================================
(q) save, configure the GSI and Quit
(c) Cancel (exit without saving or configuring)
(h) Help
========================================================================

q
Installing Globus CA certificate into trusted CA certificate directory...
Installing Globus CA signing policy into trusted CA certificate directory...
setup-gsi: Complete
root@slic15->

Click here for a fill in the blank script generator for this task. Note: the resulting script must be run as root.

Get the various required "certificates"

There are three certificates that are required to run the software installed above. One is a personal certificate. The other two are machine specific, one is for general connections and the second is for MDS clients. The machine specific certificates require root priveledge to install.

So what are certificates? Certificates are part of a Public Key Infrastructure (PKI). PKI is a new authentication and secure communication system popular on the Web and in a number of leading-edge, highly-distributed, research and development projects Certificates (along with keys) are small files that can securely identify you across an unsecure network. Ssh with passwordless login is also an example of a certificate system. See http://security.npaci.edu/help/pki/ for additional information.

User Certificate

You need to generate a certificate for yourself. This certificate is for you as a "normal" user not Globus. Modify your .cshrc file as discussed above login again as yourself. Let's assume your real name is John Smith with username jsmith. You would run the command grid-cert-request as shown below. Again, the red test is what is entered and the green is some important output that will be discussed below.

jsmith@slic15-> grid-cert-request -cn "John Smith"
A certificate request and private key is being created.
You will be asked to enter a PEM pass phrase.
This pass phrase is akin to your account password,
and is used to protect your key file.
If you forget your pass phrase, you will need to
obtain a new certificate.

Using configuration from /etc/grid-security/globus-user-ssl.conf
Generating a 1024 bit RSA private key
.............................................++++++
...........................++++++
writing new private key to '/home/jsmith/.globus/userkey.pem'
Enter PEM pass phrase:"enter your pass phrase here"
Verifying password - Enter PEM pass phrase:"enter your pass phrase here also"
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Level 0 Organization [Globus]:Level 1 Organization [Grid]:Level 2 Organization [NPACI]:Level 0 Organizational Unit [sdsc.edu]:Name (e.g., John M. Smith) []:

A private key and a certificate request has been generated with the subject:

/O=Globus/O=Grid/O=NPACI/OU=sdsc.edu/CN=John Smith

If the CN=John Smith is not appropriate, rerun this
script with the -force -cn "Common Name" options.

Your private key is stored in /home/jsmith/.globus/userkey.pem
Your request is stored in /home/jsmith/.globus/usercert_request.pem

Please e-mail the request to the Globus CA ca@globus.org
You may use a command similar to the following:

cat /home/jsmith/.globus/usercert_request.pem | mail ca@globus.org

Only use the above if this machine can send AND receive e-mail. if not, please
mail using some other method.

Your certificate will be mailed to you within two working days.
If you receive no response, contact Globus CA ca@globus.org
jsmith@slic15->$

This command creates a directory off of your home directory "~/.globus" and puts three files in it: usercert_request.pem, userkey.pem, and usercert.pem.

If you look you will see something like:

jsmith@slic15-> cd ~/.globus
jsmith@slic15-> ls -lt user*
-rw-r--r-- 1 jsmith staff 1494 Dec 20 20:12 usercert_request.pem
-r-------- 1 jsmith staff 951 Dec 20 20:12 userkey.pem
-rw-r--r-- 1 jsmith staff 0 Dec 20 20:12 usercert.pem
jsmith@slic15->

The file usercert_request.pem is what you email to ca@globus.org. Since your machine might not be set up to send email send this file using your normal email program. Paste the file in. Do not send it as an attachment.

Notice that usercert.pem is empty. The email you get back in response to your certificate request is a replacement for usercert.pem. When you get response back paste it into usercert.pem.

Do not change the protection on the file userkey.pem. It is part of your password. The other part of your password is the the pass phrase you enter when running grid-cert-request.

The string shown above

/O=Globus/O=Grid/O=NPACI/OU=sdsc.edu/CN=John Smith

is your globus username. Don't worry you will not need to type this in. But notice that it contains the information that you entered when you did the initial Globus security setup.

Gatekeeper Certificate and LDAP/MDS Certificate

These certificates are for your machine. The procedure is similar to getting a user certificate. You run a command, email off a file for each of the two requests. You will get an email back. The information in the return email is placed in a file. Generating the requests can be done as any user. Installing the some of the files requires root because they must go into the protected directory /etc/grid-security. A web page to generate a script do all of this (except the mailing) is provided below.

As an example we show the commands to run:

grid-cert-request -gatekeeper FQDN \
    -key     GLOBUS_LOCATION/etc/hostkey.pem \
    -cert    GLOBUS_LOCATION/etc/hostcert.pem \
    -request GLOBUS_LOCATION/etc/hostcert.req


grid-cert-request -cn ldap/FQDN \
     -cert GLOBUS_LOCATION/etc/server.cert \
     -key  GLOBUS_LOCATION/etc/server.key \
     -req  GLOBUS_LOCATION/etc/server.request -nopw \
     -dir  GLOBUS_LOCATION/etc
     

Where FQDN is the fully qualified domain name of the host that will run the MDS/ldap server and GLOBUS_LOCATION is the actual value of $GLOBUS_LOCATION in your environment.

Again you are asked to email the files hostcert.req and server.request to ca@globus.org to request the certificates.

Replace, the files server.cert and hostcert.pem with the information in the returned email.

As root, move the files hostkey.* to /etc/grid-security. The ownership and permissions for these files should root with be 644, 600, and 644 or:

-rw-r--r--    1 root     root host.req
-r--------    1 root     root hostkey.pem
-rw-r--r--    1 root     root hostcert.pem

The server.* files stay where they are but they should be owned by root. Set the permission for server.cert to 600 the others can be 644.

Again, a set of two scripts is provided to do these tasks. Click here.

Put things where they need to be

There are two more important parts of the Globus Security Infrastructure. These are the files that describe the certification authority policies and the file that gives the mapping between distinguished name and user names. The certification authority policies are put in the directory /etc/grid-security/certificates and the map file is /etc/grid-security/grid-mapfile.

There is a certification authority policy for each organization from which you will accept certification. The description is actually in two parts, the actual policy and a public security key. The key is stored in a file with a name that ends in ".0 and the policy is stored in a file that ends in ".signing_policy".

When you did the initial security setup the policy information for globus.org was put in /etc/grid-security/certificates/42864e48.0 and 42864e48.signing_policy. A script is provided to add additional policy files (from NCSA, NPACI, and NASA) to this directory. Click here

The grid map file /etc/grid-security/grid-mapfile contains entries, one per line, of the form:


"Distinguished name" user name

for example


"/C=US/O=NPACI/OU=SDSC/CN=Sid Karin/USERID=skarin" skarin

The first part, the "Distinguished name" must be in quotes. A script is provided to add names to /etc/grid-security/grid-mapfile. This script gives the option to add people in various groups within UCSD. As more people start using Globus additional gropus will added to the script. Click here

Modify various system configuration files

There are a few system configuration modificaitons that need to be done. These are described here and as before a web page is provided to create a script to do the modifications.

There is a minor problem in the LDAP configuration file $GLOBUS_LOCATION/etc/grid-info-slapd.conf. The line

        modulepath      /usr/local/globus/libexec/openldap/gcc32dbg

should be

        modulepath      /usr/local/globus/libexec/openldap/gcc32dbgpthr

Finally, you need to configure your system so that the Globus service run as root so every one can use them. This involves adding port mappings to the /etc/services file and adding configuration files to /etc/xinetd.d for the gatekeeper and Grid-FTP. The lines to be added to /etc/services are:


globus-gatekeeper     2119/tcp                # Globus Gatekeeper
globus-gatekeeper     2119/udp                # Globus Gatekeeper
gsiftp        2811/tcp                        # Grid-FTP server (wu-ftpd)

In the directory /etc/xinetd.d we add the two files shown below, replacing GLOBUS_INSTALL_PATH with its actual value

gsi-wuftpd service gsiftp { instances = 1000 socket_type = stream wait = no user = root server = GLOBUS_INSTALL_PATH/sbin/in.ftpd server_args = -l -a -G GLOBUS_INSTALL_PATH log_on_success += DURATION USERID log_on_failure += USERID nice = 10 disable = no }

globus-gatekeeper service globus-gatekeeper { socket_type = stream protocol = tcp wait = no user = root server = GLOBUS_INSTALL_PATH/sbin/globus-gatekeeper server_args = -conf GLOBUS_INSTALL_PATH/etc/globus-gatekeeper.conf disable = no }
Again, a script is provided to make these modifications as root. Please click here

Run some simple tests

Now you're ready to test the installation. As yourself, run

Command Notes
grid-proxy-init This "logs" you into globus. You will be asked for your pass phrase you entered when you created your certificate.
globusrun -o -r localhost '&(executable=/bin/date)' This runs "date" on your machine.
gsincftp localhost Tests the GSI-enabled version of ncftp FTP. You will be "Logged in to localhost."
$GLOBUS_LOCATION/sbin/SXXgris start This command starts the OpenLDAP 2.0 slapd server for the GRIS. Note that there is a single slapd instance for both GRIS and GIIS.
$GLOBUS_LOCATION/bin/grid-info-search -anonymous -L Sends a test query to GRIS and GIIS.

Deal with some strangeness

Globus Personal Gatekeeper

The globus-personal-gatekeeper allows you to set up your own personal copy of the globus authentication services. This is done simply by typing:

globus-personal-gatekeeper

This does two things. It sets up a directory with all of the configuration information. And it returns a GRAM contact string.

The format for the GRAM contact string is:

machine_name:port_number_:distinguished_name

When you run a globus command through the personal gatekeeper you use the GRAM contact string as the machine name. To test to see if globus is running on this machine using the indicated port your should be able to type


globusrun -a -r "machine_name:port_number_:distinguished_name"

Unfortunately this most likely will not work. Why? There might be a problem with one of the configuration files that is created when globus-personal-gatekeeper is launched.

A new directory for configuration files is created every time you run globus-personal-gatekeeper. It is removed when the globus-personal-gatekeeper is killed. The directory is created under your ~/.globus directroy. (Recall your certificate files are in the ~/.globus directory also.) The name of the directory changes each time you run globus-personal-gatekeeper but it has the form .personal-gatekeeper* where the * is the machine name followed by an interger.

To see the name of the directory do a:


ls -1a ~/.globus | grep-gatekeeper

To see the files in the directory do a:


ls ~/.globus/.personal-gatekeeper*

You need to edit ~/.globus/.personal-gatekeeper*/gridmap

This "should" contain your distinguished_name in quotes followed by your user name. More, likely the USERNAME field of the distinguishedname is missing and is replaced with a set of numbers. Replace these numbers with the text USERNAME.

The globusrun test shown above should now work. To stop the gatekeeper do a:

globus-personal-gatekeeper -killall

Purge Globus build

A script is provided to remove most of the files generated in the install. Click here. There is an optional backup that can be done that iwll save the configuration information and globus certificates. This does not save any "user" certificates but they are not deleted. Also your .cshrc and .profile files are not changed.




Author: Timothy Kaiser, Ph.D.
tkaiser@sdsc.edu
Revised: