Frequently Asked Questions about MDS 2.2

Q: How does one generate multiple gatekeeper and user certificates as well as a fewLDAP service certificates?

Write a wrapper script around grid-cert-request, to generate a key and a certificate request (each pair per machine/user can be specified in a separate file). Then mail each request separately to the ca@globus.org. This makes it easy for the Globus Certificate Authority (CA) to issue certificates and do bookkeeping without too much additional effort.

Look at the command line options in grid-cert-request given below: 

grid-cert-request [-help] [ options ...]

Example Usage:

Creating a user certificate:

grid-cert-request

Creating a host or gatekeeper certificate:

grid-cert-request -host [my.host.fqdn]

Creating an LDAP service certificate:

grid-cert-request -service ldap -host [my.host.fqdn]

Options:

-cn <name>
Common name of the user.
-service <service>
Create certificate for a service. Requires the -host option and implies that the generated key will not be password protected (i.e., implies -nopw).
-host <FQDN>
Create certificate for a host named <FQDN>.
-dir <dir_name>
Changes the directory the private key and certificate request will be placed in. By default, user certificates are placed in /yourpathname/.globus, host certificates are placed in /etc/grid-security, and service certificates are placed in /etc/grid-security/<service>.
-prefix <prefix>
Causes the generated files to be named <prefix>cert.pem, <prefix>key.pem, and<prefix>cert_request.pem.
-nopw
Create certificate without a password.
-verbose
Don't clear the screen.
-int[eractive]
Prompt user for each component of the DN.
-force
Overwrites pre-existing certificates.
-c
Will ask which CA is to be used (interactive).
-ca <hash>
Will use the CA with hash value <hash>.

Q: How do I generate and where do I install the LDAP service certificate? Also how do I distinguish between the gatekeeper certificate and LDAP certificate?

For authenticated access to MDS, each Globus Toolkit user requires a user certificate and each server requires an LDAP service certificate. These certificates allow the user and the server, respectively, to participate in the authentication process.(Each host requires a host certificate if it is running a gatekeeper or other service besides MDS.)

As described in the Globus Toolkit 2.4 Installation Instructions, you obtain a user, host, or service certificate with the grid-cert-request command.For your user certificate, you issue this command from the client machine. For the LDAP service certificate required by MDS, you issue this command from the server machine in the following form: 

%  grid-cert-request –service ldap –host my.host.fqdn

In the command format above:

  • -service ldap specifies that a certificate be created for the LDAP server.

  • -host my.host.fqdn specifies the fully qualified domain name of the host that will run the LDAP server.

For a description of all the options available for grid-cert-request, enter the command with the –help or –usage options.

After running this command, the service key is generated immediately and placed in /etc/grid-security/ldap/ldapkey.pem.The key file should be readable only by its owner. The certificate request is stored in /etc/grid-security/ldap/ldapcert_request.pem.

After you enter the grid-cert-request command for the LDAP service certificate, do the following:

  1. Use your regular user mail agent to send an e-mail to ca@globus.org; copy and paste the contents of /etc/grid-security/ldap/ldapcert_request.pem into the message. Do not include this file as an attachment. Do not be concerned about the 0 byte file named ldapcert.pem that is generated by the grid-cert-request program. It is just a place holder; your real certificate will arrive in the mail.

  2. Within two business days, your LDAP service certificate will be mailed to you. When it arrives, read the contents of the e-mail and save the entire e-mail to /etc/grid-security/ldap/ldapcert.pem. This file must be owned by the user account that will run MDS. The file should have permissions 600.

Note that the Globus CA mentioned above is provided for convenience only. For the most effective security, your environment should have its own CA.

Refer to the Globus Toolkit 2.4 Installation Instructions for more details on e-mailing your certificate request and receiving your certificate via return e-mail.

Q: What is the security model for MDS 2.2?

MDS 2.2 uses OpenLDAP, which is based on LDAPv3. The security in OpenLDAP (as of version 2.0) is provided by SASL, which uses GSSAPI.

GSSAPI is a convenient generic interface for secure application development. By itself, GSSAPI does not provide any security. It relies on underlying technologies to provide the actual identity authentication and message protection services desired by applications communicating over a network. Such a security technology is called a GSSAPI security mechanism. Applications may install and request the use of particular mechanisms or use a default mechanism provided by the GSSAPI implementation.

The GSSAPI security mechanism developed for MDS 2.2 is GSI, which enables the use of Globus certificates and grid-mapfiles to provide authorization services to information objects. MDS 2.2 also supports the OpenLDAP access control features, as described in the OpenLDAP 2.0 Administrator's Guide.

A security mechanism is identified by a globally unique Object Identifier (OID). For example, the OID for GSI is 1.3.6.1.4.1.3536.1.1.

The main security enhancement to LDAPv3 (in comparison to LDAPv2) is that new authentication mechanisms have been added to the bind operation: password protection using a hash function, and certificate-based digitally signed token. These are fully compatible with the definitions found in X.509, so that an LDAP to X.500 gateway can pass them directly to the X.500 server. The protocol is extensible to allow for privately defined or future additional authentication mechanisms, including challenge-response and multi-stage binds, and the server can return credentials to the client.

A problem with LDAPv2 deployment was that the string syntaxes for certificates and certificate revocation lists (CRLs) were not reversible, and that it was not always possible to obtain the DER binary encoding of values with these syntaxes in order to be able verify signatures. In contrast, LDAPv3 clients can request to receive CRLs, certificates and distinguished names in binary (DER encoded ASN.1) form, which eliminates the ambiguity.

Q: Do I need root access to install and run MDS 2.2?

No, but if you want to set up MDS 2.2 to automatically start up at system boot time, you do need root access. This can be requested of system staff.

Q:Do you recommend running MDS 2.2 as root?

No, for the following reasons:

  • Under UNIX, programs should not be run as root unless required. There is no reason that MDS needs to run as root.

  • MDS uses OpenLDAP, and security holes have been identified therein. Although we at the Globus Project patch OpenLDAP accordingly when we find out about these security holes, running MDS as root would make them much more dangerous.

  • MDS itself might have buffer overflow problems that could cause it to start arbitrary sub-processes, and not running MDS as root mitigates this danger.

  • Once MDS is installed, the grid-info-resource-ldif.conf configuration file (see MDS 2.2 Configuration Files) identifies information providers by their path. MDS runs these as sub-processes, and there is no reason for an information provider to run as root.If such a requirement existed, the information provider could be made setuid on its own.This could open backdoors for hackers.

  • Overall, there are only a very few pieces of the Globus Toolkit that need to have root access.The rest should just be run as an ordinary non-privileged user.

Q: I have everything up and running, but when I try to do a grid-info-search I get an error message:

ldap_sasl_interactive_bind_s: local error

What's wrong?

In MDS 2.2, clients perform a GSI-GSSAPI bind to the MDS server. The error message seen above occurs if the client does not have a valid proxy, or: 

$ grid-info-search -h 1linux -p 2167 -s sub -b "o=Grid" "objectclass=*"

SASL/GSSAPI authentication started

ldap_sasl_interactive_bind_s: Local error

If the "-x" option is used, we get the following:

$ grid-info-search -h 1linux -p 2167 -s sub -b "o-Grid" "objectclass=*" -x

version:2

#
# Filter: objectclass=*
# requesting: ALL
#

# search result
search: 2
result: 53 DSA is unwilling to perform
text: access denied: gridmap lookup failed

# numResponses: 1

This occurs whether we turn anonymous bind on or off, and whether we have run grid-proxy-init or not. Also, the grid-mapfile, owned by root, contains:

"/C=US/O=UNIV/L=COMP/CN=John Doe/Email=john.doe@comp.univ.edu"
globus

In the file /opt/globus-mds/share/certificates/ca-signing-policy.conf, both Globus and the UNIV CA appear.

What could the problem be?

The failure is happening at 'bind' time; the check on grid-mapfile occurs at the back end, after the bind goes through.

The grid-mapfile should include the user's DN and their local account name. The grid-mapfile should contain entries for users only.

The server account does not need to do a grid-proxy-init. In fact, make sure that there isn't an old proxy in /tmp that belongs to globus.

Regular user accounts need to have a valid proxy, which can be obtained using grid-proxy-init.

To check if a proxy is still valid, use grid-proxy-info. Make sure that the user's X509_USER_PROXY is pointing to a valid proxy.

See MDS 2.2 Bind Error Troubleshooting for additional information.

Q: Do I need to run grid-cert-request on the same machine as MDS 2.2?

No. The syntax for grid-cert-request is: 

grid-cert-request -service ldap -host <FQDN> -dir <dir_name>

This can be executed anywhere, on any machine. The -dir can be local to that machine, or on an accessible remote machine. Once received, the certificate and key can be moved to a new location. The FQDN must be the fully qualified domain name of the machine on which the MDS 2.2 server will be running. The certificate authority does not require that the certificate request for machine A must come from machine A.

Q: I downloaded the Globus Toolkit 2.2 release and installed MDS. All of the installation was successful.

I applied the LDAP server certification, and I put ldapcert.pem and ldapkey.pem in /etc/grid-security/ldap.

Then I started MDS with the globus-mds start (formerly SXXgris start) command.

I could do anonymous (anonymous flag set; without SASL) queries, as follows:

grid-info-search -x -b "mds-vo-name=local, o=grid"
grid-info-search -x -b "mds-vo-name=site, o=grid"

But I could not do security-enabled (with SASL) queries, as follows:

% grid-info-search -b "mds-vo-name=local, o=grid" ldap_sasl_interactive_bind_s: No such attribute

If anonymous queries work but authenticated queries do not, it means the server is up and running.

The problem could be that the wrong server port is being used, i.e., the port is not set up for authentication, or that there are no certificates in the certificate directory.

First, make sure the client has a valid proxy to start with, and second, that the server certificate permission is correct.The key is probably not world-readable or writable.You can look into /etc/grid-security/ldap/.

If all seems to be correct, try queries like:

grid-info-search -x -s base -h their-host -p their-port -b "" supportedsaslmechanisms

on the server to see if the mechanism is correct.

Sometimes, it might be because SASL_PATH is not set properly, so client and server cannot find the SASL plug-ins. If you put -d 255 on the grid-info-search query, it will give a lot more debugging information and can eliminate some other missteps.

For more details on debugging information, refer to the Increasing debugging information in grid-info-system.log FAQ item and to the MDS 2.2 User's Guide (the Diagnostics and Logging section of Chapter 3, Using MDS).

Refer to MDS 2.2 Bind Error Troubleshooting for additional information on LDAP bind errors.

Also there is an FAQ page on Globus security you might want to check here.

Q: What are the OID namespaces and schemas for MDS 2.2?

Refer to the MDS 2.2 Schemas page for an index of schema definitions and descriptions of the schema object classes and attribute types.

Q: Why does no CPU information appear in my GRIS running on Solaris?

The Solaris CPU information provider uses SysInfo, a third-party product. For further details of SysInfo, please see http://www.MagniComp.com/sysinfo

The sysinfo executable should be in the path when MDS is installed.

Q: Why am I getting bogus load information from MDS on SGI IRIX?

You may have a version of the uptime command that does not provide system load information.

Bad uptime output looks like this:

$ uptime
10:19amup 7 days, 18:28,12 users

Good uptime output looks like this:

$ uptime
10:19amup 7 days, 18:29,12 users,load average: 0.75, 0.73, 0.77

Locate an uptime executable that provides the correct output, and ensure that it is in the path when you install MDS.

If you have already installed MDS, you can correct this bug by modifying $GLOBUS_LOCATION/sbin/globus-sh-tools-vars.sh as follows:

  1. Locate the line that defines GLOBUS_SH_UPTIME, e.g., GLOBUS_SH_UPTIME="/usr/freeware/bin/uptime".
  2. Modify the path to point to the correct uptime location.

Q: The grid-info-search (or ldapsearch) command is not returning all of the data objects I am expecting. I get the message:

Size limit exceeded (4)

How can I get all the results I expect from the command line query?

There are three different sizelimit parameters involved here. All three of them have defaults, and all three of them affect the number of LDIF data objects returned from a query. These are:

  • The sizelimit option on the ldapsearch command (to which grid-info-search is a wrapper).

  • The sizelimit option in the grid-info-slapd.conf configuration file, which controls the number of objects returned by the slapd server to the client.

  • The sizelimit option for each of the information provider programs in the grid-info-resource-ldif.conf configuration file, which controls the number of objects returned by each provider to the slapd server.

Depending on your environment, you or your system administrator will need to increase one or more of these sizelimit values above their defaults in order to get the number of data objects you are expecting.

Q: I’ve made configuration changes to my MDS setup, but they do not take effect when I start MDS.

You or someone else may have a slapd server from a previous startup of MDS already running on the system, preventing a new slapd from starting. For example, you and another user may both be trying to run slapd servers on the same port. Or there may be a slapd running from a previous version or installation of MDS.

This situation may also cause the slapd server to not start at all when you enter the globus-mds start (formerly SXXgris start) command to invoke MDS. However, there may be other reasons for the slapd server not starting, such as not setting or sourcing your environment, not having the right certificates, and so forth. Refer to the MDS 2.2 User’s Guide for more information on those topics.

Check your environment for the slapd servers that are currently running. To see which slapd servers are currently running on Linux, enter the following command:

% ps uawx | grep slapd

From the output of this command, you can see if any old or conflicting slapd servers are running, and you can remove them as appropriate with the kill command.

Note that the globus-mds stop (formerly SXXgris stop) command may not kill the slapd instance you had intended. Use the kill command instead. globus-mds stop is appropriate for when you want to stop MDS, make a configuration change, and restart MDS via globus-mds start, assuming there are no conflicting slapd servers running at that point in time.

Q: How do I view MDS data via a web page?

Since MDS uses OpenLDAP as its front-end query engine, you can take advantage of all the industry standard and community driven software designed to work with LDAP servers. This includes a number of software packages created to expose LDAP server data via an html/http interface. The following are two examples:

http://ldapweb.sourceforge.net/

This is an LDAP web interface developed using Apache, PHP, and SSL tools. It can be used to manipulate Microsoft's Active Directory. It provides search, add, delete, and modify capabilities, as well as schema recognition.

http://freshmeat.net/projects/ldapwebxplorer/?topic_id=90

This is a PHP-based LDAP web interface. It provides browse, edit, delete, search, and authentication capabilities. It has some multilingual capabilities, a friendly interface, and an extendable configuration.

Additional browser and web interface resources can be accessed from Consuming Data From MDS 2.2.

Q: I am currently setting up a hierarchical GIIS. Everything works fine except registration from one GIIS to another GIIS. Could some character in the mds-vo-name be the cause of the problem (seems like the syntax required by the schema should not accept '_' - underscores)?
The attribute type of "mds-vo-name" is "printable string," which doesn't allow '_' (underscore).

Q: I’m trying to solve a problem by looking at the logfile grid-info-system.log, but it barely says anything. How can I raise the log level to see what is happening there?

In the globus-mds (formerly SXXgris) file in $GLOBUS_LOCATION/sbin, there is a call to grid-info-soft-register, with a parameter "-d 0". You can change this to, for example, “-d 255” to get lots of debugging information.

You can also use the Unix syslog daemon, configured in /etc/syslog.conf, to log OpenLDAP messages and GIIS/GRIS back end messages.

You put an entry like:

local4.debug local-log-file

in the /etc/syslog.conf file. Once you’ve done this, you’ll need to restart your syslog daemon, which is operating system dependent.

The syslog restart command publishes messages into the log file name indicated by the local4.debug entry in the syslog.conf file.

The logged messages show slapd server messages for items such as machine registration status, success or failure of a search, and slapd operational status.

Refer to the MDS 2.2 User’s Guide (the Diagnostics and Logging section of Chapter 3, Using MDS) for more information on the use of syslog.conf and examples of log file output.

Q: I’m trying to set up Globus Toolkit 2.2. Everything seems to be working, except I can only do anonymous MDS searches. If I enter grid-info-search without the “-x” option, I get this output:

SASL/GSI-GSSAPI authentication started
ldap_sasl_interactive_bind_s: Local error

I think I have everything set up correctly: the ldapcert.pem and ldapkey.pem files are in /etc/grid-security/ldap, the grid-mapfile is in /etc/grid-security, the usercert.pem and userkey.pem files are in $HOME/.globus, and all the environment variables that I can see are pointing to the right locations (including SASL_PATH). Remote jobs run fine, as do anonymous MDS queries.

Assuming everything else is in fact set up properly, the problem may have to do with how the host names of the machines are being reported. When a host name is looked up on a machine without that host name in its /etc/hosts file, it is looked up through DNS and hopefully returns the correct result. If the machine does have a host name in its /etc/hosts file but the host name is not in fully qualified format, or is in that format but does not appear first on the line referencing it, it fails to resolve correctly and the Globus Toolkit is not able to authenticate its certificate.

For example, a hosts file appearing like this will not allow authenticated access because the fully qualified name appears second on each line:

122.0.0.1 host01 host01.univ.edu localhost
125.213.49.87 host01 host01.univ.edu

A hosts file appearing like this will allow authenticated access because the fully qualified name appears first on each line:

122.0.0.1 host01.univ.edu host01 localhost
125.213.49.87 host01.univ.edu host01

If correcting the hostnames in /etc/hosts does not solve your authentication problem, refer to the other FAQs on this issue:

Bind error on grid-info-search command
Can do anonymous but not security-enabled queries

Also see MDS 2.2 Bind Error Troubleshooting.

Q: I have two hosts, Host1 and Host2. I’ve registered a GRIS on Host2 to a GIIS on Host1; together they constitute a VO called "org". When I start MDS and issue an authenticated grid-info-search command from Host2, I expect that all the data objects from both hosts will be displayed. However, I only see the data from the GIIS on Host1, as if the GRIS on Host2 had never registered to the GIIS on Host1. What can I do to get all the data I am expecting?

There are several things you can try, as follows:

  • Can Host2 ping Host1 (without its qualified name), and vice versa?

  • If the hosts can ping each other, then try the following grid-info-search queries, both anonymous (-x option) and authenticated:

    - From Host2 to Host2
    - From Host1 to Host1
    - From Host2 to Host1
    - From Host1 to Host2

  • If bind errors occur from any of the above queries, refer to MDS 2.2 Bind Error Troubleshooting for suggestions.

  • Check the system time on Host2. Is it out of sync with Host1?

    Note that the timing and caching of registration messages sent between GRIS and GIIS clients/servers in an MDS infrastructure are controlled by various parameters in the grid-info-resource-register.conf file.

    The proper operation of these parameters depends on the synchronization of the system clocks on the hosts involved. If the clocks are not synchronized, registration messages may be lost, and partial data or no data may be returned.

    Refer to MDS 2.2 Configuration Files for more details on grid-info-resource-register.conf.

If no problems are found with trying the above, then try outputting debuggingmessages by replacing "-d 0" with "-d 255" in the globus-mds (formerly SXXgris) file. This will return much more detailed debugging information. You can also use the Unix syslog daemon, configured in /etc/syslog.conf, to log OpenLDAP messages and GIIS/GRIS back end messages. Refer to the Increasing debugging information in grid-info-system.log FAQ for more details.

Q. I upgraded from MDS 2.1 to MDS 2.2 and copied my grid-info-resource-register.conf file from MDS 2.1. Now my upgraded GRIS doesn't seem to be registering to any GIIS servers. What happened?

MDS 2.2 provides mutual authentication between GIIS and GRIS servers and well as between GIIS servers in a hierarchy. Mutual authentication is enabled via a new parameter, bindmethod, in the grid-info-resource-register.conf file. Without this parameter, registration won’t work.Therefore, you need to add the bindmethod parameter to any grid-info-resource-register.conf file you have copied from an existing
MDS 2.1 installation.

On the end of each registration block in grid-info-resource-register.conf, add the line:

bindmethod: ANONYM-ONLY

This is the default value, which specifies the use of anonymous binding only.For more details on mutual authentication and the use of the bindmethod parameter, refer to MDS 2.2: Creating a Hierarchical GIIS. Refer to MDS 2.2 Configuration Files for an example of a grid-info-resource-register.conf file.

Q. What performance can I expect from a query to an MDS 2 GRIS or GIIS?

The performance of a query to a GRIS is dependent upon the performance of the information providers that the GRIS accesses, as well as the TTL data for the information they deliver:

  • When a query to a GRIS arrives, if the data requested is live and cached, the query will be answered very quickly.

  • If the data requested has been flushed from the cache because it has expired, then the GRIS will invoke the information provider or providers that supply the information required by the query.If these providers start up and deliver information quickly, then the GRIS can in turn answer the client query relatively quickly, though it will be a little slower than if the data had been in cache.

  • If an information provider takes a long time to deliver its information, either because there is a lot of overhead for it to start up or acquire the data, or because there is a lot of data to deliver, then this will negatively impact the performance the client query sees.

  • Further, the GRIS will allow a certain, configurable, time window for the information provider to finish its delivery. As a result, if the provider does not finish within the time window, the GRIS will terminate its link to the provider and move on to the next provider, if any. As a result, data from that provider may never appear in the GRIS.To resolve this, either extend the time window that the GRIS waits, or send less data to the GRIS. In this case, though, the client query to the GRIS will take a long time to return, since it will wait for the timeout window and the amount of time to access any other information providers before it can answer the client query.

  • To extend this time window for the information provider, try increasing the value of the timelimit: parameter for the relevant provider in the etc/grid-info-resource-ldif.conf file. (Refer to MDS 2.2 Configuration Files for more details on and an example of this file.)

The performance of a query to a GIIS is dependent upon the performance of the GRIS's that it accesses, as well as the TTL data for the information they deliver:

  • When a query to a GIIS arrives, if the data requested is live and cached, the query will be answered very quickly.

  • If the data requested has been flushed from cache because it has expired, then the GIIS might query a GRIS that supplies that information. The performance of this sub-query is like any client querying a GRIS, as described above.

  • Alternatively, in a GIIS hierarchy, a GIIS query for un-cached data might in turn query a subsidiary GIIS. The performance of this sub-query to another GIIS is based upon whether it has cached data, and if not, the performance of its sub-hierarchy.

In summary, there is no a priori formula for predicting the performance of a query to MDS. The more complex the GIIS hierarchy, the more unpredictable the performance, and in general, the longer that a query might take to answer, depending upon TTL data.

It is possible to mitigate this performance variability by attempting to insure that data queried is normally in cache.One can do this either by increasing the TTL value for the data (but that might not make sense), or by causing the cache to fill by periodically running a script that triggers this.This can be done at any or all levels of a GIIS hierarchy, and at the GRIS level as well.If the caches are filled regularly by scripts running periodically, then client queries will most likely find the data they are looking for in cache.

Q. I am trying to add to MDS a subtree that holds new information.For this subtree, my script tries directly to input data into the DIT using the LDAP API but is unsuccessful.Why am I getting error messages about failure to add a new entry?

You cannot add data into the MDS backend databases using LDAP adds.The MDS backends are read-only databases (from the client’s point of view).That’s why you are getting these errors.You may be able to achieve the desired result by using the LDBM backend with OpenLDAP.

Information will only appear in a GRIS from an executable information provider configured into grid-info-resource-ldif.conf that conforms to our information provider API (for example, by outputting LDIF to stdout).See the Creating New Information Providers document for more details.