Softpanorama

May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

Environment Modules

News Enterprise Unix System Administration Recommended Links Unix Configuration Management Tools Installation of environment modules<   C3 Tools parallel
rsync rdist Cluster SSH Mussh Job schedulers Unix System Monitoring Grid Engine Perl Admin Tools and Scripts
SSH Power Tool Tentakel Multi Remote Tools pdsh pdcp      
SSH Usage in Pipes Password-less SSH login scp sftp Tips History Humor Etc

The Environment Modules package provides the dynamic modification of a shell environment variables and first of all $PATH and $LD_LIBRARY_LOAD.

They are alsmost standard package on HPc clusters but can be used for other purposes too. In all cases when you deal with multiple version of sotare they are very helpful/

this package was written in the early 1990s and is used at universities and research centers to deploy multiple versions of application packages. The key functionality -- ability to use multiple versions of the program or package from the same account by just loading proper module is more useful and withstand the text of time.

The key idea is trivial: package manipulates shell environment giving the user the capability to modify environment to suit partioicylar package (or versin of the package) by using command load.  And than when you no longer need to restore environment ot previous state by using  command unload. In pronicple this functionality fdoes not reure anything fancy. loading can be done by sourcing bash scripts and unloading by restoring environment from the file saved when you loaded the module.

Still the implementation was done in TCL and that makes in much less tranparent then if it would be woirttin in pure bash.

Typical installation consists of multiple module files stored in tree of directories, one for each version of software used. After the modules software package is initialized, the environment can be modified on a per-module basis using the module command, which interprets module file and modify environment variables such as $PATH,  $LD_LIBRARY_LOAD, $MANPATH, and others.  Unlike modified for specific application .bashrc all module files can be shared by all  users on a system. You can think about module files as the more sophisticated method of sourcing bash scipts that set variables for application.  All other functionality is peripheral ot his key functionality of the package. 

As a user, you can add and remove module files from the current environment, but in reality this is really done (most users are really afraid of TCL ;-). In most installation module files are maintained centrally, which ahs its advantages and disadvantages. But that's how it is. 

The environment changes contained in a module file can also be summarized through the module show command.  yo can also diff environment befor and after loading of a particular module.

If you use a particular application all the time you can invoked the necessary modules modules from .bashrc file saving some typing.  Othersize you can create source files for each application and store them in some directory in your home directory. for example /Rclib. In this case all modules that you need to load for say version 4.1 of CP2K application can be loaded with the command . ~/Rclib/cp2k

Environment Modules are "shell agnostic" and work with all major shells such as bash, ksh, zsh, sh, tcsh, and csh. But in reality bash dominates the field and trhat capability is not that useful.

They also implement  the concept of "default" version, which is helpful.

All this explains why Environment Modules are now a part of most HPC cluster setup. It also inspired several alternative implementation such as lmod from University of Texas, which is written in LUA instead of TCL. LUA is more popular language that is known probably by more administrors then TCL but still it far from mainstream scripting language such as Perl or Python. 

modulefiles

The modules system is based on modulefiles. Each modulfile specifies groups of environment settings that need to be set together. Modulefiles can be installed in a central location for general use, or in a user directory for personal use. Environment Modules modulefiles are written in the Tcl (Tool Command Language) and are interpreted by the modulecmd program.

modulefiles are created on per application per version basis. They can be dynamically loaded, unloaded, or switched. Along with the capability of using multiple

default modulefiles directory

The default modules search path is in a hidden configuration file you can display with:

cat ${MODULESHOME}/init/.modulespath

The /etc/modulefiles directory used by some distributions (or any other directory) can be used after a build from source by modifying the ${MODULESHOME}/init/.modulespath file.

Add your own modules to the path

The ${MODULESHOME}/modulefiles/use.own module essentially performs these steps.

mkdir $HOME/privatemodules
cp ${MODULESHOME}/modulefiles/null $HOME/privatemodules
module use-append $HOME/privatemodules

there can be serveral diredctores from which you can load milule fles

The default strorage directory for modules is /etc/modulefiles directory.  and it is populated with several modules during the installation of the package from RPM. The $HOME/privatemodules or another directory can be used instead along with "module use --append" or modification of the configuration file specifying the default modules search path.

The default modulefiles directory is empty initially. Copy the null module to the default modulefiles directory to have it shown by "module avail". The following uses the null and module-info modules to show use of a version file within a hierarchical organization and their affect on module avail and module show:

 mkdir /etc/modulefiles/test
 cp ${MODULESHOME}/modulefiles/null /etc/modulefiles/test/2.0
 cp ${MODULESHOME}/modulefiles/module-info /etc/modulefiles/test/1.0
 module avail
 module show test

Set the first version as the default:

 echo '#%Module' > /etc/modulefiles/test/.version
 echo 'set ModulesVersion "1.0"' >> /etc/modulefiles/test/.version
 module avail
 module show test
 module show test/2.0

Switch to the newer version as the default with:

 rm /etc/modulefiles/test/.version
 echo '#%Module' > /etc/modulefiles/test/.version
 echo 'set ModulesVersion "2.0"' >> /etc/modulefiles/test/.version
 module avail
 module show test

After the above following a fresh install from source you would see:

$ module avail

-------------------- /usr/local/Modules/3.2.10/modulefiles ---------------------
dot         module-git  module-info modules     null        use.own

------------------------------- /etc/modulefiles -------------------------------
null              test/1.0          test/2.0(default)

--------------------------- /home/user/privatemodules --------------------------
null
$ module show test
-------------------------------------------------------------------
/etc/modulefiles/test/2.0:

module-whatis    does absolutely nothing 
-------------------------------------------------------------------
$ module show test/1.0
-------------------------------------------------------------------
/etc/modulefiles/test/1.0:

module-whatis    returns all various module-info values 
+++ module-info +++++++++++++++++++++++++++++++
flags                   = 2
mode                    = display
name                    = test/1.0
specified               = test/1.0
shell                   = bash
shelltype               = sh
version                 = test/1.0
user                    = advanced
trace                   = 0
tracepat                = -.*
symbols                 = *undef*
+++ info ++++++++++++++++++++++++++++++++++++++
hostname                = localhost
level                   = 1
loaded null             = 0
library                 = /usr/local/lib/tcl8.6
nameofexecutable        =
sharedlibextension      = .so
tclversion              = 8.6
patchlevel              = 8.6.1
+++++++++++++++++++++++++++++++++++++++++++++++
-------------------------------------------------------------------

Automatic modules initialization via /etc/profile.d

Environment Modules on Scientific Linux, CentOS, and RHEL distributions in the environment-modules package include modules.csh and modules.sh scripts for the /etc/profile.d directory that make modules initialization part of the default shell initialization.

This makes managing complex environments a bit less complicated.

For a source build the automation for all users can be manually configured.

/etc/profile.d/modules.sh from the etc/global/profile.modules file in the 3.2.10 modules build directory.

 trap "" 1 2 3
 
 case "$0" in
     -bash|bash|*/bash) . /usr/local/Modules/default/init/bash ;; 
        -ksh|ksh|*/ksh) . /usr/local/Modules/default/init/ksh ;; 
        -zsh|zsh|*/zsh) . /usr/local/Modules/default/init/zsh ;;
                     *) . /usr/local/Modules/default/init/sh ;; # default
 esac
 
 trap 1 2 3

Copy the etc/global/profile.modules file from the 3.2.10 modules build directory to the system initialization directory:

 sudo cp etc/global/profile.modules /etc/profile.d/modules.sh

Add a version 3.2.10 symbolic link for the above generic addresses:

 cd /usr/local/Modules
 sudo ln -sT 3.2.10 default

tcsh, csh automatic modules initialization

A /etc/profile.d/modules.csh symbolic link to the etc/global/profile.modules file in the 3.2.10 modules build directory can enable automatic modules initialization for these users.

Module Command

module help

To get a usage list of module options type the following (the listing has been abbreviated to only those commands discussed on this page) :

% module help  
  Available Commands and Usage:
     +  add|load     modulefile [modulefile ...]
     +  rm|unload    modulefile [modulefile ...] 
     +  switch       modulefile1 modulefile2
     +  display      modulefile [modulefile ...] 
     +  avail        path [path]
     +  list      
     +  help         modulefile [modulefile ...]

To get help on a specific module use:

 % module help [modulefile]


module list

% module list

This lists all the modulefiles that are currently loaded into your environment.

module avail

% module avail

This option lists all the modulefiles that are available to be loaded. Notice that many of them have version numbers associated with them and that where there is more than one version, one is labeled as the default. You can also restrict this command to a single package; for example,

module avail visit

module display

 % module display [modulefile]

Use this command to see exactly what a given modulefile will do to your environment, such as what will be added to the PATH, MANPATH, etc., environment variables.  This is synonymous with

 % module show [modulefile]

module load

% module load [modulefile1] [modulefile2] ...

This command adds one or more modulefiles to your current environment. It does so silently, unless there is a problem with a modulefile. If you load the generic name of a module, you will get the default version. To load a specific version, load the modulefile using its full specification.  For example,

 % module load visit

will do the same thing as

% module load visit//2.1.2

module unload

% module unload [modulefile]

This removes the listed module from the user's current environment. Modulefiles can be removed in any order.  Note that this command will fail silently if the modulefile you specify is not already loaded.

module switch

% module switch [modulefile_old] [modulefile_new]

This command demonstrates the true advantage of using modules. Different versions of entire software packages can be replaced with a single module command. 

This is synonymous with

 % module swap [modulefile_old] [modulefile_new]

Note that the following command would be completely correct and appropriate if, before issuing it, you had issued a "module load" command for a non-default version of the VisIt software and you wish to switch to the default version:

module swap visit visit

Please refer to "module" man page for more details.

Loading Modules into Your Default Environment

You can modify your environment so that certain modules are loaded whenever you log in. Put your changes in one of the following files, depending on your shell.

Here is an example of a .cshrc.ext file with commands to load modules. Notice that, modules specific to a platform are put inside of if-then blocks.

if ($NERSC_HOST == "hopper") then
# Replace the following line with personal settings for Hopper
   module load fftw
endif
Back to Top

Install your own customized modules

You can create and install your own modules for your convenience or for sharing software among collaborators. The module definition files can be in a project directory, your home directory, or any available file system.  Make sure the UNIX file permissions grant access to all users who want to use the  software. (Note: do not give write permissions in your home directory to anyone else!)

An example of such a modulefile "1.2.7" for the customized module "myzlib" is located at /global/project/projectdirs/mpccc/usg/modulefiles/hopper/myzlib.  This modulefile includes a nice feature to detect the current compiler environment on Cray systems (such as pgi or intel, with the definifition of $LCcompiler), and adds the corresponding include and libaray paths automatically (with the definition of $XXX_POST_LINK_OPTS and $XXX_INCLUDE_OPTS) to the compiler wrappers.  To use this module, just do the following:

% module use /global/project/projectdirs/modulefiles/hopper
% module load myzlib/1.2.7 

Note that the "module use" command adds this new directory before other module search paths (defined as $MODULEPATH), so modules defined in this custom directory will have precedence if there are other modules with the same name in the module search paths.  If you prefer to have the new directory added at the end of $MODULEPATH, use "module use -a" instead of "module use" in the above command.

Recommended Links

Google matched content

Softpanorama Recommended

Top articles

Sites

Top articles

Sites

...

 

  1. Jump up ^ Furlani, John L. (September 30 – October 3, 1991). "Modules: Providing a Flexible User Environment". Proceedings of the Fifth Large Installation Systems Administration Conference (LISA V) (San Diego, CA): 141–152. Retrieved 9 February 2014.
  2. Jump up ^ "NICS User Support Kraken Modules Description". The National Institute for Computational Sciences (NICS). Oak Ridge National Laboratory - Department of Energy. 22 May 2011. Retrieved 9 February 2014.
  3. Jump up ^ "NERSC Modules Software Environment". The National Energy Research Scientific Computing Center (NERSC). Office of Science - Department of Energy. 2014-02-07. Retrieved 19 February 2014.
  4. Jump up ^ "Use of Modules for Accessing Multiple Versions of Software". Baseline Configuration. DoD High Performance Computing Modernization Program (HPCMP). 20 Mar 2008. Retrieved 9 February 2014.
  5. Jump up ^ "DoD High Performance Computing Modernization Program Baseline Configuration". Baseline Configuration. DoD High Performance Computing Modernization Program (HPCMP). Retrieved 9 February 2014.
  6. Jump up ^ John L. Furlani & Peter W. Osel (July 2009). "modulefile - files containing Tcl code for the Modules package". SourceForge. man page. Retrieved 9 February 2014.
  7. Jump up ^ John L. Furlani & Peter W. Osel (July 2009). "module - command interface to the Modules package". SourceForge. man page. Retrieved 9 February 2014.

 

  1. http://modules.sourceforge.net/ - home page for the Environment Modules project
  2. John L. Furlani, Modules: Providing a Flexible User Environment Proceedings of the Fifth Large Installation Systems Administration Conference (LISA V), pp. 141–152, San Diego, CA, September 30 - October 3, 1991.
  3. http://lmod.sourceforge.net - alternative implementation using LUA instead of Tcl
  4. http://www.lysator.liu.se/cmod/ - alternative implementation using C only
  5. PennState Environment Modules User Guide
  6. Drag your design environment kicking and screaming into the '90s with Modules! - SNUB Boston 2001 - Erich Whitney, Axiowave Networks, Mark Sprague, ATI Research

 

Environment Modules

SourceForge.net