Softwares

This section presents the various kinds of software products (binaries, libraries, scripts and tools) available on the center. It is organized as follows. After generally explaining the different kinds of software on the center, we will focus on specific cases (restricted and purchased products). Finally, we will talk about the software life cycle, from the installation request to the product removal.

Generalities on software

Lots of tools and products are available on the supercomputer. A distinction is made between the OS and CCC software.

OS software

They are software products provided by the operating system. They are installed by packages (rpm) and are locally stored on each node.

All common Linux software products enter in this category, such as:

shells (bash, tcsh),
common scripting tools (cat, sed, awk, …)
editors (vi, emacs, gedit, …),
minimal installations of common scripting language (perl, python).

Updates occur during maintenance or during production only if they do not impact the operating system of the node.

CCC software

CCC software products are the third-party dedicated tools or dedicated compilations. They are shared by all center nodes and are installed by TGCC-CCRT dedicated service(s) during production. We prefer the term CCC software products over center software which is ambiguous.

Contrary to the OS software products, CCC software products are not within the standard operating system paths for binaries or libraries (eg. /usr/bin or /usr/lib64).

Each product is installed in a dedicated directory and requires an update of environment variables like and before use. These environment updates are handled by the module tool detailed in the Environment management section.

Updates occur during production as they do not interrupt nor interfere with the operating system installation.

Software and module

A CCC software product requires an environment update before use. The module command is the tool for it, so a module exists for each CCC software product.

An OS software product is usually not referred to by a module, except when:

it is usually the case in other computing centers (ex: tcl)
there is a CCC software version of it (ex: valgrind): it helps to avoid conflicts between the OS and CCC installation.

Products referenced by module have been organized into categories:

applications: simulation products
environment: defines the user/group environment
tools: development tools
graphics: visualization or image manipulation tools
parallel: parallel execution software such as MPI
libraries: third-party libraries
compilers: the various compilers

A domain-oriented view is provided by module search <keyword>. To list the valid keywords use:

module help|show products/keywords

Software toolchains

Warning

Please refer to internal technical documentation to get information about available toolchains.

This approach provides a variety of compilation / runtime environments to users with consistency across products. To consult the list of available builds for a given product, type module help <product>.

The selection between toolchains is made with two modules flavor/buildcompiler/<compiler>/<version> and flavor/buildmpi/<mpi>/<version>. It allows you to change the toolchain used for any loaded product regardless of the Intel compiler and MPI implementation used at runtime, which is determined by your current intel and mpi modules loaded.

More information on the flavor mechanism can be found on the dedicated section.

The module flavor/buildcompiler/<compiler>/<version> lets you choose the version of the Intel compiler whereas the module flavor/buildmpi/<mpi>/<version> lets you choose the MPI implementation and version.

Product Life cycle

Here are the phases of a software product life cycle, explained chronologically from its installation request to its deprecation and removal.

Installation request

New products and/or product update(s) can be requested by mail to the TGCC-CCRT hotline.

Requests are usually accepted if:

the product is the last stable version,
the product is useful to many users,
the product has no license issue.

There are also special cases, namely:

an installation for a group of persons is not possible since these persons are not part of the same user container,
or the product access should be restricted (see Installation request and restricted products),
or the product requires a license server (see Installation request and purchased products).

Installation phase

If the installation request is approved, we will proceed to the installation itself:

for an OS software product, it should be done at the next maintenance (or before if this is deemed safe).
and for an CCC software product, it should take several days.

Note

Those are estimated installation times. If technical difficulties occur the installation may take longer (example: licensed products)

Newly installed CCC software maybe listed with:

module help|show products/newinstall

Production phase

During the production, the software product should be operational. If you detect an issue, a bug or a missing feature, please report it to the TGCC-CCRT hotline.

End-of-life and removal

An OS software product is considered obsolete when either the distribution considers it as such or when an update has been planned for the next maintenance.

For CCC software products, we try to respect the following rules. For each CCC software product, the center favors versions with the least bugs, the most features and the best performances. Hence the last stable version of a product is very welcome (new features, better performances and bug corrections). So when a new version of a product is installed, previous versions may become obsolete. To be more specific, we will only keep:

the latest version,
its last known stable(s) version(s) (for reference purpose),
and the version(s) before retro-compatibility issues (mainly API break).

Python

Available versions and toolchains

To display all Python versions, use the command module av -t python3:

python3/x.y.z
python3/x.y.z(stable)
python3/x.y.z(unstable)

The (stable) version only changes for security issues. The (unstable) version changes everytime a user asks for a new package or an update. We higly recommand the (stable) one for virtualenv users.

There are at least three types of Python distributions:

System:
- Only packages that could be compiled with the operating system GCC
GCC+OpenMPI:
- Compiled with a recent GCC and Open MPI
GCC+OpenMPI+CUDA
- Compiled with a recent GCC, Open MPI and CUDA
- For AI applications (PyTorch, Tensorflow…)

Display available configurations with the command module help python3/<version>.

-------------------------------------------------------------------
 Module Specific Help for /ccc/etc/modulefiles/tools/python3/<version>:

   Software description:
     Name          : Python lang
     Description   : Python programming language
     Version       : <versions>

   Software configuration(s):

     Copy/paste one of the following line ("module purge" could be required).

     Setup(s) for users based on GCC compiler:

       0 : module load gcc/<versions> mpi/openmpi/<versions> flavor/python3/cuda-<version> python3/<version>
       1 : module load gcc/<versions> mpi/openmpi/<versions> python3/<version>

     Setup(s) without specific requirement:

       2*: module load python3/<version>

 -------------------------------------------------------------------

For instance, to load the GCC+Openmpi+CUDA distribution:

$ module load gcc/<version> mpi/openmpi/<version> flavor/python3/cuda-<version> python3/<version>

To list all available Python packages in your current environment, run pip3 list.

Adding new module

Adding new modules in the distribution

In order to request the installation of one or several python modules, please refer to the Installation request part.

Adding new modules locally, through a virtual environment

Load the desired Python environment:

$ module purge ; module load [...] python3/<version>

Create a virtual environment based on this environment:

$ python3 -m venv <my_virtual_env> --system-site-packages

--system-site-packages option indicates that the original Python site-packages will be available to the new environment. Without this option, the new virtual environment will be empty and the environment should be rebuilt from scratch.

Activate it:

source <my_virtual_env>/bin/activate

Import the source code of the Python modules you need to install, for instance: maze-3.0.0.tar.gz (see Data transfers).

Extract the folder from the archive:

tar xfz <module.tar.gz>

Go in the folder containing the installation files:

cd <module>

Install the python module:

python3 setup.py install

You may need to install one or several dependencies to complete the installation. Proceed in the same way for each of those. It is recommended to identify beforehand the aforementioned dependencies either on a local computer and/or by reading the requirements.txt file that should be at the root of the Python package.

Machine learning and artificial intelligence

Pytorch

In order to use Pytorch, you may use the following environment:

$ module purge
$ module load gcc/<version> mpi/openmpi/<version> flavor/python3/cuda-<version> python3/<version>
$ python3
Python 3.x.y (main, XXX XX 20XX, XX:XX:XX) [GCC X.Y.Z] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>>

Tensorflow

In order to use Tensorflow, you may use the following environment:

$ module purge
$ module load gcc/<version> mpi/openmpi/<version> flavor/python3/cuda-<version> python3/<version>
$ python3
Python 3.x.y (main, XXX XX 20XX, XX:XX:XX) [GCC X.Y.Z] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow
>>>

Jupyter-lab

Jupyter-lab is available on irene. You may use the following commands to start Jupyter-lab:

$ module purge
$ module load python3/<version>
# or module load gcc/<version> mpi/openmpi/<version> python3/<version>
# or module load gcc/<version> mpi/openmpi/<version> flavor/python3/cuda-<version> python3/<version>
$ jupyter-lab

Warning

Jupyter-lab is not available with the Intel toolchain.

It is recommended to use NiceDCV for better stability.

Alphafold

We provide access to the DeepMind’s Alphafold software via a container. Included are the OCI image, the complete database and a set of scripts for easier usage.

Available versions

The version available is 2.3.1, and the database includes the latest versions of all files as of December 2022.

Using AlphaFold on TGCC

AlphaFold requires access to the |default_GPU_partition| partition. It may works on other GPU partition. Upon loading the module, you will receive a path to our AlphaFold Readme file.

$ module load alphafold

First import the image using pcocc-rs.

$ pcocc-rs image import docker-archive:${ALPHAFOLD_IMAGE_ROOT}/alphafold-2022.10.24.tar user:alphafold-231

Next, please copy the scripts we have provided to utilize AlphaFold via pcocc-rs.

$ cp -r ${ALPHAFOLDRUN_ROOT}/run_squashfs-flat ${CCCSCRATCHDIR}/<my_test_case>

You have the option to select between monomer or multimer cases. This can be configured in the submission script, with monomer being the default setting. The output directory should exist before sumbitting the script.

$ cat ${ALPHAFOLDRUN_ROOT}/run_squashfs-flat/submit_script_alphafold-231_squashfs-flat.sh
#!/bin/bash
#MSUB -e AF_%J.e
#MSUB -o AF_%J.o

#MSUB -n 1 # nombres de tâches MPI
#MSUB -c |cores_per_GPU_on_default_partition| # number of cores to be booked. You need |cores_per_GPU_on_default_partition| cores to get one GPU.
##MSUB -x #node exclusivity, to be add for big cases. (if you encounter a memory issue)

#MSUB -T 20000
#MSUB -q |default_GPU_partition|    # hybrid may be used as well
#MSUB -A <group>
#MSUB -m work,scratch

module purge
module load alphafold
export CCC_ALPHAFOLD_DATABASE
export CCC_ALPHAFOLD_IMAGE="alphafold-231"

# Use the database stored on hdd to test the impact
#export ALPHAFOLD_USE_DB_HDD=true

# the output directory should exist before launching the job. Adapt to your need.
export ALPHAFOLD_OUTPUT="$(pwd)/output-squash-flat"

# to edit to match your need
export ALPHAFOLD_INPUT=${ALPHAFOLDRUN_ROOT}/exemple_CCRT/input.fasta

# Replace ${ALPHAFOLDRUN_ROOT}/run_pcocc-rs_squashfs-flat_|default_GPU_partition|.sh by the path to your submission script if you want to use a tailored version
ccc_mprun ${ALPHAFOLDRUN_ROOT}/run_squashfs-flat/run_pcocc-rs_squashfs-flat_|default_GPU_partition|_monomer.sh --max_template_date 2020-05-14 --fasta_paths $ALPHAFOLD_INPUT > AF-231-squash-flat.log

Finally, simply submit the script as usual.

$ ccc_msub submit_script_alphafold-231_squashfs-flat.sh

The database includes the following files.

/ccc/products/alphafold-db-2022.10.24/system/default/db/
├── alphafold-db-compressed_hdd.sqsh
├── alphafold-db-compressed.sqsh
├── alphafold-db-flat_hdd.sqsh
├── alphafold-db-flat_t1.sqsh
├── bfd
│   ├── bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt_a3m.ffdata
│   ├── bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt_a3m.ffindex
│   ├── bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt_cs219.ffdata
│   ├── bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt_cs219.ffindex
│   ├── bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt_hhm.ffdata
│   └── bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt_hhm.ffindex
├── mgnify
│   └── mgy_clusters.fa
├── params
│   ├── LICENSE
│   ├── params_model_1_multimer_v3.npz
│   ├── params_model_1.npz
│   ├── params_model_1_ptm.npz
│   ├── params_model_2_multimer_v3.npz
│   ├── params_model_2.npz
│   ├── params_model_2_ptm.npz
│   ├── params_model_3_multimer_v3.npz
│   ├── params_model_3.npz
│   ├── params_model_3_ptm.npz
│   ├── params_model_4_multimer_v3.npz
│   ├── params_model_4.npz
│   ├── params_model_4_ptm.npz
│   ├── params_model_5_multimer_v3.npz
│   ├── params_model_5.npz
│   └── params_model_5_ptm.npz
├── pdb70
│   ├── md5sum
│   ├── pdb70_a3m.ffdata
│   ├── pdb70_a3m.ffindex
│   ├── pdb70_clu.tsv
│   ├── pdb70_cs219.ffdata
│   ├── pdb70_cs219.ffindex
│   ├── pdb70_hhm.ffdata
│   ├── pdb70_hhm.ffindex
│   └── pdb_filter.dat
├── pdb_mmcif
│   ├── mmcif_files
│   └── obsolete.dat
├── pdb_seqres
│   └── pdb_seqres.txt
├── uniclust30
│   ├── UniRef30_2022_02_a3m.ffdata
│   ├── UniRef30_2022_02_a3m.ffindex
│   ├── UniRef30_2022_02_cs219.ffdata
│   ├── UniRef30_2022_02_cs219.ffindex
│   ├── UniRef30_2022_02_hhm.ffdata
│   ├── UniRef30_2022_02_hhm.ffindex
│   └── UniRef30_2022_02.md5sums
├── uniprot
│   ├── uniprot_sprot.fasta
│   └── uniprot_trembl.fasta
└── uniref90
    └── uniref90.fasta

Note

For advanced use of AlphaFold, you may modify run_pcocc-rs_squashfs-flat_|default_GPU_partition|_monomer.sh and run_pcocc-rs_squashfs-flat_|default_GPU_partition|_multimer.sh as needed.

Note

AlphaFold is accessible to members of the alphafold UNIX group. To join, please contact support at please refer to internal documentation to get hotline email or please refer to internal documentation to get hotline phone.

Products list

This section catalogs all the software accessible on irene. For further details about these offerings, please engage with our support team. A number of applications within this list have the capability to execute computations on GPUs, as denoted by the “GPU Support” column. To clarify, this designation is exclusive to end-user applications. However, this does not preclude the possibility of other categories of products, such as tools and libraries, from being capable of running on GPUs. Should you require a GPU-aware version of any application not currently indicated as such, we encourage you to reach out to the support team for support.

The list is available in the dedicated section.

Specific software

The center has specific rules for restricted products and purchased products.

Restricted software products

A restricted product is referenced by module, but its access is restricted to a specific Unix group. Although the reasons may vary, this usually comes from a license requirement.

For example, a licensed product may restrict its usage to academic purposes, or the purchased license is an academic one.

Warning

Please refer to internal technical documentation to get information about this subject.

Installation request and restricted products

Warning

Please refer to internal technical documentation to get information about this subject.

Products with purchased license

Some products (such as Intel compilers) require a license server. We refer to them as licensed products.

Module and licensed products

Here are several adaptations of module for licensed products:

A licensed product module depends on a license module to find its license server.
Although the license module is not shown at load time, a module list can display it.
license module are named licsrv/<server> and are listed by:

module avail licsrv

Installation request and purchased products

While requesting the installation of a purchased product, you may:

have a license not requiring a dedicated server,
have the license and want to restrict its usage to a group of people (because you only have a few tokens),
have a global license (limited in time),
have no license and ask for the center to purchase it.

The two first cases are handled by defining it as a restricted product.

The last case must be clearly justified. If so, you must at least:

estimate the number of user,
specify its application domain,
tell us if you need the academic or the commercial license,
and tell us if you need a global license and/or how many tokens are necessary.

Please note that, even if such installation follows the usual validation process, you may have additional delays, namely:

if you need a non-FlexNet dedicated license server,
if you need to generate some license server identification (done by us and filtered for security reasons) before getting the license.