Running AMUSE on The Cartesius supercomputer

The Cartesius is the Dutch national supercomputer. For more information on e.g. obtaining an account, see the SURFSara website: https://userinfo.surfsara.nl/systems/cartesius.

Using AMUSE on the Cartesius is relatively straightforward. Below is a tested method for installing and using AMUSE using the prerequisites, though other options (e.g. using only software pre-installed on the machine) should also be possible. For a generic description of the installation of prerequisites, see the Installation of the prerequisite software section.

Obtaining AMUSE

We assume a copy of AMUSE has been downloaded to an amuse folder in the users home. For instance using git:

> cd /home/USERNAME

> git clone https://github.com/amusecode/amuse.git

Environment settings

Since we will be using the AMUSE prerequisites software, we need to set some environment variables. Make sure the following lines are present in your .bashrc file:

#file: ~/.bashrc

#load java 1.7 needed for some codes
module load java/oracle

export PREFIX=~/amuse/prerequisites
export PATH=${PREFIX}/bin:${PATH}
export LD_LIBRARY_PATH=${PREFIX}/lib:${LD_LIBRARY_PATH}

#use gfortran
export FC=gfortran
export F77=gfortran

Also make sure that .bashrc is loaded in your .bash_profile. This way, the enviroment is always set correctly, both in interactive and non-interactive mode.

#file: ~/.bash_profile

if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi

Note: be sure to re-connect to the machine for these changes to take effect.

Install AMUSE prerequisites

Next, we will install all prerequisites of amuse using the AMUSE supplied scripts.

# create a directory for the prerequisites
> mkdir ~/amuse/prerequisites

# go to the <doc/install> directory
> cd ~/amuse/doc/install

# Start the installation script for Python.
> ./install-python.sh

# Download the prerequisite packages.
> ./install.py download

# Install prerequisites. Use hydra as the default MPI process manager. May take a while...
> ./install.py --hydra install

# Optionally also install matplotlib
> ./install.py --matplotlib install

Configure and build AMUSE

Configuring and building amuse is now as normal.

# go to the amuse directory
> cd ~/amuse

# configure amuse
> ./configure MPIEXEC=mpiexec.hydra

# build amuse
> make

# optionally also install codes requiring downloading files
> make DOWNLOAD_CODES=1

Test the installation

To test your AMUSE installation, run nosetests.

Note: do not run simulations on the frontend of the cartesius. This is not allowed!

# go to the amuse directory
> cd ~/amuse

> mpiexec.hydra -n 1 nosetests -v tests

Running on a Cartesius node

Running on the Cartesius is typically done by submitting a slurm script. See the surfsare site for more info:

https://userinfo.surfsara.nl/systems/cartesius/usage/batch-usage

Below is a simple example script for running amuse on Cartesius.

#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -p short
#SBATCH -t 10

cd ~/amuse

mpiexec.hydra -n 1 ./amuse.sh examples/syllabus/gravity_simple.py

Submit using sbatch, get status using squeue, cancel using scancel

#submit a script
> sbatch example-script

#list jobs of current user
> squeue -u $USER

#cancel job 505224
scancel 505224

#cancel all jobs of the current user
> scancel -u $USER

Using multiple nodes should also work, by specifying this to slurm. MPI will automatically pickup on this and spread workers over all nodes.

#!/bin/bash
#SBATCH -N 2
#SBATCH -n 10
#SBATCH -p short
#SBATCH -t 10

cd ~/amuse

#note that this simple example uses only a single worker
mpiexec.hydra -n 1 ./amuse.sh examples/syllabus/gravity_simple.py