It is possible to run AMUSE on multiple machines simultaneously. The AMUSE script itself always runs on a users’ local machine, while workers for codes can be “send out” to remote machines such as workstations, clusters, etc.
Deploying workers one remote machines requires a full installation of AMUSE on each machine. For each code “sockets” support needs to be present. This is done by default, and should be available for all codes. Note that older versions of Fortran lack the necessary features to support sockets workers.
On each machine, the distributed code also needs to be build. Distributed AMUSE requires a Java Development Kit (JDK), preferably Oracle Java version 7 or 8. The
configure script tries to locate the JDK, but you may need to specify it by hand. For details, see:
> ./configure --help
To build distributed amuse run the following at the amuse root:
> make distributed.code
To check if the installation is set-up properly, run all the tests related to the worker interface:
> cd $AMUSE_DIR > nosetests -v test/codes_tests/test*implementation.py
Note that Distributed AMUSE is mostly tested with the version of MPI includes in the amuse “prerequisites”. If you get MPI errors while running remote (parallel) workers, try using the install.py script included in AMUSE to install the prerequisites.
Usage of Distributed Amuse is (by design) very close to the usage of any other code in AMUSE. The main difference being it contains resources, pilots, and jobs, instead of particles.
- Description of a machine capable of running jobs.
For each resource distributed AMUSE will launch a support process (HUB) to facilitate communication and coordination between the workers and AMUSE
- Process running on a resource (often within a reservation on a resource) waiting for jobs to run on behalf of AMUSE.
Can consist of multiple machines.
- Worker process.
Will search for a suitable pilot to run on.
In general, a user will first define resources, then deploy pilots on these resources, and finally create codes that make use of the machines offered by the pilots.
Initializing the Distributed AMUSE code¶
Distributed Amuse can be initialized like any other code:
>>> from amuse.community.distributed.interface import DistributedAmuseInterface, DistributedAmuse >>> from amuse.community.distributed.interface import Resource, Resources, Pilot, Pilots >>> >>> #initialize code, print output of code to console >>> instance = DistributedAmuse(redirection='none')
Distributed AMUSE supports a few parameters to adjust settings. All parameters need to be set before any resource, pilot or job is made to have effect.
Overview of settings:
- Boolean parameters, defaults to False.
If true/enabled, will output additional debugging information and logs, both in the code output, and in a distributed-amuse-logs folder on any target machine used.
- Port on which a simple webinterface is available for monitoring.
Defaults to “0”, for a port determined automatically.
- To facilitate communication across different networks (with for instance firewalls), as hub is by default started on each resource.
This can be turned off if needed, for instance if all resources are within the same network.
- The user is responsible for making sure enough slots are available to run a worker.
If not, it will end up in the queue. The time the worker will wait before giving up can be set using this parameter.
- The distributed code starts AMUSE workers running the actual codes.
This can take a while on some machines. If needed, this parameter can be used to increase the time waited.
>>> instance.parameters.debug = True >>> instance.parameters.webinterface_port = 5555 >>> instance.commit_parameters() >>> >>> print instance.parameters.webinterface_port
Distributed Amuse has a small build-in webinterface for monitoring. A utility function is available to get the url:
>>> import webbrowser >>> >>> webbrowser.open(instance.get_webinterface_url())
In order to use a remote machine, AMUSE needs to have some information about this resource such as the host name, type of machine, username to gain access, etc. This can be specified by creating a “Resource” in Distributed AMUSE. As a side effect, a communication hub is also started on the (frontend of) the resource.
>>> resource = Resource() >>> resource.name = "some.resource" >>> resource.location = "email@example.com" >>> resource.scheduler = "ssh" >>> resource.amuse_dir = "/home/user/amuse" >>> >>> instance.resources.add_resource(resource)
Overview of all options:
- Some user chosen name for the resource
- Address of the resource. Usually a hostname (e.g. somehost.somedomain.com). Could also be an IP address
- Location of amuse on the remote machine (e.g. /home/user/amuse-svn)
- Where all temporary files will be put on the remote machine
- Sometimes a machine is not reachable directly due to firewalls and such. Use this setting to provide an intermediate resource to route traffic via. This resource should already have been created.
- The type of scheduler present on the remote machine. Defaults to ‘ssh’ useful for single machines. Current supported scheduler types: ‘ssh’, ‘sge’, ‘slurm’
- Normally the support process is started on the front end. However, it can also be submitted to a queue by specifying it here.
- When a hub is submitted, this option denotes the time the hub will be available.
The next step in running jobs remotely is to start a so-called pilot job on the resource specified previously. This pilot will submit a job to the resource, create necessary communication channels with the main amuse application, and wait for jobs to be started (currently mostly workers)
Note that pilots may not be started for a while. A function is available to wait until all created pilots have started.
>>> pilot = Pilot() >>> pilot.resource_name='local' >>> pilot.node_count=1 >>> pilot.time= 2|units.hour >>> pilot.slots_per_node=22 >>> pilot.label='local' >>> >>> instance.pilots.add_pilot(pilot) >>> >>> print "Pilots:" >>> print instance.pilots >>> >>> print "Waiting for pilots" >>> instance.wait_for_pilots()
Overview of all options:
- name of the resource to start the pilot on
- queue to use to run the pilot (cluster specific, not used in case of ssh)
- number of nodes to start the pilot on
- time to keep the pilot active
- number of workers to start on a node. Usually the number of cores, but could be less if memory is a limiting factor, or workers are multi-core capable
- label to attach to the pilot. Can be used when starting workers to run workers on specific pilots
- Additional options. Usually not required.
When running remote workers, they can be started as normal. However, AMUSE needs to be signalled to use the distributed code to start them instead of the normal process. A function is available to enable and disable this.
>>> print "starting all workers using the distributed code" >>> instance.use_for_all_workers()
>>> print "not using distributed workers any longer" >>> instance.use_for_all_workers(enable=False)
Alternatively, you can also explicitly enable the distributed code per worker
>>> print "using this distributed instance for all distributed workers" >>> instance.use_for_all_distributed_workers(enable=True) >>> worker = Hermite(channel_type='distributed')
Or, even pass the instance of the distributed code you would like to use, in the rare case you have multiple distributed codes
>>> worker = Hermite(channel_type='distributed', distributed_instance=instance)
This section lists all the relevant worker options for Distributed AMUSE. Most are new, some are also supported in the other channel implementations. You are normally not required to use any options.
- Number of worker processes started (thus working as normally the case).
Each worker takes up a slot of the pilot (see above)
- Label of the pilot to use. By default any pilot with enough free slots found will be used to start this worker.
Using the labels an explicit selection can be done.
- Number of threads used in the process.
This can be used to explicitly set the OMP_NUM_THREADS environment variable in the worker
- Set this to “distributed” to start workers using the distributed code.
Alternatively, use the use_for_all_workers functions as described above to set this by default
- This is a reference to the distributed instance used to start the worker, in the rare case you have multiple distributed codes.
- Boolean option stating if this code is a dynamic python code.
If so, all .py files in the worker directory will be copied to the remote machine before starting the code.
By default workers are started on any available pilot with enough slots available. However, sometimes you would like to have more control over which worker is started where, for instance if special hardware is present on some machines.
The concept of labels can be used within Distributed AMUSE to get this functionality. If a label is attached to a worker (one of the parameters when starting a worker, see above), only pilots with exactly the same label (specified when the pilot is started) are considered candidates for running the worker. The name of labels is completely up to the user.
For instance, say a simulation uses a number of workers running on a CPU, and a single GPU worker. The following code will put all the cpu workers on one machine, and the single gpu worker on another.
>>> cpu_pilot = Pilot() >>> cpu_pilot.resource_name='machine1' >>> cpu_pilot.node_count=1 >>> cpu_pilot.time= 2|units.hour >>> cpu_pilot.slots_per_node=30 >>> cpu_pilot.label='CPU' >>> instance.pilots.add_pilot(cpu_pilot) >>> >>> gpu_pilot = Pilot() >>> gpu_pilot.resource_name='machine2' >>> gpu_pilot.node_count=1 >>> gpu_pilot.time= 2|units.hour >>> gpu_pilot.slots_per_node=1 >>> gpu_pilot.label='GPU' >>> instance.pilots.add_pilot(gpu_pilot) >>> >>> ... >>> worker1 = Hermite(label='CPU') >>> worker2 = Bonsai(label='GPU') >>> >>> #will not start due to a lack of slots. >>> worker3 = Bonsai(label='GPU')
AMUSE contains a number of examples for the distributed code. See examples/applications/
Gateways can be used in case of connectivity problems between machines, such as firewalls and private IP addresses. This is for instance the case at the LGM. A gateway is started like any other resource (and thus require a valid installation of AMUSE on each gateway). This resource can then be specified to be a “gateway” to another resource. In this case all ssh connections will be made via the gateway, so make sure you can login from the gateway to the target machine without using a password, as well as from your local machine.
Commonly Encountered Problems¶
Most initial setup problems with the Distributed AMUSE code can be solved by checking:
Can you login to each machine you plan to use using ssh without using a password? See for instance here on how to set this up: https://www.thegeekstuff.com/2008/11/3-steps-to-perform-ssh-login-without-password-using-ssh-keygen-ssh-copy-id/
Did you configure a Java JDK version 1.7 or higher using ./configure? Check the content of config.mk to see which java is used, and what version was detected. Make sure to do a “make clean” and “make” in case you make any changes. This should also be done on all machines.
Is AMUSE configured properly on each and every machine? Running the code implementation tests is a good way of spotting issues:
>>> nosetests -v test/codes_tests/test_*_implementation.py
Are the settings provided for each resource correct (username, amuse location, etc)
Have you set the correct mpiexec in ./configure? This setting is normally not used by AMUSE, so you may only now notice it is misconfigured
In case this does not help, it is probably best to check the output for any errors. Normally worker output is discarded by most scripts. Use ‘redirect=none’ to see the output of the workers, a lot of errors show up in this output only. There is also a “debug” parameter in Distributed Amuse. If enabled, output for each pilot will be in a “distributed-amuse-logs” folder in the home of each remote machine used, and additional information is printed to the log from the local AMUSE script.