File input and output
AMUSE can read and write to/from several data formats. The main routines are write_set_to_file and read_set_from_file, explained below.
The AMUSE framework provides a generic way of reading and writing sets of entities (these entities can be particles, gasclouds or gridpoints). AMUSE provides a function to write a set to a file and a function to read a set from a file. These functions need the name of the file and the format of the data in the file. We will describe these functions first in this chapter. The functions can throw 3 kinds of exceptions, these are described next.
To write a data set to a space separated text file do:
>>> from amuse.io import write_set_to_file
>>> from amuse.datamodel.core import Particles
>>> from amuse.units import units
>>> x = Particles(100)
>>> x.mass = 10.0 | units.MSun
>>> x.radius = range(1.0,101.0) | units.RSun
>>> write_set_to_file(x, "test.csv","txt", attribute_types = [units.MSun, units.RSun])
Write a set to the given file in the given format.
filename – name of the file to write the data to
format – name of a registered format or
a FileFormatProcessor
subclass (must be a
class and not an instance)
All other keywords are set as attributes on the fileformat processor. To
determine the supported options for a processor call
get_options_for_format()
Registered file formats:
Process an HDF5 file
Process text files with AMUSE header format
Process comma separated files
Process a Starlab binary structured file
Process a Gadget binary data file
Process an HDF5 file
Process a NEMO binary structured file
Process a Starlab binary structured file
Process a NEMO binary structured file
Process a text file containing a table of values separated by a predefined
Process a text file containing a table of values separated by a predefined character
Process a text file containing a table of values separated by a predefined character
Read a set from the given file in the given format.
filename – name of the file to read the data from
format – name of a registered format or
a FileFormatProcessor
subclass (must be a
class and not an instance)
All other keywords are set as attributes on the fileformat processor. To
determine the supported options for a processor call
get_options_for_format()
Registered file formats:
Process an HDF5 file
Process text files with AMUSE header format
Process comma separated files
Process a Starlab binary structured file
Process a Gadget binary data file
Process an HDF5 file
Process a NEMO binary structured file
Process a Starlab binary structured file
Process a NEMO binary structured file
Process a text file containing a table of values separated by a predefined
Process a text file containing a table of values separated by a predefined character
Process a text file containing a table of values separated by a predefined character
Retuns a list of tuples, each tuple contains the name of the option, a description of the option and the default values.
format – name of a registered format or
a FileFormatProcessor
subclass (must be a
class and not an instance)
Raised when the given format is not supported by AMUSE.
Raised when the given format cannot save data (only reading of data is supported for the format)
Raised when the given format cannot read data (only saving of data is supported for the format)
AMUSE provides it’s own data format based on hdf5 files. The HDF5 file storage provides support for storing particle set and grids in the same file. It also retains the particle references and can store multiple versions of the same set.
>>> from amuse.support import io
>>> from amuse.ic.plummer import new_plummer_model
>>> particles = new_plummer_model(1000)
>>> io.write_set_to_file(
particles,
'plummer.hdf5',
'hdf5',
)
>>> io.read_set_from_file('plummer.hdf5', 'hdf5')
If set to True, data can be written to the file, equivalent of append_to_file even if read_set_from_file is used
If set to True, new data is appended to HDF5 files. If set to False, the existing file is removed and overwritten. Only relevant for write set to file. (default: False)
If set to True, the file is closed after reading, unless you set copy_history to True no previous savepoints will be returned
If set to False, don’t use compression, otherwise use specified compression mode (defaults to False, True means ‘gzip’)
Compression level to use if using compression (0-9, defaults to None)
If set to True, the savepoint history is read from file into memory.
Extra attributes to store with the data set. Some formats (moste notably the amuse native format) can store extra attributes with the set in file. The ‘write_set_to_file’ function will collect all keyword arguments that do not match to an option into the extra attributes dictionary.
A list of names to save the data under. If filled this will load the sets or grids with the given names and return this as a list. When saving the names will be used to save each set, a list of sets is needed in the write_set_to_file function (default: [])
If set to True, overwrite file if it exists, otherwise writing an existing file generates an exception
If set to True, will return a context manager instead of the loaded set(s). This context manager will take care of properly closing any connected resources. To access the set, use the with statement. .. code-block:: python with load_set_from_file(“example.h5”, “amuse”) as particles: print particles Usefull for cases where close_file == False. (default: False)
AMUSE storage version to use, needs to be >= ‘2.0’ if you want to store links between particles and grids (default: ‘2.0’)
AMUSE has support for reading and writing text (.txt
, .csv
) files.
You can specify the txt format by entering “txt” (for space separated files)
or “csv” (for comma separated files)
as the format:
>>> from amuse.support import io
>>> particles = io.read_set_from_file(
'plummer.txt',
'txt',
attribute_types = (units.m, units.m, units.kms),
attribute_names= ('x', 'y', 'vx')
)
>>> io.write_set_to_file(particles, 'plummer.txt', 'txt')
List of the data types of the attributes. If not given, the data types will be float64.
List of the names of the attributes to load or store
List of the units of the attributes to store. If not given the units will be derived from the units stored in the attribute set. When derived from the set, the units will always be converted to the base units.
Separator between the columns
Comments to add to the header
Extra attributes to store with the data set. Some formats (moste notably the amuse native format) can store extra attributes with the set in file. The ‘write_set_to_file’ function will collect all keyword arguments that do not match to an option into the extra attributes dictionary.
format specification string to convert numbers to strings, see format_spec in python documentation
Lines starting with this character will be handled as part of the footer
If true, store the decription of the attributes and units in the header, better format than must_store_units_in_header provides
Lines starting with this character will be handled as part of the header
Output most precise text output for floats (9 digits) and doubles (17 digits)
Column for the key (default is -1, no key stored/loaded). Keys will be interleaved with the data if set other than 0. No attribute type or name can be given for the key.
The maximum number of lines to convert and write out in one go, larger values may be a little faster but will use a lot more memory
Store line with parsable units in the header of the text file. This line can be used to restore the units on reading the file. When this line is added to the file you do not need to specify the attribute_types on reading the file
The precision is a decimal number indicating how many digits should be displayed after the decimal point
List of vector quantities, each vector quantity is one column in the text file. By default this list will be derived from the particles set. When this option is given it will override the particle set data.
“Set the stream to output to
By default, text files are stored with some unit information on a comment line in the header. Unfortunately, amuse cannot depend on this line to read the file. When reading a text file you always need to specify the units of every column in the file. For example, to read a file with particle positions where all positions are stored in parsec, do:
particles = io.read_set_from_file(
'positions.txt',
'txt',
attribute_types = (units.parsec, units.parsec, units.parsec),
attribute_names= ('x', 'y', 'z')
)
When writing a text file, you do not need to specify the units. If
you don’t specify the units the write_set_to_file
function will
default to the units of the quantities being saved. For reliability
and reproducibility we suggest to always specify the units and
the names of the attributes to save when saving a text file:
particles = io.write_set_from_file(
particles,
'txt',
attribute_types = (units.parsec, units.parsec, units.parsec),
attribute_names= ('x', 'y', 'z')
)
The amuse text file routines are very generic and since version 6.0, you can also specify a list of vector quantities instead of a particle set:
particles = io.write_set_from_file(
None,
'txt',
attribute_types = (units.parsec, units.parsec, units.parsec),
attribute_names = ('x', 'y', 'z'),
quantities = [
[0.1,0.2] | units.parsec,
[0.3,0.4] | units.parsec,
[0.5,0.6] | units.parsec,
]
)
AMUSE has support for reading and writing starlab (.dyn
) files.
You can specify the starlab format by entering “dyn” or “starlab”
as the format:
>>> from amuse.support import io
>>> particles = io.read_set_from_file('plummer.dyn','starlab')
>>> io.write_set_to_file(particles, 'output.dyn', 'dyn')
The starlab format support sevaral options, listed below. You can
use these options by adding additional keyword arguments to
the read_set_from_file()
or write_set_to_file()
functions.
For example:
>>> from amuse.support import io
>>> particles = io.read_set_from_file('plummer.dyn','starlab', must_scale = False, return_children = False)
The length fields in the dynamics section of a starlab file can be in parsec or in scaled units , defaults to scaled units (nbody_system.length) When set to scaled units, AMUSE will convert the units if scaling parameters are also given in the file. See the `must_scale` option to turn this scaling off.
The m field in the dynamics section of a starlab file can be in MSun or in scaled units , defaults to scaled units (nbody_system.mass). When set to scaled units, AMUSE will convert the units if scaling parameters are also given in the file. See the `must_scale` option to turn this scaling off
The time fields in the dynamics section of a starlab file can be in Myr or in scaled units , defaults to scaled units (nbody_system.time). When set to scaled units, AMUSE will convert the units if scaling parameters are also given in the file. See the `must_scale` option to turn this scaling off.
Extra attributes to store with the data set. Some formats (moste notably the amuse native format) can store extra attributes with the set in file. The ‘write_set_to_file’ function will collect all keyword arguments that do not match to an option into the extra attributes dictionary.
If True use the scaling values from the file, if False do not scale the stellar dynamics properties. Only used when no nbody to si converter has been set.
Starlab datafiles store stellar dynamics properties in scaled nbody values, provide a converter to store si data (defaults to None). Value None means no converter, or use scaling values provided in the file
If True returns the children of the root node, if False returns the root node (defaults to True)
If True also return the converter when reading a set using read_set_from_file
The units of the values in the star (stellar properties) section of a starlab file are always in derived S.I units (solar mass, million years, solar luminocity etc.).
The masses given in de the dynamics section of a starlab file
are usually in nbody units. Some starlab tools will set the
mass values in Solar mass units (for example the makemass
tool will return the masses in solar mass units). To read these
files you need to set the dynamics_mass_units
.
> makeplummer -n 1000 > plummer1.dyn
> cat plummer1.dyn | makemass -f 1 -x -2.0 -l 0.1 -u 20 > plummer2.dyn
> cat plummer2.dyn | add_star -Q 0.5 -R 5 > plummer3.dyn
> cat plummer3.dyn | scale -s > plummer4.dyn
> cat plummer4.dyn | kira -S > plummer5.dyn
The plummer1.dyn
, plummer4.dyn
and plummer5.dyn
files
will provide masses (and all other dynamical properties) in scaled
nbody units. The plummer2.dyn
and plummer3.dyn
files
will have masses in solar masses. To read each file in AMUSE, and
return the particles with S.I. units, you need to do:
>>> from amuse.support import io
>>> from amuse.units import nbody_system, units
>>> converter = nbody_system.nbody_to_si(1 | units.MSun, 1 | units.parsec)
>>> particles1 = io.read_set_from_file('plummer1.dyn','starlab', nbody_to_si_converter = converter)
>>> particles2 = io.read_set_from_file('plummer2.dyn','starlab', dynamics_mass_units = units.MSun, nbody_to_si_converter = converter)
>>> particles3 = io.read_set_from_file('plummer3.dyn','starlab', dynamics_mass_units = units.MSun, nbody_to_si_converter = converter)
>>> particles4 = io.read_set_from_file('plummer4.dyn','starlab')
>>> particles5 = io.read_set_from_file('plummer5.dyn','starlab')
Note
No nbody converter object is needed for the last files, as the scale factors given in the files will be used.
The plummer1.dyn
, plummer4.dyn
and plummer5.dyn
can also
be read in nbody units. In the following example the returned
particles have dynamic attributes (mass, radius, velocity, acceleration)
in nbody units:
>>> from amuse.support import io
>>> particles1 = io.read_set_from_file('plummer1.dyn','starlab')
>>> particles4 = io.read_set_from_file('plummer4.dyn','starlab', must_scale = False)
>>> particles5 = io.read_set_from_file('plummer5.dyn','starlab', must_scale = False)
AMUSE has support for reading and writing nemo (.tsf
) files.
You can specify the starlab format by entering “nemo” or “tsf”
as the format:
>>> from amuse.support import io
>>> particles = io.read_set_from_file('plummer.tsf','nemo')
>>> io.write_set_to_file(particles, 'output.tsf', 'tsf')
The nemo format support several options, listed below. You can
use these options by adding additional keyword arguments to
the read_set_from_file()
or write_set_to_file()
functions.
For example:
>>> from amuse.support import io
>>> from amuse.units import nbody_system, units
>>> converter = nbody_system.nbody_to_si(1 | units.MSun, 1 | units.parsec)
>>> particles = io.read_set_from_file('plummer.nemo','tsf', nbody_to_si_converter = converter)
Extra attributes to store with the data set. Some formats (moste notably the amuse native format) can store extra attributes with the set in file. The ‘write_set_to_file’ function will collect all keyword arguments that do not match to an option into the extra attributes dictionary.
NEMO datafiles store nbody data, provide a converter to store si data (None means no converter)
AMUSE has support for reading and writing gadget 2 files. You can specify the gadget format by entering “gadget” as the format:
>>> from amuse.support import io
>>> gas, halo, disk, bulge, stars, bndry = io.read_set_from_file('plummer.dat','gadget')
>>> io.write_set_to_file(particles, 'output.dat', 'gadget')
The gadget file format reader will return a tuple of gas, halo, disk, bulge, stars and boundary particles. This is different form other readers which only return one set. The write_set_to_file can take a particles set (will be saved as halo particles) or a tuple of gas, halo, disk, bulge, stars and boundary particles.
The gadget format support several options, listed below. You can
use these options by adding additional keyword arguments to
the read_set_from_file()
or write_set_to_file()
functions.
For example (will read a gadget file with timestep information):
>>> from amuse.support import io
>>> particles = io.read_set_from_file('plummer.nemo','gadget', has_timestep = True)
Set to True to convert the w=sqrt(a)*dx/dt to (comoving) velocity in comoving integrations
The endianness of the binary date stored in the file
If filled with an array with masses > 0.0 assume equal mass for the corresponding set
Extra attributes to store with the data set. Some formats (moste notably the amuse native format) can store extra attributes with the set in file. The ‘write_set_to_file’ function will collect all keyword arguments that do not match to an option into the extra attributes dictionary.
Set to true if the file has a acceleration block
Set to true if the file has a potential energy block
Set to true if the file has a block with the rate of change of the entropic function of each gas particle
Set to true if the file has a block with the individual timestep for each particle.
The format of the header structure of the gadget file.
Set to True if the file contains correct keys. Set to False to generate the keys in amuse and prove an id attribute for the id’s in the gadget file
Set to true the ids will be written as longs in the gadget file
Set to true if the file contains initial conditions. An initial conditions gadget file contains less data.
Set to True to return both the particles and the header from the gadget file
Pass a namedtuple to store non-default values in the header of the gadget file