cpuadoc

cpuadoc(1) General Commands Manual cpuadoc(1)
NAME
 cpu.adoc - Description of computers for parallel processing in Etomo
SYNOPSIS
 /usr/local/ImodCalib/cpu.adoc
DESCRIPTION
 The cpu.adoc file contains descriptions of the computers you wish to
 use when running parallel processes through Etomo with
 processchunks. It is used by Etomo to build the parallel processing
 table and to run processchunks. If you are just using a single
 computer with multiple processors, there are two simple alternatives to
 using the cpu.adoc file: 1) enable parallel processing and enter the
 number of processors in the Settings dialog opened from the Etomo
 Options menu; or 2) define the environment variable IMOD_PROCESSORS to
 be the number of processors (see imodenv). This file also describes
 the GPUs available for processing in parallel with multiple graphics
 cards. If you have a single computer with multiple GPUs, you need to
 use a cpu.adoc; see EXAMPLES below.
 The cpu.adoc file must be placed in the IMOD calibration directory.
 The location of this directory is defined by the environment variable
 IMOD_CALIB_DIR, which has a default value of /usr/local/ImodCalib with
 a default installation of IMOD. You will probably need to create this
 directory. Then, prepare a cpu.adoc file and place it there. There is
 an example cpu.adoc file in the autodoc directory of the IMOD
 distribution; you should be able to use this as a starting point.
 Computer Sections and Attributes
 [Computer = computer_name]
 Each section describes one computer. Computer_name will be used
 by Etomo and IMOD to contact the computer via ssh. As an
 alternative to listing an actual computer name, the name can
 simply be "localhost", in which case programs will be run
 directly rather through ssh. A computer should be listed only
 once; do not have one section with an actual name and one
 section with "localhost".
 Computer attributes are name/value pairs found under a computer
 section, they have the following syntax:
 name = value
 The order of the attributes is not important. All attributes
 are optional; some are entered in order to change the default
 value of an attribute, while others are purely descriptive. For
 the descriptive attributes, you can put anything in them that
 you want. However, any attribute you do not use will be omitted
 from the table, so not using attributes can save screen space in
 Etomo. If all of the users are familiar with the attributes of
 the computers or you are planning to use all of the computers
 every time you run a parallel process, then you do not need to
 use anything but the number attribute.
 number = number_of_cores
 The maximum number of processor cores available for simultaneous
 use on this system. If you do not enter this attribute a single
 core is assumed. Hyperthreading is often not very effective for
 computational processes, so this number should generally be
 limited to the number of true cores. For a report of the number
 of physical cores, enter
 imodqtassist -t
 (For use on clusters / queue servers, please see Queue Sections
 and Attributes, below, for a different interpretation applicable
 to those cases).
 gpu[.local] = 1
 This attribute means that the GPU (graphics processing unit) of
 a video card installed in this computer is available for running
 compatible IMOD processes. In fact, any value besides 1 also
 indicates that there is a GPU, so the entry needs to be
 commented out or removed to make a GPU unavailable. If the
 local modifier is used, then the GPU is only available to an
 Etomo that is being run on this computer. Only video cards with
 CUDA can be used in this way.
 gpu.device = GPU#[,GPU#...]
 Use this attribute instead of gpu = 1 when multiple GPUs are
 available. GPUs will be added to the processchunks call in the
 same order as they are listed in this attribute. GPUs are
 numbered from one in this entry, as in the entries to various
 programs, because "0" refers to the best available GPU rather
 than a specific one. This differs from the numbering (from 0)
 in the output from nvidia-smi.
 type = CPU_type
 This attribute goes into the table under the heading CPU Type
 and can contain a description of the computer's processor.
 exclude-interface =
 batchRunTomo|join|peet|pp|recon|serialSections|tools
 Excludes a computer from the Parallel Processing table depending
 on the interface in use. Only one interface can be excluded per
 computer. "batchRunTomo" refers to the Batch Tomograms
 interface. "pp" refers to the Nonlinear Anisotropic Diffusion
 interface, and the Generic Parallel Process interface. "recon"
 refers to the Build Tomogram interface. "tools" refers to the
 Flatten Value interface, and the Test GPU interface.
 users = user_name[,user_name...]
 A list of the users who have login priviledges on the computer.
 If a user is not on this list, the computer will be excluded
 from the Parallel Processing table when the user runs Etomo.
 This attribute is optional. If it is omitted, the computer will
 not be excluded based on user name.
 speed = CPU_speed
 This attribute goes into the table under the heading Speed and
 can contain the speed of the computer. If you want to put units
 for this attribute in the column header, see the Global
 Attributes section below.
 memory = amount_of_RAM
 This attribute goes into the table under the heading RAM and can
 contain the amount of memory the computer has. If you want to
 put units for this attribute in the column header, see the
 Global Attributes section below.
 os = operating_system
 This attribute goes into the table under the heading OS and can
 contain a description of the operating system the computer is
 running.
 gpu.type = GPU_model
 This attribute goes into the GPU table under the heading Type
 and can contain a model number for the graphics card.
 gpu.ncores = Number_of_GPU_cores
 This attribute goes into the GPU table under the heading Cores
 and can contain the number of CUDA cores.
 gpu.speed = GPU_clock_rate
 This attribute goes into the GPU table under the heading Speed
 and can contain the clock rate of the GPU. If you want to put
 units for this attribute in the column header, see the Global
 Attributes section below.
 gpu.memory = amount_of_GPU_RAM
 This attribute goes into the GPU table under the heading RAM and
 can contain the amount of memory the graphics card has. If you
 want to put units for this attribute in the column header, see
 the Global Attributes section below.
 mountname = name_for_mounting_this_computers_directories_remotely
 This attribute specifies the name substituted for %mountname in
 a mount rule (see below) when running on this current computer
 as the
 mountrule.#.local = local_directory_path
 mountrule.#.local = remote_directory_path
 See the section below on Mount Rules. Mount rules can be
 entered as computer attributes in order to specify a rule that
 applies only for a specific machine or to override a global
 mount rule.
 Queue Sections and Attributes
 [Queue = queue_name]
 Each section describes one queue in a cluster. Queue_name will
 be displayed in the parallel processing table and sent to
 processchunks with the -Q option, but it does not need to match
 the actual name of a queue. Some Computer section attributes
 (e.g. number) are also valid in Queue but sections can have
 altered meaning; most will have no effect. The gpu attribute can
 be used to indicate that a queue will assign one GPU per job,
 which is suitable for use with the processing steps that can use
 a GPU in the Tomogram Reconstruction interface of Etomo. The
 gpu.local and gpu.device attributes will have no effect, and
 various descriptions for the table will not show up anywhere.
 Note that the key name in the section header line does not need
 to be "Queue". If you start Etomo with the --queuesection
 option specifying an alternative key name, then Etomo will load
 sections into the queue table starting with these key names
 instead of sections that start with "Queue". This feature
 allows you to define available queues for more than one
 situation in the same cpu.adoc. For example, with a section
 starting:
 [AlpineQueue = 4Core-1GPU]
 and the option --queuesection AlpineQueue, the table would show
 a queue named "4Core-1GPU".
 number = number_of_processing_units
 The maximum number of jobs to be run simultaneously on the queue
 (required).
 Jobs run from the Etomo Tomogram Reconstruction or PEET
 interfaces are single-threaded and will each use only a single
 core. In these cases, the maximum number of jobs and the maximum
 number of cores are identical, and cores may be scattered across
 multiple nodes.
 Conversely, jobs created with Batchruntomo can be multi-
 threaded and can each use multiple cores if coresPerClusterJob
 is defined. In this case, the maximum total number of cores used
 will be number times coresPerClusterJob.
 For a slurm queue running in exclusive access mode, entire nodes
 must be allocated, so number must be evenly divisible by
 coresPerNode, and the resulting quotient will be the number of
 nodes requested at initialization.
 command = command_to_run_queuechunk
 Required. Queuechunk is a script in IMOD that allows Etomo to
 work with specific types of cluster software, but can be
 modified or replaced to work with any cluster software. See the
 man page for Queuechunk for the list of supported cluster
 types and the specifications that a modified script must meet.
 If the version in IMOD is modified or replaced, it should be
 given a different name, and placed somewhere on the IMOD user's
 path other than IMOD/bin. That name would be used in place of
 "queuechunk" in the command defined in this section.
 A PBS example where there is only one queue:
 command = queuechunk -t pbs
 A PBS example where there are multiple queues:
 command = queuechunk -t pbs -q queue_name
 This command will be passed in quotes to processchunks. It will
 also be used to get the load from the queue.
 initialize = command_to_send_queuechuck_at_start
 Required for a slurm queue set up in exclusive allocation mode.
 Queuechunk will run this command on the cluster machine at
 the beginning of a Processchunks run. For a slurm queue,
 this is needed to allocate the desired number of nodes. To use
 the script slurmInit.sh distributed with IMOD, enter:
 initialize = slurmInit.sh %{nodes} QOS
 where QOS is the quality of service, which is used like a queue
 name in slurm, and %{nodes} is substituted with a value computed
 as explained below for the coresPerNode entry.
 deinitialize = command_to_send_queuechuck_at_end
 Required for a slurm queue set up in exclusive allocation mode.
 Queuechunk will run this command on the cluster machine at
 the end of a Processchunks run. For a slurm queue, this is
 needed to release the allocation. To use the script
 slurmCleanup.sh distributed with IMOD, enter:
 initialize = slurmCleanup.sh
 coresPerNode = #_of_cores_per_allocated_unit
 When using exclusive allocation mode, this entry is used to
 compute the number of nodes to request in the initialization
 command for a slurm queue from the number of cores that the user
 requests. The computed number is substituted for the %{nodes}
 variable in the initialize command.
 coresPerClusterJob #_of_cores_available_to_job
 This entry is used to indicate the number of cores available to
 a job running on a particular queue; it is used when running
 Batchruntomo in parallel. Do not use this entry with a slurm
 queue running in exclusive allocation mode.
 gpusPerClusterJob = #_of_GPUs_available_to_job
 This entry is used to indicate the number of GPUs available to a
 job running on a particular queue; this is used when running
 Batchruntomo in parallel. The attribute gpu = 1 can be used
 instead when there is only one GPU. Also, this attribute was
 originally named gpusPerNode prior to IMOD 4.12.40, and that
 name is still accepted. Do not use this entry with a slurm
 queue running in exclusive allocation mode.
 pc-option.option = value
 This entry will pass the specified option to Processchunks
 with the given value only when running on this queue. See the
 pc-option.option description in the the Global Attributes
 section below. An entry here will override a value specified in
 the Global section.
 Global Attributes
 Global attributes are name/value pairs found at the top of the file,
 before any computer sections. They typically have the following
 syntax:
 nameA.nameB = value
 The order of the attributes is not important.
 Version = 1.2
 The Version attribute is usually required. It refers to the
 syntax used in .adoc files and should be set to 1.2.
 units.speed = units_for_the_speed_attributes
 The value appears in sub-heading row in the Etomo parallel
 processing table under the heading Speed.
 units.memory = units_for_the_memory_attributes
 The value appears in sub-heading row in the Etomo parallel
 processing table under the heading RAM.
 units.gpu.speed = units_for_the_GPU_speed_attributes
 The value appears in sub-heading row in the Etomo GPU parallel
 processing table under the heading Speed.
 units.gpu.memory = units_for_the_GPU_memory_attributes
 The value appears in sub-heading row in the Etomo GPU parallel
 processing table under the heading RAM.
 units.load = queue_load_header[,queue_load_header...]
 Shows how many comma-separated values will be returned by
 queuechunk when it returns a queue's load and provides header(s)
 for them in the Etomo parallel processing table. The default is
 "Load".
 max.tilt = maximum_number_of_cores_users_should_use_for_tilt
 When this attribute is in use, a reminder not to use too many
 cores will appear next to the tilt parallel processing checkbox.
 This number must be a whole number. The purpose of this
 attribute is to discourage users from using too many cores and
 from experiencing diminishing returns because of bottlenecks.
 We are currently setting max.tilt to 12, but this is an old
 limit.
 max.volcombine =
 maximum_number_of_cores_users_should_use_for_volcombine
 Similarly to max.tilt, this attribute will cause a
 recommendation on the maximum number of cores to appear next to
 the volcombine parallel processing checkbox. This step is much
 more susceptible than tomogram generation to diminishing returns
 from I/O bottlenecks. We are currently setting max.volcombine
 to 8.
 separate-chunks = value
 This attribute can be used to force programs to write chunks
 into separate files that will be reassembled at the end of
 processing. Without this setting, different instances of the
 Tilt program will all write to one output file simultaneously,
 which has given problems in one Mac installation. Any value
 other than 0 activates the feature.
 min.nice = minimum_nice_value
 The minimum value of the Nice spinner. The default is 0.
 users-column = [0|1]
 When this attribute is present and not set to 0, the Users
 column will be included in the Parallel Processing table.
 mountrule.#.local = local_directory_path|remote_directory_path
 See the section below on Mount Rules.
 pc-option.option = value
 This and other pc-option attributes allow the specified option
 to be passed to Processchunks with the given value. Such
 options are placed at the end of the argument list so that they
 can override the value passed by Etomo, if any. The option
 would be specied without a leading dash. Meaningful options
 that would not conflict with the ones managed by Etomo would be
 d, e, C, T, L, v, V, and possibly n and O. An option specified
 with this entry would be passed on all Processchunks runs,
 although the value may be overridden by other more specific
 entries.
 pc-option.computer.option = value
 An option specified with this entry will be passed only on
 Processchunks runs with computers, not queues. The value
 would override a value specified with a global pc-option.option
 entry.
 pc-option.queue.option = value
 An option specified with this entry will be passed only when
 running on queue. The value would override a value specified
 with a global pc-option.option entry.
 Mount Rules for Local to Remote Path Translations
 In order to use parallel processing in IMOD, all computers must be able
 to access the directory where the data and command files are located.
 However, it is not necessary that the directory be referred to by the
 same name on the different computers. When these names differ, you
 must provide Etomo with information about how to translate the current
 working directory path on the local computer into a path that can be
 used to access the directory on the remote computers. This gets tricky
 because the true path of a directory, as revealed by a pwd command, may
 not be the same as the path that the user enters to get there. Thus,
 in setting up path translations, you need to change to a typical
 directory and then use pwd to find out what the official path to the
 directory is. This is the path that Etomo will see on the local
 machine, so you need to work out how this needs to be translated so
 that it can be accessed on the remote machines.
 As a simple example, each Linux machine in our laboratory used to have
 a directory named /localscratch which was accessed from any machine as
 /scratch/computer_name (where computer_name is the name of a machine,
 without any domain). The required mount rules were entered as:
 mountrule.1.local = /localscratch
 mountrule.1.remote = /scratch/%mountname
 Where %mountname is entered exactly as written and will be substituted
 for the appropriate mount name. In our example, the mount name is just
 the computer name, but a mount name different from the computer name
 can be entered for an individual computer using the mountname
 attribute.
 For a complicated example, we had a Macintosh running OSX 10.4, and it
 mounted our Linux home directories (/home, /home1, /home2) under the
 same names. It mounted the Linux machine scratch directories under
 /scratch/computer_name. However, when we were running on the Mac and
 cd'd to a user's home directory and entered pwd, we got, e.g.,
 /private/var/automount/home1/username. When we cd'd to a Linux scratch
 directory and entered pwd, we got /private/var/automount/computer_name.
 The correct translations can be accomplished with:
 mountrule.2.local = /private/var/automount/home
 mountrule.2.remote = /home
 mountrule.3.local = /private/var/automount
 mountrule.3.remote = /scratch
 The numbers specify the order in which the rules are applied. Note
 that it is important to apply the rule for home first to avoid having
 /private/var/automount/home get translated to /scratch/home. Also
 note that this one rule works for /home, /home1, and /home2. The
 automount names no longer do this on OSX 10.5 and higher, but the
 example is still good for illustrating how to deal with complex
 situations.
 Our Linux machines also used to access the home directories under
 /Users on the Mac, by mounting these directories as
 /computer_name/username. So we had another mount rule:
 mountrule.4.local = /Users
 mountrule.4.remote = /%mountname
 All of the rules in our two examples are compatible, so they could all
 be listed as global mountrules in the same cpu.adoc. If this were not
 the case, we could still maintain one file by listing some rules as
 local rules, inside the section for a particular computer.
 Here are some other facts about mount rules. The current directory is
 checked for substitution against one rule at a time, and if it matches
 a rule then the substitution is made and no other rule is checked.
 Local rules for the current host machine, if any, are checked before
 the global rules.
 It is required to have a local rule and a remote rule with the same
 number and in the same area (global attributes area or Computer
 section). Each mount rule attribute must have a value.
 When %mountname is used, then a Computer section for the current host
 computer must exist, or there must be a Computer section called
 localhost. In the latter case, a mountname attribute is required for
 that section.
EXAMPLES
 A cpu.adoc for a standalone four-core system would be just:
 Version = 1.2
 [Computer = localhost]
 number = 4
 A cpu.adoc for a standalone 16-core system with 4 equivalent GPUs would
 be:
 Version = 1.2
 [Computer = localhost]
 number = 16
 gpu.device=1,2,3,4
 Both of these files are in fact unnecessary because they can be
 configured in the Etomo Options-Settings dialog. See
 $IMOD_DIR/autodoc/cpu.adoc and Splitbatch for further examples.
LIMITATIONS
 Windows computers may not be placed in the same cpu.adoc parallel
 processing table as Linux and Macintosh computers.
 All computers in the cpu.adoc will be loaded into a scrollable table in
 Etomo and ssh connections will be opened to each one to monitor its
 load. A cpu.adoc with many tens of computers may slow down Etomo too
 much.
SEE ALSO
 queuechunk(1)
IMOD 5.2.6 cpuadoc(1)