batchruntomo

batchruntomo(1) General Commands Manual batchruntomo(1)
NAME
 batchruntomo - Align tilt series and generate tomograms automatically
SYNOPSIS
 batchruntomo [options] directive_files...
DESCRIPTION
 Batchruntomo runs all the operations required to align a tilt series
 and build a tomogram, following a set of instructions called "direc-
 tives" provided in a text file. These directives indicate either
 parameter settings or which pathway to follow at various points in the
 processing sequence. The program will perform operations in a fixed
 order determined by these choices; the order of the directives does not
 affect this order. The program can work on more than one data set and
 can also move the tilt series from their current location into a sepa-
 rate directory for each data set, which is created if necessary. It
 accepts tilt series with extensions ".mrc", ".st", ".hdf", ".tif", and
 ".tiff". Depending on the image file naming style specified, it may
 change the extension to ".st" so that other programs will work cor-
 rectly.
 Directives
 Directives consist of key-value pairs separated by equals signs, where
 the key has 2 to 4 components separated by periods. The first compo-
 nent is a prefix that indicates the type of directive or the stage at
 which it is applied. The value may be blank if appropriate but the
 equals sign must be present. There are currently three prefixes:
 "setupset", for directives involved when setting up a data set; "com-
 param", for directives used to set options in command files; and "run-
 time", for directives interpreted by Batchruntomo or Etomo. A descrip-
 tion of all directives is contained in the file
 "IMOD/com/directives.csv", which is in comma-separated values format.
 It is accessible in a more readable form from the IMOD help index via
 the link "Directives for batch processing and Etomo templates" (avail-
 able from "imodhelp" or when accessing the html package of IMOD docu-
 mentation with a browser).
 "setupset" directives have two forms. The majority have the form
 setupset.copyarg.<copytomocoms short option name>=value
 These directives will be processed mostly generically by Etomo and sent
 to Copytomocoms as command line arguments. The short option name
 must be given in full. Unlike for the other two kinds of directives,
 whenever there is a pair of directives corresponding to a pair of
 options for each axis of a dual axis set, a directive for the first or
 only axis does not apply to the second axis. A separate "b" directive
 must be supplied.
 Other "setupset" directives are interpreted exclusively by Batchruntomo
 or Etomo and have the form
 setupset.<parameter or option>=value
 "comparam" directives are interpreted generically when a command file
 is created either by Copytomocoms or by Makecomfile. They have
 the form
 comparam.<root name of file>.<program>.<Long option name>=value
 The root name of the command file may include an axis letter ("a" or
 "b"). For a single axis data set, directives without a letter and with
 an "a" are treated as equivalent and ones with "b" are ignored. For a
 dual axis data set, a directive without a letter can apply to either
 axis, but will be overridden by a directive with a letter matching the
 specific axis, regardless of which one occurs later in the processing
 sequence. The long option name must be given in full. Options to
 Tilt must match the case shown in the Tilt man page, even though
 Tilt accepts case-insensitive options.
 Because "comparam" directives are treated generically, they are not
 restricted to the ones listed in the table of directives. Any valid
 option can be used, although some would be meaningless in the context
 of automation and some would conflict with the management provided by
 Copytomocoms or be overridden by Makecomfile.
 "runtime" directives have the form
 runtime.<Processing step>.<axis>.<parameter or option>=value
 Here the processing step and parameter have names that have been made
 up just for the directive. The convention is for the processing step
 to be capitalized and the parameter to start with lower case. The axis
 must be either "a", "b", or "any". For a single axis data set, direc-
 tives with "any" or "a" are treated as equivalent and ones with "b" are
 ignored. For a dual axis data set, a directive with "any" can apply to
 either axis, but will be overridden by a directive with a letter match-
 ing the specific axis, regardless of which one occurs later in the pro-
 cessing sequence.
 Boolean directives must have a value of either 0 or 1: 1 to turn the
 option on or 0 to turn it off; no other values (including blank) are
 allowed. If a boolean directive is not present then the option is off,
 unless it has been turned on by an entry in a template file (see
 below). For any other kind of directive, the value can be blank, which
 causes the option in question to be removed from consideration.
 Specifically, for a "comparam" directive, a blank value will cause the
 option to be removed from the command file if present. For other kinds
 of directives, a blank value removes the setting of the parameter by a
 previous directive in a template file.
 If a processing step is being skipped or bypassed by an alternative
 method of processing, then all the directives related to it have no
 effect on the processing that is done. It is thus generally safe to
 leave unused directives in a file. Such directives with prefix "com-
 param" will be used to set parameters in command files produced by
 Copytomocoms, but directives for parameters in other command files
 will have no effect because those files will not be produced. Direc-
 tives with prefix "runtime" are simply ignored if they are not relevant
 to the chosen processing sequence.
 Template Files
 Template files can contain parameters that can be used meaningfully for
 multiple, similar data sets. Directives exist to define a microscope
 template file, a system template file, and a user template file. The
 idea is for the microscope file to have parameters specific to a micro-
 scope or microscope/camera combination, and to have the other template
 files contain other parameters that would not be dataset-specific.
 Each file consists of a subset of directives just like those described
 for defining options for the batch processing. There is a heirarchy
 among these directives. They are always processed in the order scope
 template, system template, user template, and batch directive file, and
 entries later in this sequence override ones that appeared earlier.
 Batchruntomo reads the template files and uses their contents along
 with those of the directive file to determine the values for parame-
 ters. It also passes whatever template files exist, plus the batch
 file, to Etomo for it to use when setting arguments for calling Copyto-
 mocoms(1), and Etomo passes the files on to Copytomocoms to use for
 setting parameters. Batchruntomo passes them to Makecomfile to use
 similarly when making optional command files, as well as to Setupcom-
 bine(1).
 Another directive file, "batchDefaults.adoc" from the IMOD "com" direc-
 tory, provides some default values needed when running with batch pro-
 cessing. As of IMOD 4.10.15, these are read in to Batchruntomo and
 processed prior to the templates. Prior to that, Etomo appended these
 directives to the batch directive file unless there was an overriding
 value in a template. Unfortunately, if such a file is used as a start-
 ing batch file in the Etomo interface, these directives are indistin-
 guishable from others and override corresponding template values, if
 any, instead of being overridden. Batchruntomo now recognizes this
 situation and will issue a warning if any such directives do override a
 template value.
 The program checks all directives after it reads them, including the
 ones from a template file, against the master list in "directives.csv".
 All "setupset" and "runtime" directives must match ones in this file.
 It is possible to enter "comparam" directives not listed in this file,
 provided that the command file name and program name match those of
 other directives in the master file. The program will consult the
 autodoc file for the program to determine whether the option is valid.
 It will issue a warning if an autodoc cannot be found; otherwise, it
 will be able to determine validity definitively.
 Processing Steps Available
 This section describes some of the capabilities and current limitations
 of various steps.
 X-ray removal is optional; if selected, automatic X-ray removal will be
 done, and "clip stats" will be run on the fixed stack. There is also a
 directive to run Archiveorig to make a compressed difference between
 corrected and original stacks, then delete the original stack. A man-
 ual replacement model can be specified. It is suggested that object 1
 contain patches specified in the default way, by a point on each pixel;
 object 2 contain lines, if any; object 3 contain patches specified by
 boundary contours, if any; and that directives be added to specify
 these choices and that objects 1-3 are all-section objects. These sug-
 gestions will be enforced when there is a GUI for setting up batch
 directives. If the manual replacement model is located elsewhere, it
 will be copied to the dataset and renamed to the standard name for such
 models, so that the model can be opened and added to when running in
 Etomo.
 Detection of blank images and images with large dark regions at the
 highest tilt angles can be done before further processing by including
 the preprocessing directive "endExcludeCriterion" with a fractional
 value such as 0.5. (The directive is ignored for montaged tilt
 series.) The program will look at the standard deviation (SD) of
 images at each end of the tilt series and find ones whose SD is less
 that the criterion times the mean SD of the five adjacent less-tilted
 images. Up to three images can be excluded on each end of the series
 by this analysis. The program will then do a histogram analysis with
 Clip to see whether the intensity distribution has two peaks and how
 much of the area falls in the lower peak. A view is excluded if inten-
 sity at the lower peak is less than 0.17 times that at the upper peak
 and more than 0.33 of the pixels are in the lower peak (these defaults
 can be modified with "darkExcludeRatio" and "darkExcludeFraction"
 directives). Views can be eliminated up to 6 tilts from either end in
 this way, but with either analysis, only consecutive views will be
 eliminated. View skipping/exclusion lists will be added or adjusted in
 xcorr.com, track.com, align.com, and tilt.com.
 Coarse alignment is always performed. The binning of a prealigned
 stack can be specified.
 Fiducialless alignment can be chosen, in which case tomogram thickness
 must be specified. The tomogram will not be reoriented at all (no
 angle offset or X-axis tilt applied).
 Patch tracking can be chosen. Again, tomogram thickness must be speci-
 fied, but in this case the tomogram will be reoriented based on angles
 found by Tiltalign. Also, if you specify gold erasing after 3D bead
 finding, the thickness must be specified for the tomogram used to find
 the beads. Patch tracking can be improved by using Varypatchtrack(1)
 to vary patch size and filtering, among other parameters. It can be
 run with default parameters using the runtime directive PatchTrack-
 ing.varyPatchTrack, and various "comparam" directives can be used to
 control its operation.
 Gold fiducial tracking can be done either with automatic seed-finding
 then bead tracking, or with RAPTOR followed by optional bead tracking.
 For a second axis, Transferfid can be run, followed by automatic
 seed-finding to fill in the resulting model. RAPTOR can be run instead
 for the second axis (unlike in Etomo), although there will then be no
 list of corresponding fiducials. Beadtrack can be run more than
 once, because this completes a model more reliably than selecting fur-
 ther rounds of tracking in a single run.
 Fine alignment with Tiltalign can include local alignments in all
 cases. There is a directive to enable the stretching (distortion)
 solution. When enabled, stretching will be solved for only if there
 are gold fiducials on two surfaces, and if there are enough on the
 minority surface overall and relative to the total number. Stretching
 may be solved for in the global solution but not in the local one if
 there are too few fiducials on the minority surface per local area. In
 addition to making these decisions, Batchruntomo also runs the program
 Restrictalign, which will use cross-validation to test whether the
 number of variables being solved for should be reduced. This poten-
 tially lengthy process will be run once with global variables, then
 once again if either stretching or local alignments are added.
 Restrictalign will turn off robust fitting if it gives no benefit,
 but if it does so when evaluating global variables and local alignments
 are being done, Batchruntomo will turn robust fitting on again for the
 local tests. The fine alignment produces an angle offset and an X-axis
 tilt that is applied when building the tomogram, unless there are too
 few gold. In addition, if there is gold on two surfaces, the distance
 between the gold particles provides an estimate of thickness that can
 be used as one basis for the tomogram thickness. Otherwise, if gold is
 to be erased after 3D bead finding, the thickness for the required
 tomogram must be specified.
 Tomogram positioning can be done for both plastic sections and cryosam-
 ples by setting the "Positioning.sampleType" directive to 1 or 2,
 respectively. Plastic section positioning is done with a binned-down
 whole tomogram for now. If there is no "Positioning.binByFactor"
 directive, the program will pick a binning up to 4 that brings the
 binned size under 512 in its largest dimension. If there is no "Posi-
 tioning.thickness" directive, the program will pick a thickness based
 on the largest dimension: 250 if size under 512, 400 if size under
 1024, 500 if size under 2048, and 600 otherwise. The program will run
 Findsection and try to make a model for Tomopitch with 5 pairs of
 sample lines. Cryosample positioning is done with the script Cryoposi-
 tion(1). In this case, the "Positioning.thickness" directive is
 required. Cryoposition picks an appropriate binning, but if you do
 want to set this binning, it can be done with a "cryoposition.Binning-
 ToApply" directive instead of the runtime "Positioning.binByFactor"
 directive. By default, 25 unbinned pixels will be added on each sur-
 face by Tomopitch; this can be changed with a "tomopitch.ExtraThick-
 ness" directive.
 When there is no positioning, angles are set from the Tiltalign out-
 put. The Z height of the specimen in the tomogram will be whatever
 location results from the coarse alignment; thus the specimen will be
 centered in Z if features contributing to the cross-correlations are
 evenly distributed in depth. If the correlations are dominated by fea-
 tures near one surface, the specimen will not be centered in Z. If
 there are fiducials on two surfaces, one can use a "Position.centerOn-
 Gold" directive to apply the Z shift computed by Tiltalign for cen-
 tering on the range of fiducials in Z. Another side-effect of using
 this directive is that the estimated thickness from alignment will be
 larger so that the tomogram will then contain all of the gold included
 as fiducials.
 The aligned stack can have specified binning and size. Optional modi-
 fications of the aligned stack are performed in the standard order, or
 with gold erasing first if the -erase option is entered. CTF correc-
 tion can be done with a single nominal defocus value. Alternatively,
 Ctfplotter can be used to find the defocus automatically if the
 "autofitRangeAndStep" directive is present. When Ctfphaseflip is
 run, it will use a "setname.defocus" file if it is present; otherwise
 it will use a "setname_simple.defocus" file created with the nominal
 defocus from the "setupset.copyarg.defocus" directive. (The "set-
 name.defocus" file could derive from manual or automatic running of
 Ctfplotter, or could be created by other means). Gold erasing can
 be done with automatic finding of beads in a tomogram, or by transform-
 ing the fiducial model (or the model put out by Tiltalign that has
 gaps filled in, if available). For finding beads in the tomogram, the
 "GoldErasing.thickness" directive must be supplied unless there are
 beads on two surfaces. The binning is set as the unbinned bead size
 divided by 5 and rounded to an integer, as in Etomo; thus, the "Gold-
 Erasing.binning" directive is no longer required. Both the automatic
 finding of beads in 3D, and the detection of defocus, are steps that
 one might want to check before proceeding. To allow intervention at a
 single stopping place, CTF correction and gold erasing steps are inter-
 leaved: defocus is detected, beads are found, CTF is corrected, then
 beads are erased. 2D filtering and dose weighting can be done by
 adding the directive "AlignedStack.filterStack". There are various
 directives to enter the Mtffilter parameters for dose weighting. To
 use an .mdoc file or the information inside an HDF file from SerialEM,
 add the directive "mtffilter.TypeOfDoseFile" with value 4. The entry
 for the dose information file in the command file is managed appropri-
 ately by Copytomocoms except in one case: using the old naming style
 with descriptive extensions when the input stack is an MRC file with an
 extension other than ".mrc". In this case you need the "mtffil-
 ter.DoseWeightingFile" directive with either the name of the .mdoc file
 or a generic entry that works for multiple data sets: the extension
 before the ".mdoc" in the name, typically ".st".
 The tomogram can be computed with regular weighted back-projection,
 with SIRT, or with a back-projection using a SIRT-like filter. The
 iterations to leave can be specified for SIRT. The regular back-pro-
 jection can be computed in the same run as a SIRT or SIRT-like one by
 adding the directive "Reconstruction.doBackprojAlso". The thickness
 can be chosen automatically if there are gold beads on both surfaces or
 if tomogram positioning was done, otherwise it must be specified with a
 directive. The "Reconstruction.fallbackThickness" directive is the
 most useful one because it can be used to specify a thickness to be
 used when there is no thickness available from either fine alignment or
 positioning. The fallback thickness will also be used if the thickness
 from either is less than 40% of the fallback thickness, which would be
 the case if there are gold on only one surface and no positioning was
 done.
 A 3D CTF-corrected tomogram can be computed by adding the directive
 "ctf3dsetup.SlabThicknessInNm"; "ctf3dsetup.RunSlabsInParallel" can be
 added to make slabs in parallel and "ctf3dsetup.UseUnalignedImages" can
 be added to reconstruct directly from the unaligned images (the raw
 stack). The result will be given the name of the regular weighted
 back-projection tomogram. You can make a SIRT or SIRT-like filtered
 tomogram in addition to the 3D CTF-corrected one. Steps for modifying
 the aligned stack are skipped when making just the 3D CTF-corrected
 tomogram, and they are applied during the reconstruction step.
 Tomogram combination for a dual-axis dataset is possible with all kinds
 of alignments and will proceed automatically. The program first uses
 Findsection to find the Z limits, and decides whether to get the
 initial registration between volumes with Solvematch or Dualvol-
 match(1). The patch correlations and fitting are done with Autopatch-
 fit(1); by default, it will try medium to extra-large patches and add
 target residual values of 0.4,0.45 for Findwarp on the final trial.
 There are "runtime.Combine" directives to control this behavior and to
 set a few other parameters for tomogram combination; also, "comparam"
 directives can be used to modify parameters in solvematch.com, patch-
 corr.com, and matchorwarp.com. If both SIRT or SIRT-like and regular
 back-projection reconstructions are available, the latter will be com-
 bined unless the combine "doSIRTifBoth" directive specifies otherwise.
 The decision of whether to match A to B instead of B to A is made based
 on the ratio of the B to A thicknesses determined from their Z limits.
 This ratio is compared to a criterion that can be set with the "matchA-
 toBThickRatio" directive and whose default is 0.9. With this crite-
 rion, A will be matched to B if B is thinner than A by more than ~10%.
 Lowering the criterion would make it match A to B only when the thick-
 ness difference is more extreme. Raising the criterion toward 1 would
 make it match A to B with less thickness difference. It can be set to
 a very low number (0.2) to guarantee that B is always matched to A, or
 a very high number (5) to have A always matched to B.
 Postprocessing can be done with Trimvol, and should be selected with
 the directive "runtime.Postprocess.any.doTrimvol". For compatibility
 with old behavior, Trimvol will still be run if any of the "run-
 time.Trimvol" directives "findSecAddThickness", "reorient", "thick-
 ness", "sizeInX", "sizeInY", "scaleFromX", "scaleFromY", or "scale-
 FromZ" are present. Findsection will be used to find the section
 limits if the "findSecAddThickness" directive is present. The output
 file for a final reconstruction will have the usual name, setname.rec.
 For a dual-axis data set, it is possible to trim one or both of the
 single axis reconstructions; here the output file will be named with
 "_trim.rec". If both SIRT or SIRT-like and regular back-projection
 reconstructions were made, the postprocessing "doSIRTifBoth" directive
 controls whether to trim one or both reconstructions; in the latter
 case the file for the back-projection will be named "_BP.rec". When-
 ever a SIRT reconstruction is trimmed, the result on the final itera-
 tion will be used. By default, the trimmed volume will be reoriented
 by rotating around X. The entire volume will be used unless the "find-
 SecAddThickness", "thickness", "sizeInX", or "sizeInY" directive is
 present. If one of the "scaleFrom" directives or the "scaleToMeanSD"
 directive is present, the volume will be scaled to bytes. If "scale-
 FromX" or "scaleFromY" is not present, the area used for scaling in X
 or Y will be the Trimvol default of 80% of the range; if "scaleFromZ"
 is not present, the central one-third of slices will be used. All of
 these size and scaling directives may be specified either as a number
 of pixels, or as a fraction between 0.02 and 1.
 Nonlinear anisotropic diffusion (NAD) with Nad_eed_3d can be run if
 both "runtime.NAD" directives are present for specifying the number of
 iterations and the K value. Trimvol must be run at least to create
 the input file for the operation.
 Reduction and Filtering The final volume can be reduced and/or filtered
 by Reducefiltvol, where the reduction is done with antialiased fil-
 tering and can be done by a non-integer amount. The directive "run-
 time.Postprocess.any.doReduceFilt" must given, along with at least one
 of the directives for "ReductionFactor", "LowPassRadiusSigma", or
 "DeconvolutionStrength". Trimvol must be run at least to create the
 input file for the operation.
 Data set cleanup can be done with Tomocleanup by adding the "run-
 time.Cleanup.doCleanup" directive. There are several "run-
 time.Cleanup.keep..." directives to retain particular intermediate data
 set files. The default is for Batchruntomo to include the Tomo-
 cleanup(1) option -sirt to retain the output from running a SIRT or
 SIRT-like reconstruction as well as a back-projection, but this can be
 overridden with a directive.
 Stopping and Restarting
 The -start and -end options allow datasets to be started and stopped at
 chosen steps in the sequence. The most useful step to stop at would be
 10, to check CTF plotting and/or gold detection, then restart at 11.
 To check fine alignment before going on, stop and restart at 6.
 If you are setting up a situation where parameters are being changed
 before restarting in an existing dataset, you should experiment to make
 sure the changes have the desired effect and are not overridden. Here
 are some general guidelines on that point:
 1) Changes in "comparam" directives will generally have no effect once
 a command file is created, because they are processed generically dur-
 ing command file creation. In most cases, creation occurs during tomo-
 gram setup, but some files are created with Makecomfile just when
 needed, so it would be possible to change "comparam" directives for a
 file still to be created. Some exceptions to this are "align.tiltal-
 ign.SurfacesToAnalyze" and "tilt.tilt.THICKNESS", for which the current
 directive value will be used instead of the value in the command file.
 2) Changes in "setupset" directives will also generally have no effect
 because they are interpreted during setup; the use of the defocus value
 mentioned above is an exception.
 3) Changes in "runtime" directives will generally have the desired
 effect.
 4) Changes in parameters in the command files will generally have the
 desired effect, unless that parameter is managed by Batchruntomo on the
 basis of "runtime" directives and other information. The biggest exam-
 ple of this is the treatment of "align.com" when Tiltalign is rerun;
 the -use option is provided specifically to avoid rerunning Tiltal-
 ign(1).
 Resource Usage
 The CPU list provided with the -cpus option is passed directly to Pro-
 cesschunks(1) for running operations that are divided into multiple
 chunks. However, other operations that run from a single command file
 will generally be run on the local machine, unless the -single option
 is used to direct them to the first machine in the CPU list. The
 exceptions are two time-consuming steps in dual-axis combination,
 Matchvol and Autopatchfit, which benefit from using multiple
 threads and are sent to the machine with the highest number of CPUs.
 When running single command files, the program sets a limit on the num-
 ber of threads equal to the number of CPUs designated in the CPU list
 for the machine being used, if it is known (-limit can be entered if
 this is not otherwise known for the local machine).
 Batchruntomo can be run in parallel on multiple data sets, and in this
 case it can also be run on a cluster queue. See the section OPTIONS
 FOR PROCESSING DATA SETS IN PARALLEL for the options and environment
 variables involved; see the Splitbatch man page for descriptions of
 resource usage and how to use a cluster.
 Running External Commands
 It is possible to specify an arbitrary external command to run either
 instead of a step or after one has run, with "runtime.ReplaceS-
 tep.any.#" or "runtime.RunAfterStep.any.#", respectively, where "#" is
 the step number given below for the -end option. So far this is possi-
 ble only for CTF determination (step 9), aligned stack filtering (step
 13), and post-processing (step 20), but more steps can be added easily
 if needed. The directive value must consist of a single line to run
 one command-line operation without piping or redirection. The program
 will place this command after a "$" in a command file named "replaceS-
 tep.#.com" or "runAfterStep.#.com", and a log will be created when it
 is run. The command can contain %{setname}, which will be substituted
 with the root name of the data set plus the axis letter, if any. A
 replacement command will be run even if there are no directives to
 activate the normal running of that step. However, a command to be run
 after a step will be run only if the normal step is run.
OPTIONS
 Batchruntomo uses the PIP package for input (see the manual page for
 pip). Options can be specified either as command line arguments
 (with the -) or one per line in a command file (without the -).
 Options can be abbreviated to unique letters; the currently valid
 abbreviations for short names are shown in parentheses.
 FILE, DIRECTORY, AND DATA SET OPTIONS
 These options give information about files, directories, and the data
 set name
 -directive (-di) OR -DirectiveFile File name
 Input file with directives for automating one or more data sets.
 The filename must end in .adoc. All non-option arguments will
 be taken as such input files. (Successive entries accumulate)
 -root (-ro) OR -RootName File name
 A data set name can be provided with this option to override the
 name contained in the directive file, thus allowing directive
 files to be reused without editing. If this option is used, the
 number of directive files must be either one or the same as the
 number of root names. The current location of the tilt series
 must be indicated with the -current option if only one directive
 file is entered with multiple root names. (Successive entries
 accumulate)
 -current (-cu) OR -CurrentLocation File name
 Directory where tilt series is currently located, or where it
 was originally located if it has already been delivered. This
 entry overrides the data set directory contained in the direc-
 tive file. It can be used in two different ways. If the
 -deliver or -make option is entered, this option specifies the
 directory where a tilt series will be moved from, and it must be
 entered either once (if all data sets are in the same place) or
 once for each data set. If neither of those options are entered,
 then this option specifies the directory where the data set will
 be processed and must be entered once for each data set or not
 at all. If the program is being rerun with the -deliver option
 and the data set has already been delivered on a previous run,
 the entry here can be either the original location of the stack
 or its current location; the program will recognize that its
 current location is under the delivery directory. However, if
 the -make option was used and the data set was already deliv-
 ered, this entry must be the original location of the stack
 rather than its new location; otherwise the program will move it
 into a subdirectory of its data set directory. (Successive
 entries accumulate)
 -deliver (-de) OR -DeliverToDirectory File name
 Make a subdirectory of the specified directory for each data
 set, named with the root name of the data set, and move the tilt
 series into the subdirectory. This option can be entered either
 once, or once for each data set to deliver to more than one
 location. Several other files with the same root name as the
 tilt series will be moved from the current location of the data
 set: an associated metadata file with extension ".mdoc", a log
 of acquisition with extension ".log", and a file of tilt angles
 with extension ".rawtlt". A raw boundary model for patch track-
 ing or autoseeding will also be delivered. The -current option
 must be entered either once, or once for each data set. (Suc-
 cessive entries accumulate)
 -make (-mak) OR -MakeSubDirectory
 Make a subdirectory for each data set at its current location,
 named with the root name of the data set, and move the tilt
 series into the subdirectory. Other files will be moved there
 as well, just as for the -deliver option. With this option, you
 can sort data sets into multiple top-level directories, and the
 program will take care of putting each one into its own subdi-
 rectory. This option cannot be entered with the -deliver
 option. The -current option must be entered either once, or
 once for each data set, and must specify the original location
 of the stack if the program is being rerun and the stack has
 already been delivered.
 -check (-ch) OR -CheckFile File name
 File to check for quit and finish signals. The default is
 batchruntomo.###.input in the directory where this program is
 started, where ### is the process ID. This file should not have
 the same name as the file used by Batchruntomo to control Pro-
 cesschunks when it runs jobs, which is "processchunks.cmds" in
 the data set directory. If a Q is placed in this file, it will
 terminate any running jobs and stop all processing. If an F is
 placed in the file, the program will stop after finishing the
 current data set. Termination will usually be quicker if a Q is
 placed into "processchunks.cmds" as well, with no delay between
 that action and writing Q to this check file. Thus, it should
 be done by a program or by entering a single command line with
 two commands on it.
 -frompath (-fr) OR -TranslatePathsFrom Text string
 Prefix from which to translate directive, location, and delivery
 paths. When this program is being run in parallel, the path for
 accessing the directories on the current machine may be at a
 different mount point than on the machine where the runs were
 set up. When there is a RemoteDirectory entry, Splitbatch
 will add this option and the next one to indicate how to trans-
 late the paths for directive files, current locations, and
 delivery locations to work on any machine. However, it will
 retain any existing entries and avoid duplicating any. These
 entries should be equivalent to a mountrule defined in cpu.adoc
 (see cpuadoc). Batchruntomo will use these translated paths
 in all runs (hence they must work on all machines, even if they
 are accessed through links on some machines). However, it will
 translate paths in messages to the log back to having the origi-
 nal prefixes so that Etomo can track file changes. If multiple
 translations are entered (equivalent to having multiple global
 mountrules in cpu.adoc), they must all work on all machines,
 such as by all translating to the same mount point. (Successive
 entries accumulate)
 -topath (-t) OR -TranslatePathsTo Text string
 Prefix to which directive, location, and delivery paths will be
 translated. The number of entries must be the same for this
 option as for -frompath (Successive entries accumulate)
 -remote (-re) OR -RemoteDirectory File name
 Path on remote machines to directory from which this program is
 started.
 BASIC PROCESSING RESOURCE OPTIONS
 These options give information about the processing resources and
 their limits in the simplest case of not running data sets in parallel
 -cpus (-cp) OR -CPUMachineList Text string
 Machines to use in parallel processing, or number of local
 cores. Each machine name can be followed by a ":" (colon), fol-
 lowed by the number of CPUs to use in that machine; e.g.,
 druid:4,tubule:4. Prior to IMOD 4.8.25, a "#" sign was used
 instead of ":". Using the "#" sign will now result in a warning
 when entering options on the command line and an error when
 entering options through standard input, because in the latter
 case it cannot be distinguished from the start of a comment. If
 the environment variable MULTI_PROC_CPU_LIST is set, it over-
 rides this entry to the program.
 -single (-si) OR -SingleOnFirstCPU
 Run single command files on the first machine in the CPU list,
 instead of on the local machine. The matchvol and autopatchfit
 steps in combine are not affected by this entry; they are always
 run on the machine with the most CPUs.
 -gpus (-gp) OR -GPUMachineList Text string
 Machines to use for parallel processing with a GPU, or 1 for
 local GPU. As when running processchunks with the -G option,
 multiple or specific GPUs can be specified by colon-separated
 numbers after a machine name, e.g., druid:2,tubule:1:2. If the
 environment variable MULTI_PROC_GPU_POOL is set, it overrides
 this entry to the program.
 -nice (-n) OR -NiceValue Integer
 Priority level or niceness for running jobs. Note that this
 applies to all computations, not just those run with parallel
 processing. The default is 15.
 -limit (-l) OR -LimitLocalThreads Integer
 Limit on the number of threads when running a single command
 file on the local machine. By default, this limit is set to the
 number of CPUs on the local machine indicated by the entry to
 -cpus. If that entry consists only of remote machines, then the
 program has no information about this limit, and single command
 files containing programs parallelized with OpenMP will use all
 available cores. This option allows one to enter a limit in
 this case, as well as to set a different limit than the number
 of CPUs in the CPU list.
 OPTIONS FOR PROCESSING DATA SETS IN PARALLEL
 These options control the processing of multiple data sets in paral-
 lel, including on a cluster
 -parallel (-pa) OR -ParallelBatchRootName Text string
 Root name that will be sent to Gpuallocator when this program
 is being run in parallel with other instances and needs to use
 one or more GPUs. In this case, the entry for -gpus needs to
 have the full list of GPUs available for all of the instances.
 This entry needs to be included to indicate batch is being run
 in parallel even when there are no GPUs.
 -maxGPUs (-max) OR -MaxGPUsInParallelBatch Integer
 Maximum number of GPUs to use when running this program in par-
 allel. The default is 4. See Splitbatch and Gpualloca-
 tor(1).
 -queue (-q) OR -QueueCommand Text string
 Command for running processes on a cluster queue when running in
 parallel, typically a Queuechunk command. This entry would
 be the same as the "command" entry for a queue in a cpu.adoc
 (see cpuadoc and "IMOD/autodoc/cpu.adoc" for examples). If
 the environment variable MULTI_PROC_QUEUE_COMMAND is set, it
 overrides this entry to the program. With this entry, the pro-
 gram runs all operations that can be parallelized in chunks on
 the queue. The treatment of single command files for other
 operations depends on whether MULTI_PROC_QUEUE_COMMAND was set:
 if it is, then Batchruntomo assumes it is itself running on a
 queue and runs each command file normally; otherwise it runs
 each command file on the queue. With this entry, -cpus and
 -gpus entries are not allowed.
 -jobs (-jo) OR -MaxJobsOnQueue Integer
 Maximum number of jobs to submit to the cluster queue, which
 influences how many chunks parallel operations will be divided
 into. If the environment variable MULTI_PROC_MAX_QUEUE_JOBS is
 set, it overrides this entry to the program.
 -jcores (-jc) OR -CoresPerClusterJob Integer
 Number of cores available to each job when running on a cluster;
 these cores will be used directly instead of submitting command
 files to the cluster queue. Thus, this entry and -queue cannot
 be entered together, nor can -cpus and -gpus be entered. This
 entry also sets the thread limit for processes whose command
 files are not split into chunks. If the environment variable
 MULTI_PROC_JOB_CORES is set, it overrides this entry to the pro-
 gram.
 -jgpus (-jg) OR -GPUsPerClusterJob Integer
 Number of GPUs available to each job when running on a cluster;
 these GPUs will be used directly instead of submitting command
 files to the cluster queue. This option cannot be entered
 unless -jcores is. If the environment variable
 MULTI_PROC_JOB_GPUS is set, it overrides this entry to the pro-
 gram.
 -gqueue (-gq) OR -GPUQueueCommand Text string
 Command for running GPU-using processes on a secondary cluster
 queue, typically another Queuechunk command. For such a
 process, either a single command file or separate chunks will be
 submitted to this queue, depending on how many jobs are allowed
 for this queue (the -gjobs entry). Ideally, this queue should
 provide access to just one GPU, because there is no mechanism
 for the submitted jobs to make use of multiple GPUs. If the
 environment variable MULTI_PROC_GPU_QUEUE is set, it overrides
 this entry to the program. The option can entered only if
 -queue or -jcores is entered.
 -gjobs (-gj) OR -MaxGPUJobsOnQueue Integer
 Maximum number of jobs to submit to the secondary cluster queue
 specified with -gqueue; this corresponds to the "number" entry
 in a cpu.adoc section for that queue. The maximum jobs will
 also be limited by the entry for -maxGPUs. If the environment
 variable MULTI_PROC_MAX_GPU_JOBS is set, it overrides this entry
 to the program.
 PROCESSING FLOW CONTROL
 These options allow doing a subset of processing operations
 -one (-o) OR -ProcessOneAxis Integer
 Enter 1 to do setup and process A axis, or 2 to skip setup and
 process only B axis. This entry applies to all dual-axis data
 sets being run.
 -end (-en) OR -EndingStep Floating point
 Step to end processing with in each set and axis. Steps are
 numbered as follows:
 0: Setup
 1: Preprocessing
 2: Cross-correlation alignment
 3: Prealigned stack
 4: Patch tracking, autoseeding, or RAPTOR
 5: Bead tracking
 6: Alignment
 7: Positioning
 8: Aligned stack generation
 9: CTF plotting
 10: 3D gold detection
 11: CTF correction
 12: Gold erasing after transforming fiducial model or
 projecting 3D model
 13: 2D filtering
 14: Reconstruction
 14.5: Postprocessing on a/b axis reconstruction
 15: Combine setup
 16: Solvematch
 17: Initial matchvol
 18: Autopatchfit
 19: Volcombine
 20: Postprocessing with Trimvol
 21: NAD (Nonlinear anistropic diffusion)
 21.5: Reducefiltvol
 22: Cleanup
 These numbers can be relied on not to change again since this
 is now a floating point entry; fractional values will be used if
 necessary to insert more steps without renumbering.
 -start (-sta) OR -StartingStep Floating point
 Step to start processing with. Steps are numbered as described
 for -end. Each set will start at this step unless the -first
 option is entered, in which case only the first set will start
 with this step and others will start from the beginning. In the
 latter case, add 100 to start with this step in the second axis
 of a dual-axis data set.
 -first (-fi) OR -StartForFirstSetOnly
 Skip to starting step only for the first set; the following sets
 will be run from the beginning.
 -use OR -UseExistingAlignment
 This option can be used to rerun Tiltalign instead of using the
 existing fine alignment when the -start option is used with a
 value over 6 and the program will be running any steps that
 involve the aligned stack or reconstruction. With an entry of 0
 in this situation, the program reruns Tiltalign iteratively
 in the usual fashion before proceeding (this can only be done
 with input on sequential lines, not from command line). The
 default is not to do this, to keep the program from overwriting
 an alignment based on manually adjusted parameters. Rerunning
 alignment in this situation was a necessity in early versions of
 the program but should no longer be needed; this option is being
 kept just for backwards compatibility.
 -exit (-exi) OR -ExitOnError
 Exit on error instead of going on to next axis or data set
 EMAIL, VALIDATION, RENAMING, AND TEST OPTIONS
 Aside from the email options, these are miscellaneous options for
 testing or for use with Etomo.
 -email (-em) OR -EmailAddress Text string
 Address for sending email messages either when a set is aborted
 or when the run completes.
 -SMTP (-SM) OR -SMTPserver Text string
 An SMTP server for sending email messages. The default is
 "localhost", but if the local host cannot send email, a server
 can be entered with this option.
 -validation (-v) OR -ValidationType Integer
 By default, the program will check all directives in the direc-
 tive and template files against the master list of directives in
 IMOD_DIR/com/directives.csv. Enter 1 to run the program just to
 check a directive file and exit, 2 to check a template file and
 exit, or -1 to skip this step. When checking a directive file,
 the directive file is first processed to find certain required
 values, and an error in that step will prevent the checking of
 directives. Template files referenced by the directive file
 will be checked as well. With an entry of 2, the file will be
 checked right after reading. A batch directive file is required
 to contain only directives with a Y in the "Batch" column of the
 master directives file, and a template file must contain only
 directives with a Y in the "Tmplt" column.
 -style (-sty) OR -NamingStyle Integer
 0 for output image files to have descriptive extensions like
 ".preali", 1 for extension ".mrc", or 2 for extension ".hdf".
 In the latter two cases the usual decriptive text is put before
 the extension and generated image files will have the corre-
 sponding type.
 -pcm (-pc) OR -MakeComExtensionPcm Integer
 0 or 1 to name output command files with extension ".com" or
 ".pcm". The latter is not yet supported by Etomo.
 -ext OR -StackExtension Text string
 Extension of an existing raw stack, used when just renaming
 stacks(s), omitting the period. With this entry, the program
 will perform the same renaming operations on one or two raw
 stacks in the current directory as when doing a reconstruction.
 The -root option must be entered once with the root name of the
 file, and the -style option must also be entered.
 -axis (-a) OR -AxisOfExtension Integer
 When just renaming stacks, enter 0 for a single-axis dataset, or
 1 or 2 for whether the extension provided with -ext applies to
 the a or b stack of a dual-axis set. With a non-zero naming
 style, the stack that this extension applies to will keep its
 name, and the other axis of a dual axis set will have its exten-
 sion changed to match if necessary.
 -etomo (-et) OR -EtomoDebug Integer
 Run Etomo with the given debug level
 -bypass (-b) OR -BypassEtomo
 Run Copytomocoms directly instead of Etomo for the setup step.
 The program will add options to extract tilt angles if direc-
 tives for using a raw tilt file are not present, and to specify
 axis rotation angles if those directives are not present.
 Setupset directives in the batch file only are passed as options
 to Copytomocoms, but it may still need other directives there.
 Excluded views are not removed since Etomo does that. An "edf"
 file is made with just the items that would be added by
 Batchruntomo.
 -PID Print process ID
 -help (-h) OR -usage
 Print help output
 -StandardInput
 Read parameter entries from standard input
FILES
 Batch and template files must have the extension ".adoc". The tilt
 series file must have an extension ".st" or ".mrc". When Etomo is run
 to set up the data set, it copies the batch directive file to the
 dataset directory with the name "batchDirective.adoc". If there are
 any template files, they are copied to the dataset with the the names
 "scopeTemplate.adoc", "systemTemplate.adoc", and "userTemplate.adoc".
 However, if any feature is used that requires modification of the batch
 directive file, namely use of the -root or -current option or template
 files listed without paths, Batchruntomo writes the modified file to
 the dataset directory as "batchDirective.adoc" and runs Etomo with this
 file.
AUTHOR
 David Mastronarde
BUGS
 Email bug reports to mast at colorado dot edu
SEE ALSO
 etomo
HISTORY
IMOD 5.2.6 batchruntomo(1)