subtomosetup
subtomosetup(1) General Commands Manual subtomosetup(1)
NAME
subtomosetup - Sets up command files for reconstructing multiple sub-
volumes
SYNOPSIS
subtomosetup options
DESCRIPTION
Subtomosetup creates a set of command files for directly reconstructing
multiple numbered subvolumes from an aligned tilt series. Its main use
is to allow subtomogram alignment and averaging to be done on unbinned
data after selecting particle positions on a binned-down tomogram,
without having to build a full unbinned tomogram. It also allows one
to apply CTF correction to the aligned stack for a series of Z depths
in the tomogram, and use the appropriate corrected stack in recon-
structing each subvolume. Note that if the unbinned tomogram already
exists, the program Boxstartend can be used to extract subvolumes
from it.
Rules of Operation
The program allows considerable flexibility as long as several restric-
tions are followed.
1) The program must be run from the dataset reconstruction directory,
and the command files provided to it should ideally be ones managed and
output by Etomo. Do not try to use a tilt.com constructed with what
seem to be the minimum required entries.
2) The raw tilt series stack must still be present. Alternatively, a
file with the same name as the stack containing just the header can be
used, but in this case the aligned stack must be present at the desired
size and binning.
3) The tomogram on which points were selected for reconstruction must
be available and entered with the -volume option, although it need not
be in the current directory. Alternatively, a file with just the
header of this volume can be supplied.
4) The tilt series alignment, tomogram positioning parameters, the X-
axis tilt, and the tilt angle offset in the Tomogram Generation panel
of Etomo must not be changed between the binned tomogram and subvolume
reconstructions. Whether to use a GPU may be changed, and the current
choice will determine whether a GPU is used for the reconstructions.
5) If you apply any processing to that volume outside of Etomo, the
header entries for pixel spacing, origin, and tilt angles must be pre-
served in the file entered with the -volume option. If you need to
trim some more, use Trimvol; if you need to bin, use Binvol.
6) If you need to apply other software that does not preserve these
header entries, follow these procedures: Make sure that the file ana-
lyzed has the same dimensions as the IMOD-based file from which it was
derived. Enter the name of the IMOD-based file instead of the analyzed
file with the -volume option. Supply a point coordinate file instead
of a model file for the -center option (use Model2Point(1) with the
-float option to make such a file). Other alternatives are to load a
model onto the IMOD-based file with the -m option and save it again, or
to run Imodtrans on a model with the -image option and the name of
the IMOD-based files, and use the resulting model.
To make a file with a copy of the header from the raw stack or modeled
volume, you can use the script "copyheader" with two arguments: the
name of the input image file and the file in which to write the header,
e.g.:
copyheader setname.st setname_header.st
You are free to make an aligned stack from a centered subarea or an
over-sized area and to change this area between the binned and subvol-
ume reconstructions. You can apply any trimming options in the Post-
processing panel in Etomo. Note that the subvolumes will be reoriented
the same way that the binned tomogram was.
3-D CTF Correction
3-D CTF correction is activated with the -zlevels option, which speci-
fies the number of levels in Z at which to compute CTF corrections, or
with the -extent option, which sets the range in Z over which each CTF
correction is used The total Z range of the subvolume centers will be
divided into levels based on one or the other entry, and each subvolume
will be reconstructed from the aligned stack for its Z-level. The com-
mand files are set up to process the Z-levels in order, with the CTF-
corrected stack from the previous Z-level deleted before going on to a
new Z-level. The command file for correction ("ctfcorrection.com" by
default) must already include the necessary options before running this
program, including the correct pixel size for the aligned stack. The
aligned stack must not already be CTF-corrected, but it may have gold
fiducials erased already. Alternatively, fiducials can be erased in
each corrected stack with the -erase option. If so, the command file
for erasing ("golderaser.com" by default) must also have the correct
options before running this program.
Reconstructions from the Raw Stack
Reconstructions can also be done directly from the raw stack when doing
3-D CTF correction. The aligned stack need not exist in this case, but
desired parameters must be set in the command file for CTF correction,
and the defocus file must exist. If 2-D filtering is to be done, it
will be done on the raw stack by the command files produced here, and
parameters must be set already in the command file for filtering. If
gold beads are to be erased, that will happen after creating the CTF-
corrected raw stack for each defocus level, the parameters must be set
in the gold erasing command file, and the model for erasing must exist
already.
Reconstructions from raw images cannot take account of corrections for
a distortion field or magnification gradients and will not be valid if
either of these corrections were used when making the aligned stack.
The program will examine the Newstack log and/or command files from
making both the coarse and final aligned stacks and issue a warning if
either of these contains distortion corrections.
Using One or More GPUs
A GPU can be used for both correction and reconstruction, for neither,
or for correction but not reconstruction. The -gpu option can be used
to control when the GPU is used. If that is not entered, the command
files determine what will happen. If the reconstruction command file
specifies use of the GPU, then it will be used for CTF correction as
well, regardless of the setting in the command file for correction.
However, if the reconstruction command file does not specify using the
GPU, the setting in the correction command file determines whether cor-
rection is done on a GPU. When reconstruction is run on CPUs and cor-
rection on a GPU, correction will be run on only one GPU, and the first
machine in the machine list given to Processchunks must have a GPU.
It may be more efficient to do correction on one GPU even when many
CPUs are available, because correction with CPUs will be split into
many jobs to be run in parallel, which incurs some overhead and delays.
When correction is done on CPUs, the operation will be split up by
assuming there are 8 CPUs, unless a specific number is specified with
-proc. Also, if reconstruction is set to be done on a GPU, correction
(on the GPU) will be split into jobs when -proc is entered with a value
higher than 1.
X-axis Tilt
When reconstructing from an aligned stack, if an X-axis tilt is
included in either the reconstruction or correction command files, then
the program evaluates whether it is similar enough in the two opera-
tions. If an X-axis tilt is being applied at all during correction, it
should closely match the value used for reconstruction, otherwise the
program gives a warning. If there is an X-axis tilt in reconstruction
but not correction, the program will determine the maximum difference
in defocus produced by the two different X-axis tilt values, which
occurs at the extremes of the reconstruction in Y. If this defocus is
larger than a certain fraction of the Z-level extent, the program
issues a warning.
When reconstructing from the raw stack, the X-axis tilt in the recon-
struction command file is unconditionally used for the correction.
There is no additional time involved in using the X-axis tilt in this
case because the CTF correction has to be done in diagonal strips of
full-sized FFTs anyway.
Chunks, Subvolumes, and Running the Jobs
Subtomosetup generates command files named with the root name of the
Tilt command file ("tilt" by default) plus "-sub-001.com", etc. By
default, each command file will produce ten subtomograms named "root-
name-N.mrc" where "rootname" is the dataset root name, and "N" is a
number that increases sequentially for all subvolumes computed, but
with enough leading zeros so that all files have the same number of
digits and will list in order. The numbers will be in the same order
as the points in the model or point file, but with 3-D CTF correction,
they will not be computed in order. If a position is too close to the
top or bottom of the Y-extent that can be reconstructed from the cur-
rent aligned stack, the particle will be skipped, but output files will
still be numbered sequentially unless the -skip option is entered. For
positions not quite so close to the limits in Y, a Tilt run will be
included to generate a subvolume with up to 1/6 of the extent blank in
that direction.
Multiple Tilt runs are put in each command file in order to reduce
the overhead in starting and monitoring commands, and to reduce the
number of .com and .log files in the directory. The division of runs
into jobs can be controlled by the -proc and -runs options. When doing
3-D CTF correction, runs at each Z level are apportioned into chunks
using the same strategy, but with the same target for the total number
of reconstruction chunks.
Subvolumes can be stacked into a much smaller number of volume stacks
to reduce the number of files that have to be handled. See the -vol-
stack option.
The command files can be run with Subm, Processchunks, or the
generic parallel processing interface in Etomo. For example
subm tilt-sub*.com
to run all files in sequence, or
processchunks 8 tilt-sub
to run in parallel on 8 processors, as can also be done in the Etomo
interface. A GPU will be used if specified in the original command
file. The Etomo interface can be used to run on multiple GPUs located
on different machines, but to run on multiple GPUs in one machine, pro-
cesschunks must be run directly, such as with
processchunks -G localhost:1:2:3
OPTIONS
Subtomosetup uses the PIP package for input (see the manual page for
pip). Options can be specified either as command line arguments
(with the -) or one per line in a command file (without the -).
Options can be abbreviated to unique letters; the currently valid
abbreviations for short names are shown in parentheses.
-root (-ro) OR -RootName Text string
Root name of dataset. For one axis of a dual-axis data set,
include the "a" or "b" in this name. This option is required.
-center (-ce) OR -CenterPositionFile File name
Name of an IMOD model file or a point coordinate file with the
center positions of the desired subvolumes. A point coordinate
file should have one point per line, with its X, Y, and Z coor-
dinates separated by spaces, not commas. These coordinates
should correspond to pixels in the full file specified with
-volume, where the first pixel in any dimension spans coordi-
nates from 0 to 1. This option is required.
-volume (-v) OR -VolumeModeled File name
Name of the image volume in which points were selected. If a
model file is supplied for the -center option, this entry should
be the image volume that was modeled. If a point coordinate
file is supplied with -center, the coordinates must correspond
to pixel positions in this volume file. This option is
required.
-raw (-ra) OR -RawStackFile File name
Name of raw image stack. This option is needed only if this
filename cannot be determined by analyzing command files in the
directory.
-axis (-ax) OR -AxisAngle Floating point
Angle that the tilt axis is rotated from vertical (counterclock-
wise positive) in the raw image stack, used to determine whether
the X and Y sizes of a full aligned stack are transposed from
those of the raw stack. This entry is needed only if it cannot
be found in an alignment log file.
-objects (-o) OR -ObjectsToUse List of integer ranges
If only a subset of objects in the model contain points to be
reconstructed, this option can be used to enter a list of those
objects (comma-separated ranges are allowed).
-size (-si) OR -SizeInXYZ Three integers
Final size of subvolumes in X, Y, and Z, in unbinned pixels. If
the modeled volume was reoriented by rotation or flipping, sub-
volumes will be treated similarly, and this entry specifies the
size after reorienting. If the aligned stack is binned, the
actual size will the specified size divided by its binning.
This option is required.
-dir (-d) OR -DirectoryForOutput File name
Name of directory to place subvolumes into. By default, the
subvolumes will be written into the dataset directory; this
option can be used to place them in a subdirectory or, in fact,
in a directory located anywhere. If the directory does not
exist yet, it will be created.
-chunk (-ch) OR -DirectoryForChunkFiles File name
Name of directory in which to place the command files. By
default, these chunk files will be written to the dataset direc-
tory; with this option they can be in a subdirectory or other
location. The log files will be created in the same directory.
If the directory does not exist yet, it will be created. The
command files must still be run from the dataset directory.
-skip (-sk) OR -SkipSubVolNumbers
Skip subvolume numbers whenever points near the border of the
volume are skipped, thus keeping the subvolume and point numbers
in register even when points are skipped. The default is to put
out sequentially numbered subvolumes with no gaps, which is
required when processing the subvolumes in PEET with template
specifications for a series of volumes.
-stackvols (-st) OR -MakeVolumeStacks Integer
Stack the subvolumes into one or more MRC volume stacks, each
containing up to the given number of subvolumes. The individual
subvolumes will then be deleted. For each volume stack, the
program will also produce a model file with a point at the cen-
ter of each subvolume, as required for using these stacks in
PEET. 3dmod will load a volume stack as multiple subvolumes,
each at a different time index, so the model has each point in
its own contour at the appropriate time index. The volume
stacks will be named "rootname-vol#.mrc" and the models will be
named "rootname-vol#.mod", where "#" is a value numbered from 1,
with enough leading 0's so that the numbers for all files have
the same number of decimal places.
-com (-co) OR -CommandFile File name
Starting command file for running Tilt to make the reconstruc-
tions. The default is to use "tilt.com", so this option is
needed for a dual-axis data set or to use a copy of the file.
-binali (-b) OR -NewAlignedBinning Integer
Make a new aligned stack with the given binning instead of using
the existing aligned stack. Any operations that were done on
the previous aligned stack need to be specified with options
here; they will not be redone automatically. Also note that the
program will run Newstack to create a temporary aligned stack
with one image in order to analyze its header settings. This
option cannot be entered with -unaligned.
-newstcom (-n) OR -NewstackComFile File name
Name of command file to use for making a new aligned stack,
which can be entered with or without its extension of ".com".
If this option is not entered, "newst" will be substituted for
"tilt" in the name of the command file for reconstruction.
-unaligned (-un) OR -UseUnalignedImages
Use the raw, unaligned stack for CTF correction and reconstruc-
tion. This should be the only option that is needed to use the
raw stack, other than the -reduce option if desired, when this
program is run inside a directory with a full set of processing
files from one data set. This option can be used only with 3-D
CTF correction. The option cannot be entered with -binali.
-reduce (-red) OR -FourierReduceByFactor Integer
Reduce the raw stack by the given factor by using Newstack to
crop its Fourier transform and use this reduced stack for pro-
cessing from unaligned images.
-zlevels (-z) OR -NumberOfZLevels Integer
Number of different depths in Z at which to make CTF-corrected
aligned stacks. Either this option or -thickness must be
entered, but not both. An entry of 1 is allowed and will result
in CTF correction being done at the defocus corresponding to the
midpoint of the subtomogram Z depths.
-extent (-ex) OR -ExtentOfZLevelsInNm Integer
The maximum depth in Z over which to use one CTF correction, in
nanometers. A large value can be used to do CTF correction only
once, in which case the correction will be at the defocus corre-
sponding to the midpoint of the subtomogram Z depths.
-invert (-i) OR -InvertZLevelOffsets Integer
Enter 1 to invert the sign of the offset in Z applied when doing
the CTF correction for each Z level. This would be needed if
the tomogram is upside-down from its orientation in the micro-
scope, by rotation and/or inversion in Z. Tests with the most
common sources of this problem indicate that this occurs when-
ever tilt angles need inversion in Ctfplotter and Ctfphase-
flip(1); thus the default is to use the value of the -invert
option in the command file for CTF correction, with the option
here available to override that value. See the section on .B
Inverting Tilt angles in the Ctfplotter man page for more
details.
-adjust (-ad) OR -AdjustForAlignZShift
Adjust for Z shift of tomogram away from the height that would
have been produced by the cross-correlation alignment alone.
The original cross-correlation (coarse) alignment would produce
a tomogram in which the contrast-producing material is centered
in Z, on average. If the same material is responsible for the
signals used to measure the CTF, then a CTF correction with zero
Z offset would be appropriate for the middle of a tomogram based
on that alignment. With this option, the program finds the
total shift imposed by the Z offset values in align.com and
tilt.com and adds that to the Z offset used in Ctfphaseflip
so that the subtomograms shifted by that amount have zero Z off-
set in the correction.
-ctfcom (-ct) OR -CorrectionComFile File name
Command file for CTF correction, which can be entered with or
without its extension of ".com". If this option is not entered,
"ctfcorrection" will be substituted for "tilt" in the name of
the command file for reconstruction.
-erase (-er) OR -EraseFiducials
Run a command file to erase gold, which will occur after after
CTF correction of the aligned stack, if any. Gold may be erased
in the aligned stack prior to CTF correction when using an
existing aligned stack, but this option can be used to erase
after correction, which is the preferred order of operations in
Etomo and Batchruntomo. That order is better in principle
but may make little difference in practice.
-goldcom (-go) OR -GoldEraserComFile File name
Name of command file to use for erasing gold, which can be
entered with or without its extension of ".com". If this option
is not entered, "golderaser" will be substituted for "tilt" in
the name of the command file for reconstruction.
-filter (-f) OR -FilterIn2D
Filter images in 2-D. If filtering is desired, this entry is
needed when reconstructing from a raw stack, but when using an
existing aligned stack, the filtering may be done either before-
hand or with this option.
-2dcom (-2) OR -2DFilterComFile File name
Name of command file to use for 2-D filtering. If this option
is not entered, "mtffilter" will be substituted for "tilt" in
the name of the command file for reconstruction.
-gpu (-gp) OR -WhenToUseGPU Integer
The entry overrides the default behavior described above based
on the entries in the command files. Enter 0 to use just CPUs,
1 to use GPUs for both reconstruction and CTF correction, or 2
to use a single GPU for CTF correction and CPUs for reconstruc-
tion. In the latter case, the first computer in the machine
list must have a GPU.
-pixel (-pi) OR -RawPixelSize Floating point
Pixel size of raw image stack in nanometers. This entry is
needed when using unaligned images only if the pixel size is 1.0
in the image file header, and the pixel size is not found in a
track.com file.
-xform (-x) OR -AlignTransformFile File name
File with the linear transformations that were used to align the
images. This option is needed when using unaligned images only
if this filename cannot be determined by analyzing command files
in the directory.
-reorient (-reo) OR -ReorientionType Integer
This option can be used to specify the type of reorientation
that was applied to the original reconstruction to obtain the
modeled volume, in the unlikely event that the program cannot
detect which was used. Reorientation by rotation around the X
axis can be detected by the tilt angles in the volume header;
swapping of Y and Z can be detected only if the "clip: flipyz"
title is still present. If necessary, the program will assume
that swapping occurred if the Y dimension is more than twice the
Z dimension, or that no reorientation occurred if the Z dimen-
sion is more than twice the Y dimension. If the program makes a
wrong assumption or insists that this option be used, enter a 0
for no reorientation, 1 for swapping of Y and Z, or -1 for rota-
tion around X.
-proc (-pr) OR -ProcessorNumber Integer
Number of processing units, either CPUs or GPUs, that the jobs
will be run on. The program will divide the jobs into 10 com-
mand files (chunks) per processor if this results in fewer than
1000 chunks, or with fewer chunks per processor, down to 5, in
an attempt to keep the number of chunks under 1000.
-runs (-ru) OR -RunsPerChunk Integer
Number of Tilt runs per command file (chunk). The program will
create chunks with this number of runs, or with more runs to
keep the total number of chunks under 100,000. This option can-
not be entered with -proc. The default is 10.
-help (-h) OR -usage
Print help output
-StandardInput
Read parameter entries from standard input
FILES
All temporary files are removed by each command file as it runs, and
command and log files except "-finish.log" file are removed by the
"-finish.com" file.
AUTHOR
David Mastronarde
SEE ALSO
tilt, subm, processchunks, boxstartend, trimvol, bin-
vol(1), model2point, imodtrans
BUGS
Email bug reports to mast at colorado dot edu.
IMOD 5.2.0 subtomosetup(1)