findwarp

findwarp(1) General Commands Manual findwarp(1)
NAME
 findwarp - to find a series of 3-D transformations to warp one volume
 into alignment with another
SYNOPSIS
 findwarp [options] input_file [output_file]
DESCRIPTION
 Findwarp will solve for a series of general 3-dimensional linear trans-
 formations that can then be used by Warpvol to align two volumes to
 each other. It performs a series of multiple linear regression on
 local sets of the displacements between the volumes determined at a
 matrix of positions (patches). The displacements must be contained in
 a file with at least the following form:
 Number of displacements [optional ID values]
 One line for each displacement consisting of the X, Y, and Z
 coordinates in the first volume, then the displacements in X, Y
 and Z involved in moving from the first to the second volume
 The displacement file can have additional columns of values after the
 displacements in X, Y, and Z. These extra columns are numbered from 1,
 and they may also have ID values on the first line of the file indicat-
 ing their type.
 The program has two basic modes of operation. In one mode, it will
 compute a solution for a specified number of patches to be included in
 each local fit. In the other mode, it automatically searches for the
 largest number of patches that gives local fits with mean residuals
 less than a specified criterion. These modes behave differently
 depending on whether the program is run with parameter input via the
 PIP interface or via interactive input. With parameter input, the pro-
 gram will either run one set of local fits with a given number of
 patches, or find the best warping automatically. If the program is run
 interactively, it loops on the specification of the subsets of dis-
 placements to use until the user decides to write out a particular sub-
 set. However, at any point it can be told to find the best warping
 automatically with the current set of parameters, in which case it does
 so then exits.
 The program will automatically eliminate "outliers", patch displace-
 ments that are likely to be incorrect because they are so extreme, when
 compared to the rest of the displacements. This elimination is conser-
 vative, but if for some reason it operates incorrectly, you can control
 the parameters of elimination or stop the elimination from occurring.
 By default, the program will eliminate up to 10% of the patches from
 each local fit.
 In addition to the outlier removal, the program provides several meth-
 ods for fitting to only a subset of the data; two involving manual
 steps and one automatic. One manual method is to eliminate whole rows
 or columns of patches. The other is to use a model file to specify
 which patches to include in the fit. This model can be quite simple,
 consisting of just a single contour enclosing the region where patches
 are generally good. This contour can be drawn in any Z plane of the
 volume. However, if the good region changes through the depth of the
 tomogram, you can draw contours at several Z levels. If you have two
 layers of patches, draw two contours, one near the top and one near the
 bottom of the tomogram; if you have three layers, add another contour
 in the middle, etc. For a given patch, the program will find the con-
 tour at the nearest Z level and use that one to determine whether to
 include the patch.
 Automatic removal of some data is also possible if Corrsearch3d com-
 puted some measures of structure for each patch and stored these as
 extra columns in the patch vector file. Selection of patches based on
 these values is done by entering the -extra option to indicate the kind
 of extra column, and the -select option with one or more threshold val-
 ues for the values in that column. For example, to select based on the
 fraction of analyzed boxes in each patch that have a high amount of
 structure, which has ID number 5, one might enter "-extra 5,1" and
 "-select 0.5,0.6,0.7". If multiple criteria are entered, as in this
 example, the entire autofitting process will be repeated with each
 selection criterion to try to satisfy the first residual criterion,
 then higher residual criteria will be considered. Even if a criterion
 for mean residual is satisfied with a given selection threshold, it may
 go on to a higher threshold to try to reduce the maximum residual (see
 the -desired option).
 The program will also work with a patch file from which bad patches
 have been removed by hand. This may become necessary if bad patches
 are too frequent in some location to be eliminated as outliers. To use
 this feature, use Patch2imod to convert the patch file to a model
 file where displacements are represented by vectors, examine the file
 in 3dmod, eliminate aberrant contours, and convert the model file to a
 new patch file with Imod2patch.
 If there is only one layer of patches in the thin dimension, there is
 insufficient information to solve for the full transformation, so the
 program will solve for only two of the three columns of each local
 transformation matrix, and keep the third column of each matrix fixed.
 The same procedure is used if a particular local area does not have
 sufficient data on more than one layer in the thin dimension.
 For a given arrangement of patches, the program finds a mean and maxi-
 mum residual for each of the fits. It first reports how many points
 have been eliminated as outliers, in how many fits they appeared to be
 outliers, and a summary of the distribution of their residuals. On one
 line, it next reports the average and the maximum of the mean residu-
 als. On the next line, it reports the average and maximum of the maxi-
 mum residuals. The goal is to find an arrangement that contains as
 many patches as possible in each direction yet has residuals comparable
 to those found with a volume that does not need warping (typically 0.2
 to 0.3).
 The program decides which arrangements of patches are acceptable for
 fitting by determining whether the ratio of measurements to unknown
 values falls within a given range. Originally, this ratio was evalu-
 ated based on the nominal number of patches in each dimension. In IMOD
 4.8.14, the maximum number of patches actually available in each dimen-
 sion, and the mean number of patches, is evaluated for each potential
 local fitting size. Ratios are based on these two values and used to
 determine if a fitting size is acceptable. This is a more reliable
 method for curved sections, especially when there are relatively many
 patch positions in the thickness dimension. The default is to use the
 new method for new patch data that have extra column ID values, other-
 wise to use the old method. There are two ways to override this behav-
 ior: either enter -1 or 1 for the -legacy option to force the use of
 the new or old method, respectively; or set the environment variable
 FINDWARP_LEGACY_RATIOS to -1 or 1.
 If you want to regenerate the combined volume in an older data set,
 where there are no IDs for the extra columns, you can use existing
 patches and get the same result from Findwarp as before. To get a new
 fitting result (which might be preferable), either make a new set of
 patch vectors or set the environment variable to -1.
OPTIONS
 Findwarp uses the PIP package for input (see the manual page for
 pip) but can still take input interactively for exploring the effect
 of varying parameters. The following options can be specified either
 as command line arguments (with the -) or one per line in a command
 file or parameter file (without the -). Options can be abbreviated to
 unique letters; the currently valid abbreviations for short names are
 shown in parentheses.
 -patch (-pat) OR -PatchFile File name
 Name of input file with positions and displacements. If this
 option is not entered, the first non-option argument will be
 used for the input file name.
 -output (-o) OR -OutputFile File name
 Optional output file for warping transformations. If this
 option is not entered, the second non-option argument (if any)
 will be used for the output file name.
 -region (-reg) OR -RegionModel File name
 Model file with contours enclosing the patches to be included in
 the fits.
 -volume (-v) OR -VolumeOrSizeXYZ File name
 Either the name of the file or the X, Y, and Z dimensions of the
 volume being matched to.
 -initial (-i) OR -InitialTransformFile File name
 File with initial 3D transform used to make the second volume
 for the patch correlations. This transform will be incorporated
 into the warping transformations written to the output file.
 The format of such a file is described in the Matchvol man
 page.
 -residual (-res) OR -ResidualPatchOutput File name
 Output file for positions, displacements, and mean residual val-
 ues, averaged over all the fits that include a position. This
 value will be zero for patches excluded from all fits. If the
 patch input file has correlation coefficients, they will be
 replaced by the residuals. After the residual value, this file
 will also have a value for the fraction of fits in which the
 vector was removed as an outlier.
 -target (-t) OR -TargetMeanResidual Multiple floats
 One or more mean residual values to try to reach in the auto-
 matic search for the best warping. Multiple values should be
 entered in increasing order. The program will try to find a
 warping with the largest number of included patches that gives a
 mean residual below the first value; then it tries again with
 the second value, etc.
 -measured (-me) OR -MeasuredRatioMinAndMax Two floats
 The minimum and maximum ratio of measurements to unknowns to be
 allowed in the automatic fits. The defaults are 4 and 20.
 -legacy (-l) OR -LegacyRatioEvaluation Integer
 This entry controls the evaluation of the ratio of measurements
 to unknowns, as described above. The default is 0 to use the
 new method only when there are ID values for the extra columns.
 Enter -1 or 1 to force the use of the new or old method, respec-
 tively. This entry overrides a setting of the FIND-
 WARP_LEGACY_RATIOS environment variable.
 -xskip (-x) OR -XSkipLeftAndRight Two integers
 Number of columns of patches to exclude on the left and right
 -yskip (-y) OR -YSkipLowerAndUpper Two integers
 Number of patches to exclude on the lower and upper sides in Y.
 This entry specifies either rows or slabs of patches to exclude,
 depending on whether the volume is oriented so that it is thin
 in the Z or Y dimension.
 -zskip (-z) OR -ZSkipLowerAndUpper Two integers
 Number of patches to exclude on the lower and upper sides in Z.
 This entry specifies either slabs or rows of patches to exclude,
 depending on whether the volume is oriented so that it is thin
 in the Z or Y dimension.
 -extra (-extr) OR -ExtraValueSelection Two integers
 Select only patches with extra value on one side of a criterion.
 To include patches based on values in an extra column, enter two
 numbers for this option: the ID value or the negative of the
 extra column number (numbered from 1); and 1 to select patches
 above the criterion or -1 to select ones below it. You must
 also enter one or more threshold values with the -select option.
 (Successive entries accumulate)
 -select (-se) OR -SelectionCriteria Multiple floats
 One or more criteria for selecting patches based on extra val-
 ues. Multiple criteria should be entered in order of greater
 selectivity. If there are multiple values being selected on,
 this option must be entered the same number of times as the
 -extra option and it must have the same number of criteria
 entered each time. (Successive entries accumulate)
 -desired (-des) OR -DesiredMaxResidual Floating point
 When patches are selected based on extra values and the mean
 residual is acceptable but the maximum of the maximum residuals
 in the fits is above this value, the program will go on to apply
 the next criterion, if there is one. It will still accept the
 result if the last maximum residual is above this target on the
 last criterion. The default is 8 times the last residual target
 entered with -target. Enter 0 to disable this feature.
 -rowcol (-ro) OR -LocalRowsAndColumns Two integers
 Number of rows and columns of patches to include in each local
 fit. The second value will apply to the longer of the Y and the
 Z dimension. If this option is not entered, the program will
 automatically search for the best warping.
 -slabs (-sl) OR -LocalSlabs Integer
 By default, fits include all slabs of patches in the shorter of
 the Y and Z dimensions. This entry allow fits to be done to
 subsets of patches in that dimension. If the -rowcol option is
 used, then this entry specifies the number of slabs of patches
 to include in each fit. Otherwise, the entry sets the minimum
 number of slabs of patches that will be tried in the automatic
 search for the best warping.
 -extent (-exte) OR -MinExtentToFit Integer
 Minimum number of rows and columns of patches to include in each
 local fit. When there are many (~10 or more) overlapping
 patches in the thickness dimension, an entry of 3 will keep the
 program from doing 2 by 2 or 2 by 3 fits in the other dimen-
 sions, which would only have a single patch of overlap between
 laterally adjacent fits. Such an entry would also be appropri-
 ate when the spacing between patches is unusually small, to keep
 the transformation from being solved over too small an extent.
 -maxfrac (-ma) OR -MaxFractionToDrop Floating point
 Maximum fraction of patches to drop from each fit by outlier
 elimination. Enter 0 to for no outlier elimination. The
 default is 0.1.
 -minresid (-mi) OR -MinResidualToDrop Floating point
 The minimum residual for outlier elimination; patches with
 residuals smaller than this value will be retained no matter how
 extreme they are relative to the other patches. The default is
 0.5.
 -prob (-pr) OR -CriterionProbabilities Two floats
 Two probabilities controlling outlier elimination: a criterion
 probability for a patch to be evaluated as an outlier (default
 0.01), and a criterion probability for a patch to be eliminated
 regardless of the distribution of extreme values (default
 0.002).
 -discount (-di) OR -DiscountIfZeroVectors Floating point
 Use this option to ignore local fits that have many zero vectors
 when computing the average mean residual and the average maximum
 residual. This will prevent misleading averages when analyzing
 the warping fields used to align adjacent volumes for stitching.
 Fits with the number of zero vectors bigger than the given frac-
 tion of the total vectors will be excluded from the averages.
 -debug (-deb) OR -DebugAtXYZ Three floats
 Center location at which to print debug output about fits
 -param (-par) OR -ParameterFile Parameter file
 Read parameter entries as keyword-value pairs from a parameter
 file.
 -help (-h) OR -usage
 Print help output
 -StandardInput
 Read parameter entries from standard input.
INTERACTIVE INPUTS
 If the program is started with no command line arguments, it takes
 interactive input with the following entries:
 Name of file with positions and displacements
 (At this point the program will report the number of patches in X, Y,
 and Z in this file, or complain if the data do not have the
 proper form.)
 Either the file name, or the X, Y, and Z dimensions of the volume being
 matched to.
 Name of an Imod model file with contours enclosing the patches to
 be included in the fit, or Return to use all patches.
 0 to proceed interactively, or 1 to find best warping automatically
 IF you enter 1, next enter:
 One or more target mean residuals to achieve.
 The minimum and maximum ratio of measurements to unknowns to be
 allowed in the local fits, or / to use the default values.
 0 to use all of the data, or 1 to specify a subset of patches to use
 IF you enter 1, next enter three lines:
 Number of columns of patches to eliminate from the left and right
 sides of the data, 0,0 to use all patches in X, or / to use
 previous values.
 Number of rows or slabs of patches to eliminate from the lower and
 upper extent in Y, 0,0 to use all patches in Y, or / to use
 previous values.
 Number of slabs or rows of patches to eliminate from the lower and
 upper extent in Z, 0,0 to use all patches in Z, or / to use
 previous values.
 IF you selected automatic warping, the program now proceeds by fitting
 to the largest possible area that does not exceed the maximum ratio
 of measurements to unknowns, and it tries progressively smaller
 areas until the desired mean residual is achieved. It does this
 using the parameters (e.g., row or column elimination) that were set
 on any previous interactive rounds of fitting. If the target
 residual is reached, it requests the names of the initial
 transformation file and the output file, as described below, and
 exits. If the residual is not reached, it tries the next target
 residual if any, and if no targets are met, it exits with an error.
 IF you are proceeding interactively, continue with the following:
 IF there are more than 2 patches in the thin dimension, the program
 next asks whether you always want to do the local fits to all
 patches in that dimension. Just enter 0 for the typical situation.
 0 to use default parameters for outlier elimination, or 1 to adjust
 any of these parameters. Just enter 0 unless you know better.
 IF you enter 1, then make four entries, or / to take the default:
 Maximum fraction of patches to eliminate from each fit. Set this
 to 0 to stop the outlier elimination from occurring.
 The minimum residual for elimination; patches with residuals
 smaller than this value will be retained no matter how extreme
 they are relative to the other patches.
 Criterion probability for patches to be considered candidates for
 for elimination. A smaller value will eliminate fewer patches.
 Criterion probability for patches to be eliminated regardless of
 the pattern of outliers. A higher value may force the
 elimination of more patches.
 Number of local patches in X and Y, X and Z, or X, Y, and Z to include
 in each fit. These values must be at least 2 and no larger than the
 total number of patches in the respective direction.
 The program will loop on the last entry until the same numbers are
 entered twice in a row (e.g. with a /). If you enter 0 for the number
 of patches in X, it will loop back to the query about whether you want
 to find warping automatically. When you do enter a /, it will request:
 Name of file with initial 3D transformation that was applied to one of
 the volumes before the patch correlation, or Return if there was no
 such file. The format of such a file is described in the
 Matchvol man page.
 Name of output file in which to place the transformations. These will
 be inverse transformations, ready for use by Warpvol.
 An exception to the above occurs if you specify that all available
 patches are to be used in a single fit. You would do this if you just
 needed to eliminate a row or column of patches from the fit. In this
 case, when you enter a /, the program will simply ask for the name of a
 file in which to place the single refining transformation, just as with
 Refinematch.
HISTORY
 Written by David Mastronarde 1/30/97
 12/24/98: added outlier elimination, integrated complex options.
 6/6/99: added ability to output single refining transformation.
 1/1/00: added model exclusion and automatic finding of best warp
 6/7/01: rewrote data input to handle data with missing patches
 7/20/02: rearranged input to make it easier to run automatically with
 rows or columns removed
 8/21/06: converted to PIP, made it handle either orientation of volume,
 made automatic fitting more flexible in the thin dimension,
 changed outlier output to a summary, added option for patch and
 residual output
BUGS
 Email bug reports to mast at colorado dot edu.
IMOD 5.2.0 findwarp(1)