| irproc (Mar05) | irred | irproc (Mar05) |
This task make extensive use of expressions using keywords and pixel values for various purposes. This allows providing explicit values, the values of keywords, functions of the values of keywords and pixels, and combinations of keywords. It also allows keywords to be different for different applications. The badpixel expression is a more general version of the saturation detection functionality.
Some philosophical difference with ccdproc are that "in-place" calibration or automatic processing of calibrations images are not allowed. Also support for doing all processing and producing output data using only short integers is not provided. A consequence of this is that flat fields are normalized rather than carrying along and applying a scale factor when a flat field is applied.
If a keyword is specified by the maskkey parameter the output mask name will be recorded in the header of the output image under this keyword. If no output mask is created the keyword is not changed from that in the input image.
Processing depends on the type of exposure, the part of the detector recorded in the image, the photon energies being detected, and other characteristics of the instrument. To make it easy to process mixtures of images, and to check for proper types of calibrations, and to allow configuration for instruments with different characteristics and keywords the following parameters are used to interpret the data. These make use of expressions which use keyword names as operands and evaluate to boolean (yes or no), string, or numeric values. The expression syntax is described by irprocexpr.
[The use of expressions is a different and more flexible approach than use in ccdproc based on the instrument translation file.]
The following parameters select the corrections and calibration operations to be performed. The allowed operations and the order in which they are performed is controled by the order, dorder, and forder parameters. The single letter processing code is used to identify the operation in the order parameters and in record keeping. In order for an operation to be performed the processing switch must be enabled AND the operation must be listed in the order of operations. Normally the order of operations is rarely changed and the operations are controled through the processing switches. But by setting all the switches to yes the processing steps may be controled by the order parameters. Because the order parameter may be set by an input keyword it is possible to control the operations though a header keyword as might be desired in a pipeline situation.
The processing parameters provide data for the various operations selected by the processing switches.
fit - fit a function to the bias vector (non-interactive)
ifit - fit a function to the bias vector (interactive)
mean - the mean of the biassec columns at each line
median - the median of the biassec columns at each line
minmax - the mean at each line with the min and max excluded
The "fit" and "ifit" options use the icfit routines to fit a function to the average bias vector either non-interactively or interactively respectively. The fit is then evaluated and subtracted at each line. The fitting parameters are set with the biaspars parameters. The other options statistically estimate the bias at each line using only the bias pixels for that line.
IRPROC processes detector array data, recorded as images, to correct or calibrate array defects, electronic and detector bias, signal dependent linearity and crosstalk, flat field response, and sky background. The output images may also be trimmed to specified regions to exclude bad edges and non-imaging pixels. In addition to the processed detector images it may also produce pixel masks consisting of bad pixels identified by an input mask and by a bad pixel expression computed during processing.
While this task is general, it was developed with infra-red array detectors in mind. It is designed to be efficient and relatively easy to use for the typical operations performed on images from such detectors. Processing observational data also requires identifying different types of exposures and matching calibrations, so this task also handles this bookkeeping based on information from the image headers.
In addition to allow image header keywords to identify the type of exposure, including how to match calibrations by filter or grating, there is a feature that allows any irproc parameter except the input list to be specified by header keywords. This is done by prefixing the keyword with !. This feature can be particularly useful for pipelines and for integration with data acquisition systems.
While this task is most useful when header information is provided it can still be used without requiring any keywords. This is done by setting the various keyword expressions to select all data and not use keyword. The user must then group and process the data using image lists that explicitly contain only the appropriate data. The parameters must also be set with explict values.
The first step is to select the images to be processed. The initial candidate list is specified by the input parameter. The input names may be single images or multi-extension format (MEF) files. MEF data are expanded into a list of image extensions (with non-image extensions ignored) and then processed as independent images. However, on output images from the same MEF file will be written to another MEF file with the same extension names.
After a candidate list is defined the list may be winnowed using the intype selection parameter. So one may process a subset of files either by explicitly specifying the desired files in the input list with the selection parameter set to accept all images or by specifying a more generic template list, such as "*.fits", and using the selection parameter to extract a subset which have a common characteristic.
The most common type of input selection is to process dark, bias and flat field exposures before combining them into master calibrations to be applied to the program exposures. Because this is so common the dtype and ftype parameters are provided for the selection expressions so they can be set once. Then the intype parameter is set to point to these expressions using the syntax ")dtype" or ")ftype". Note that this syntax is a less known feature of the IRAF parameter system which can be used with any task and parameter.
Another aspect of selecting calibration images using the dtype and ftype expressions is that the task can then be configured to do different operations without requiring changing the processing switches. This is done using the dorder and order parameters. These parameters set the order and operations appropriate for these types of calibration images. If an operation is not referenced in these parameters it will not be performed even if the processing switches are set for the steps for program exposures.
Another winnowing of the input list is the exclusion of images that have been previously processed by Birproc. This task adds keywords to document the processing that has been applied to the image. These keywords are checked to determine what has been done previously. Therefore, if all the steps have already been done an image will be skipped based on this processing history information. Note that if just some of the operations have been performed then the image will be processed but only the operations which have not been previously done and which are requested by the task parameters will be applied.
The override parameter may be used to override this prohibition against repeating operations that have been previously performed. Note that a valid alternative is to delete the header keywords. There are some operations that do make sense to repeat, albeit with different parameters of calibrations. These include corrections to the flat field, detecting and fixing additional bad pixels, and using additional data for sky subtraction.
The processing operations to be performed on the selected input data are set by boolean (yes/no) parameters. The allowed steps and their order are specified by the order parameter. The dorder and forder parameters will be used instead if the image is identified as a dark or flat field by the dtype or ftype parameters. It is important to realize that in order for an operation to be performed it must appear in the order parameter and have the appropriate processing switch set. This allows two modes of operation. One may enable all the switches and select the steps with the order parameter or one may set the order parameter to enable all the steps and switch operations on or off with the switches. One method may be better for interactive work and the other better for pipelines.
In addition to providing alternate ways to adjust parameters, the order parameters are important for defining the order of processing. Different types of detector array data, such as CCD vs IR arrays, require similar instrumental calibrations but are performed in a different order. Each processing operation has a switch parameter which has a single letter code. This is indicated in the PARAMETER section. The order parameter is a string of these codes which are applied from left to right. If there is a malformed order string an error will result and the task will abort. One type of error which may not be obvious is that if an operation may not appear more than once.
The processing steps have related parameters which must be set. These are things like image sections defining the bias pixels and trim region and calibration image lists. These parameters and how the various operations are performed will be described below in subsections.
The output image names are specified by the outputfIR parameter. One can provide a list which gives a name for each image in the input list. However, when processing large input lists or automatically processing data with a pipeline or in a data acquisition environment it is useful to generate output names from the input names. This can be done with pattern substitutions. The fIoutputfIR parameter provides a simple form of pattern substitution with the $I special string. Note that there is another form of template substitution which is a feature of the template lists which is described in the Ifiles help topic.
If the input parameter has the form "@inet:<number>[:<host>]" then the task will accept connections on the port given by <number> and, optionally, on the host specified by <host>. In this way the task may be used as a processing server. Logically this is simply a infinite input list which irproc reads through sequentially.
When a connection is made the task polls for image names and when one is received it processes that image. When processing is completed it returns to the polling step for additional images. The connection may be transient without error so that clients may connect, pass an image name, and disconnect.
The main reason to use this task in this way is to maintain an efficient running sky median for background subtraction. To do this the skysub parameter should be enabled, the sky parameter should be null to use the input list as the sky list, and the skymode parameter should be set to the median of some number of preceding images. The "nearest" option might also be useful in quick reduction applications so that every input image is subtracted by the preceding image. Note that in this mode the first image will not be sky subtracted.
Bad pixels may be identified during processing by an integer valued expression, specified by the badpixel parameter, evaluated on the input pixel values. Typically the expression would identify pixel values above some saturation threshold given by a header keyword. An example of this is
$I > saturate ? 4 : 0
This says if the input pixel value is greater than the value of the saturate keyword the result is the value 4 otherwise it is zero. The value 4 is used as the value in the output bad pixel mask if one is created. However, more complex expressions may be defined which may evaluate to different integer flag values.
The neighbors of a bad pixel may be flagged as bad with the same value by using the bpgrow parameter.
Note that the expression is evaluated on the input image pixels before any corrections are made. If the image has been partially processed earlier then the criteria may not be equivalent to identifying the saturated pixels in the completely unprocessed image.
The erraction parameter is provided to control whether an error will abort the task, and so any calling scripts that don't catch the abort, or if only a warning or error message is printed to the standard error output. The distinction is most useful when processing a list of images. In some cases one would like the processing to stop immediately, including a parent script, to fix the source of the error. But in more automated situations one may want to continue on to other images in the list and deal with the warning messages later. The "quit" mode is intermediate between an abort and a warning in that it lets the task exit immediately but without aborting a parent script. The script can monitor the standard error output to detect the exit.
The crosstalk correction consists of subtracting some small fraction, called the crosstalk coefficient, of the signal in one pixel from another pixel. The relationship between the pixels and the crosstalk coefficients are specified by a crosstalk calibration file specified by the xtalkfile parameter. Details of the algorithm are TBD.
The bias pixel correction consists of estimating a bias value for an image line, from either the bias pixels from that line or from all the bias pixels, and subtracting that value from all the pixel in the line. There are three types of parameters defining this correction.
The first is specifying the bias pixels with the biassec parameter. This requires the bias pixels to be recorded in the image data as a contiguous strip.
The second is specifying the algorithm with the biastype parameter. There two types of algorithms. One is where only the bias pixels for an image line are used for that line. The other is were all the bias pixels are used.
The line-only algorithms are "mean", "median", and "minmax". The mean algorithm uses the mean of all the bias pixels on the line for the bias value. Because there may be bad bias pixels the "median" option uses the median for the bias value. But the median is not a very statistically efficient estimate (meaning for a given sample size it has larger uncertainty than the mean if bad data is not present). So the last algorithm is to assume that bad pixels are rare so that by excluding the highest and lowest values from the mean the statistical estimate is not affected by bad data. This estimate for the bias value has the option name "minmax".
The algorithm that combines all the bias pixels collapses the bias pixels at each line to a value using a simple mean. The collection of bias values as a function of the line is the "bias vector". Because the individual bias values at each line are noisy due to small number sampling the bias vector is smoothed by fitting a function. This assumes that the line-by-line trends in the bias are smooth. Note that one type of function is a constant which is equivalent to averaging all the bias pixels into a single value for the image. The bias types for this algorithm are "fit" and "ifit" for an interactive and non-interactive fit respectively.
The fit uses the standard IRAF curve fitting package icfit. The parameters are organized in a parameter set (pset) and are described by the help topic biaspars. Psets are just a way to organize paraemters and help topics. However, the task sees a single set of parameters and any parameter may be set explicitly on the command line.
The linearity correction replaces pixel values by a new value which is a function of the orginal pixel value. The correction is specified by a replacement expression given by the paraemeter linexpr. The replacement expression is in terms of the uncorrected pixel value, represented in the expression by the operand $I, and the result is the corrected pixel value.
The expression is typically monotonicly increasing polynomial. The coefficients of the polynomial are constants, keywords pointing to constants, or pixel values from a coefficient image. In the first two cases all pixels use the same coefficients and so the non-linearity correction is the same. In the last case linearity correction can be different for each pixel.
The expression is implemented with a general expression evaluator described under the topic irprocexpr. The expressions can use any of the standard arithmetic operators and mathematical functions, the conditional evaluation operator, and numerical constants. The operand $I is used to reference the input pixel value. The operands $C1, $C2, ... are used to reference values in an optional linearity coefficient image given by the parameter linimage. The column and line are matched with the input pixel being corrected so the linearity image must be the same size in the first two dimensions. A third dimension may be used for coefficients C2 and higher, where the index is the value of the third dimension. Note that it is not required to use a linearity image if the expression does not depend on pixel position.
A typical expression with and without positional dependence might be a polynomial such as
"$I + (1 + $C1 * $I)" # With linearity image
"$I * (1 + 0.01 * $I)" # Without linearity image
"$I * (1 + LINCOEFF * $I)" # With coefficient in header
Fairly complex expressions can be built up, particularly using the conditional evaluation operator:
"$I*(1+$I*($I<100?$C1:($I<25000?$C1+$C2*($I-100) : 2.5)))"
"$I*(1+$I*($I<100?0.01 : ($I<25000?0.01+0.0001*($I-100) : 2.5)))"
Sky subtraction is one of the key uses of irproc. Sky subtraction consists of subtracting an estimate of the sky background at each pixel. There are a number of ways that the sky background may be estimated.
Most of the methods are based on using other images. This requires an ordered list of images. The list is specified by the sky parameter or, when a separate sky list is not defined, the list of input images is used. The list is broken down into filter lists as defined by the filter parameter. The ordering (within each filter list) is set either by the sortkey parameter or by the position in the list when no sort key is specified. The ordered list for the filter matching the image being processed is called the "ordered filter list".
To use the position in the list, the images being processed must appear in the list. However, the image being processed obviously should not be used in estimating the sky for that image and the task will automatically enforce this. For dithered observations of relatively sparse fields all images are typically used to contribute to the sky estimation of other images.
For observations of extended sources, or for chopping type of observations, it is desirable to select the images to be used. This may be done using the skykey parameter. This is any keyword which has a specific value for those to be used as sky images. Note that the sortkey and skymode parameters still apply to compute a sky using only data within a window of the image to be corrected.
The sky estimation method to be used is specified by the skymode parameter. The choices are:
If the maskkey parameter points to a keyword whose value is a bad pixel mask then the non-zero values in the mask will exclude pixels from the sky estimate. If all images exclude the same pixel then no pixels will be excluded. Note that for "nearest" sky this means that a bad pixel mask has no effect.
Dark and flat field calibration images are specified by list of image names. A list is allowed because each input image may require different calibration images or there may be several choices from which irproc can choose. The most appropriate image is chosen based on exposure time (exptime parameter) for dark images or filter and grating setting (filter parameter) for flat field images. If more than one calibration is equally good then the one closest based on the sorkey parameter is used. If there is no sorting criterion then the first one in the list order is applied.
Unlike ccdproc there is no scaling applied to the flat field. Therefore, if the flat field is not normalized the flat fielded output will have a significantly different mean level. For efficiency there is no check made for division by zero. Therefore, one should make sure that there are no bad values in the master flat fields when they are created. One way to do this is to apply the fixpix correction to the flat fields to replace bad pixels specified in a mask with interpolated values.
Note that there is no check made that the images specified in the list are actually of the appropriate type and that they have been processed. It is up to the user to insure the calibration images are properly prepared. As noted previously, it is possible to define calibration images by header keywords. In this case the value will not be a list but the single calibration designated by some other mechanism through the header keyword.
Bad pixels are replaced by linear interpolation from neighboring lines and columns when the parameter fixpix is set. The algorithm is the same as used in the task fixpix. There are two sources of bad pixels which are interpolated. One is the bad pixel mask from the input image as specified by the header keyword defined by the maskkey parameter. The other are bad pixels identifed by the bad pixel expression given by the badpixel parameter.
When the parameter trim is set the input image will be trimmed to the image section given by the parameter trimsec. This trim should, of course, be the same as that used for the calibration images.
irprocexpr, biaspars, ccdproc, fixpix, icfit