NAME

USAGE

DESCRIPTION

The task batsurvey performs basic analysis of BAT "survey" data, also known as detector plane histograms or DPHs. As opposed to event data, BAT survey data is accumulated in histograms on-board the spacecraft, with typical integration times of 300 seconds. An 80-channel binned spectrum is recorded for each of the 32768 detectors and saved in the DPH files.

The batsurvey task reduces a set of "raw" observed DPHs. Most importantly, it performs data screening that the BAT team has found vital for obtaining good quality results. It produces sky images and source fluxes for each independent "snapshot," corresponding to a single pointed visit by Swift. Users may choose a set of independent energy bins, and the batsurvey will record the images and fluxes in each of those bands separately. A practical number of energy bands is eight to twelve.

INPUTS

batsurvey operates on a single Swift observation. It relies on the "bat" and "auxil" data. If you wish to reduce multiple observations, then you must call batsurvey once for each observation.

By default batsurvey will detect any new sources above the specified blind signal-to-noise threshold. However, you may wish to enter an analysis catalog which specifies any additional sources that you expect *might* be detectable, regardless of the blind threshold. The format of this catalog is defined in the help-file for batcelldetect and batclean. Upon completion, the fluxes of all known and newly detected sources will be recorded in the *.cat output files.

batsurvey places its results in the directory specified by the 'outdir' parameter. The results can be divided into two categories: global results which apply to the entire analysis, and exposure-specific results.

The global results are stored in subdirectories underneath the 'outdir' directory. The important global results are:

1. gti - contains various good time intervals identified by the prcessing script. Most of these intervals are derived by the various tests of batsurvey-gti. The combination of all temporal filters is stored in gti/master.gti.
2. dph - contains the energy-corrected survey DPHs, as produced by batsurvey-erebin. There should be two files in this subdirectory for each DPH file: a corrected DPH file named like swNNNNNNNNNNN_erebin.dph, and a detector quality mask file.
3. outventory.dat - an ASCII text file containing the summary of the processing for each eposure.

The exposure-specific results are stored in subdirectories underneath the 'outdir' directory. In the case of batsurvey, an exposure is considered to be one contiguous exposure of minimum duration. While one Swift snapshot might be a single exposure, it is also possible for a snapshot to be divided into two or more exposures if the script detects something problematic during the snapshot.

The snapshot directory names are formatted like "point_YYYYDDDhhmm" where YYYY, DDD, hh, mm are the year, day of year, hour and minute of the start of a snapshot.

There are many outputs within the each exposure subdirectory. The following outputs are documented to be present. If the output is not documented here, then you must not assume it will be present in future releases of the batsurvey software.

1. point_YYYYDDDhhmm_1.dpi - detector plane images, one for each energy band.
2. point_YYYYDDDhhmm_2.detmask - final detector plane quality map for this image. NOTE: currently this is a single map which applies to all energy bands, but in the future, there may be a quality map for each energy band.
3. point_YYYYDDDhhmm_2.bkgdpi - final detector background map for this detector image, one for each energy band.
4. point_YYYYDDDhhmm_2.img - final sky flux map constructed from the above detector/quality maps, one for each energy band. The units are "RATE."
5. point_YYYYDDDhhmm_2.var - final sky noise map constructed from the sky image, one for each energy band.
6. point_YYYYDDDhhmm_2.cat - flux catalog for sources in the sky image. This catalog contains sources listed in the input catalog as well as sources detected by blind search. The flux columns will be vectors, one measurement for each energy band.
7. point_YYYYDDDhhmm_chi.fits - Chi-square value for each detector plane image.
8. point_YYYYDDDhhmm.occimg - occultation map for this exposure.
9. point_YYYYDDDhhmm.poivar - sky noise map based on propagation of Poisson errors. This should be the lower limit for sensitivity.
10. point_YYYYDDDhhmm.att - cleaned attitude file for this exposure.
11. point_YYYYDDDhhmm_pnt.gti - good time interval for this exposure.
12. point_YYYYDDDhhmm_status.txt - ASCII file, containing status of processing.

Many of the outputs listed above have a file name like "*_2.*". These files are the result of two processing iterations, which defaults to two (ncleaniter=2). If you request more iterations, then this suffix number will be different. For example, if you choose ncleaniter=3, then all of the "*_2.*" files listed above will actually be "*_3.*".

NOTE: if processing succeeded for a given exposure, then the _status.txt file will contain the string 'status="SUCCESS"'. For a processing failure, the status file will not contain this string. A failure will be indicated by a particular reason="XXXX" code.

ANALYSIS METHODS

Unlike most FTOOLS, which are usually intended to be basic "building block" type tasks, the batsurvey is a complex pipeline task which has many steps and logical paths. The basic processing steps are: energy correction (batsurvey-erebin); temporal filtering (batsurvey-gti and batsurvey-aspect); spatial filtering (batsurvey-detmask); cleaning; and generation of sky maps. Unfortunately, it is difficult to describe all of these steps in this help file. We refer the user to the BAT User Manual for more detailed information of the analysis procedures.

IMPORTANT NOTES

While batsurvey performs many important steps required for the processing of BAT survey data, there are some issues that need attention.

This task has many parameters controlling the filtering and adjustment of the data. Most of the defaults have been set by the BAT team after some considerable experience with processing much data. Generally, users should not need to modify these parameters. The most important parameters are: 'incatalog', which allows the user to specify their a priori sources of interest; 'energybins', to select the energy bins (but see below); and 'timesep' to determine the coarseness of time sampling.

Before using any processing results, you must check the status of a pointing to make sure of the success conditions (as described above). Also, the point_YYYYDDDhhmm_chi.fits file can be used to filter any images with poor chi-square value, which indicates a bad-quality fit.

The image and catalog results for off-axis sources will be distorted by the known effects of passive absorption in the BAT imaging system, as already documented on the BAT Analysis Issues section of the BAT Digest web page. Therefore, at the present, the BAT team recommends users to limit analysis to on-axis sources until a more generic approach can be made available.

Regarding energy bins, the BAT team recommends either 4 energy bins ("14-24,24-50,50-100,100-195") or 8 energy bins ("14-20,20-24,24-35,35-50,50-75,75-100,100-150,150-195"). Other energy bin selections are certainly possible, however, the above selections will allow the most support from the BAT team in the future. Choosing fewer than four energy bins is not recommended since, the response of the instrument changes with energy and angle; lumping all energies together will lower sensitivity. Choosing many energy bins is also not recommended. With many energy bins, the number of counts per detector per energy bin becomes small, and the statistical properties of this condition have not been fully investigated.

Memory usage: this task is heavily memory intensive! Expect to require about 50 megabytes of available memory per energy band listed in 'energybins'.

OUTPUTS

PARAMETERS

indir [directory]

Name of Swift observation directory. This directory should have the layout and contents of a single standard Swift observation segment. The directories "indir/bat" and "indir/auxil" are required.

outdir [directory]

Name of output directory where the processed results are stored. If necessary, this directory will be created. The structure and contents of the output directory are documented above.

(incatalog="NONE") [string]

Name of input analysis catalog. This catalog should be in the format that batcelldetect accepts.

(ncleaniter=2) [integer]

Number of cleaning iterations to perform. Setting ncleaniter=1 will prevent bright sources from being removed. The BAT team has found that a setting of 3 or more provides no additional benefit over ncleaniter=2.

(energybins = "14-20,20-24,24-35,35-50,50-75,75-100,100-150,150-195") [string]

List of energy bins, in keV and comma-separated. The bins must not overlap. See above for a discussion of limitations regarding energy bins. Note that if you change the lower- or upper-most bin then you should also adjust the 'elimits' parameter to match.

(elimits = "14-195") [string]

Energy limits for DPI quality checking, in keV. The energy limits should match bin edges specified by the 'energybins' parameter.

(timesep = "SNAPSHOT") [string]

Method of time separation, either "SNAPSHOT" or "DPH". For "SNAPSHOT", snapshot observations are processed separately, corresponding to the coarsest possible time sampling. For "DPH", the individual survey DPHs are processed separately, corresponding to the finest possible time sampling.

(keepbits = "7") [string]

Integer number of bits of precision to keep in the output images. A value of ALL means keep all bits. A positive value indicates that 'keepbits' bits of precision should be preserved, and the other bits should be set to zero. See the documentation for 'batfftimage' for more information.

(keep_sky_images = "LAST") [string]

Number of sky images to keep, one of "ALL" "NONE" or "LAST". A value of "ALL" indicates that sky images from all clean iterations should be kept. A value of "LAST" indicates that only the last iteration should be kept (the default). A value of "NONE" indicates that sky images should be discarded.

(poivarmap = "YES") [boolean]

Indicates whether a Poisson-based noise map should be created. The Poisson-based noise map is not used in the analysis, but may be of interest in post-processing. Regardless of this setting, batsurvey always creates and uses a noise map from batcelldetect.

(pointing_check = "YES") [boolean]

Indicates whether periods of bad spacecraft pointing should be filtered. Here "bad pointing" means that the spacecraft was drifting or settling a significant distance from the final stable pointing direction (as determined by the "point_toler" and "roll_toler" parameters). Regardless of this setting, batsurvey always excludes data where the spacecraft declares it is slewing, and where a star tracker problem is indicated in the telemetry.

(filter_midnight = "YES") [boolean]

Indicates whether DPHs which cross the midnight boundary should be excluded. The purpose of this filter is to remove cases where Swift executes a "roll-in-place" maneuver at the hand-over from one daily observing plan to the next.

(ratemaxthresh = 12000) [real]

Maximum allowed count rate in a DPH, in units of counts/sec. The purpose of this filter is to exclude periods of high background.

(rateminthresh = 3000) [real]

Minimum allowed count rate in a DPH, in units of counts/sec. The purpose of this filter is to exclude anomalously low count rates.

(detthresh = 22000) [integer]

The minimum allowed number of enabled detectors. The purpose of this filter is to exclude periods of time where much of the BAT array was disabled.

(detthresh2 = 16000) [integer]

The minimum allowed number of unmasked detectors. The purpose of this filter is to exclude data sets where too many detectors were masked. This filtering occurs after detectors are masked by various quality checks, whereas the filtering associated with "detthresh" occurs before.

(filtnames = "all") [string]

Comma-separated list of standard time filtering methods to apply. See the help file for batsurvey-gti for a list of possible methods, or use 'all' to activate all appropriate filtering techniques.

(gtifile = "NONE") [string]

User requested good time interval filter, or NONE. This good time interval filter is intersected with the standard quality filtering done by batsurvey-gti. In other words, the user-requested gtifile can only reduce exposure, not increase it. A value of "NONE" indicates no extra temporal filtering beyond the standard filtering. NOTE that small amounts of time outside of the 'gtifile' may be accepted for processing, as controlled by the parameters 'min_dph_frac_overlap', 'min_dph_time_overlap' and 'max_dph_time_nonoverlap' of the task batbinevt.

(filtexpr = "NONE") [string]

Additional filtering expression based on filter file, beyond the standard methods specified by 'filtnames' and 'saofiltexpr'. A value of "NONE" indicates no extra temporal filtering beyond what is done by batsurvey-gti.

(saofiltexpr = "ELV > 30.0") [string]

Filter expression based on columns in the "prefilter" file, usually based on spacecraft position or pointing.

(stlossfcnthresh = 1.0E-9) [real]

Maximum allowed star tracker "loss function." The loss function provides an indicator of the quality of the star tracker attitude solution.

(expothresh = 150.0) [real]

Minimum allowed exposure per snapshot, in seconds.

(timeslop = 0.0) [real]

Amount to enlarge each snapshot GTI. It is not recommended to change this setting.

(snrthresh = 5.0) [real]

Minimum signal-to-noise threshold for detecting new sources in a sky image.

(cleansnr = 9.0) [real]

Minimum signal-to-noise threshold for cleaning. Sources detected at a greater significance will be cleaned via batclean, while sources with a smaller significance will not be cleaned.

(cleanexpr = "NONE") [string]

Selection expression used to enable cleaning on additional sources, using standard CFITSIO filtering criteria (see fhelp for 'rowfilter'). This expression is applied to the output catalog from batcelldetect; any columns produced by batcelldetect - including those columns supplied by the user in their input catalog - are available to be filtered on. Sources to be cleaned will have either SNR > 'cleansnr' OR 'cleanexpr' true. A value of "NONE" indicates only filtering by 'cleansnr'.

(balance = "ShortEdges,LongEdges,InOut") [string]

Type of balancing to do when calling batclean. See documentation for batclean for more information.

(brightthresh = 0.017) [real]

Minimum source count rate to activate mask edge removal, in count/s/det. For sources brighter than this intensity, the shadows of the perimeter of the mask are removed. The purpose of this spatial filter are to remove difficult-to-model shadows associated with passive materials at the edge of the BAT mask structure.

(pcodethresh = 0.05) [real]

Minimum partial coding for sources detected in sky images, where partial coding is measured as a fraction.

(bkgpcodethresh = 0.01) [real]

Minimum partial coding for background estimates from sky images, where partial coding is measured as a fraction.

(badpixthresh = 4.0) [real]

Maximum allowed residuals after cleaning, in units of sigma. Detectors with larger residuals indicate faulty cleaning, and those detectors are excluded in subsequent cleaning iterations.

(copy_cleaned_sources = "YES") [boolean]

After one cleaning iteration, bright sources will have been removed from the sky images. Setting this parameter to "YES" will transfer the bright sources from the previous cleaning iteration to the current image, thus keeping "all" of the source intensity information. For each source more significant than "cleansnr", a circular disk of radius "copy_cleaned_radius" is transferred from the previous iteration to the current iteration. Note that since a bright source can induce its own noise, setting this parameter to "YES" can make bright sources appear to have unphysical significances.

(copy_cleaned_radius = 0.008) [real]

Radius around each cleaned source to copy, in radians, when copy_cleaned_sources is set to YES.

(batclean_backexp = NO) [boolean]

If set, then the diagnostic 'backexp' output of batclean is saved for each snapshot. The name of the file is point_YYYYDDDhhmm.backexp. See the documentation for batclean for more information. If batclean_backexp=NO (the default), then no batclean diagnostic file is saved.

(point_toler = 0.025) [real]

The "bad pointing" threshold radius, in degrees. When the spacecraft pointing is more than "point_toler" degrees from the final pointing direction, it is declared "bad".

(roll_toler = 0.083) [real]

The "bad pointing" threshold in roll, in degrees. When the spacecraft roll is more than "roll_toler" degrees from the final roll angle, it is declared "bad".

(pointerr_frac_time = 0.15) [real]

For a given DPH, if the pointing is "bad" for a fractional duration that exceeds "pointerr_frac_time" then that DPH is excluded. For example, if the pointing is "bad" for 14% of the exposure of a DPH, then the DPH would be kept with the default setting, and rejected if pointerr_frac_time=0.10.

(pointerr_abs_time = 35.0) [real]

For a given DPH, if the pointing is "bad" for a duration that exceeds "pointerr_abs_time" then that DPH is excluded. pointerr_abs_time is measured in seconds.

(min_dph_frac_overlap = 0.75) [real]

When deciding whether to include or exclude a DPH in a snapshot based on the GTI filter, the amount of overlap between the DPH and the GTI are determined. If the fractional overlap exceeds "min_dph_frac_overlap" then the DPH is kept, otherwise it is discarded.

(min_dph_time_nonoverlap = 40.0) [real]

When deciding whether to include or exclude a DPH in a snapshot based on the GTI filter, the amount of non-overlap between the DPH and the GTI are determined. If the non-overlap exceeds "min_dph_time_nonoverlap" (in seconds) then the DPH is discarded, otherwise it is kept.

(bsurseq = "NONE") [string]

String value to be assigned to the BSURSEQ keyword in output files. The contents of the BSURSEQ are not used by BAT software; the user may use this keyword to identify data for their own purposes. A value of NONE indicates that the BSURSEQ keyword should not be written.

(aperture = "CALDB") [string]

Name of the BAT mask aperture to use, or "CALDB".

(mask_edge_aperture = "CALDB") [string]

Name of the BAT mask edge aperture to use, or "CALDB".

(global_pattern_map = "NONE") [string]

Name of global detector pattern map to use for bias subtraction of each image. The revised count rate is computed as NEW_RATE = (OLD_RATE - PATTERN_MAP). The number of images in the file must match the number of requested energy bands. A value of "NONE" indicates no bias subtraction.

(global_pattern_mask = "NONE") [string]

Name of global quality map file, to be used for spatial filtering of known "bad" detectors. This should be a single standard detector quality map which applies to all energy bands. Note that detectors already flagged as bad by batdetmask do not need to be re-flagged.

(alignfile = "CALDB") [string]

Spacecraft alignment file name, or "CALDB".

(off_axis_corr = "NO") [boolean]

Whether to correct for known off-axis effects. This parameter is not fully implemented, and should always be set to NO.

(offcorrfile = "NONE") [string]

Name of off-axis correction file. This parameter is not fully implemented, and should always be set to NONE.

(check_centroiding = "NO") [boolean]

Verify centroiding process. This parameter is not fully implemented, and should always be set to NO.

(clobber = YES) [boolean]

If the output files already exist, then setting "clobber = yes" will cause it to be overwritten. Note that files will always be clobbered. Setting clobber=NO will have no effect.

(chatter = 2) [integer, 0 - 5]

Verbosity level of output. Note that this setting is of limited use, since most sub-tasks are run with a standard level of verbosity.

(history = YES) [boolean]

Ignored.

EXAMPLES

1. Run BAT survey processing on observation 00035025015 and place the results in the 00035025015-results directory. The default energy bins are used.

     batsurvey 00035025015 00035025015-results

SEE ALSO

batsurvey-aspect, batsurvey-detmask, batsurvey-erebin, batsurvey-detmask, batsurvey-gti

LAST MODIFIED