IRAF V2.12.2-EXPORT Release Notes February 6, 2004 These release notes provide a summary of the major changes in the V2.12.2 patch update. This is primarily a bugfix and platform-support release to address a number of bugs affecting IRAF tasks and other packages, and to keep the system running on the various (and many) new OS versions that have become available since the last general release. This patch updates Sun/IRAF and PC-IRAF systems only, we will consider requests for additional platform support but since the next patch is expected later this summer we will put off updating the remaining (DNUX, IRIX, and HPUX) ports until then. The V2.12.2 release is a RECOMMENDED update for all sites, especially those sites running on a Linux distribution released in the last year (e.g. Redhat 9/Fedora/RHEL, Slackware 9.x, SuSE 9.x, etc). This release is a REQUIRED update for those users wishing to reduce data using the GEMINI V1.5 package, or platforms such as Slackware 9.1 or Redhat Enterprise Linux where problems can keep IRAF from running at all. More detailed technical documentation of all system changes will be found in the 'notes.v212' file in the iraf$local directory, detailed revisions notes for each application package are in the package directories in a file called Revisions, e.g. onedspec$Revisions. Please see also the 'v212revs.txt' file in this directory for a complete description of the V2.12 release. If you are upgrading from a V2.11 system, additional features of V2.12 are described in that document but not here. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Contents: ========= 1. Highlights of This Release 1.1 Specific Bugs Fixed by This Release 2. Platform Support 2.1 Special Note to Fedora and RHEL Users 2.2 Special Note to Linux IMFORT Users 2.3 Linux PPC Support 2.4 Optimization Changes for Mac OS X 3. Core IRAF Revisions Summary - New Tasks - Existing tasks with new parameters/defaults - Existing tasks with new capabilities 4. NOAO Package Revisions Summary - New NOAO Package Tasks - Existing tasks with new parameters/defaults - Existing tasks with new capabilities 5. Parameter File Change Summary A. Details of Major Task Changes - Improvements to AUTOIDENTIFY Task for Non-Linear Dispersions - New CL Builtin Functions - XC Compiler Changes B. System Changes Which May Affect You - login.cl file changes - External Package Recompilation - Parameter File Changes ============================================================================ 1. Highlights of This Release: ============================================================================ o Fixed a number of bugs in the FITS kernel related the keyword header cache, image timestamps, bscale/bzero scaling and header corruption. o XC Compiler Changes. The XC compiler was modified on PC-IRAF systems to add new command-line flags and environment variables to support alternative compilers. Additionally, the commandline for the under- lying GCC compilation was modified as part of needed platform support for PC-IRAF. o New CL Builtin Functions. A number of new builtin functions were added to the CL scripting language. These provide complements to existing functions as well as new string handling capability. Other functions may be added in the future, suggestions for needed capabilities are welcome. See below for details. o DATAIO linked statically. The DATAIO binary no longer uses the IRAF shared library on Sun/IRAF systems to avoid memory limitations when dealing with large files. o "login.cl" File Changes. The file was modified slightly to allow the user-defined loginuser.cl file to be loaded later in the startup process, allowing the user to override or make use of system packages in their private definitions. 1.1 Specific Bugs Fixed by This Release: ------------------------------------------ The following items have been mentioned repeatedly in user mail and are listed here as a quick reference of bugs now fixed. Please see the detailed system or package Revisions notes for a complete list of changes and bug fixes. o Ctrl-C would cause the system to hang on OS X 10.2 (Janguar) systems. o Writing graphics to the image display using the 'imd' device would not display until a 'gflush' was issued. This bug was especially a problem for tasks like ELLIPSE in the STSDAS.ISOPHOTE package but would affect any tasks trying to overlay the image display. o The DATAIO package binary is now linked non-shared (i.e. will not use the IRAF shared library) on Solaris system to avoid memory limitations seen in the IMPORT/EXPORT tasks with large images. o The 'gamma' character in the SGIKERN greek font set has been restored. o File copies using iraf networking has been fixed. ============================================================================ 2. Platform Support: ============================================================================ IRAF V2.12.2 supports the following platforms: Sun/IRAF - supports SunOS 4.1 and Solaris 5.5.1 thru Solaris 9 PC-IRAF - supports RedHat 6/7/8/9 and Fedora/RHEL (RHUX) - supports Mac OS X 10.2 and 10.3 (MACX) - supports SuSE 6.2 thru Suse 9 (SUSE) - supports Slackware 8 and higher (LNUX/x86) - supports FreeBSD 4.2 and higher (FBSD) - supports Solaris 7 for x86 and higher (SSOL) - supports Yellow Dog Linux 3.0 for PPC (LNUX/ppc) Note that PC platforms not mentioned here specifically may still be supported by one or more of the distributions (e.g. Mandrake can use RHUX, Debian can use LNUX or RHUX, etc). 2.1 Special Note to Fedora and RHEL Users ------------------------------------------ Preliminary testing under Redhat Fedora and Enterprise Linux systems has revealed a potential problem with the interaction between the MEMIO interface and user resource settings. We do not yet know whether this will affect other distributions using the same newer glibc and kernel versions, or if this is a problem peculiar to Redhat. In either case the workaround will be similar and the problem will be addressed more fully in the next release. Specifically, pointers allocated in the normal course of a task may occassionally be at an address outside the user's per-process stack space, resulting in a "memory has been corrupted" or "segmentation violation" error. This problem was seen during the beta test period with IMFORT tasks and was originally thought to be a problem with the "exec-shield" feature of Fedora, but has also appeared on RHEL systems without exec-shield. The workaround is to remove the stacksize limit in the user's shell with a command such as limit stacksize unlimited # for tcsh users ulimit -s unlimited # for bash users As a preventive measure against this problem, the CL startup script was modified to implement this change and so most users who only use IRAF from the CL will not need to take any special action. This remains an issue for IMFORT tasks however, and users may need to use one of the above commands to get the tasks to run correctly. 2.2 Special Note to Linux IMFORT Users --------------------------------------- In addition to the stack size problems above, platform support for this release was further complicated by changes to glibc and the 'ld' loader on some newer linux distributions, which resulted either in unresolved symbols or a segfault from the loader itself. Users would see various combinations of those problems depending on the distribution being used. To fix the unresolved symbol problem the compatability library 'libcompat.a' (found in the iraf$unix/bin.) was updated to include the missing symbols from older glibc versions and the XC compiler modified to use this library on more platforms. The loader segfault is caused by the definition of the "Mem common" symbol "mem_" at address zero in the one and only assembler routine in IRAF (all iraf pointers are relative to this address). To fix this problem it was necessary to define the symbol on the GCC command-line instead, again by modifying the XC compiler to do this automatically. These changes will be transparent to people compiling external packages, SPP sources using the XC compiler directly, or IMFORT programs using the FC command under the CL and only affect Linux platforms. However, users who build their IMFORT programs by calling the Fortran/C compiler directly with absolute paths to the needed IRAF libraries, say from a Makefile, will need to adjust their link line to include the compatability library and the extra linker argument. To summarize, an imfort program built with F77/G77, and *not* the iraf XC/FC compilers, must now be linked as something like g77 myprog.f \ -L/iraf/iraf/bin. \ <-- define iraf paths -L/iraf/iraf/unix/bin. \ <-- define iraf paths -Wl,--defsym,mem_=0 \ <-- NEW flag, fixes 'ld' segfault -lcompat \ <-- NEW flag, fixes unresolved syms -limfort -lsys -lvops -los <-- link imfort libs Users with questions or problems should contact site support. 2.3 Linux PPC Support (***NEW***) ---------------------------------- V2.12.2 is the first release to fully support Linux for the PPC platform as part of the PC-IRAF system. The port was done a while ago but never fully integrated into the V2.12 release until a disk failure prompted us to configure a dual-boot machine. This upgrade contains the completed port which was built using Yellow Dog Linux V3.0.1 on a PowerMac G4 733Mhz system. The port appears to be stable in our limited testing but we would be interested in working with users on this platform in finding any problems. Initial benchmarking of IRAF V2.12.2 shows that on average IRAF is roughly 30% *faster* under YDL than on the same machine running OS X 10.3.2. The exact cause of this difference is not yet fully understood, see below for benchmark results offering some interesting results. It is also not known at this time whether the binaries will work on other PPC Linux systems such as Redhat, Debian or SuSE (and support for G5 systems is even more in doubt). The system should build cleanly from source on other distributions however some tweaking may be required. If you are interested in building IRAF for a distribution other that YDL, please see the "IRAF Site Manager's Guide" for details on the complete build steps, or contact IRAF site support (iraf@noao.edu). We are grateful to Terra Soft Solutions (www.terrasoftsolution.com) and Yellow Dog Linux (www.yellowdoglinux.com) for their continued interest and support in getting IRAF ported to this platform. 2.4 Optimization Changes For MAC OS X -------------------------------------- The default compiler flags for PC-IRAF systems were re-examined as part of this release with an eye towards improving performance. As part of this, the default optimization level under Mac OS X was raised to "-O3" despite providing only a relatively slight improvement in speed on the same hardware. Similar changes were investigated for Linux systems but had to be abandoned due to a (as yet untraced) compiler optimizer bug present in GCC 2.95 thru GCC 3.2 which produce incorrect results. This bug is present in both x86 and ppc linux systems, however the GCC 3.3 compiler under OS X 10.3 seems have fixed it. Sample benchmarks (including LinuxPPC for comparison) are shown below. It's important to understand that these tests provide only a very crude benchmark, but show systematic differences between the platforms tested. All tests were conducted on the same physical machine. XREGISTER Tests: A CPU intensive test using the following commands: cl> blkrep dev$pix test100.fits 4 4 cl> $xregister test100.fits test100.fits "[*,*]" test1 Results: Version Opt Flag Build OS Time ------- -------- -------- ---- (Times were about equal V2.12.1 -O OSX 10.1 1:19 for both FITS and IMH V2.12.2 -O OSX 10.3.2 1:02 image formats) V2.12.2 -O3 OSX 10.3.2 0:52 V2.12.2 -O YDL 3.0.1 0:52 BENCH script: A crude script which approximates a CCD reduction and analysis. Uses both CPU and Disk-intensive tasks to generate and operate on the images. See script at ftp://iraf.noao.edu/pub/bench.cl Results: V2.12.1/-O V2.12.2/-O V2.12.2/-O3 V2.12.2/YDL 3.0 imh fits imh fits imh fits imh fits Make 7 imgs 41 48 38 50 40 52 26 33 Proc 5 imgs 28 36 25 37 25 36 6 15 Combine 5 imgs 16 16 14 17 12 16 12 11 Median 1 img 32 32 29 33 28 32 27 17 Total time 117 133 106 137 106 137 71 77 Following the V2.12.2 release we will examine the potential gains for further tuning the OS X binaries for G5 systems. If these are found to be significant a separate set of binaries for G5 will be released. ============================================================================ 3. CORE IRAF REVISIONS SUMMARY ============================================================================ This section describes changes to tasks in the IRAF core system other than routine bug fixes. New Tasks --------- system.touch - Change file access and modification proto.ringavg - Compute pixel averages in concentric rings about a center Existing Tasks with New Parameters or New Parameter Defaults ------------------------------------------------------------ plot.pradprof - new params 'az1' and 'az2' to select profile azimuth ranges immatch.geoxytran - new param 'direction' for evaluating transformation type Existing Tasks with New Capabilities ------------------------------------ IMAGES GEOXYTRAN - new param to allow evaluating the transformation in either the forward direction (the previous behavior and default with the new parameter) or the backward direction. IMALIGN - restructured to avoid 'goto' stmts for PyRAF compatability PLOT PRADPROF - new params to select range of azimuths for profile UTILITIES URAND - modified seed algorithm to avoid 1 sec granularity ============================================================================== 4. NOAO PACKAGE REVISIONS SUMMARY ============================================================================== This section describes changes to tasks in the NOAO package tasks other than routine bug fixes. New NOAO Package Tasks ---------------------- longslit.fceval - Evaluate coordinates using the FITSCOORDS solutions nproto.mkms - Create multispec from 1D spectra including associated bands obsutil.cgiparse - Parse STRING_QUERY environment variable into task parameters Existing Packages and Tasks with New Parameters or New Parameter Defaults ------------------------------------------------------------------------- ace.diffdetect - quoted prompt strings apextract.apflat1 - expanded abbreviations of redirected param ccdtime.ccdtime - minimum seeing changed to 0.001 identify.aidpars - 'rms' param now specified in units of 'fwidth' param obsutil.specpars - aperture type now 'circular' instead of 'circle' obsutil.sptime - added new 'minexp' minimum exposure time param Existing Tasks with New Capabilities ------------------------------------ ARTDATA MKHEADER - task now truncates header cards to 80 chars NPROTO FINDGAIN - restructured to avoid 'goto' stmts for PyRAF compatability FINDTHRESH - restructured to avoid 'goto' stmts for PyRAF compatability ONEDSPEC AUTIDENTIFY - Improved algorithm to better support non-linear dispersions which would previously fail to find a solution SMW - The SMW (spectral wcs) interface was modified such that - if CUNITn is specified in velocity (m/s or km/s) and if CTYPEn is VELO (or VELOCITY) then internally the velocity zero point reference of 21 centimenters will be added. - Experience has shown that data with no units that users want to import is mostly in Angstroms. So rather than use the old FITS standard that units are meters it will now assume Angstroms. TWODSPEC APEXTRACT - Support for multiextension FITS data was added LONGSLIT TRANSFORM - Features in the IDENTIFY database with zero weight are now ignored. ============================================================================== 5. Parameter File Change Summary ============================================================================== The parameter file changes below are for modifications between V2.12.1 and V2.12.2. In the tables below each parameter change is identified with one of the following codes followed by task_name.parameter_name and the description of the change. * n = new parameter * c = changed/modified parameter * d = deleted parameter Core IRAF System Changes: ------------------------- IMMATCH: n geoxytran.direction: added new param 'direction' PLOT: n pradprof.az1: added new param 'az1' - the starting azimuth (in deg) n pradprof.az2: added new param 'az2' - the ending azimut (in deg) PROTO: n ringavg.par: New task parameter file SYSTEM: n touch.par: New task parameter file NOAO Package Changes: --------------------- ACE: c diffdetect.par: Quoted the prompt strings APEXTRACT: c apflat1.par: expanded abbreviation of 'apflat' to task 'apflatten' DIGIPHOT.PHOTCAL: c photcal.catdir: Fixed typo in 'catdir' parameter value NPROTO.MKMS: n mkms.par: New task parameter file OBSUTIL: CGIPARSE: n cgiparse.par: New task parameter file CCDTIME: c ccdtime.seeing: changed default value from 0.1 to 0.001 arcsec SPTIME: c specpars.aptype: modified option 'circle' to 'circular' c sptime.skysub: added new 'shuffle' option n sptime.minexp: added new min exposure param ONEDSPEC.IDENTIFY: c aidpars.par: new parameters to support improved dispersion finding algorithm TWODSPEC.LONGSLIT: n fceval.par: New task parameter file ============================================================================== A. Details of Major Task Changes ============================================================================== * IMPROVEMENTS TO AUTOIDENTIFY TASK FOR NON-LINEAR DISPERSIONS The AUTOIDENTIFY algorithm and parameters were modified in this release to be more robust and flexible in automating the determin- ation of dispersion solution. In particular, the task will work more reliably with spectra having a greater degree of non-linearity. These changes affect the primary task AUTOIDENTIFY and also IDENTIFY and REIDENTIFY since they may be used to apply the same algorithm. The new parameters are "crquad" and "maxnl" in the AIDPARS parameter set. In addition, the default values for "cddir", "ntarget", "ndmax", "rms", and "fmatch" were changed. The algorithm changes include searching a longer list of potential candidate line identifications and dispersions with a greater degree of possible non-linearity and trying patterns with smaller number of lines when larger patterns fail. * NEW CL BUILTIN FUNCTIONS Several new functions were added to the CL to improve string handling and the provide complementary functions to those which already exist. Additional functions may be added in the next release, suggestions for missing capabilities are also solicited. These new functions include: isindef(expr) Can be used to check for INDEF values in expressions. INDEF values may be tested for equality, however when otherwise used in a boolean expression the result of the boolean is also INDEF. This function can be used to trap this particular case, or for INDEF strings/variable directly. Result is a boolean yes/no. strldx(chars,str) Complement to the stridx which returns the last occurance of any of 'chars' in 'str'. Returns index of last char or zero if not found. strlwr(str) Convert the string to lower case, returns a string. strupr(str) Convert the string to upper case, returns a string. strstr(str1,str2) Search for first occurance of 'str1' in 'str2', returns index of the start of 'str1' or zero if not found. strlstr(str1,str2) Search for last occurance of 'str1' in 'str2', returns index of the start of 'str1' or zero if not found. The new string functions are particularly use for dealing with pathnames where one needs to find and extension, separate a file from a path prefix, and so on. New builtin functions may be added in the next release if needed. Also, modified the substr() function to allow a 'last' index greater than a 'first' index, in which case the returned string is reversed. * XC COMPILER CHANGES The XC compiler was modified to allow better support for alternative host Fortran/C compilers and for user-specified flags. In particular: - The environment variables "XC-CC", "XC-F77" and "XC-LINKER" which can be used to define a different compiler command may now also be specified using underscores instead of a dash. Recent 'tcsh' and 'bash' shells no longer permit a dash in environment variable names, this change retains the previous functionality. - Three new environment variables, "XC-CFLAGS", "XC-FFLAGS", and "XC-LFLAGS" (and the same with underscores instead of dashes), are now supported to allow user-defined compiler/linker flags to be defined. These can be used for instance to reset the optimization level or pass thru additional flags to the host compiler. These differ slightly from the "-/" passthru flag as they can be used to target a particular compiler stage, and since they are applied after the hardwired flags they can be used to override the default flags. - A new '-C' XC flag has been added and is required on LinuxPPC systems for compiling host programs with a C program main(). [PC-IRAF only] - A new '-G' XC flag has been added to force the link to be done using the system libg2c.a rather than the default libf2c.a distributed with IRAF. Platforms such as LinuxPPC require this library (but this is done automatically) and it's possible other linux systems will in the near future. This flag is also required when using alternative compilers such as Absoft Fortran. ============================================================================== B. System Changes Which May Affect You ============================================================================== * LOGIN.CL FILE CHANGES There were slight modifications to the default login.cl file with this update which will cause the system to prompt you with a warning that the file should be regenerated. Specifically, the user's "loginuser.cl" file is now loaded *after* the default CL package, allowing the file to override or make use of the core system packages earlier in the login process. Additionally, the 'touch' foreign command has been removed to avoid confusion with the SYSTEM.TOUCH task. * EXTERNAL PACKAGE RECOMPILATION The V2.12.2 release contains changes to the IMIO interface header files used by numerous applications, as well as numerous bug fixes to FITS kernel. Relinking of *all* external packages is recommended to pick up these changes. In the worst case, not doing so could lead to a runtime error due to the incompatibilities, at the least bugs fixed in this release may still be present in external package code. * PARAMETER FILE CHANGES As with all releases, we recommend that you do a MKIRAF and delete all your old parameter files after the IRAF upgrade. You may choose not to do this if you are in the midst of a project and have setups that may be difficult to reproduce, or if you use none of the tasks listed above.