Rebuilding CIAO 4.8 source with pre-built OTS binaries on Linux
![[CXC Logo]](../../imgs/cxc-logo.gif)
CIAO 4.8 Science Threads
Overview
Synopsis:
CIAO depends on a number of external software components we refer to as OTS (off-the-shelf) software. The following instructions let you build CIAO 4.8 from source on Linux systems, but assume you are using pre-built binaries for the OTS software. With these directions, you can rebuild CIAO entirely, or rebuild selected tools in the source code.
Related Links:
- Building CIAO from source on OS X: to build CIAO from source with pre-built OTS binaries on Mac OS X systems.
- Downloading CIAO: for a quick, automated installation of CIAO through the ciao-install script.
- Manually Installing CIAO: for manually installing CIAO using the binary packages.
Last Update: 30 11 2015 - Updated instructions for ciao 4.8 release
Contents
- A few notes before you start
- CIAO Source Build Quickstart
- Requirements
- Build Preparation
- Build Configuration
- Build Instructions
- AHELP Indexing
- Smoke Tests
- Troubleshooting
- History
A few notes before you start
To prevent any old variables from affecting the CIAO build environment, it is advised to open up a new terminal before continuing.
Where should CIAO be installed?
CIAO can be installed in any location and does not need to be installed with any root or super-user privileges.
Off-the-shelf software
The CIAO off-the-shelf packages include adapted OTS and some dependencies on specific versions of OTS. For example, we use a modified version of VTK for CIAO, as well as a specific snapshot of gtkglext. As such, it is highly beneficial (and will save you a lot of headaches) to build CIAO with our OTS binaries. If you do need to build CIAO completely from source, contact the CXC Helpdesk.
You can view the list of the OTS and their versions here.
CIAO Source Build Quickstart
The following are the basic directions for building CIAO 4.8 from source with pre-built OTS binaries on Linux operating systems (Only 64 bit is supported). If you are only rebuilding selected directories, skip step 4. For detailed information and explanation of the steps, please continue to the following sections.
- Download GCC and gfortran, if not already installed
-
Download and unpack CIAO binaries and source code
via ciao-install; cd into
ciao-4.8
% cd <path-to>/ciao4.8/
-
Create $ASCDS_INSTALL variable pointing to location of ciao-4.8 directory
% export ASCDS_INSTALL=`pwd` (for bash) % setenv ASCDS_INSTALL `pwd` (for tcsh)
-
Delete contents of lib/ and binexe/.
If only rebuilding selected directories, skip this step.
% rm -rf binexe/* lib/*
-
Set $PKG_CONFIG_PATH to include paths
to ${ASCDS_INSTALL}/ots/lib/pkgconfig and the path to your system pkgconfig:
For some newer Redhat based builds (e.g., CentOS 6), you need to add /usr/share/pkgconfig to $PKG_CONFIG_PATH.
% export PKG_CONFIG_PATH=${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig (for bash) % setenv PKG_CONFIG_PATH ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig (for tcsh)
% export PKG_CONFIG_PATH=${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig (for bash) % setenv PKG_CONFIG_PATH ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig (for tcsh)
-
Setup the CIAO environment and run configure. If the source code and binaries were unpacked by hand (i.e., using
bash ciao-install --download-only), then configure must first
be run without any arguments. ciaorun must be run in line with configure.
These four setps are needed ONLY IF you unpacked the source code and binaries by hand. % mv src src.save % ./configure % mv src.save src % bash src/config/ciao-python-fix (start ciao session) % . bin/ciao.sh (for bash) % source bin/ciao.csh (for tcsh) % ciaorun ./configure --with-ascii --with-fits --<other-options>
-
Re-run ciao-python-fix
% bash src/config/ciao-python-fix
-
Run make install
If only rebuilding a certain tool, cd into that tool's directory and run make install there.
% ciaorun make install
-
Run ciao-python-fix
% bash src/config/ciao-python-fix
-
Generate AHELP pages
(start CIAO session) % . bin/ciao.sh -o (for bash) % source bin/ciao.csh -o (for tcsh) % ahelp -r
-
Run smoke tests
% cd test % make -k test 2>&1 | tee ~/ciao_test.log (for bash) % make -k test |& tee ~/ciao_test.log (for tcsh)
Requirements
Operating Systems
Building CIAO with the pre-built OTS is supported on 64-bit Linux systems.
Note: we also provide instructions for building CIAO with the pre-built OTS on Mac OS X 10.8-10.10.
Compilers
Building CIAO requires C, C++ and FORTRAN compilers. We strongly recommend using GCC and gfortran. Building CIAO on Linux system has been tested with GCC and gfortran versions 4.2.x, 4.4.7, 4.8.x and 5.2.
Check to see if you have gfortran and GCC on your system:
% gcc --version % gfortran --version
If either gives you an error, you will have to download and install that compiler.
Source files + OTS binaries
Building CIAO from source using the pre-built binary OTS requires the ciao-4.8-src-*.tar.gz packages as well as the platform specific ciao-4.8-bin-Linux64.tar.gz binaries.
First, install CIAO using the ciao-install script. Go to the Downloading CIAO page. Under Custom Installation, choose your platform and check the "Standard CIAO installation" and "Full source build" boxes. This will select all of the necessary binaries (which contain the OTS) and will download the source code. CALDB, the background event files and the science scripts package are optional; include or exclude these packages by checking the boxes next to them. Finally, click the "Download the ciao-install Script" button.
From a terminal, run the ciao-install script. ciao-install will unpack all the files for you [recommended].
% bash <path-to>/ciao-install
If you want to unpack the files yourself, use the --download-only switch:
% bash <path-to>/ciao-install --download-only
and then unpack the tar.gz packages. For example, the commands for unpacking the "Standard CIAO installation" Linux 64-bit packages are shown below.
% tar -xzf ciao-4.8-bin-chips-Linux64.tar.gz % tar -xzf ciao-4.8-bin-core-Linux64.tar.gz % tar -xzf ciao-4.8-bin-graphics-Linux64.tar.gz % tar -xzf ciao-4.8-bin-obsvis-osx64.tar.gz % tar -xzf ciao-4.8-bin-prism-Linux64.tar.gz % tar -xzf ciao-4.8-bin-sherpa-Linux64.tar.gz % tar -xzf ciao-4.8-bin-tools-Linux64.tar.gz % tar -xzf ciao-4.8-contrib-DEV.tar.gz % tar -xzf ciao-4.8-src-chips.tar.gz % tar -xzf ciao-4.8-src-core.tar.gz % tar -xzf ciao-4.8-src-obsvis.tar.gz % tar -xzf ciao-4.8-src-prism.tar.gz % tar -xzf ciao-4.8-src-sherpa.tar.gz % tar -xzf ciao-4.8-src-tools.tar.gz % tar -xzf caldb_4.6.5_main.tar.gz
Please note that the list of files to untar depends on the selections made when downloading the ciao-install script.
Unpacking caldb_4.7.0_main.tar.gz will create directories data/, docs/, and software/. You will need to create a CALDB directory and move data/, docs/ and software/ into CALDB/, then softlink CALDB to CIAO after the build.
> # in directory where caldb_4.7.0_main.tar.gz is > mkdir CALDB > mv docs/ data/ software/ CALDB/
Note: ciao-4.8-bin-obsvis-<plat>.tar.gz is NOT needed as no OTS are packaged in that binary distribution.
Note: If you are having trouble with the ciao-install script, refer to "Downloading the Software" in Manually Installing CIAO.
Build Preparation
-
Untarring the ciao-4.8-*.tar.gz files creates a directory ciao-4.8; cd into it:
You will make many references to the ciao-4.8 directory during the build configuration. Creating an environment variable for the path to ciao-4.8 will make the build commands easier:% cd ciao-4.8 % ls bin config.h.in COPYING.LIB LICENSE README.obsvis VERSION_obsvis binexe config.log data LICENSE.SAO share VERSION_prism CALDB config.status doc Linux_build.html src VERSION.prop_tk ciao_installed config.sub include Mac_build.html test VERSION_sherpa ciao-type configure INSTALL_BINARY Makefile VERSION VERSION_tools confdefs.h configure.in install-sh ots VERSION_chips config contrib INSTALL_SOURCE param VERSION_core config.guess COPYING lib README VERSION_graphics
If you only want to rebuild certain directories of the CIAO source code, skip to step (3).% export ASCDS_INSTALL=`pwd` (for bash) % setenv ASCDS_INSTALL `pwd` (for tcsh) % echo $ASCDS_INSTALL /<path-to>/ciao-4.8
-
To make sure that libraries and executables are newly created by the
source build on your system, delete the contents of
the lib/, and binexe/ directories:
% rm -rf lib/* binexe/*
-
CIAO uses GTK for chips and prism. The necessary GTK files are packaged
with the CIAO graphics binary tarfile in the pkgconfig directory. We
discovered that combining the system and CIAO GTK produces a more stable
system than the CIAO GTK alone. Therefore, you must build with both the
system and the CIAO GTK pkgconfig. To do this, set the following
variable $PKG_CONFIG_PATH:
% export PKG_CONFIG_PATH=${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig (for bash) % setenv PKG_CONFIG_PATH ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig (for tcsh)
-
Note: On Redhat Based builds (CentOS 6), you need to add
/usr/share/pkgconfig to $PKG_CONFIG_PATH:
% export PKG_CONFIG_PATH=${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig (for bash) % setenv PKG_CONFIG_PATH ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig (for tcsh)
-
Note: On Redhat Based builds (CentOS 6), you need to add
/usr/share/pkgconfig to $PKG_CONFIG_PATH:
Build Configuration
Before building CIAO, we must configure the build script for our system. This involves choosing which compilers to use, setting library locations, and adding other options to the build. All of this is handled by a configuration script.
Basic configuration
-
If you unpacked the source code and binaries by hand,
run the configuration script without any arguments before
continuing. This
links the CIAO Python and other CIAO tools to the CIAO build; skipping
this step would result in a CIAO environment without access to
all the CIAO tools.
Otherwise, skip to Step 2.% mv src src.save % ./configure % mv src.save src % bash bin/ciao-python-fix
-
Start a CIAO session.
% . bin/ciao.sh (for bash) % source bin/ciao.csh (for tcsh)
-
Run the configuration script. The most basic configuration call to
have a successful CIAO build using the default build options
(i.e., using the Python distribution packaged with CIAO):
The --with-fits and --with-ascii switches include the FITS and ASCII kernels when building the CXC DataModel (necessary to build CIAO).
% ciaorun ./configure --with-fits --with-ascii (using the default build options)
Additional arguments, described in the following subsections, may be used in the configure call to further specify the configuration.
If configure fails, see the Troubleshooting section. A likely cause of failure is the environment variables $PATH and $PKG_CONFIG_PATH are not properly set, or that there are multiple compilers on your $PATH and configure is picking-up the wrong compiler(s).
The following subsections contain details on setting the compilers, the configuration options, and compiler optimization levels.
Setting the compilers
Specifying a specific path to a compiler can be achieved by using the appropriate keyword:
- CC - Name and location of the c compiler
- CXX - Name and location of c++ compiler
- F77 - Name and location of FORTRAN compiler
- FC - Name and location of FORTRAN compiler
Usage example:
% ciaorun ./configure --with-ascii --with-fits \ FC=<path-to>/bin/gfortran-4.2 \ F77=<path-to>/bin/gfortran-4.2 \ CC=<path-to>/bin/cc \ CXX=<path-to>/bin/c++
(Exclude ciaorun from the command if building with your system Python.)
To specify the FORTRAN compiler, the FC and F77 variables must be the same.
The compiler must also be in your PATH. If <path-to-compiler> is not in $PATH, add it using
% export PATH=${PATH}:<path-to-compiler> (for bash) % setenv PATH ${PATH}:<path-to-compiler> (for tcsh)
Configuration flags/options
Here is the list of configure options available for CIAO:
Option | Description |
---|---|
--prefix=<location> | Directory to install CIAO. Default location will be '.' |
--x-includes=<DIR> | directory DIR in which the X include files are located |
--x-libraries=<DIR> | directory DIR in which the X library files are located |
--with-debug | Compile with debug mode enabled |
--with-opt=<level> | Set optimization level (default=-O3) |
--with-fits |
Include FITS kernel when building CXC DataModel.
Note: This switch is required to build the DataModel. It indicates that the setup is for a compile and not just a binary install. |
--with-ascii |
Include the ASCII kernel when building the CXC
DataModel.
Note: This switch is required if you need to read ASCII files into ChIPS. |
--with-f95loc=<DIR> | Location of libf95 (if using g95) |
--with-gfortranloc=<DIR> | Location of libgfortran (if using gfortran) Unnecessary if gfortran and libgfortran are in the same tree (e.g. if gfortran is in /usr/bin, and libgfortran is in /usr/lib, they’re in the same tree) |
--with-imagerdir=<saord> | Path to directory/tree containing DS9 |
The flags --with-ascii and --with-fits are REQUIRED for CIAO to build correctly. All other flags are optional.
Optimization and compilers
Optimization is turned on by default, -O3 for GCC. If you wish to use a different optimization level or turn off optimization, use the --with-opt= switch. To turn off optimization, use --with-opt=-O0. See GNU’s Optimization Options for more details.
Build Instructions
Now, the source tree should be properly set up to build. The directions here are for rebuilding the entire CIAO source code (Rebuilding CIAO), or for rebuilding a selected directory (Rebuilding selected source code).
Rebuilding CIAO
To build and install the code, type "make install". CIAO currently requires that "make install" be issued as a single command rather than as separate commands "make; make test; make install". This is due to build dependencies on installed libraries and packages.
% ciaorun make install
Note that this will take a few minutes.
Rebuilding selected source code
There are a few tools that depend on local libraries that if not installed may require rebuilding first. Most src directories can be rebuilt after running configure by simply stepping into the directory and running "make install".
For example, say a user rebuilt libfftw to take advantage of their specific hardware. After placing the libfftw files in $ASCDS_INSTALL/ots/lib, they’d need to re-link the tools in $ASCDS_INSTALL/src/da to their system by doing
% cd ${ASCDS_INSTALL}/src/da % ciaorun make install
to rebuild the da tools with their custom libfftw build.
Post-processing
After the build completes, run the script ciao-python-fix:
% bash src/config/ciao-python-fix
This post-processes your configuration, to byte-compile included Python files for your installation, and to ensure all CIAO packages with Python interfaces can be built. Once this script is run, the CIAO installation cannot be moved or relocated to a different directory without repeating the configure and ciao-python-fix steps.
Linking to CALDB: if you downloaded CALDB and unpacked the tarfile by hand (e.g. ran ciao-install with the --download-only switch), you can link it to CIAO at this point by creating a softlink to the CALDB directory in $ASCDS_INSTALL:
% pwd /<path-to>/ciao-4.8 % ln -s <path-to>/CALDB CALDB
AHELP Indexing
After installation you will need to generate an index file used by the CIAO help documentation application (ahelp). The ciao.*sh script needs to run to set up dynamic library paths used by ahelp.
% . bin/ciao.sh -o (for bash) % source bin/ciao.csh -o (for tcsh)
Then run the ahelp command with the -r (re-index) option.
% ahelp -r
This will generate the help indexes. If this completes successfully, you can execute the smoke tests.
Smoke Tests
To run the tests, you should do something like:
% cd test % . ../bin/ciao.sh (Note: Only needed if you have not setup for CIAO in a prior step) % make -k test 2>&1 | tee ~/ciao_test.log (for bash)
or
% cd test % source ../bin/ciao.csh (Note: Only needed if you have not setup for CIAO in a prior step) % make -k test |& tee ~/ciao_test.log (for tcsh)
A successful test will look like:
[1/42] Running test FOO-smoke001 ... PASS
where 'FOO' is the name of an individual test. If you see a message that reports:
[1/42] Running test FOO-smoke001 ... FAIL
then there is an unexpected problem with the tests. Note that if you see additional warnings/errors and the test reports PASS, then the test is good (tests sometimes do test error conditions).
All the test outputs are written to $ASCDS_WORK_PATH/smoke.$LOGNAME where $LOGNAME is your system login name. The tests need about 60 Mb to run. The test scripts clean-up the disk space upon successful completion. Tests which fail leave their test output so you may check the results. When all of the tests PASS, the smoke test scripts should remove $ASCDS_WORK_PATH/smoke.$LOGNAME and all its sub-directories.
$ASCDS_WORK_PATH is set in the $ASCDS_INSTALL/bin/ciao.*sh setup scripts as /tmp by default. If the machine does not have a /tmp or you don't have write permission, you will need to edit the appropriate ciao.*sh script to change the $ASCDS_WORK_PATH location or set the $ASCDS_WORK_PATH environment variable. If this variable is set, ciao.*sh will not modify it.
If a test FAILS, you may want to double-check that you’re using the right versions of GCC and gfortran, and that your environment variables $PATH and $PKG_CONFIG_PATH are set correctly. Also check Bugs: Installation & Smoke Tests for other documented problems and solutions; if you run into any other problems, please contact the CXC Helpdesk. If you decide to re-run the build process, repeat the configuration and building process in a new terminal to prevent the old environment from affecting the new build.
Troubleshooting
Below are some common problems you may run into while building and installing CIAO. You can also check Bugs: Installation & Smoke Tests for other documented problems and solutions. If you run into any other problems, please contact the CXC Helpdesk.
Configure script
-
Error "cfitsio is not found"
checking for pkg-config... no configure: error: cfitsio is not found. Please specify the path using '--with-cfitsio=<<path>' or 'setenv PKG_CONFIG_PATH <path>/lib/pkgconfig'
This error shows that the $PKG_CONFIG_PATH variable is not set correctly. $PKG_CONFIG_PATH should contain(for Linux 64-bit) ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig (for 64-bit Cent OS 6) ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig
-
For other errors, check that:
- your compilers are in your $PATH
- $PKG_CONFIG_PATH is set correctly
- ${ASCDS_INSTALL}/ots/bin is in your $PATH
- If you need to change your $PKG_CONFIG_PATH you will also need to re-run configure.
Build
If the build fails at any point during "make install", check that
- you have compatible GCC and gfortran compilers (see Compilers for details)
- your compilers are in your $PATH.
- ${ASCDS_INSTALL}/ots/bin is in your $PATH.
- $PKG_CONFIG_PATH is set correctly
Then, run "make -k clean", then remove the contents of lib/ and binexe/ again,
% rm -rf binexe/* lib/*
re-configure the build with
% . bin/ciao.sh -o (for bash) % source bin/ciao.csh -o (for tcsh) % ciaorun ./configure --with-fits --with-ascii (Exclude "ciaorun" from the command if building with your system Python.)
and run make install again. If it still fails, try opening-up a new terminal so that you have a clean environment to work with. Check that you have compatible compilers and that you’ve set $PKG_CONFIG_PATH and your $PATH variables correctly defined.
History
08 Dec 2014 | First issue. Merges INSTALL_SOURCE text file into CIAO website. |
30 11 2015 | Updated instructions for ciao 4.8 release |