Rebuilding CIAO 4.8 source with pre-built OTS binaries on OS X
![[CXC Logo]](../../imgs/cxc-logo.gif)
CIAO 4.8 Science Threads
Overview
Synopsis:
CIAO depends on a number of external software components we refer to as OTS (off-the-shelf) software. The following instructions let you build CIAO 4.8 from source on Mac systems, but assume you are using pre-built binaries for the OTS software. With these directions, you can rebuild CIAO entirely, or rebuild selected tools in the source code.
Related Links:
- Building CIAO from source on Linux: to build CIAO from source with pre-built OTS binaries on Linux systems.
- Downloading CIAO: for a quick, automated installation of CIAO through the ciao-install script.
- Manually Installing CIAO: for manually installing CIAO using the binary packages.
Last Update: 30 11 2015 - Updated instructions for ciao 4.8 release
Contents
- A few notes before you start
- CIAO Source Build Quickstart
- Requirements
- Build Preparation
- Build Configuration
- Build Instructions
- AHELP Indexing
- Smoke Tests
- Troubleshooting
- History
A few notes before you start
To prevent any old variables from affecting the CIAO build environment, it is advised to open up a new terminal before continuing.
Where should CIAO be installed?
CIAO can be installed in any location and does not need to be installed with any root or super-user privileges.
Off-the-shelf software
The CIAO off-the-shelf packages include adapted OTS and some dependencies on specific versions of OTS. For example, we use a modified version of VTK for CIAO, as well as a specific snapshot of gtkglext. As such, it is highly beneficial (and will save you a lot of headaches) to build CIAO with our OTS binaries. If you do wish to build CIAO completely from source, contact the CXC Helpdesk.
You can view the list of the OTS and their versions here.
CIAO Source Build Quickstart
The following are the basic directions for building CIAO 4.8 from source with pre-built OTS binaries on Mac OS X. Building CIAO 4.8 is supported on OS X 10.8-10.10. If you are only rebuilding selected directories, skip step 4. For detailed information and explanation of the steps, please continue to the following sections.
- Download GCC and gfortran, if not already installed
- Install Command Line Tools for Mac, if not already installed
-
Download and unpack CIAO binaries and source code
via ciao-install; cd
into ciao-4.8
% cd <path-to>/ciao4.8/
-
Create $ASCDS_INSTALL environment variable pointing to location of ciao-4.8 directory
% export ASCDS_INSTALL=`pwd` (for bash) % setenv ASCDS_INSTALL `pwd` (for tcsh)
-
Delete contents of lib/ and binexe/.
If only rebuilding selected directories, skip this step.
% rm -rf binexe/* lib/*
-
Set $PKG_CONFIG_PATH to the OTS pkgconfig and your system's
pkgconfig in X11. If you have xQuartz installed, the X11 libraries are
located in /opt/X11.
% export PKG_CONFIG_PATH=${ASCDS_INSTALL}/ots/lib/pkgconfig:/opt/X11/lib/pkgconfig (for bash) % setenv PKG_CONFIG_PATH ${ASCDS_INSTALL}/ots/lib/pkgconfig:/opt/X11/lib/pkgconfig (for tcsh)
-
Setup the CIAO environment and run configure. If the source code and binaries were unpacked by hand (i.e., using
bash ciao-install --download-only), then configure must first
be run without any arguments. Also, if using the CIAO Python distribution
(i.e., you do not have Python 2.7.x), ciaorun must be run in-line with configure.
These four step are needed ONLY IF you unpacked the source code and binaries by hand % mv src src.save % ./configure % mv src.save src % bash bin/ciao-python-fix (start ciao session) % . bin/ciao.sh (for bash) % source bin/ciao.csh (for tcsh) % ciaorun ./configure --with-ascii --with-fits --x-includes=/opt/X11/include --x-libraries=/opt/X11/lib --<other-options>
-
Run make install
If only rebuilding a certain tool, cd into that tool's directory and run make install there.
% ciaorun make install
-
Re-run ciao-python-fix
% bash src/config/ciao-python-fix
-
Generate AHELP pages
(start CIAO session) % . bin/ciao.sh -o (for bash) % source bin/ciao.csh -o (for tcsh) % ahelp -r
-
Run smoke tests
% cd test % make 2>&1 | tee ~/ciao_test.log (for bash) % make |& tee ~/ciao_test.log (for tcsh)
Requirements
This section describes the necessary libraries and files needed to build CIAO.
Operating Systems
Building CIAO with the pre-built OTS is supported on 64-bit OS X 10.8-10.10.
Note: we also provide instructions for building CIAO with the pre-built OTS on Linux systems.
Compilers
Building CIAO requires C, C++ and FORTRAN compilers. We strongly recommend using GNU GCC and gfortran. Building CIAO on Mac Intel systems has been tested with GCC 4.2.x, 4.6.x, and 4.8.x, and gfortran 4.2.x and 4.8.x.
gfortran
Most Mac systems are not packaged with FORTRAN compilers. Check to see if you have gfortran by checking for the version in a terminal:
% gfortran --version
If you do not have gfortran, you can download it from www.macports.org .
GCC
-
For Xcode users: If you have Xcode with a version
between 3.1.4 and 4.6.3, you have a compatible version of GCC.
If you have Xcode 5.0 or higher, you may need to download GCC. Type "gcc" into a terminal. If you see "clang", like shown belowyou have an incompatible GCC compiler. Here are a few "how-to's" on installing GNU GCC for Xcode users via Macports or Homebrew.% gcc clang: error: no input files
- For those without Xcode:You can install a GNU version of gcc-4.2 using osx-gcc-installer or Homebrew.
Command Line Tools
You will need Command Line Tools (CLT), which allow you to run "make" commands from a terminal. To see if CLT are installed, type "make --version" into a terminal window:
% make --version
If you receive a message like "-bash: make: command not found," then you do not have CLT installed.
Users with Xcode <4.2.1 or >5.0 already have CLT installed.
If you have OS X 10.8 and Xcode 4.3 or above, you can download CLT through Xcode. Start-up Xcode and choose "Preferences." Click "Downloads" --> "Components" and then hit the "Install" button next to "Command Line Tools." It will ask you for your Apple ID.
If you do not have Xcode, you can get Command Line Tools from osx-gcc-installer, the Apple Developer site (free login required), or from the command line with:
% xcode-select --install
X11
To build CIAO 4.8 on a Mac, you need X11. From 10.8 and up, X11 was no longer included. Users had to download xQuartz, the X11 package for OS X, which installs in /opt/X11. To see if you have X11, you can type which xterm on the command line:
% which xterm /opt/X11/bin/xterm
If nothing is returned, you will need to download xQuartz.
Source files + OTS binaries
Building CIAO from source using the pre-built binary OTS requires the ciao-4.8-src-*.tar.gz packages as well as the platform specific ciao-4.8-bin-osxml64.tar.gz binaries. users should grab the osxml64 binaries. Currently we are not supporting source builds on El Capitan.
First, install CIAO using the ciao-install script. Go to the Downloading CIAO page. Under Custom Installation, choose your platform and check the "Standard CIAO installation" and "Full source build" boxes. This will select all of the necessary binaries (which contain the OTS) and will download the source code. CALDB, the background event files and the science scripts package are optional; include or exclude these packages by checking the boxes next to them. Finally, click the "Download the ciao-install Script" button.
From a terminal, run the ciao-install script. ciao-install will unpack all the files for you [recommended].
% bash <path-to>/ciao-install
If you want to unpack the files yourself, use the --download-only switch:
% bash <path-to>/ciao-install --download-only
and then unpack the tar.gz packages. For example, the commands for unpacking the OS X 10.8-10.9-compatible files from the "Standard CIAO Installation" are shown below.
% tar -xzf ciao-4.8-bin-chips-osxl64.tar.gz % tar -xzf ciao-4.8-bin-core-osxl64.tar.gz % tar -xzf ciao-4.8-bin-graphics-osxl64.tar.gz % tar -xzf ciao-4.8-bin-obsvis-osx64.tar.gz % tar -xzf ciao-4.8-bin-prism-osxl64.tar.gz % tar -xzf ciao-4.8-bin-sherpa-osxl64.tar.gz % tar -xzf ciao-4.8-bin-tools-osxl64.tar.gz % tar -xzf ciao-4.8-contrib-DEV.tar.gz % tar -xzf ciao-4.8-src-chips.tar.gz % tar -xzf ciao-4.8-src-core.tar.gz % tar -xzf ciao-4.8-src-obsvis.tar.gz % tar -xzf ciao-4.8-src-prism.tar.gz % tar -xzf ciao-4.8-src-sherpa.tar.gz % tar -xzf ciao-4.8-src-tools.tar.gz % tar -xzf caldb_4.7.0_main.tar.gz
Please note that the list of files to untar depends on the selections made when downloading the ciao-install script.
Unpacking caldb_4.7.0_main.tar.gz will create directories data/, docs/, and software/. You will need to create a CALDB directory and move data/, docs/ and software/ into CALDB/, then softlink CALDB to CIAO after the build.
> # in directory where caldb_4.7.0_main.tar.gz is > mkdir CALDB > mv docs/ data/ software/ CALDB/
Note: ciao-4.8-bin-obsvis-osxml64.tar.gz is NOT needed as no OTS are packaged in that binary distribution.
Note: If you are having trouble with the ciao-install script, refer to "Downloading the Software" in Manually Installing CIAO.
Build Preparation
-
Untarring the ciao-4.8-*.tar.gz files creates a directory ciao-4.8; cd into it:
You will make many references to the ciao-4.8 directory during the build configuration. Creating an environment variable for the path to ciao-4.8 will make the build commands easier:% cd ciao-4.8 % ls CALDB/ Linux_build.html VERSION_chips bin/ config.h.in data/ share/ COPYING* Mac_build.html VERSION_core binexe/ config.log doc/ src/ COPYING.LIB* Makefile VERSION_graphics ciao-type config.status* include/ test/ INSTALL_BINARY README VERSION_obsvis ciao_installed config.sub* install-sh* INSTALL_SOURCE README.obsvis VERSION_prism confdefs.h configure* lib/ LICENSE VERSION VERSION_sherpa config/ configure.in* ots/ LICENSE.SAO VERSION.prop_tk VERSION_tools config.guess* contrib/ param/
If you only want to rebuild certain directories of the CIAO source code, skip to step (3).% export ASCDS_INSTALL=`pwd` (for bash) % setenv ASCDS_INSTALL `pwd` (for tcsh) % echo $ASCDS_INSTALL /<path-to>/ciao-4.8
-
To make sure that libraries and executables are newly created by the
source build on your system, delete the contents of the lib/,
and binexe/ directories:
% rm -rf lib/* binexe/*
-
CIAO uses GTK for chips and prism. The necessary GTK files are packaged
with the CIAO graphics binary tarfile in the pkgconfig directory. We
discovered that combining the system and CIAO GTK produces a more stable
system than the CIAO GTK alone. Therefore, you must build with both the
system and the CIAO GTK pkgconfig.
pkgconfig is located in the X11 library. For users who installed xQuartz, X11
is located in /opt/X11/.
To include both the system and CIAO GTK, add the paths to both pkgconfig's in the environment variable $PKG_CONFIG_PATH:
% export PKG_CONFIG_PATH=${ASCDS_INSTALL}/ots/lib/pkgconfig:/opt/X11/lib/pkgconfig (for bash) % setenv PKG_CONFIG_PATH ${ASCDS_INSTALL}/ots/lib/pkgconfig:/opt/X11/lib/pkgconfig (for tcsh)
-
The tool pkg-config is used to find the libraries and header files needed to build parts of CIAO. Macs do not come with the pkg-config tool by default. The graphics binary distribution is packaged with pkg-config so that you can use it to build CIAO. You must add the path to pkg-config to your $PATH.
If using the Python packaged with CIAO (the default build), then prepend ${ASCDS_INSTALL}/ots/bin to $PATH:% export PATH=${PATH}:${ASCDS_INSTALL}/ots/bin (for bash) % setenv PATH ${PATH}:${ASCDS_INSTALL}/ots/bin (for tcsh)
% export PATH=${ASCDS_INSTALL}/ots/bin:${PATH} (for bash) % setenv PATH ${ASCDS_INSTALL}/ots/bin:${PATH} (for tcsh)
Build Configuration
Before building CIAO, we must configure the build script for our system. This involves choosing which compilers to use, setting library locations, and adding other options to the build. All of this is handled by a configure script.
Basic configuration
-
If you unpacked the source code and binaries by hand,
run the configuration script without any arguments before
continuing. This links the CIAO Python and other CIAO tools
to the CIAO build; skipping this step would result in a CIAO
environment without access to all the CIAO tools.
Otherwise, skip to Step 2.% mv src src.save % ./configure % mv src.save src % bash bin/ciao-python-fix
-
Start a CIAO session.
% . bin/ciao.sh (for bash) % source bin/ciao.csh (for tcsh)
-
Run the configuration script. The most basic configuration call to
have a successful CIAO build using the default build options is:
or, if using your system Python distribution:
% ciaorun ./configure --with-fits --with-ascii --x-includes=/opt/X11/include --x-libraries=/opt/X11/lib (using default build options)
The --with-fits and --with-ascii switches include the FITS and ASCII kernels when building the CXC DataModel, which are necessary to build CIAO. The arguments --x-includes= and --x-libraries= tell the system where to find the X11 library and include files.% ./configure --with-fits --with-ascii --x-includes=/opt/X11/include --x-libraries=/opt/X11/lib (using system Python)
Additional arguments, described in the following subsections, may be used in the configure call to further specify the configuration.
If configure fails, see the Troubleshooting section. A likely cause of failure is the environment variables $PATH and $PKG_CONFIG_PATH are not properly set, or that there are multiple compilers on your $PATH and configure is picking-up the wrong compiler(s).
The following subsections contain details on setting the compilers, the configuration options, and compiler optimization levels.
Setting the compilers
Specifying a specific path to a compiler can be achieved by using the appropriate keyword:
- CC - Name and location of the c compiler
- CXX - Name and location of c++ compiler
- F77 - Name and location of FORTRAN compiler
- FC - Name and location of FORTRAN compiler
Usage example:
% ciaorun ./configure --with-ascii --with-fits \ FC=/opt/local/bin/gfortran-mp-4.8 \ F77=/opt/local/bin/gfortran-mp-4.8 \ CC=/opt/local/bin/gcc-mp-4.8 \ CXX=/opt/local/bin/g++=mp-4.8
(Exclude ciaorun from the command if building with your system Python.)
To specify the FORTRAN compiler, the FC and F77 variables must be the same.
The compiler must also be in your $PATH. If <path-to-compiler> is not in $PATH, add it using
% export PATH=${PATH}:<path-to-compiler> (for bash) % setenv PATH ${PATH}:<path-to-compiler> (for tcsh)
Configuration flags/options
Here is the list of configure options available for CIAO:
Option | Description |
---|---|
--prefix=<location> | Directory to install CIAO. Default location will be '.' |
--x-includes=<DIR> | directory DIR in which the X include files are located |
--x-libraries=<DIR> | directory DIR in which the X library files are located |
--with-debug | Compile with debug mode enabled |
--with-opt=<level> | Set optimization level (default=-O3) |
--with-fits |
Include FITS kernel when building CXC DataModel.
Note: This switch is required to build the DataModel. It indicates that the setup is for a compile and not just a binary install. |
--with-ascii |
Include the ASCII kernel when building the CXC
DataModel.
Note: This switch is required if you need to read ASCII files into ChIPS. |
--with-f95loc=<DIR> | Location of libf95 (if using g95) |
--with-gfortranloc=<DIR> | Location of libgfortran (if using gfortran) Unnecessary if gfortran and libgfortran are in the same tree (e.g. if gfortran is in /usr/bin, and libgfortran is in /usr/lib, they’re in the same tree) |
--with-imagerdir=<saord> | Path to directory/tree containing DS9 |
The flags --with-ascii and --with-fits are REQUIRED for CIAO to build correctly. All other flags are optional.
Optimization and compilers
Optimization is turned on by default, -O3 for GCC. If you wish to use a different optimization level or turn off optimization, use the --with-opt= switch. To turn off optimization, use --with-opt=-O0. See GNU’s Optimization Options for more details.
Build Instructions
Now, the source tree should be properly set up to build. The directions here are for rebuilding the entire CIAO source code (Rebuilding CIAO), or for rebuilding a selected directory (Rebuilding selected source code).
Rebuilding CIAO
To build and install the code, use "make install". CIAO currently requires that "make install" be issued as a single command rather than as separate commands "make; make test; make install". This is due to build dependencies on installed libraries and packages.
% ciaorun make install
Note that this will take a few minutes.
Rebuilding selected source code
There are a few tools that depend on local libraries that if not installed may require rebuilding first. Most src directories can be rebuilt after running configure by simply stepping into the directory and running "make install".
For example, say a user rebuilt libfftw to take advantage of their specific hardware. After placing the libfftw files in $ASCDS_INSTALL/ots/lib, they’d need to re-link the tools in $ASCDS_INSTALL/src/da to their system by doing
% cd ${ASCDS_INSTALL}/src/da % ciaorun make install (using default build options) % make install (using system Python)
to rebuild the da tools with their custom libfftw build.
Post-processing
After the build completes, run the script ciao-python-fix:
% bash src/config/ciao-python-fix
This post-processes your configuration, to byte-compile included Python files for your installation, and to ensure all CIAO packages with Python interfaces can be built. Once this script is run, the CIAO installation cannot be moved or relocated to a different directory without repeating the configure and ciao-python-fix steps.
Linking to CALDB: if you downloaded CALDB and unpacked the tarfile by hand (e.g. ran ciao-install with the --download-only switch), you can link it to CIAO at this point by creating a softlink to the CALDB directory in $ASCDS_INSTALL:
% pwd /<path-to>/ciao-4.8 % ln -s <path-to>/CALDB CALDB
AHELP Indexing
After installation you will need to generate an index file used by the CIAO help documentation application (ahelp). The ciao.*sh script needs to run to set up dynamic library paths used by ahelp.
% . bin/ciao.sh -o (for bash) % source bin/ciao.csh -o (for tcsh)
Then run the ahelp command with the -r (re-index) option.
% ahelp -r
This will generate the help indexes. If this completes successfully, you can execute the smoke tests.
Smoke Tests
To run the tests, you should do something like:
% cd test % . ../bin/ciao.sh (Note: Only needed if you have not setup for CIAO in a prior step) % make -k test 2>&1 | tee ~/ciao_test.log (sh, ksh, bash)
or
% cd test % source ../bin/ciao.csh (Note: Only needed if you have not setup for CIAO in a prior step) % make -k test |& tee ~/ciao_test.log (tcsh)
A successful test will look like:
[1/42] Running test FOO-smoke001 ... PASS
where 'FOO' is the name of an individual test. If you see a message that reports:
[1/42] Running test FOO-smoke001 ... FAIL
then there is an unexpected problem with the tests. Note that if you see additional warnings/errors and the test reports PASS, then the test is good (tests sometimes do test error conditions).
All the test outputs are written to $ASCDS_WORK_PATH/smoke.$LOGNAME where $LOGNAME is your system login name. The tests need about 60 Mb to run. The test scripts clean-up the disk space upon successful completion. Tests which fail leave their test output so you may check the results. When all of the tests PASS, the smoke test scripts should remove $ASCDS_WORK_PATH/smoke.$LOGNAME and all its sub-directories.
$ASCDS_WORK_PATH is set in the $ASCDS_INSTALL/bin/ciao.*sh setup scripts as /tmp by default. If the machine does not have a /tmp or you don't have write permission, you will need to edit the appropriate ciao.*sh script to change the $ASCDS_WORK_PATH location or set the $ASCDS_WORK_PATH environment variable. If this variable is set, ciao.*sh will not modify it.
If a test FAILS, you may want to double-check that you’re using the right versions of GCC and gfortran, and that your environment variables $PATH and $PKG_CONFIG_PATH are set correctly. Also check Bugs: Installation & Smoke Tests for other documented problems and solutions; if you run into any other problems, please contact the CXC Helpdesk. If you decide to re-run the build process, repeat the configuration and building process in a new terminal to prevent the old environment from affecting the new build.
Troubleshooting
Below are some common problems you may run into while building and installing CIAO. You can also check Bugs: Installation & Smoke Tests for other documented problems and solutions. If you run into any other problems, please contact the CXC Helpdesk.
Configure script
-
Building CIAO with the Clang compiler is not supported. If you are
having trouble configuring CIAO, type "gcc" into the
terminal. If you see
then you are using an incompatible GCC compiler. Please refer to the GCC section in Compilers for information on installing the right compiler for your system.
% gcc clang: error: no input files
-
Error "cfitsio is not found"
checking for pkg-config... no configure: error: cfitsio is not found. Please specify the path using '--with-cfitsio=<<path>' or 'setenv PKG_CONFIG_PATH <path>/lib/pkgconfig'
This error shows that the $PKG_CONFIG_PATH variable is not set correctly. $PKG_CONFIG_PATH should contain% export PKG_CONFIG_PATH=${ASCDS_INSTALL}/ots/lib/pkgconfig:/opt/X11/lib/pkgconfig (for bash) % setenv PKG_CONFIG_PATH ${ASCDS_INSTALL}/ots/lib/pkgconfig:/opt/X11/lib/pkgconfig (for tcsh)
-
For other errors, check that:
- your compilers are in your $PATH
- $PKG_CONFIG_PATH is set correctly
- ${ASCDS_INSTALL}/ots/bin is in your $PATH
Build
If the build fails at any point during "make install", check that
- you have compatible GCC and gfortran compilers (see Compilers for details)
- your compilers are in your $PATH.
- ${ASCDS_INSTALL}/ots/bin is in your $PATH.
- $PKG_CONFIG_PATH is set correctly
Then run
% make -k clean
remove the contents of lib/ and binexe/ again,
% rm -rf binexe/* lib/*
re-configure the build with
% . bin/ciao.sh -o (for bash) % source bin/ciao.csh -o (for tcsh) % ciaorun ./configure --with-fits --with-ascii --x-includes=/opt/X11/include --x-libraries=/opt/X11/lib (Exclude "ciaorun" from the command if building with your system Python.)
and run make install again. If it still fails, try opening-up a new terminal so that you have a clean environment to work with. Check that you have compatible compilers and that you’ve set $PKG_CONFIG_PATH and your $PATH variables correctly defined.
History
08 Dec 2014 | First issue. Merges INSTALL_SOURCE text file into CIAO website. |
30 11 2015 | Updated instructions for ciao 4.8 release |