How to install COIN-OR?

We needed a fast, dedicated server to solve optimization problems. So let’s install COIN-OR CBC (the MLP optimization solver we use). We start from a stock Ubuntu 13.10 machine, as it will happen to you when deploying on Amazon EC2 Spot Instances, like we do.

The easiest way to get COIN-OR OS and CBC installed on a common Linux platform is to install the “full release” binary packages, which contain the binaries from all major COIN-OR projects. Just download the latest COIN-OR-*.tgz package from the CoinAll repo, unpack it, and link the files in its bin/ subdirectory into your path with something like ln -s bin/* /usr/local/bin/.

However, we want to install from source here, which gives us some advantages (mostly following reasoning found here):

• The most important reason is that COIN-OR cannot distribute GPL-licenced source or binaries with its code. It includes download scripts for these however. So to get goodies like ASL (the AMPL input format library we desperately need for CBC here), GMPL, command completion, command history, Haskell libraries etc. one has to install from source.
• The installation uses the Debian package management system for simple modifications, updates and removal. (Some COIN-OR packages are available in the Ubuntu archives, but not all from the COIN-OR distribution – not OS for example, which we need here. Also these packages tend to be a bit outdated.)
• We can optimize COIN-OR to our needs with the many compilation options of COIN-OR. For example, enabling multi-threading for the CBC solver, or optimizatios for specific CPUs / hardware.
• Bug fixes can be incorporated immediately instead of waiting for a new COIN-OR release.

How to install all major COIN-OR packages from source, according to the CoinAll distribution build notes and the guidlines from the T. K. Ralphs presentation:

1. Install some needed developer tools. Note that csh is needed for DyLP/doc/build_dylpdoc, which is called in make distclean for example.
   sudo apt-get install patch make pkg-config wget subversion checkinstall csh gcc g++ gfortran
2. Install some libraries needed by some COIN-OR packages. (matplotlib is needed for the GiMPy tests to complete.)
   sudo apt-get install python-matplotlib
3. Download the latest stable version. This probably will get you the latest maintenance release, so probably currently 1.7.4 as seen in the CoinAll list of binaries.
   svn co http://projects.coin-or.org/svn/CoinBinary/CoinAll/stable/1.7 CoinAll-1.7
4. Add necessary and desired third-party source code. Some code could not be included into the COIN-OR distribution because of licence incompatibility (for example with the GPL). But COIN-OR provides nice download scripts to add these components:
   1. Add the MUMPS public domain linear solver. If present, it will be used in the compilation of Ipopt. If not present, Ipopt will be compiled without solvers, which will cause make test fail for Ipopt with a message like “selected linear solver ma27 not available” [source].
      cd ~/CoinAll-1.7/ThirdParty/Mumps
./get.Mumps
   2. Add Glpk. This is needed by CBC in order to read GMPL input – that’s exactly what we need, since it is the only way to use an algebraic modeling language for comfortable, high-level input to CBC. GMPL (also called MathProg) is actually a subset of the widespread AMPL [source], so many models in AMPL should also be expressable with it.
      cd ~/CoinAll-1.7/ThirdParty/Glpk
./get.Glpk
   3. Add the ASL library. Without this being present, CBC would be built without AMPL .nl input format support – trying to call it with a .nl file would then result in an error “nl file specified locally but ASL not present”. But we want to have the nice simple AMPL input format. So to get it included, we do according to the CBC FAQ:
      cd ~/CoinAll-1.7/ThirdParty/ASL
./get.ASL
   4. Add BLAS. Adding Blas (andLapack, see below) to CBC gives better numerical stability to CBC’s LP solver [source, p.5] – whatever that means. Blas is also needed for compiling one of the other COIN-OR projects; it can be added with apt-get install libblas3 libblas-doc libblas-dev but let’s keep here with the mechanism supplied by COIN-OR to get the latest version:
      cd ~/CoinAll-1.7/ThirdParty/Blas
./get.Blas
   5. Add Lapack. Second component in addition to Blas for improving the numerical stability of CBC. To add it:
      cd ~/CoinAll-1.7/ThirdParty/Lapack
./get.Lapack
   6. (optional) Add METIS. METIS is a library for graph partitioning, and I don’t know if CBC can profit from it. But since it’s the last of the ThirdParty packages that are free to use for all, let’s go ahead:
      cd ~/CoinAll-1.7/ThirdParty/Metis
./get.Metis
5. Configure it all. In the configure command below, the LDFLAGS=-Wl,--no-as-needed; is to solve problems caused probably by some library symbols now being versioned in current versions of Ubuntu. These problems show up with error messages saying “undefined reference to dlopen, dlclose, dlerror, dlsym”. The LDFLAGS setting is a solution from the ipopt mailing list, and while it works for me I don’t quite understand it. It is only needed for ./configure in the sub-projects OS, DyLP and IPopt. But it does not hurt to apply it globally, so for simplicity we do just that. Also, we configure CBC to support parallel execution. This is done by adding the --enable-cbc-parallel compile-time option [source]. Again, it does not hurt to add it to the global ./configure, it will only affect CBC (and should result in a line “Cbc multithreading enabled” in build/Cbc/configure.log).
   mkdir ~/CoinAll-1.7/build;
cd ~/CoinAll-1.7/build/;
../configure --enable-gnu-packages --enable-cbc-parallel -C --prefix=/usr/local/share/coin-or LDFLAGS=-Wl,--no-as-needed;
6. Compile it all.
   cd ~/CoinAll-1.7/build/;
make -j 2;
7. Adapt the tests for a headless server. Some tests will fail without an X Server. We run this on a headless server, so we have to modify the tests (or use Xvfb, but that is a hack as well …).
   1. In ~/CoinAll-1.7/GiMPy/test/test_algorithms.py, remove line produce_graphs() (actually the very last one).
   2. Solve a similar problem with ~/CoinAll-1.7/GrUMPy/test/test_bb.py.
   3. TODO There may be more issues like this. I did not finish trying to get all tests to run.
8. Run the tests. (Might still fail at some point when used on a headless server, as per the last point. But you will get through with most at least.)
   make test;
9. Create a directory for the installation.
   sudo mkdir /usr/local/share/coin-or
10. Create a Debian package and install it. Installing is included in the following command if all works well. In total, it can take a while, 20 minutes even, depending on your machine. I named mine coinor-coinall.
    sudo checkinstall make install;
11. Put the binaries into the path. Because this is not done by the installation. Let’s use symlinks:
    cd /usr/local/bin && sudo ln -s ../share/coin-or/bin/* .

TODO The packages built this way can be installed on any Ubuntu 13.10, 14.04 or perhaps newer version, but they miss to list some dependencies, so these should be added manually in the “checkinstall make install” step. Until then, make sure to provide the packages yourself with:

sudo apt-get install gfortran