How to install the NEOS server 4.0?

These are the instructions for installing a generic NEOS server, which you can use for many purposes.

WARNING: These instructions guide you through a successul installation of NEOS itself, but stop there. NEOS is not usable without adding one or more solvers, and for that you have to develop one XML config file and one wrapper script each. These are not provided within the NEOS server software package, not even for open source solvers like COIN-OR CBC. For that reason, I finally abandoned NEOS and moved to the 12 years newer and much cleaner COIN-OR Optimization Services (OS) alternative solution. (The last NEOS 4.0 release is from 2002-10-25, while COIN-OR OS is in active development and meant to be the "next-generation NEOS server" [source].) Also NEOS turned out to be really messy, old Perl software with confusing and inconsistent documentation, so that it is a pain to work with it. You have been warned. Proceed at your own risk 🙂

You can test the NEOS interface by using the public NEOS server, as we did too. So installing your own NEOS server is only reasonable when you want to avoid overloading their servers and / or prohibit all your job submissions from becoming public, as happens with the public NEOS server.

Choosing a machine for your server

For our purposes, using Amazon EC2 Spot Instances seems the most economic and scalable solution. For Economy App, we are starting with one of the CPU power optimized c1.medium instances. It is available more or less permanently for 0.028 USD/h in the South America data center (sa-east-1) – that's 20 USD/month when running full-time, while we only need it to run 25% of the time at first.

As our userbase will grow, we will quickly need a lot or calculation power, and then have to care for where to get it economically. And there's lots of optimization potential – see the initial CloudHarmony cloud hosting comparisons, and their current cloud hosting benchmarks for example.

Installing the NEOS server 4.0

Here we are exploring how to install it on a Linux server (Ubuntu 10.13, to be exact), based on the original installation instructions. It is assumed that your web server and the NEOS server will run on the same machine (else, see this hint). We also assume that you start with only one server – NEOS can handle multiple workstations running solvers though [source].

  1. Download NEOS. Download and extract one of the NEOS server packages. The newst is NEOS server 4.0 from 2002-10-25. (Except you want to go for NEOS server 5.0 from 2005, which however appears largely undocumented.) So for example, do:
    cd /var/local/;
    sudo wget ftp://ftp.mcs.anl.gov/pub/neos/Server/server-4.0-102502.tar.gz;
    sudo tar -xzf server-4.0-102502.tar.gz;
    sudo mv server-4.0 neos;
    sudo rm server-4.0-102502.tar.gz;
  2. Provide the directory structure. You have to create web-accessible directories for CGI files and public web pages and a kind of NEOS workspace directory [source]. The latter is called "directory for variable-length files", and if web-accessible, will make all job submissions to your NEOS server public [source]. In our case, we create these directories below the NEOS installation base dir (originally server-4.0). The workspace dir and document root dir are siblings, so job submissions will not be public.
    sudo mkdir /var/local/neos/cgi-bin/ /var/local/neos/htdocs/ /var/local/neos/workspace/;
  3. Adjust file ownership. Change file ownership rights to the user who will install and run the NEOS server. For our Ubuntu 13.10 AMI running as an Amazon EC2 Spot Instance, we use for example:
    sudo chown -R ubuntu:ubuntu /var/local/neos;
  4. Provide Mail. The NEOS Makefile expects a program Mail (yes, caps M), which in old times was used for an extended version of mail [source]. Let's at least provide mail as a surrogate and see if that works:
    apt-get install mail && sudo ln -s $(which mail) /usr/local/bin/Mail
  5. Configure NEOS. Execute make and answer its config questions according to the NEOS FAQ instructions:
    cd /var/local/neos/config && make
  6. Configure crash recovery. Configure NEOS so it will be automatically restarted via cron when it crashes. For that, execute the following command as the user who should run the NEOS server (but if crontab -l shows you have a crontab already, edit it manually since the following command would replace it): crontab /usr/local/neos/crontabfile.
  7. Install legacy libs. The NEOS server restart script uses flush.pl, a legacy Perl library. This creates a warning about being deprecated and imminent removal from the next major release, so we resolve that warning by installing the library which will provide it in the future: sudo apt-get install libperl4-corelibs-perl.
  8. Start your NEOS server.
    /var/local/neos/bin/restart
  9. Check that the NEOS server is running. Just check in ps aux output that the above command started the initializer.pl, scheduler.pl and socket-server.pl daemons successfully [source].
  10. Confirm that the XML-RPC interface to your server works. By default it resides at port 3333, you configured it with make above. For that, just visit that URL with a browser (something like http://example.com:3333/ ). If it works, the NEOS server will server you a message like this:
    121
    your-server-name: ERROR: "Host: example.com:3333" is not recognized by this server.
  11. Install and configure a web server. It will serve the NEOS web interface. In this case, we use lighttpd.
    1. Install lighttpd.
      sudo apt-get install lighttpd
    2. Enable CGI. Enable the CGI module that you will use to serve the NEOS Perl application:
      sudo lighty-enable-mod cgi
    3. Don't serve CGI files as source! A little security optimization for the NEOS server files: in /etc/lighttpd/lighttpd.conf, change this line
      static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
      to this
      static-file.exclude-extensions = ( ".php", ".pl", ".fcgi", ".cgi" )
    4. Add a configuration section for the NEOS server. This goes to /etc/lighttpd/lighttpd.conf (in vhost style though it's our only site).
      $HTTP[“host”]  =~ "example.com" {
        server.document-root = "/var/local/neos/htdocs"
        alias.url += ( "/cgi-bin" => "/var/local/neos/cgi-bin/" )
        $HTTP[“url”] =~ "^/cgi-bin" {
            cgi.assign = ( ".cgi" => "" )
        }
      }

      [References: nixCraft lighttpd setup for Perl programs; lighttpd Configuration File Options reference.]
    5. Reload the lighttpd configuration. Execute: sudo service lighttpd force-reload
    6. Adjust file permissions. We need to give the web server user (by default www-data for lighttpd) read and write permissions to some directories:
      cd /var/local/neos;
      sudo chown -R :www-data .;
      sudo chmod g+r+w htdocs/jobs workspace/spool/WEB workspace/databases workspace/tmp
      ;
    7. Test the web interface. Make sure that the NEOS web interface works by visiting the base URL in a browser. It would be http://example.com/ for the above case. Also visit http://exampel.com/cgi-bin/check-status.cgi to make sure your CGI config works.
  12. Configure your system so that the NEOS server will be started at system start. This is especially relevant when using an Amazon EC2 spot instance, since these are occasionally shut down and restarted depending on your bid pricing (and yes, this means system shutdown, not virtual machine hibernating). For that add the following to the content of /etc/rc.local (and disable the default exit 0 if still present):
    /var/local/neos/bin/restart

Registering solvers at your NEOS server

Your NEOS server 4.0 is installed now, but so far it does not have any knowledge about what solvers are available and how to contact them. The NEOS Solvers FAQ explains the process of registering solvers; see also this short list of steps and the HelloNEOS example.

The problem is that the configurations for registering all the solvers of the public NEOS server are all missing – not included in the NEOS server software as they are considered parts of the installation. You would have to write a complex XML file and a wrapper script to (for example) make your NEOS server talk to the free software COIN-OR CBC solver.  In the time needed for this, you can just as well install the alternative solution COIN-OR OS, which comes already pre-configured for being used with CBC. Which is why I abandoned using NEOS at this point.

Installing a solver and making it talk to your NEOS server

At this point your NEOS server knows how to talk to the CBC solver, but neither is your CBC solver installed nor did we add anything to let it know how to talk back to the NEOS server. So let's do both of this now.

(Note: You might see the neos-comms packages for download. However, do not use these. Reasoning: These files are from 1998, while code with equivalent functionality is available inside the latest NEOS 4.0 2002-10-25 server package. That code is clearly newer and better, with comments and all. The neos-comms package contained a neos-comms-4.01/bin/client file that is not available in the new code, but seemingly is just an added start script for the comms daemon and Tcl/Tk interface that got now integrated into the daemon and Tcl/Tk application themselves.)

  1. Start the comms daemon. This is the "Communications daemon" which has to run on the solver workstation (same machine in our case) and mediates between the NEOS server on one side and the solver (here CBC) on the other. It can be started with the "Communications GUI", available as a Tcl/Tk application within the NEOS server software installation. For details, see comms-help.txt (also available within the equivalent place in your NEOS server installation).

Links to NEOS documentation


Posted

in

,

by

Tags:

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.