The idea explored here is to separate your data physically from the computer that you are using them on. This has the following advantages: (1) you are quite safe from theft because you can keep carrying your data with you always, even when having to let your notebook at home or in a hotel because it is too heavy; (2) you can use your data on any computer (like a friend's computer when travelling, an Internet cafe computer etc.); (3) you can hand over data items quickly and easily because every computer has a USB interface.

You have to decide if the this idea is worth its benefits for you, as all solutions below are much slower and more expensive than a bigger standard SATA III SSD (as of 2014-06, about 100 EUR new incl. shipment for a 256 GB SSD, with data transfer rates up to 600 MB/s, though the continuous transfer rates have to the researched).

However, here are the solutions I could come up with. By adequacy for my requirements – which included to be 128 GB or more in size:

  • SD card in SD card reader. This is the bes option on notebooks that come with a card reader, since it does not block any ExpressCard expansion port (reading your digital camera card is easily possible in the USB ports as well with SanDisk Ultra II SD Plus USB cards). However, it has to be tested still if the typical notebook card reader can utilize the speed of highest speed SD cards used here (else a ExpressCard/34 based card reader would be needed). Because for proper speed, we need the latest "U3" class SD cards. These are just being introduced around 2014-06, so prices will drop lateron. Currently available products:
  • SD card in ExpressCard/34 card reader. For data exchange, all readers can expose a SD card as a USB mass storage device, so no drivers on the host system are needed. The additional advantage of using a SD card is that you can also use them for other purposes (recording photos and HD video etc.). For proper speed, we need the U3 speed class cards again, see above.
  • Compact Flash card in ExpressCard/54 card reader. This is a good solution, since the speeds of Comact Flash cards are comparable to the ExpressCard SSDs while the Compact Flash format makes it simpler to find an adapter to USB mass storage (namely, a simple card reader). However, Compact Flash does not fit by width into ExpressCard/34, so your notebook will need the larger ExpressCard/54 slot [see]. Speeds of Compact Flash are great, and for data exchange you can carry a USB card reader that exposes the Compac Flash card as USB mass storage (means it needs no driver). But high-speed, large-capacity cards are also expensive; recommendable products:
    • A moderately priced one is the Hama CF 128 GB (up to 150 MB/s read, so maybe 40 MB/s continuous write?) for about 145 EUR.
  • ExpressCard/34 form factor SSD, plus USB adapter. This is the best option in terms of speed. It can be connected to other computers using a USB to ExpressCard adapter (overview; also see DeLock 61714 etc.). ExpressCard allows cards to use an USB 2.0, USB 3.0 or PCIe mode, and not all these adapters also support PCIe (the Digigear PCU10 does however, but needs manual configuration for PCIe mode). So check first which mode is used by the ExpressCard SSD – anything above 60 MB/s means USB 3.0 or PCIe mode. For the connectors etc., see the ExpressCard standards document.
    Available products 64 GB and larger:​

     

  • Small "knobby" USB flash drive. If your notebook has a USB 3.0 port or you can add one via an ExpressCard/34 slot or similar, the achievable speed of this solution can be good, too. It seems that there is still no 128 GB flash drive in "knob" size. A hack to get over this would be to combine two of them physically so that they fit exactly into two adjacent USB ports on your notebook (and with a small cable-type USB hub to other computers' USB ports). Available options:
  • microSD card in "knobby" microSD card reader. This option works even up to 128 GB (with 128 GB microSD cards being available). However, these cards tend to be slow. Available models: just one so far
  • Compact Flash card in 1.8" or 2.5" SATA drive bay. This is essentially using a Compact Flash card as a removable, rugged SSD. There are Compact Flash to SATA adapters for both 1.8" and 2.5" SATA SSD form factor (example: StarGat SAT2C). Also this solution will work only if your notebook can be modified in a way to allow drawing out the card from the HDD bay. And it's not really clear how the computer or operating system might react when extracting the Compact Flash card for example during ACPI Suspend mode. Also when using the system disk bay for this type of SSD, we cannot physically separate operating system and personal files as intended here.
  • miniPCIe SSD in ExpressCard and SATA external enclosure. This is more a theoretical option since you do not want to handle a bare PCB with your data on it just to connect it to a computer that does not have an ExpressCard slot. But it would work, as there are converters for miniPCIe to ExpressCard and miniPCIe to SATA.

If you are using Plesk as a server control panel and took part in all updates starting with some Plesk 10 version, you will notice that your recent version (Plesk 11.5 currently) has a total mess with respect to the web hosting directory layout. That's because during some of these updates, Plesk migrated its hosting directory structure, and now the old and new ones are mixed.

The new Plesk web hosting directory layout

Say you have two subscriptions (subscribers being the same or different customers, or even you as an admin). Each subscription has one subdomain and one additional domain. Then the web hosting directory layout would be like this (using placeholder domain names here):

  • /var/www/vhosts/subscr1-maindomain.com/subscr1-maindomain.com/
  • /var/www/vhosts/subscr1-maindomain.com/sub.subscr1-domain.com/
  • /var/www/vhosts/subscr1-maindomain.com/subscr1-otherdomain.com/
  • /var/www/vhosts/subscr2-maindomain.com/subscr2-maindomain.com/
  • /var/www/vhosts/subscr2-maindomain.com/sub.subscr2-domain/
  • /var/www/vhosts/subscr2-maindomain.com/subscr2-otherdomain.com/

Plus, in /var/www/vhosts/system/ there will be a directory for each domain and subdomain, named with its domain name, and containing subdirectories conf/, logs/ and statistics/.

Recommendation for using this directory layout

One subscription per (sub)domain. We want a directory structure that is logical and consistent, so it's not acceptable to find directories for some but not all domains inside /var/www/vhosts/. The only solution I could come up with so far is creating a new subscription for every single domain and subdomain that you want to host. And then to name the document root directory for hosting the main domain of that subscription always htdocs rather than with the domain name.

Problem: if you have a domain-restricted Plesk licence. With this method, you will run into problem depending on your Plesk licence. For example, if you have the typical "10 domains" licence. Because in this setup, every subdomain is created as its own Plesk domain record, so taking up one of the 10 available domains. While creating Plesk subdomain records for cases where their parent domain is also hosted in Plesk would not take up anything from your licence. But in my view: cleanliness first 🙂 And it's another reason to move to a recommendable, free and open source web hosting control panel.

Removing old, legacy elements. The following elements of the directory structure are probably legacy ones from the earlier Plesk versions and can be safely removed. However, I did not test this, so rename them first and check if everything still works before removing them:

  • /var/www/vhosts/example.com/subdomains/
  • /var/www/vhosts/example.com/conf/ (a symbolic link)
  • /var/www/vhosts/sub.example.com/ (in cases where this contains only conf/, logs/, statistics/ for a domain or subdomain that is an add-on domain for a subscription with another main domain, and has its document root inside that main domain's directory)
  • /var/www/vhosts/httpsdocs/ (probably not needed, as the new place seems to be /var/www/vhosts/default/httpsdocs/)

This is my list of Android applications that I recommend everyone to have and install on own phones and phones prepared for clients. Of course everybody will need more apps for specialized purposes, but these ones should be relevant for every user.

Selection criteria for this list:

  • only one selected best app for every purpose
  • any functional open source alternative is preferred to a polished proprietary software; proprietary licences are marked in red to indicate looking for a free & open replacement still
  • decentralized open communication protocols (e-mail, XMPP, SIP) are preferred to commercial centralized applications (Skype, Facebook)
  • communication apps should be privacy-enabled
  • the app selection should enable self organization and group organization, esp. for grassroots movements like social change activists

The "state" column contains my personal installation state info – just ignore 😉

state name FDroid Google Play description
notes
licence
x 3G/4G Speed Optimizer   Play Allows settings for faster 3G download and upload speeds, and for adjusting distribution between both. freeware
x 4EXT Recovery Control n/a Play For managing the 4EXT Recovery mode from inside an app. 4EXT Updater is a simpler, gratis alternative. commercial
x AdAway F-Droid   Removes all the pesky in-app advertisements. GPLv3
  Advanced Task Killer Free        
  ADW        
x Alarm Klock F-Droid   Full-featured alarm clock to help you wake up. Apache2
  aLogcat        
  AnaCam        
  AndroSS        
  AndStatus        
x Angulo F-Droid   For measuring angles and distances, also useable as a water-level. GPLv3
  AnyMemo        
  AnySoftKeyboard        
x APV PDF Viewer F-Droid     GPLv3
x Arity F-Droid   Scientific calculator. Apache2
x Barcode Scanner F-Droid   Complete barcode and QR code scanning app, employed by other apps as well. Apache2
  BLUE LINE        
n/y BlueGps F-Droid n/a Allows to connect an external GPS device and make it usable in e.g. OsmAnd~ via the "mock location provider" option. Means faster, more precise and higher coverage GPS, longer battery runtime, and allows placing the GPS receiver outside the car. To be evaluated. GPLv3
x Bluetooth GPS n/a Play Same as BlueGps, but non-free. Good quality though. freeware
  Busybox   Play Installer for Busybox, a boiled down version of GNU Core Utils. Needed for many other root apps, incl. 4EXT Recovery Updater. GPL 2
  CamPainter Lite        
x ChatSecure F-Droid   Encrypted XMPP chat client, including off-the-record features. Apache2
x Clock n/a n/a This app (com.android.deskclock) comes pre-installed with Android and is well done, including clock, alarm clock, countdown clock and stop watch. Only the alarm clock misses some features, so we also have Alarm Klock. open source
  ColorDict        
  Compass        
  ConnectBot        
  Cool Reader        
  Diktofon        
x Evernote   Play Note taking and organizing system that syncs to desktop and web. freeware
  Every Locale        
x F-Droid F-Droid   Package manager / marketplace for all the Android open source software recommended here. GPLv3+
  Fake Dawn        
  FAST App Search Tool        
x Faster GPS F-Droid   Enables getting faster GPS fixes by selecting a NTP server near you. GPLv3
x Firefox F-Droid   Web browser, mobile edition. MPL2
  FolderSync        
  Freemind Viewer        
x GetBack GPS F-Droid   Finding a way back to a "home location" with GPS guidance. Hopefully better than Point, which can be inaccurate. GPLv3+
  Google Play services        
x Google Play Store        
x K-9 Mail F-Droid   Full-featured e-mail client. Apache2
  KeepTrack        
  Ki Freemind        
  Maniana        
x Mathdroid F-Droid   Handy calculator with history. GPLv3
  MobileOrg        
  MobileWebCam        
x MrWhite F-Droid      
x Network Discovery F-Droid   A network utility: discover hosts inside a network you are connected to, and open ports on these hosts. GPLv2
  Notification Plus        
  Obsqr        
  Offline Calendar        
  OS Monitor no-gmaps        
x Osmand~ F-Droid   Super great offline map viewing and routing application. Uses OpenStreetMap data. GPLv3
x OSMTracker F-Droid   GPX tracker that allows note-taking, esp. also meant as a help for OpenStreetMap mapping work. GPLv3
x Pedometer F-Droid     GPLv3
  Blockchain        
  QuickDic        
  Remuco        
  Ruler        
x SatStat F-Droid   GPS, sensor and network status information. So for most purposes, a more low-level graphing app like Sensor Readout is not needed. GPLv3+
  Sca Lite        
  Search based launcher        
  Search Light F-Droid   Flashlight using the camera LEDs. Screen is on at the same time, so MrWhite is also installed as a less energy hungry alternative.  
  Speech Trainer        
  Super Ruler Free        
  TallyBee        
  Tallyphant        
  Terminal Emulator        
  Theme Chooser Themes        
  Timesheet        
  Units        
  Vanilla Music        
  Who Has My Stuff?        
  WiGLE Wifi Wardriving        
  WikiDroyd        
  Wikipedia        
x Word Lens Translator n/a Play Travel app that translates words recognized in the live camera view between five languages and English. All happens offline, no network connection is used for anything. commercial
n/y WordPress F-Droid     GPL
  YAAB        
  YouTube        

I am still looking for apps for:

  • SMS backup application (there is SMS Backup+, but it cannot save to SD card)
  • accessing microblogging services incl. Identi.ca and Twitter
  • App for power saving while travelling: switch mobile network off (airplane mode) while the phone moves by car or train, as determined by GPS from speed and location. And only check every 30 min for missed calls.
  • Find a powerful, high usability text editor for Android. Ideally with hierarchical mode that is compatible with Freemind's double-indent text exports.
  • Find or develop an app that unifies all possible action on clipboard content, can be called from every app and stores clipboard history, and analyzes clipboard content and history to suggest relevant actions (like "send wire payment to").

Usually, you can have mentions with e-mail notifications using the Drupal mentions module, or mentions with auto-suggest using the Drupal ckeditor_mentions module, but not both. But it turns out you can, indeed, by combining both modules (see also). Here are detailed instructions how to set thus up.

Recommended: Using Message Notify for message sending

  1. Install and enable required modules: rules, mentions, ckeditor_mentions, message_notify.
  2. Configure the mentions module (in /admin/config/content/mentions) to  use "Input: Prefix: @" and "Input: Suffix: " (nothing). This corresponds to how usernames are displayed after letting ckeditor_mentions insert a suggestion.
  3. Make sure your usernames are set up to not contain spaces, because the @username format does not allow usernames with spaces to be recognized.
  4. Enable permission "Bypass Rules access control" for your user's user group, or you will get a "Access violation! You have insufficient access permissions to edit this configuration." error when trying to create the "mention:entity of type …" Rule condition below.
  5. In the admin menu, go to "Structure -> Message types" (/admin/structure/messages) and create a new message type. It is simplest to clone it from one that is meant for use in e-mails already, such as any Commons Notify message type (if you are a Drupal Commons user).
  6. Edit your new message type to your needs. You can use the following replacement tokens in both subject and body fields:
    1. [message:user:name] Username of the user to whom the message is sent.
    2. [mention:author:name] Username of the user who created the mention.
    3. [mention:entity:…]
  7. Create a new Rule (say, named message_on_mention_in_node) like this:
    1. Specify the rule's trigger to be: "React on event: After a new mention is created".
    2. Add a condition specifying "mention:entity is of type: Node".
    3. Add an action "Create a new entity". [TODO]
    4. Add an action "Provide a data value". [TODO]
    5. Add an action "Save entity". [TODO]
    6. Add an action "Send message with Message Notify" [TODO]
    7. The four actions in your rule should now look similar to this screenshot.
  8. Create a new Rule messgae_on_mention_in_comment, in analogy to the above but with the condition using entity type "comment".

Alternative: Using PHP code for message sending

It is also possible to create a rule that will send the required e-mail using custom PHP code. However, since this requires enabling the "PHP Filter (php)" module, it is not considered good practice security-wise. However, if you want to use this route, you can do so as follows:

  1. Follow steps 1-3 from the above list.
  2. Create a Rule email_on_mention_with_php.
    1. Specify the rule's trigger to be: "React on event: After a new mention is created".
    2. Add an action "Send mail", set its "To:" field to "mention:user:mail" and the Subject field to something similar to the following text:

Hello [mention:user:field-name-first] [mention:user:field-name-last],

you got mentioned by [mention:author:field-name-first] [mention:author:field-name-last] here:

<?php
if ($mention->entity_type == 'comment') {
  echo '"' . $mention->entity->subject . '"' . "\n";
  echo $mention->entity->comment_body[‘und’][0][‘safe_value’];
}
elseif ($mention->entity_type == 'node') {
  echo '"' . $mention->entity->title . '"' . "\n";
  echo $mention->entity->body[‘und’][0][‘safe_value’];
}
else {
  echo '<no preview available>';
}
?>

You can view the mention and comment on it here:
<?php
if ($mention->entity_type == 'comment') {
  $cid = $mention->entity->cid;
  echo("http://edgeryders.eu/comment/$cid#comment-$cid");
}
elseif ($mention->entity_type == 'node') {
  $nid = $mention->entity->nid;
  echo("http://edgeryders.eu/node/$nid");
}
else {
  echo '<no URL available>';
}
?>

best regards,
xxx

<?php
// echo("——————-\n" . 'Debugging information: $mention = ');
// print_r($mention);
?>

 

Forwarding web traffic

This can come in very helpful when switching servers and wanting to avoid any downtime while the updated DNS entries propagate over 48 hours through the Internet. This is esp. important for websites with

  1. Make sure you meet the requirements. You need:
    1. Your old server has to be a Linux / Unix based server, so that rinetd can be installed.
    2. root access to your old server for installing rinetd
  2. Copy over your websites and databases. You might want to put community websites into maintenance mode first because it's impractical and spammy to write hundreds of users an e-mail saying they should not add content over the next half an hour …
  3. Install rinetd. On your old server, execute (assuming Debian or Ubuntu): apt-get install rinetd
  4. Configure rinetd. For web traffic, the following lines in /etc/rinetd.conf are enough (see the docs for details, and use your new server's IP address instead of xxx.xxx.xxx.xxx of course):
    0.0.0.0  80  xxx.xxx.xxx.xxx  80
    0.0.0.0 443  xxx.xxx.xxx.xxx 443
  5. Stop your webserver. So, do something like service apache2 stop. rinetd can only work successfully if nothing else is connected to the ports it shoud bind to. If there is something else listening, it will result in no redirections and a huge rinetd log file containing messages like 22/Mar/2013:19:35:47 0.0.0.0 0 (null) 0 0 0 accept-failed – [source].
  6. Start rinetd. That's simply service rinetd restart.
  7. Test it. If your websites still work, they are now served by the new server, since you disabled the webserver on the old one.
  8. Change your DNS records. To make everyones requests switch to the new server, point your sites DNS records to the new server.

Forwarding e-mail traffic

You can now migrate e-mails with a similar approach, something along these lines:

netstat -t -l -p
service courier-imapd stop
service courier-imaps stop
service courier-pop3d stop
service courier-pop3s stop
service postfix stop

Then add more forwarding rules in /etc/rinetd.conf – in my case I now have these in total:

# any private mail system
0.0.0.0         24        xxx.xxx.xxx.xxx   24
# SMTP
0.0.0.0         25        xxx.xxx.xxx.xxx   25
# Secure SMTP
0.0.0.0         26        xxx.xxx.xxx.xxx   26
# HTTP
0.0.0.0         80        xxx.xxx.xxx.xxx   80
# POP3
0.0.0.0        110        xxx.xxx.xxx.xxx  110
# IMAP
0.0.0.0        143        xxx.xxx.xxx.xxx  143
# HTTPS
0.0.0.0        443        xxx.xxx.xxx.xxx  443
# IMAP over TLS
0.0.0.0        993        xxx.xxx.xxx.xxx  993
# POP3 over TLS
0.0.0.0        995        xxx.xxx.xxx.xxx  995

And finally restart rinetd to make it pick up the new config:

service rinetd restart

Alternative approaches

  • Using iptables. It also is about redirecting traffic at NAT / TCP level. However, this it is more difficult to understand and to configure, since it is meant as a full-featured software firewall.
  • Using remote databases. If all your sites support using a database on a different host (Drupal, WordPress etc. all can do that), you can do it as follows. The advantage is, you can move your sites one by one, so can take time to iron out any issues on the new server without having to prepare that for all sites at once (which would mean copying over databases twice, once to test and once to take in new changes). A disadvantage of this is that it only works for websites, not for e-mails. Step by step:
    1. Put your site in maintenance mode for a short time.
    2. Move your site and its database over to the new server.
    3. Test your site on the new server by bending DNS at your own machine (edit /etc/hosts if you are on Linux).
    4. Edit your site configuration on your old server to use the remote databases on the new one.
    5. Optionally (if your site has user-generated files), set up an rsync / scp / ftp synchronization job to push or pull new and updated files to the new server, called by a cron job every 15 minutes or so.
    6. Disable maintenance mode again and test if the site on the old server still works (remember to disable DNS bending again!).
    7. Adapt your DNS configuration to point  to the new server. After 48 hours at latest, nobody will access the old server any more.

 

We needed a fast, dedicated server to solve optimization problems. So let’s install COIN-OR CBC (the MLP optimization solver we use). We start from a stock Ubuntu 13.10 machine, as it will happen to you when deploying on Amazon EC2 Spot Instances, like we do.

The easiest way to get COIN-OR OS and CBC installed on a common Linux platform is to install the “full release” binary packages, which contain the binaries from all major COIN-OR projects. Just download the latest COIN-OR-*.tgz package from the CoinAll repo, unpack it, and link the files in its bin/ subdirectory into your path with something like ln -s bin/* /usr/local/bin/.

However, we want to install from source here, which gives us some advantages (mostly following reasoning found here):

  • The most important reason is that COIN-OR cannot distribute GPL-licenced source or binaries with its code. It includes download scripts for these however. So to get goodies like ASL (the AMPL input format library we desperately need for CBC here), GMPL, command completion, command history, Haskell libraries etc. one has to install from source.
  • The installation uses the Debian package management system for simple modifications, updates and removal. (Some COIN-OR packages are available in the Ubuntu archives, but not all from the COIN-OR distribution – not OS for example, which we need here. Also these packages tend to be a bit outdated.)
  • We can optimize COIN-OR to our needs with the many compilation options of COIN-OR. For example, enabling multi-threading for the CBC solver, or optimizatios for specific CPUs / hardware.
  • Bug fixes can be incorporated immediately instead of waiting for a new COIN-OR release.

How to install all major COIN-OR packages from source, according to the CoinAll distribution build notes and the guidlines from the T. K. Ralphs presentation:

  1. Install some needed developer tools. Note that csh is needed for DyLP/doc/build_dylpdoc, which is called in make distclean for example.
    sudo apt-get install patch make pkg-config wget subversion checkinstall csh gcc g++ gfortran
  2. Install some libraries needed by some COIN-OR packages. (matplotlib is needed for the GiMPy tests to complete.)
    sudo apt-get install python-matplotlib
  3. Download the latest stable version. This probably will get you the latest maintenance release, so probably currently 1.7.4 as seen in the CoinAll list of binaries.
    svn co http://projects.coin-or.org/svn/CoinBinary/CoinAll/stable/1.7 CoinAll-1.7
  4. Add necessary and desired third-party source code. Some code could not be included into the COIN-OR distribution because of licence incompatibility (for example with the GPL). But COIN-OR provides nice download scripts to add these components:
    1. Add the MUMPS public domain linear solver. If present, it will be used in the compilation of Ipopt. If not present, Ipopt will be compiled without solvers, which will cause make test fail for Ipopt with a message like “selected linear solver ma27 not available” [source].
      cd ~/CoinAll-1.7/ThirdParty/Mumps
      ./get.Mumps
    2. Add Glpk. This is needed by CBC in order to read GMPL input – that’s exactly what we need, since it is the only way to use an algebraic modeling language for comfortable, high-level input to CBC. GMPL (also called MathProg) is actually a subset of the widespread AMPL [source], so many models in AMPL should also be expressable with it.
      cd ~/CoinAll-1.7/ThirdParty/Glpk
      ./get.Glpk
    3. Add the ASL library. Without this being present, CBC would be built without AMPL .nl input format support – trying to call it with a .nl file would then result in an error “nl file specified locally but ASL not present”. But we want to have the nice simple AMPL input format. So to get it included, we do according to the CBC FAQ:
      cd ~/CoinAll-1.7/ThirdParty/ASL
      ./get.ASL
    4. Add BLAS. Adding Blas (andLapack, see below) to CBC gives better numerical stability to CBC’s LP solver [source, p.5] – whatever that means. Blas is also needed for compiling one of the other COIN-OR projects; it can be added with apt-get install libblas3 libblas-doc libblas-dev but let’s keep here with the mechanism supplied by COIN-OR to get the latest version:
      cd ~/CoinAll-1.7/ThirdParty/Blas
      ./get.Blas
    5. Add Lapack. Second component in addition to Blas for improving the numerical stability of CBC. To add it:
      cd ~/CoinAll-1.7/ThirdParty/Lapack
      ./get.Lapack
    6. (optional) Add METIS. METIS is a library for graph partitioning, and I don’t know if CBC can profit from it. But since it’s the last of the ThirdParty packages that are free to use for all, let’s go ahead:
      cd ~/CoinAll-1.7/ThirdParty/Metis
      ./get.Metis
  5. Configure it all. In the configure command below, the LDFLAGS=-Wl,--no-as-needed; is to solve problems caused probably by some library symbols now being versioned in current versions of Ubuntu. These problems show up with error messages saying “undefined reference to dlopen, dlclose, dlerror, dlsym”. The LDFLAGS setting is a solution from the ipopt mailing list, and while it works for me I don’t quite understand it. It is only needed for ./configure in the sub-projects OS, DyLP and IPopt. But it does not hurt to apply it globally, so for simplicity we do just that. Also, we configure CBC to support parallel execution. This is done by adding the --enable-cbc-parallel compile-time option [source]. Again, it does not hurt to add it to the global ./configure, it will only affect CBC (and should result in a line “Cbc multithreading enabled” in build/Cbc/configure.log).
    mkdir ~/CoinAll-1.7/build;
    cd ~/CoinAll-1.7/build/;
    ../configure --enable-gnu-packages --enable-cbc-parallel -C --prefix=/usr/local/share/coin-or LDFLAGS=-Wl,--no-as-needed;
  6. Compile it all.
    cd ~/CoinAll-1.7/build/;
    make -j 2;
  7. Adapt the tests for a headless server. Some tests will fail without an X Server. We run this on a headless server, so we have to modify the tests (or use Xvfb, but that is a hack as well …).
    1. In ~/CoinAll-1.7/GiMPy/test/test_algorithms.py, remove line produce_graphs() (actually the very last one).
    2. Solve a similar problem with ~/CoinAll-1.7/GrUMPy/test/test_bb.py.
    3. TODO There may be more issues like this. I did not finish trying to get all tests to run.
  8. Run the tests. (Might still fail at some point when used on a headless server, as per the last point. But you will get through with most at least.)
    make test;
  9. Create a directory for the installation.
    sudo mkdir /usr/local/share/coin-or
  10. Create a Debian package and install it. Installing is included in the following command if all works well. In total, it can take a while, 20 minutes even, depending on your machine. I named mine coinor-coinall.
    sudo checkinstall make install;
  11. Put the binaries into the path. Because this is not done by the installation. Let’s use symlinks:
    cd /usr/local/bin && sudo ln -s ../share/coin-or/bin/* .

TODO The packages built this way can be installed on any Ubuntu 13.10, 14.04 or perhaps newer version, but they miss to list some dependencies, so these should be added manually in the “checkinstall make install” step. Until then, make sure to provide the packages yourself with:

sudo apt-get install gfortran

These are the instructions for installing a generic NEOS server, which you can use for many purposes.

WARNING: These instructions guide you through a successul installation of NEOS itself, but stop there. NEOS is not usable without adding one or more solvers, and for that you have to develop one XML config file and one wrapper script each. These are not provided within the NEOS server software package, not even for open source solvers like COIN-OR CBC. For that reason, I finally abandoned NEOS and moved to the 12 years newer and much cleaner COIN-OR Optimization Services (OS) alternative solution. (The last NEOS 4.0 release is from 2002-10-25, while COIN-OR OS is in active development and meant to be the "next-generation NEOS server" [source].) Also NEOS turned out to be really messy, old Perl software with confusing and inconsistent documentation, so that it is a pain to work with it. You have been warned. Proceed at your own risk 🙂

You can test the NEOS interface by using the public NEOS server, as we did too. So installing your own NEOS server is only reasonable when you want to avoid overloading their servers and / or prohibit all your job submissions from becoming public, as happens with the public NEOS server.

Choosing a machine for your server

For our purposes, using Amazon EC2 Spot Instances seems the most economic and scalable solution. For Economy App, we are starting with one of the CPU power optimized c1.medium instances. It is available more or less permanently for 0.028 USD/h in the South America data center (sa-east-1) – that's 20 USD/month when running full-time, while we only need it to run 25% of the time at first.

As our userbase will grow, we will quickly need a lot or calculation power, and then have to care for where to get it economically. And there's lots of optimization potential – see the initial CloudHarmony cloud hosting comparisons, and their current cloud hosting benchmarks for example.

Installing the NEOS server 4.0

Here we are exploring how to install it on a Linux server (Ubuntu 10.13, to be exact), based on the original installation instructions. It is assumed that your web server and the NEOS server will run on the same machine (else, see this hint). We also assume that you start with only one server – NEOS can handle multiple workstations running solvers though [source].

  1. Download NEOS. Download and extract one of the NEOS server packages. The newst is NEOS server 4.0 from 2002-10-25. (Except you want to go for NEOS server 5.0 from 2005, which however appears largely undocumented.) So for example, do:
    cd /var/local/;
    sudo wget ftp://ftp.mcs.anl.gov/pub/neos/Server/server-4.0-102502.tar.gz;
    sudo tar -xzf server-4.0-102502.tar.gz;
    sudo mv server-4.0 neos;
    sudo rm server-4.0-102502.tar.gz;
  2. Provide the directory structure. You have to create web-accessible directories for CGI files and public web pages and a kind of NEOS workspace directory [source]. The latter is called "directory for variable-length files", and if web-accessible, will make all job submissions to your NEOS server public [source]. In our case, we create these directories below the NEOS installation base dir (originally server-4.0). The workspace dir and document root dir are siblings, so job submissions will not be public.
    sudo mkdir /var/local/neos/cgi-bin/ /var/local/neos/htdocs/ /var/local/neos/workspace/;
  3. Adjust file ownership. Change file ownership rights to the user who will install and run the NEOS server. For our Ubuntu 13.10 AMI running as an Amazon EC2 Spot Instance, we use for example:
    sudo chown -R ubuntu:ubuntu /var/local/neos;
  4. Provide Mail. The NEOS Makefile expects a program Mail (yes, caps M), which in old times was used for an extended version of mail [source]. Let's at least provide mail as a surrogate and see if that works:
    apt-get install mail && sudo ln -s $(which mail) /usr/local/bin/Mail
  5. Configure NEOS. Execute make and answer its config questions according to the NEOS FAQ instructions:
    cd /var/local/neos/config && make
  6. Configure crash recovery. Configure NEOS so it will be automatically restarted via cron when it crashes. For that, execute the following command as the user who should run the NEOS server (but if crontab -l shows you have a crontab already, edit it manually since the following command would replace it): crontab /usr/local/neos/crontabfile.
  7. Install legacy libs. The NEOS server restart script uses flush.pl, a legacy Perl library. This creates a warning about being deprecated and imminent removal from the next major release, so we resolve that warning by installing the library which will provide it in the future: sudo apt-get install libperl4-corelibs-perl.
  8. Start your NEOS server.
    /var/local/neos/bin/restart
  9. Check that the NEOS server is running. Just check in ps aux output that the above command started the initializer.pl, scheduler.pl and socket-server.pl daemons successfully [source].
  10. Confirm that the XML-RPC interface to your server works. By default it resides at port 3333, you configured it with make above. For that, just visit that URL with a browser (something like http://example.com:3333/ ). If it works, the NEOS server will server you a message like this:
    121
    your-server-name: ERROR: "Host: example.com:3333" is not recognized by this server.
  11. Install and configure a web server. It will serve the NEOS web interface. In this case, we use lighttpd.
    1. Install lighttpd.
      sudo apt-get install lighttpd
    2. Enable CGI. Enable the CGI module that you will use to serve the NEOS Perl application:
      sudo lighty-enable-mod cgi
    3. Don't serve CGI files as source! A little security optimization for the NEOS server files: in /etc/lighttpd/lighttpd.conf, change this line
      static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
      to this
      static-file.exclude-extensions = ( ".php", ".pl", ".fcgi", ".cgi" )
    4. Add a configuration section for the NEOS server. This goes to /etc/lighttpd/lighttpd.conf (in vhost style though it's our only site).
      $HTTP[“host”]  =~ "example.com" {
        server.document-root = "/var/local/neos/htdocs"
        alias.url += ( "/cgi-bin" => "/var/local/neos/cgi-bin/" )
        $HTTP[“url”] =~ "^/cgi-bin" {
            cgi.assign = ( ".cgi" => "" )
        }
      }

      [References: nixCraft lighttpd setup for Perl programs; lighttpd Configuration File Options reference.]
    5. Reload the lighttpd configuration. Execute: sudo service lighttpd force-reload
    6. Adjust file permissions. We need to give the web server user (by default www-data for lighttpd) read and write permissions to some directories:
      cd /var/local/neos;
      sudo chown -R :www-data .;
      sudo chmod g+r+w htdocs/jobs workspace/spool/WEB workspace/databases workspace/tmp
      ;
    7. Test the web interface. Make sure that the NEOS web interface works by visiting the base URL in a browser. It would be http://example.com/ for the above case. Also visit http://exampel.com/cgi-bin/check-status.cgi to make sure your CGI config works.
  12. Configure your system so that the NEOS server will be started at system start. This is especially relevant when using an Amazon EC2 spot instance, since these are occasionally shut down and restarted depending on your bid pricing (and yes, this means system shutdown, not virtual machine hibernating). For that add the following to the content of /etc/rc.local (and disable the default exit 0 if still present):
    /var/local/neos/bin/restart

Registering solvers at your NEOS server

Your NEOS server 4.0 is installed now, but so far it does not have any knowledge about what solvers are available and how to contact them. The NEOS Solvers FAQ explains the process of registering solvers; see also this short list of steps and the HelloNEOS example.

The problem is that the configurations for registering all the solvers of the public NEOS server are all missing – not included in the NEOS server software as they are considered parts of the installation. You would have to write a complex XML file and a wrapper script to (for example) make your NEOS server talk to the free software COIN-OR CBC solver.  In the time needed for this, you can just as well install the alternative solution COIN-OR OS, which comes already pre-configured for being used with CBC. Which is why I abandoned using NEOS at this point.

Installing a solver and making it talk to your NEOS server

At this point your NEOS server knows how to talk to the CBC solver, but neither is your CBC solver installed nor did we add anything to let it know how to talk back to the NEOS server. So let's do both of this now.

(Note: You might see the neos-comms packages for download. However, do not use these. Reasoning: These files are from 1998, while code with equivalent functionality is available inside the latest NEOS 4.0 2002-10-25 server package. That code is clearly newer and better, with comments and all. The neos-comms package contained a neos-comms-4.01/bin/client file that is not available in the new code, but seemingly is just an added start script for the comms daemon and Tcl/Tk interface that got now integrated into the daemon and Tcl/Tk application themselves.)

  1. Start the comms daemon. This is the "Communications daemon" which has to run on the solver workstation (same machine in our case) and mediates between the NEOS server on one side and the solver (here CBC) on the other. It can be started with the "Communications GUI", available as a Tcl/Tk application within the NEOS server software installation. For details, see comms-help.txt (also available within the equivalent place in your NEOS server installation).

Links to NEOS documentation