You can follow all the steps of the official HowTo “The Perfect Server – Ubuntu 13.04 (Apache2, BIND, Dovecot, ISPConfig 3)“. To get it to work afterwards on Ubuntu 13.10, use these additional steps:

  1. Switch to a .vhost naming scheme for Apache2 site configurations. In Apache2 2.4, only the sites-enabled/*.conf files will be loaded automatically, missing out ISPConfig’s /etc/apache2/sites-enabled/*.vhost. Usually we would rename them to .conf, but because .vhost files are also generated automatically for every website created in ISPConfig, we instead adapt what Apache2 loads. Because that is less work than adapting the source code of ISPConfig. Note in the following commands, both for sites and further down for config snippets, that it’s important to keep the original apps.vhost and ispconfig.vhost included at the start of the site inclusion process, so let’s keep the “000-” and “100-” prefixes used by ISPConfig [source].
    1. In /etc/apache2/apache2.conf, replace line “IncludeOptional sites-enabled/*.conf” with “IncludeOptional sites-enabled/*.vhost“.
    2. mv /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/000-default.vhost;
    3. mv /etc/apache2/sites-enabled/000-default.conf /etc/apache2/sites-enabled/000-default.vhost;
    4. rm /etc/apache2/sites-enabled/000-apps.vhost;
    5. mv /etc/apache2/sites-available/apps.vhost /etc/apache2/sites-available/000-ispconfig-apps.vhost;
    6. ln -s ../sites-available/000-ispconfig-apps.vhost /etc/apache2/sites-enabled/000-ispconfig-apps.vhost;
    7. rm /etc/apache2/sites-enabled/000-ispconfig.vhost;
    8. mv /etc/apache2/sites-available/ispconfig.vhost /etc/apache2/sites-available/000-ispconfig.vhost;
    9. ln -s ../sites-available/000-ispconfig.vhost /etc/apache2/sites-enabled/000-ispconfig.vhost;
  2. Move and rename the ISPConfig Apache2 config file. In addition, the ISPConfig .conf files which are also present actually contain configuration information and not vhost directives for sites, so a2ensite and a2dissite cannot handle them. They are now expected in /etc/apache2/conf-available, to be handled by a2enconf and a2disconf. So let’s rearrange things a bit to solve this (note that we use a “.local” suffix to indicate local configuration additions, not provided via OS packages):
    1. rm /etc/apache2/sites-enabled/000-ispconfig.conf;
    2. mv /etc/apache2/sites-available/ispconfig.conf /etc/apache2/conf-available/000-ispconfig.local.conf;
    3. a2enconf 000-ispconfig.local;
  3. (optional) Fix a2ensite and a2dissite. Due to our transition to the “.vhost” naming scheme for Apache2 site configurations, a2ensite / a2dissite do not work any more because they expect a .vhost suffix. Not too bad because one will enable and disable most sites within ISPConfig anyway. However, we can also repair this by supplying a local alternative with a fix in the source code:
    1. cp /usr/sbin/a2ensite /usr/local/bin;
    2. ln -s a2ensite /usr/local/bin/a2dissite;
    3. In file /usr/local/bin/a2ensite, replace line 60 “$sffx   = ‘.conf’;” with “$sffx   = ‘.vhost’;“.
    4. Ensure with which a2ensite and which a2dissite that your local alternatives are in use now.
    5. Unfortunately, this fix only works for sites not handled by ISPConfig. To also handle those created by ISPConfig, one has to find a solution for the fact that ISPConfig creates the symlinks with a “100-” prefix while the site configurations in sites-available/ miss this. The simplest is to provide an alternative with that prefix in sites-available/ as well, using something like:
      ln -s /etc/apache2/sites-available/;
  4. Fix a syntax error about directory options. Apache would now complain about a syntax error: “Either all Options must start with + or -, or no Option may.” So add a “+” character before all options missing a “+” or “-” prefix in:
    /etc/apache2/sites-available/000-ispconfig-apps.conf line 30
    /etc/apache2/sites-available/000-ispconfig.conf line 20
  5. Remove the NameVirtualHost directives. They don’t really hurt, but are effectless and deprecated in Apache 2.4, generating warnings when restarting Apache. So remove them from:
    /etc/apache2/sites-available/000-ispconfig.conf line 8
    /etc/apache2/conf-available/000-ispconfig.conf lines 53 and 54
  6. Reload the Apache configuration.
    service apache2 reload;
  7. Test ISPConfig and configure it. If you have enabled SSL for the ISPConfig site during the installation configure script, try visiting your server’s FQDN at port 8080 in your browser (like browsing to Initial username and password are “admin” and “admin” – remember to change them immediately. The remaining configuration happens from inside the ISPConfig 3 panel.

We needed a fast, dedicated server to solve optimization problems. So let’s install COIN-OR CBC (the MLP optimization solver we use). We start from a stock Ubuntu 13.10 machine, as it will happen to you when deploying on Amazon EC2 Spot Instances, like we do.

The easiest way to get COIN-OR OS and CBC installed on a common Linux platform is to install the “full release” binary packages, which contain the binaries from all major COIN-OR projects. Just download the latest COIN-OR-*.tgz package from the CoinAll repo, unpack it, and link the files in its bin/ subdirectory into your path with something like ln -s bin/* /usr/local/bin/.

However, we want to install from source here, which gives us some advantages (mostly following reasoning found here):

  • The most important reason is that COIN-OR cannot distribute GPL-licenced source or binaries with its code. It includes download scripts for these however. So to get goodies like ASL (the AMPL input format library we desperately need for CBC here), GMPL, command completion, command history, Haskell libraries etc. one has to install from source.
  • The installation uses the Debian package management system for simple modifications, updates and removal. (Some COIN-OR packages are available in the Ubuntu archives, but not all from the COIN-OR distribution – not OS for example, which we need here. Also these packages tend to be a bit outdated.)
  • We can optimize COIN-OR to our needs with the many compilation options of COIN-OR. For example, enabling multi-threading for the CBC solver, or optimizatios for specific CPUs / hardware.
  • Bug fixes can be incorporated immediately instead of waiting for a new COIN-OR release.

How to install all major COIN-OR packages from source, according to the CoinAll distribution build notes and the guidlines from the T. K. Ralphs presentation:

  1. Install some needed developer tools. Note that csh is needed for DyLP/doc/build_dylpdoc, which is called in make distclean for example.
    sudo apt-get install patch make pkg-config wget subversion checkinstall csh gcc g++ gfortran
  2. Install some libraries needed by some COIN-OR packages. (matplotlib is needed for the GiMPy tests to complete.)
    sudo apt-get install python-matplotlib
  3. Download the latest stable version. This probably will get you the latest maintenance release, so probably currently 1.7.4 as seen in the CoinAll list of binaries.
    svn co CoinAll-1.7
  4. Add necessary and desired third-party source code. Some code could not be included into the COIN-OR distribution because of licence incompatibility (for example with the GPL). But COIN-OR provides nice download scripts to add these components:
    1. Add the MUMPS public domain linear solver. If present, it will be used in the compilation of Ipopt. If not present, Ipopt will be compiled without solvers, which will cause make test fail for Ipopt with a message like “selected linear solver ma27 not available” [source].
      cd ~/CoinAll-1.7/ThirdParty/Mumps
    2. Add Glpk. This is needed by CBC in order to read GMPL input – that’s exactly what we need, since it is the only way to use an algebraic modeling language for comfortable, high-level input to CBC. GMPL (also called MathProg) is actually a subset of the widespread AMPL [source], so many models in AMPL should also be expressable with it.
      cd ~/CoinAll-1.7/ThirdParty/Glpk
    3. Add the ASL library. Without this being present, CBC would be built without AMPL .nl input format support – trying to call it with a .nl file would then result in an error “nl file specified locally but ASL not present”. But we want to have the nice simple AMPL input format. So to get it included, we do according to the CBC FAQ:
      cd ~/CoinAll-1.7/ThirdParty/ASL
    4. Add BLAS. Adding Blas (andLapack, see below) to CBC gives better numerical stability to CBC’s LP solver [source, p.5] – whatever that means. Blas is also needed for compiling one of the other COIN-OR projects; it can be added with apt-get install libblas3 libblas-doc libblas-dev but let’s keep here with the mechanism supplied by COIN-OR to get the latest version:
      cd ~/CoinAll-1.7/ThirdParty/Blas
    5. Add Lapack. Second component in addition to Blas for improving the numerical stability of CBC. To add it:
      cd ~/CoinAll-1.7/ThirdParty/Lapack
    6. (optional) Add METIS. METIS is a library for graph partitioning, and I don’t know if CBC can profit from it. But since it’s the last of the ThirdParty packages that are free to use for all, let’s go ahead:
      cd ~/CoinAll-1.7/ThirdParty/Metis
  5. Configure it all. In the configure command below, the LDFLAGS=-Wl,--no-as-needed; is to solve problems caused probably by some library symbols now being versioned in current versions of Ubuntu. These problems show up with error messages saying “undefined reference to dlopen, dlclose, dlerror, dlsym”. The LDFLAGS setting is a solution from the ipopt mailing list, and while it works for me I don’t quite understand it. It is only needed for ./configure in the sub-projects OS, DyLP and IPopt. But it does not hurt to apply it globally, so for simplicity we do just that. Also, we configure CBC to support parallel execution. This is done by adding the --enable-cbc-parallel compile-time option [source]. Again, it does not hurt to add it to the global ./configure, it will only affect CBC (and should result in a line “Cbc multithreading enabled” in build/Cbc/configure.log).
    mkdir ~/CoinAll-1.7/build;
    cd ~/CoinAll-1.7/build/;
    ../configure --enable-gnu-packages --enable-cbc-parallel -C --prefix=/usr/local/share/coin-or LDFLAGS=-Wl,--no-as-needed;
  6. Compile it all.
    cd ~/CoinAll-1.7/build/;
    make -j 2;
  7. Adapt the tests for a headless server. Some tests will fail without an X Server. We run this on a headless server, so we have to modify the tests (or use Xvfb, but that is a hack as well …).
    1. In ~/CoinAll-1.7/GiMPy/test/, remove line produce_graphs() (actually the very last one).
    2. Solve a similar problem with ~/CoinAll-1.7/GrUMPy/test/
    3. TODO There may be more issues like this. I did not finish trying to get all tests to run.
  8. Run the tests. (Might still fail at some point when used on a headless server, as per the last point. But you will get through with most at least.)
    make test;
  9. Create a directory for the installation.
    sudo mkdir /usr/local/share/coin-or
  10. Create a Debian package and install it. Installing is included in the following command if all works well. In total, it can take a while, 20 minutes even, depending on your machine. I named mine coinor-coinall.
    sudo checkinstall make install;
  11. Put the binaries into the path. Because this is not done by the installation. Let’s use symlinks:
    cd /usr/local/bin && sudo ln -s ../share/coin-or/bin/* .

TODO The packages built this way can be installed on any Ubuntu 13.10, 14.04 or perhaps newer version, but they miss to list some dependencies, so these should be added manually in the “checkinstall make install” step. Until then, make sure to provide the packages yourself with:

sudo apt-get install gfortran

These are the instructions for installing a generic NEOS server, which you can use for many purposes.

WARNING: These instructions guide you through a successul installation of NEOS itself, but stop there. NEOS is not usable without adding one or more solvers, and for that you have to develop one XML config file and one wrapper script each. These are not provided within the NEOS server software package, not even for open source solvers like COIN-OR CBC. For that reason, I finally abandoned NEOS and moved to the 12 years newer and much cleaner COIN-OR Optimization Services (OS) alternative solution. (The last NEOS 4.0 release is from 2002-10-25, while COIN-OR OS is in active development and meant to be the "next-generation NEOS server" [source].) Also NEOS turned out to be really messy, old Perl software with confusing and inconsistent documentation, so that it is a pain to work with it. You have been warned. Proceed at your own risk 🙂

You can test the NEOS interface by using the public NEOS server, as we did too. So installing your own NEOS server is only reasonable when you want to avoid overloading their servers and / or prohibit all your job submissions from becoming public, as happens with the public NEOS server.

Choosing a machine for your server

For our purposes, using Amazon EC2 Spot Instances seems the most economic and scalable solution. For Economy App, we are starting with one of the CPU power optimized c1.medium instances. It is available more or less permanently for 0.028 USD/h in the South America data center (sa-east-1) – that's 20 USD/month when running full-time, while we only need it to run 25% of the time at first.

As our userbase will grow, we will quickly need a lot or calculation power, and then have to care for where to get it economically. And there's lots of optimization potential – see the initial CloudHarmony cloud hosting comparisons, and their current cloud hosting benchmarks for example.

Installing the NEOS server 4.0

Here we are exploring how to install it on a Linux server (Ubuntu 10.13, to be exact), based on the original installation instructions. It is assumed that your web server and the NEOS server will run on the same machine (else, see this hint). We also assume that you start with only one server – NEOS can handle multiple workstations running solvers though [source].

  1. Download NEOS. Download and extract one of the NEOS server packages. The newst is NEOS server 4.0 from 2002-10-25. (Except you want to go for NEOS server 5.0 from 2005, which however appears largely undocumented.) So for example, do:
    cd /var/local/;
    sudo wget;
    sudo tar -xzf server-4.0-102502.tar.gz;
    sudo mv server-4.0 neos;
    sudo rm server-4.0-102502.tar.gz;
  2. Provide the directory structure. You have to create web-accessible directories for CGI files and public web pages and a kind of NEOS workspace directory [source]. The latter is called "directory for variable-length files", and if web-accessible, will make all job submissions to your NEOS server public [source]. In our case, we create these directories below the NEOS installation base dir (originally server-4.0). The workspace dir and document root dir are siblings, so job submissions will not be public.
    sudo mkdir /var/local/neos/cgi-bin/ /var/local/neos/htdocs/ /var/local/neos/workspace/;
  3. Adjust file ownership. Change file ownership rights to the user who will install and run the NEOS server. For our Ubuntu 13.10 AMI running as an Amazon EC2 Spot Instance, we use for example:
    sudo chown -R ubuntu:ubuntu /var/local/neos;
  4. Provide Mail. The NEOS Makefile expects a program Mail (yes, caps M), which in old times was used for an extended version of mail [source]. Let's at least provide mail as a surrogate and see if that works:
    apt-get install mail && sudo ln -s $(which mail) /usr/local/bin/Mail
  5. Configure NEOS. Execute make and answer its config questions according to the NEOS FAQ instructions:
    cd /var/local/neos/config && make
  6. Configure crash recovery. Configure NEOS so it will be automatically restarted via cron when it crashes. For that, execute the following command as the user who should run the NEOS server (but if crontab -l shows you have a crontab already, edit it manually since the following command would replace it): crontab /usr/local/neos/crontabfile.
  7. Install legacy libs. The NEOS server restart script uses, a legacy Perl library. This creates a warning about being deprecated and imminent removal from the next major release, so we resolve that warning by installing the library which will provide it in the future: sudo apt-get install libperl4-corelibs-perl.
  8. Start your NEOS server.
  9. Check that the NEOS server is running. Just check in ps aux output that the above command started the, and daemons successfully [source].
  10. Confirm that the XML-RPC interface to your server works. By default it resides at port 3333, you configured it with make above. For that, just visit that URL with a browser (something like ). If it works, the NEOS server will server you a message like this:
    your-server-name: ERROR: "Host:" is not recognized by this server.
  11. Install and configure a web server. It will serve the NEOS web interface. In this case, we use lighttpd.
    1. Install lighttpd.
      sudo apt-get install lighttpd
    2. Enable CGI. Enable the CGI module that you will use to serve the NEOS Perl application:
      sudo lighty-enable-mod cgi
    3. Don't serve CGI files as source! A little security optimization for the NEOS server files: in /etc/lighttpd/lighttpd.conf, change this line
      static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
      to this
      static-file.exclude-extensions = ( ".php", ".pl", ".fcgi", ".cgi" )
    4. Add a configuration section for the NEOS server. This goes to /etc/lighttpd/lighttpd.conf (in vhost style though it's our only site).
      $HTTP[“host”]  =~ "" {
        server.document-root = "/var/local/neos/htdocs"
        alias.url += ( "/cgi-bin" => "/var/local/neos/cgi-bin/" )
        $HTTP[“url”] =~ "^/cgi-bin" {
            cgi.assign = ( ".cgi" => "" )

      [References: nixCraft lighttpd setup for Perl programs; lighttpd Configuration File Options reference.]
    5. Reload the lighttpd configuration. Execute: sudo service lighttpd force-reload
    6. Adjust file permissions. We need to give the web server user (by default www-data for lighttpd) read and write permissions to some directories:
      cd /var/local/neos;
      sudo chown -R :www-data .;
      sudo chmod g+r+w htdocs/jobs workspace/spool/WEB workspace/databases workspace/tmp
    7. Test the web interface. Make sure that the NEOS web interface works by visiting the base URL in a browser. It would be for the above case. Also visit to make sure your CGI config works.
  12. Configure your system so that the NEOS server will be started at system start. This is especially relevant when using an Amazon EC2 spot instance, since these are occasionally shut down and restarted depending on your bid pricing (and yes, this means system shutdown, not virtual machine hibernating). For that add the following to the content of /etc/rc.local (and disable the default exit 0 if still present):

Registering solvers at your NEOS server

Your NEOS server 4.0 is installed now, but so far it does not have any knowledge about what solvers are available and how to contact them. The NEOS Solvers FAQ explains the process of registering solvers; see also this short list of steps and the HelloNEOS example.

The problem is that the configurations for registering all the solvers of the public NEOS server are all missing – not included in the NEOS server software as they are considered parts of the installation. You would have to write a complex XML file and a wrapper script to (for example) make your NEOS server talk to the free software COIN-OR CBC solver.  In the time needed for this, you can just as well install the alternative solution COIN-OR OS, which comes already pre-configured for being used with CBC. Which is why I abandoned using NEOS at this point.

Installing a solver and making it talk to your NEOS server

At this point your NEOS server knows how to talk to the CBC solver, but neither is your CBC solver installed nor did we add anything to let it know how to talk back to the NEOS server. So let's do both of this now.

(Note: You might see the neos-comms packages for download. However, do not use these. Reasoning: These files are from 1998, while code with equivalent functionality is available inside the latest NEOS 4.0 2002-10-25 server package. That code is clearly newer and better, with comments and all. The neos-comms package contained a neos-comms-4.01/bin/client file that is not available in the new code, but seemingly is just an added start script for the comms daemon and Tcl/Tk interface that got now integrated into the daemon and Tcl/Tk application themselves.)

  1. Start the comms daemon. This is the "Communications daemon" which has to run on the solver workstation (same machine in our case) and mediates between the NEOS server on one side and the solver (here CBC) on the other. It can be started with the "Communications GUI", available as a Tcl/Tk application within the NEOS server software installation. For details, see comms-help.txt (also available within the equivalent place in your NEOS server installation).

Links to NEOS documentation

AWS CLI is the Amazon Web Services command line interface tool, the new unified utility to manage your cloudy Amazon things.

People are usually told to install it with pip install awscli (including in the official docs), but this is a hateable solution for Linux package system fans because

  • You get one more package system (pip, for Python packages) where you have to care for updates, and where you will forget just that. Not that it would be any worse than having own package management systems already for Firefox add-ons, Chrome extensions, Gnome extensions, Ruby gems, Drupal modules, and WordPress plugins. All of that is just plain bad. Grrrr.
  • You no longer have a single point of control and overview for what software is installed on your system.

So, let's try installing the AWS CLI from packages. Fortunately, there are fairly recent (awscli 1.2.9, from 2014-01) packages for upcoming Ubuntu 14.04 (Trusty Tahr). We are on Ubuntu 13.10 however, but we can fix it by adapting these instructions for Debian to our situation:

  1. Add a package source for Ubuntu trusty, by adding a line like this (with your Ubuntu mirror) to the bottom of /etc/apt/sources.list:
    deb trusty main universe
  2. Create a preferece for Ubuntu trusty packages that will allow to install them when specifying the distribution, but will not select them automatically even when the version is newer than the local one. For that, create a file /etc/apt/preferences.d/ubuntu-trusty.pref with the following content:
    Package: *
    Pin: release a=trusty
    Pin-Priority: 200
  3. Install awscli and its dependencies: sudo apt-get update; sudo apt-get install -t trusty awscli.

You are ready to use it now (try aws --version). Note that they include the functional equivalent of the Amazon EC2 CLI tools, and many more Amazon CLI tools – you will very probably not need to install any Amazon specific CLI tools any more, regardless of what outdated how-tos are telling you.

Also see Amazon's official AWS CLI documentation.


This issue started to occur right after installing Ubuntu 13.10 on a ThinkPad T61 with an "Intel Corporation 82801H (ICH8 Family) HD Audio Controller (rev 03)" according to lspci. So, whenever I do one of these:

  • adjust volume with the hardware volume silence / up / down buttons, only while not in a Skype call
  • adjust volume with an equivalent software mixer feature, only while not in a Skype call

the effect is this:

  • the next Skype call will be completely silent (both output and input do not work)
  • the Skype call after that will play a randomized variant of rattling noise for output, while input probably works fine (judged by the level bar indicator); interestingly, the closing sound of the Skype call will play correctly, which indicates that only one of the audiostreams is not initialized correctly (the Gnome sound settings panel shows that two are in use during a call)
  • the Skype call after that will play also the output, but with lots of stuttering in between
  • the Skype call after that will play the output correctly, and the input will also work fine

For this to be repeatable, allow for some 3-5 seconds of separation between calls. At times, these four steps will also be reduced to three, or the "rattling noise" or "stuttering" step could also occur two or more times.

In addition, there is a similar problem with Skype chats. Whenever I do one of these:

  • send a message

the effect is this:

  • a randomized variant of permanent, rattling noise (the "message sent" sound will not be sent)

This noise will persist also through Skype calls initiated afterwards. It will (in most cases) change when sending another message, and after some messages will also completely disappear again. The surefire way to make it disappear is though when your chat partner sends you a message.


The reason for this behavior is that Ubuntu 13.10 ships with PulseAudio 4.0, and Skype does not properly support that so far [source].

The proper solution is to simply install Skype from the Canonical Partner repositories (so, not by manually downloading a Skype .deb via its website). It contains a patch to the skype.desktop file that starts Skype now with a proper workaround as "env PULSE_LATENCY_MSEC=60 skype" [source].

However, if like me you're used to start applications via the Alt+F2 mini-terminal, you might even have this patched Skype package installed, and it has still no effect to fix your audio in Skype. As you probably want to keep starting Skype by hitting Alt+F2 and typing skype, here is a way to do so:

  1. Createa file /usr/local/bin/skype, with the following content:


    # Workaround for the Skype incompatibility with PulseAudio 4.0, as explained in
    # This is already installed in the Ubuntu Saucy Skype package from Canonical Partner repos, however their
    # solution of prepending an environment variable in the .desktop file is not used when starting Skype
    # via the Alt+F2 mini-terminal. To apply the solution for that too, we need this file to supersede the normal
    # Skype command. Use "which skype" to confirm that the skype command afterwards indeed refers to
    # /usr/local/bin/skype instead of /usr/bin/skype.

    env PULSE_LATENCY_MSEC=60 /usr/bin/skype

  2. Give proper execution rights to that file.
  3. Check with which skype to make sure Skype is now called from /usr/local/bin/skype instead of from /usr/bin/skype.

There are also some dirty workarounds (here mentioned only to learn something about Skype and Pulseaudio, not to use them):

  1. Do not use volume buttons except when in a Skype call. This is not practical of course.
  2. Disable Skype sound events for "Call Connecting" and "Chat Message Sent", and maybe all others. This can be done in the Skype "Options -> Notifications" menu item. This however can result in Skype calls falling completely silent, and this will also not fix itself in the next call then. (It can however be fixed, right during the call even, by playing some seconds of sound from a different application.)
  3. Create the permanent random noise, then mute it via the "System Sounds" stream. This works as follows:
    1. Disable the "System Sounds" PulseAudio stream by going to the Gnome Sound Settings dialog, there to tab "Sound Effects" and set "Alert Volume: Off". Alternatively, in Skype go to "Options -> Sound Devices -> Open Pulse Audio Volume Control -> Playback", and click the "Mute audio" button for stream "System sounds".  In both cases, you will notice that the Skype sound channel in the "Applications" tab (if there is one, usually only after step 2) also goes silent, and with it the Skype notifications in chats, and your volume level adjustment feedback sound goes silent as well (explanation see below).
    2. Let Skype create a sound mess. You have to create that permanent random noise, either by using the "send chat message" technique from above (with a notifocation sound enabled of course), or by test playing any notification sound via Skype's "Options -> Notifications -> Test Event" button. (You can not use the random noise generated by a Skype call with broken audio from above.)
    3. Now do whatever you want in Skype, the problem with corrupted sound in calls will not appear any more, even when using volume adjustments in between of calls.
    4. You have to repeat step 2 after a restart of Skype. (The muting of the System Sounds stream stays active after Skype quits because it is not a Skype feature; however Skype is only immune against creating corrupt audio in calls once it has corrupted the System Sounds audio by executing step 2.)

Explanation Attempt

Let me try a little explanation (just from observations, I do not know how PulseAudio works internally): The problem of Skype seems to be that it cannot properly write to the "System Sounds" stream if another application has written to it in between (including the feedback sounds of the volume change buttons). When Skype tries to write to the System Sounds stream in this situation, it results in that noise (as exemplified by the "send chat message" case). Or for some reason, it can also result in all Skype sounds being muted, and on second try in that noise (as exemplified by the phone call example; it's really due to the notification sounds played at the start of the phone call, since there is no such problem when disabling the notification sounds).

So, the third workaround above works by letting Skype try (creating the noise), but muting that noise away (before or afterwards). As Skype's attempt to write to the channel never returns, it is blocked and Skype has to use a different (newly created) channel for the notification sounds when starting a phone call, as can be seen in the "Applications" tab of Gnome sound settings. That might be why now, playing the notification sound no longer results in corrupted audio. That indeed Skype tries permanently to write to the System Sounds channel (and only creating noise doing so) can be recognized from the fact that the noise stops when Skype exits: then, at last, it stops its desperate attempts of writing to System Sounds.

Other Issues: Silent Input and Output

Muted input. For some reason, at times it happens that Skype shuts off the microphone input when exiting. (This is possibly related to letting Skype access ones input mixer levels in its options dialog.) It can be fixed by going into the Gnome Sound Settings dialog, to tab "Input", and switching "Input volume" off and on again. When you see the input level bar moving when sound is present, all is well again.

Muted output. In other cases (as seen above), Skype output might be completely muted, while still working for other applications. Playing any sound from another application will fix this. And probably, going to Gnome Sound Settings and switching "Output volume" off and on again will also help.

Checkvist is a nice, web-based ouline editor. Since it uses a hierarchical content structure, and a mindmapping software like Freemind does the same, interfacing between them can work well in theory. There are some quirks and tips though that we will explore here in practice.

I selected Checkvist among alternative solutions for the following reasons:

  • Work fast with large amounts of text. Large amounts means here something like 400k characters (400 pages A4), which Freemind can handle easily. The web-based, open source mindmapping tool Wisemapping for example was only able to work with some few pages of text before getting sluggish).
  • Unlimited lists, list items and other features in the gratis version. In contrast, Workflowy is also nice but offers only 250 free list items per month … .
  • Collaborative editing in the free version. Because that is what I need it for: a real-time collaborative interface for some content I developed so far in Freemind mindmaps.
  • Public sharing in the free version.
  • Comfortable importing from Freemind 1.0. In contrast, Wisemapping for example would support only imports from Freemind 0.9 directly, so you would need Freemind 0.9 installed as well to copy, paste and save your Freemind 1.0 mindmap in 0.9 format before uploading it.

How to import Freemind content into Checkvist

  1. You can only import plain text. No icons, colors, HTML rich text formatting of nodes etc., but you do not have to remove them before either.
  2. Make sure you do not have multiple paragraphs of text in any one node. Because the second and following paragraphs would start without indentation in the text version, leading to hierarchy level errors during the import. So, split every node that currently has multiple paragraphs using this technique:
    1. Position the node selection at the root node of the branch you want to export.
    2. Do "Navigate -> Unfold All" (Ctrl + Shift + End).
    3. Do "Edit -> Select Visible Branch" (Ctrl + Shift + A).
    4. Do "Format -> Use Plain Text". This will convert bulleted and numbered lists into normal paragraphs, as else "Split Node" would not be able to break them up.
    5. Do "Tools -> Split Node".
    6. Do "Navigate -> Fold All" (Ctrl + Shift + Home).
  3. Copy the content you want to import. Select all nodes you want to appear on the first level after the import, and do "Edit -> Copy" (Ctrl + C).
  4. In Checkvist, select "Import" and paste the clipboard content.
  5. Click "Import tasks". It will import your Freemind content as indented text.


Wait, the *Secure* Socket Layer in HTTPS can be insecure? Yep, in the age of total surveillance, it can.

Good news: To the best of our knowledge, there is also secure SSL still. (But don't trust me on these instructions with your life or the life of your website users – you have to become your own expert!). The considerations below take into account both secrecy and server performance. The tips are in decreasing order of importance.

(1) Your users need an uncompromized computer!

Because if we'd start with compromized hardware, all is lost anyway. The malware can simply grab your communications from the browser screen and send it to the surveillance body. No need to break SSL, then. But of course, that would also be simple: that malware would also have an easy job to hide man-in-the-middle attacks by preventing the tools mentioned below from detecting SSL certificate changes.

The best first tip for having a non-compromized computer is having two: one for daily work, one for only the high-value communications. And you would not go near any threat on the Internet with the second one.

(2) Enforce HTTPS for all connections

For performance reasons, you might think about using SSL connections only for login (password transmission), or at least only while users are logged in (also protecting the content they post, and the session cookie which else can be used for session hijacking). However, surveillance can derive lots of metadata, behavioral data etc. also from looking at what people read while not being logged in. With proper SSL speed optimization, the server load of enforcing SSL everywhere should be manageable.

(3) Throw out insecure SSL cipher suites

Configure your webserver to not use:

  • the old utmost crap "export" cipher suites
  • plain DES (triple DES is ok though)
  • RC2
  • RC4 (which is kind of broken)

See the source for these recommendations. Note: The SSL Labs SSL Test is a nice site to check if your configuration works as intended.

From the remaining ones, all ciper suites that use at least 128 bit keys for symmetrical keys are ok. This is roughly equivalent to the security of 3072 bits RSA keys [source, p. 64], which are a sufficient protection against brute force attacks even beyond 2030 (as we will see below). Which means, encrypted data recorded now could only be broken some time after 2030. This is further protected by the fact that the symmetrical keys are only used for one SSL session, and using brute force attacks to decrypt one such small session from 20+ years ago is almost certainly not worth the effort in 2040 or so.

In a few years, when all browsers will support higher grade AES cipher suites and so on, you would of course switch to only allow at least 192 or 256 bits of security. Which is equivalent to 7680 resp. 15360 bit RSA keys [source, p. 64] and comes at relatively neglible performance costs of needing about 30% more CPU time for the same data throughput [source].

(4) Use only DHE Perfect Forward Secrecy key cipher suites

We want to use perfect forward secrecy (PFS) cipher suites. PFS means: when the private key of the server is leaked at some time, recorded communication of the past still cannot be decrypted. It only allows the attacker to impersonate the server for negotiating keys for new sessions, until the SSL certificate expires (which is hopefully soon).

Here is how to configure your webserver for using PFS cipher suites.

However, not all PFS ciphers are the same. As Bruce Schneier writes: "Prefer conventional discrete-log-based systems over elliptic-curve systems; the latter have constants that the NSA influences when they can." [source] This means, use Diffie-Hellman key exchange (DHE) cipher suites, not Elliptic-curve Diffie-Hellman key exchange (ECDHE) [source]. The danger of weak ECDHE ciphers is that recorded, encrypted communication could be broken later with limited effort attacks (albeit only session by session). The downside of DHE, on the other side, is "only" that it is slower.

(If you really want to use ECDHE for performance reasons, offer it only with elliptic curves that are safe. I am not sure if browsers support any safe curves [see] or how to restrict what curves your OpenSSL installation will offer to the client [see]. Tell us in the comments when you find it out.)

(5) Always store your private key in encrypted form

Which should mean, store it in encrypted form. This will require you to enter a passphrase when restarting your webserver process. But assuming that your webserver is stable, this would only be done when you are at the server anyway. It is quite common practice to store the private key in plain text so that it is only readable by root, but that is a severe vulnerability that would allow a remote attacker or somebody with physical access to the server to do a man-in-the-middle attack that does not need a certificate change.

(6) Use a 2048 bit private key

Because 1024 bit RSA private keys are considered too weak already.

A 2048 bit RSA private key for your sever is however enough. You want to avoid a needlessly longer key because SSL handshake performance exponentially degrades with increasing key length [source; test results].

2048 bits are enough because: With only forward secrecy cipher suits available, a broken or compromized private key of the server only means that an attacker can impersonate the server from that moment on (that is, can do a man-in-the-middle attack without causing a SSL certificate change). But he can not decipher past recorded communications. So the key strength has only to be enough to prevent brute force attacks during the certificate's validity. Which means 2048 bits until 2030, and 3072 bits afterwards [source]. But keep the validity period short of course (say, a year).

(7) Let your users monitor for SSL certificate changes

The problem with powerful surveillance bodies is that they are powerful: it is credibly alleged that three letter agencies can deploy man-in-the-middle attacks effortlessly on large scales by having backdoors in consumer DSL routers [source]. This also allows to compromise SSL connections, as follows. The router usually acts as a DNS server, and forwards requests to DNS servers of the telecommunications provider. When enabled by the backdoor, this allows DNS requests to be deflected to another DNS server, resulting (for example) in a site with a fraudulent SSL certificate being served to you (that will forward to the real site, but monitor your communications) [source, p. 23].

Such an attack might use a different SSL certificate signed by the same Certification Authority (CA) – or a different CA, it does not really matter since browsers by default do not notify users when a site's CA changed. But after all, the certificaate is different, and that is how it can be detected. Because: they simply cannot use these man-in-the-middle attacks on SSL for anything beyond targeted operations [source]. When used permanently, the attacks would be detected for example by site owners who notice the difference between wha certificate their site should serve to the world and what they see when visiting it in their browser.

In any case, it means that we should assume the CA-based certification mechanism to be completely broken, and should not rely on it. Until it is going to be replaced with a distributed, reliable mechanism (maybe a PGP style trust network? or registering the public key / domain name pairs on the Namecoin blockchain?), we have to make do with verifying the SSL certificate yourself (see below), or where this is too much effort, by tracking certificate changes.

Tracking certificate changes provides a decent protection against man-in-the-middle attacks, since these require exchanging the certificate, as shown above. You should monitor both the differences from what certificate everone else sees (using the Perspectives Firefox plugin) and from what certificate you saw in the past (using the Certificate Patrol Firefox plugin). From practical experience, Certificate Patrol is however not useful on large websites (Google, Facebook Twitter etc.) since these tend to use multiple certificates, from multiple CAs, and exchange them frequently. This makes using Certificate Patrol annoying; it would be better to have an option in it that lets you switch it on only for sites that you "do not want to be surveilled on".

(8) Let your users verify the SSL public key themselves

It is alleged that three-letter agencies collude with Certification Authorities (CAs) to get a second, different certificate signed by them for every new one they sign [source, p. 23]. Which will allow the man-in-the-middle attacks on SSL explained above. It is not known if this extends to any CA outside the US (where they can be subpoenaed …). But for practical purposes, let's just assume that all CAs are compromised.

A HTTPS certificate merely says that a website's traffic goes to whoever controls the domain plus whoever controls the CA. Even worse: since browsers by default do not notify a user when the HTTPS certificate changed and the new comes from a different CA, without manual checks at each page load a user can never be sure if the traffic does not perhaps go to certain three-letter agencies, as it can be safely assumed that at least some CAs are controlled by three-letter agencies. So, while HTTPS certificates are still good enough to exclude ordinary criminals, they are not up against massive surveillance.

(Also of course, you cannot trust the CAs yourself: never let them generate the private key for you, or upload your private key to them. With that precaution, even using a CA that is controlled by a three-letter agency is not a problem. They do not get you server's private key, they just sign your server's public key to attest it belongs to your server. Which is a correct statement, and not affected by being compromised. The compromised CA could create a duplicate certificate for a spy agency's own private key, but whether they use that or one from a different CA is just a matter of taste since a browser does not warn users against CA changes.)

In practical terms: You may proceed using a certfificate from a compromised CA for your "normal" website visitors, while at the same time also warning everyone that the HTTPS CA scheme is broken and users should not rely on it, instead should verify the SSL public key themselves. Tell your users to not simply trust the CA certificate represented by a happy Firefox displaying a lock to them. Tell them to verify by themselves, at the first time using the site, that the certificate is the one issued by the site operators instead of possibly a man-in-the-middle. Together with being alerted about certificate changes (see above), this provides a very decent protection against man-in-the-middle attacks.

Verifying can be difficult, depending on the size of your user community, but as tracking SSL certificate changes is usually enough by itself it does not have to be a very thorough vaidation. So you have different options. Here are some proposals, by increasing security:

  • Publish the fingerprint of the correct SSL certificate on the same website.
  • In the signup message, include the SSL certificate's fingerprint. This protects against later, dynamic modification of your website content by attackers.
  • As above, but also sign the signup message with the organization's GPG key, and publish the GPG public key on keyservers.
  • As above, but let users manually verify your GPG public key fingerprint (and also the current SSL certificate fingerprint while you're at it). This can be done without personal meeting of business card handovers etc., simply by creating a video live link, making sure it's not a pre-recorded video (some talking …) and letting the site representative both speak and sign the finger print characters at the same time. This of course mandates that the site representative is well-known, ideally by having met in person before.

(9) Use a self-signed certificate

With the above technique of still using CA-certified public keys but warning users against trusting them, most users would not see or simpply not follow the warning. They can be forced to deal with untrustable certificates though if you simply use a self-signed certificate. It will cause the browser to promt users to add a security exception, so they will have to verify the certificate before doing so.

(10) Let your users monitor IP address changes

If an attacker can get hold of the server's original SSL private key, he can impersonate the server without the tools above detecting an SSL certificate change. However, a similar tool would detect an IP address change. And you would announce the IP address of your server, and changes to it, for users to verify them, in analogy to how you want them to verify the SSL certificate fingerprint above. (And when you're at it, you might even want to switch to a self-signed certificate that would then also include the IP address, for free. At least on a second, synonymous domain or subdomain, for the users who know what they are doing.)

(11) Exchange your SSL private key and certificate frequently

Not sure about this one. It seems better to invalidate a SSL private key after a month than a year, which will prohibit an attacker from going on for long with man-in-the-middle attacks using your original private key that are undetectable by the above SSL certificate change monitoring. However, it introduces a task for users to manually verify the fingerprint of a new SSL certificate every month or so. Which is unrealistic for nearly all public-facing websites. Also it might not be needed because a man-in-the-middle attack could still be detected by IP address change monitoring proposed above. (However maybe even IP addresses can be bent by the secret services? I just don't know.)

(12) Put your server at a secure location

Even if you have encrypted your private key as stored on the server, there is a chance that it might be read from a memory dump. Which can be obtained when having physical access to your machine, or remote access to the virtualization system if you are on a VPS host (virtual private host). So at least do not rent a server in a country where three-letter agencies have easy access to company secrets, and also don't host at large hosting companies. Ideally of course, place the server physically at your home. And there into an intrusion-protected room. With lots of concrete around. In your basement. But ahh well … sorry, now I became paranoid about it all 😛