Powerline is a technology for transmitting data over the AC grid. All devices provide an RJ45 Ethernet plug (and some also wifi and USB), so they support client devices of all operating systems. Basic configuration of encryption is likewise OS independent, as nearly all of them use a "pairing" button for that. However, to set the encryption key manually, to make more detailed settings, to read out speed statistics and to update the firmware most come with configuration tools that only work under Windows.

So, can we find a device that can be fully managed under Linux, ideally with free software? After a long search, this turns out to be a simple task. Just use any device that supports the Homeplug AV or Homeplug AV2 standard, and manage it with Open PLC Utils, a free software full-featured management tool developed by AV/AV2 chipset maker Atheros. It  allows full device configuration, info reading and upgrading for all INT6000/6300/6400/AR7420 devices, even tampering with the parameter set in the firmware. The only alternative are DS2 chipset based devices, since at least most of them can be fully configured via a built-in web interface (such as the COMTREND Powergrid 9020, their most modern device). However, a tool like Open PLC Utils is preferred here since it effectively replaces a part of the DS2 management software with free software, allowing scripting, modifications etc., and is more powerful in general.

Recommendable devices

It is said that it does not matter so much in terms of connection speed and quality which exact device you choose, as long as you choose the right chipse. Here, we focus on the Homeplug AV chipsets (INT6000, INT6300, INT6400) and there on the most modern one (INT6400). We avoid the Homeplug AV2 chipsets because they seem not yet mature (see below) and the DS2 chipsets because they are not as readily configurable under Linux.

Recommendation for Germany: Speedport Powerline 100. The best recommendation is so far: Telekom Speedport Powerline 100. It has the INT6400 chipset (most modern one for the Homeplug AV 200 Mbit/s devices), and it is dirt cheap (ca. 20 EUR incl. shipment for a pair of them, via ebay.de) because Telekom made Germany awash with these devices. It also is good quality, since it is a whitelabel product from well-known French brand LEA Networks – see also the very detailed test on tomsnetworking.de (in German). One can use the LEA-made software tool SOFTPLUG for settings beyond simple pairing for encryption, but it runs only under Windows. It is not needed however, as the free software tool Open PLC Utils can likewise be used, also running under Linux.

Discussion of alternatives: Devolo dLAN AVsmart+. These devices are quite nice, as they have an extended status display on the device – compare the manual. They are also readil available in used condition on eBay, but not as dirt cheap as the Speedport Powerline 100. However, they use the older INT6000 chipset [source], leading to a third less speed in long-range applications [source].

Discussion of alternatiives: COMTREND PowerGrid 9020. This is not a Homeplug AB/AV2 device, but uses the non-interoperable DS2 chipset. Reviews on Amazon are good. However, it can be fully managed under Linux as all settings are available within a web interface, as argued for and against above. The Powergrid 9020 is the most modern Powerline device of Madrid-based manufacturer COMTREND so far, and avaible in UK and Europe type plug versions. The UK plug versions are very readily available online, as they are provided by UK based large ISP Britich Telecom (BT) to their customers. The Europe plug versions are quite hard to find though (but here are some). Also see the installation guide and the  full manual.

Discussion of alternatives: Homeplug AV2 devices. There are the even more modern 500 Mbit/s Homeplug AV2 devices (using the AR7420 etc. chipsets), however as of 2014-09 this generation of technology seems to not be mature still, often suffering from connection breakdowns, low throughput, the devices running hot etc. (as judged from reviews on Amazon). So we avoid them here, also because they are still more expensive on the second-hand market. But your priorities and mileage may vary. They also can be fully managed with Open PLC Utils.

Linux software for powerline adapters

While Open PLC Utils is clearly the winner, the following list is all the Linux-based powerline software I came across:

  • Open PLC Utils. As said, clearly the winner: full-featured and free software.
  • Faifa. A manufacturer independent, free software Homeplug AV/AV2 utility for Linux. Allows low-level functions such as control frame dumps. Considered to be the successor to the older plconfig utility, see below.
  • plconfig. An older, simple Linux based configuration utility. For download on Github. Superseded by Faifa now.
  • dlanlist, dlanpasswd. Open source software that can be downloaded and compiled under Linux and is meant to list (and set the password of) Devolo dLAN devices. Since these conform to the Homeplug AV/AV2 standard, the software might be used to also configure other devices, after some simple modifications.
  • devolo Cockpit. A large configuration softwre for Devolo dLAN devices. Seemingly not available as a source version, so adaptations to other devices are not possible.
  • Intellon device manager for 3.x firmware (Windows software). Allows full management of INT6x00 chipset devices when using a 3.x firmware version, incl. editing firmware parameters. Its use is described in this article.
  • Intellon device manager for 4.x firmware (Windows software). In contrast to the version for 3.x firmwares, this does not support full management of the of INT6x00 chipset devices any more.

Background information

The following in-depth articles provide relevant background information about Powerline technology and their successful use:

Modding and hacking powerline adapters

At 10 EUR for a used device or less, powerline adapters are so cheap that they offer themselves to several non-standard uses. At least teh following come to mind:

  • Powerline noise filters. Devices that include an AC plug with noise filter (such as the SpeedportPowerline 100 recommended above) are the cheapest option to use them as frequency filters to prevent contamination of the 50 Hz frequency from other digital devices, mobile phone chargers etc..
  • Powerline bridge. It is also possible to create a simple bridge by connecting two of them with a crossover Cat5e cable (or a bridge device in between, if necessary). This should allow crossing phase boundaries in the home cabling, if necessary.
  • VDSL P2P modem replacements. And then, of course, these cheap devices are a natural candidate for creating a cheap DIY replacement for pairs of VDSL modems (usually 150 EUR per piece!). It works by connecting twisted phone wire to the signal (via soldering) before it gets modulated on the AC mains power. It is possible to use this for transmitting data over several hundred meters at least.

 

When you use FTP for recursive downloads or uploads and the FTP server has a firewall installed, you might get blocked due to "too many connections". Nearly all of those connections would be in TIME_WAIT state, and are only visible on the server side (not when you check on your client with e.g. netstat -anp --tcp | grep <ip-address>). The exact threshold of "too many" depends on firewall configuration (but can be something like 300 – 800). You will probably see no error message at all, and cannot reach the FTP server any more, either temporarily or permanently. What causes this problem, and how to avoid it?

First, this is a completely normal feature of the FTP protocol in passive mode: "Some existing protocols, such as FTP, make use of implicit signalling, and cannot be retrofitted with TIME_WAIT controls." – source; additionally, FTP does not provide for TCP connection reuse as HTTP/1.1 does, so we are indeed stuck with using one TCP socket for transferring one file via FTP). Perusing lots of server ports is not nice of the FTP protocol though (they stay in TIME_WAIT for up to four minutes!), since in extreme casesit can lead to the server runnig out of ports in the ephemeral port range for incoming connections. That's why the firewall is jumping in …

To understand how to work around this, we first have to know a bit about FTP active / passive mode and TCP TIME_WAIT:

"In active mode, the client establishes the command channel (from client port X to server port 21) but the server establishes the data channel (from server port 20 to client port Y, where Y has been supplied by the client).
In passive mode, the client establishes both channels. In that case, the server tells the client which port should be used for the data channel."
[Stackoverflow answer by paxdiablo, CC-BY-SA-3.0]

Now, TIME_WAIT is a state that a socket enters for around 4 minutes after a TCP connection was closed cleanly. It exists for two reasons (preventing delayed segments from being misinterpreted as part of a new connection, and reliable full-duplex connection termination, as explained in a great article in detail).

Fix 1: Use active mode FTP

The usual recommendation (like here) for avoiding being blocked by a firewall due to TIME_WAIT connections is to switch from passive to active FTP mode. Here's my (preliminary) understanding of why this works:

In the FTP protocol, one TCP data connection is used per file transfer, in both passive and active modes. However, in active mode the port used for these TCP connections on the server side is always 20, while in passive mode it is a random port number for each new connection, which will then be in TIME_WAIT state for 2-4 minutes after the connection ends. That is why "the downside to passive mode is that it consumes more sockets on the server [and] could eventually lead to port exhaustion" [source]. Not exactly though: a socket in TIME_WAIT state does not have to block the whole port from reuse, but only the exact socket (which  is a combination of two server addresses and two port numbers), though many operating systems naively implement it to block the whole port [source]. And in line with this naive way of implementing it, firewalls naively assume that the port is still blocked (even if it's not) and hence consider TIME_WAIT connections under somehow active connections. Now if you switch to active mode FTP, the server uses port 20 for all the outgoing data connections, with different sockets for each though. So the same number of sockets are consumed, and the same number are in TIME_WAIT state for the same amount of time, but since they all belong to one port, the firewall will not see it, and not block you.

This is obviously a relatively crude way of solving the issue. A better way would be firewall whitelisting (if accessible to you), or a deep inspection firewall with rules not counting TIME_WAIT connections resulting from passive mode FTP.

Fix 2: Limit the number of transfers per minute

If a firewall configuration allows us (say) 400 connections, and sockets stay in TIME_WAIT state for 4 minutes, it allows for 100 file transfers (each with its own TCP connection) per minute without being blocked. Technically, recursive download commands could support such a limit, but I am not aware of any FTP client that does. Not even lftp has this: it has the net:connection-limit option, but that is only for active connections. A connection that is in TIME_WAIT on the server is considered closed by the client already, so will not count into this limit.

However, a relatively nice approximation is this: Use the mirror --script=FILE command to just generate a script for downloading the files, file by file. Edit that script and insert lftp sleep 4m commands after a number of files that are close to triggering the firewall blocking. Then execute that script in lftp. All that could be wrapped in a nice script as well …

As an approximation, one might perhaps use a very restrictive data rate limit option to stay below the "files per minute" threshold. In lftp, that would be net:limit-total-rate (compare its manpage). Some people also reported that limiting the max. number of parallel active connections to one also helped [source] – it works the same way as rate limiting, but is far from guaranteed to help in your case, of course.

Yesterday, I nearly bricked my phone (HTC Desire HD), but got it back up and running again. Here is how – it is a relatively simple technique but I did not find it on the web, so I thought I'd share it.

Assume your phone (any Android phone for that matter) is in the following condition:

  • The bootloader works. Means, the HBOOT and FASTBOOT modes of the bootloader that appear when pressing "Volume down + Power" on a HTC Desire HD.
  • Android does not boot. This makes it impossible to unlock the Bootloader with the HTCDev method, since it usually requires a firmware update first that is done with a fully booted Android and a RUU .exe file on a Windows computer. For the same reason, installing any stock firmware in its regular RUU .exe distribution format is likewise impossible. It also makes it impossible to repair anything via adb of course.
  • The recovery solution does not work. For example because of a failed attempt to flash it, failing due to the "signatture verificattion" error mentioned above. So there's no ClockworkMod Recovery or 4ET Recovery Touch with Nandroid backup functionality that would allow you to restore Android to an earlier, working version. You also cannot flash a recovery solution because this is a custom flash that requires an unlocked bootloader.
  • The bootloader is locked. In this case, it still had "SHIP S-ON" showing at its top, means it is not posssible to flash any custom firmware etc. via fastboot since then it will turn that down with a "signature verification fail" error. For the same reason, you also cannot use the HBOOT PD98IMG.zip method for flashing a custom image (see below).
  • You cannot root or S-OFF your phone. That will be because most rooting methods require adb access or a booted Android version.

Solution: Install a stock firmware with the PD98IMG.zip method:

  1. Find a stock firmware for your phone in PD98IMG.zip format. For the HTC Desire HD, I tried PD98IMG-GB3.zip from this list, and it worked.
  2. Flash that stock firmware with HBOOT. That is, place the firmware to flash as PD98IMG.zip on the SD card and then boot your phone into the bootloader in HBOOT mode. It will detect the PD98IMG.zip and ask you if to flash it. And since it's a stock firmware, it will pass the signature verification. Usually, such a firmware also includes a (stock) recovery solution, so that will work too (though still without Nandroid backups …).
  3. Boot into Android. Should work again now.
  4. Root and S-OFF the device. Should be possible again now, since you have a working Android version again.
  5. Install a custom recovery solution. I recommend 4EXT Recovery Touch. Can be flashed now, since your bootloader is S-OFF now.
  6. Restore your system from a Nandroid backup. Works now, since you have a custom recovery solution that can handle Nandroid backups.

All software that I found (esp. the many Android apps developed for this purpose) use the webcam by taking a single picture and improving it digitally to be a "scan". This however leads to low resolutions. A better way is to use several pictures and stitch them together. This would be possible most comfortably with a dedicated software for documents from webcams, but that does not seem to exist so far. As an intermediate solution, let's use normal panorama software for this purpose.

Alternative 1: Panorama from images

My propsed scenario looks like this now, but I haven't tested or implemented it further so far:

  1. Add lighting. Add a light source that will illuminate the piece of your document that each webcam photo will cover. Such as a clipping a USB flashlight to the upper display edge of a notebook.
  2. Add your page on a stiff "background template" with markers for photo areas. Usually, one would scan A4 pages, so partitioning it in A7 elements and marking them on the "template" along the edges of the A4 page that you add in the middle will result in the option to take 8 photos per A4 page in a way that allows you to select the proper area for each, as each A7 section has an edge in common with the A4 page. 8 images at (say) 2 MPx webcam resolution results in 16 MPx per scan, which is more than enough.
  3. Create images. You would have a viewfinder with live video, hold up your page so that the next A7 section fits into the viewfinder, and click a key to shoot the next image. Ideally, the viewfinder should have an on-screen display showing a somewhat smaller rectangle for placing the A7 area in, as the edges will then be used for overlap finding when stitching the images together.
  4. Get the images stitched together. This should be a fully automated process and is possible with one of the many "panorama maker" software applications out there. Some are available for Linux.

Alternative 2: Panorama from video

This is what I tried at first: creating a panning video of your document, grabbing it with the webcam, and using video-to-panorama software to create a high-resolution image from it. However I now think that there are inherent quality limitations, as panning video images are never as sharp as still images since the objects are in motion. However, for other purposes this is a worthwhile technique, and I explored it in its own blog post "How to create a panorama from video under Linux?".

Option 1: Autopano Pro

Autopano Pro is a great sophisticsted panorama software that comes with a Linux version (.deb package even!). It is commercial software for about 100 USD, but has a demo version that will do about the same, just watermarking the generated panoramas.

They have a an article "Fun : Stitching video frames" that explains in detail how to create a panorama from a video. I was successfully able to do it like this, on Ubuntu 14.04.

Option 2: Using Panorama Maker

Panorama Maker is a gratis, Windows and Mac software  that can create panoramas from videos automatically. I was successfully able to run the software under Linux using Wine, but could not get the video-to-panorama feature to work. Probably I just did not know the correct video format to use.

For HTML files, you can use the command-line tool html2text (even with several files at once), together with wc:

html2text *.html | wc --words

For XML files, you can use the command-line tool xml_grep instead, again together with wc. It allows to specify an XPath-like expression to define which parts of the XML text to count. Examples for TMX files:

xml_grep --text_only --cond "tuv[@xml:lang='de']/seg" languages.tmx | wc --words

xml_grep --text_only --cond seg languages.tmx | wc --words

I also shared this solution at Stackoverflow; see there for alternative approaches as well.

The mouse stopped clicking at times – it seemed to happen at arbitrary moments. The pointer would still move, but it was no longer possible to click in any way (on my ThinkPad X201 Tablet meaning: trackpoint buttons, touchpad buttons, touchpad tapping, pen input, finger input). The reason for this behavior was probably a defective keyboard (which includes the trackpoint). This post is about temporary workarounds for this issue when it happens though.

Restart OpenBox

The so far least invasive way to re-initialize the mouse (compared to restarting X or rebooting) is to restart the window manager. When you can open a terminal without the mouse working, use this command in a X virtual terminal window [source]:

openbox --replace

Alternatively, switch to a console (Ctrl+Alt+F1) and use this command:

DISPLAY=:0.0 openbox --replace