When you use FTP for recursive downloads or uploads and the FTP server has a firewall installed, you might get blocked due to "too many connections". Nearly all of those connections would be in TIME_WAIT state, and are only visible on the server side (not when you check on your client with e.g. netstat -anp --tcp | grep <ip-address>). The exact threshold of "too many" depends on firewall configuration (but can be something like 300 – 800). You will probably see no error message at all, and cannot reach the FTP server any more, either temporarily or permanently. What causes this problem, and how to avoid it?

First, this is a completely normal feature of the FTP protocol in passive mode: "Some existing protocols, such as FTP, make use of implicit signalling, and cannot be retrofitted with TIME_WAIT controls." – source; additionally, FTP does not provide for TCP connection reuse as HTTP/1.1 does, so we are indeed stuck with using one TCP socket for transferring one file via FTP). Perusing lots of server ports is not nice of the FTP protocol though (they stay in TIME_WAIT for up to four minutes!), since in extreme casesit can lead to the server runnig out of ports in the ephemeral port range for incoming connections. That's why the firewall is jumping in …

To understand how to work around this, we first have to know a bit about FTP active / passive mode and TCP TIME_WAIT:

"In active mode, the client establishes the command channel (from client port X to server port 21) but the server establishes the data channel (from server port 20 to client port Y, where Y has been supplied by the client).
In passive mode, the client establishes both channels. In that case, the server tells the client which port should be used for the data channel."
[Stackoverflow answer by paxdiablo, CC-BY-SA-3.0]

Now, TIME_WAIT is a state that a socket enters for around 4 minutes after a TCP connection was closed cleanly. It exists for two reasons (preventing delayed segments from being misinterpreted as part of a new connection, and reliable full-duplex connection termination, as explained in a great article in detail).

Fix 1: Use active mode FTP

The usual recommendation (like here) for avoiding being blocked by a firewall due to TIME_WAIT connections is to switch from passive to active FTP mode. Here's my (preliminary) understanding of why this works:

In the FTP protocol, one TCP data connection is used per file transfer, in both passive and active modes. However, in active mode the port used for these TCP connections on the server side is always 20, while in passive mode it is a random port number for each new connection, which will then be in TIME_WAIT state for 2-4 minutes after the connection ends. That is why "the downside to passive mode is that it consumes more sockets on the server [and] could eventually lead to port exhaustion" [source]. Not exactly though: a socket in TIME_WAIT state does not have to block the whole port from reuse, but only the exact socket (which  is a combination of two server addresses and two port numbers), though many operating systems naively implement it to block the whole port [source]. And in line with this naive way of implementing it, firewalls naively assume that the port is still blocked (even if it's not) and hence consider TIME_WAIT connections under somehow active connections. Now if you switch to active mode FTP, the server uses port 20 for all the outgoing data connections, with different sockets for each though. So the same number of sockets are consumed, and the same number are in TIME_WAIT state for the same amount of time, but since they all belong to one port, the firewall will not see it, and not block you.

This is obviously a relatively crude way of solving the issue. A better way would be firewall whitelisting (if accessible to you), or a deep inspection firewall with rules not counting TIME_WAIT connections resulting from passive mode FTP.

Fix 2: Limit the number of transfers per minute

If a firewall configuration allows us (say) 400 connections, and sockets stay in TIME_WAIT state for 4 minutes, it allows for 100 file transfers (each with its own TCP connection) per minute without being blocked. Technically, recursive download commands could support such a limit, but I am not aware of any FTP client that does. Not even lftp has this: it has the net:connection-limit option, but that is only for active connections. A connection that is in TIME_WAIT on the server is considered closed by the client already, so will not count into this limit.

However, a relatively nice approximation is this: Use the mirror --script=FILE command to just generate a script for downloading the files, file by file. Edit that script and insert lftp sleep 4m commands after a number of files that are close to triggering the firewall blocking. Then execute that script in lftp. All that could be wrapped in a nice script as well …

As an approximation, one might perhaps use a very restrictive data rate limit option to stay below the "files per minute" threshold. In lftp, that would be net:limit-total-rate (compare its manpage). Some people also reported that limiting the max. number of parallel active connections to one also helped [source] – it works the same way as rate limiting, but is far from guaranteed to help in your case, of course.

Yesterday, I nearly bricked my phone (HTC Desire HD), but got it back up and running again. Here is how – it is a relatively simple technique but I did not find it on the web, so I thought I'd share it.

Assume your phone (any Android phone for that matter) is in the following condition:

  • The bootloader works. Means, the HBOOT and FASTBOOT modes of the bootloader that appear when pressing "Volume down + Power" on a HTC Desire HD.
  • Android does not boot. This makes it impossible to unlock the Bootloader with the HTCDev method, since it usually requires a firmware update first that is done with a fully booted Android and a RUU .exe file on a Windows computer. For the same reason, installing any stock firmware in its regular RUU .exe distribution format is likewise impossible. It also makes it impossible to repair anything via adb of course.
  • The recovery solution does not work. For example because of a failed attempt to flash it, failing due to the "signatture verificattion" error mentioned above. So there's no ClockworkMod Recovery or 4ET Recovery Touch with Nandroid backup functionality that would allow you to restore Android to an earlier, working version. You also cannot flash a recovery solution because this is a custom flash that requires an unlocked bootloader.
  • The bootloader is locked. In this case, it still had "SHIP S-ON" showing at its top, means it is not posssible to flash any custom firmware etc. via fastboot since then it will turn that down with a "signature verification fail" error. For the same reason, you also cannot use the HBOOT PD98IMG.zip method for flashing a custom image (see below).
  • You cannot root or S-OFF your phone. That will be because most rooting methods require adb access or a booted Android version.

Solution: Install a stock firmware with the PD98IMG.zip method:

  1. Find a stock firmware for your phone in PD98IMG.zip format. For the HTC Desire HD, I tried PD98IMG-GB3.zip from this list, and it worked.
  2. Flash that stock firmware with HBOOT. That is, place the firmware to flash as PD98IMG.zip on the SD card and then boot your phone into the bootloader in HBOOT mode. It will detect the PD98IMG.zip and ask you if to flash it. And since it's a stock firmware, it will pass the signature verification. Usually, such a firmware also includes a (stock) recovery solution, so that will work too (though still without Nandroid backups …).
  3. Boot into Android. Should work again now.
  4. Root and S-OFF the device. Should be possible again now, since you have a working Android version again.
  5. Install a custom recovery solution. I recommend 4EXT Recovery Touch. Can be flashed now, since your bootloader is S-OFF now.
  6. Restore your system from a Nandroid backup. Works now, since you have a custom recovery solution that can handle Nandroid backups.

All software that I found (esp. the many Android apps developed for this purpose) use the webcam by taking a single picture and improving it digitally to be a "scan". This however leads to low resolutions. A better way is to use several pictures and stitch them together. This would be possible most comfortably with a dedicated software for documents from webcams, but that does not seem to exist so far. As an intermediate solution, let's use normal panorama software for this purpose.

Alternative 1: Panorama from images

My propsed scenario looks like this now, but I haven't tested or implemented it further so far:

  1. Add lighting. Add a light source that will illuminate the piece of your document that each webcam photo will cover. Such as a clipping a USB flashlight to the upper display edge of a notebook.
  2. Add your page on a stiff "background template" with markers for photo areas. Usually, one would scan A4 pages, so partitioning it in A7 elements and marking them on the "template" along the edges of the A4 page that you add in the middle will result in the option to take 8 photos per A4 page in a way that allows you to select the proper area for each, as each A7 section has an edge in common with the A4 page. 8 images at (say) 2 MPx webcam resolution results in 16 MPx per scan, which is more than enough.
  3. Create images. You would have a viewfinder with live video, hold up your page so that the next A7 section fits into the viewfinder, and click a key to shoot the next image. Ideally, the viewfinder should have an on-screen display showing a somewhat smaller rectangle for placing the A7 area in, as the edges will then be used for overlap finding when stitching the images together.
  4. Get the images stitched together. This should be a fully automated process and is possible with one of the many "panorama maker" software applications out there. Some are available for Linux.

Alternative 2: Panorama from video

This is what I tried at first: creating a panning video of your document, grabbing it with the webcam, and using video-to-panorama software to create a high-resolution image from it. However I now think that there are inherent quality limitations, as panning video images are never as sharp as still images since the objects are in motion. However, for other purposes this is a worthwhile technique, and I explored it in its own blog post "How to create a panorama from video under Linux?".

Option 1: Autopano Pro

Autopano Pro is a great sophisticsted panorama software that comes with a Linux version (.deb package even!). It is commercial software for about 100 USD, but has a demo version that will do about the same, just watermarking the generated panoramas.

They have a an article "Fun : Stitching video frames" that explains in detail how to create a panorama from a video. I was successfully able to do it like this, on Ubuntu 14.04.

Option 2: Using Panorama Maker

Panorama Maker is a gratis, Windows and Mac software  that can create panoramas from videos automatically. I was successfully able to run the software under Linux using Wine, but could not get the video-to-panorama feature to work. Probably I just did not know the correct video format to use.

For HTML files, you can use the command-line tool html2text (even with several files at once), together with wc:

html2text *.html | wc --words

For XML files, you can use the command-line tool xml_grep instead, again together with wc. It allows to specify an XPath-like expression to define which parts of the XML text to count. Examples for TMX files:

xml_grep --text_only --cond "tuv[@xml:lang='de']/seg" languages.tmx | wc --words

xml_grep --text_only --cond seg languages.tmx | wc --words

I also shared this solution at Stackoverflow; see there for alternative approaches as well.

The mouse stopped clicking at times – it seemed to happen at arbitrary moments. The pointer would still move, but it was no longer possible to click in any way (on my ThinkPad X201 Tablet meaning: trackpoint buttons, touchpad buttons, touchpad tapping, pen input, finger input). The reason for this behavior was probably a defective keyboard (which includes the trackpoint). This post is about temporary workarounds for this issue when it happens though.

Restart OpenBox

The so far least invasive way to re-initialize the mouse (compared to restarting X or rebooting) is to restart the window manager. When you can open a terminal without the mouse working, use this command in a X virtual terminal window [source]:

openbox --replace

Alternatively, switch to a console (Ctrl+Alt+F1) and use this command:

DISPLAY=:0.0 openbox --replace

 

The idea explored here is to separate your data physically from the computer that you are using them on. This has the following advantages: (1) you are quite safe from theft because you can keep carrying your data with you always, even when having to let your notebook at home or in a hotel because it is too heavy; (2) you can use your data on any computer (like a friend's computer when travelling, an Internet cafe computer etc.); (3) you can hand over data items quickly and easily because every computer has a USB interface.

You have to decide if the this idea is worth its benefits for you, as all solutions below are much slower and more expensive than a bigger standard SATA III SSD (as of 2014-06, about 100 EUR new incl. shipment for a 256 GB SSD, with data transfer rates up to 600 MB/s, though the continuous transfer rates have to the researched).

However, here are the solutions I could come up with. By adequacy for my requirements – which included to be 128 GB or more in size:

  • SD card in SD card reader. This is the bes option on notebooks that come with a card reader, since it does not block any ExpressCard expansion port (reading your digital camera card is easily possible in the USB ports as well with SanDisk Ultra II SD Plus USB cards). However, it has to be tested still if the typical notebook card reader can utilize the speed of highest speed SD cards used here (else a ExpressCard/34 based card reader would be needed). Because for proper speed, we need the latest "U3" class SD cards. These are just being introduced around 2014-06, so prices will drop lateron. Currently available products:
  • SD card in ExpressCard/34 card reader. For data exchange, all readers can expose a SD card as a USB mass storage device, so no drivers on the host system are needed. The additional advantage of using a SD card is that you can also use them for other purposes (recording photos and HD video etc.). For proper speed, we need the U3 speed class cards again, see above.
  • Compact Flash card in ExpressCard/54 card reader. This is a good solution, since the speeds of Comact Flash cards are comparable to the ExpressCard SSDs while the Compact Flash format makes it simpler to find an adapter to USB mass storage (namely, a simple card reader). However, Compact Flash does not fit by width into ExpressCard/34, so your notebook will need the larger ExpressCard/54 slot [see]. Speeds of Compact Flash are great, and for data exchange you can carry a USB card reader that exposes the Compac Flash card as USB mass storage (means it needs no driver). But high-speed, large-capacity cards are also expensive; recommendable products:
    • A moderately priced one is the Hama CF 128 GB (up to 150 MB/s read, so maybe 40 MB/s continuous write?) for about 145 EUR.
  • ExpressCard/34 form factor SSD, plus USB adapter. This is the best option in terms of speed. It can be connected to other computers using a USB to ExpressCard adapter (overview; also see DeLock 61714 etc.). ExpressCard allows cards to use an USB 2.0, USB 3.0 or PCIe mode, and not all these adapters also support PCIe (the Digigear PCU10 does however, but needs manual configuration for PCIe mode). So check first which mode is used by the ExpressCard SSD – anything above 60 MB/s means USB 3.0 or PCIe mode. For the connectors etc., see the ExpressCard standards document.
    Available products 64 GB and larger:​

     

  • Small "knobby" USB flash drive. If your notebook has a USB 3.0 port or you can add one via an ExpressCard/34 slot or similar, the achievable speed of this solution can be good, too. It seems that there is still no 128 GB flash drive in "knob" size. A hack to get over this would be to combine two of them physically so that they fit exactly into two adjacent USB ports on your notebook (and with a small cable-type USB hub to other computers' USB ports). Available options:
  • microSD card in "knobby" microSD card reader. This option works even up to 128 GB (with 128 GB microSD cards being available). However, these cards tend to be slow. Available models: just one so far
  • Compact Flash card in 1.8" or 2.5" SATA drive bay. This is essentially using a Compact Flash card as a removable, rugged SSD. There are Compact Flash to SATA adapters for both 1.8" and 2.5" SATA SSD form factor (example: StarGat SAT2C). Also this solution will work only if your notebook can be modified in a way to allow drawing out the card from the HDD bay. And it's not really clear how the computer or operating system might react when extracting the Compact Flash card for example during ACPI Suspend mode. Also when using the system disk bay for this type of SSD, we cannot physically separate operating system and personal files as intended here.
  • miniPCIe SSD in ExpressCard and SATA external enclosure. This is more a theoretical option since you do not want to handle a bare PCB with your data on it just to connect it to a computer that does not have an ExpressCard slot. But it would work, as there are converters for miniPCIe to ExpressCard and miniPCIe to SATA.