I am back in Germany after four months in Nepal. First impression? Germany is well-organized, super quiet, and populated by zombies. The living dead, you know. (Oh and, Germany is so flat. Very strange. Why u not hilly?)

It just does not make sense. If something like Germany is the endpoint of developing areas' development, what is development for? It has no meaning at all. Essentially, after all the development, people in Nepal would be

  • just as unhappy as people in Germany
  • all their earnings will be used up for so-called "living"
  • people won't talk to people anymore
  • there will be just as few options for alternative lifestyles as in Germany, where this is systematically minimized by forcing people into consuming and paying
  • people will work in brainwashing, marketing, SEO etc. to force-feed each other the useless products of their industries … consuming up the world's resources …

Essentially, all the Germans (and Europeans and U.S. Americans, for that matter) do is keeping their country up and runnning, with insane amounts  of energy and worktime. Their countries are "optimized" to maximize the amounts of energy and worktime to invest for just keeping the country up and running. The first is called "economic growth", the second "full employment". And all these aspirations to "wealth" that is not wealth and "wellbeing" that is not wellbeing in people make them miserable. Directly and indirectly. They never have enough. They are brainwashed by advertising since decades, and the advertising comes from capitalist monopolies whom they serve with their lifetime and money.

And then we have the audacity to call ourselves a "developed nation", to be imitated by the "developing nations" … . These "developed" nations do not contribute any significant amount to a better future for the planet. They are not engaged in any concerted effort to construct a meaningful and sustainable future. Instead, they just consume. And much more than their share. While villages in Nepal are much more sustainable, since most stuff is made from fieldstones and bamboo. In terms of our global future, that is more developed than the developed "nations" … . But currently, developed nations devour the planet, and fast. Wow. (Now the good thing about Nepal is, it has so many so remote places that global capitalism will not reach them in my lifetime cheeky Like Rolpa. It currently takes three days and nights of travelling by bus from Kathmandu.)

Ok … so if development according to the paradign of "developed" nations makes no sense, what makes sense?

  • Minimize resource and worktime usage in "developed" nations, to the levels in Nepal and below. No more economic growth, no more full employment. There is such a ton of interesting work to do for that, including medical research, frugality research, full transition to a minimal energy economy, etc. etc..
  • Real development has to be integrated. Like, set up and keep up a sustainably developed commune in a remote village in Nepal, or transform a whole village into such a sustainable model village with both low resource usage and a high standard of living. Serving as an example to be imitated in the thousands of other villages. It means for an educated group of people (from Nepal, from abroad, or mixed) to go there to live there. For the long term. To make that commune able to sustain itself through sustainable production. And help it minimize the resource and time efforts needed for that. (I collected a lot of relevant tech and knowledge for small-scale self-sustainable living in EarthOS already.)

For a crisis-mapping project after the 25 April 2015 earthquake in Nepal, we needed a collaborative online database that is easy to set up and maintain. I found the product on obvibase.com to be the best existing solution. Much much better than the misuse of Google Spreadsheets that people usually engage in.

However, the Obvibase software is not free software, and it is not perfect either. So here's my list of improvements that I just submitted to them as feedback, after a month of frequent and in-depth usage of their database. If somebody wants to do so: a free software clone of obvibase.com plus the suggestions below would get us pretty close to the best collaborative ad-hoc database solution ever smiley

  • Alt+Arrow Left should work for "page back", but does not.
  • The "Main menu → More actions … → Restore …" action's form should list the person who has done an edit in another column, and also what changes were done (showing the before and after version of the affected cell).
  • For better privacy protection and because people are used to it already, sharing should work like in Google Docs (adding Google Accounts who get access, with different access levels per account). Otherwise, sharing the access link accidentally gives people read-write access, which cannot be revoked again.
  • The "Page Up" and "Page Down" keys should work in the list view.
  • Pressing the space bar on the first selectable column (with the double arrow) should select the record, adding a checkmark to the very first column.
  • It should be possible do do simple styling of columns (background color, font color, bold font, italics font). Then, one can mark up one column as more important, and speed up visual navigation.
  • Column titles should be formatted in bold and / or with a background color, to speed up visual navigation on the screen.
  • There should be a way to create a new database in a new tab. If the menu link is a normal link that can be opened in a new tab, it will be enough.
  • There should be a sharing mode where people with the link can add records and edit or delete their own records, but noth others. It will have to require login with a Google Account. Because, the general read-write access mode link can not be shared publicly to avoid access by destructive individuals who might delete every record, or overwrite it with spam.
  • In the web-published, read-only version, long text cells should have an on-hover box showing the full text, as done for all other versions.
  • It should be possible to mark individual columns as non-public in the column settings, which would exclude them form display in the public version that is read-only accessible to all web visitors. This would allow to collect public and confidential information (like, personally identifiable information) in the same database.
  • Full export to CSV. There is currently no simple way how to export and re-import the full database incl. all nested records to one or some CSV files. It is however needed for making local backups. The problem is that exporting the main database table misses out records from any nested table, and exporting a nested table misses out records from any other nested table, and also any record from the main database table that does not have a record in the exported nested table.
  • CSV exporting from nested tables should not repeat records from the main table. This happens currently, but redundancy is most always a bad idea. Instead, the nested record should refer to the ID of its parent record ("foreign key relation").
  • SQLite3 export. Would be good for offline usage, as an interface to other tools, as a way to run complex SQL queries on the dataset, for comfortable backups (unlike CSV export, which misses out some data) etc..
  • SQLite3 import, including reconciliation. To allow people "in the field" to contribute without Internet access, there should be an SQLite3 import feature. Concurrent edits would have to be reviewed before finishing the import.
  • ODS export. Not that important, but would be good for having an alternative to the SQLite3 format export. The export would include multiple sheets, one for the main database and one for each nested table. Embedded scripts would be used for filtering when following a link from the main database table to the associated records in the nested table.
  • When pressing Ctrl+V in a cell while it is not in edit mode, the current clipboard content should be entered into the cell. Currently, the Paste / Import window is shown, which is confusing because Ctrl+V is a clipboard operation in all applications, so people will use it intuitively (forgetting to go to "Edit mode" before). In the current way it functions, there are also two issues: iun addition to showing the Paste / Import window, currently "v" is inserted into the cell when pressing "Ctrl+V", and the focus in on the cell rather than in the text field in the Paste / Import window (which prevents the Esc key to be usable for closing that window).
  • When trying to change the filter value for a text column by clicking on the header, the caret is initially at the beginning of the value. It would be more useful to position it at the end. More importantly, even though the caret is in the filter value text box, the "Pos1" and "End" keys navigate to the first and last menu entry instead. They should however move the cursor.
  • To reset a text column to "no filter value", it should also be possible to click the column header, delete the filter value in the text field and then press the "Apply" button or Enter key. Because that is what people intuitively expect. Currently, a message will appear saying "No text to search for".
  • Search by transliteration and equivalent Latin characters. To find the records one is looking for fast, there should not be a need to enter special characters. For example, when a record contains the name "Matjaž", it should be possible to find it via "Matjaz" and also by any transliteration of "Matjaž" to basic Latin characters.
  • In case of a synchronization error, currently a message will appear that "some records have been rolled back". This means data loss, maybe for max. 30 s of editing. It is not due to concurrent edits, but rather due to intermittent connectivity problems. Instead of causing data loss, another solution should be possible. Like for example, asking if syncing should be tried again, and finally offering the rolled back data in CSV format to copy and add it again quickly.
  • The database becomes hard to navigate if there are many columns, exceeding the screen width. For this quite frequent case, the database should allow several lines per record, distinguishing between vertically stacked columns by using different text formats.
  • The textbox showing values for multiple-choice columns has too big paddings left and right, causing line breaks where there should not be any. Also, it should be always at least the same width as the column itself, for the same reason.
  • When changing selectable values in the settings dialogue for multiple choice columns, it should be possible to delete existing values and create new ones, or to edit existing values (which will edit all records that use the old value accordingly, saving a lot of work).
  • For powerusers, the filter field when clicking on column headers should allow regexp searches (and storing them in a "recent regexp searches" sub-menu). This can help with analysing columns that have comma-separated values, hierarchical text values and other constructions in them.
  • URLs in all text columns should be automatically made into clickable hyperlinks (and automatically shown abbreviated when not in edit mode). Else, it's additional work to go into edit mode and select and copy the URL manually.
  • Subtable records should be reachable with no or very little additional work. This is esp. important when storing contact information there, as it frequently contains hyperlinks. Proposal: When hovering over the "[…] records" entry that links to a subtable, an on-hover window should appear that shows the subtable records, incl. columns, hyperlinks etc..
  • Currently, reverting to a prior version (via "Main menu → More actions → Restore …") will lead to a database that uses a new URL, without notifying the team about this. This means starting a new branch from the version one reverted to, without knowing, and without a way to merge changes into the main branch that the rest of the team is working on. This should be considered a bug.
  • It would be good to be able to see who's editing the database at the same time. Just like in Google Docs and Google Spreadsheets, where there are user icons showing up in the top right then. It could be the same user icons as used for the Google accounts, since login happens via a Google account anyway.
  • In the form to create a comment, it should be possible to press "Ctrl + Enter" instead of clicking "Save" with the mouse. It's faster. And like it's done in the Google Doc comments as well.
  • For more comfortable and faster visual navigation in long tables, it should be possible to style each column in the column settings separately. A combination of several options for influencing the style would be available: font size, font weight, font color, overflow / line break behavior etc..

This draft-state article explores the implications of pilot-wave theory (see) on the "big questions" in life, assuming that pilot-wave theory is indeed adequate and that the universe is deterministic on a physical level.

Pilot-wave theory is simply chaos theory taking to the particle level: the deterministic laws of nature are not out of force, but "everything interacts with everything else", so we can't see them in action because it's all a mess …

Note that pilot-wave theory requires a universe with infinite resolution for the amplitude of the pilot wave function, and potentially also infinite resolution on the time scale, as else all the observed "randomness" cannot be explained.

It might be a good imagery to thing of the universe as a three-dimensional pulsating field of energy, with mass particles being nothing but reasonably stable artifacts of energy, like the droplets in those pilot-wave droplet experiments. In the end, mass is just energy, and can be converted back to energy (even elementary particles, by combining with their antimatter particles). That energy then leads to higher local pulsing of the energy field, creating "splashes" that then again become reasonably stable artifacts of energy (particles). Not all energy is bound in particles though (see "dark energy").

Maybe Planck's constant h can be interpreted as the basic "frequency" that is applied to the energy of the universe, in analogy to the frequency to apply to the silicone oil bath in pilot-wave droplet experiments to enable stable droplets? Because h indeed influences the masses of all elementary particles that can exist. But if that's the case, who applies or applied this frequency to the universe's energy? God?

Also, if the universe is deterministic, there cannot have been a "Big Bang" causes by a random quantum fluctuation of nothing. There has to be a reason for the universe instead. That's why putting the pilot-wave theory to the test and finding out if it is true is so important.

Another interesting implication of pilot-waave theory is that the universe is deterministic, but non-predictable at the same time. Assuming that the energy field (or whatever carries the wave function of the universe) is objective reality that exists independently from God, and that God "only" implemented matter in a certain configuration with this energy on creation, it would imply that not even God can predict the course of the universe. This (somewhat silly) nothing would follow form the Heisenberg uncertainty principle: "The positions and momenta of the particles are considered to be the hidden variables. However, the observer not only doesn't know the precise value of these variables, but more importantly, cannot know them precisely because any measurement disturbs them – as stipulated by the Heisenberg uncertainty principle." [source].

Assuming pilot-wave theory, the whole course of actions in the universe would be determined by the initial positions and momenta of particles. Or rather, of wave functions, as there can be empty wave functions [source]. The initial phase space of the wave functions would encode the whole history of the universe, then including the abiogenesis of life. In total, the initial states in the phase space could be randomly distributed, according to the quantum equilibrium hypothesis [source]. What is relevant is not  the distribution of phase states, but instead each individual phase state at each individual point of space. These, taken together, would be the variables that deterministically define the whole history of the universe at particle level. Or taken the other way round, by choosing a proper configuration of these (if you were God you could), you could choose any (or at least one of many different) history of the world that is possible within the laws of nature. The Heisenberg uncertainty principle would not be a limiting factor, as it just prohibits you from measuring the phase space values in action, not from setting them in the first place. But, such a clockwork universe would be a boring one … now free will!

Both pilot-wave theory and orthodox Copenhagen interpretation quantum theory are interpretations for the exact same experiment measurements. So for predicting results and for engineering, both are just as good, and preference would be a matter of style. The real difference is however the question if the universe is deterministic, or not. But even that has not too huge metaphysical interpretations, as we show below that free will is initially incompatible with both interpretations.

Can there be "free will" in a deterministic universe?

The naive notion of free will is of no use, and makes no sense. It assumes a monadic, atomic source of high-level decisions / goals, coming out of that source "out of free will" (which for an observer is equivalent to saying "for no reason", "guided by pure randomness"). Such a source, if existing, would be no useful notion of free will at all, as (1) randomness or unreasonableness that the "self" cannot influence cannot be a justification to judge the self for ethical / unethical behavior and more importantly (2) decisions are always complex results of complex processes, so you need a system to make them, not a transcendent atomic "source" that can, if anything, output a binary value.

However, there is room for a meaningful notion of "free will" even in a deterministic universe. As follows:

  • Like atoms etc. are reasonable stable energy wave patterns in the pilot wave world, memes (thought systems, visions etc.) are reasonably stable artifacts of brains. This is esp. true for "conscience", the emergent condition of "knowing, processing and understanding" enough about the world as to be able to be aware of ones own existence as part of the world, ones own options to interact with the world, and their potential outcomes.
  • Given conscience, ethical behavior is possible and can be expected as an emergence of conscience (namely as "feedfront": anticipated learning by feedback). But unethical behavior is also possible.
  • Now what causes the difference between ethical and unethical behavior in people, and is it justified to judge them / punish them based on this difference?
  • The hypothesis is here: choosing between ethical and unethical behavior is guided by "identity", a concept encompassing everything a person is. So it can be influenced by everything, including (occasionally) variations in bain chemistry that are caused by energy wave events outside of the person. However, the component that is justified to judge is the part of a person's identity that is in "thoughtspace". Namely, values, their vision of the world how they want it to be etc.. These "thoughtspace artifacts" develop slowly, over time: they are reasonably stable, isolating them from being destroyed by small external events. The hypothesis here is that "thoughtspace" is non-deterministic, because it consists of reasonably stable artifacts (thoughts etc.) that interact with other rules than the deterministic matter with which they are implemented. Means: they interact with the rules of information, which are "spiritual".
  • This above hypothesis that people "choose" to be good or bad over time is of course refuted by history: everyone chooses to be bad, to some degree. From a Christian viewpoint, people can only choose / accept to be redeemed, not to be good. But that would again be an act of "free will": the outcome depends on the person (their aggregate identity), not on any single cause that can be determined, and not based on rules that can be determined. There are patterns, but we can't explain all the variations of why this or that individual person has decided this or that way. This is largely because every case is unique, and there is no way to "repeat a life", or to simulate it so that all conditions that make it unique are incorporated. So in the end, nobody knows (also not God) how a person will decide, as there is no calculation capacity (no "parallel universe") to simulate the pilot wave conditions in for predicting the outcome. All of the universe is needed to calculate it, there's no space capacity.
  • Or even simpler: There is free will because there is conscience. Conscience implies that a person understands the world she is in, and the potential consequences of her actions, and hen acts a certain way. So to the degree that a person knew the consequences before, and could process the outcome, judging the behavior afterwards is justified. However, perfectly knowing the consequences and being able to process options perfectly would always result in perfect behavior (right?), which means that there is no justification for judging non-perfect behavior. But that logic might exactly be the logic of grace: God does not want to judge anyone for being and acting bad, as all the people were missing the light to be better … . So if God does in the end still judge people, it must mean that these people had the option to decide better (the only instance being, to accept Christ as savior). But then, how could they decide better, in a deterministic universe? Only if in "thoughtspace", there is an entity ("spirit", "heart", "moral conscience" etc.) which is self-referential and able to choose and thereby influence its own moral quality. Means, people who are evil (esp. by not accepting Christ against better knowledge) are so knowingly, and continuously, while still being able to be different.
  • The hypothesis that thoughtspace artifacts can enable non-deterministic behavior in a deterministic world can be tested by trying to create a software implementation of a conscience and trying to observe non-deterministic behavior even though letting the software start from completely identical input and adding no extra input over time and also not giving the software access to randomness. If multiple runs of that sofware lead to different results (like in reasoning, developing a vision for the world etc.), then thoughtspace is non-deterministic. It requires however (maybe) a level of intelligence on par with humans, including extensive training to understand the world. However, without random influence it is logically impossible that runs from the same starting conditions on a deterministic machine lead to non-deterministic results. So we need randomness. If both instances receive the same input as randomness, they would again lead to the same results. Which means, now results depend on the random input, not on the algorithm. And randomness cannot be called "free will". So even if quantum theory is correct and the world is fuzzy and random, there could still be no free will??
  • However, then again "being creative" is clearly an effect of free will. From that we see that, if free will should be a reasonable concept at all, it has to include both aspects of randomness (the input and inspirations that creatives seek) and reasoning (the thoughtspace activity that creates something meaningful out of the random input). This is however only the perspective of the individual: if there is no real randomness (as in pilot wave theory), then the behavior is still predetermined.
  • It seems necessary to understand the role that information plays for agency ("free will"). Human beings cannot emerge by chance, they have been created. That act of creating them changed the world. The big question is, did it change the world in a deterministic or non-deterministic manner?
  • Another approach: The question if there is free will is not relevant because God and also we ourselves would not stop judging and punishing actions even if that would not be justified. We would do it because we don't like these action and don't want them to spread. Like putting sexual offenders in jail. Or like a holy God not wanting to have community with sinners. It's against his character. Thus ultimately, judging behavior is an expression of a conflict, and does not need free will or agency. Like we pull up weeds: not because they have free will and deserve to be judged for wrongdoing, but because we don't want them to grow where they grow. From this obviously follows that there would not be any active punishment ("torture") of sinners for their wrongdoing, just being removed from God's presence. And to be fair, it would have to be annihilation, or at least a bearabe mixture of good and bad as on earth. Also, this approach makes the ultimate question not about free will, but about why the world did not stay holy as God wanted it to be. The question is about the origin of evil then! (And really, who needs "real" free will, if we can be creative and have fun doing so.) Without free will, the world could indeed be seen as a war of evil against good, each side having seemly limited (or self-limited?) options in that war, and trying to win it. Both good and evil would be "contagious".
  • So, what about the origin of evil? The Bible does not really give an answer, since the story of the Fall of Satan is very figurative and could mean something else (it talks about a certain historic king in the first place, anyway). Potentially, Satan ("evil") is eternal just as God ("good") is. Or potentially, the origin of evil is the only non-deterministic event that ever happened, and is a mystery even to God …
  • But anyway, interpreting judging and punishment as war (without free will) rather than moral act (for an act of free will) does not help either, as war in a deterministic (rather: materialistic) world is not a meaningful concept: there would not be a will to war, just the ticking of a clockwork …
  • Another way of introducing free will into the Copenhagen interpretation quantum theory is (perhaps) to claim that the mind acts like
  • Of course, materialists will always argue that there is still no free will because its thought processes are implemented deterministically, or with randomness elements at best. Perhaps this is true in the sense that it allows to refute the idea that the free will we observe in ourself could ever have appeared via evolution: there has to be free will at the start to continue having free will. (But if so, how do we explain that newly conceived human beings obviously acquire their free will at some point?)
  • So the big question is, after all: does matter drive the mind, or does mind drive matter? From neurological studies (effects of drugs etc.) it is apparent that matter drives the mind at least in part. But also on the overall, highest level of "consciousness"? Because apparently, I can move a finger when I want it … means, mind drive matter, right? To reconcile this intuitive notion of free will with physics (in whatever interpretation of quantum theory) it could be argued that the mind becomes gradually independent from matter, and being able to drive matter rather than the other way round, by learning more about matter. That is, by understanding the world around you. With an operationally complete understanding of the world, the word has a full (or sufficiently full) representation in the mind, which allows detailed simulations in thoughtspace (though processes, decision making etc.) before then implementing them in the real world. This capacity to reason about the world, including both creativity and repeated self-feedback of results, would be the way how "free will" is implemented. It can be considered free even if individual thought processes are deterministic, since humans tend to think something over multiple times, each time starting from a slightly different starting point (even if the variation is just more or less hunger etc.) and to come up with different results accordingly. And then, to valuate these results, and that is where the freedom comes in. Because the decision for one of the alternative options is not based on randomness or simple determinism, but on an understanding of the different options. Understanding is a spiritual activity, in thoughtspace, even though implemented in matter. The algorithm for it is non-deterministical as it includes feedback: "try to understand until you understand".
  • Possibly, relativistic effects (what is considered instantaneous from which point of view) introduce non-determinism without randomness, and thus the opportunity for free will, into a pilot-wave theory universe?

In the end, it is obvious that we don't know anything …

The most interesting idea from this discourse is the brazen (because logically impossible) hypothesis that conscious information systems (human brains) could derive non-deterministic results from deterministic input and with deterministic means. If we can prove that experimentally, it is for certain that history even in a deterministic pilot-theory universe is not boring at all 🙂 Because people could enact these non-deterministic results of their thoughspace processes, introducing non-determinism into the physical space. However, this question if deterministic brains can produce non-deterministic results is well-known in philosoph aready, and still debated [source].

The other interesting finding here is that both orthodox quantum theory (with real randomness) and pilot-theory (with full determinism) have no obvious or non-obvious but proven space for free will. If you want to uphold free will (or the idea of spirit, relationships, contact to God etc.) you have to dive into physics and find the blind spot.

Downloading your OSMAnd~ maps to a computer and installing them to your Android phone from there has several advantages:

  • Use cheaper or faster network access. In a case where, for example, there is no wifi available but your computer is connected by wired Ethernet network, downloading the maps to your computer is probably much faster and cheaper than using your mobile data plan. (An alternative is configuring tethering at your computer. USB tethering and Bluetooth tethering would work but wifi tethering can be a challenge to set up since most wifi hardware in notebooks does not support access point mode, and Android might not support ad-hoc mode.)
  • Avoid tracking. You avoid being tracked by the OSMAnd~ maps server, which else "will send your device and application specs to an Analytics server upon downloading the list of maps you can download" [source].

So, how to do this?

  1. Just download the relevant zip archives from the OsmAnd maps index.
  2. Connect your Android phone and unpack the downloaded zip archives into the osmand folder on the SD card. (Alternatively, use adb push filename.ext /sdcard/0/osmand/ according to these answers).
  3. Start OsmAnd~. It should read and index the files (which for the first startup will take a bit of time).

One of my websites was constantly throwing "Internal Server Error" errors, and that error appeared as follows in /var/log/apache2/error.log:

Thu Oct 16 17:59:14 2014 (19446): Fatal Error Unable to allocate shared memory segment of 67108864 bytes: mmap: Cannot allocate memory (12)

And that even though 9 GiB of memory was free. Also, only one website was affected, others did run fine. The error appeared even when requesting files that did not exist at all (independent of file type: JPG, PHP etc.). After ten minutes or so, the error would disappear on its own, for a while. Also, after restarting Apache the problem disappeared, for a while.

Reason

The problem was due to hitting the system limit of shmpage (shared memory allocation), similar to this case. This limit effectively does only exist in virtualized (VPS) environments, not on physical machines. Hitting this limit can be confirmed by running "cat  /proc/user_beancounters", which in our case would output right after the above error situation: shmpages held=247122 [...] limit=262144 failcnt=10067. At 4 kiB page size, the max. allowed 262144 shared memory pages correspond to (262144*4096)/1024^3 = 1 GiB of shared memory.

When restarting Apache, the problem would be temporarily resolved because shared memory usage would initially be lower: it reduced the number of shared memory pages held form about 247000 to 148000, as per cat /proc/user_beancounters.

The number of 67108864 bytes of shared memory to allocated, mentioned in the error message, gives a hint what consumes this much shared memory: it is just the default value of PHP's opcache size of 64 MiB, configured in /etc/php5/cgi/php.ini (because 67108864 / 1024 / 1024 = 64). The problem is not that PHP-FCGI would try to allocate 64 MiB of opcache shared memory once, or once per site, but once per process [source]. PHP-FastCGI processes are reused for several requests [source], so the opcache caching makes still some sense, contrary to this opinion. However, they are relatively short-lived as their number varies depending on site load, so the caching does not add much benefit. And worse, the opcache caches are not shared between the PHP-FastCGI processes, but each one gets its own. With about 10 processes per site, each consuming 64 MiB of shared memory, we quickly hit the 1 GiB shared memory limit of the VPS, like above. To illustrate, these were the values in my case: the free command indicated the following shared memory usage:

  • 69 MB some seconds after stopping apache2
  • 136 MB immediately after starting apache2 (service apache2 start; free)
  • 500 – 700 MB some 30 – 180 s after starting apache2
  • 989 MB typically when apache2 is running for a long time, very close to the 1 GiB limit already

Nearly all this shared memory usage is created by the php-cgi processes, as can be seen in top output (use "b" to toggle to background highlighting, "x" to toggle to highlighting the sort column, and "<" / ">" to switch to SHR as the sort column). Namely, when 991 MB shared memory are consumed, 620 MB (= 9 * 64 MB + 2 * 40 MB) of this was consumed by 11 php-cgi processes.

Solution

The solution is to use PHP-FPM instead of PHP-FastCGI [source]. Contrary to that source, this works independently of the webserver, also working with Apache2.

After deploying this solution, you can compare the "Zend OPcache" section of phpinfo() scripts from different sites on your server. As you can see from the "Cache hits", "Cache misses", "Cached scripts" and "Cached keys" numbers, there is only one single OPcache for all your PHP sites. However, you can configure opcache parameters different for different sites (like memory usage etc.), and this is also reflected in the phpinfo() output. I can't really make sense of that so far, but assume that memory usage etc. are indeed configured per PHP-FPM process pool. So 128 MiB for one site and 64 MiB for another would allow for 192 MiB total shared memory usage for opcache.

The shared memory situation after switching the site to PHP-FPM was only a 70 MiB max. difference in shared memory between Apache2 and php-fpm running and not running, compared to the 900 MiB earlier. This was generated by five php-fpm processes running simultaneously, with 40-50 MB shared memory "each". So clearly, the shared memory is indeed shared between php-fpm processes instead of each having its own.

(Another tip: there are other important ways to optimize OPcache, see this blog post.)

Working solution

Finding this solution was quite a nightmare. But, here it is.

This assumes you have a working setup of Panels in Drupal 7 already.

  1. Use a group path prefix. Set up pathauto to include your group's path as a prefix for group content. This is a challenge by itself due to various bugs and changes in Drupal, but here is my solution.
  2. Edit the node template panel page. In the CTools page manager (at /admin/structure/pages), enable the node_view (node template) page, if not already done, and then edit it (at /admin/structure/pages/edit/node_view).
  3. Create a new Panel variant.
  4. Create a selection rule by path. Click "Selection rules" in teh left sidebar for your new Panel variant. Then select from the list "String: URL path", add it, and configure it to use your group's path prefix with a wildcard, like groupname/*.
  5. Reorder your variant. Click "Reorder variants" at the top right (should be /admin/structure/pages/nojs/operation/node_view/actions/rearrange). Reorder the variants to make yours come after "Node panelizer" and before "Fallback panel". Drupal will work through the vriants and select the first where teh selection rules match, so placing it after "Fallback" would make it never show up.
  6. Test. Go to some non-panelized group content that has your group's path prefix, and see if the panel variant gets applied. To see an effect, you will have to add content to the panel variant of course.

Alternative solutions

The following solution should also work:

Panelize display modes, set display mode per group. This is a combination of the following:

  • ds_extras: needed to create additional view modes for content types
  • context: for general context management, needed by context_og
  • context_og: needed in context to detect when a node belongs to a certain group
  • contextual_view_modes: to set a view mode for a content type based on its context
  • panelizer: to panelize a view mode

You would define a context for each group for which you want a default panel, triggered by a node belonging to the group. Then you would configure the content types (with the options added by contextual_view_modes) to set a specific view mode for each of these contexts, which you created via ds_extras. And use Panels (or rather Panelizer I think) to create default panels for these new view modes.

This solution is semantic, since it is not dependent on URL patterns. However, it is also more complex, clutters the view modes namespace, and requires one new panel for each content type / view mode combination, rather than just one in the node template.

Non-working solutions

The following solutions that I tried did not work:

Selecting a Panel variant by group membership. This should work – see my instructions here. But currently (2014-09) it does not due to Drupal issue #2242511 "How to create panel variant with selection rule for groups audience field". There is an impractical workaround available, but "somebody" should go in there and fix it for real …

Using og_panels. This would be the most comfortable variant, as it avoids also the need to be admin to change a group's default layout. However, there is no Drupal 7 port of the og_panels module. See: Drupal issue #990918.

Selecting panel variants via contexts. Proposal: Create a solution from a combination of the following:

  • context: for general context management, needed by context_og. See: https://www.drupal.org/project/context
  • context_og: to detect when a node belongs to a certain group. See: https://www.drupal.org/project/context_og
  • panels: to set a panel variant in the Panels selection rules, based on the context detected via context_og. See: https://www.drupal.org/project/panels

Problem: Panelizer does not list the context / context_og contexts in the list of contexts of the "Context exists" selection rule. This is because these are simply two different things called "context": the Panels module does not even depend on the context module, nor vice versa.

The "Context exists" selection rule will list the "Panels contexts" that are defined in the context section above the selection rules section, if and only if that context field contains data. See this wording in the "Context exists" selection rule: "Check to see if the context exists (contains data) or does not exist (contains no data).", and the reproduction instructions from Drupal issue #1263896. This means that in effect, "context" in Panels means the same as "contextual filters" in views: it is a way to pass in arguments that can be grabbed by the view resp. the content elements on the panel, and using them in "Context exists" is rather a side use.

Using context_panels_layouts. Proposal: Create a solution from a combination of the following:

  • context: for general context management, needed by context_og
  • context_og: to set a conext when a node is part of a certain group
  • context_panels_layout: to set a panel as a context reaction
  • panels: to set a panel variant in the Panels selection rules, based on the context detected via context_og

Problem: This does not work, as context_panels_layout can only be used to set Panels layouts, not actual panel variants that also include the content added to the layout.

Powerline is a technology for transmitting data over the AC grid. All devices provide an RJ45 Ethernet plug (and some also wifi and USB), so they support client devices of all operating systems. Basic configuration of encryption is likewise OS independent, as nearly all of them use a "pairing" button for that. However, to set the encryption key manually, to make more detailed settings, to read out speed statistics and to update the firmware most come with configuration tools that only work under Windows.

So, can we find a device that can be fully managed under Linux, ideally with free software? After a long search, this turns out to be a simple task. Just use any device that supports the Homeplug AV or Homeplug AV2 standard, and manage it with Open PLC Utils, a free software full-featured management tool developed by AV/AV2 chipset maker Atheros. It  allows full device configuration, info reading and upgrading for all INT6000/6300/6400/AR7420 devices, even tampering with the parameter set in the firmware. The only alternative are DS2 chipset based devices, since at least most of them can be fully configured via a built-in web interface (such as the COMTREND Powergrid 9020, their most modern device). However, a tool like Open PLC Utils is preferred here since it effectively replaces a part of the DS2 management software with free software, allowing scripting, modifications etc., and is more powerful in general.

Recommendable devices

It is said that it does not matter so much in terms of connection speed and quality which exact device you choose, as long as you choose the right chipse. Here, we focus on the Homeplug AV chipsets (INT6000, INT6300, INT6400) and there on the most modern one (INT6400). We avoid the Homeplug AV2 chipsets because they seem not yet mature (see below) and the DS2 chipsets because they are not as readily configurable under Linux.

Recommendation for Germany: Speedport Powerline 100. The best recommendation is so far: Telekom Speedport Powerline 100. It has the INT6400 chipset (most modern one for the Homeplug AV 200 Mbit/s devices), and it is dirt cheap (ca. 20 EUR incl. shipment for a pair of them, via ebay.de) because Telekom made Germany awash with these devices. It also is good quality, since it is a whitelabel product from well-known French brand LEA Networks – see also the very detailed test on tomsnetworking.de (in German). One can use the LEA-made software tool SOFTPLUG for settings beyond simple pairing for encryption, but it runs only under Windows. It is not needed however, as the free software tool Open PLC Utils can likewise be used, also running under Linux.

Discussion of alternatives: Devolo dLAN AVsmart+. These devices are quite nice, as they have an extended status display on the device – compare the manual. They are also readil available in used condition on eBay, but not as dirt cheap as the Speedport Powerline 100. However, they use the older INT6000 chipset [source], leading to a third less speed in long-range applications [source].

Discussion of alternatiives: COMTREND PowerGrid 9020. This is not a Homeplug AB/AV2 device, but uses the non-interoperable DS2 chipset. Reviews on Amazon are good. However, it can be fully managed under Linux as all settings are available within a web interface, as argued for and against above. The Powergrid 9020 is the most modern Powerline device of Madrid-based manufacturer COMTREND so far, and avaible in UK and Europe type plug versions. The UK plug versions are very readily available online, as they are provided by UK based large ISP Britich Telecom (BT) to their customers. The Europe plug versions are quite hard to find though (but here are some). Also see the installation guide and the  full manual.

Discussion of alternatives: Homeplug AV2 devices. There are the even more modern 500 Mbit/s Homeplug AV2 devices (using the AR7420 etc. chipsets), however as of 2014-09 this generation of technology seems to not be mature still, often suffering from connection breakdowns, low throughput, the devices running hot etc. (as judged from reviews on Amazon). So we avoid them here, also because they are still more expensive on the second-hand market. But your priorities and mileage may vary. They also can be fully managed with Open PLC Utils.

Linux software for powerline adapters

While Open PLC Utils is clearly the winner, the following list is all the Linux-based powerline software I came across:

  • Open PLC Utils. As said, clearly the winner: full-featured and free software.
  • Faifa. A manufacturer independent, free software Homeplug AV/AV2 utility for Linux. Allows low-level functions such as control frame dumps. Considered to be the successor to the older plconfig utility, see below.
  • plconfig. An older, simple Linux based configuration utility. For download on Github. Superseded by Faifa now.
  • dlanlist, dlanpasswd. Open source software that can be downloaded and compiled under Linux and is meant to list (and set the password of) Devolo dLAN devices. Since these conform to the Homeplug AV/AV2 standard, the software might be used to also configure other devices, after some simple modifications.
  • devolo Cockpit. A large configuration softwre for Devolo dLAN devices. Seemingly not available as a source version, so adaptations to other devices are not possible.
  • Intellon device manager for 3.x firmware (Windows software). Allows full management of INT6x00 chipset devices when using a 3.x firmware version, incl. editing firmware parameters. Its use is described in this article.
  • Intellon device manager for 4.x firmware (Windows software). In contrast to the version for 3.x firmwares, this does not support full management of the of INT6x00 chipset devices any more.

Background information

The following in-depth articles provide relevant background information about Powerline technology and their successful use:

Modding and hacking powerline adapters

At 10 EUR for a used device or less, powerline adapters are so cheap that they offer themselves to several non-standard uses. At least teh following come to mind:

  • Powerline noise filters. Devices that include an AC plug with noise filter (such as the SpeedportPowerline 100 recommended above) are the cheapest option to use them as frequency filters to prevent contamination of the 50 Hz frequency from other digital devices, mobile phone chargers etc..
  • Powerline bridge. It is also possible to create a simple bridge by connecting two of them with a crossover Cat5e cable (or a bridge device in between, if necessary). This should allow crossing phase boundaries in the home cabling, if necessary.
  • VDSL P2P modem replacements. And then, of course, these cheap devices are a natural candidate for creating a cheap DIY replacement for pairs of VDSL modems (usually 150 EUR per piece!). It works by connecting twisted phone wire to the signal (via soldering) before it gets modulated on the AC mains power. It is possible to use this for transmitting data over several hundred meters at least.