Federating the Onions

April 23, 2018 - Reading time: 3 minutes

Within Freedombone it has long been possible to view fediverse instances via an onion address. That has applied to GNU Social, postActiv and more recently Pleroma. But this is really just the client to server part of the communications pipeline and federation between instances (server to server) remained exclusively via the clearnet.

A couple of years ago I did do some investigation of whether I could get GNU Social to federate via onion addresses, which would have the advantage of being independent of the DNS and certificate authority systems. There are a few php Tor proxying examples out there on Github, but none of my experients with federating GNU Social via onion addresses worked out the way I had hoped and I expect that fixing this would require a more involved level of php hacking than I'm currently familiar with.

Recently it has become possible to proxy Pleroma through Tor so that the servers can federate using Tor's DNS resolver, so I've added this as the default behavior both for the ordinary version of Freedombone and also the "onion only" version which, as the name implies, only uses onion addresses to access apps. If you're using Freedombone then this is all automatic, but if you're not the changes needed are quite simple.

If you're using Debian 9.x (the current stable) then you may want to install the tor daemon from backports. This will give you access to the shiny new version 3 onion addresses which have better performance and security properties.

apt-get -yq -t stretch-backports install tor

Create an onion address for your Pleroma instance. Within /etc/tor/torrc:

HiddenServiceDir /var/lib/tor/hidden_service_pleroma/
HiddenServiceVersion 3
HiddenServicePort 80

And restart tor to generate the address:

systemctl restart tor

To find out what the onion address is:

cat /var/lib/tor/hidden_service_pleroma/hostname

Create an nginx configuration for your site. Something like:

proxy_cache_path /tmp/pleroma-media-cache levels=1:2 keys_zone=pleroma_media_cache:10m max_size=100m inactive=80m use_temp_path=off;

server {
    listen default_server;
    server_name yoursiteonionaddress;

    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Permitted-Cross-Domain-Policies none;
    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;

   access_log /dev/null;
   error_log /dev/null;

   root /etc/pleroma;
   index index.html;

   gzip_vary on;
   gzip_proxied any;
   gzip_comp_level 6;
   gzip_buffers 16 8k;
   gzip_http_version 1.1;
   gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript application/activity+json application/atom+xml;

   location / {
       client_max_body_size 15m;
       client_body_buffer_size 15m;

       limit_conn conn_limit_per_ip 50;
       limit_req zone=req_limit_per_ip burst=50 nodelay;

       add_header 'Access-Control-Allow-Origin' '*' always;
       add_header 'Access-Control-Allow-Methods' 'POST, GET, OPTIONS' always;
       add_header 'Access-Control-Allow-Headers' 'Authorization, Content-Type' always;
       if ($request_method = OPTIONS) {
           return 204;

       proxy_http_version 1.1;
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       proxy_set_header Host $http_host;
       #proxy_set_header X-Forwarded-Proto $scheme;
       proxy_pass http://localhost:4000;

  location /proxy {
      client_max_body_size 15m;
      client_body_buffer_size 128k;

      limit_conn conn_limit_per_ip 10;
      limit_req zone=req_limit_per_ip burst=10 nodelay;

      proxy_cache pleroma_media_cache;
          proxy_cache_lock on;
          proxy_pass http://localhost:4000;

Where in the above case the Pleroma daemon is running on port 4000.

Now edit your secret.exs Pleroma configuration file and add the following line:

config :pleroma, :http, proxy_url: {:socks5, :localhost, 9050}

You will then need to recompile Pleroma.

cd where_you_installed_pleroma
sudo -u pleroma mix clean
sudo -u pleroma mix deps.compile
sudo -u pleroma mix compile

And restart the pleroma daemon.

systemctl restart pleroma

You should now be able to access Pleroma from the onion address and also federate with other instances which also support server to server onion addresses via a tor proxy.

Improving read/write performance

April 22, 2018 - Reading time: 2 minutes

I've been running small computers on EMMC or microSD cards since 2010, so I know that when it comes to reading or writing things to disk performance can be quite sluggish. This is one of the reasons why I usually turn logging off, because anything which might be often writing to disk is going to slow things down.

Recently I was wondering whether there were any software or kernel tweaks which I could do to try to improve microSD performance. I found a few things which may make a small difference, but the overall answer is that there isn't much you can do other than buy the best quality Sandisk or Samsung microSD you can find. Running some benchmarks I found that microSD read speed can vary wildly from about 22MB/s all the way down to 5MB/s, with the variations happening for no obvious reason but probably due to the underlying microcontroller logic on the memory card itself as it does things like wear leveling.

I bought a Cubieboard 2 in 2014 and it has been used (and abused) quite extensively for testing Freedombone image builds. It has the option to connect an external drive via a SATA cable, but I had never tried doing that because I assumed that a secondary power supply would also be needed. It turns out though that isn't the case, so I was wondering whether I could actually boot the file system from SATA. According to the Cubieboard wiki the answer appeared to be no, or at least not directly.

Normally single board computers only boot from EMMC or microSD, but there's a way to work around this limitation. What you can do is have a boot partition on the microSD card which then points to the main rootfs on a SATA connected SSD or hard drive. This is fairly easy to do by modifying the root path within a file called boot.cmd. I modified the freedombone-image command with an extra --sata option to do this, and so long as you copy the resulting image both to the microSD and the SSD then it works.

This is one of those things which is so ridiculously simple that I wonder why I didn't do it four years ago, but I was mostly focussed upon the Beaglebone Black as a platform, Debian support for Cubieboard was extremely basic and there were a ton of much higher priority problems to solve at the time.

The end result is that running on an SSD the disk performance on Cubieboard 2 is at least an order of magnitude faster, and this is quite pleasing since it means that using these A20 based boards for things like NextCloud or Syncthing is a much more attractive proposition.

Tidying up Tor

April 20, 2018 - Reading time: 1 minutes

Something which I noticed recently is that on the rare occasions where the Debian tor package gets updated it's potentially possible to lose the hidden service settings within your torrc configuration file if you make the wrong choice and say yes to use the maintainer's version of that file. Especially if you were running the "onion only" version of Freedombone this could be pretty bad. You wouldn't lose the keys, but it would still be very inconvenient and stop you from accessing apps.

Fortunately, there's a solution to this. The torrc file can define an include path where secondary configurations can be loaded. So this is now what happens in the current version of Freedombone. The configuration automatically gets migrated to use an include path to the settings. So no matter what happens when the tor package gets updated you shouldn't lose your settings or access to your apps via onion addresses.

What is the future of programming?

April 17, 2018 - Reading time: 2 minutes

I was watching a talk by Bob Martin about "The Future of Programming". It's an amusing tale, but it says far more about the past than the future. I was expecting him to say at the end that the future is going to resemble the past and that software will mostly be written by highly discipined, minimally supervised middle aged professionals of no particular gender who have all signed up to a code of ethics. But he didn't say that and instead he ends up predicting what happens a couple of years later with Zuckerberg testifying before the US congress and the impending regulation of "big tech".

So what is the future of programming? I don't think it will be fully automated, but it will probably be partially automated. There's still a lot of hype about Deep Learning right now, but none of the fashionistas are paying any attention to genetic programming. I think that user interface design will become more of a well defined thing and that there will be genetic programming systems which can largely automate the creation of user interfaces. If you can define in vague terms the kind of thing you want then a system can automatically produce candidate screen arrangements from which you then choose from and itterate until you are satisfied with the result.

But much of programming - especially the behind the scenes stuff which is more detailed and requires more knowledge of hardware specifics - I expect to continue to require human programmers.

The code of ethics which Bob Martin mentions isn't a new idea, but it's interesting to follow through with what the implications are. Codes of ethics don't come out of nowhere and they usually aren't mass adopted because a clever CEO somewhere wrote them down on a blog. Those types of things imply professional organizations, such as guilds or unions. Organisations bigger than any particular company and capable of applying pressure to rogue bosses to keep them in line. We don't have anything like that yet for software professionals, but there are signs that it might be on the way. Not long ago there was the first strike by software engineers attempting to join a union at a startup company in the US, and although it seems that the outcome in that case was not a good one it does set a precedent for future organizing of that type. If there were to be a global union of software engineers capable or organised collective actions then I expect the gender problem and maybe some other problems not mentioned in the talk would quickly go away. Perhaps that's too optimistic and the gender problem is actually more complex than that, but with a significant weight of organized labour behind it I suspect that progress on these problems could be made far more rapidly than otherwise.

Freedombone 3.1: Prospects for a better internet

April 16, 2018 - Reading time: 3 minutes

It has been quite a while since the last official release, so it's about time for another. Freedombone 3.1 continues on a Debain 9.x (stretch) base and there have been a few new applications added since last year, the most notable of which are Pleroma and PeerTube. Both of those apps possibly might have a big future if current trends play out the way I think they will.

This release also includes significant improvements to the mesh version, allowing you to change protocols on the fly. Presently there doesn't seem to be any clear winner in the battle of mesh routing protocols, so it makes sense to include the most common ones and have the user decide. The mesh system is now also pure IPv6 and I like to think that this system is a kind of proof of concept for what the internet could become if supporting legacy software and the client/server paradigm wasn't an obstacle.

There has also been a change of logo. The graffiti style logo was used from the beginning and although I still like this logo I wanted something which was more consistent with the ASCII header of source files and the message of the day within the software itself. So the new logo is really just a colored in version of the ASCII logo. An early criticism was that perhaps the logo should be just an icon of some kind, because it's possible that the system will end up being used in non-English speaking areas. I think that's a reasonable concern and although it hasn't been a problem so far it might be worth investing in some new logo artwork in future.

A question I always ask myself when putting out new software is "is this still relevant?". The world of software moves quickly and things which were once important become no longer so as the technological landscape changes. Freedombone is one of those curious cases where it's not me that's aligning with the world but the world that's coming to meet where I am instead. The issues which motivated the creation of this system are becoming more relevant over time, rather than less. Things like net neutrality under threat, censorship, W3C approving DRM, infrastructure centralization and fragility, growing realization of how out of touch Silicon Valley companies are with most people's lives, aggressive demonetization and the end of the idea that advertising can be a "win-win situation" for creators of web content.

Change is obviously needed, but what kind of change? Just "writing to your MP", as Open Rights Group frequently recommends might be necessary sometimes but isn't sufficient. I think the public have to take matters into their own hands and reclaim the internet as a platform for everyday life rather than just as a vehicle to be used cynically to increase the size of Zuckerberg's bank account. Hosting web systems at an individual or community level can be part of that, and although it's not yet consumer grade easy it is becoming more feasible for more people.

Zuckerberg versus the US congress

April 11, 2018 - Reading time: 4 minutes

I watched most of Mark Zuckerberg's testimony to the US congress while also thinking of the "great Facebook trial" mentioned in the Freedom in the Cloud talk from 2010. This isn't such a trial. In fact, it's the opposite. Almost without exception, congress critter after congress critter were lining up to praise Mr Zuckerberg and his site in glowing terms. They were falling over themselves to thank him for coming, for the wonderful blessings which his site had bestowed upon the world. They praised his intelligence, his business acumen and even his patriotism.

The Zuckerberg appearance was a display of America's most powerful engaging in mutual admiration. That congress people felt it necessary to ingratiate themselves before him says a lot about where power really resides and who has the goods on who. Zuckerberg is one of the top ten wealthiest people in the world - a multi-billionaire. It's possible that if he can steer Facebook out of it's current difficulties then he may go on to become the world's richest man and own a large fraction of the planet's resources. If current trends continue then at some time in the foreseeable future one man will have more wealth than half the world's population. Things really are looking that bad.

The predictions about what would happen in this inquiry are turning out to be accurate. There will be mild criticism of Facebook. Zuckerberg will make some superficial changes to privacy settings and terms of service documents. He might pay a small fine, relative to his profits. And then the business model will continue as if nothing had happened.

One of the great tricks of the inquiry process was to have four minute slots for questions. I'm not highly familiar with the internals of US politics but when I saw this I thought it was a stroke of genius. The questioners usually took 3.5 minutes to ask their question, leaving mere seconds for Zuckerberg to give only the briefest and most boilerplate answer, or just avoid answering altogether due to running out of time. This allowed him to dodge many of the more interesting questions, such as exactly how "shadow profiles" get created and whether it's possible to opt out of them (it isn't), or how Facebook tracks web site visits via "like" buttons or "tracking pixels".

Zuckerberg himself appeared slightly nervous at first, but mostly his demenor was confident and assertive. This wasn't someone being brought to heel for transgressions by the mythical "checks and balances". It was more like the master telling the slaves what the new Facebook terms of service will look like in the next year or two.

So other than theatre, is there anything which can be learned from this?

Zuckerberg says that not just political campaigns but anyone with a Facebook group and some significant number of followers will be required to submit formal identity documents to the company. This could result in another round of purges and an exodus from Facebook into either Twitter or the fediverse.

I had expected it, and may have even mentioned it in past blogs, but now there's no doubt. Zuckerberg made clear that he is going to use AI to moderate Facebook. At one point he said that there is no number of employees he could hire in order to review the content of the site. It can only be done with AI. This is a high risk strategy, but for a centralized silo system there aren't any other options which wouldn't break the business model. I expect that there's a large probability that AI based moderation will go badly. It might even be Facebook's ultimate undoing.

Really the only way to moderate communities is with human oversight by people who hopefully are in posession of some form of wisdom and who are embedded within those communities and understand their history. Only people have the knowledge and context to be able to evaluate social events within diverse cultures and situations. A single centralized system is going to struggle with this, no matter how advanced the AI algorithms are. Context collapse could be rampant. A lot of folks might get purged or miscategorized.

As the titans of the state and the oligarchy slug it out in a grudge match for supermacy over personal data, this potential looming catastrophe could be a huge opportunity for the fediverse to demonstrate that community moderation is superior to centralized AI moderation, and is the only truly scalable way of supporting meaningful social relations in the 21st century.


The blog of Bob Mottram, a Free Software hacker and maintainer of the Freedombone project.

Web site

Email/XMPP: bob@freedombone.net

Matrix: @bob:matrix.freedombone.net