Cover Image

For the Birbs

May 3, 2018 - Reading time: 2 minutes

The first continuous integration (CI) system I saw in use was back in the mid 2000s. Back then I don't think it was even called "continuous integration". It was just a way to test that builds worked, run unit tests and the send alerts if things had broken. For long builds that would usually be run overnight. It also wasn't anything formal, just some arbitrary and completely bespoke scripts on a server.

More recently I was discussing whether it would be a good idea to have continuous integration for the Freedombone project. Since making images involves building a minimal Debian and then installing and configuring other apps this can take quite a while. A couple of hours on a fairly powerful machine. At first I thought there wouldn't be much to be gained by doing this, but on second thoughts maybe it might be useful if I can find or requisition an old desktop machine to do it. At present testing of Freedombone builds is very manual and quite laborious.

I know that Lars Wirzenius has been developing a CI system called ick. As a side note you might also want to check out his Getting Things Done for Hackers. Ick looks ok and I could probably use it, but the code looks rather complicated and I'm not a huge fan of yarns as a software validation methodology, although I can understand why developers might be attracted down the path of making things as similar to natural language as possible. I was thinking that for my purposes CI doesn't need to be this complicated, since there's only a small set of things which it needs to do.

So I spent roughly a day writing a CI system and the result is BirbCI. This is really a single command implemented as a bash script which can set up systemd daemons to run builds and then report the results as a html page. It's about the simplest possible CI system at roughly 330 lines of script including comments. I also blinged it up a bit with a background image and coloured icons and text representing build status, so that it doesn't look too sparse and utilitarian.

This should be good enough to start thinking about making some sort of Freedombone build server. It's not a super-high priority, but maybe something to work on as a background task in the next few weeks or months.


Federating the Onions

April 23, 2018 - Reading time: 3 minutes

Within Freedombone it has long been possible to view fediverse instances via an onion address. That has applied to GNU Social, postActiv and more recently Pleroma. But this is really just the client to server part of the communications pipeline and federation between instances (server to server) remained exclusively via the clearnet.

A couple of years ago I did do some investigation of whether I could get GNU Social to federate via onion addresses, which would have the advantage of being independent of the DNS and certificate authority systems. There are a few php Tor proxying examples out there on Github, but none of my experients with federating GNU Social via onion addresses worked out the way I had hoped and I expect that fixing this would require a more involved level of php hacking than I'm currently familiar with.

Recently it has become possible to proxy Pleroma through Tor so that the servers can federate using Tor's DNS resolver, so I've added this as the default behavior both for the ordinary version of Freedombone and also the "onion only" version which, as the name implies, only uses onion addresses to access apps. If you're using Freedombone then this is all automatic, but if you're not the changes needed are quite simple.

If you're using Debian 9.x (the current stable) then you may want to install the tor daemon from backports. This will give you access to the shiny new version 3 onion addresses which have better performance and security properties.

apt-get -yq -t stretch-backports install tor

Create an onion address for your Pleroma instance. Within /etc/tor/torrc:

HiddenServiceDir /var/lib/tor/hidden_service_pleroma/
HiddenServiceVersion 3
HiddenServicePort 80 127.0.0.1:8011

And restart tor to generate the address:

systemctl restart tor

To find out what the onion address is:

cat /var/lib/tor/hidden_service_pleroma/hostname

Create an nginx configuration for your site. Something like:

proxy_cache_path /tmp/pleroma-media-cache levels=1:2 keys_zone=pleroma_media_cache:10m max_size=100m inactive=80m use_temp_path=off;

server {
    listen 127.0.0.1:8011 default_server;
    server_name yoursiteonionaddress;

    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Permitted-Cross-Domain-Policies none;
    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;

   access_log /dev/null;
   error_log /dev/null;

   root /etc/pleroma;
   index index.html;

   gzip_vary on;
   gzip_proxied any;
   gzip_comp_level 6;
   gzip_buffers 16 8k;
   gzip_http_version 1.1;
   gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript application/activity+json application/atom+xml;

   location / {
       client_max_body_size 15m;
       client_body_buffer_size 15m;

       limit_conn conn_limit_per_ip 50;
       limit_req zone=req_limit_per_ip burst=50 nodelay;

       add_header 'Access-Control-Allow-Origin' '*' always;
       add_header 'Access-Control-Allow-Methods' 'POST, GET, OPTIONS' always;
       add_header 'Access-Control-Allow-Headers' 'Authorization, Content-Type' always;
       if ($request_method = OPTIONS) {
           return 204;
       }

       proxy_http_version 1.1;
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       proxy_set_header Host $http_host;
       #proxy_set_header X-Forwarded-Proto $scheme;
       proxy_pass http://localhost:4000;
  }

  location /proxy {
      client_max_body_size 15m;
      client_body_buffer_size 128k;

      limit_conn conn_limit_per_ip 10;
      limit_req zone=req_limit_per_ip burst=10 nodelay;

      proxy_cache pleroma_media_cache;
          proxy_cache_lock on;
          proxy_pass http://localhost:4000;
  }
}

Where in the above case the Pleroma daemon is running on port 4000.

Now edit your secret.exs Pleroma configuration file and add the following line:

config :pleroma, :http, proxy_url: {:socks5, :localhost, 9050}

You will then need to recompile Pleroma.

cd where_you_installed_pleroma
sudo -u pleroma mix clean
sudo -u pleroma mix deps.compile
sudo -u pleroma mix compile

And restart the pleroma daemon.

systemctl restart pleroma

You should now be able to access Pleroma from the onion address and also federate with other instances which also support server to server onion addresses via a tor proxy.


Improving read/write performance

April 22, 2018 - Reading time: 2 minutes

I've been running small computers on EMMC or microSD cards since 2010, so I know that when it comes to reading or writing things to disk performance can be quite sluggish. This is one of the reasons why I usually turn logging off, because anything which might be often writing to disk is going to slow things down.

Recently I was wondering whether there were any software or kernel tweaks which I could do to try to improve microSD performance. I found a few things which may make a small difference, but the overall answer is that there isn't much you can do other than buy the best quality Sandisk or Samsung microSD you can find. Running some benchmarks I found that microSD read speed can vary wildly from about 22MB/s all the way down to 5MB/s, with the variations happening for no obvious reason but probably due to the underlying microcontroller logic on the memory card itself as it does things like wear leveling.

I bought a Cubieboard 2 in 2014 and it has been used (and abused) quite extensively for testing Freedombone image builds. It has the option to connect an external drive via a SATA cable, but I had never tried doing that because I assumed that a secondary power supply would also be needed. It turns out though that isn't the case, so I was wondering whether I could actually boot the file system from SATA. According to the Cubieboard wiki the answer appeared to be no, or at least not directly.

Normally single board computers only boot from EMMC or microSD, but there's a way to work around this limitation. What you can do is have a boot partition on the microSD card which then points to the main rootfs on a SATA connected SSD or hard drive. This is fairly easy to do by modifying the root path within a file called boot.cmd. I modified the freedombone-image command with an extra --sata option to do this, and so long as you copy the resulting image both to the microSD and the SSD then it works.

This is one of those things which is so ridiculously simple that I wonder why I didn't do it four years ago, but I was mostly focussed upon the Beaglebone Black as a platform, Debian support for Cubieboard was extremely basic and there were a ton of much higher priority problems to solve at the time.

The end result is that running on an SSD the disk performance on Cubieboard 2 is at least an order of magnitude faster, and this is quite pleasing since it means that using these A20 based boards for things like NextCloud or Syncthing is a much more attractive proposition.


Tidying up Tor

April 20, 2018 - Reading time: 1 minutes

Something which I noticed recently is that on the rare occasions where the Debian tor package gets updated it's potentially possible to lose the hidden service settings within your torrc configuration file if you make the wrong choice and say yes to use the maintainer's version of that file. Especially if you were running the "onion only" version of Freedombone this could be pretty bad. You wouldn't lose the keys, but it would still be very inconvenient and stop you from accessing apps.

Fortunately, there's a solution to this. The torrc file can define an include path where secondary configurations can be loaded. So this is now what happens in the current version of Freedombone. The configuration automatically gets migrated to use an include path to the settings. So no matter what happens when the tor package gets updated you shouldn't lose your settings or access to your apps via onion addresses.


What is the future of programming?

April 17, 2018 - Reading time: 2 minutes

I was watching a talk by Bob Martin about "The Future of Programming". It's an amusing tale, but it says far more about the past than the future. I was expecting him to say at the end that the future is going to resemble the past and that software will mostly be written by highly discipined, minimally supervised middle aged professionals of no particular gender who have all signed up to a code of ethics. But he didn't say that and instead he ends up predicting what happens a couple of years later with Zuckerberg testifying before the US congress and the impending regulation of "big tech".

So what is the future of programming? I don't think it will be fully automated, but it will probably be partially automated. There's still a lot of hype about Deep Learning right now, but none of the fashionistas are paying any attention to genetic programming. I think that user interface design will become more of a well defined thing and that there will be genetic programming systems which can largely automate the creation of user interfaces. If you can define in vague terms the kind of thing you want then a system can automatically produce candidate screen arrangements from which you then choose from and itterate until you are satisfied with the result.

But much of programming - especially the behind the scenes stuff which is more detailed and requires more knowledge of hardware specifics - I expect to continue to require human programmers.

The code of ethics which Bob Martin mentions isn't a new idea, but it's interesting to follow through with what the implications are. Codes of ethics don't come out of nowhere and they usually aren't mass adopted because a clever CEO somewhere wrote them down on a blog. Those types of things imply professional organizations, such as guilds or unions. Organisations bigger than any particular company and capable of applying pressure to rogue bosses to keep them in line. We don't have anything like that yet for software professionals, but there are signs that it might be on the way. Not long ago there was the first strike by software engineers attempting to join a union at a startup company in the US, and although it seems that the outcome in that case was not a good one it does set a precedent for future organizing of that type. If there were to be a global union of software engineers capable or organised collective actions then I expect the gender problem and maybe some other problems not mentioned in the talk would quickly go away. Perhaps that's too optimistic and the gender problem is actually more complex than that, but with a significant weight of organized labour behind it I suspect that progress on these problems could be made far more rapidly than otherwise.


Freedombone 3.1: Prospects for a better internet

April 16, 2018 - Reading time: 3 minutes

It has been quite a while since the last official release, so it's about time for another. Freedombone 3.1 continues on a Debain 9.x (stretch) base and there have been a few new applications added since last year, the most notable of which are Pleroma and PeerTube. Both of those apps possibly might have a big future if current trends play out the way I think they will.

This release also includes significant improvements to the mesh version, allowing you to change protocols on the fly. Presently there doesn't seem to be any clear winner in the battle of mesh routing protocols, so it makes sense to include the most common ones and have the user decide. The mesh system is now also pure IPv6 and I like to think that this system is a kind of proof of concept for what the internet could become if supporting legacy software and the client/server paradigm wasn't an obstacle.

There has also been a change of logo. The graffiti style logo was used from the beginning and although I still like this logo I wanted something which was more consistent with the ASCII header of source files and the message of the day within the software itself. So the new logo is really just a colored in version of the ASCII logo. An early criticism was that perhaps the logo should be just an icon of some kind, because it's possible that the system will end up being used in non-English speaking areas. I think that's a reasonable concern and although it hasn't been a problem so far it might be worth investing in some new logo artwork in future.

A question I always ask myself when putting out new software is "is this still relevant?". The world of software moves quickly and things which were once important become no longer so as the technological landscape changes. Freedombone is one of those curious cases where it's not me that's aligning with the world but the world that's coming to meet where I am instead. The issues which motivated the creation of this system are becoming more relevant over time, rather than less. Things like net neutrality under threat, censorship, W3C approving DRM, infrastructure centralization and fragility, growing realization of how out of touch Silicon Valley companies are with most people's lives, aggressive demonetization and the end of the idea that advertising can be a "win-win situation" for creators of web content.

Change is obviously needed, but what kind of change? Just "writing to your MP", as Open Rights Group frequently recommends might be necessary sometimes but isn't sufficient. I think the public have to take matters into their own hands and reclaim the internet as a platform for everyday life rather than just as a vehicle to be used cynically to increase the size of Zuckerberg's bank account. Hosting web systems at an individual or community level can be part of that, and although it's not yet consumer grade easy it is becoming more feasible for more people.


About

The blog of Bob Mottram, a Free Software hacker and maintainer of the Freedombone project.

Web site

Email/XMPP: bob@freedombone.net

Matrix: @bob:matrix.freedombone.net