04 March 2018

The European Union, explained to geeks

I've started describing the EU to geek-friends as a set of services built over decades.

  • You need to coordinate energy supplies across the company departments (nations)? We built a service (ECSC) that does that. Worked great, no more fighting over the last bit of coal!
  • Need atomic development? EURATOM service. Worked great, again.
  • And so on and so forth...
  • At some point somebody said hey, we need a management tool for all this stuff! "European Council" service, with a set of dedicated subthreads for the real work (European Commission).
  • But that service will run amok at times, let's add some monitoring and security checks! EuroParliament service - took a few rewrites to get right, nobody really likes to work on monitoring tools; but like systemd on Linux, the EP service should eventually take over anything that talks to real world I/O, so it's pretty important.
  • Now all this stuff needs to communicate, with common formats that avoid parsing and reparsing umpteen different types of data back and forth, and ways to look up the right service for a given job - so we created a "CommonMarket SDK", optionally turbocharged with options like Schengen. When exceptions are thrown, the SDK will automatically invoke the ECJ service to resolve matters; and it will self-update by talking to the management services. Once everyone adopts the SDK, then it should be easier to make more radical changes through that (ECB, Euro, common fiscal policy...). But in the end nobody likes change, it's always hard to break backward-compatibility.

Now, across the company/continent, various departments/nations have adopted some or all of these services, but most of them ended up relying on the SDK one way or the other, so it became basically mandatory. At one point we had to give a name to the whole framework, and "EU" it was.

It's definitely not a monolith, but there are so many moving parts that the management services are now essential. Some departments have renounced their Write access (Iceland, Norway, Switzerland...) and some were never granted that privilege (Turkey); some departments were forced to change their processes to suit the framework (Italy, Greece, the Eastern countries...). Things are still pretty shaky, developers are still very much at work, but it's getting better with time. It's making more and more services possible and even *easy* to bootstrap (EMA, EFSA, Erasmus...). Bugs creep in and out, we keep adding more and more fault-tolerance, the workload is not yet distributed fairly, etc etc; but it's accomplishing some very heavy tasks that are absolutely mission-critical, if we want to keep the company running and competing with the big boys.


Then, one day, a department said they'd like to go back to pen and paper. Except for a few services, for which they want to hand-craft packets individually, but those services should just assume the data is still as good as before, and never throw exceptions - they will ignore any response from the ECJ service anyway. Some of what their department does depends on another department, which has no intention to go back to pen and paper, but they say they will somehow give them bits and receive paper, without anyone actually doing the transformation, and without any friction at all. And they'd like to retain write access to the management services too, thank you very much, plus veto powers, so nobody can change SDK formats without consulting them.

Cue facepalming.

25 January 2018

Windows Survival Kit for OSX exiles

After 5 long years, I was gently forced back to Windows for everyday work. The experience has been less terrible than I thought, but was pretty frustrating at the beginning.

To help anyone in the same predicament, I put together a small list of tips to make the move a bit more tolerable.

  • Windows does not have hot corners. Hello, 1995! Anyway, the solution here is using a small app called, unsurprisingly, HotCornersApp. It works well enough, and it's fine with multiple screens too. Despite the basic website, it's legit and even opensource, you can compile it yourself (although beware - apparently the very latest updates may not build properly).
  • Windows does not do text substitution, which is one of those things that you don't know how much you love it until they take it away from you. As far as I can see, there is no free utility on Windows to do this, or nothing that actually works well enough. So, I paid for Breevy. It feels a bit retro (it looks blurry on high-def screens...), but it works very well and has all sorts of options and special features.
  • Windows doesn't really deal well with multi-language support, aka typing accents from a US keyboard. Sure, you can use US-International, and deal with ' and " becoming meta-keys, but for a techie/programmer typing those characters on their own more often than accents, it's extremely annoying. The solution was again Breevy: you can define a combo that will not be triggered unless you type a special character afterwards. For example, I defined `e to become รจ after I press Ctrl. It works absolutely everywhere, although it's nowhere as elegant as the OSX popup. Same story for special characters like £ , € etc.
    • For best results, check if your keyboard supports custom macro. Mine does, so I mapped the blank side-keys to accents and so on. After muscle-memory starts kicking in, this solution is actually superior to stock OSX.
  • I was not going to reformat my external hard drives to NTFS, which is a pain to use back on OSX; so again I had to pay for Paragon HFS+ for Windows. The UI is garbage (and does not work properly with multi-monitor setups), but the actual driver seems to work perfectly.
  • Microsoft has basic print-to-PDF support. If you need to concatenate documents, PrimoPDF does it, and it's free (do not download the Nitro version). The interface is not great though.
  • For the developer types out there who rely on Dash, a good equivalent on Windows is Zeal. It supports the same format, even fetching docsets directly from Dash repositories.
  • Also for developer/sysadmin types, the Windows equivalent of homebrew is now Chocolatey. Whoever came up with that word should give up trying to name things, but the software does work. You can use it to install 7zip (to get the latest beta with proper security patches, choco install 7zip.install --pre -y) windirstat and so on, it will make it easy to upgrade them when necessary (rather than having through the usual website-download-install dance, just choco upgrade them all).
  • There are quite a few apps that do what Fluid does, i.e. making websites into "native" apps. I know, I know, they are aberrations; but I got used to having a few sites (Gmail, Trello, Hangouts...) accessible this way. After a bad experience with WebCatalog (it was working ok, but then it got stuck trying to upgrade itself), I installed nativefier. This is again more for the geek types; it requires npm and it's a command-line app with a few quirks.
  • UPDATE 1: to get back the "preview on pressing Space" experience, there is a free app called Seer. You can also pay for a license, but it's not clear what the difference might be, the free app is more than enough for my needs. I had to remove the Ctrl-Alt-S shortcut in Settings in order to make it work properly.

Did I miss anything? Feel free to suggest other useful tidbits in comments.

17 November 2017

How to root Nexus 7 (2012) and flash it to LineageOS, from Mac OSX

I've finally got fed up enough with the old Nexus 7 (2012) that we originally bought to let the kids play - it's never really worked properly, getting laggy every 5 minutes and running out of battery after an hour. The kids moved on to an iPad, so I "installed" this crappy tablet to the kitchen wall, hoping to use it for calendaring and podcasts, but it was still horrendously laggy. I've decided to give a chance to LineageOS (previously CyanogenMod) to see if it helps. I'd be interested to know if anybody else uses a tablet wall-mounted in their kitchen, and what apps they installed - at the moment I have Paprika, the BBC players and weather, and PodcastAddict.

Here are the full steps required for flashing - I write them here because some of these are at risk to disappear from the internet for good after the demise of CyanogenMod wiki.


First you need to install the Android tools. I recommend doing this with homebrew, install it if you don't have it yet (it's extremely useful in many many cases).
Then open a terminal on your mac and type:
  • brew tap caskroom/cask
  • brew tap caskroom/versions
  • brew cask install java8
  • brew cask install android-sdk
Enable Developer Mode on the tablet by tapping 7 times on the build number under Settings -> About tablet.
Back to Settings, tap on Developer Options and enable USB Debug mode and Stay Awake.
Go back to your mac terminal and type:
  • sudo adb start-server
This starts the android debugger server.
Connect the tablet with a real USB cable (beware charging-only cables! Those won't work.)
On the tablet screen, you should be prompted to authorize the device; do it.
Back to terminal:
  • adb reboot bootloader
once the device comes back in bootloader mode:
  • fastboot oem unlock
Accept the disclaimer on the tablet by clicking the power button when Yes is highlighted (you can move with the volume buttons).
If it doesn't reboot on its own, select Start (again with the power button). At this point you should see an unlocked pad at the bottom of the screen as Android loads.
Note: if you plan to flash another OS, there is no point in going through the whole setup at this point, skip as much as you can.
Re-enable Developer mode and USB Debugging.

At this point the machine is rooted. The following steps are necessary only if you want to install LineageOS or other hacks; at the very minimum, though, you should install Trimmer (fstrim) from the Play store and use it liberally. Anyway, on with the flashing...

Get the latest image for "grouper" from https://twrp.me/, and unzip it.
Back to the terminal:
  • cd folder/where/you/have/your-downloaded-image.img
  • fastboot flash recovery your-downloaded-image.img
  • adb reboot bootloader
At this point you have a custom recovery image with a bunch of nifty features that make it very easy to hack the device. Tap Wipe and select system, cache, and dalvik.
Get the image you want to install. I'm currently trying this but in general anything after CM10 / Android 4.1 may be on the slow side for the original Nexus. You probably want the Google Apps from opengapps.org - choose ARM / 7.1 and the Micro option. Save both the apps zip and the image zip in the same folder.
Back to Terminal, let's copy these files to the device:
  • cd folder/where/you/have/your/zips
  • adb push your-downloaded-image.zip /
  • adb push your-downloaded-gapps.zip /
On the tablet, in the recovery screen, select Install, tap on Install ZIP, then select the image file and install. Repeat for the gapps file. If gapps refuse to go in because of space constraints, download this file, rename it gapps-config.txt, then copy it to the device in the same location as the gapps zip:
  • adb push gapps-config.txt /
and try again to install the gapps image.
Reboot the tablet. You might have to re-enable developer mode and usb debugging.
Install the Trimmer (fstrim) app, which is absolutely necessary. Launch it and click Trim Now. Click on the settings icon and enable autotrim as frequently as possible. In fact, every time you install an app or write a bunch of files, before you launch the new app, open Trimmer and do a trim, or it will all get laggy again.
Another good app is Android SSH Server, although it's a bit old it still works fine and it's easy to configure. You just have to use the options -t -oHostKeyAlgorithms=+ssh-dss -p XXXXX (where XXXXX is the custom port configured in the app, Android won't allow the classic one) when connecting.

And that's it.

10 October 2017

Fighting Cliqz in Firefox

Mozilla recently announced that some of their German users downloading Firefox will receive a version that tracks some of their web activity, reporting it to Cliqz.com. In the wake of this development, which is pretty awful from a privacy perspective, I went spelunking into my version of FF to see if anything had been enabled already in my build.

To my horror, cliqz.com is already mentioned in a few places, despite me never installing anything related to it. If you enter about:config in the address bar and search for cliqz, a few entries will pop up:

Straight out of the gate, it looks like something from cliqz.com has been whitelisted. What is it?

The social.* preferences are related to SocialAPI Services, a sort of framework to integrate social networks into Firefox. It was introduced several years ago but very few people actually use it or know about it. If you look up the related preferences, a few more entries are present:

If you are a fan of this sort of thing, you can go to that address https://activations.cdn.mozilla.net and install some of the available providers. I have no idea whether they still work or not; among others, delicious.com has recently been sold and it's in read-only mode, so that's unlikely to be useful.

I personally disabled it all (that social.remote-install.enabled freaked me out) by double-clicking all boolean properties (turning them to false) and double-clicking then clearing out social.whitelist.

The other mentions of cliqz seem to be special-casing whitelists aimed at making an extension work around some of the recent changes in the Firefox extensibility framework. If I understand correctly, dom.ipc.cpows.allow-cpows-in-compat-addons allows Cross-Process Object Wrappers (an internal communication mechanism that should slowly be removed) to be used even if the extension is marked as compatible with the new multiprocess architecture; and extensions.legacy.exceptions exempts the listed extensions from being marked as Legacy.

In those whitelists, there is one called testpilot@cliqz.com. Test Pilot is an official Firefox add-on from Mozilla that will periodically publish some proposed additions to the browser, allowing users to enable them, test them out, give feedback and so on. I personally like it, some of the proposed features are actually pretty good (although it doesn't look like any of them ever made it to the main build); considering its complexity (an extension-installing extension) it makes sense that it might require some special privileges, and after all it's an official Mozilla feature so why not?

Well, it looks like Mozilla is using cliqz.com to gather remote info about TestPilot usage, which is very disappointing. TestPilot was marketed as an official Mozilla project, there was no mention of third parties involved. I won't disable TestPilot, but I am again very disappointed in their cavalier attitude with my usage data.

To conclude, it looks like the "synergy" between Mozilla and Cliqz was already underway before the latest announcement. It's likely that more defensive hacks will be required in the near future to keep user-tracking at bay in Firefox. As a long-time fan of Mozilla since the '90s, this development is disappointing.

31 July 2017

How to build waifu2x command-line version on osx

UPDATE 23/10/2017: recent changes seem to be incompatible with clang. I have updated the steps below to check out the last commit that is known as working.
Waifu2x is a popular image converter backed by neural-network models, typically used to upscale images from anime. The various web versions are easy to use but typically don't allow for batch-processing, because it's a relatively intensive operation. However, there is a command-line version that is fairly easy to compile on Windows and Linux. OSX support was notably absent... until now.
These are the steps required to get it working on Mac:
  1. brew tap science && brew install opencv3
    (if you don't have Homebrew, go install it now - it's wonderful)
  2. git clone https://github.com/DeadSix27/waifu2x-converter-cpp.git
  3. cd waifu2x-converter-cpp
  4. git checkout d69313040b0784662465fb1d2eca81a2b1ebccb2
  5. cmake -DOVERRIDE_OPENCV=1 -DOPENCV_PREFIX=/usr/local/Cellar/opencv3/<your version here> . (remember the dot at the end!)
  6. make -j4
At this point, you have the executable waifu2x-converter-cpp in the directory; you can make install if you really want to (I prefer to keep custom stuff in my home dir). To test it's working, the following command should return info on your system:

./waifu2x-converter-cpp --list-processor

If you need some images to test, here you can get quite a few stills from the gorgeous 5 Centimeters Per Second. Note: you might have to rename the folder "models_rgb" into "models" before converting.


If you hate the command-line, there is a simple QT wrapper. To compile it you will need Qt-Creator and Qt installed and configured. This used to be very complicated, but both items have now been packaged for Homebrew so it's all awesomely simple (well, compared to what it was, at least). Note that you still have to do the steps above - the wrapper needs the waifu2x executable to be already compiled.

Install and configure Qt and Qt-Creator

  1. brew install qt && brew cask install qt-creator
  2. Launch Qt Creator from Launchpad, then go to Qt Creator -> Preferences (Command + ,)
  3. Select Build & Run, then the Qt Versions tab
  4. Click on Add... and select “Macintosh HD”
  5. Press Command + Shift + . to show hidden directories
  6. select /usr/local/Cellar/qt5/<your version>/bin/qmake then OK and OK again to close the Preferences window
  7. Reopen Preferences -> Build & Run, select the Kits tab and click Add
  8. Set the following options:
    • Name to something like "Qt CLang Desktop" 
    • Both Compiler options to Clang X86 64bit
    • Qt Version to the version you just created
    • Click on Make Default then OK to close Preferences.

Compile waifu2x-converter-qt

  1. git clone https://github.com/toyg/waifu2x-converter-qt.git (the original repo looks abandoned, so I forked it and tweaked it a bit)
  2. cd waifu-converter-qt && open waifu2x-converter-qt.pro (this should open Qt Creator; if it doesn't, launch it manually and File -> Open Project -> select the .pro file).
  3. Select Build -> Run (or Command + R). The resulting .app bundle should now be in a folder called build-waifu2x-converter-qt-<something something>, just beside your main project folder.
When you first launch the GUI, you should probably go to Preferences and set the path to your waifu2x-converter-cpp executable.