04 January 2014

How to run Twister on ARM (Debian) and OSX

I'm growing very fond of Twister, a new project from Brazilian developer Miguel Freitas. Miguel basically leveraged Bitcoin and Bittorrent concepts (and code!) to build a fully-decentralised Twitter clone. The P2P architecture means that it's completely immune to censorship and "chilling effects": nobody in the network can hand your account over to the authorities or disclose who you are, nobody can delete your posts -- and neither can you, so be careful what you post :)

It's definitely early days (it's still a bit hacky to set up, and the html interface needs work), but I've decided to help a bit, so I managed to make it work on OSX and now on my little ARM-based always-on home server, the CuBox. Like most lightweight P2P services, Twister is perfect for long-running, low-power ARM instances (Raspberry Pi etc), so this is what you'll need to know if you want to set it up there.

  1. Download all libdb4.8* packages from Bittylicious. Berkeley DB 4.8 is a Bitcoin requirement, and at this point in time there's no way around it. Unfortunately, recent distributions (i.e. Debian Wheezy and newer) replaced it with 5.1, so apt-get won't help you (NOTE: if you don't care about wallet compatibility with already-running instances, you can just use libdb5.1). You can compile libdb4.8 from source if you want, but they'll require about 500 MB of dependencies (java, X etc); the Bittylicious packages worked fine for my vanilla Debian install. There's a chance that BDB as a whole will be dropped at some point in the future.
  2. Download other dependencies: openssl-dev, libboost-all-dev, miniupnpc (miniUPNPc is technically optional but strongly recommended, it makes it a breeze to go through firewalls). These should come down fine with apt-get. Note that OpenSSL should be version 1.0+ (check by running openssl version). NOTE: it's been reported that default Fedora openssl packages (built without support for Elliptic Curve crypto) will make Twister crash at the moment. Make sure you find alternatives until the bug is fixed.
  3. clone Twister repositories:
    git clone https://github.com/miguelfreitas/twister-core.git
    git clone https://github.com/miguelfreitas/twister-html.git
  4. prepare your configuration (note: DON'T edit user and pwd, they're hardcoded elsewhere at the moment):
    mkdir ~/.twister
    echo -e "rpcuser=user\nrpcpassword=pwd" > ~/.twister/twister.conf
    ln -s /path/to/your/twister-html ~/.twister/html
  5. Follow these instructions to build twisterd. You might want to link it under /usr/local/bin once done.
  6. Launch it. Give it 5 to 10 minutes to download the full blockchain, then connect to http://your-server:28332/home.html . You'll be prompted to create an ID - do so.
  7. After creation, make sure you backup your key by doing:
    ./twisterd dumpprivkey my-user > my-key.txt
    Backup my-key.txt (or just its content) in a safe place, it's your all-important key: you lose that, you lose access to your ID and there is no way to get it back!
  8. You should also backup ~/.twister/user_data as it contains your direct messages; these can theoretically be retrieved if lost, but it takes a while, so it's probably good practice to just back them up regularly.
  9. Enjoy! The community is very small at the moment, so feel free to go on a follow-spree. There are two mailing lists, Twister-Users (more active) and Twister-Dev (more technical). The project is brand new (Miguel released it at the end of November 2013) so expect a bumpy ride, but the community is very friendly and in need of help with HTML and jQuery as well as C++, if you can spare some cycles :)

For OSX, the process is slightly more convoluted at the moment, so I documented it in the official doc/build-osx.md. Note that I'm not a make guru, I'm certain that there's plenty of room for improvement there -- feel free to contribute!

02 November 2013

How to easily create custom networks in VmWare Fusion 5 Standard

For a long time, creating custom networks in VmWare Fusion has been a challenge. For some bizarre reason, for years Fusion lacked the network-management GUI commonly found in the equivalent VmWare Workstation product for Windows.

Since switching to OSX last year I've had to edit each VMX file, forcing interfaces to use a custom network definition, but I've recently come across a lovely tool that greatly simplifies this task AND finally clarified for me how Fusion stores network definitions.

This tool is UBER Network Fuser, by Nicholas Weaver. It was released in 2012 when he wasn't working at VmWare yet, and it's not been updated since -- likely because Fusion 6 now offers similar functionality in the twice-as-expensive Professional version (which I think is ridiculous -- Workstation comes in one version with all features, why does the overall-less-powerful OSX equivalent have two?) but it still runs fine on OSX 10.8.5 Mountain Lion.

The tool itself is really easy to use: just go to the Networks tab to add or edit subnets, specifying whether you want to use the Fusion-provided dhcp and NAT services and to run through a virtual adapter. For some reason it won't allow you to customise subnets (you get a random 192.168.x.x net with mask 255.255.255.0), but that can be hacked-in as I'll show you later.

Next, go to the Configuration tab and click on the path to your Virtual Machines (which might already look correct, but won't work until you actually set it yourself). Leave the Preferences Path alone.

Now go to the Virtual Machines tab, refresh the list, select a machine to edit, select the interface you want to assign to your custom network, and select the network. Done!

This is already a nice set of features, but the real kicker is still to come. Close the tool, start Fusion, edit another image, select your network adapter, scroll the option list… your custom network is now an option for all machines! This is because Fusion stores network definitions in /Library/Preferences/VMware Fusion/networking - open it with a text editor and you'll see it contains old and new network definitions. UBERNF manipulates this file, and Fusion is happy to go along with it. You could just change subnet values here, I believe.

Unfortunately Nicholas didn't release any source code for UBERNF (I wouldn't be surprised if some of it ended up in Fusion Professional, to be honest), so there is no way to improve it or even just fix little defects that are probably due to OSX changes between 10.7 / Lion (the version he compiled it on) and 10.8 / Mountain Lion. It would be nice if somebody could write a similar app for Mountain Lion / Mavericks and release it as open-source, or simply charging a few dollars less than the difference in cost between Fusion Standard and Professional. Silly, silly VmWare...

16 September 2013

21 July 2013

How to make pull requests on GitHub or BitBucket

Github and Bitbucket make it very easy to submit pull requests; probably too easy, in fact. The temptation to just fork somebody's repo, fix the obvious mistake and submit a pull request is quite strong. But that's not how you're supposed to work! So this is basically a dummy's guide to mark this procedure into my brain.

A pull requests marries the entire branch to the one you're targeting; if you keep making changes here and there, a pull request will include all these new changes, regardless of when you originally opened it. It makes your request basically un-mergeable by upstream repositories, in most cases.

So the real procedure is:

  1. Fork the upstream repository. Now you have your own master branch. Clone it locally as usual.
  2. Create a new branch, either via web or (easier) from the command line with git branch FixBugBranch && git checkout -b FixBugBranch. 
  3. Make your changes on this FixBugBranch. Make only the minimum amount of changes necessary to fix a specific issue, then commit and push.
  4. On GitHub, create a pull request from FixBugBranch towards the original repository
  5. When/if your pull request is accepted upstream, you can delete FixBugBranch.
If you want to make further changes, you can either create a further branch from FixBugBranch, or create and merge a pull request from FixBugBranch into your master. The important thing is that you don't touch FixBugBranch anymore, so that upstream maintainers won't receive all your extra commits but only ones relevant to the particular bug you raised. 

18 July 2013

Unicode URL-handling in web.py

Web.py is a lovely tool I'm currently using for a silly project (warning: explicit Italian language). Unfortunately, it does some clever things to support URLs containing Unicode, but then drops the ball when it comes to actually do anything with them (i.e. dispatch/route them as expected, using regular expressions that actually match Unicode objects).

This was a real problem in my app, so I came up with a quick and dirty patch, which may or may not work for you and may or may not break other things. Basically I've tracked down the regex operations on URLs, and added Python's re.UNICODE flag to them, so that unicode characters will be matched as "\w" etc.

Feel free to tell me where I'm going wrong -- I'm not a web.py guru by all means -- but this little patch significantly improved my quality of life today, so to speak.