OpenAir API shock

From the NetSuite OpenAir API documentation (PDF):

Since we are using HTTP, each connection is isolated, and must go through authorization each time. This authorization consists of sending the server an XML data structure consisting of company name, user name, and user password.

... really? In 2014? Ever heard of tokens? I'm not asking for full OAuth, but a simple header-based token mechanism is banal, faster and much more secure than sending XML with user and password for each request.

Oh, your API endpoint is a Perl script. That explains it, I guess... you are not "using HTTP", you are using CGI. Badly.

After this gem, I'm not surprised to learn that they implement simple data-retrieval actions with POST (or PUT -- what?) rather than GET, that the whole API basically consists in exposing database tables as they are, and that their XML is entirely custom. Excuse me, I think I've just thrown up in my mouth...

Python and cmd.exe on Windows - a world of pain.

As I mentioned a few days ago on Hacker News in a Ruby thread, CPython support for Windows is, overall, extraordinarily good for a runtime with clear Unix roots. This said, occasionally you'll eventually hit a wall and find yourself cursing Guido & Bill under your breath. Yesterday was one of those times for me.

I'm currently working on a project using Python 3.3.5 on Windows 2008 r2, building a program that will talk to Weblogic 10.3 via the Jython-powered WLST interface (it actually does more than that: it leverages Jython to also instantiate several complex Java classes, launch VBS scripts and so on and so forth). In order to correctly set up WLST/Jython, I have to launch a batch file which in turns calls several other batch files in order to set up all sorts of environment variables. These are all pure-DOS batch files doing very little except creating or reading environment variables, but they're nested two or three levels deep from the entry-point batch.

For some reason, when I launched this batch with Popen(), variables were not set correctly. In fact, it looked like "call" statements in the batch were just silently ignored. I tried using shell=True, and it made no difference whatsoever. I put it down to some weird cmd.exe behaviour; tried to switch extension from .bat to .cmd and things started to move a bit more (so much for all those posts saying there is no difference between the two) but still some stuff wouldn't work, so I eventually settled for reimplementing the whole batch chain in Python (which is terrible and will likely bite me a year down the line as the version of Weblogic changes, but beggars can't be choosers).

The most frustrating thing, however, was that opening an interactive pipe to test and do exploratory programming was just too difficult. There are a lot of examples out there talking about .send(), .communicate() and stdin=subprocess.PIPE, but nobody seems to mention what I experienced: as soon as you call communicate() on a cmd.exe launched with Popen(), all pipes are closed and there is no obvious way to reopen them. I don't think this is due to cmd.exe, because the last output I got was always ">More ?"; I think this is just CPython being too eager to clean up.

Luckily, I found a solution in WinPexpect, a fork of Pexpect that actually deals with Windows weirdness. Processes launched with winpexpect.winspawn() actually keep their stdin pipes up long enough for me to figure out enough stuff to fully re-implement the batch chain.

The result is a Python script three times as long as the original batch and likely to break the first time Oracle changes a line here or there. It will do for now, but the experience left a sour taste in my mouth, so to speak; cmd.exe is a crappy shell, but it shouldn't be that hard to open a long-running prompt-like process piping stuff to it. If I'm missing an obviously-better solution, please let me know and I'll happily blog about it, because clearly Google and DuckDuckGo need to know about it.

How to run Twister on ARM (Debian) and OSX

I'm growing very fond of Twister, a new project from Brazilian developer Miguel Freitas. Miguel basically leveraged Bitcoin and Bittorrent concepts (and code!) to build a fully-decentralised Twitter clone. The P2P architecture means that it's completely immune to censorship and "chilling effects": nobody in the network can hand your account over to the authorities or disclose who you are, nobody can delete your posts -- and neither can you, so be careful what you post :)

It's definitely early days (it's still a bit hacky to set up, and the html interface needs work), but I've decided to help a bit, so I managed to make it work on OSX and now on my little ARM-based always-on home server, the CuBox. Like most lightweight P2P services, Twister is perfect for long-running, low-power ARM instances (Raspberry Pi etc), so this is what you'll need to know if you want to set it up there.

1. Download all libdb4.8* packages from Bittylicious. Berkeley DB 4.8 is a Bitcoin requirement, and at this point in time there's no way around it. Unfortunately, recent distributions (i.e. Debian Wheezy and newer) replaced it with 5.1, so apt-get won't help you (NOTE: if you don't care about wallet compatibility with already-running instances, you can just use libdb5.1). You can compile libdb4.8 from source if you want, but they'll require about 500 MB of dependencies (java, X etc); the Bittylicious packages worked fine for my vanilla Debian install. There's a chance that BDB as a whole will be dropped at some point in the future.
2. Download other dependencies: openssl-dev, libboost-all-dev, miniupnpc (miniUPNPc is technically optional but strongly recommended, it makes it a breeze to go through firewalls). These should come down fine with apt-get. Note that OpenSSL should be version 1.0+ (check by running openssl version). NOTE: it's been reported that default Fedora openssl packages (built without support for Elliptic Curve crypto) will make Twister crash at the moment. Make sure you find alternatives until the bug is fixed.
3. clone Twister repositories:
   git clone https://github.com/miguelfreitas/twister-core.git
   git clone https://github.com/miguelfreitas/twister-html.git
4. prepare your configuration (note: DON'T edit user and pwd, they're hardcoded elsewhere at the moment):
   mkdir ~/.twister
   echo -e "rpcuser=user\nrpcpassword=pwd" > ~/.twister/twister.conf
   ln -s /path/to/your/twister-html ~/.twister/html
5. Follow these instructions to build twisterd. You might want to link it under /usr/local/bin once done.
6. Launch it. Give it 5 to 10 minutes to download the full blockchain, then connect to http://your-server:28332/home.html . You'll be prompted to create an ID - do so.
7. After creation, make sure you backup your key by doing:
   ./twisterd dumpprivkey my-user > my-key.txt
   Backup my-key.txt (or just its content) in a safe place, it's your all-important key: you lose that, you lose access to your ID and there is no way to get it back!
8. You should also backup ~/.twister/user_data as it contains your direct messages; these can theoretically be retrieved if lost, but it takes a while, so it's probably good practice to just back them up regularly.
9. Enjoy! The community is very small at the moment, so feel free to go on a follow-spree. There are two mailing lists, Twister-Users (more active) and Twister-Dev (more technical). The project is brand new (Miguel released it at the end of November 2013) so expect a bumpy ride, but the community is very friendly and in need of help with HTML and jQuery as well as C++, if you can spare some cycles :)

For OSX, the process is slightly more convoluted at the moment, so I documented it in the official doc/build-osx.md. Note that I'm not a make guru, I'm certain that there's plenty of room for improvement there -- feel free to contribute!

How to easily create custom networks in VmWare Fusion 5 Standard

For a long time, creating custom networks in VmWare Fusion has been a challenge. For some bizarre reason, for years Fusion lacked the network-management GUI commonly found in the equivalent VmWare Workstation product for Windows.

Since switching to OSX last year I've had to edit each VMX file, forcing interfaces to use a custom network definition, but I've recently come across a lovely tool that greatly simplifies this task AND finally clarified for me how Fusion stores network definitions.

This tool is UBER Network Fuser, by Nicholas Weaver. It was released in 2012 when he wasn't working at VmWare yet, and it's not been updated since -- likely because Fusion 6 now offers similar functionality in the twice-as-expensive Professional version (which I think is ridiculous -- Workstation comes in one version with all features, why does the overall-less-powerful OSX equivalent have two?) but it still runs fine on OSX 10.8.5 Mountain Lion.

The tool itself is really easy to use: just go to the Networks tab to add or edit subnets, specifying whether you want to use the Fusion-provided dhcp and NAT services and to run through a virtual adapter. For some reason it won't allow you to customise subnets (you get a random 192.168.x.x net with mask 255.255.255.0), but that can be hacked-in as I'll show you later.

Next, go to the Configuration tab and click on the path to your Virtual Machines (which might already look correct, but won't work until you actually set it yourself). Leave the Preferences Path alone.

Now go to the Virtual Machines tab, refresh the list, select a machine to edit, select the interface you want to assign to your custom network, and select the network. Done!

This is already a nice set of features, but the real kicker is still to come. Close the tool, start Fusion, edit another image, select your network adapter, scroll the option list… your custom network is now an option for all machines! This is because Fusion stores network definitions in /Library/Preferences/VMware Fusion/networking - open it with a text editor and you'll see it contains old and new network definitions. UBERNF manipulates this file, and Fusion is happy to go along with it. You could just change subnet values here, I believe.

Unfortunately Nicholas didn't release any source code for UBERNF (I wouldn't be surprised if some of it ended up in Fusion Professional, to be honest), so there is no way to improve it or even just fix little defects that are probably due to OSX changes between 10.7 / Lion (the version he compiled it on) and 10.8 / Mountain Lion. It would be nice if somebody could write a similar app for Mountain Lion / Mavericks and release it as open-source, or simply charging a few dollars less than the difference in cost between Fusion Standard and Professional. Silly, silly VmWare...

Character shifting "encryption" in Windows program -- any idea...?

I could do with some help on this StackOverflow question I just posted. It's one for Windows programmers and/or amateur codebreakers.