Apple Mail + Exchange WebServices = Madness

I bought my first Mac last year, and I've been using the default Mail client ever since. Despite all its shortcomings, it's one of the very few clients supporting Exchange, and probably the only one  I know (apart from Outlook) that supports even its WebService-mode my employer uses. OSX even supports shared calendars!

However, there is one particularly annoying problem I keep encountering.

Exchange has an "auto-discovery" feature, which means that you pass a web address to Mail, the app connects to that URL and it receives the addresses of all necessary servers involved. Say you are a responsible sysadmin, and you run TWO Exchange servers to get some redundancy; when Mail does the auto-discovery dance, it gets told that there are indeed two servers. Mail will try the first one, prompt the user for his credentials, and if successful, it will save server address and credentials in its settings and use it. When the first server goes down, it will switch to the second one, and the user will live happily ever after.

Now, OSX has a system-wide keychain for password storage; basically, applications store all passwords (and other securable objects) in a central, protected place in encrypted form. In order to discern which password is used by which app, they are organized in different "items" with name, description, etc.

It appears that Mail will write each server/user/pass combo in a different item. So when it connects to server1, it will create an item saying "for server 1, connect with username X and password Y"; when it connects to server2, even though it came from the same auto-discoverable Exchange setup, it will write a new item saying "for server2, connect with username X and password Y". When you change your password in Preferences, Mail will update only the last-used item. Which means, the next time you switch server, your authentication will fail at least once. It appears also that Calendar (or any other program using the accounts specified in System Preferences) will randomly pick one of the two.

If you also have the misfortune to install Outlook 2011 for Mac (which you might want to do because some niche Exchange features are not supported by Apple programs), that program will write yet another item.

The result: you have three items for one account -- and this doesn't even include other devices! (phone etc). Whenever your password changes, you have to make sure all of them are updated, or authentication will likely fail somewhere. Because these subsystems poll servers very frequently, and don't give up very easily when refused access, there is a very high chance that a password change will result in your apps triggering a lock-out by constantly trying to use the old password. This is what regularly happens to me! Note that simply changing your password in Preferences might not be enough, because Mail will update only one of the two items.

Workaround: turn off *all* your clients (quit Mail, Calendar, Outlook, shut down your iPhone etc), change the password on Exchange, then open the Keychain ( /Applications/Utilities/Keychain Access.app ), select Passwords, search your username and update all necessary items, then reboot for good measure. Now all apps will use the correct credentials, and not lock you out.

on Aaron Swartz

I've lived through my fair share of internet lore in the last 15 years. Since 2001, Aaron's name kept popping up here and there; he belonged to that special pantheon of people whose genius is so clear, it makes other people despair that they will never, ever measure up to it. When I found out he was A FUCKING KID, well, it wasn't an easy day for my ego. His code was always, invariably fantastic; at a time when people were saying "Python will never match Java for raw performance", he (re)built Reddit to sustain inordinate amounts of traffic, and gave away the code for good measure. He kept contributing his flawless logic to umpteen projects, from Markdown to Django, helping them succeed with grace and selflessness. And then he started going really political, and damn, was he fighting all the good fights. A lot of people claim to be "talented" or be "good", but you could see he was the real deal. He was the sort of person I wanted to be if I only I could rewind my life 20 years and start again.

Back in the '90s, we used to think there was such a thing as "internet culture". We used to think the online world was free, the 'net community would have been immune to the evils of offline corruption, information wanted to be free, "they" couldn't fight disruption, progress was inevitable. 

Lies, all of them. 

We are losing this social revolution like our fathers lost in the '60s and '70s. Established players have made clear, over and over, that they will crush our lofty ideals as soon as we hit their wallets. They will ruin our lives, bankrupt us, force us into exile. Aaron is the last casualty of this counter-revolution. Sooner or later there will be a Vienna Congress moment, indeed it almost happened last month. 

Aaron was the best we could be, and even him couldn't bear the pain of living while fighting the good fight in this corrupt world. How are we supposed to cope? The small ones with simple jobs, normal families, little talent, broken dreams, and our own baggage of bad life choices and mediocrity -- how are we supposed to still believe?

Ah fuck. What a sad day.

Why I've never really liked the Facebook API

The other day, I got an email from Virgin Media stating that my connection had been "upgraded to 100Mb/s". I went to a bunch of speed-testing websites, and reported speeds were indeed much higher than in the past. I was tempted to brag about it on Facebook then I remembered that, last time I did something similar, I was humbled by a bunch of Dutch friends with "big pipes". I wondered what sort of speed they reported then, so I went to Facebook to search for that old status. And that's where my problems started.

The standard FB search UI failed to return anything even vaguely related, as usual. So I started googling for apps that would allow me to search my previous posts, and found a few which just wanted to gather all my personal data (on FB -- you don't say!). Then I found that you can actually request a complete download of all your data from FB (under Settings) and launched the request, but it looked like it would take a long time (for the record, I finally got it about 24 hours later). So I thought "hey, surely I can work with the FB API". How naive of me!

There is, in fact, a straightforward API call to get your statuses: /me/statuses. By default, it will return some 25 records, and links to further paginated results. Except pagination is ridiculously buggy: after the first 50 records, it will just return a blank page. If you try to use the limit parameter, it will return a maximum of 100 records per page, and again it will stop after the second page (i.e. max 200 results, which it's actually 199 because everybody knows "there are only two hard things in computer science"). Time-based parameters (until, since) didn't seem to work at all. Using wrappers rather than direct calls didn't seem to make any difference. Being very late, I gave up and went to sleep.

A day later, still incredulous and obviously fairly frustrated, I googled harder and finally found a relevant question on StackOverflow, which pointed to a Facebook bug about pagination. As the bug says, you can work around the problem by always using offset rather than relying on 'next' and 'previous' links returned in JSON responses. I verified and that's actually the case. By now, my export was available for download anyway. You can imagine how happy I am (not).

Lessons learnt from this whole debacle:

• The unofficial facebook-sdk for python doesn't work with Python 3. There is an experimental fork with very limited testing (i.e. it passes 2to3 and that's it).
• the json module in Python 3 Standard Library, as used by facebook-sdk, chokes on Facebook data. Don't even ask me how I found out. Trying with a more up-to-date version from the original upstream doesn't help. There is a Python 3 fork which didn't help. Juggling between json.load and json.loads didn't seem to help, and I didn't want to rip the guts out of facebook-sdk in fear of dropping compatibility with 2.x (although I cringed at times: using "file" as variable name? Really?). No wonder @kennethreitz rolled his own JSON parser in Requests.
• facebook-sdk should probably be rewritten from scratch in Python 3 using Requests. Not that I'll ever do it.
• After so many years and botched revamps, the Facebook API is still terrible. For something reportedly so essential to "2.0" internet infrastructure, and with so many uber-smart people on their payroll, the whole thing still feels incredibly hackish.

Public therapy

I just experienced my first real professional failure in almost two years of consulting for my current employer. I'm not saying I've always been perfect before, but it's the first time a customer basically told me to give up and go home.

As tempted as I am to blame the damn tool (which I had never seen before in my life, and unsurprisingly refused to do my bidding), the hard truth is that:

• I failed to properly and fully "sniff out" customer requirements in advance. This should have been a huge red flag, but I thought I was good enough to just deal with it. Lesson 1: there is a reason hubris is a cardinal sin.
• On finding myself in trouble, I kept hacking at the problem for days when I should have just taken a step back straight away. I kept googling for a magic bullet, when I should have admitted that I did not know how the product was supposed to work, should have gone back to studying from first principles, and should have built a local proof-of-concept before attempting a real-world deployment. Despite being completely honest with the customer at all times, I ended up over-promising and under-delivering, which is the exact opposite of what I always try to do. Lesson 2: if you find your axe is blunt, hitting faster and from all directions will not compensate; just stop and sharpen up, there is no shame in it.
• Because of this "just a little hack" attitude, deep down I was not fully committed and concentrated on the problem. I kept assuming that solving a second main task would "make up" for failure on the first one. Unfortunately, this second task depended in part on other people, who also failed to deliver on time. Lesson 3: deus ex-machina is a literary device, not an action plan and certainly not a plan B. Also Lesson 4: be brutally honest with yourself at all times.

Obviously, I'm not happy today. Regardless of the actual task at hand, I failed at thinking strategically and being self-aware, and at my age this should not happen; I'm pretty sure I've learnt all these lessons when I was 22, and still managed to forget them. I hope this little recap will help me focus... my next engagement looks like a slam-dunk and I owe to myself to make it so.

"Mark of the Ninja" does not work under VmWare Fusion 5

I installed "Mark of the Ninja" in an XP image under VmWare Fusion 5, hoping the game would have been "simple enough" to run. The image has 4 cores and 4GB of RAM, so plenty of resources. Unfortunately, it looks like it will just crash XP soon after launch. Turning 3D support off, the game doesn't even start. Ah well, i'll have to dig out some old laptop...