tag:blogger.com,1999:blog-36764482024-03-14T08:15:09.438+00:00SubclassedPythonaro, oh-oh-oh.toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.comBlogger365125tag:blogger.com,1999:blog-3676448.post-43991106565466711082022-06-23T09:43:00.004+01:002022-06-23T09:48:10.217+01:00How to add a tachometer to a Triumph Street Twin (2016-2019)<h2>The Problem</h2>
<p>I recently gave in to family tradition: I'm originally from Bologna, home of <a href="https://www.ducati.com/" target="_blank">Ducati</a> and heart of <a href="https://www.bolognawelcome.com/en/blog/the-motor-valley" target="_blank">the Italian "Motor Valley"</a>, so sooner or later I had to buy a motorbike (or two, but nevermind).</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiC3KW81DlbE6pbwtuDdllRiEv7JbtxyDCFeCuGrNaLrcSl1uuhK8fockkzhFkxNA5PuaSdG-CRqAPjv7DmZoJc1VzBZCyPgA9wJGLFiz0ZNQzNYgsT3H5t60RgP0WZYCvAUgye8l8t2HBejFKzv9Pf5OLGR214kiGzv9XMwbajK-NnO051vQ/s1556/IMG20220507121340.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1167" data-original-width="1556" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiC3KW81DlbE6pbwtuDdllRiEv7JbtxyDCFeCuGrNaLrcSl1uuhK8fockkzhFkxNA5PuaSdG-CRqAPjv7DmZoJc1VzBZCyPgA9wJGLFiz0ZNQzNYgsT3H5t60RgP0WZYCvAUgye8l8t2HBejFKzv9Pf5OLGR214kiGzv9XMwbajK-NnO051vQ/s400/IMG20220507121340.jpg"/></a></div>
<p>As the main workhorse I went for a <a href="https://www.motorcycle.com/specs/triumph/standard/2018/street-twin/base/detail.html" target="_blank">2018 Triumph Street Twin</a>, because it's practical but still fun, fairly cheap, and simply beautiful. I love it to bits, but one thing I really missed: a tachometer ("rev counter"). While trying their hardest to segment the Bonneville range, Triumph clearly thought that people would pay a few grand more and get a Speed Twin just so that they could see RPMs. It took them about 4 years to understand the silliness of this position (or to produce enough Bonneville variations that nobody cares about that bit anymore), so post-2020 Street Twins display RPMs on the digital display - in a pretty small way, but at least it's there.</p>
<h2>The Solution</h2>
<p>So, how could we go about adding a tachometer to pre-2020 models? Various web searches produced a few different methods, with a number of pitfalls. There are a few electrical schematics floating around, but I'm not good at that sort of thing and comments seemed to be that they are easy to get wrong.<p>
<p>A much easier method is to attach an OBDII ("OBD2") reader to the related port, which is available under the seat.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjo-pgOhrHFGfhReyY4tZmgOvvZeM202CU_NbTyf9TCycg4pdF6vnWZ9I7-xvTkyDsQggfAWK7j4pB9heFUfL4N5JnmsgHpoBj9pRNsRqBLlenGtxUnpHtPJkn3vGwQrfEeKsYYWIgk3ZnXBTPgZtqT_A-ekkVU9hvEud-Js7sW9r-GMRVfMQ/s1771/IMG20220622124158.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1328" data-original-width="1771" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjo-pgOhrHFGfhReyY4tZmgOvvZeM202CU_NbTyf9TCycg4pdF6vnWZ9I7-xvTkyDsQggfAWK7j4pB9heFUfL4N5JnmsgHpoBj9pRNsRqBLlenGtxUnpHtPJkn3vGwQrfEeKsYYWIgk3ZnXBTPgZtqT_A-ekkVU9hvEud-Js7sW9r-GMRVfMQ/s400/IMG20220622124158.jpg"/></a></div>
<p>OBDII is a standard interface for debugging systems in vehicles; it's recently been mandated by the European Union, so it's available in pretty much any vehicle produced in the last decade or so. Once you connect a reader, you can see a wealth of information about the vehicle, from fuel consumption to engine load to pretty much any sensor available... including RPMs! So that's how we can get a tachometer without effectively modifying anything in the bike itself.<p>
<p>There are two ways to show the extracted information: wired, and wireless. Each comes with pros and cons.</p>
<h3>Wireless (Bluetooth)</h3>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5fTo8JdHmSZ7KVt6YhOGix60G-SCwL2boEN-G2VeLY4FDOUc3HwI9sfJbmausnpipnxXvWel99UN18vBiqU_FyCtZcs1-IKmGJArWGwf3JbTBsec0Dbg5eThuvf5Uar7EfdhxusOPrHL0F_T-OFFTY6YRD2d7fMLV1IJhIlwdaojALDHoSA/s1319/IMG_20220623_093457.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="320" data-original-height="1266" data-original-width="1319" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5fTo8JdHmSZ7KVt6YhOGix60G-SCwL2boEN-G2VeLY4FDOUc3HwI9sfJbmausnpipnxXvWel99UN18vBiqU_FyCtZcs1-IKmGJArWGwf3JbTBsec0Dbg5eThuvf5Uar7EfdhxusOPrHL0F_T-OFFTY6YRD2d7fMLV1IJhIlwdaojALDHoSA/s320/IMG_20220623_093457.jpg"/></a></div>
<p>Wireless is less intrusive but fiddlier: a compact reader stays entirely under the seat, and communicates with a phone or tablet over Bluetooth (as far as I know, there are no dedicated wireless displays). It's described in <a href="https://amzn.to/2CoSARQ" target="_blank">this Youtube video</a>, and the dongles are <a href="https://amzn.to/3zUxlFj" target="_blank" rel="sponsored">easy to find on Amazon</a>. As a permanent solution, it suffers from lag and the requirement that the phone screen stays on all the time, set on the specific app. (Side note: I found the best iOS app to be <a href="http://www.fourstroke.ca/" target="_blank">FourStroke</a> - most of them are focused on cars and might not even work with cheap aftermarket readers.) I have an old phone I mount on handlebars, but I typically want to use it to display GPS stuff and play music, so that just didn't work for me. You also have to manually open the app and connect every time, after you start the bike, which is annoying.</p>
<h3>Wired</h3>
<p>A wired solution, on the other hand, requires a bit more setup, but you do it once and that's it, you don't have to worry about it ever again; readings are a bit less laggy (although still not as quick as hardware ones), and leave your phone free to do other stuff. They typically also feature hardware controls, which are easier to use with gloves than touchscreens (yes yes, I have modern gloves - in practice, they are still too big to be precise).</p>
<p>There are a few OBDII readers on the market, largely meant for the car market. I grabbed <a href="https://www.aliexpress.com/item/32886750530.html" target="_blank">this cheap thing from AliExpress</a>, but there are <a href="https://amzn.to/3NgV4Cm" target="_blank" rel="sponsored">equivalents on Amazon</a> and so on. Setup is trivial: you connect it to the port, turn on the bike, and it just works, letting you choose what to display. You turn the bike off and the display shuts down automatically, so your battery is not drained. Nice!</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjo-pgOhrHFGfhReyY4tZmgOvvZeM202CU_NbTyf9TCycg4pdF6vnWZ9I7-xvTkyDsQggfAWK7j4pB9heFUfL4N5JnmsgHpoBj9pRNsRqBLlenGtxUnpHtPJkn3vGwQrfEeKsYYWIgk3ZnXBTPgZtqT_A-ekkVU9hvEud-Js7sW9r-GMRVfMQ/s1771/IMG20220622124158.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1328" data-original-width="1771" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjo-pgOhrHFGfhReyY4tZmgOvvZeM202CU_NbTyf9TCycg4pdF6vnWZ9I7-xvTkyDsQggfAWK7j4pB9heFUfL4N5JnmsgHpoBj9pRNsRqBLlenGtxUnpHtPJkn3vGwQrfEeKsYYWIgk3ZnXBTPgZtqT_A-ekkVU9hvEud-Js7sW9r-GMRVfMQ/s400/IMG20220622124158.jpg"/></a></div>
<p>Finding a location for the wire is basically the same as for the USB you also find under the seat; if you've wired it up to a handlebar mount for phones, as I have, you can basically do the same - it's actually easier here, since you can push it further back with the other cables.</p>
<div class="separator" style="clear: both; text-align:center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrvmjeQRKPM4cXzFqIM06XDE95AaOKdR0SRFW2FRTEkBt10IfNp5waTqTubmrg96TsxUmsepsnRpbsdiaDPqvUKLAIFmQ-7v3eXY2oApEtP2QjFHQsc060hMh4wCbpzNA6d9Iwh_bTo2Mfx77jWpmgkAYa-1aXEHu0V6Iu9Kdrl7RezaeV9g/s2070/IMG20220622124022.jpg" style="padding: 1em 0; text-align: center; "><img alt="" border="0" width="200" data-original-height="1552" data-original-width="2070" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrvmjeQRKPM4cXzFqIM06XDE95AaOKdR0SRFW2FRTEkBt10IfNp5waTqTubmrg96TsxUmsepsnRpbsdiaDPqvUKLAIFmQ-7v3eXY2oApEtP2QjFHQsc060hMh4wCbpzNA6d9Iwh_bTo2Mfx77jWpmgkAYa-1aXEHu0V6Iu9Kdrl7RezaeV9g/s200/IMG20220622124022.jpg"/></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjl3kBpOirJr2cCBAdpXVdu2FXx9ECPOiK7vEE7JdSlnB2cWX7Pz-66IDEuwrTCgncQLznqD1aVovtJhaTvS46jh6HVkw86lO5FBe-15yNnqZ0l_jUzt5f3XGj2BiAH59e21ckvBqIytsoYp1zPdCKCb99h9VEIWWDHTEca6edLpoeC8FVZwQ/s1757/IMG20220622123838.jpg" style="padding: 1em 0; text-align: center; "><img alt="" border="0" width="200" data-original-height="1318" data-original-width="1757" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjl3kBpOirJr2cCBAdpXVdu2FXx9ECPOiK7vEE7JdSlnB2cWX7Pz-66IDEuwrTCgncQLznqD1aVovtJhaTvS46jh6HVkw86lO5FBe-15yNnqZ0l_jUzt5f3XGj2BiAH59e21ckvBqIytsoYp1zPdCKCb99h9VEIWWDHTEca6edLpoeC8FVZwQ/s200/IMG20220622123838.jpg"/></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrLbkuzfUkVWeSvfpjt7gQocIctWaRFjGRd2BdjRylxorH-E3C6WI_EeuI6jthhOL92cZh8Pi4Wila4yQrv6Nw_ia2n08sPZ2SGrNdfEQ1PQ-021YLqkRqvFPHh05z-3ajS4DFOiMG1sf-JEnVGk-WEPPBcz4O9MgDiV7qPIvdTbvpTtvf-Q/s1942/IMG20220622123833.jpg" style="padding: 1em 0; text-align: center; "><img alt="" border="0" width="200" data-original-height="1457" data-original-width="1942" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrLbkuzfUkVWeSvfpjt7gQocIctWaRFjGRd2BdjRylxorH-E3C6WI_EeuI6jthhOL92cZh8Pi4Wila4yQrv6Nw_ia2n08sPZ2SGrNdfEQ1PQ-021YLqkRqvFPHh05z-3ajS4DFOiMG1sf-JEnVGk-WEPPBcz4O9MgDiV7qPIvdTbvpTtvf-Q/s200/IMG20220622123833.jpg"/></a></div>
<h4>Unresolved issues</h4>
<p>There are a few annoyances though, due to the fact that this is fundamentally a product for cars:</p>
<ul>
<li>The display can be a bit hard to see in daylight, particularly when the sun shines directly on it.</li>
<li>You'll need to hack together a mounting system; the stand that comes with the product is meant for car dashboards. You can see my solution in the pic of the display above - not the most beautiful, but it will do for now; I plan to revisit it at some point.</li>
<li>It's not waterproof. In the rainy North of England, this is a significant problem. I'm trying to think about solutions; if you have any suggestion, feel free to drop a comment.</li>
</ul>
<p>All these issues could be solved if manufacturers started making readers dedicated to the motorbike market. I hope that happens at some point, because it would be awesome - it would free consumers from the tiranny of OEM displays.</p>
<h3>Hot Engine is Hot</h3>
<p>
Last bit of advice: while you're testing, be careful with your dangling cables! As you can see from the pic below, I mistakenly left mine over the exhaust for a few seconds and... well. Amazingly, it still works!</p><div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJjV_V5dAhpn5FCS_dPbQeoWX_ozansXl6QI_FWWKcLYPB91i4jFZdjBx0Wz2ZHY3Y6TmoR2UhaC4AU3KjabrlkZO8E3Tb5UyDzec3BP1C8tBuSMmdXWym0jmq_uwpV2v9QXgMoeLi6yAawkgQtqkUppVLCXhWV3yaGhN5bKbP3mO_d1vL0w/s1600/signal-2022-06-22-12-26-49-634-1.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" height="400" data-original-height="1600" data-original-width="900" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJjV_V5dAhpn5FCS_dPbQeoWX_ozansXl6QI_FWWKcLYPB91i4jFZdjBx0Wz2ZHY3Y6TmoR2UhaC4AU3KjabrlkZO8E3Tb5UyDzec3BP1C8tBuSMmdXWym0jmq_uwpV2v9QXgMoeLi6yAawkgQtqkUppVLCXhWV3yaGhN5bKbP3mO_d1vL0w/s400/signal-2022-06-22-12-26-49-634-1.jpg"/></a></div>toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0tag:blogger.com,1999:blog-3676448.post-65831821408036161312022-05-30T12:45:00.002+01:002022-05-30T12:49:39.031+01:00On OneStream<p>If you follow <a href="https://www.linkedin.com/in/glacava/" target="_blank">me on LinkedIn</a>, you might have noticed that, about two years ago, I joined <a href="https://onestreamsoftware.com/" target="_blank">OneStream</a>.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgprludVrKqcaFLet9ZN1li-iNlMV9_leN_4TCqrtJzyiszuaC-oGVg9ytKNvmpxWNBropI2sra5BdMjxXDsOsthgYcmPnJf_R1YKF4fHbjyEKf55rjYxXwDgi4dfFwEFduaRq6tRWgYE8BAPEHIM3olxgkcvivunmTrX5Usv79CIf8E8hltA/s3568/cropped-OS-LogoTM-Horizontal-FC-RGB.webp" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="320" data-original-height="664" data-original-width="3568" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgprludVrKqcaFLet9ZN1li-iNlMV9_leN_4TCqrtJzyiszuaC-oGVg9ytKNvmpxWNBropI2sra5BdMjxXDsOsthgYcmPnJf_R1YKF4fHbjyEKf55rjYxXwDgi4dfFwEFduaRq6tRWgYE8BAPEHIM3olxgkcvivunmTrX5Usv79CIf8E8hltA/s320/cropped-OS-LogoTM-Horizontal-FC-RGB.webp"/></a></div>
<p>I've since refrained from writing about it, for a number of reasons: the product is massive, so it took a while to get to grasp with it; my new role kinda constrained what I could talk about; and I thought I wasn't particularly well-qualified yet to speak about the subject.</p>
<p>I recently attended the <a href="https://splash.onestreamsoftware.com/" target="_blank">Splash</a> conference for the first time. One of the things I brought home from San Antonio (together with a certain virus most people thought defeated) was the belief that, by now, I actually know a few things about OneStream - and there is a big hunger for that knowledge among clients and partners. The leadership is aware of this, so it was easy to get the greenlight on a few related projects.</p>
<p>This means that I'll be writing a bunch of posts in the next couple of months, on OneStream-related subjects. They might not be published here, but wherever they end up, I'll make sure to link them from here. I'm a geek, not a marketer, so they will be technical posts about getting stuff done; there is already<a href="https://onestreamsoftware.com/resources/" target="_blank"> plenty of material on why you should use OneStream for your planning and financial consolidation needs</a> - what people need is to learn <b>how</b> you can do that, and that's where I'm going to help.</p>
<p>In the meantime, if you'd like me to cover a particular topic, feel free to reach out (here, <a href="https://www.linkedin.com/in/glacava/" target="_blank">on LinkedIn</a>, or at my OneStreamSoftware.com address (glacava@).</p>
toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0tag:blogger.com,1999:blog-3676448.post-24529984721121060852022-03-12T01:15:00.001+00:002022-03-12T01:21:18.928+00:00Detecting Badger2040 boards and automating uploads<p>I recently bought a bunch of <a href="https://shop.pimoroni.com/products/badger-2040" target="_blank">Pimoroni Badger2040</a> boards, and they are a lot of fun.</p>
<p>The Badger is basically a small microcontroller (the <a href="https://www.raspberrypi.com/documentation/microcontrollers/raspberry-pi-pico.html" target="_blank">Raspberry Pico</a>) with an <a href="https://en.wikipedia.org/wiki/E_Ink" target="_blank">eInk</a> display, in the size of a typical office badge. It has a few buttons you can interact with, when powered, but because of eInk it doesn't actually need to be powered all the time - you can just set it to the desired screen, turn off the battery, and the screen will stay as it was more or less forever.</p>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhQjfsx9yvx5S3ZwufBblUw4f2s0RKSV4KFs6FW4uDXwIknQMsKLqpYrOfUaZAgDrDEh9qHUsW7qeydKdyERK5ewr4Jkrl8ovd_6AhmMmqN5Tws8y-4hmX81M7ArFP6z7-SLQEGjhjFpXB6bGptN6qGf6ZCbskvG_bs7xH3Gpc2--1P9HTr1w=s2799" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="320" data-original-height="1698" data-original-width="2799" src="https://blogger.googleusercontent.com/img/a/AVvXsEhQjfsx9yvx5S3ZwufBblUw4f2s0RKSV4KFs6FW4uDXwIknQMsKLqpYrOfUaZAgDrDEh9qHUsW7qeydKdyERK5ewr4Jkrl8ovd_6AhmMmqN5Tws8y-4hmX81M7ArFP6z7-SLQEGjhjFpXB6bGptN6qGf6ZCbskvG_bs7xH3Gpc2--1P9HTr1w=s320"/></a></div>
<p>The fun bit is that it can run <a href="https://micropython.org/" target="_blank">MicroPython</a>, so programming it is a breeze. You don't have to deal with all the scary vagaries of C/C++; just write your Python scripts, save them to the board, and run them. Sweet!</p>
<p>There is already <a href="https://learn.pimoroni.com/article/getting-started-with-badger-2040" target="_blank">a fairly comprehensive tutorial on how to get started with Badger2040</a>, but (like most Pico-related documentation out there) it assumes you're happy to use <a href="https://thonny.org/" target="_blank">Thonny</a>, an editor focused on the micropython ecosystem, in order to move files to the board. With all due respect, Thonny is a very limited editor, and it gets recommended only because it's the most intuitive when it comes to managing files on the Pico. I'm much happier when I live in my beloved PyCharm, but its MicroPython plugin is somewhat limited and requires manual interaction, so I investigated a strategy to automate the basic stuff directly from Python on my laptop.</p>
<p>The first step is detecting the board. It appears to the operating system as a serial port, so we have to list the available ports and find the one that looks like our guy.</p>
<pre> # badgerutils.py
import serial.tools.list_ports as list_ports
from serial.tools.list_ports_common import ListPortInfo
def is_badger(port: ListPortInfo):
""" decide if the port looks like a Badger2040 """
# mac, but other systems will probably be similar,
# just add other "if" blocks for windows etc
if sys.platform.startswith('darwin'):
# you should be more thorough,
# might want to check VID etc, but this will do for dev
if port.manufacturer and \
port.manufacturer.lower().startswith('micropython'):
return True
return False
def get_badger():
""" loop through all the ports and find our board """
ports = list(list_ports.comports())
for p in ports:
if is_badger(p):
return p
</pre>
<p>The next step is where things get a bit hairy. Interacting over the serial port is not everyone's idea of fun, so we better stand on the shoulder of geeky giants if possible. We could dig through Thonny's code, but it's long and complicated and meant to support a lot of scenarios we don't really care about. Instead, we can reuse a little utility called <a href="https://github.com/scientifichackers/ampy" target="_blank">ampy</a>, which is slightly old but fairly robust and (more importantly) self-contained and easy to understand.</p>
<p>Ampy includes a couple of modules to interact with a micropython board. You can have a look at the functions found in its <code>cli</code> module to figure how to wrap them, but here's a simple approach to start pushing files to the board - some of the code is lifted almost entirely from <a href="https://github.com/scientifichackers/ampy/blob/master/ampy/cli.py" target="_blank">ampy.cli</a>, but it's MIT-licensed, so you can do that (just mention the original copyright notice somewhere, if you publish it!).</p>
<pre># BadgerManager.py
from serial.tools.list_ports_common import ListPortInfo
from ampy.files import Files, DirectoryExistsError
from ampy.pyboard import Pyboard
class MyBadger(Pyboard):
def __init__(self, port: ListPortInfo):
super(MyBadger, self).__init__(port.device)
self.files = Files(self)
def upload(self, file_path: Path, dest_path: Path):
""" upload file or directory to board """
if file_path.is_dir():
# Directory copy, create the directory and walk all children
# to copy over the files.
for parent, child_dirs, child_files in os.walk(file_path):
# Create board filesystem absolute path to parent directory.
remote_parent = posixpath.normpath(
posixpath.join(dest_path, os.path.relpath(parent, file_path))
)
try:
# Create remote parent directory.
self.files.mkdir(remote_parent)
except DirectoryExistsError:
# Ignore errors for directories that already exist.
pass
# Loop through all the files and put them on the board too.
for filename in child_files:
with open(os.path.join(parent, filename), "rb") as infile:
remote_filename = posixpath.join(remote_parent,
filename)
self.files.put(remote_filename, infile.read())
else:
# File copy
# check if in subfolder
if len(dest_path.parents) > 1:
# subfolder was specified
# each parent has to be created individually,
# because of ampy limitations
for d in sorted(dest_path.parents)[1:]: # first is /, discard
self.files.mkdir(d)
# Put the file on the board.
with open(file_path, "rb") as infile:
self.files.put(dest_path.absolute(), infile.read())
def ls(self, dirname='/', recurse=True):
""" List files on board """
dirpath = dirname if type(dirname) == Path else Path(dirname)
return self.files.ls(dirpath.absolute(),
long_format=False, recursive=recurse)
</pre>
<p>Putting both things together we can interact very easily with the board like this:</p>
<pre>
from badgerutils import get_badger
from BadgerManager import MyBadger
# Note: in real life, remember to manage error conditions !
port = get_badger()
board = MyBadger(port)
board.upload("./something.txt", "/something.txt")
assert('/something.txt' in board.ls())
</pre>
<p>Happy hacking!</p>toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com1tag:blogger.com,1999:blog-3676448.post-48751603022935191912020-06-06T14:11:00.019+01:002021-03-19T07:10:46.923+00:00Better access to special characters with AutoHotkey on Windows <p>EDIT 2020-06-21: I tweaked the layout a bit, and updated screenshot and scripts.</p>
<p>When you're trying to improve your typing skills, there are quite a few things you can do: <a href="https://www.keybr.com/">learning to touchtype</a>, <a href="http://blog.pythonaro.com/2015/01/kinesis-freestyle2-multichannel.html">getting an ergonomic / split keyboard</a>, or <a href="https://colemak.com/">moving to a better layout than QWERTY</a>. However, if you're like me (small hands, short pinkie), chances are that none of these will be of much help when you have to type a lot of special characters, for example in programming. That's because special characters are typically hard to reach. Most layouts banish them to the edges of town, leaving them almost entirely to the right pinkie and to shift+<number>, which forces your hands to wander very far from the home-row on which your muscle-memory is based.</p>
<p>I'm currently experimenting with a solution to this state of things. Thanks to a wonderful little program called <a href="https://www.autohotkey.com/">AutoHotkey</a>, you can tweak your keyboard in great ways; what I decided to do was to leverage the largely-unused (but very easy to reach) CapsLock. I basically turned CapsLock into a new meta key (which is not Ctrl, Alt, AltGr, Win or Cmd), allowing me to get a completely blank layer that is independent of any existing key or shortcut. I then associated the most easily-reachable keys to the most common (and hardest to reach with typical layouts) special characters I need.</p>
<p>The result is that, by pressing capslock+<home-row-key>, I now get special characters with less effort and less wandering.</p>
<blockquote class="imgur-embed-pub" lang="en" data-id="a/4iDsebe" data-context="false" ><a href="//imgur.com/a/4iDsebe"></a></blockquote><script async src="//s.imgur.com/min/embed.js" charset="utf-8"></script>
<p>What you see above is the layout I'm currently using. It's not perfect, but the principles are:</p>
<ul>
<li>optimize the position of keys I find least-reachable and most-used on a regular layout</li>
<li>privilege right-hand keys, which are the most natural companions to a left-hand meta</li>
<li>privilege opening brackets, as editors typically auto-close them</li>
<li>try to minimize "wandering" of hands from home-row as much as possible</li>
</ul>
<p>I've also added a numpad on CapsLock+Shift, which is useful on laptop keyboards. Yes, you often have a hardware NumLock mode, but I never use it because I find it risky (if you mistakenly leave it on and the screen locks, good luck typing your password).</p>
<p>NOTE: the <code>ALT+`</code> combo is a "mac-ism" - it's actually <code>AltGR+`</code> on Windows (or <code>Ctrl+Shift+Alt+`</code>). It's the shortcut to prepend to a vowel to get a grave-accented character. I'm Italian, so I use it to type accents on a US keyboard.</p>
<p>I wish someone would come up with a "standard" meta-layout like this, with some real thought to ergonomics and frequencies; then again, programming languages can vary so much (for example there are lots of $ in Perl, but very few in Python) that I guess it would be difficult to appease everyone.</p>
<p>Here is a <a href="https://gist.github.com/toyg/35f08f85ca5cc98341ac32f2cb2e7fb4">AutoHotkey script for QWERTY</a> and <a href="https://gist.github.com/toyg/a9128d08fe2fb9e6ab574a8247cf4fe6">AutoHotkey script for COLEMAK</a> (that's actually what I use). If you install AutoHotkey, just save the script as <code>AutoHotkey.ahk</code> in the resulting installation folder and it will be automatically executed when you start the program (<a href="https://www.autohotkey.com/docs/Program.htm#run">it can also be run at startup</a>).</p>
<p>If you are on macOS/OSX, things are a bit more awkward; I might cover that in another post at some point, but my solution there relies on a smart external keyboard. Happy hacking!</p>toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com1tag:blogger.com,1999:blog-3676448.post-86468272052436730722020-04-03T22:15:00.001+01:002020-04-03T22:15:34.736+01:00Django + PostgreSQL + Docker-Compose: a few gotchas<p>A lot of the tutorials out there make it look like it's trivial to set up a development environment with Django, Postgres and Docker. That never quite matched my experience; you always end up knowing too much about docker, and there are a few gotchas that most people typically fail to mention. The following are a few specifically related to Postgres and Django, which I'm writing here because I tend to forget them every time I start a new project...</p>
<h2>A running container is not a running database</h2>
<p>Docker-compose will happily report a container as "up" even though it's busy doing init work. With postgres, this means that a container might look "up" when really it's still creating the actual db instance, so app connections might well fail.</p>
<p>A good workaround is to use <code>pg_isready</code>, like this :</p>
<pre>
#!/usr/bin/env sh
docker-compose up
until pg_isready -d your_pg_db -h your_pg_host \
-p your_pg_port -U your_pg_superuser
do
echo "Waiting for db to be available..."
sleep 2
done
# now we can do actual work, like db migrations
...
</pre>
<h2>Don't run; exec</h2>
<p>A lot of howtos state, more or less, "if you want to run something in an instance, use <code>docker-compose run some_machine some_command</code>". This is misleading. <code>run</code> will create a <i>new</i> ancillary container, which will run in parallel to any other container of the same type that might already be up. If you want to execute an ancillary process <i>inside an already-running container</i>, use <code>docker-compose exec some_machine some_command</code> instead. This will ensure you are "logged on" the running container.</p>
<h2>While coding, don't copy; mount</h2>
<p>Many will tell you that you need to ensure reproducibility; and as such, your code should be copied or checked out to the instance in Dockerfile, i.e. at build stage. That is a huge drag on development, since you need to rebuild the whole image on every minor change. It is annoying and slow even with multi-stage builds.</p>
<p>Instead, you can mount your actual source directory as a volume, and exploit all the goodies that make development tolerable, like Django's autoreload features. Make your docker-compose.ymllook like this instead:</p>
<pre>
services:
your_app_machine:
volumes:
- type: bind
source: /host/location/of/src
target: /container/location/of/app
...
</pre>
<p>When you want to run tests or go to production, use a second Dockerfile that inherits from the first (with FROM) and actually copies data (or more likely checks it out via git), without the volume definition.</p>
<h2>Know your tools</h2>
<p>This is not really specific to docker! Master your tools in depth, it will help. I honestly didn't know that JetBrains PyCharm can now configure the interpreter running in a Docker container as the main one for the project, which makes a lot of things easier (debugging, REPL etc). Extremely helpful!</p>toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com1tag:blogger.com,1999:blog-3676448.post-18860615735405079762020-02-06T17:59:00.005+00:002020-12-14T01:05:16.637+00:00Announcing the AutoEPM Patch Tracker<p>UPDATE: This service has been discontinued; this post is left as historical reference. If you need this sort of thing, <a href="https://linkedin.com/in/glacava" target="_blank">contact me on LinkedIn</a>.</p>
<p>With the recent release of EPM 11.2, I thought it would be a good time to make public a tool I've built and used "behind the scenes" for quite some time, <a href="https://tracker.autoepm.co">the AutoEPM Patch Tracker</a>.</p>
<p>It's a patch tracker focused on EPM products, automatically aggregating updates and pointing to readmes, hopefully a bit faster to peruse than My Oracle Support. It also recommends Critical Product Updates that can be applied to the EPM stack, and the most common "standard" updates (only for 11.1.2.4 at the moment). With a free account via LinkedIn sign-on, all lists can be downloaded in CSV, JSON, and XML (Atom).</p>
<p>It's only a small showcase of what we can do at <a hrif="https://autoepm.com">AutoEPM</a> by Targlet. If you'd like to find out how to integrate this level of automation and intelligence to your EPM enviroments, feel free to <a href="mailto:info@autoepm.com">reach out</a>! I'll be at OpenWorld London next week as well, always happy to have a chat.</p>toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0tag:blogger.com,1999:blog-3676448.post-42698080645056611482020-01-10T17:01:00.000+00:002020-01-10T17:37:53.716+00:00Oracle Hyperion EPM 11.2 - first impressions<p>Last month, Oracle finally released the long-awaited on-premises release of EPM 11.2, rumoured to be the last hurrah for the venerable suite. After a couple of test installations, I have a few thoughts from an infrastructure perspective.</p>
<p> First, the bad news:</p>
<ul>
<li>This is very much a .0 release, in many ways the first since 11.1.2.0 (the last time so much of the infrastructure had substantially changed). It is rough in quite a few places, and configuration is not for the faint of heart.</li>
<li>I have encountered what, to me, looks like a blocking bug in multi-server configurations. I filed a support request with Oracle and will see how it goes. At the moment I simply wouldn't be able to recommend it in situations where high-availability is necessary.</li>
<li>Even after configuration, it is harder to operate by non-techies than in the past. Even the "start" and "stop" links provided, which used to be ok for single-machine use at least, are not sufficient anymore - some extra scripting will be necessary to make it useable even in the most basic scenarios.</li>
<li>Documentation has not fully caught up with 12c, particularly in the area of additional configuration tasks for Financial Close Manager (which was already pretty lacking). Some of the additions, to deal with the now-necessary Repository Creation Utility, are ambiguous.</li>
<li>Weblogic seems hungrier for memory than in previous releases. This might be due to 12c, or to the fact that it's now using Java 8 under the hood - I had already measured an increase in this area when replacing Java 6 with 7 on 11.1.2.4.</li>
<li>Essbase seems to have been shipped almost unmodified from the latest 11.1.2.4 version. Among other things, this makes it currently impossible to do a full-SSL configuration if Essbase is in the picture. It's disappointing, particularly considering how Essbase customers are typically among the few who often require fully-encrypted solutions.</li>
<li>The documentation seems to state that running any "Configure Database" task more than once will end in failure. I suspect this is true only for certain components, but I have not had the time yet to explore this area. This makes it even harder than before to migrate databases after configuration, so make sure you take design choices that will stand the test of time.</li>
<li>EPMA lives! Well, not really - the product is gone, but somehow it's still avilable in the Provisioning screen and you can assign roles for it. 🤷🏻♂️</li>
<li>This is the end of the line for IIS stalwarts. Using IIS as front-end webserver will simply not work (according to docs).</li>
<li>It's not particularly clear whether Oracle 18c and 19c are officially supported as database. In practice they seem to work (after all, they are really 12.2.0.2 and .03 rebranded), but it would be nice if the matrix was clarified.</li>
</ul>
<p>But there are also some good news!</p>
<ul>
<li>The products themselves are basically the same, with the occasional extra feature. Users shouldn't need any substantial retraining, with the notable exception of any EPMA or Interactive Reporting power-users still around.</li>
<li>The 12c stack, in theory, unlocks a number of attractive possibilities for system administrators and implementors - containers, no-downtime patching, and so on. It will take some time to fully unpack how EPM can take advantage of these goodies.</li>
<li>Recent Windows Server releases are supported.</li>
<li>In my test, everything seems to work even with the free editions or Oracle and SQLServer (XE 18c and Express 2017), which is nice for consultants and other professionals.</li>
<li>Both 12c and Java 8 support modern crypto standards, which should make things less awkward when risk-assessing and pentesting.</li>
<li>Java 8 brings a lot of improvements to the language and the ecosystem. Whole classes of problems have been removed (MaxPermGen, anyone?), which should make services more reliable and performant. Custom integrations will also be easier and faster to develop and run.</li>
</ul>
<p>In the past, I would have expected such a release to be followed by a quick stream of fixes, necessary to make it actually fit for production use. We will have to see if it is going to be the case here, considering how long it took to debut and the overall tone of the Oracle roadmap (mostly cloud-focused). Should Oracle really commit to improving the on-premise, this release would soon prove itself as a substantial improvement over 11.1.2.4.</p>
<p>If you'd like to know more and discuss your options in the EPM space, make sure to <a href="https://autoepm.com">get in touch with us</a>.</p>toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com8tag:blogger.com,1999:blog-3676448.post-24924960919401818652019-06-18T12:43:00.000+01:002019-06-18T12:43:38.073+01:00How to dump macOS security-rights database<p>macOS handles some security items in a custom database, which may or may not be SQLite. The official way to interact with such database, outside of Objective-C, is the <code>/usr/bin/security</code> utility, with the parameter <code>authorizationdb</code> and specifying the required operation on the class of rights, e.g. <code>security authorizationdb read system.login.console</code><p>
<p>Unfortunately, there is no way (that I could find) to simply list all classes. I was trying to uninstall something that might have had references in that db, but I wasn't sure about the class it might be registered under, so I wanted to dump them all.</p>
<p>Luckily I found what looks looks like <a href="http://www.dssw.co.uk/reference/authorization-rights/index.html">a comprehensive list of rights</a>. After a quick scraping job with Python, I had a list that I could use like this:</p>
<code>cat osxrights.txt | xargs -I % sh -c 'sudo security authorizationdb read %' | grep -B 10 myAnnoyingItemToRemove</code>
<p>... and the bundle wasn't anywhere, so I could just chuck it. </p>
<p>(Note: had the item been found, I would have had to dump the whole security class to a .plist file with <code>security authorizationdb read the.class > my.plist</code> , edited the file to remove it, then write it back to the db with <code>security authorizationdb write the.class < my.plist </code>)</p>toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0tag:blogger.com,1999:blog-3676448.post-62269035436654606592019-03-14T12:32:00.001+00:002019-03-14T12:32:28.843+00:00How to remove all bars in fullscreen mode in Firefox on MacOS/OSX ("kiosk mode")<h2>Procedure</h2>
<p>If you want to remove all toolbars from Firefox in fullscreen mode on Mac, this is the procedure:
<ol><li>Type <code>about:config:</code> in the location bar to open advanced settings. (If it is the first time you do it, you might have to click "confirm" or something like that in the scary screen that comes up first.)</li>
<li>search for <code>full-screen-api.allow-trusted-requests-only</code> and double-click the line that is found, so the value changes to <code>false</code></li>
<li>Bring up the Bookmark Toolbar if you don't have it (menu <i>View -> Toolbars -> Bookmark Toolbar</i>)</li>
<li>Right-click on empty space in the Bookmark Toolbar, and select <i>New Bookmark...</i></li>
<li>In the <em>Name</em> field, type something like "Enter Fullscreen" (it doesn't really matter, just don't leave it blank)</li>
<li>In the <i>Location</i> field, enter the following line:
<pre>javascript:document.getElementsByTagName('html')[0].mozRequestFullScreen();void(0)</pre>
</li>
<li>Click <i>Add</i> to save the bookmark.</li>
</ol>
Now, when you click on that bookmark, Firefox will go "real fullscreen", hiding all toolbars. To get them back, you will have to press ESC.</p>
<p>Now, there is a bit of a snag. That preference in <code>about:config:</code> is set to true by default for a reason: it is a bit unsafe to allow any website to use fullscreen-enabling javascript. For this reason I would suggest to only do this for limited periods, then switch it back once you're done.</p>
<h2>Long-winded explanation</h2>
<p>FF has a preference, <code>browser.fullscreen.autohide</code>, that <em>in theory</em> should govern the behaviour of toolbars when going fullscreen. It is set to true by default, meaning toolbars should disappear. However, for some reason this property has (almost) always failed to work on MacOS/OSX builds of Firefox.</p>
<p>There used to be a few extensions that corrected that behaviour, but they've all gone away when Firefox switched to <a href="https://browserext.github.io/browserext/">WebExtension APIs</a>, which basically sandboxed extensions more rigidly and blocked them from changing the browser interface as deeply as they could before. This was the price for the dramatic performance improvement seen in Firefox 57+.</p>
<p>This workaround hence relies on the new <a href="https://developer.mozilla.org/en-US/docs/Web/API/Fullscreen_API">Fullscreen WebAPI</a>, which is supposed to be a new, standard way to enable fullscreen via Javascript on any browser (well, any browser that implements it - just a few at the moment). However, because of security concerns, this API is locked down by default; so, in order to use it, we first have to relax checks as described above. Once that is done, we can launch a bit of JS from a bookmark that triggers "real" fullscreen.</p>
<p>Credit to <a href="https://apple.stackexchange.com/a/238118/39619">this AskDifferent post</a> where I first found the trick.</p>toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0tag:blogger.com,1999:blog-3676448.post-56837656848344510332019-02-11T18:35:00.001+00:002019-06-18T12:43:52.603+01:00How to customise Jetty embedded in Spark Java framework<p>EDIT: I uploaded a full working example, ready to be customized, <a href="https://github.com/toyg/spark-custom-jetty">on github</a>. If you just want a working solution, go there. Otherwise, keep reading for the explanation.</p>
<p>When it comes to Java microframeworks for API development, I've been using <a href="http://sparkjava.com/">Spark</a> for some time in a couple of different projects. For most casual needs, it works out of the box; however, there are circumstances where the "easy" options for configuring the embedded <a href="https://www.eclipse.org/jetty/">Jetty</a> instance, are simply not enough. In that case, you have to take control and basically replace the instance with one that you completely control.</p>
<p>Due to how Jetty works, there are multiple moving parts: the server takes care of things like thread pools, the socket is responsible for SSL and other low-level protocol stuff, and the handler is where you do high-level middleware (changing headers etc). This means that you might need several factories to take care of all these. The good news is that Spark is granular enough that you can (more or less) limit yourself to what you need. The main pattern is: there will be a factory for each element, with a <code>create()</code> method returning an interface; you implement the interface and swap out the default factory with your own. However, these factories at the moment (Spark 2.8) are somewhat nested, so depending on where you need to work, you might have to replace a bunch of classes before you hit the point you're interested in.</p>
<p>The main entry point is <code><a href="http://static.javadoc.io/com.sparkjava/spark-core/2.8.0/spark/embeddedserver/EmbeddedServers.html#add-java.lang.Object-spark.embeddedserver.EmbeddedServerFactory-">EmbeddedServers.add(identifier, serverFactory)</a></code>. This is what you'll call from the <code>main()</code> method you use for all the <code>post()</code> and <code>get()</code> configuration directives. It is important that EmbeddedServers should appear right at the top, before any other configuration directive, otherwise Spark will use its default factory. It should look more or less like this:</p>
<pre>EmbeddedServers.add(
EmbeddedServers.Identifiers.JETTY,
new MyWonderfulServerFactory());</pre>
<p>For the identifier, we re-use the default one because otherwise Spark will take over again. Nothing to do there.</p>
<p>The server factory is where real work begins. Your class (or lambda) needs to implement the <code><a href="http://static.javadoc.io/com.sparkjava/spark-core/2.8.0/spark/embeddedserver/EmbeddedServer.html">spark.embeddedserver.EmbeddedServerFactory</a></code> interface, which has one method:</p>
<pre>public EmbeddedServer create(
Routes routeMatcher,
StaticFilesConfiguration staticFilesConfiguration,
ExceptionMapper exceptionMapper,
boolean hasMultipleHandler)
</pre>
<p>As a starter, you can copy the content of <code><a href="https://github.com/perwendel/spark/blob/master/src/main/java/spark/embeddedserver/jetty/EmbeddedJettyFactory.java">spark.embeddedserver.jetty.EmbeddedJettyFactory</a></code> as-is. You don't need constructors (unless you want them). Strictly speaking you don't need withThreadPool() and withHttpOnly() methods either, but I suggest you keep them anyway; just change the return type to match MyWonderfulServerFactory.</p>
<p>Create() does three things:
<ol>
<li>Initializing the route matcher, which you probably don't want to touch;</li>
<li>Initializing the Jetty handler for that matcher, which you may want to configure for things like header manipulation and other middleware;</li>
<li>Creating the embedded server, which is likely what you are after.</li>
</ol>
<p>I will assume we want to tweak n.3. The main place where to pay attention is this line at the end of create():</p>
<pre>return (new EmbeddedJettyServer(this.serverFactory, handler)
).withThreadPool(this.threadPool);</pre>
<p>This will return the default configuration, and we don't want that. So we change it to something like:</p>
<pre>return (new MyWonderfulEmbeddedServer(handler)
).withThreadPool(this.threadPool);</pre>
<p>Now we need a MyWonderfulEmbeddedServer class, which should implement <code><a href="http://static.javadoc.io/com.sparkjava/spark-core/2.8.0/spark/embeddedserver/EmbeddedServer.html">spark.embeddedserver.EmbeddedServer</a></code>. Again, as a starting point, you can copy <code><a href="https://github.com/perwendel/spark/blob/master/src/main/java/spark/embeddedserver/jetty/EmbeddedJettyServer.java">spark.embeddedserver.jetty.EmbeddedJettyServer</a></code>, and change the return types to match. I also suggest you get rid of the factory parameter in constructor and the related field, which adds a bit of unnecessary complexity. That factory is actually used only in one place:</p>
<pre>if (this.threadPool == null) {
this.server = this.serverFactory.create(
maxThreads, minThreads, threadIdleTimeoutMillis);
} else {
this.server = this.serverFactory.create(this.threadPool);
}</pre>
<p>Which you can replace with the following (straight from <a href="https://github.com/perwendel/spark/blob/master/src/main/java/spark/embeddedserver/jetty/JettyServer.java">spark.embeddedserver.jetty.JettyServer</a>):</p>
<pre>if (this.threadPool == null) {
if (maxThreads > 0) {
int min = minThreads > 0 ? minThreads : 8;
int idleTimeout = threadIdleTimeoutMillis > 0 ? threadIdleTimeoutMillis : '\uea60';
server = new Server(new QueuedThreadPool(maxThreads, min, idleTimeout));
} else {
server = new Server();
}
} else {
this.server = threadPool != null ? new Server(threadPool) : new Server();
}</pre>
<p>It's a bunch of stuff related to the amount of threads and timeouts. If that's what you were trying to configure, btw, this is where you can do it. To be honest, I believe Spark developers intended that the "proper" way, to do that particular customisation, would be to keep the factory parameter as it is, implement your own alternative to the badly-named JettyServer class -- it should be something like JettyThreadConfigFactory, really -- which implements the similarly badly-named JettyServerFactory, and pass it to the constructor in MyWonderfulServerFactory.create().</p>
<p>In my case, though, I was after an SSL customisation, and for that I had to swap out yet another factory, <a href="http://static.javadoc.io/com.sparkjava/spark-core/2.8.0/spark/embeddedserver/jetty/SocketConnectorFactory.html">spark.embeddedserver.jetty.SocketConnectorFactory</a>, mentioned in the second half of this block:</p>
<pre>ServerConnector connector;
if (sslStores == null) {
connector = SocketConnectorFactory.createSocketConnector(
this.server, host, port);
} else {
connector = SocketConnectorFactory.createSecureSocketConnector(
this.server, host, port, sslStores);
}
</pre>
<p>In this case there is no interface, you can just extend the existing class with what you need. I wanted to enforce a particular set of SSL ciphers and protocols, so I overrode createSecureSocketConnector() adding the following bits:</p>
<pre>sslContextFactory.setExcludeProtocols("SSLv3", "SSLv2", "TLSv1.2");
// first we clear existing exclusions
sslContextFactory.setExcludeCipherSuites(new String[]{});
// then we re-add what we need
sslContextFactory.setIncludeCipherSuites(bigArrayOfCipherNames);
</pre>
<p>And that's it. Now you know how to instantiate your own Jetty instance for Spark. It's a bit convoluted, and hopefully a "spark 3" will give us a better architecture to work with, one day. In the meantime, this is how you can do it.</p>toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0tag:blogger.com,1999:blog-3676448.post-24976998857669111442018-11-26T14:41:00.001+00:002018-11-26T14:41:57.505+00:00How to get Google Home / Google Assistant to call your iOS iPhoneIf you have an iPhone, Google Assistant and Google Home won't be able to call or ring your phone out of the box, since it's an Android-only feature. If you are not in the US, you cannot even use the workaround of calling your number. So what can you do, if you tend to misplace your phone around the house ? (ahem)<br />
<ol>
<li>Get an account on <a href="https://ifttt.com/">IFTTT</a></li>
<li>Install the <a href="https://itunes.apple.com/us/app/ifttt/id660944635">IFTTT app</a> on your phone</li>
<li>create a new applet, click on "<i>this</i>" and select <i><b>Google Assistant</b></i></li>
<li>select "<i>say a simple phrase</i>" and enter the details you prefer (e.g. "make a noise on my phone" or "ping my phone". Note the most common "ring my phone" or "where is my phone" are reserved by Google and won't work).</li>
<li>click on "<i>then</i>" and select <i><b>VoIP Calls</b></i></li>
<li>enter a message the phone will tell you if you pick up (e.g. "Glad you found me!")</li>
<li>Save the applet.</li>
</ol>
Now, when you say that phrase to Google Home / Goole Assistant, it will ring your phone, so you can dig it out of the sofa or the Lego box (ahem).toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0tag:blogger.com,1999:blog-3676448.post-15047162169210090212018-08-15T16:52:00.000+01:002018-10-04T13:42:41.784+01:00how to load initial data and test data in Django 2+<p>There are two ways to automatically load data in Django:
<ul>
<li>for data you need while running tests, place xml/json/yaml files in <code>yourapp/fixtures</code>.</li>
<li>for data you need while setting up the database from scratch, or at specific points in time, you must create a Migration</li>
</ul>
</p>
<p>This is a bit annoying, because chances are these locations will get out of sync sooner or later, and it duplicates effort if you do reproducible builds, docker, and stuff like that.</p>
<p>The solution is to create a migration that actually loads fixtures. So:
<ol><li>Create your fixtures: <code>manage.py dumpdata --output yourapp/fixtures/yourmodel.json yourapp.YourModel</code></li>
<li>Create an empty Migration: <code>manage.py makemigrations --empty yourapp</code><li>
<li>Edit the resulting migration (the last file created under <code>yourapp/migrations</code>, making it look like this:
<pre>
from django.db import migrations
def load_fixtures(apps, schema_editor):
# This is what will be executed by the migration
from django.core.management import call_command
# this is the equivalent of running manage.py loaddata yourmodel.json
for fixture_name in ['yourmodel']: # add any additional model here
call_command("loaddata", fixture_name)
# add other calls if you have multiple models
def rollback(apps, schema_editor):
# This will be executed if you rollback the migration, so you want to clean up
for model_name in ["YourModel"]: # add any additional model here
model = apps.get_model("yourapp", model_name)
model.objects.all().delete()
class Migration(migrations.Migration):
dependencies = [
# ... don't touch anything here ...
]
operations = [
migrations.RunPython(load_fixtures, rollback),
]
# -*- coding: utf-8 -*-
</pre>
</li>
<li>Profit</li>
</ol>
<p>Note that this does not remove the option to have data that is available only in certain situation: just don't list the fixtures you don't want in the migration, and vice-versa.</p>toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0tag:blogger.com,1999:blog-3676448.post-77242606643067142562018-05-24T22:04:00.001+01:002018-05-24T22:04:11.225+01:00How to securely wipe an NVMe driveNVMe drives are great: they are fast and they are huge. That huge size, however, can be a pain when it comes to securely erasing data. Old-school commands like <code>wipe</code> are simply not up to the task; and even if they were, they work on assumptions that do not map properly to a solid-state world. Writing random data over and over is going to dramatically reduce the lifespan of a solid-state drive, and it's pointless when all NVMe disks already have built-in tools that can take care of this task quickly and safely.<br />
<br />
So what do you do when you want to wipe a NVMe drive? <br />
<ol><li>
Download a recent Linux distribution. I would recommend Debian/Ubuntu or one of their smaller derivatives (like Knoppix). Burn it on a cdrom or USB drive and boot the system from it.</li><li>Make sure your package manager is up-to-date (under Debian/Ubuntu, <code>sudo apt-get update</code>), then install <code>nvme-cli</code> (<code>sudo apt-get install nvme-cli</code>)</li><li>If your drive is a Samsung, it now has to be put to sleep (you can do that with <code>sudo systemctl suspend</code>) and then woken up. This is a weird bug that <a href="https://github.com/linux-nvme/nvme-cli/issues/84">Samsung doesn't seem in any hurry to fix</a>.</li><li>Now you can securely wipe the disk: <code>sudo nvme format -s1 /dev/nvme0n1</code></li></ol>
For the curious: the -s option triggers <b>Secure Erase</b> mode, which can be set to 1 (wipe) or 2 (delete encryption keys for encrypted data). 1 looks like the safest option, because it will automatically do what 2 does if it detects that all data is encrypted. Reference <a href="https://www.mankier.com/1/nvme-format">here</a>.<br />
<br />
The latest NVMe specification adds other commands, to scrub every nook and cranny (bus caches etc), but as far as I know they have not been implemented yet. toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com5tag:blogger.com,1999:blog-3676448.post-66040240774280821302018-05-15T22:59:00.000+01:002018-05-15T22:59:15.639+01:00New BeginningsSo, after 7 intense years at Infratects (now Inlumi), it's time for me to move on.<br />
<br />
I have a few ideas about what to do next, but nothing set in stone yet. <a href="https://www.linkedin.com/in/glacava/">My LinkedIn profile</a> could do with more details, but it's a decent primer for what I do for a living - Hyperion/EPM, Python, Java, Weblogic and thereabout.<br />
<br />
I was a web developer in a previous life, so I enjoy hacking and automating everything, getting dirty with infrastructure and the cloud; and in 13 years working on Hyperion products, I've absorbed a pretty good amount of knowledge related to financial processes (consolidation rules, metadata, cube performance and so on) as well as a deep understanding of the innards of the EPM suite. I can tweak your database, hack your Weblogic, integrate your cloud-based authentication, script your exports and migrations, and so on; and if there is something the tools won't do... I'll build you a new tool! That's where I make the difference: at the intersection of technology and Finance, boosting the productivity of accountants. <br />
<br />
If this sounds interesting, ping me on <a href="https://www.linkedin.com/in/glacava/">LinkedIn</a> and we can have a chat.toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0tag:blogger.com,1999:blog-3676448.post-69954369111134613142018-04-16T20:30:00.000+01:002018-04-16T20:30:23.071+01:00Nespresso Blends - a comprehensive spreadsheetAs a faithful <a href="https://www.nespresso.com/" target="_blank">Nespresso</a> user, I was a bit shocked last month when I discovered one of my favourite blends contained Robusta. I had always assumed all blends were 100% Arabica and alas, that was not the case. So I started looking up what is what, but I was quickly overwhelmed - I wasn't going to browse through 24 pages of slow-loading images to read all blurbs. Enter Python.<br />
<br />
As it often happens, <a href="https://xkcd.com/974/" target="_blank">the situation degenerated</a> quickly. The result is <a alt="Nespresso Blend Data Spreadsheet" href="http://static.pythonaro.com/coffee0.xlsx">an Excel spreadsheet </a>(also available <a href="https://docs.google.com/spreadsheets/d/e/2PACX-1vQHAG1wLaUR2d1oLt35SCu__pSFStX0MfppWD-ZLr2R-JYgbqOeYgv7EOlsoPC_V_A5AIJG67OrBImC/pubhtml" target="_blank">in Google Docs</a>), listing all attributes of all blends so that you can filter out what you need. Decaffeinated varieties are highlighted, because that's not real coffee 😁toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0tag:blogger.com,1999:blog-3676448.post-33434808886079067202018-03-28T15:37:00.000+01:002018-05-24T07:41:39.970+01:00How to implement custom BotStorage class for Microsoft BotFrameworkSince launch, the MS BotFramework has been changing very rapidly. So rapidly, in fact, that I recently gave up trying to keep up with my handrolled Python, and embraced (sigh) their NodeJS SDK. At least now I won't have to worry when they break stuff, right ?<br />
One of the most recent BF changes is the deprecation and eventual removal of the default persistence API. You are now supposed to either use one of the pre-built Azure services (paying $$), or provide your own implementation. In pure Microsoft style, they state that rolling your own is very easy... but never actually provide a sample or even tell you which interface should be implemented.<br />
I had to scavenge through their code to figure it out (at least they opensource their stuff these days), but I thought I'd save others a bit of aggravation, and here it is:<br />
<pre>var MyBotStorage = (function () {
// optional constructor
function MyBotStorage(options) {
this.options = options;
}
MyBotStorage.prototype.getData = function (iBotStorageContext,
callback){
// IBotStorageData interface
var data = {
userData: {},
conversationData: {},
privateConversationData: {}
};
// the callback MUST be invoked
// signature: callback(Error, iBotStorageData )
callback(null, data);
}
MyBotStorage.prototype.saveData = function (iBotStorageContext,
iBotStorageData,
errorCallback){
// the callback MUST be invoked
errorCallback(null);
}
return MyBotStorage;
}());
var botStorage = new MyBotStorage({});
var bot = new builder.UniversalBot(connector, function (session) {
// your bot code here
}).set('storage', botStorage);
</pre>
As you can see, it is indeed trivial, once you know how. It's sad that MS somehow, in the haste of deprecating their older interfaces, couldn't find the time to put this sample in their otherwise-extensive documentation. I suspect the fact that Azure is not mentioned anywhere might have something to do with it, but <i>I'm sure</i> I'm just assuming excessive malice and there is a perfectly-plausible explanation that does not involve greed. Or is there?toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0tag:blogger.com,1999:blog-3676448.post-13408764312739118652018-03-26T08:00:00.000+01:002018-03-26T08:00:12.975+01:00Appstores need a Trial Mode<p>I've been looking for a while for a Windows-native Twitter client that could sync with my other non-Windows devices (<abbr title="also known as">aka</abbr> "supporting TweetMarker"). I've found one that claims to do that, <a href="https://www.microsoft.com/store/productId/9NBLGGH62LKC">Tweet It!</a>, but there is a problem: it costs £3.5 upfront. That's a relatively high amount of money to throw down a well hoping that my wish will come true. However, because this feature is so rare, <i>if it worked</i> I'd be happy to pay three times as much, no questions asked. I just have no way to find out.</p>
<p>Appstores need a simple Trial Mode. All the current hacks (In-App-Purchases, subscriptions, refunds etc) are just that, clumsy workarounds to this glaring omission, which is why app prices are so squeezed down - to the chagrin of indie developers. Trial Mode would also bring huge collateral benefits like reducing reliance on SaaS services (something that Microsoft in particular should relish) and creating viable alternatives to the free-to-play bubble that has turned online gaming into a socially-accepted form of gambling addiction.</p>
<p>I doubt Apple will ever introduce Trial Mode - they are the dominant player and have little incentive to change the rules; and Android is a far west where the Play Store is almost irrelevant. But Microsoft should experiment with something like this, while the Windows Store is still young and evolving. Both developers and users would love it, they have been asking for something like this for years.</p>toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0tag:blogger.com,1999:blog-3676448.post-39841424453094465572018-03-04T04:22:00.001+00:002018-05-24T07:43:09.734+01:00The European Union, explained to geeksI've started describing the EU to geek-friends as a set of services built over decades.<br />
<ul>
<li>You need to coordinate energy supplies across the company departments (nations)? We built a service (<a href="https://en.wikipedia.org/wiki/European_Coal_and_Steel_Community">ECSC</a>) that does that. Worked great, no more fighting over the last bit of coal!</li>
<li>Need atomic development? <a href="https://en.wikipedia.org/wiki/European_Atomic_Energy_Community">EURATOM</a> service. Worked great, again.</li>
<li>And so on and so forth...</li>
<li>At some point somebody said hey, we need a management tool for all this stuff! "<a href="https://en.wikipedia.org/wiki/European_Council">European Council</a>" service, with a set of dedicated subthreads for the real work (<a href="https://en.wikipedia.org/wiki/European_Commission">European Commission</a>).</li>
<li>But that service will run amok at times, let's add some monitoring and security checks! <a href="https://en.wikipedia.org/wiki/European_Parliament">EuroParliament</a> service - took a few rewrites to get right, nobody really likes to work on monitoring tools; but like systemd on Linux, the EP service should eventually take over anything that talks to real world I/O, so it's pretty important.</li>
<li>Now all this stuff needs to communicate, with common formats that avoid parsing and reparsing umpteen different types of data back and forth, and ways to look up the right service for a given job - so we created a "CommonMarket SDK", optionally turbocharged with options like <a href="https://en.wikipedia.org/wiki/Schengen_Area">Schengen</a>. When exceptions are thrown, the SDK will automatically invoke the <a href="https://en.wikipedia.org/wiki/European_Court_of_Justice">ECJ</a> service to resolve matters; and it will self-update by talking to the management services. Once everyone adopts the SDK, then it should be easier to make more radical changes through that (ECB, Euro, common fiscal policy...). But in the end nobody likes change, it's always hard to break backward-compatibility.</li>
</ul>
Now, across the company/continent, various departments/nations have adopted some or all of these services, but most of them ended up relying on the SDK one way or the other, so it became basically mandatory. At one point we had to give a name to the whole framework, and "EU" it was.<br />
It's definitely not a monolith, but there are so many moving parts that the management services are now essential. Some departments have renounced their Write access (Iceland, Norway, Switzerland...) and some were never granted that privilege (Turkey); some departments were forced to change their processes to suit the framework (Italy, Greece, the Eastern countries...). Things are still pretty shaky, developers are still very much at work, but it's getting better with time. It's making more and more services possible and even *easy* to bootstrap (EMA, EFSA, Erasmus...). Bugs creep in and out, we keep adding more and more fault-tolerance, the workload is not yet distributed fairly, etc etc; but it's accomplishing some very heavy tasks that are absolutely mission-critical, if we want to keep the company running and competing with the big boys.<br />
...<br />
Then, one day, a department said they'd like to go back to pen and paper. Except for a few services, for which they want to hand-craft packets individually, but those services should just assume the data is still as good as before, and never throw exceptions - they will ignore any response from the ECJ service anyway. Some of what their department does depends on another department, which has no intention to go back to pen and paper, but they say they will somehow give them bits and receive paper, without anyone actually doing the transformation, and without any friction at all. And they'd like to retain write access to the management services too, thank you very much, plus veto powers, so nobody can change SDK formats without consulting them.<br />
Cue facepalming.toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0tag:blogger.com,1999:blog-3676448.post-68060657747322423382018-01-25T19:30:00.000+00:002018-05-24T07:42:01.758+01:00Windows Survival Kit for OSX exilesAfter 5 long years, I was gently forced back to Windows for everyday work. The experience has been less terrible than I thought, but was pretty frustrating at the beginning.<br />
To help anyone in the same predicament, I put together a small list of tips to make the move a bit more tolerable.<br />
<ul>
<li>Windows does not have hot corners. Hello, 1995! Anyway, the solution here is using a small app called, unsurprisingly, <a href="https://sites.google.com/site/bytecar/home/hotcornersapp" target="_blank">HotCornersApp</a>. It works well enough, and it's fine with multiple screens too. Despite the basic website, it's legit and even opensource, you can <a href="https://github.com/bytecar/HotCornersApp" target="_blank">compile it yourself</a> (although beware - apparently the very latest updates may not build properly).</li>
<li>Windows does not do <a href="http://osxdaily.com/2010/07/23/set-up-text-substitution-in-mac-os-x/" target="_blank">text substitution</a>, which is one of those things that you don't know how much you love it until they take it away from you. As far as I can see, there is no free utility on Windows to do this, or nothing that actually works well enough. So, I paid for <a href="http://www.16software.com/breevy/" target="_blank">Breevy</a>. It feels a bit retro (it looks blurry on high-def screens...), but it works very well and has all sorts of options and special features.</li>
<li>Windows doesn't really deal well with multi-language support, aka <i>typing accents from a US keyboard</i>. Sure, you can use US-International, and deal with <code>'</code> and <code>"</code> becoming meta-keys, but for a techie/programmer typing those characters on their own more often than accents, it's extremely annoying. The solution was again Breevy: you can define a combo that will not be triggered unless you type a special character afterwards. For example, I defined <code>`e</code> to become <code>è</code> after I press Ctrl. It works absolutely everywhere, although it's nowhere as elegant as the OSX popup. Same story for special characters like £ , € etc.</li>
<ul>
<li>For best results, check if your keyboard supports custom macro. <a href="https://gaming.kinesis-ergo.com/fsedge/" target="_blank">Mine does</a>, so I mapped the blank side-keys to accents and so on. After muscle-memory starts kicking in, this solution is actually superior to stock OSX.</li>
</ul>
<li>I was not going to reformat my external hard drives to NTFS, which is a pain to use back on OSX; so again I had to pay for <a href="https://www.paragon-software.com/home/hfs-windows/" target="_blank">Paragon HFS+ for Windows</a>. The UI is garbage (and does not work properly with multi-monitor setups), but the actual driver seems to work perfectly.</li>
<li>Microsoft has basic print-to-PDF support. If you need to concatenate documents, <a href="http://www.primopdf.com/" target="_blank">PrimoPDF</a> does it, and it's free (do not download the Nitro version). The interface is not great though.</li>
<li>For the developer types out there who rely on <a href="https://kapeli.com/dash" target="_blank">Dash</a>, a good equivalent on Windows is <a href="https://zealdocs.org/" target="_blank">Zeal</a>. It supports the same format, even fetching docsets directly from Dash repositories.</li>
<li>Also for developer/sysadmin types, the Windows equivalent of homebrew is now <a href="https://chocolatey.org/" target="_blank">Chocolatey</a>. Whoever came up with that word should give up trying to name things, but the software does work. You can use it to install 7zip (to get the latest beta with proper security patches, <code>choco install 7zip.install --pre -y</code>) windirstat and so on, it will make it easy to upgrade them when necessary (rather than having through the usual website-download-install dance, just <code>choco upgrade</code> them all).</li>
<li>There are quite a few apps that do what <a href="http://fluidapp.com/" target="_blank">Fluid</a> does, i.e. making websites into "native" apps. I know, I know, they are aberrations; but I got used to having a few sites (Gmail, Trello, Hangouts...) accessible this way. After a bad experience with WebCatalog (it was working ok, but then it got stuck trying to upgrade itself), I installed <a href="https://github.com/jiahaog/nativefier" target="_blank">nativefier</a>. This is again more for the geek types; it requires <a href="https://www.npmjs.com/" target="_blank">npm</a> and it's a command-line app with a few quirks.</li>
<li>UPDATE 1: to get back the "preview on pressing Space" experience, there is a free app called <a href="https://sourceforge.net/projects/ccseer/">Seer</a>. You can also pay for a license, but it's not clear what the difference might be, the free app is more than enough for my needs. I had to remove the Ctrl-Alt-S shortcut in Settings in order to make it work properly.</li>
</ul>
Did I miss anything? Feel free to suggest other useful tidbits in comments.toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0tag:blogger.com,1999:blog-3676448.post-61013088003424532962017-11-17T00:55:00.000+00:002017-11-17T00:56:59.172+00:00How to root Nexus 7 (2012) and flash it to LineageOS, from Mac OSXI've finally got fed up enough with the old Nexus 7 (2012) that we originally bought to let the kids play - it's never really worked properly, getting laggy every 5 minutes and running out of battery after an hour. The kids moved on to an iPad, so I "installed" this crappy tablet to the kitchen wall, hoping to use it for calendaring and podcasts, but it was still horrendously laggy. I've decided to give a chance to LineageOS (previously CyanogenMod) to see if it helps. I'd be interested to know if anybody else uses a tablet wall-mounted in their kitchen, and what apps they installed - at the moment I have Paprika, the BBC players and weather, and PodcastAddict.<br />
<br />
Here are the full steps required for flashing - I write them here because some of these are at risk to disappear from the internet for good after the demise of CyanogenMod wiki.<br />
<br />
Note: <b>THIS WILL WIPE YOUR DEVICE, YOU WILL LOSE EVERYTHING THAT IS ON IT</b> so first make backups etc etc.<br />
<br />
First you need to install the Android tools. I recommend doing this with <a href="https://brew.sh/" target="_blank">homebrew</a>, install it if you don't have it yet (it's extremely useful in many many cases).<br />
Then open a terminal on your mac and type:<br />
<ul>
<li>brew tap caskroom/cask </li>
<li>brew tap caskroom/versions</li>
<li>brew cask install java8</li>
<li>brew cask install android-sdk</li>
</ul>
Enable Developer Mode on the tablet by tapping 7 times on the build number under <i>Settings -> About tablet.</i><br />
Back to Settings, tap on <i>Developer Options</i> and enable USB Debug mode and Stay Awake.<br />
Go back to your mac terminal and type:<br />
<ul>
<li>sudo adb start-server</li>
</ul>
This starts the android debugger server. <br />
Connect the tablet with a real USB cable (beware charging-only cables! Those won't work.)<br />
On the tablet screen, you should be prompted to authorize the device; do it.<br />
Back to terminal:<br />
<ul>
<li>adb reboot bootloader</li>
</ul>
once the device comes back in bootloader mode:<br />
<ul>
<li>fastboot oem unlock</li>
</ul>
Accept the disclaimer on the tablet by clicking the power button when <i>Yes</i> is highlighted (you can move with the volume buttons).<br />
If it doesn't reboot on its own, select <i>Start</i> (again with the power button). At this point you should see an unlocked pad at the bottom of the screen as Android loads.<br />
Note: if you plan to flash another OS, there is no point in going through the whole setup at this point, skip as much as you can. <br />
Re-enable Developer mode and USB Debugging.<br />
<br />
<b>At this point the machine is rooted. </b>The following steps are necessary only if you want to install LineageOS or other hacks; at the very minimum, though, you should install Trimmer (fstrim) from the Play store and use it liberally. Anyway, on with the flashing... <br />
<br />
Get the latest image for "grouper" from <a href="https://twrp.me/">https://twrp.me/</a>, and unzip it.<br />
Back to the terminal:<br />
<ul>
<li>cd folder/where/you/have/your-downloaded-image.img</li>
<li>fastboot flash recovery your-downloaded-image.img</li>
<li>adb reboot bootloader</li>
</ul>
At this point you have a custom recovery image with a bunch of nifty features that make it very easy to hack the device. Tap Wipe and select system, cache, and dalvik.<br />
Get the image you want to install. I'm currently trying <a href="https://forum.xda-developers.com/nexus-7/development/rom-lineageos-14-1-nexus-7-2012-t3530261" target="_blank">this</a> but in general anything after CM10 / Android 4.1 may be on the slow side for the original Nexus. You probably want the Google Apps from opengapps.org - choose ARM / 7.1 and the Micro option. Save both the apps zip and the image zip in the same folder.<br />
Back to Terminal, let's copy these files to the device:<br />
<ul>
<li>cd folder/where/you/have/your/zips</li>
<li>adb push your-downloaded-image.zip /</li>
<li>adb push your-downloaded-gapps.zip /</li>
</ul>
On the tablet, in the recovery screen, select Install, tap on Install ZIP, then select the image file and install. Repeat for the gapps file. If gapps refuse to go in because of space constraints, download <a href="https://www.androidfilehost.com/?fid=745425885120692225" target="_blank">this file</a>, rename it <i>gapps-config.txt</i>, then copy it to the device in the same location as the gapps zip:<br />
<ul>
<li>adb push gapps-config.txt /</li>
</ul>
and try again to install the gapps image. <ul>
</ul>
Reboot the tablet. You might have to re-enable developer mode and usb debugging.<br />
Install the <a href="https://play.google.com/store/apps/details?id=com.fifthelement.trimmer" target="_blank">Trimmer (fstrim)</a> app, which is absolutely necessary. Launch it and click Trim Now. Click on the settings icon and enable autotrim as frequently as possible. In fact, every time you install an app or write a bunch of files, before you launch the new app, open Trimmer and do a trim, or it will all get laggy again.<br />
Another good app is <a href="https://play.google.com/store/apps/details?id=com.icecoldapps.sshserver&hl=en" target="_blank">Android SSH Server</a>, although it's a bit old it still works fine and it's easy to configure. You just have to use the options <i>-t -oHostKeyAlgorithms=+ssh-dss -p XXXXX</i> (where XXXXX is the custom port configured in the app, Android won't allow the classic one) when connecting.<br />
<i></i><br />
<br />
And that's it.toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0tag:blogger.com,1999:blog-3676448.post-21858133994336207272017-10-10T14:14:00.001+01:002017-10-10T14:18:31.850+01:00Fighting Cliqz in Firefox<p>Mozilla recently <a href="https://news.ycombinator.com/item?id=15421708" target="_blank">announced</a> that some of their German users downloading Firefox will receive a version that tracks some of their web activity, reporting it to Cliqz.com. In the wake of this development, which is pretty awful from a privacy perspective, I went spelunking into my version of FF to see if anything had been enabled already in my build.</p>
<p>To my horror, cliqz.com is already mentioned in a few places, despite me never installing anything related to it. If you enter about:config in the address bar and search for cliqz, a few entries will pop up:<br />
<a href="https://1.bp.blogspot.com/-lHLryF014dQ/Wdy9TeC2o9I/AAAAAAAABMM/iXARezgj6U46vc7VhUCssWXUk1ioRrUagCLcBGAs/s1600/Screen%2BShot%2B2017-10-10%2Bat%2B13.29.39.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="203" data-original-width="1600" height="80" src="https://1.bp.blogspot.com/-lHLryF014dQ/Wdy9TeC2o9I/AAAAAAAABMM/iXARezgj6U46vc7VhUCssWXUk1ioRrUagCLcBGAs/s640/Screen%2BShot%2B2017-10-10%2Bat%2B13.29.39.png" width="640" /></a><br />
Straight out of the gate, it looks like something from cliqz.com has been whitelisted. What is it?</p>
<p>The <code>social.*</code> preferences are related to <a href="https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Social_API">SocialAPI</a> Services, a sort of framework to integrate social networks into Firefox. It was introduced several years ago but very few people actually use it or know about it. If you look up the related preferences, a few more entries are present:<br />
<a href="https://1.bp.blogspot.com/-obf_a6fnr3I/Wdy-DP7HvxI/AAAAAAAABMU/dxt5Wp7ekF4dpIYGQMrrMxWxNMUXtA0WQCLcBGAs/s1600/Screen%2BShot%2B2017-10-10%2Bat%2B13.32.15.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="295" data-original-width="1600" height="118" src="https://1.bp.blogspot.com/-obf_a6fnr3I/Wdy-DP7HvxI/AAAAAAAABMU/dxt5Wp7ekF4dpIYGQMrrMxWxNMUXtA0WQCLcBGAs/s640/Screen%2BShot%2B2017-10-10%2Bat%2B13.32.15.png" width="640" /></a><br />
If you are a fan of this sort of thing, you can go to that address <a href="https://activations.cdn.mozilla.net/">https://activations.cdn.mozilla.net</a> and install some of the available providers. I have no idea whether they still work or not; among others, delicious.com has recently been sold and it's in read-only mode, so that's unlikely to be useful.</p>
<p>I personally disabled it all (that <code>social.remote-install.enabled</code> freaked me out) by double-clicking all boolean properties (turning them to false) and double-clicking then clearing out social.whitelist.</p>
<p>The other mentions of cliqz seem to be special-casing whitelists aimed at making an extension work around some of the recent changes in the Firefox extensibility framework. If I understand correctly, <code>dom.ipc.cpows.allow-cpows-in-compat-addons</code> allows Cross-Process Object Wrappers (an internal communication mechanism that should slowly be removed) to be used even if the extension is marked as compatible with the new multiprocess architecture; and <code>extensions.legacy.exceptions</code> exempts the listed extensions from being marked as Legacy.</p>
<p>In those whitelists, there is one called testpilot@cliqz.com. <a href="https://testpilot.firefox.com/">Test Pilot</a> is an official Firefox add-on from Mozilla that will periodically publish some proposed additions to the browser, allowing users to enable them, test them out, give feedback and so on. I personally like it, some of the proposed features are actually pretty good (although it doesn't look like any of them ever made it to the main build); considering its complexity (an extension-installing extension) it makes sense that it might require some special privileges, and after all it's an official Mozilla feature so why not?</p>
<p>Well, it looks like Mozilla is using cliqz.com to gather remote info about TestPilot usage, which is very disappointing. TestPilot was marketed as an official Mozilla project, there was no mention of third parties involved. I won't disable TestPilot, but I am again very disappointed in their cavalier attitude with my usage data.</p>
<p>To conclude, it looks like the "synergy" between Mozilla and Cliqz was already underway before the latest announcement. It's likely that more defensive hacks will be required in the near future to keep user-tracking at bay in Firefox. As a long-time fan of Mozilla since the '90s, this development is disappointing.</p>toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0tag:blogger.com,1999:blog-3676448.post-61976962163231354532017-07-31T20:00:00.000+01:002017-10-23T03:49:54.809+01:00How to build waifu2x command-line version on osx<div><b>UPDATE 23/10/2017:</b> recent changes seem to be incompatible with clang. I have updated the steps below to check out the last commit that is known as working. <br/></div>
<div>
<a href="http://waifu2x.udp.jp/index.html" target="_blank">Waifu2x</a> is a popular image converter backed by neural-network models, typically used to upscale images from <i>anime</i>. The various web versions are easy to use but typically don't allow for batch-processing, because it's a relatively intensive operation. However, there is a command-line version that is fairly easy to compile on Windows and Linux. OSX support was notably absent... until now.</div>
<div>
These are the steps required to get it working on Mac:
<br />
<ol>
<li><code>brew tap science && brew install opencv3</code><br />(if you don't have <a href="https://brew.sh/" target="_blank">Homebrew</a>, go install it now - it's wonderful)</li>
<li><code>git clone https://github.com/DeadSix27/waifu2x-converter-cpp.git</code></li>
<li><code>cd waifu2x-converter-cpp</code></li>
<li><code>git checkout d69313040b0784662465fb1d2eca81a2b1ebccb2</code></li>
<li><code>cmake -DOVERRIDE_OPENCV=1 -DOPENCV_PREFIX=/usr/local/Cellar/opencv3/<your version here> .</code> (remember the dot at the end!)</li>
<li><code>make -j4</code></li>
</ol>
At this point, you have the executable <code>waifu2x-converter-cpp</code> in the directory; you can <code>make install</code> if you really want to (I prefer to keep custom stuff in my home dir). To test it's working, the following command should return info on your system:<br />
<br />
<code>./waifu2x-converter-cpp --list-processor</code>
</div>
<br />
If you need some images to test, <a href="https://imgur.com/gallery/dSpHk" target="_blank">here</a> you can get quite a few stills from the gorgeous 5 Centimeters Per Second. Note: you might have to rename the folder "models_rgb" into "models" before converting.<br />
<br />
<h3>
BONUS: Qt GUI</h3>
<div>
If you hate the command-line, there is <a href="https://github.com/toyg/waifu2x-converter-qt" target="_blank">a simple QT wrapper</a>. To compile it you will need Qt-Creator and Qt installed and configured. This used to be very complicated, but both items have now been packaged for Homebrew so it's all awesomely simple (well, compared to what it was, at least). Note that you still have to do the steps above - the wrapper needs the waifu2x executable to be already compiled.</div>
<br />
<h4>
Install and configure Qt and Qt-Creator</h4>
<ol>
<li><code>brew install qt && brew cask install qt-creator</code></li>
<li>Launch <i>Qt Creator </i>from Launchpad, then go to <i>Qt Creator -> Preferences</i> (Command + ,)</li>
<li>Select <i>Build & Run</i>, then the <i>Qt Versions</i> tab</li>
<li>Click on <i>Add...</i> and select “Macintosh HD”</li>
<li>Press Command + Shift + . to show hidden directories</li>
<li>select <code>/usr/local/Cellar/qt5/<your version>/bin/qmake</code> then OK and OK again to close the Preferences window</li>
<li>Reopen <i>Preferences -> Build & Run</i>, select the Kits tab and click Add</li>
<li>Set the following options:
<ul>
<li><i>Name</i> to something like "Qt CLang Desktop" </li>
<li>Both <i>Compiler</i> options to Clang X86 64bit</li>
<li><i>Qt Version</i> to the version you just created</li>
<li>Click on <i>Make Default</i> then OK to close Preferences.</li>
</ul>
</li>
</ol>
<h4>
Compile waifu2x-converter-qt</h4>
<ol>
<li><code>git clone https://github.com/toyg/waifu2x-converter-qt.git</code> (<a href="https://github.com/khws4v1/waifu2x-converter-qt">the original repo</a> looks abandoned, so I forked it and tweaked it a bit)</li>
<li><code>cd waifu-converter-qt && open waifu2x-converter-qt.pro</code> (this should open Qt Creator; if it doesn't, launch it manually and <i>File -> Open Project</i> -> select the .pro file).</li>
<li>Select <i>Build -> Run</i> (or Command + R). The resulting .app bundle should now be in a folder called <code>build-waifu2x-converter-qt-<something something></code>, just beside your main project folder.</li>
</ol>
<div>
When you first launch the GUI, you should probably go to Preferences and set the path to your waifu2x-converter-cpp executable.</div>
toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com4tag:blogger.com,1999:blog-3676448.post-41825618944941670152017-07-31T10:24:00.001+01:002017-07-31T20:03:56.460+01:00Windows "vintage" default desktop coloursNote to self: these are the default desktop background colours for old Windows versions.<br />
<ul>
<li><span style="background-color: teal;"> </span> <b>008080</b> (RGB 000-128-128) - Windows 95/98</li>
<li><span style="background-color: #3a6ea5;"> </span> <b>3A6EA5</b> (RGB 058-110-165) - Windows 2000</li>
</ul>
(I know you love them, even if setting a wallpaper was the first thing you did after logging on a new PC. It's called nostalgia.)toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0tag:blogger.com,1999:blog-3676448.post-88187299181143922272017-05-18T10:01:00.000+01:002017-05-18T10:01:00.962+01:00How to get Ctrl-arrow shortcuts working in RDP sessions on macOS / OSX<p>It looks like recent versions of the official <a href="https://itunes.apple.com/us/app/microsoft-remote-desktop/id715768417">Microsoft Remote Desktop client</a> are a bit shy when it comes to sending CTRL to the remote Windows session. I discovered this because I use Ctrl-Arrow very often to move from word to word, and it just stopped working in RDP.</p>
<p>The solution is to make sure you don't have any overlapping OSX shortcut - typically, CTRL is used for Mission Control actions. Just disable them from <i>System Preferences -> Keyboard -> Shortcuts -> Mission Control</i> and you should be good to go. Any recent Apple keyboard will have real keys dedicated to those actions anyway.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://2.bp.blogspot.com/-00MQX9CK1BQ/WR1iq5tM0jI/AAAAAAAABLc/Tn8w-c1YzsIPH177olrG53u-RkCICe0LACLcB/s1600/Screen%2BShot%2B2017-05-18%2Bat%2B09.51.48.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://2.bp.blogspot.com/-00MQX9CK1BQ/WR1iq5tM0jI/AAAAAAAABLc/Tn8w-c1YzsIPH177olrG53u-RkCICe0LACLcB/s320/Screen%2BShot%2B2017-05-18%2Bat%2B09.51.48.png" width="320" height="212" /></a></div>
<p>(Come on Microsoft, show some guts - have an option to bypass Mac shortcuts entirely!)</p>toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com1tag:blogger.com,1999:blog-3676448.post-27523948287259255522017-05-03T15:42:00.000+01:002017-05-03T15:42:13.634+01:00Ambushwhack, a Warhammer Quest helper<p>Last week someone on Hacker News linked a site dedicated to HeroQuest. This quickly resulted in me buying <i><a href="http://amzn.to/2qrMdHG">Warhammer Quest: Shadows over Hammerhal</a></i> to play with my kids. I've not dabbled in fantasy miniatures for 20 years, but <i>Hammerhal</i> looked like a self-contained dungeon crawler in the same style as HQ, and indeed that's what it is: a gamemaster arranges a dungeon for a party of 2-5 adventurers, who then proceed to clear 8 levels full of bad guys.</p>
<p>The box includes 31 miniatures; one <i>will</i> need <a href="http://amzn.to/2pwBtGS"> some decent glue</a>, a sharp boxcutter, and a few hours to assemble them... but it's worth it. The kids loved it so much, we're now playing it every day before bedtime - it's a great incentive to get them to do homework, clean up their rooms, and brush their teeth quickly and without fuss.</p>
<p>The game requiring a gamemaster is good in the sense that I can tweak the game and keep the fun flowing. My son is 5 and doesn't really enjoy complex rules, so we started with a simplified set: one move+attack turn for each character, all characters moving 6 spaces, no hero dice or destiny dice, rolling only to hit, and rolling for unexpected events only when they enter a room. I used the Gryph-hound character as an NPC to show them these mechanics, and I'm slowly replacing introducing a few real rules every couple of sessions. So it's all great, right? Well, kinda.</p>
<p>Being a classic dungeon master inevitably involves some dice-rolling. <i>Hammerhal</i> is not too bad in that regard, leaving most of the rolling to adventurers, but one notable exception is ambushes. *Every* turn an adversary is not on the board, the gamemaster is supposed to roll for ambush; if the ambush happens (1 in 6 chances), he has to then roll further for character quantity and behaviour, which means between 2 and 7 throws depending on level and results. That can significantly slow down proceedings, and I found myself "forgetting" to roll quite a few times to keep things going.</p>
<p>So I did what any geek would do: I automated it away. The result was <a href="http://static.pythonaro.com/ambushwhack/" title="Ambushwhack, a Warhammer Quest helper">Ambushwhack</a>, a simple and mostly self-contained webpage that does all the rolling for you. I made buttons big enough to be usable on phones and tablets, added a few pics of my unpainted miniatures (since Games Workshop are infamous for aggressively defending their IP, they would likely come after me if I used their own pics), and now I use it every day.</p>
<p>I posted it to a few Warhammer Quest forums (boardgamegeek etc) and people asked to support <a href="http://amzn.to/2qyyL20">Silver Tower</a>, the other version of Warhammer Quest currently available, which has a similar (but different) set of ambush rules. A kind soul sent me most of the necessary details, so <a href="http://static.pythonaro.com/ambushwhack/silvertower.html" title="Ambushwhack for Silver Tower">here it is</a>. It lacks pictures and action titles but it's usable, at least until I can justify splurging on another big box of toys :)</p>
<p>I hope someone else will find it useful and maybe volunteer to redesign it a bit - my CSS skills are almost as bad as my miniature-painting ones. Because I'm lazy, I've only tested it on Firefox and Safari, but my javascript style is so '90s (no jQuery!) that it should work flawlessly everywhere these days. Each page is entirely self-contained (except for pics) so it can be used offline, and I slapped in an iOS icon so you can add it to your home screen. I will probably put it on github when I get some time. Have fun!</p>toyghttp://www.blogger.com/profile/04499664589774142384noreply@blogger.com0