Showing posts with label django. Show all posts
Showing posts with label django. Show all posts

03 April 2020

Django + PostgreSQL + Docker-Compose: a few gotchas

A lot of the tutorials out there make it look like it's trivial to set up a development environment with Django, Postgres and Docker. That never quite matched my experience; you always end up knowing too much about docker, and there are a few gotchas that most people typically fail to mention. The following are a few specifically related to Postgres and Django, which I'm writing here because I tend to forget them every time I start a new project...

A running container is not a running database

Docker-compose will happily report a container as "up" even though it's busy doing init work. With postgres, this means that a container might look "up" when really it's still creating the actual db instance, so app connections might well fail.

A good workaround is to use pg_isready, like this :

#!/usr/bin/env sh

docker-compose up
until pg_isready -d your_pg_db -h your_pg_host \
                 -p your_pg_port -U your_pg_superuser
do
    echo "Waiting for db to be available..."
    sleep 2
done
# now we can do actual work, like db migrations
...

Don't run; exec

A lot of howtos state, more or less, "if you want to run something in an instance, use docker-compose run some_machine some_command". This is misleading. run will create a new ancillary container, which will run in parallel to any other container of the same type that might already be up. If you want to execute an ancillary process inside an already-running container, use docker-compose exec some_machine some_command instead. This will ensure you are "logged on" the running container.

While coding, don't copy; mount

Many will tell you that you need to ensure reproducibility; and as such, your code should be copied or checked out to the instance in Dockerfile, i.e. at build stage. That is a huge drag on development, since you need to rebuild the whole image on every minor change. It is annoying and slow even with multi-stage builds.

Instead, you can mount your actual source directory as a volume, and exploit all the goodies that make development tolerable, like Django's autoreload features. Make your docker-compose.ymllook like this instead:

services:
   your_app_machine:
      volumes:
         - type: bind
           source: /host/location/of/src
           target: /container/location/of/app
   ...

When you want to run tests or go to production, use a second Dockerfile that inherits from the first (with FROM) and actually copies data (or more likely checks it out via git), without the volume definition.

Know your tools

This is not really specific to docker! Master your tools in depth, it will help. I honestly didn't know that JetBrains PyCharm can now configure the interpreter running in a Docker container as the main one for the project, which makes a lot of things easier (debugging, REPL etc). Extremely helpful!

15 August 2018

how to load initial data and test data in Django 2+

There are two ways to automatically load data in Django:

  • for data you need while running tests, place xml/json/yaml files in yourapp/fixtures.
  • for data you need while setting up the database from scratch, or at specific points in time, you must create a Migration

This is a bit annoying, because chances are these locations will get out of sync sooner or later, and it duplicates effort if you do reproducible builds, docker, and stuff like that.

The solution is to create a migration that actually loads fixtures. So:

  1. Create your fixtures: manage.py dumpdata --output yourapp/fixtures/yourmodel.json yourapp.YourModel
  2. Create an empty Migration: manage.py makemigrations --empty yourapp
  3. Edit the resulting migration (the last file created under yourapp/migrations, making it look like this:
    from django.db import migrations
    
    def load_fixtures(apps, schema_editor):
        # This is what will be executed by the migration
        from django.core.management import call_command
        # this is the equivalent of running manage.py loaddata yourmodel.json
        for fixture_name in ['yourmodel']: # add any additional model here
            call_command("loaddata", fixture_name)
        # add other calls if you have multiple models
    
    def rollback(apps, schema_editor):
        # This will be executed if you rollback the migration, so you want to clean up
        for model_name in ["YourModel"]:  # add any additional model here
            model = apps.get_model("yourapp", model_name)
            model.objects.all().delete()
    
    class Migration(migrations.Migration):
        dependencies = [
          # ... don't touch anything here ...
        ]
    
        operations = [
            migrations.RunPython(load_fixtures, rollback),
        ]
    # -*- coding: utf-8 -*-
    
  4. Profit

Note that this does not remove the option to have data that is available only in certain situation: just don't list the fixtures you don't want in the migration, and vice-versa.

19 November 2014

HOWTO Setup gunicorn on OpenBSD 5.6

As part of a silly web project of mine, I'm having fun getting reacquainted with OpenBSD.

Today I wanted to set up Gunicorn, a powerful webserver that will eventually run my Django application. It was not a straightforward process, so I decided to document it for my fellow geeks.

Preamble

This procedure assumes that you've already set up the following:
  • installed Python 3 (I tested with 3.4.1)
  • created a custom myuser account and logged on as such
  • created a virtualenv with pyvenv /home/myuser
  • installed Django in the above-mentioned virtualenv
  • created a Django project at /home/myuser/myproject

Let's go!

  1. activate the virtualenv: source bin/activate
  2. install Gunicorn: pip install gunicorn
  3. create /home/myuser/gunicorn_config.py with the following contents:
    command = '/home/myuser/bin/gunicorn'
    pythonpath = '/home/myuser/myproject'
    chdir = '/home/myuser'
    pidfile = '/home/myuser/.gunicorn_myproject.pid'
    user = 'nobody'
    worker_tmp_dir = '/tmp'
    errorlog = '/var/log/gunicorn_myproject'
    # tweak the following lines to suit your setup
    bind = 'localhost:8001'
    workers = 4
    You can add any further option as necessary, but it's important you keep pidfile and user.
  4. sudo to root (I'd recommend to do that from another account, i.e. don't add myuser to wheel) and create /etc/rc.d/gunicorn_myproject with the following contents:
    #!/bin/sh
    daemon="/home/myuser/bin/python"
    daemon_flags="/home/myuser/bin/gunicorn -c /home/myuser/gunicorn_config.py -D myproject.wsgi"
    
    rc_stop() {
        kill `cat /home/myuser/.gunicorn_myproject.pid`
    }
    
    . /etc/rc.d/rc.subr
    
    rc_cmd $1
    Note how we are specifying daemon to be the python executable. This is the only way I've found to keep the rest of the machinery in rc.subr working smoothly.
  5. chmod 555 /etc/rc.d/gunicorn_myproject
  6. now you can start and stop Gunicorn with standard OpenBSD 5.x commands: /etc/rc.d/gunicorn_myproject start
  7. Typically, you'd then set nginx to proxy and cache requests, and firewall gunicorn from the outside.
Note how gunicorn starts as root and then spawns processes as nobody; if I were to start it as myuser, it would not work. This is because gunicorn tries to change ownership of files and processes after startup, and OpenBSD doesn't seem to like it. This looks suboptimal to me, but I couldn't find a workaround.

Note also how we're writing the main process PID to file, and then using it to stop that same main process. Standard rc.subr machinery would expect me to specify a regex in the pexp variable, which would then be passed to pgrep to find the process; but this doesn't work with gunicorn because pgrep simply cannot distinguish between master and worker processes, since they appear to have the exact same command line.

If you know anything else I should do to further secure this setup, please let me know in comments!

04 June 2012

Django 1.4 help file CHM version (and how to build your own)

UPDATE 2012-06-04: God save the Queen! Thanks to the long "Jubilee Holiday Weekend", I got around generating an updated version for Django 1.4. Here it is: Djangodocs 1.4 in CHM format. Enjoy!

----------

This is a funny story.

I happen to think Microsoft's proprietary CHM format is lovely. So I went looking for a CHM version of docs for Django, and google found it for me on this blog. I duly downloaded it, tried to to open it and... it wouldn't display. I could only see the TOC, but not the actual documents. I thought this might be a corrupted version, and it was for an alpha release of Django anyway, so I though I'd compile a version myself. After all, these docs are built with Sphinx, which apparently can generate all sorts of formats...

So here's the procedure to compile django's docs:

  1. download and install Sphinx.
    Easy_install Sphinx
    was all I needed. Hurrah for Python.
  2. ADDITIONAL STEP for v1.4: modify _theme\djangodocs\layout.html to remove all javascript tags, otherwise you'll get jQuery-related errors in the final output. This is a known bug.
  3. Run Sphinx to generate the initial files:
    cd Django-1.1/docs
    mkdir _build/html
    %PYTHONDIR%\scripts\sphinx-build.exe -b htmlhelp -d _build\doctrees . _build\html
    
  4. Download and install Htmlhelp.exe from the Microsoft site. This will give you the HTML Help Workshop. Note: it doesn't matter if you get a final message saying you already have a more recent version.
  5. launch the workshop, File -> Compile..., select the file Djangodoc.hhc which should now be in _build/html, and this will produce the chm.
  6. ...??? Profit!

... Then I found out the reason that downloaded CHM didn't work was a stupid patch from Microsoft. Ouch.

Anyway, if you need it, here's the file: Djangodocs 1.1 in CHM format. If it doesn't work, make sure you follow this suggested procedure, and save yourself some time...

16 September 2008

on PyconUK 2008

Great conference, like last year. Interesting talks, nice people, spotless organization. Go to the PyconUK wiki if you want slides and (in a few days) recorded audio.

Next year the same bunch of great guys will host EuroPython, in June. I'm sure it will be fantastic, but I don't know whether I'll be able to attend, as it will happen during weekdays and I probably couldn't justify the absence from work. Apparently it's all because the French don't go to conferences during weekends; it's always their fault, isn't it ;)

Anyway, it was all good energy to start messing again with Django on a magnatune-inspired project. Also, we'll try to "reboot" Python North-West and see if we can stabilise it a bit. It's all fun :)

14 September 2008

Fact.

Jacob Kaplan-Moss looks like a young Robin Williams.

15 August 2008

django on jython

You can now run Django on Jython, as announced by Leonardo Muñoz. Lovely. I really should go back hacking django a bit.

08 April 2008

Google App Engine

Google App Engine. Write applications in Python using a WSGI-compatible application framework, then host them on Google’s highly scalable infrastructure.
(via Simon Willison's Links)

"Holy s**t" was Ryan Tomayko's comment, and my first thought as well. Google is realising the promises of WSGI, and what a sight it is.
On one hand, this is fantastic; it made me think of "J2EE done right". On the other hand, if you use GAE, you are handing your entire infrastructure and data (!) over to Google, which might not be the smartest move for lots of companies (any FTSE100 for example, and probably any NASDAQ-listed as well).

Anyway, Python rocks.

03 October 2007

Post-meeting

I published my django presentation slides on the Files section of the list page (in ODP and PDF).

My notes on the meeting are in the relevant post at Happenings in Python Usergroups.

I certainly enjoyed it, and I hope the other guys did too. The presentation was ok (even though I slightly fudged the demo, and didn't really go too deep into details); I think people enjoyed the overview, and I tried to communicate what django was all about, more than showing clever tricks or how to satisfy obscure requirements.

Who knows, maybe one day I'll actually make a living out of this stuff...

08 September 2007

Liveblogging Pycon UK 2007 - first notes

Lots of people, fantastic atmosphere, lots of kudos to the Birmingham user group. "Msdn magazine" in the goodie bag sounds like MS really wants to invest in Python, that's probably why Sun decided to go on Ruby instead.

First talk on SQLALchemy by Paul Johnston, very very interesting, too bad it was a bit squeezed, another 15 minutes would have helped. Must investigate the reflection vs autoupdate stuff & migrate (django doesn't manage db changes very well at the moment).

Second talk: Mr. Voidspace (Michael Foorde) on Silverlight & IronPython -- lots of possibilities there, but still very very early. 1Mb of local space is very little for serious usage, there are accessibility issues (but possibly less than Flash) and limitations (1 Canvas only). But having an embedded mini-CLR/DLR in every browser (currently IE/FF/Safari, with Opera in the works) gives me a feeling of "ActiveX done right", something Java should have done 5 years ago.

Made contact with other Manchester pythonistas, it's really true that pythons hide under rocks! Looking forward to build a community in Manc when we go back, people really wants to invest serious time on Python apparently. Oh, and Resolver is hiring, but it's London-based and they do Xtreme (pair) programming, so no telecommuting, but if you are in the area and you fancy "coding the way Guido indented it" give it a go, they seem very nice guys.

Time for Django stuff with Simon Willison!

UPDATE: Simon rocks. Fast as lightning and to the point, lots of goodies for serious django usage, I hope the session was filmed because it was really worth it.

UPDATE: I met Phil Thompson, the creator of PyQt! Jeez, I probably sounded like a fanboy (that I am). And I also met a guy not just from around Manchester -- from Stockport! Astonishing, the world is so small these days. Am now on the PyQt tutorial from Mark Summerfield (who has a book finally coming out on PyQt! fantastic), Trolltech provided some very nice freebies. I feel in geek heaven.

UPDATE: Mark was great, but 2 hours straight are a bit much, so in the end the class was clearly a bit tired. Will definitely go back to his presentation very soon. Break now, then on to the lightning talks -- the list looks endless, might not do them all. Organizers expected about 100 people, got more than 200...

UPDATE: the first lightning talks: Open Spaces (weird, not sure I got it), a lovely chap trying to convert the Hansard (which is getting XML already) in RDF, Jeff Tupholme on putting javascript inside python (crazy) within LiveConnector.

UPDATE: pydoctor statically analyses code to generate docs and then uses a pseudo-wiki interface to correct typos and generate diffs (sounds nice); a lexer parsing thing which went over my head; a freakily-dressed guy from ACCU on how to pitch Python to C(++) shops (use "high-level"!); Software Freedom Day next week (eek! I'm in Oslo)!