MySQL Named Locks in Python Context Managers

I’ve been using MySQL (and recently MariaDB) for many years – it must be something like 14 by now – but every now and then I learn something new about it. Recently, I’ve learned about named locks and how you can use to use your already-there MySQL server as a mean to create distributed locks which are not related to a specific DB transaction.

Here is an example of a Python function who’s internal code will never execute concurrently, even in a multi-process, multi-machine distributed environment, as long as all processes talk to the same MySQL database:

NOTE: this code uses SQLAlchemy-like session semantics, but can easily be applied to any Python MySQL client.

That’s very nice! This code will try to obtain a lock for 5 seconds before resuming execution. If the lock is obtained (meaning no other MySQL client has requested to lock this specific lock), execution will resume and when finished the lock will be released. If the lock cannot be obtained within the given timeout, meaning some other client is currently running this code, an exception will be thrown. In any case the code will never run more than once at any given time. Oh, and MySQL named locks are connection-bound, meaning they are released if the connection dies or is explicitly closed – but again this should not happen while the code is executing, but will keep us safe if the entire program crashes, for example.

Before I knew about this feature, we used to do some custom logical locking in our code (which never feels like a solid solution) or use transaction-level row / table locking which coupled our application’s logic with DB operations too much; MySQL named locks are decoupled of any actual data in your tables – its just a mean to get centralized app-level locking. And while there might be other, more lean mechanisms to achieve that, if you already use MySQL I believe this is a very good solution. To clarify, the Python code between GET_LOCK and RELEASE_LOCK can be anything and does not need to tie in to the database.

However, the code example above is not very clean and has a few disadvantages:

  • It does not handle exceptions properly. If an exception is thrown after the lock was acquired and before it was released, we are most likely going to end up with the lock not being released until the MySQL connection is closed, and we don’t know when that’s going to happen. Not good.
  • No clear separation of concerns – we have a single function that handles both application logic (the part between the locks) and provides the locking implementation. This can be solved in several ways, but I believe the way I’ll demonstrate below to be most elegant.
  • No code reusability, which is somewhat tied to the previous point. We cannot reuse the locking mechanism in other code paths very easily, and need to retype it. We also cannot reuse the application logic between the locks in a non-locked context – or even in unit tests for that matter.

We can solve all these issues very elegantly using the with statement and context managers. These features are one of my favorites idioms more or less unique to the Python programming language, and it’s these sort of features that I believe really help make Python code very clean and elegant without being too verbose.

We’ll start by creating a context manager for MySQL named lock:

And then proceed to use it in our function:

So what does this do? The @contextmanager annotation help us easily create a context-managed resource using generator-like semantics; The wrapper ensures that no matter what happens, the lock is released as we leave the managed context whether it is because the code executed successfully or because an exception was thrown.

The semantics of using the locking_context.named_lock context manager are extremely simple and readable, and reusing the locking context manager is a matter of an import statement and a single line of code. By injecting a mock or monkey-patched object as the first argument of named_lock(), we can also easily test the context manager itself and any code using it. In addition, if we ever need to switch from MySQL-based locking to some other implementation, it can be done more easily.

While the same flow can be achieved in many other languages supporting, for example, try / finally semantics, in most cases I’m aware of one will need to use more complex and less readable flow control structures such as callables to accomplish (yes, if you’re a JavaScript programmer this might make sense to you, but remember that pyramids, while look impressive from the outside, are really tombs with mummies and traps on the inside). I believe it is features like context managers that make Python a language that encourage writing clean code.

Imagick: Maintain (fake) transparency when saving as JPEG

I haven’t blogged in a while (have been busy you know), so I’ve decided to share this small piece of knowledge I’ve obtained by experimenting. I wrote a small test app (it’s for a feature of the next version of Zend Server – maybe I’ll share it one day when the API is stable), which does some image manipulation with the ImageMagick extension.

For those of you who don’t know ImageMagick allows one to preform pretty cool stuff on images – except for the usual drawing, conversion, rotation, rescaling etc., it also exposes some API to easily preform neat effects, like drop shadow, round corners and my newest favorite (apparently only available in the very latest builds of the extension) – the Polaroid effect.

In his blog Mikko Koppanen, the author of the ImageMagick PHP extension, shows how to create drop shadows (as well as other neat things – you should check out his blog!), but in his examples Mikko will always save as PNG, which is something I dare to say most web users will not do, and prefer saving as JPEG.

Problem with many of those effects, is that they leave parts of the image as transparent. When saving the picture as JPEG (as I do, since saving as PNG produces too big files), these transparent areas appear as black.

So after some experimenting, I’ve found out that the way to work around this is to composite another opaque layer as your background layer, filled with your background color of choice (white in my case). You will of course loose the ability to place the picture on other background colors and still have a nice “transparency” look – but as long as you stick to the background color you’ve set, it will look great.

Here is a code sample producing the same thumbnail + drop shadow as in Mikko’s example, but saving it with white matte color as JPEG:

<?php

$bgColor = '#ffffff'; // End result will have a white background

/* This was taken from Mikko's example */
$im = new Imagick( 'strawberry.png' );
$im->thumbnailImage( 200, null );
$im->roundCorners( 5, 5 );

$shadow = $im->clone();
$shadow->setImageBackgroundColor( new ImagickPixel( 'black' ) );
$shadow->shadowImage( 80, 3, 5, 5 );
$shadow->compositeImage( $im, Imagick::COMPOSITE_OVER, 0, 0 );

/* My addition: clone the entire image again to create the background layer */
$bg = $shadow->clone();

/* I'm using colorFloodFiilImage with high tolerance to paint it all white - maybe there are 'cleaner' ways to do it though */
$bg->colorFloodFillImage($bgColor, 100, '#777777', 0, 0);
$bg->compositeImage($shadow, Imagick::COMPOSITE_OVER, 0, 0);
$bg->setImageFormat('jpeg');
$bg->flattenImages();

/* Display the image */
header( "Content-Type: image/jpeg" );
echo $bg;

While there’s another step in the way, and the image will only look good on white backgrounds, you can now save it as a JPEG file with good compression and acceptable file size.

PHP Error Reporting Levels – WTF is 6134?

In PHP, the error reporting level (whether errors go to the log or to the screen or whatever) is determinded by the error_reporting INI directive (or at runtime using the error_reporting() function). Both take an integer as their value – and usually this integer is represented by error level constants like E_ALL, E_STRICT or E_USER_WARNING.

So in order to set the error reporting to anything but notices and strict errors, you would set something like this in php.ini:

error_reporting = E_ALL & ~E_NOTICE & ~E_STRICT

(actually, E_ALL does not really include E_STRICT but I put it here just to be more explicit)

This is actually great – one of the more easy to use and understand APIs in my opinion (yeah, I really like bitwise operations).

However, what I really hate is that sometimes I need to work with the integer value of the error reporting level (like 1 for E_ERROR or 84 for E_PARSE | E_CORE_ERROR | E_COMPILE_ERROR) and it’s very hard for me to remember what arbitrary numbers like 6134 mean.

So, today I wrote this tiny CLI script that helps me understand what an arbitrary error_reporting level integer might mean:

<?php

$errorLevels = array(
        'E_COMPILE_ERROR'     => E_COMPILE_ERROR,
        'E_COMPILE_WARNING'   => E_COMPILE_WARNING,
        'E_CORE_ERROR'        => E_CORE_ERROR,
        'E_CORE_WARNING'      => E_CORE_WARNING,
        'E_ERROR'             => E_ERROR,
        'E_NOTICE'            => E_NOTICE,
        'E_PARSE'             => E_PARSE,
        'E_RECOVERABLE_ERROR' => E_RECOVERABLE_ERROR,
        'E_STRICT'            => E_STRICT,
        'E_USER_ERROR'        => E_USER_ERROR,
        'E_USER_NOTICE'       => E_USER_NOTICE,
        'E_USER_WARNING'      => E_USER_WARNING,
        'E_WARNING'           => E_WARNING,
);

if (! isset($_SERVER['argv'][1])) {
        fprintf(STDERR, "Usage: {$_SERVER['argv'][0]} <error_level>n" .
                        "  Where <error_level> is a PHP error reporting leveln");
        exit(1);
}

$level = $_SERVER['argv'][1];

echo "Error level $level includes:n";
foreach ($errorLevels as $k => $v) {
        if ($level & $v) echo "t $k n";
}

echo "n";

To use, just run the script with a single value parameter, like so:

shahar.e@wintergreen ~ $ php error_level.php 6134
Error level 6134 includes:
	 E_COMPILE_ERROR 
	 E_COMPILE_WARNING 
	 E_CORE_ERROR 
	 E_CORE_WARNING 
	 E_PARSE 
	 E_RECOVERABLE_ERROR 
	 E_USER_ERROR 
	 E_USER_NOTICE 
	 E_USER_WARNING 
	 E_WARNING 

Enjoy!