What are Generators?

I like to describe Generators as special functions which are iterable and maintain state. Think of a function that instead of returning once and destroying its state (local variables) after returning, can return multiple times, while maintaining the state of local variables, thus allowing iteration over an instance of that function state. In fact, a call to a generator function creates a special Generator object which can be iterated. The object maintains the internal state of the generator, and on each iteration generates a new value. The same result can be achieved by implementing a Traversable class, but with much less code.

This is very different from the way we are used to think of functions, so maybe an example is the best way to demonstrate this. I will use a simplified example based on the one given in the documentation:

function xrange($start, $end, $step = 1)
{
  for ($i = $start; $i <= $end; $i += $step) {
    yield $i;
  }
}

$start = microtime(true);
foreach (xrange(0, 1000000) as $i) {
  // do nothing
}
$end = microtime(true);

echo "Total time: " . ($end - $start) . " sec\n";
echo "Peak memory usage: " . memory_get_peak_usage() . " bytes\n";

In the example above, the xrange function is a Generator which operates in a similar yet simplified version of the range() PHP function (just like in Python!). The main thing to notice is the yield keyword – this tells the function to yield a value – which means a value is “returned” but the state of the generator is maintained.

When iterating over a generator function, as you can see in the foreach loop, iteration continues as long as a value is yielded. Once the function returns without yielding (as xrange in our example would do once the inner for loop is done), iteration stops. We get a behaviour which is (almost) equivalent to range in the sense that it allows us to iterate over numbers – but, without allocating the entire array of numbers in advance. In our example, we save a lot of memory and in fact execution is faster when a generator is used.

To demonstrate, here is the output of the script above (ok, I added some formatting to the output, but the results are real!):

$ /usr/local/bin/php /tmp/with-generators.php
Total time: 0.20149302482605 sec
Peak memory usage: 234,256 bytes

This is on a one-million integers “array” (unlike range, no real array is allocated so we can’t do random access on members, but during iteration it behaves just like an array).

By comparison, executing the same code with range() instead of xrange(), results in the following:

$ /usr/local/bin/php /tmp/without-generators.php
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 32 bytes) in /private/tmp/generators.php on line 12

Ok, we reach our memory limit. Lets try to go crazy (not a good idea in production):

$ /usr/local/bin/php -d memory_limit=200M /tmp/without-generators.php
Total time: 0.31754398345947 sec
Peak memory usage: 144,617,256 bytes

After increasing the memory limit to 200 MB, the script runs: but it takes longer (honestly, to my surprise), and consumes an order of magnitude more memory.

Pretty cool, huh?

Just to demonstrate, calling var_dump on a generator would result in this:

var_dump(xrange(0, 100));
// Output:
// object(Generator)#2 (0) {
// }

But I can do the same thing with Iterator interfaces, no?

Yes! pretty much anything you can do with Generators can be done by creating class which implements either the Iterator or IteratorAggregate interfaces. But in many cases, a lot of boilerplate code can be removed if a Generator is used instead. For example, a class equivalent to the xrange generator above would look like this:

class XrangeObject implements Iterator
{
  private $value = 0;
  private $start = 0;
  private $end   = 0;
  private $step  = 1;

  public function __construct($start, $end, $step = 1)
  {
    $this->value = (int) $start;
    $this->start = (int) $start;
    $this->end   = (int) $end;
    $this->step  = (int) $step;
  }

  public function rewind()
  {
    $this->value = $this->start;
  }

  public function current()
  {
    return $this->value;
  }

  public function key()
  {
    return $this->value;
  }

  public function next()
  {
    return ($this->value += $this->step);
  }

  public function valid()
  {
    return $this->value <= $this->end;
  }
}

$start = microtime(true);
$xrange = new XRangeObject(0, 1000000);
foreach ($xrange as $i) {
  // do nothing
}
$end = microtime(true);

echo "Total time: " . ($end - $start) . " sec\n";
echo "Peak memory usage: " . memory_get_peak_usage() . " bytes\n";

Wow, that’s much more code for something we achieved very simply with a generator. BTW, the results are:

$ /usr/local/bin/php /tmp/with-iterator.php
Total time: 0.61971187591553 sec
Peak memory usage: 240,968 bytes

As you can see, memory usage is comparable to a Generator. Run time is more than 3 times slower, but in most realistic use cases this time is usually negligible – in any case unless we would have seen an order of magnitude of difference, performance is not a major issue here. The interesting thing really is the amount of boilerplate code we had to use when creating an iterator – most of this code is just generic boring stuff and not what we really care about. With Generators, the implementation is much shorter.

How about a realistic use case?

Ok, so we have used a generator to iterate over numbers. Woopti-doo. We can just drop the generator and use the for loop inside it to achieve the same thing. How about a more realistic use case?

Take a look at the following example, which I believe can be pretty useful and still has fairly straightforward code: a generator which combines the efficiency of XMLReader with the simple API of SimpleXML to bring you an efficnet yet easy to use XML reader function for possibly large XML streams with repeating structure – for example, RSS or Atom feeds.

function xml_stream_reader($url, $element)
{
  $reader = new XMLReader();
  $reader->open($url);

  while (true) {
    // Skip to next element
    while (! ($reader->nodeType == XMLReader::ELEMENT && $reader->name == $element)) {
      if (! $reader->read()) break(2);
    }

    if ($reader->nodeType == XMLReader::ELEMENT && $reader->name == $element) {
      yield simplexml_load_string($reader->readOuterXml());
      $reader->next();
    }
  }
}

The xml_stream_reader() generator defined above will use XMLReader to open and read from an XML stream. Unlike PHP’s SimpleXML or DOM extensions, it will not read an entire XML document into memory, thus avoiding potential blowups on very large XML files. To keep things simple for the user however, whenever it encounters the XML element searched by the user (e.g. the item element in RSS feeds), it will read the entire element into memory (assume each item is small but there are potentially thousands of items) and return it as a SimpleXMLElement object – thus still providing the ease of use of SimpleXML for the consumer.

Here is how it can be used:

$feed = xml_stream_reader('http://news.google.com/?output=rss&num=100', 'item');
foreach($feed as $itemXml) {
  echo $itemXml->title . "\n";
}

While I couldn’t find a large-enough XML file to test this on, even with 2mb files, this can be much more efficient than DOM or SimpleXML, and without too much more coding.

So I’m really happy about the addition of generators – it’s a cool feature. Not one you’d use every day, but in some places where complex Iterators had to be implemented (and where OO features such as polymorphism are not required), generators can be a real neat, concise and maintainable solution.

Even as an experienced developer with good knowledge of the LAMP stack setup, setting up new vhosts and other configuration for each new project is sometimes a pain in the ass.

There are of course good news – starting from PHP 5.4, the Command Line Interface (CLI) version of PHP comes with a build-in Web server that can be used to serve PHP apps in development. This Web server is very easy to use – you just fire it up in the right place and it works, serving your PHP files. While it is by no means a viable production solution (it is a sequential, no-concurrency server meaning it will only serve one request at a time), it is very convenient for development purposes.

While it “just works” for simple “1-to-1 URL <-> File” apps, it can work almost as easily for rewrite based MVC apps, including Zend Framework 1.x and 2.x apps and probably for other frameworks as well.

As a start, here is how you launch the Built-in Web server (assuming you have PHP 5.4 in your PATH):

$ php -S localhost:8080

The -S flag means “start the built-in server and listen on localhost port 8080″ (which of course can be changed, but usually in development you want it listening to local requests only). You can always hit Ctrl+C to quit the server.

This tells PHP to treat the current directory as the document root, and serve files as they are requested – basically if $_SERVER['REQUEST_URI'] matches a local file it will be served (static file extensions are identified and sent as-is, while other files are parsed as PHP files and are executed). If the request URI does not match a local file, a 404 error will be returned.

This works well for simple no-rewrite apps – but what about more complex MVC apps? That’s easy to – all you have to do is give the Web server a file which will be used as a catch-all “router” script, so in a ZF app for example, you can do the following:

$ php -S localhost:8080 public/index.php

This means all requests will now go through public/index.php. If you try it with your ZF app, you will see it works almost perfectly – ZF views are rendered very nicely. The only problem is that static file references (images, CSS files and JavaScript mostly) do not work. That’s because these requests are routed through index.php as well – and it does not know how to handle those.

Of course, you could write a complex plugin to intercept such requests and handle them from within ZF, serving static content using, for example, PHP’s readfile() function. But this seems like a huge overhead for something which is only needed in development environments (in production, you will have the proper rewrite rules set up for your production-grade web server of choice, of course).

My solution came from reading a little bit more about the built-in Web server. It turns out that if the router script returns FALSE, the web server will fall back to serving the file directly, as if no router script was specified in the command line. So I wrote this tiny little wrapper PHP script that I now use in my ZF app. You can even include it in the source tree as it is a virtually harmless file to have even in a production environment:

Save this file in your project as public/builtin-ws-wrapper.php and then run the built-in Web server like so:

$ php -S localhost:8080 public/builtin-ws-wrapper.php

In a way very similar to how the recommended ZF rewrite rules work, the script checks if the file being requested is a real file (and is under the document root) and if so returns FALSE – this tells the built-in Web server to handle it from here. Otherwise, it simply includes the framework’s main entry file, index.php, and lets Zend Framework take care of the rest.

So what are PHP extensions?

An extension in PHP is in fact a module providing some functionality to the PHP Engine. While the term makes it sound like extensions provide some kind of special functionality, in reality many of the language’s most basic functions and classes are provided in extensions. Many extensions are shipped as part of the default PHP distribution, and some are in fact compiled into PHP in such way that they cannot even be unloaded. Come to think of it, perhaps it is best to think of PHP extensions as “language modules”.

Another important aspect of Extensions (in the PHP world) is that like PHP they are written in C (and on rare occasions C++) and are compiled and loaded into PHP as a shared object or DLL, or compiled statically into the PHP binary. This is in contrast with libraries, components and frameworks which in most cases are written and distributed as PHP code.

Some examples you are most likely familiar with are the mysql extension providing all the mysql_* functions, or the gd extension providing image processing functions. Both of these extensions are almost always included with the default PHP distribution. More examples include the session and SPL extensions, which are hardly separable from PHP but are considered extensions because mechanically they use PHP’s internal Extension API.

Most extensions provide new user-space API (PHP functions, classes and constants). Some extensions may not provide new APIs directly, but interact with other extensions or with PHP’s core to provide added functionality – for example, the PDO extension provides new API, but the PDO_MySQL and PDO_OCI extensions do not – they merely add capabilities to the PDO extension.

How are extensions shipped?

As mentioned above, PHP extensions are written in C or C++, which means they are usually shipped as C/C++ source code and sometimes as pre-compiled, OS and CPU architecture specific binaries. Many extensions are shipped with PHP – and if they are not already enabled, all you need to do is enable them in order to use them (more on that later).

There are also many extensions which are not shipped with PHP but are developed in PHP’s official extension repository, called PECL (the PHP Extension Community Library). PECL extensions can be downloaded, compiled, and added to your PHP installation provided that you are running the right version of PHP. This can be done automatically using a command line tool called ‘pecl’, or can be done manually using a set of standard build tools. For Windows users, you may compile extensions manually but pre-built extension DLLs can sometimes be found around the Internet for many of the popular extensions.

If you have the right build tool chain installed, installing an extension using pecl is usually dead simple:

$ pecl install apc

And that’s more or less it. If your extension builds on a 3rd party C library (as do many extensions), you will need to have that library installed as well.

In order to be accepted into PECL, extensions must adhere to certain standards set by the PHP development team. This means you can usually trust PECL extensions to work well, provided that they are actively maintained and are considered stable by their development teams.

For example, the APC, gnupg and sphinx extensions are all relatively popular extensions provided through PECL.

In addition to PECL, PHP extensions may be development outside of PECL – sometimes due to legal / licensing reasons, and sometimes for other reasons. You may find some PHP extensions outside of PECL – but it is recommended to be more cautious with PHP extensions not available through PECL, as these may not adhere to the standards of the PHP development team.

Enabling and Disabling PHP extensions

Once you have an extension installed (this usually mean you have this extension’s DLL or .so file placed in your PHP’s extension directory), you will need to enable it in order to use it.

First, check the output of phpinfo() to see if the extension is not already enabled.

If it is not enabled, you will need to edit your php.ini and make sure that the following is set:

extension_dir=<path to the directory containing your extensions>
extension=myext.so

Where “myext.so” is the file name, without the path, of the extension’s .so or DLL file. You will notice your php.ini probably includes multiple extension= lines. This is fine – there should be one for each extension you want to load. You can of course comment out lines referring to extensions you don’t need by adding a semicolon (;) at the beginning of the line.

After making changes to your php.ini, you need to restart your Web server (or your PHP process pool if using FPM / FastCGI) to apply the changes.

To ensure your new extension is loaded, check phpinfo() again. Check your PHP error log (or Web server error log) for any errors during restart which might indicate a problem loading the extension.

For production servers, it is recommended that you disable any extension which you do not use. This may save you a little bit of server memory and CPU cycles, and most of all will reduce the chances of unknown bugs or security issues in those extensions affecting your server.

Keep in mind that some extensions may be build statically into PHP. You will see these listed as extensions in php.ini but you will not be able to disable them, and in most cases you will not see an extension= line referring to them in php.ini or an .so / .DLL files. Removing statically compiled extensions requires recompiling PHP itself, and in most cases this is hardly needed as most statically compiled extensions tend to include core functionality which rarely needs to be removed.

Creating Extensions

If you are a C/C++ programmer and want to create your own extension, you will need to learn about the Zend Engine 2 API. Unfortunately, there isn’t a lot of documentation on the subject, and what is available as part of the PHP manual has long been outdated.

It is recommended you check out Sara Golemon’s excellent book Extending and Embedding PHP. It has been around for quite a few years now and does not cover the most recent APIs, but it is definitely a good starting point. In addition, you should look for extension creating presentations from recent years PHP conferences – almost every conference has one of these by someone from the PHP Core team. Of course, looking at other extensions source code is always the best place to learn

sudo useradd -m -G `groups ubuntu | cut -d" " -f4- | sed 's/ /,/g'` -s/bin/bash newuser

Of course ‘ubuntu’ is the user you want to copy, and ‘newuser’ is the name of the new user.

Note that the new user will be in the admin group but will still require a password when using sudo (that’s because in the EC2 images ‘ubuntu’ is the only user with NOPASSWD privileges. I personally believe this is a good thing, but if you want you can always add NOPASSWD on the admin group in /etc/sudoers.

I won’t repeat the reasons why methods such as bcrypt are preferred (read the comments on the previous post to learn why). However, I will note that starting from PHP 5.3 bcrypt is in fact built-in to PHP – so if you do not require portability to older versions of PHP, bcrypt-hasing could be done very easily, using the useful but a bit enygmatic crypt function:

/**
 * Hash a string using bcrypt with specified complexity
 *
 * @param  string $password input string
 * @param  integer $complexity bcrypt exponential cost
 * @return string
 */
function bcrypt_hash($password, $complexity = 12)
{
  if ($complexity < 4 || $complexity > 31) {
    throw new InvalidArgumentException("BCrypt complexity must be between 4 and 31");
  }

  // CRYPT_BLOWFISH salts must be 22 alphanumeric characters long
  $random = get_random_alnum_salt(22);

  // The crypt function decides which algorithm to use (we need Blowfish) based on
  // the format of the salt parameter
  $salt = sprintf('$2a$%02d$%s', $complexity, $random);

  return crypt($password, $salt);
}

/**
 * This generates a random alphanumeric string of length $length.
 *
 * This may not be a cryptographic grade random string generation
 * function - but it is good enough for our example
 *
 * @param  integer $length random string length
 * @return string
 */
function get_random_alnum_salt($length)
{
  static $chars = null;
  if (! $chars) {
    $chars = implode('', array_merge(range('a', 'z'), range('A', 'Z'), range(0, 9)));
  }

  $salt = '';
  for ($i = 0; $i < $length; $i++) {
    $salt .= $chars[mt_rand(0, 61)];
  }
  return $salt;
}

/**
 * Check a password against a hashed string
 *
 * @param  string $password cleartext password
 * @param  string $hashed bcrypt format hashed string
 * @return boolean
 */
function bcrypt_check_hash($password, $hashed)
{
  // Do some quick validation that $hashed is indeed a bcrypt hash
  if (strlen($hashed) != 60 || ! preg_match('/^\$2a\$\d{2}\$/', $hashed)) {
    throw new InvalidArgumentException("Provided hash is not a bcrypt string hash");
  }
  return (crypt($password, $hashed) === $hashed);
}

As you can see, the code is pretty simple – hashing is done internally using the crypt function. It is interesting to note that the returned hash string is of a very particular format – it already contains the complexity and salt parts. So comparing hashed string to cleartext originals is also easy and done nicely by crypt – you give it the hashed string as the salt parameter, and it takes care to only take the complexity and salt prefix of the entire hashed string into account. If the result is the same string (after all the same complexity and salt were used), you will get the exact same hashed string back. Nice!

It’s important to note that bcrypt is intentionally slow – on my laptop it hashes a medium-sized password with cost factor 12 at about 0.3 seconds. It means this is a very good solution for password hashing (because you only do it once in a while at user registration and login but not in the critical path of any frequently repeating action), but not so useful for general purpose string hashing – for such uses, you can still use MD5, SHA-1 and friends.

Also keep in mind that increasing the cost factor increases the time it takes to hash string dramatically, so as computing cost is reduced you can simply increase the cost factor to ensure you are future-proof.

Since Shoppimon is based on Zend Framework 2.0, it also heavily relies on the ZF2.0 autoloader stack. Class autoloading is convenient, and was shown to greatly improve performance over using require_once calls. However, different autoloading strategies have pros and cons: while PSR-0 based autoloading (the so called Standard Autoloader from ZF1 days) works automatically and doesn’t require updating any mapping code for each new class added or renamed, it has a significant performance impact compared to classmap based autoloading.

Fortunately, using ZF2′s autoloader stack and Phing, we can enjoy both worlds: while in development, standard PSR-0 autoloading is used and the developer can work smoothly without worrying about updating class maps. As we push code towards production, our build system takes care of updating class map files, ensuring super-fast autoloading in production using the ClassMapAutoloader. How is this done? Read on to learn.

Before auto-generating class map files, you need to make sure your code supports both using a class map based autoloader, and falling back to the standard autoloader. This is how it is done in the main Shoppimon bootstrap file:

// Set up the autoloader
require_once 'Zend/Loader/AutoloaderFactory.php';
Zend\Loader\AutoloaderFactory::factory(array(
  'Zend\Loader\ClassMapAutoloader' => array(
    include 'library/Shoplift/autoload_classmap.php'
  ),
  'Zend\Loader\StandardAutoloader' => array(
    'namespaces' => array(
      'Shoplift' => realpath('library/Shoplift')
    )
  ),
));

Note that ‘Shoplift’ is the namespace for our in-house library used in addition to Zend Framework 2.0. We have similar code in the Module.php file of each one of our modules:

public function getAutoloaderConfig()
{
  return array(
    'Zend\Loader\ClassmapAutoloader' => array(
      __DIR__ . '/autoload_classmap.php'
    ),
    'Zend\Loader\StandardAutoloader' => array(
      'namespaces' => array(
        __NAMESPACE__ => __DIR__ . '/src'
      )
    ),
  );
}

Both code segments reference a class map file named autoload_classmap.php – these exists for our common library and for each one of our modules – and this is what they look like:

<?php

/**
 * Autoloader classmap
 *
 * Keep the classmap empty - it is auto-generated at build time
 */

return array();

Yes – an empty classmap array is returned. This is how this file is committed to SCM, and we even add it to our .gitignore file to ensure it is not mistakenly overwritten with actual data. Returning an empty class map ensures that the first ClassMapAutoloader always fails, and that the StandardAutoloader is always used. That is, unless a classmap file is properly generated at build time.

We’ve decided to use Phing for building our PHP apps – it wasn’t a hard decision, and was based on the fact that Phing is well known and widely used, it is very similar in concepts and syntax to Ant which we were already using for some of our non-PHP stuff (yes, we have some Java and Python as well, not everything is PHP you know…) and, unlike Ant, we can easily extend it based on our needs using PHP – which is exactly what we’ve done in order to ensure class maps are populated at build time.

Here is a segment from our Phing build.xml file responsible for generating class maps:

<target name="generate-classmap" depends="copyfiles">
  <taskdef classname="phing.tasks.ZfClassmapTask" name="classmap" />
  <classmap outputFile="${builddir}/data/library/Shoplift/autoload_classmap.php">
    <dirset dir="${builddir}/data/library">
      <include name="Shoplift" />
    </dirset>
  </classmap>
  <classmap outputFile="${builddir}/data/module/Frontend/autoload_classmap.php">
    <dirset dir="${builddir}/data/module" includes="Frontend" />
  </classmap>
  <classmap outputFile="${builddir}/data/module/InternalApi/autoload_classmap.php">
    <dirset dir="${builddir}/data/module" includes="InternalApi" />
  </classmap>
</target>

The “generate-classmap” build target takes a set of files (previously copied to a working directory in the “copyfiles” target), scans them and generates several classmap files – one for the library/Shoplift directory, and one for each one of our modules – “Frontend” and “InternalApi”. This is done after defining a new task type: the “classmap” task, defined in the phing/tasks/ZfClassmapTask.php file, which looks like this (this is where the magic is):

We place this file in the ‘phing/tasks’ directory under our source tree, but it can actually be placed anywhere in the include path (see comment inside the file) as long as the classname=”…” attribute of the <taskdef> element is adjusted accordingly.

This is a Phing task class, and was built by adapting the classmap_generator.php tool included in ZF2 to Phing. It scans the directories mentioned in the <dirset> elements, finds PHP classes, and adds them to the generated class map file. This is done automatically at build time, takes a couple of seconds, and allows us to enjoy uninterrupted development as well as good production performance.

BTW as you can see the Phing class was posted as a Gist in Github – so feel free to fork it, fix it and improve it – and please share the results.

The idea of Shoppimon is simple – we want to provide Web monitoring and availability analysis which will be useable by, and useful to “normal” people – not only the tech guy, the programmer or the IT specialist, but the site owner, the business owner or even the marketing guy – in other words the real stake holder.

Shoppimon focuses on synthetic monitoring – it simulates real users going through the store and logs their “experience” – any errors encountered, overall time to complete certain actions, etc. – as such it complements real-user monitoring that products like Zend Server provide). In comparison to existing synthetic Web monitoring services, we want Shoppimon to be a snatch to get started with, an inexpensive solution that would be useful to even the smallest commercial site owners, and most important using it should not require one to be a tech savvy person.

Shoppimon is focusing on Magento – a popular PHP / Zend Framework based eCommerce solution. Magento has proved to be a popular eCommerce solution and has grown a rich ecosystem of developers and service providers around it, and has a rich community of users. But as most PHP programmers know, managing Magento’s availability and performance isn’t easy. It’s a heavy application, one of the heaviest PHP apps ever built, and the codebase is somewhat complex. Our goal is to help Magento store owners find (and fix) problems in their stores, and show them how their store compares to others.

This is why we decided to focus Shoppimon on Magento and provide objective scientific data (which we present in an easy-to-digest way) that would help Magento store owners (and in turn their developers and hosting providers) to enjoy smooth sailing.

Technically, Shoppimon is pretty interesting (hey, otherwise I wouldn’t have done it) – it has parts written in different programming languages. It is entirely based on Cloud technologies. The front-end runs on Zend Framework 2.0 (it is possibly the first commercial app to be out there running on ZF 2.0) and on Zend Server. The backend parts mix PHP, Java and Python (a long story…) and uses some very interesting technology to simulate real shoppers browsing around on the Magento sites we test.

In the next few weeks I’ll probably be posting more about how stuff work at Shoppimon under the hood. Stay tuned!

So, the topic in question is “what is the right way to store user passwords in my DB”. To be clear, I am talking specifically about the passwords users will use to log in to your application, not some 3rd party password you need to store for whatever reason. This is something almost any application out there requires – unless you interface with some external authentication mechanism (OAuth, openId, your office LDAP or Kerberos server), there’s a very high chance you’ll need to authenticate users against a self-stored user name and password.

In order to figure out what is the best solution, let’s start by going over the problems we might face if we simply take the naive approach and store passwords in their original, clear text form:

• If our database gets hacked (for example if we are exposed to an SQL injection attack through some 3rd party app we have installed on our server), passwords could get stolen. Clear-text passwords could easily be used to hijack our users’ accounts. In many cases, even if we ignore the risk of a hacked database – a clear-text password can be stolen by a disgruntled worker with access to the DB.
• Moreover, users tend to use the same passwords for all sorts of different services. If I know someone’s password to one service, chances are I can use the same password or a similar one to impersonate that user in other sites as well. Most users do not consider the fact (but you should!) that when they type in a password for your silly site with funny kitten pictures, there’s a good chance they are entrusting the password to their bank or PayPal account in your hands.

Given the points above, it is our responsibility as web developers to ensure that nobody, not even us, could get a clear-text version of the user’s password. So what’s the right way to do that?

Best solution: avoid the problem all together

The best way is, of course, not to store passwords at all. If functional and business terms allow, you should consider using some other authentication mechanism like OAuth or OpenID – basically let someone else worry about storing sensitive data. Many users have Google or Facebook profiles, and you can find several examples of pretty big websites that simply let users authenticate through these providers instead of through their own identity management service. This is an elegant solution, but clearly it does not work in all cases. If you still need to store passwords, read on.

Password Hashing

One aspect of passwords is that really, you have no use for their actual value. A password can be anything, and as long as the user repeats the same initial password when asked to sign in, you do not care what the password’s textual value is. This is important, because it means we don’t need to store the original value of the password: it is safe to store a product of the password, and as long as we know how to reproduce this product from a typed-in password, we can compare the products and not the password itself.

Sounds confusing? Here is a simple example: assume that instead of passwords, our site uses numbers to authenticate users. Each user has to provide a user name and a random number as a password. Before we store this “password” in our DB, we pass it through this simple mathematical function:

    f(x) = x * 5

So if a user types in “4709″ as a password, we multiply that number by 5 and store the value “23545″ in our DB. When the user attempts to sign in again, we pass the value typed in as password through the same function. If we get “23545″ as the product of the typed in password passed through our function, we know the user typed in the right password.

This is good, because now the value stored in our DB is not the actual password, but an obscure value (well, sort of). If someone steals our DB, they can type in “23545″ into the sign-in form all day – but that’s not the right password (remember, we multiply whatever is typed in by 5 before comparing it!).

Unfortunately, things are not that easy. First, it’s enough for someone to know that we multiply numbers by 5 to obfuscate them, and they can easily reverse engineer passwords by dividing the data stored in the DB by 5 – our function is reversible. Second, a smart enough hacker looking at our list of stored password values would probably notice that all of them are multiples of 5 – so even without knowing what we do in advance, reverse engineering our “security” method is quite easy.

As it turns out, the approach is right but the function we are using is too simple. What we need is a function that:

• Is irreversible, or at least is impossible to reverse in the real world. In other words, even if one knows both the result and the function used, computing the input value will be impossible or impractical.
• Is indeed a function in the mathematical sense – that is given the same input value, only a single outcome is possible, and the same outcome is always produced for the same input. For example, a computer function which multiplies the input value by a random number is no good for us.
• Produces a distinct, single value for each input value – or at least provides a very low risk of producing the same result for two input values. This is important because we want to make sure that only one typed-in password matches the obscured value stored in the DB.

Of course, most of us are not mathematicians – so you’d be happy to know that  such functions exist, and are usually referred to as “hash functions”. Hash functions use used extensively by programmers for all sorts of uses, and cryptography is most definitely one of them. A few popular and useful cryptographic-grade hash functions are MD5, SHA-1 and SHA-2. We will not go into the mathematical definitions of these functions (I have very little knowledge of how these functions actually work!) – it’s enough to say pretty much every popular programming language  out there has at least one implementation of these functions.

As an example, the MD5 function produces a 128-bit “hash value” for an input value provided to it. The hash is always 128 bit long, regardless of the input size. It will always produce the same hash value for the same input. The most successful known collision attack (that is an attack producing the same hash value for a modified input value) on MD5 took about 2 million execution attempts which makes it quite bad for validating SSL certificates (which it used to be used for), but still sort of Ok (although not great) for password hashing.

To compute an MD5 hash value in PHP, you can do the following:

    php > echo md5("my name is Inigo Montoya");
    d9937edae7d26a399d41dda16f137e42

As you can see the MD5 value of the string “my name is Inigo Montoya” is “d9937edae7d26a399d41dda16f137e42″ (this is in fact a hexadecimal representation of the MD5 value, which is a 128 bit number – this is the standard way to present hash values of various functions). On the other hand:

    php > echo md5("my name is Indigo Montoya");
    ae7cd5e68c73f9f44df66030cc9d1c06

Even a slight change in the input text produces a completely different MD5 hash.

It’s time to stop using MD5 for cryptographic purposes

While MD5 was the de-facto standard for storing hashed passwords for some time, it is now becoming clear that it may not be suitable for cryptographic purposes (it is definitely suitable for other things). In 2009 it was shown that producing collisions for MD5 can be done within seconds or minutes on commodity hardware – this does not mean it is easy to reverse engineer password values stored in your DB as MD5 hash values, but it does mean that if a highly skilled hacker wants to specifically target your site, they have a better chance of succeeding in doing so. In addition, it is safe to assume additional vulnerabilities will be detected in the future.

Unless you are somehow limited (not if you’re a PHP developer!), switching to stronger hash functions such as SHA-1 or SHA-256 is highly recommended.

Throughout the rest of this article I will use SHA-1 in examples. SHA-1 is not collisions free, but so far the best known theoretical collision attack on SHA-1 took 2 to the power of 51 attempts to perform (that’s a number with 16 digits!), and until now nobody has been able to show an actual successful attempt to do so. SHA-1 produces a 160-bit digest values, and can be computed in PHP like so:

    php > echo sha1("my name is Inigo Montoya");
    b208946a9c3c4b26a4d6bb87c3f630f996146ee

So, I should just store the password hash in the DB?

Well, yes and no.

Yes – because that’s the first step. By storing a SHA-1 hashed version of the password in your DB you ensure nobody can compute the password by simply stealing your users DB table data. When a user types in their password, you compare the stored hash to the SHA-1 hash of the typed in string, and if they match, you grant access. Simple and effective.

But wait… that’s still not good enough.

Stupid but Effective: Dictionary Attacks

All hash functions are vulnerable to a type of attack sometimes referred to as rainbow attacks or dictionary attacks.

These attacks take advantage of the fact that in most cases, humans are humans – and the passwords they use are of limited size (how many people can you think of that use 12 or even 10 character long passwords?) and are composed of a limited set of characters (remember that even power users that use punctuation, numbers and mixed-case characters in their passwords are still confined to the ~75 characters or so on their keyboards).

Dictionary attacks are stupid but effective: the idea is to create a dictionary (basically a big key -> value table) of predictable passwords (dictionary words, expected combinations of key strokes, all permutations of what’s on your keyboard up to 8 characters long) and their MD5 or SHA-1 values. Once such a table exists (creating it may take several hours on commodity hardware, but this is a one-time effort), you can search for an original password using it’s hash value.

A dictionary attack allows me to reverse-engineer the original password from it’s hash value not by smart computation (which, given a good hash function, is impossible or impractical), but through a simple query to a ready-made “dictionary” mapping hash values to original strings.

But it’s even easier than that: nowdays there are services that offer such dictionary lookup in their existing databases. It’s not even required to do the work of building the dictionary.

One good solution to dictionary attacks is forcing your users to mix punctuation, upper and lower-case characters and numbers in their at-least 12 character long passwords. However, we all know that in many cases this means expecting too much from your users.

The practical solution to dictionary attacks is quite simple, and is called salting.

Just Add Salt

Salting is a simple yet effective method to improve the security of stored passwords and prevent dictionary attacks. The idea is that instead of expecting a long, random password from the user, you take whatever password the user provides and add additional random noise (referred to as “salt”) to it yourself. You store that random noise next to the password, and use it to compute the hash when checking passwords.

Once a long enough and random enough salt is added, comparing hash values stored in your DB to a dictionary becomes very hard: an attacker will need to build an entire database of hash values for each different salt + password combination, effectively requiring the creation of a table with hundreds of billions of records to crack a single password.

Make sure a different random salt is added to each password: otherwise a single DB of salted hash values can be created for your application – it won’t be useful for other apps, but if someone wants to target your app they can definitely achieve their goals.

As an example, let’s assume a user who’s password is ‘inigo2001′. Here is how this user’s password will be stored in the DB without salting:

 +-------------------+------------------------------------------+
 | user              | password_hash                            |
 +-------------------+------------------------------------------+
 | inigo@montoya.com | e40900c950cc6011297b2b392b42c29688b33ac7 |
 +-------------------+------------------------------------------+

An attacker with a good dictionary can figure out that the password_hash value is in fact the SHA-1 digest of “inigo2001″. However, if we add salt:

 +-------------------+------------------------------------------+------------------------------+
 | user              | password_hash                            | password_salt                |
 +-------------------+------------------------------------------+------------------------------+
 | inigo@montoya.com | fe66f3eb9c0afc8c935dc9f3f26dbea68d48ccc1 | 9ljYI+xMaVOSloDwt9ahzTpqMHA= |
 +-------------------+------------------------------------------+------------------------------+

Guessing that password_hash is the SHA-1 digest of “indigo20019ljYI+xMaVOSloDwt9ahzTpqMHA=” is quite hard – one would need to build a huge dictionary just to figure out this one password, assuming they also have insight into our code and have figured out that we have concatenated the password_salt value after the original password value and passed that through SHA-1.

Note that the password_salt value in this case is a base-64 encoded string of 20 random bytes – using a random enough and long enough salt is important, otherwise there’s a good chance your password + salt value happens to already exist in the attacker’s DB.

Example Time

To summarize things, here is an actual example of a few PHP functions that store user information in the DB in a secure manner and verify passwords against that stored information.

Our users table in the database is assumed to look something like:

mysql> DESCRIBE users;
+---------------+---------------------+------+-----+---------+----------------+
| Field         | Type                | Null | Key | Default | Extra          |
+---------------+---------------------+------+-----+---------+----------------+
| id            | int(10) unsigned    | NO   | PRI | NULL    | auto_increment |
| email         | varchar(50)         | NO   | UNI | NULL    |                |
| password      | char(40)            | NO   |     | NULL    |                |
| password_salt | binary(16)          | YES  |     | NULL    |                |
+---------------+---------------------+------+-----+---------+----------------+

The password field is a 40 byte long CHAR (SHA-1 hashes in hexadecimal representation are always 40 byte long). The password_salt field is a 16 byte BINARY field – it will contain some random bytes with no particular encoding so it shouldn’t be a CHAR or VARCHAR field.

As new users register, the following functions are used to set the user’s password in the DB:

class User
{
  /**
   * This will contain a hashed version of the user's password
   *
   * @var string
   */
  protected $password = null;

  /**
   * This will contain the salt value used to add noise to the password hash
   *
   * @var string
   */
  protected $password_salt = null;

  /**
   * Set the user's password
   *
   * @param string $password
   */
  public function setPassword($password)
  {
    // Test that password is at least 6 characters mixing letters and digits
    if (! preg_match('/^.*(?=.{6,})(?=.*[a-z])(?=.*[A-Z])(?=.*\d).*$/')) {
        throw new \ErrorException("Password is not strong enough");
    }

    $this->password_salt = $this->generateRandomSalt();
    $this->password = sha1($password . $this->password_salt);
  }

  /**
   * Generate a random salt value, 16 bytes long
   *
   * This relies on OpenSSL being available. If it is not available, any
   * cryptographic-grade random string generation function would work. On
   * UNIX machines, you can just read 16 bytes from /dev/urandom.
   *
   * @return string
   */
  protected function generateRandomSalt()
  {
    return openssl_random_pseudo_bytes(16);
  }
}

To check a given password, we add the following function to the same class (assume that the protected values are populated from values fetched from the DB):

  /**
   * Check if password is correct
   *
   * @param  string $password
   * @return boolean
   */
  public function checkPassword($password)
  {
    $hashed = sha1($password . $this->password_salt);
    return ($hashed === $this->password);
  }

As you can see, this class (assuming a working database access layer) will do the work of properly salting and hashing passwords before saving them, and of comparing given clear-text passwords to a salted, hashed value stored in the DB. This practically ensures stealing passwords from you is near impossible.

What’s next?

I hope that this article pointed out some good practices in storing passwords in the DB. It is important to remember that while your site may not be very interesting to hack into, hacking into your users’ accounts could be a first step towards identity theft or the hijacking of accounts on another site holding much more sensitive data. Implementing the measures described here would mean you are at least treating your users’ password with the right care.

There are additional aspects to password security which you should look into: using proper security on the transport channel when asking for passwords (HTTPS with a valid certificate), requiring strong enough passwords from your users, avoiding session fixation attacks (session_regenerate_id at login) and more. I also did not touch procedures of replacing lost passwords, which are also a common weak point vulnerable for phishing attacks. There is quite a lot of material to read out there on these topics, and if I see an interest is raised I might cover some of them myself in the future.

Go with the Flow: PHP’s Userspace Streams API

Almost every PHP application out there needs to read data from files or write data to files – or things that look like files but are not quite files – these unstructured blobs of data are commonly referred to as “streams”. Stream functions allow a scalable, portable and memory efficient way to handle data, and pretty much any PHP developer out there knows how to read data from or write data to a steam. The best part is that you don’t have to be an extension author in order to provide access to any data source as if it was just a regular file. PHP’s userspace streams API allows you do to exactly that, and this article will show you how.

If you’re a subscriber, feel free to read the article and send me your feedback. If not, go ahead an buy the issue

First, know that I was lucky: for this method to properly work, you need a few things:

• The machine must be EBS based
• You need to be able to afford a couple of minutes of downtime
• You need to be able to withstand the effects of restarting the machine – for example, if you do not have an Elastic IP address associated with the machine, its public address will change. In some situations this is not acceptable.

After trying some different approaches, what worked for me was to do the following:

1. Generate a new keypair for yourself, and import the public key to your EC2 account
2. Start a new, clean, cheap machine (this will only be needed to do very simple things, so I recommend using a tiny machine) in the same availability zone as the affected machine
3. Stop the affected machine (do not terminate, STOP it – this is only possible with EBS machines)
4. Detach the root device from the affected machine (by default attached as /dev/sda1)
5. Attach the detached device to the new clean machine
6. SSH into the clean machine and mount the affected machine’s root filesystem somewhere (e.g. in /mnt/fs)
7. Now you can edit /mnt/fs/root/.ssh/authorized_keys (or on official Ubuntu machines /home/ubuntu/.ssh/authorized_keys) and add your new public key to it
8. Unmount the volume and terminate the clean machine – you no longer need it
9. Re-attach the root device to the affected machine (which should be stopped) – ensure to attach it as the same device it was before (e.g. /dev/sda1)
10. Re-start your old machine – you should now be able to use your new key!

Another approach which could work but I gave up on after a couple of attempts (I think it really depends on the init scripts in the machine you are using), is to stop the machine and change the User Data of it to a shell script that sets a new public key in the right place, then start it again.

And really, you should backup your keys!