Replacing a lost SSH key on an Amazon EC2 machine

Due to an unfortunate shmelting accident (read: poor backup practices), I lost the SSH private key granting me the only way to access one of my EC2 hosted servers. Being unable to access the server, and unable to easily set a new public key through Amazon’s interfaces, I panicked for a few seconds. Then I started trying to hack my way in, and eventually found a way to set a new public key to my user. Here is what I did.

First, know that I was lucky: for this method to properly work, you need a few things:

  • The machine must be EBS based
  • You need to be able to afford a couple of minutes of downtime
  • You need to be able to withstand the effects of restarting the machine – for example, if you do not have an Elastic IP address associated with the machine, its public address will change. In some situations this is not acceptable.

After trying some different approaches, what worked for me was to do the following:

  1. Generate a new keypair for yourself, and import the public key to your EC2 account
  2. Start a new, clean, cheap machine (this will only be needed to do very simple things, so I recommend using a tiny machine) in the same availability zone as the affected machine
  3. Stop the affected machine (do not terminate, STOP it – this is only possible with EBS machines)
  4. Detach the root device from the affected machine (by default attached as /dev/sda1)
  5. Attach the detached device to the new clean machine
  6. SSH into the clean machine and mount the affected machine’s root filesystem somewhere (e.g. in /mnt/fs)
  7. Now you can edit /mnt/fs/root/.ssh/authorized_keys (or on official Ubuntu machines /home/ubuntu/.ssh/authorized_keys) and add your new public key to it
  8. Unmount the volume and terminate the clean machine – you no longer need it
  9. Re-attach the root device to the affected machine (which should be stopped) – ensure to attach it as the same device it was before (e.g. /dev/sda1)
  10. Re-start your old machine – you should now be able to use your new key!

Another approach which could work but I gave up on after a couple of attempts (I think it really depends on the init scripts in the machine you are using), is to stop the machine and change the User Data of it to a shell script that sets a new public key in the right place, then start it again.

And really, you should backup your keys!

Bitbucket: Converting Hg repositories to Git

Recently I started using Bitbucket for private repository hosting for a project I’m working on. While I had no experience with Mercurial, I figured it can’t be that tricky – and Bitbucket offers free private hosting which is what this project needed (couldn’t go public, couldn’t pay, didn’t have the time to set up self-hosted SCM hosting).

All in all I like Bitbucket (although I have to admit on most aspects they seem to fall behind GitHub), but not so much using Mercurial – for all sorts of reasons it felt quirky and less polished than Git, which honestly I have much more experience with.

So following Bitbucket’s big announcement on Git support, I’ve decided to migrate my repositories from Hg to Git, while keeping them on Bitbucket and maintaining repository history. I’m happy to say it was relatively a piece of cake to successfully achieve. Here is what I did:

Step 1: Set up your repositories

First, I renamed my old Hg repository through Mercurial’s web interface, to something like “MyProject” to “MyProject Hg”. This changes the repository URL, but since I wasn’t planning on using it anymore that doesn’t really matters – plus you can always rename back if things go bad.

Then, I created a new Git repository with the name of the previous repository, e.g. “MyProject”. Again, that can be easily done from Bitbucket’s Web interface.

Step 2: Install the Mercurial hggit plugin

The hggit plugin allows your Mercurial command line tool hg to talk to git repositories – that is push and pull from Git. Installing it is easy, as it is probably available from your package manager. On Mac, if you use Macports, you can run:

  $ sudo port install py26-hggit

While on Ubuntu, run:

  $ sudo aptitude install mercurial-git

Then, make sure to load the plugin by adding the following lines to your ~/.hgrc file:


Congratulations: Your hg command now speaks Git!

Step 3: Push your code into your Git repository

To push your code into your new Git repository, you basically need to run two commands:

First, create a Mercurial bookmark that references master to your default branch. This will help Git create the right refs later on:

  $ cd ~/myproject-hg-repo/
  $ hg bookmark -r default master

Next, simply push your code into the newly created Git repository:

  $ hg push git+ssh://

Of course, make sure to change the repository URL to the URL of your new Git repository. To make sure hg understands you’re referring to a Git repository, if using SSH add the git+ssh:// prefix to the URL. This should push your entire repository to the new Git repository, and within a few seconds up to a few minutes (depending on how big your repository is), you should be able to see all your old commits in the new Git repo.

Step 4: Switch your local repository to use Git

Now that your new Git repo is up at Bitbucket, you’ll need to switch to using Git locally. There are two paths you can take here: the safe one, is to simply git clone your code to a new working directory and work from there. It’s safe, and will work well. However, if you’re a cowboy like me, and are too lazy to create a new IDE project on a different directory, you can in fact simply switch to working with Git on the same directory (but I still seriously recommend you ensure Bitbucket really does have your code as backup…).

Here is how to do it. From the local repository directory, run:

  $ git init
  $ git remote add origin
  $ git pull origin master
  $ git reset --hard HEAD

Again, replace the repository URL with your own. This will “merge” everything in your Git repo into the local working directory. You will need to create a new .gitignore file if needed – and can now simply delete the .hg directory, as it is no longer needed. You can now happily use Git with your Bitbucket code.

While there shouldn’t be any problems, I also recommend keeping your old Hg repository around on Bitbucket for a few days, just to make sure nothing blows up – and delete it from Bitbucket’s web interface once you’re sure everything works well.

Imagick: Maintain (fake) transparency when saving as JPEG

I haven’t blogged in a while (have been busy you know), so I’ve decided to share this small piece of knowledge I’ve obtained by experimenting. I wrote a small test app (it’s for a feature of the next version of Zend Server – maybe I’ll share it one day when the API is stable), which does some image manipulation with the ImageMagick extension.

For those of you who don’t know ImageMagick allows one to preform pretty cool stuff on images – except for the usual drawing, conversion, rotation, rescaling etc., it also exposes some API to easily preform neat effects, like drop shadow, round corners and my newest favorite (apparently only available in the very latest builds of the extension) – the Polaroid effect.

In his blog Mikko Koppanen, the author of the ImageMagick PHP extension, shows how to create drop shadows (as well as other neat things – you should check out his blog!), but in his examples Mikko will always save as PNG, which is something I dare to say most web users will not do, and prefer saving as JPEG.

Problem with many of those effects, is that they leave parts of the image as transparent. When saving the picture as JPEG (as I do, since saving as PNG produces too big files), these transparent areas appear as black.

So after some experimenting, I’ve found out that the way to work around this is to composite another opaque layer as your background layer, filled with your background color of choice (white in my case). You will of course loose the ability to place the picture on other background colors and still have a nice “transparency” look – but as long as you stick to the background color you’ve set, it will look great.

Here is a code sample producing the same thumbnail + drop shadow as in Mikko’s example, but saving it with white matte color as JPEG:


$bgColor = '#ffffff'; // End result will have a white background

/* This was taken from Mikko's example */
$im = new Imagick( 'strawberry.png' );
$im->thumbnailImage( 200, null );
$im->roundCorners( 5, 5 );

$shadow = $im->clone();
$shadow->setImageBackgroundColor( new ImagickPixel( 'black' ) );
$shadow->shadowImage( 80, 3, 5, 5 );
$shadow->compositeImage( $im, Imagick::COMPOSITE_OVER, 0, 0 );

/* My addition: clone the entire image again to create the background layer */
$bg = $shadow->clone();

/* I'm using colorFloodFiilImage with high tolerance to paint it all white - maybe there are 'cleaner' ways to do it though */
$bg->colorFloodFillImage($bgColor, 100, '#777777', 0, 0);
$bg->compositeImage($shadow, Imagick::COMPOSITE_OVER, 0, 0);

/* Display the image */
header( "Content-Type: image/jpeg" );
echo $bg;

While there’s another step in the way, and the image will only look good on white backgrounds, you can now save it as a JPEG file with good compression and acceptable file size.

Subversion: Finding the “base” revision of a branch

I use Subversion a lot – but today I’ve learned something new:

You can easily find the “base” revision of a branch or a tag (i.e. the revision in which the branch or tag was created) by issuing the following command:

svn log -v --stop-on-copy

The last revision you see in the log (in this case from one of my own Glista project’s branches) is the revision the svn copy command was issued on, i.e. the branch was made.

This can then be used when merging the same branch back into trunk.