Fixing logrotate errors and other MySQL issues on Ubuntu / Debian

For our MySQL databases on EC2, we back up the data by taking hourly snapshots of the volume that stores the MySQL data folder. This is a common practice, and in fact we do it using the popular ec2-consistent-snapshot script by Alestic. For backups this works great; in addition, having volume snapshots of MySQL data is very useful when we want to quickly launch a new MySQL server with a fairly recent state of our data – whether as a way to speed up synchronization of a new DB node or for testing purposes.

When launching a new EC2 instance we simply mount a new volume created from the latest DB snapshot on /var/lib/mysql and then start MySQL, and everything “just works” from there.

However, there are some quirks to this method: recently, I’ve encountered some errors from daily logrotate tasks which fail to flush the logs on a post-rotate script (this is from the logs of cron that runs logrotate):

Continue reading

Monitoring EC2 instance memory usage with CloudWatch

At Shoppimon we’ve been relying a lot on Amazon infrastructure – it may not be the most cost effective option for larger, more stable companies but for small start-ups that need to be very dynamic, can’t have high up-front costs and don’t have a large IT department its a great choice. We do try to keep our code clean from any vendor-specific APIs, but when it comes to infrastructure & operations management, AWS (with help from tools like Chef) has been great for us.

One of the AWS tools we use is CloudWatch – it allows us to monitor our infrastructure and get alerted when things go wrong. While its not the most flexible monitoring tool out there, it takes care of most of what we need right now and has the advantage of not needing to run an additional server and configure tools such as Nagios or Cacti. With its custom metrics feature, we can even send some app-level data and monitor it using the common Amazon set of tools.

However, there’s one big missing feature in CloudWatch: it doesn’t monitor your instance memory utilization. I suppose Amazon has all sorts of technical reasons not to provide this very important metric out of the box (probably related to the fact that their monitoring is done from outside the instance VM), but really if you need to monitor servers, in addition to CPU load and IO, memory utilization is one of the most important metrics to be aware of.

Continue reading

Quickly Creating a New Admin User on Ubuntu

Working quite a lot on the Ubuntu Server EC2 images, I am often faced with a need to create one or more additional admin users, which have the same permissions as the first user (“ubuntu” in these images). I did a little bit of searching but didn’t find any way to easily add a new user with the same groups as the Ubuntu user, so I crafted a little command. I’m pasting it here mostly for future self reference, and also in hope this helps someone:

sudo useradd -m -G `groups ubuntu | cut -d" " -f4- | sed 's/ /,/g'` -s/bin/bash newuser

Of course ‘ubuntu’ is the user you want to copy, and ‘newuser’ is the name of the new user.

Note that the new user will be in the admin group but will still require a password when using sudo (that’s because in the EC2 images ‘ubuntu’ is the only user with NOPASSWD privileges. I personally believe this is a good thing, but if you want you can always add NOPASSWD on the admin group in /etc/sudoers.


Replacing a lost SSH key on an Amazon EC2 machine

Due to an unfortunate shmelting accident (read: poor backup practices), I lost the SSH private key granting me the only way to access one of my EC2 hosted servers. Being unable to access the server, and unable to easily set a new public key through Amazon’s interfaces, I panicked for a few seconds. Then I started trying to hack my way in, and eventually found a way to set a new public key to my user. Here is what I did.

First, know that I was lucky: for this method to properly work, you need a few things:

  • The machine must be EBS based
  • You need to be able to afford a couple of minutes of downtime
  • You need to be able to withstand the effects of restarting the machine – for example, if you do not have an Elastic IP address associated with the machine, its public address will change. In some situations this is not acceptable.

After trying some different approaches, what worked for me was to do the following:

  1. Generate a new keypair for yourself, and import the public key to your EC2 account
  2. Start a new, clean, cheap machine (this will only be needed to do very simple things, so I recommend using a tiny machine) in the same availability zone as the affected machine
  3. Stop the affected machine (do not terminate, STOP it – this is only possible with EBS machines)
  4. Detach the root device from the affected machine (by default attached as /dev/sda1)
  5. Attach the detached device to the new clean machine
  6. SSH into the clean machine and mount the affected machine’s root filesystem somewhere (e.g. in /mnt/fs)
  7. Now you can edit /mnt/fs/root/.ssh/authorized_keys (or on official Ubuntu machines /home/ubuntu/.ssh/authorized_keys) and add your new public key to it
  8. Unmount the volume and terminate the clean machine – you no longer need it
  9. Re-attach the root device to the affected machine (which should be stopped) – ensure to attach it as the same device it was before (e.g. /dev/sda1)
  10. Re-start your old machine – you should now be able to use your new key!

Another approach which could work but I gave up on after a couple of attempts (I think it really depends on the init scripts in the machine you are using), is to stop the machine and change the User Data of it to a shell script that sets a new public key in the right place, then start it again.

And really, you should backup your keys!

Experimenting with Glista on OS X

I haven’t blogged in a while, probably because I was too busy. I’ve been working, started to take some university classes (Philosophy & Computer Science), and… I’m doing most of my work on Mac OS X now. Don’t worry, I’m still a Linux guy – but mostly for work purposes (and out of curiosity) I decided to ask Zend for a Macbook when my Thinkpad was starting to die.

Unfortunately the negative side effect of this is that I had to put Glista on hold – since I didn’t have a Gtk+ based desktop anymore there wasn’t much point in actively working on it.

However, in the last couple of days (following some patches that came in from ananasik, for whom I immediately gave commit access) my fingers started itching, and I decided to play with porting Glista to OS X – and found this project.

After some hours of tinkering, crashing, building, rebuilding and breaking things again, I now have a somewhat working (albeit ugly, and not so OS X friendly) working Application bundle running on my own 32 bit OS X 10.6:

Glista running on native OS X for the first time!


Glista in the Dock!

If you’re really up for it, you can get a Disk Image here.

You can also build it from source by checking out and doing the following:

  1. Make sure you have all the nescesary build environment (XCode is usually a good start!)
  2. Install all the gtk-osx tools and libraries including ige-mac-builder and gtk-quartz
  3. cd into the source directory and run (in a jhbuild shell after installing osx-gtk) ./configure –prefix=$PREFIX
  4. Note that some things do not work on OS X yet (or will never work) like libunique integration, gtk-spell, libnotify integration etc. – that’s normal for nowRun ‘make’, don’t (!!) run ‘make install’ (well you can, but there’s no need, you’ll just pollute your system
  5. cd into dist/mac/ and run ‘make dist-mac’. If everything is ok this should create in that directory.
  6. Move that .app into /Applications (or anywhere else) and enjoy!

So far, it looks like it’s going to be a long time before Glista will work smoothly on Mac – and most of it is because Gtk+ is not really that portable, and making it use OS-native widgets and rendering seems to be quite a challenge. I also don’t feel I know enough about the internals of Gtk+, Quartz or OS X in general in order to help with that effort – but who knows, maybe I’ll be able to help somehow?

BTW I’m not sure if that binary will work on anything but OS X 10.6 on Intel 32 bit. If you try, let me know!

NetworkManager: Auto-HTTP login to a Wifi network

One of the cafés in my area where I frequently drink / work requires you to pass through an annoying web page forcing you to agree to some terms before allowing you to access the Internet through their Wifi network. It’s free – but they still annoy you with this silly HTTP gateway. This is actually a frequent thing in Israel – most cafés offer free Wifi access, but some will require to to log-in nevertheless.

So today I figured out how to get NetworkManager to automatically work around this HTTP gateway for me whenever I connect to Arcaffe’s Wifi network. Since it’s super cool, and since I bet lots of people are annoyed by these sort of Wifi gateways, here’s how to do it:

Apparently, NetworkManager allows you to create special post-connect or post-disconnect scripts that are executed when a network interface is brought up or down. Here is what I did:

I Created the following script and saved it at /etc/NetworkManager/dispatcher.d/



if [ "x$1" = "x$IFACE" ] && [ "$2" = "up" ]; then
    # Figure out the wifi SSID
    SSID=$(/sbin/iwconfig $IFACE  | grep ESSID | cut -d: -f2 | sed 's/^s*"(.*)"s*$/1/')

    case "$SSID" in
           COOKIE='Cookie: JSESSIONID=uc54121j305s; cookies=true'

           curl -d "$DATA" -H "$COOKIE" -H "$REFERER" "$URL" > /tmp/arcaffe.last 2> /tmp/arcaffe.last.err

Don’t forget to make the file executable – I did it by running chmox +x /etc/NetworkManager/dispatcher.d/

Some things you should note:

  • “012-ArCaffe” is the ESSID of the network I’m logging in to. This of course work for ArCaffe in Israel, but you should change that with your network’s ESSID.
  • Replace the value of IFACE with the name of your wireless interface
  • $1, the first parametter passed by NetworkManager to the script, is the network interface that was just connected or disconnected
  • $2, the second parameter, is “up” or “down” – the status of the interface.
  • The code I have inside the case block is where the magic happens. In this case, I send an HTTP POST with the correct parameters, Cookie and Referer headers and URL. This causes ArCaffe’s gateway to log me in
  • I use curl – but I could have also used wget or any other tool to do the job
  • The -d flag sends the POST data, the -H flags set a header
  • I figured exactly what request to send using LiveHttpHeaders – but you can also use tcpflow or any other packet sniffing or HTTP sniffing tool
  • You can add more options to the ‘case’ statement for more networks that need that sort of treatment. With a little of bash-fu that should be no problem.

That’s it! Man I love Linux today :D

Notify me when emerge is done

As a Gentoo user, I frequently install new software or update existing packages using emerge. Unlike binary package managers, building packages from source using emerge takes time, and I prefer running it inside a detached screen session, because (a) if I close the session it continues in the background and (b) it actually runs faster when it doesn’t need to show all the compilation output in an X terminal.

This has an annoying side effect: I sometimes start a long update process and forget I did it. If things fail (or even if they succeed) I don’t know about it.

Today, I solved this problem using notify-send, the libnotify binary client. I’ve created a short shell script wrapper to emerge which sends me a notification once emerge finishes, along with the original command line arguments and exit status code (0 = ok, failure otherwise):



/usr/bin/emerge "$ARGS"
if [ "$STATUS" = "0" ]

/usr/bin/notify-send --expire-time=0 --urgency=$LEVEL "emerge finished" "Exit Code: $STATUS
Emerge args: $ARGS"

If you save this code in a file named emerge-notify (and of course remember to chmod +x this file) you could then do this (from a GNOME terminal or any other X terminal):

$ ./emerge-notify -puD world

And you’ll get a nice notification when it’s done:

"emerge finished" notification

Of course, you need libnotify the notify-send binary and dbus for this.

Glista 0.3 Released

Thanks to the very good holiday layout this year*, I finally got to release the next preview release of Glista – my super simple Gtk+ based to-do list manager for your desktop.

The major improvement in this release is category support. For a while, I didn’t want to add any features that will make the UI more complex than it is. Then I noticed that I tend to add ad-hoc categories to my tasks in order to sort them – that is instead of typing “Fix ZF bug #1234″ I type “ZF: Fix bug #1234″. This is a very natural way to organize your tasks for me, and I assume that it is for most people. So I decided to add category support by recognizing this colon-separated syntax and breaking any item typed in this way into a “cateogry: item” structure.
Continue reading