Development

Getting HTTPS going on your local POW instance using Nginx



Security. We’ve all heard and read about it. But many of us forget to do something about it, until something nasty happens. In Oct 2010, the blogosphere erupted with the demonstration of Firesheep pointing out that one of the most basic things any web developer can do to improve their application security is to use HTTPS.

For a Rails developer, isn’t this just a matter of adding a gem to your Gemfile, and running bundler?

If only it were that simple.

For this post, I’ll go through some of the steps that I took to get HTTPS going on a Rails application running locally under POW. Hopefully, it’ll be of use to somebody.

What’s POW?

So you have a rails application, which you typically fire up using:

POW

What happens if you have multiple rails applications? Or for that matter sinatra, merb or any other Rack-based application? Do you fire them all up with separate ports?

Here’s where POW comes in really handy. POW! is a zero-config rack server for Mac OS X. It’s dead easy to install and get running, and makes it possible to host multiple Ruby Rack applications without each Rack application having to run in a separate web server on a separate port.

POWer me up!

To install POW, run the following in the terminal:

Here’s a trap that I found. The documentation for POW typically tell you to download and install POW with one command, namely; curl piped to sh. However, in my experience, this did not (for some reason) configure OS X’s system resolver properly so requests were not being forwarded to POW. I had to specify the Top Level Domains (TLD) in a config file and reinstall POW. So, save yourself the trouble and set it first with the POW_DOMAINS variable in the .powconfig file before you install.

Keep the POWer on

When you start using POW, you may notice that some refreshes take a really long time to come back. This happens because POW tends to spin down the rails instance after a timeout, making it painful when you refresh your page as this causes POW to reinitialise the rails application from scratch (and we all know how long rails 3 takes to initialise). To fix this, edit your ~/.powconfig to include the following:

The POW_TIMEOUT sets POW to keep the process running for a long time, which is typically what you want, given that in development, rails will automatically reload (mostly) the altered files.

So now, you can throw away the rails s command as POW will fire up the instance when you need it.

How Do I Bolt On HTTPS To POW?

Nginx Logo
AFAIK – you can’t. I looked, and I couldn’t find any way to install a certificate to get HTTPS going. So, the easiest way to bolt on HTTPS is to setup Nginx as a reverse proxy. And although it sounds hard, it isn’t.

Nginx is fast HTTP server. And like any capable HTTP server, it is able to accept HTTPS connections and can be configured to proxy requests through to POW.

Bolt In Nginx

To install Nginx on OS X, you have two choices:

  1. Install it via Brew
  2. Install it the old fashion way

Brew me an Nginx

The easy way is to use Brew. Just run

Done.

I Don’t Have Brew

If you’re like me, and you’re already using MacPorts and haven’t made the switch to brew, then you have to do things the hard way. I followed instructions on Kevin Worthington’s Blog but specifically, I had to do the following:

  1. Install XCode(I already had XCode installed previously so I didn’t have to do anything). Installing XCode is a matter of downloading it and installing it. In fact, if you’re running Snow Leopard or Lion, you can download it and install it directly via the Mac App Store.
  2. Install PCREThis required me to download the source and install it manually. The steps I took were as follows:

  3. Install NginxInstalling Nginx is a matter of downloading the source, building and installing it. I followed the following steps:

Intermission. Time For Certificates.

Ok. Time for a break. Time to break out openssl to generate some certificates.

HTTPS sits on top of Secure Sockets Layer (SSL). For this to work, the encrypted communication requires certificates to be installed at the web server end. In production, you would purchase the certificate from a Certificate Authority (CA) like Verisign or Digicert. The CA’s role is to issue you with a certificate that verifies the identity of the entity the certificate is for (e.g. the domain name owner).

But for development, you don’t need a certificate from a CA. All you need is openssl to generate a certificate which is self-signed. There are countless articles on the web describing this process in detail. www.akadia.com’s article seems to be pretty self explanatory.

Once you’re done generating the certificates, you will need to either copy or symlink them to the appropriate Nginx location so that Nginx can find them. To do this, I did the following:

I Have Nginx. Now to configure it.

If you followed my install instructions above, then Nginx’s conf file is located in

You will need to edit the nginx.conf file to configure Nginx to:

  1. Use the certificates you generated
  2. Listen to port 443 to handle the HTTPS protocol

Here’s what I added to the nginx.conf file to configure Nginx as a reverse proxy.

[text]
server {
listen 443 ssl;
server_name myapp.dev;

ssl on;
ssl_certificate ssl/server.crt;
ssl_certificate_key ssl/server.key;

keepalive_timeout 60;

ssl_session_timeout 5m;

ssl_protocols SSLv2 SSLv3 TLSv1;
ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
ssl_prefer_server_ciphers on;

location / {
proxy_pass http://127.0.0.1;
### force timeouts if one of backend is died ##
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;

### Set headers
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

### Most PHP, Python, Rails, Java Apps can use this header
proxy_set_header X-Forwarded-Proto https;

### By default we don’t want to redirect it
proxy_redirect off;
}
}
[/text]

The things that you may need to alter are:

  • server_name – enter the domain that matches what you would enter to hit POW
  • ssl_certificate – enter the location of your server.crt file. If you copied the certificates
    to /usr/local/conf/ssl, then enter ssl/server.crt since the
    nginx.conf resides in /usr/local/conf
  • ssl_certificate_key - enter the location of your server.key file. This should be similar
    to that for ssl_certificate

The rest of the configuration tells Nginx to proxy the request to localhost on port 80, which is where POW will
kick in and handle the request for you.

Configured!

Once you're done, fire up Nginx using /usr/local/sbin/nginx and test it by navigating to https://myapp.dev.

"You gotta blog about this man..." cried Alan after I checked in. In acknowledgement, I nodded. Fast forward four weeks later, after CrowdHired has launched... I now have some breathing space to really blog about it. So hopefully this has been of help to you and saved you some time along the way.

Tips To Optimise Your Software Builds

So, you’ve gotten into the TDD/BDD swing of things, and you’re creating specs all over the shop… in a systematic way (of course). Now you’ve built up a suite of tests, and everything’s going well… things are green.

One important thing to start realising on a green-field project is to always take time to improve. And one area that developers should always focus on is the build.

The importance of builds

Builds are an important safeguard in any project. On most projects, developers typically run builds before they checkin. Indeed, it is good practice to make sure everybody gets into this habit. If your team takes a “don’t care” attitude, checking in regardless of whether the continuous integration server is green, this can quickly lead to massive development pain.

So, let’s assume that your team is building before checkins. Great.

Pay attention to your build times

If you’ve been on a sizeable project (duration, scope, people), most people suffer from bloated builds. Sometimes they take 2 hours to turn around, from commit to deploy. When that situation occurs, most of us start loathing the build.

Which is why on any new initiative, you may start out with every intention to minimise build times. But when do you start factoring this activity in?

In my opinion, as soon as you can.

Why? Because you will start to learn something new. You will begin to learn what makes the build take too long, and start rolling in steps to stop them from occurring in the future. You will also learn what testing methods are slow, and therefore take the effort to use them only when needed.

And the sooner you do this, the faster your team will be in addressing these problems. It’s much easier to address these problems earlier (when you have 50 tests) rather than later (when you have 5000 tests, and your build is taking 1 hour).

Learning is the bottleneck

Many lean/agile books will promote the idea that “Learning is the bottleneck”. Note that this isn’t coding. If it was coding, then we should be optimising our practices to code faster. The problem here is that coding faster is entirely predicated on how quickly your team learns and adapts. So focus on giving your team the learning opportunity it needs to avoid pitfalls and problems.

And a key to that is learning how to write good tests in the environment. Learning to make sure that the feedback you get is appropriate and quick, which drives the next cycle of learning.

Case study: 161 tests taking 5 sec

On a recent project I am on, it was a new initiative. So I had built up a stead suite of specs. Thing is, I realised that the 161 tests I had was starting to take a little too long ~ 5 seconds.

Now 5 seconds for a build isn’t long at all by any means. You could put up with it. But I know, that on an large project in the same environment, they ran 9,000 or more tests within 45 seconds. If you do the math, that’s 200 tests per second.

Here, I was taking 5 seconds for 161 tests. If I project it out, at 9000 tests, that would take me 80 hours to complete!

Step 1: Profile your tests

The first thing you should do, is profile your tests, and get an idea of the tests that are taking the longest. With a small number of tests, this is easier to find out, and you can quickly focus on a cluster of tests that are doing similar things. Most unit testing libraries have some means of timing how fast tests take, so getting this information shouldn’t be difficult.

Step 2: Experiment, and validate

Then, experiment. Think about what could be causing the issue. Try something, and re-profile the test. Make sure that what you’re doing leads to an improvement. Don’t just guess and hack. You need to validate as well.

Step 3: Commonize and disseminate

Once you have a “fix”, codify it. Write it into a helper method/class and refactor your tests that have the problem to use the new solution. Then socialise it. Announce it at a standup. Send out an email. Talk about it.

This last step is really important. The time you have invested into learning and fixing an issue is now going to help you many times over. By codifying it, you’re encoding the knowledge into the code repository, which will stick around long after you have left the project. Codifying it also helps others out, so they don’t have to walk into the same pit-hole you did.

Socialising it helps your team establish a general guideline around how tests should be written/approached. In may mean that that particular kind of test isn’t warranted under all circumstances. Establishing a baseline in your team for how this is approached is key to ensuring you have the right kind of tests.

Summary

Here’s the 3 point summary:

  1. Pay attention to your build times, even with a few tests.
  2. Aim for a 200 tests per second (or better) test/rate
  3. Profile, experiment, validate, codify and disseminate the information

What are your test rates? Can you beat 200 tests per second?

RVM, Mac OSX 10.6 and Ruby 1.8.7 p302 Install Error

RVM LogoWell… it’s amazing how fast things move on in the Ruby community. I haven’t been doing much Ruby work until recently and discovered my Ruby interpreter was out of date. Looks like the latest 1.8.7 is p302.

With RVM, it’s a piece of cake to upgrade your ruby interpreter. And this is no different.

Step 1: Upgrade rvm
The first important step is to tell rvm to update to the latest release. This is dead easy to do.

That’s it!. Updating rvm is important as it updates rvm’s knowledge of what is latest and greatest for each ruby version.

Step 2: Install 1.8.7
You might be thinking the next most obvious step is to simply run

Upon running this, you’ll find the following appears:

If you then check make.error.log, you’ll see this

It’s about this point you’re blue sky, easy going install has just come crashing down in a heap. After some digging, it looks like the problem is to do with compiling the ruby runtime on the 10.6 platform. It doesn’t seem to know where the readline library is, which is used by irb and other parts of the ruby platform.

On 10.6, readline is typically found at /usr/local/include/readline. If you don’t have it installed, you can find further instructions here on how to install readline 0.6 for 10.6 from source.

Unlike Tim’s post, I’m going to explain what you need to do to tell rvm to tell the ruby configure tool where to look for readline. You won’t need to move anything or do any special thing. Just simply pass in an argument.

The magic command is…

The –with-arch simply tells the configure tool to compile it as an x86_64 platform. Since I’m running on one of the latest and greatest Mac hardware, this is the right setting. The next argument tells the configure tool where readline is.

Once you run this, you’ll see this

That’s it!. No magic. No need to alter the source that was downloaded. No need to move files. All those hacks you simply can do without, and just pass in the right path and everything works as advertised.

%CD% – the pwd for cmd/dos on Windows

cmd promptHave you ever wondered what the cmd equivalent for the unix command pwd?

To those unfamiliar with pwd, it’s a UNIX shell command you type on a terminal screen to get the Present Working Directory (pwd). Why is this useful? Well it can be used as a shortcut for operations on the current directory you’re in. It’s also useful when it comes down to calling shell scripts and passing directory refences around to properly locate other files that you expect in specific locations relative to the present working directory.

So what’s the cmd equivalent?

Open Windows Explorer
Ever been stuck in the terminal screen and wanted to open Windows Explorer in the current director? This command will save you multiple mouse clicks and help you visualise the directory quickly.

If you run explorer on the command prompt, it opens Windows Explorer but doesn’t navigate to the spot where your directory is. I have found this to be a great annoyance and just not that useful. Passing in a directory will cause explorer to open that directory up. This is a little more useful, but the folder view in XP seems to be hidden. You have to click on folders to see something useful.

So, to address this, run this instead

The /e argument causes explorer to open in its default view. A comma is needed to delimit arguments to the explorer program. This opens the explorer in a much more familiar view. If you’re after more options, check out the following page on the command-line arguments for Explorer.exe.

Making Snow Leopard (10.6.3), PostgreSQL 8.4, RVM and TextMate Play Nice

Finder IconGreat, you just bought a new mac to replace your old, sluggish machine that keeps running out of space. Now comes the joy of migrating to a brand new spanking machine. These are the key steps that I took to migrate. In the midst of my journey, I didn’t find one consistent spot which had all the information I needed. So, to help those that may be doing what I did, here are my steps to get my tool stack back up and running….

Step 1: Use Migration Assistant

The mac tools make it easy to migrate from one machine to another and the tool to use is Migration Assistant. There is just one caveat – on your new machine, make sure you run Migration Assistant as part of the install process. This is incredibly important, as it allows the installer to copy the entire user from your old machine to the new – complete with .bash_profiles and .bashrc files and all. If you fail to do this, you can end up in a real mess. I did. In fact, I ran Migration Assistant, and tried to overwrite my current user – bad idea. I had to reinstall Snow Leopard.

Once you have selected the directories to move, let the tool take care of the rest. By in large, you will usually select most things. You may decide to omit Applications (as I did) and reinstall your applications from scratch. This is usually a better outcome.

Step 2: Install Applications

Now you have all your data, you need to reinstall your key applications (TextMate, Quicksilver, Firefox etc). This is fairly straightforward – and will test your memory to find all those license keys you need to reinstall each piece of software!

In this phase, don’t install Postgres using the DMG image as provided on the postgres site. The dmg contains a 32bit compiled version which will run on Snow Leopard but doesn’t work with trying to build libraries against it, since much of the building mechanisms require 64bit libraries.

Step 3: Install XCode 3.2.2

The XCode toolset is required to build any native software on the mac. It comes with gcc and other tools which makes building things from source a lot better. Make sure you get v3.2.2 since the version that ships on the Snow Leopard disk has a bug in it.

Head to http://developer.apple.com/technologies/xcode.html and click on Mac Dev Center -> XCode 3.2.2 and iPhone SDK 3.2. If the link sends you into a spin (it did for me), then go the iPhone Dev Center and click on the same link. You just need XCode 3.2.2. You will be required to register in order to download the SDK. Once you have it, install it.

Step 3: Re-install MacPorts and Install Postgres8.4

I had MacPorts running on my previous system. That won’t do you much good now if you’re coming from a 32bit environment. To remove MacPorts, run this:

This will remove the MacPort binaries. Head over to http://distfiles.macports.org/MacPorts/ to download the MackPorts-1.8.2-10.6-SnowLeopard.dmg file and install that. Then install the following:

This will download the dependencies, and the source files, compile and build the binaries appropriate to the current OS platform you are running. Installing wget gives you the similar tools on unix (it’s a curl equivalent). It also installs a number of useful dependencies. At the end of the install, you will need to run the following to complete the installation.

Create the postgres user (if you haven’t got one already) and run the following to setup the defaultdb for postgres:

Then you will need to install postgres as a service:

Edit your shell profile and append postgres binaries to your path. The binary files live here:
/opt/local/lib/postgresql84/bin
Now fire up postgres as a service by running:

To test postgres is install and running propertly, first, ensure that your current user has administrative privileges over postgres, run the following:

Then test that you can create a new database:

You should see positive confirmation that you could login to the test_db instance. Now drop the database:

The last statement should show that there is no such database called ‘test_db’.

Step 5: Install RVM
RVM is a tool which allows you to switch between ruby environments. I use to do my own hot switching using symlinks and shell commands, but this tool takes care of everything in a very neat package.

The RVM install is relatively straightforward. I chose to install it as a user, making it easier to remove if required. I followed the RVM installation instructions as provided, with the following notes:

  • I installed it as a user – to make it easier to remove later if required.
  • I had to alter my .bash_profile and just appended the command to source $HOME/.rvm/scripts/rvm

Step 6: Making TextMate use RVM
My editor of choice is TextMate, which has internal support for Ruby and for running Ruby Unit Tests. Most of this magic requires you to tell TextMate where your Ruby runtime is, and the rest works fine. This causes TextMate to require the right files, and you should then be able to run Ruby tests directly with TextMate using CMD-R.

However, TextMate wasn’t really “designed” to handle multiple Ruby runtimes easily. Thankfully, as of 0.1.45 of RVM, you can simply get TextMate to point TM_RUBY to a pre-specified script.

There were two key things that need to do to get TextMate to play nice:

  1. Set TM_RUBY to ~/.rvm/bin/rvm-auto-ruby
  2. Rename /Applications/TextMate.app/Contents/SharedSupport/Support/lib/Builder.rb to something else

These TextMate-RVM integration instructions provide the detail of what needs to be done.

After all that, you’re done! Snow Leopard, PostgreSQL, RVM and TextMate all under one roof.