Reuse your tdd skills to build an application cluster with vagrant and chef: iteration 1

Lately, at the office, my team and I were willing to improve our release process and reach Continuous Delivery state.

So automating our application’s infrastructure (1 database, 2 appservers, 1 search engine, 1 proxy) meant:

  • automating the database migration: index, column, constraints, table, view, data, etc,
  • automating the webapp deployment: the easy part,
  • automating the search engine migration: aliases, indices, mappings, full reindex, partial reindex.
  • automating the proxy configuration: register/unregister members.

When you add a Zero Downtime (french blog entry with english references) constraint you’re basically left with 2 choices:

  • act on the existing cluster: migrate database, then index, then sequentially migrate the webapps. This technique is known as the Blue Green Deployment. This technique works as soon as versions N+1 and version N are fully compatible. You must maintain at least 2 versions of your code until version N is no longer used. Only then will you be able to cleanup what’s remaining of the version N in your code.
  • act on a brand new cluster: reproduce an identical cluster, populate the new database and the new index with the current data, then switch to the new cluster (Immutable server à la Netflix). A little easier as you don’t have to maintain 2 versions of codebase. You still need to write search engine and database migration scripts.

While the first method is used pretty often, the second requires more recent tools and skills (mostly ops ones) making most dev teams uncomfortable with them. Should they really be? Being able to reproduce  identical environments at will really looked like magic to me just 3 weeks ago (I have a strong java background). I got dragged into the huge but totaly addictive automation and configuration management ecosystem. I chose to give Chef and Vagrant a try to build the above cluster while using my TDD knownledge.

Continue reading “Reuse your tdd skills to build an application cluster with vagrant and chef: iteration 1”

Spice-up your application: add elasticsearch geo feature

Lately I’ve been busy working on elasticsearch features for my company.
In the process I came accross the shiny “geo search” feature. While not being that sensitive to shiny and new technologies (don’t get me wrong, I don’l like dusty ones) I still wanted to test elasticsearch geo capabilities for further adoption.
The reference documentation on the subject is a post from Shay Banon on the elasticsearch website. Geo search is made possible by indexing coordinates that conform to “geo_point” structure, then sort by “_geo_distance” or filter by “geo_distance“.
The other very useful resource was that post from Gauthier Lemoine‘s blog . The author’s app could find the nearest stations to the Eiffel Tour.
The the example is clear and includes index feed from the ratp’s open data, then a search based on the Eiffel Tour coordinates. It is written in python.

I added a geocoding capability that allows a user to provide any location which is a little more convenient. I also tried to use maven’s exec plugin to setup/feed the index based on the file available on data.ratp.fr which contains the complete list of paris stations: trains, subways and bus.
The maven build drops/creates the index, generates stations data and bulk insert them, starts the webapp then searches against a provided location like “35 avenue daumesnil, 75012 paris”.

Let’s take a look at the relevant parts of the solution
Continue reading “Spice-up your application: add elasticsearch geo feature”

Safely re-index with elasticsearch

I’ve been using Elasticsearch in production for about 1 year. The component does a very good job at indexing/searching but lacks a built-in solution for continuity of service: you can’t hold incoming write request an resume them at will.

In my team, we started thinking about a home-made solution. We really wanted to keep our webapp as robust as possible and avoid to much sub-system dependencies such as queues, webservices, etc. Elasticsearch was already a big one, from an ops perspective (mainly because it’s young). We also really needed the interruption-free feature because we could not afford even a minute of service interruption.

After some time we had to face it: with no built-in solution around we had to build one and guess what? We could no more avoid queues because they really are robust component and are a perfect match for a {pause/resume|consumer/producer} paradigm.

Continue reading “Safely re-index with elasticsearch”

Selenium from scratch

Hi reader,

If you’re concerned with testing then you’ve already came accross the difficult task of UI testing.
We all agree that fat client model is different from thin client’s but they share common characteristics: screens and transitions between screens. Testing them is different but seems equaly difficult. We will cover thin UI ie web site pages testing.
Before thinking about the “how”, we’ll think of the “what”.
Every UI is a set of screens (we pages in our case). Pages display data in UI components. Displaying data as a result of an action is something we might want to test:

Given some persisted messages
When I navigate to the "list messages" page
Then the UI should display the persisted messages

Given I input the "create message" form with invalid phone number
When I submit the "create message" form
Then the UI should display "Invalid phone number" error

The other very important behaviour to test is the transitions between screens.
In an HTTP context, requesting a resource with the “GET” method might not be allowed. In a security context, accessing a resource might not be allowed at all. From the UI perspective preventing a transition from screen A to B often translates into removing/hidding a link/button from the page:

Given I provide "admin/admin" authentication
When I navigate to the "catalog"
Then the page should display the "delete" link

Given I provide "user/user" authentication
When I navigate to the "catalog"
Then the page should not display the "delete" link

We will test 2 transitions: a successful form submission and a failure.
Continue reading “Selenium from scratch”