Spice-up your application: add elasticsearch geo feature

Lately I’ve been busy working on elasticsearch features for my company.
In the process I came accross the shiny “geo search” feature. While not being that sensitive to shiny and new technologies (don’t get me wrong, I don’l like dusty ones) I still wanted to test elasticsearch geo capabilities for further adoption.
The reference documentation on the subject is a post from Shay Banon on the elasticsearch website. Geo search is made possible by indexing coordinates that conform to “geo_point” structure, then sort by “_geo_distance” or filter by “geo_distance“.
The other very useful resource was that post from Gauthier Lemoine‘s blog . The author’s app could find the nearest stations to the Eiffel Tour.
The the example is clear and includes index feed from the ratp’s open data, then a search based on the Eiffel Tour coordinates. It is written in python.

I added a geocoding capability that allows a user to provide any location which is a little more convenient. I also tried to use maven’s exec plugin to setup/feed the index based on the file available on data.ratp.fr which contains the complete list of paris stations: trains, subways and bus.
The maven build drops/creates the index, generates stations data and bulk insert them, starts the webapp then searches against a provided location like “35 avenue daumesnil, 75012 paris”.

Let’s take a look at the relevant parts of the solution…

Safely re-index with elasticsearch

I’ve been using Elasticsearch in production for about 1 year. The component does a very good job at indexing/searching but lacks a built-in solution for continuity of service: you can’t hold incoming write request an resume them at will.

In my team, we started thinking about a home-made solution. We really wanted to keep our webapp as robust as possible and avoid to much sub-system dependencies such as queues, webservices, etc. Elasticsearch was already a big one, from an ops perspective (mainly because it’s young). We also really needed the interruption-free feature because we could not afford even a minute of service interruption.

After some time we had to face it: with no built-in solution around we had to build one and guess what? We could no more avoid queues because they really are robust component and are a perfect match for a {pause/resume|consumer/producer} paradigm.

Selenium from scratch

Hi reader,

If you’re concerned with testing then you’ve already came accross the difficult task of UI testing.
We all agree that fat client model is different from thin client’s but they share common characteristics: screens and transitions between screens. Testing them is different but seems equaly difficult. We will cover thin UI ie web site pages testing.
Before thinking about the “how”, we’ll think of the “what”.
Every UI is a set of screens (we pages in our case). Pages display data in UI components. Displaying data as a result of an action is something we might want to test:

Given some persisted messages
When I navigate to the "list messages" page
Then the UI should display the persisted messages

Given I input the "create message" form with invalid phone number
When I submit the "create message" form
Then the UI should display "Invalid phone number" error

The other very important behaviour to test is the transitions between screens.
In an HTTP context, requesting a resource with the “GET” method might not be allowed. In a security context, accessing a resource might not be allowed at all. From the UI perspective preventing a transition from screen A to B often translates into removing/hidding a link/button from the page:

Given I provide "admin/admin" authentication
When I navigate to the "catalog"
Then the page should display the "delete" link

Given I provide "user/user" authentication
When I navigate to the "catalog"
Then the page should not display the "delete" link

We will test 2 transitions: a successful form submission and a failure.

Android from scratch, part 2: use android-maven-plugin

Hi people,

I wasn’t sure to write again on Android because the first time I tried the platform I identified 3 knowledge levels:
– academic: knowing what Android is without ever building anything on that platform;
– basic: reading a lot about the topic and sorting information to finally extract the minimum/essential (there are a lot of incomplete/inaccurate information that make them unusable ‘as is’) required to build and deploy an app on a device (cf previous post);
– professional: involves build skills (stable, repeatable, customized per env), developement skills (fat client knowledge is a plus), release/distribution skills (branding, publishing on Google Play, etc)

That last step is mean for professionnals but I’ll try to present the building part. With the right tools we can reach a pretty nice result.

Android from scratch: the basics

Hi people,

When the Open Handset Alliance decided to make the Android Platform available I was happy: I could already see myself building crazy mobile apps.
Well since then I have been hit hard by reality: before producing any working piece of code on my I had to get familiar with that new ecosystem.

Android ecosystem is very wide because it covers various aspects from installing the development platform to deploying publicly on Google Play®.
The official documentation is complete thus quite dense. Yet knowing everything is not required when one want to get quick visual result.
The purpose of the post is to help one deploying an application on an Android device (be it virtual or real). A simplified deployment process could be:
- download and install the SDK
- create a virtual device or plug a real one
- create a project
- build the project
- deploy the project to your device
- enjoy your application
This simple workflow is functionnal but misses more advanced steps like testing with elaborate tools (lint, monkey, etc), preparing artefact for production (obfuscation, zipalign, signing), plublishing artefact on Google Play® (prepare marketing assets, etc).
In this post we’ll focus on the simple workflow. I may write another post on the advanced one.

DISCLAIMER: I was not couragous enough to decline the example for windows systems. All samples are in bash.

RESTful pagination with cucumber-jvm, jersey and elasticsearch

Hi all,

It’s been a while since my last post. There are good reasons for that. I had many topics in mind :
- RESTful pagination. This one already hides 2 wide concepts : pagination and HATEOAS.
- Cucumber. I already know 2 Cucumber implementations : Java and Ruby. It’s not trivial to choose when provided with that much options.
I ended up building a pagination system with cucumber-jvm as acceptance test framework, jersey as REST framework, jetty as container, in-memory h2 database and in-memory elasticsearch searchengine.

Spring managed event listeners with JPA

Hi readers,

I wanted to set things right. I’ve  been approximative in a previous post. I wrote an incomplete assertion: “using spring-managed listeners with JPA is not possible“. I want to clarify things:

- using spring managed listeners with JPA is not possible if you absolutly want to use JPA interface and ignore the underlying implementation,

- if your implementation is hibernate, and depending on hibernate version, your solution may be more or less elegant.

Add search features to your application, try Elasticsearch part 4 : search

Elasticsearch relies on Lucene engine.
The good news is that Lucene is really fast and powerful. Yet, it’s not a good idea to expose such power to the user. Elasticsearch acts as a first filter but remains quite complete.
When you don’t master an API, a good practice is to have control over what you expose to the user. But this comes with a cost, you’ll have to:
- implement a query language
- implement a language parser
- implement a query translator (translate into something understandable by elasticsearch)
- run search
- translate Elasticsearch results into a custom structure

The task seems daunting but no worry: we’re going to take a look at each step.

Add search features to your application, try Elasticsearch part 3 : attaching indexation to events

Now that we are able to index, we should think of when whe should trigger indexing tasks.

A simple answer would be : whenever some indexed data has changed. “Changed” means change cardinality (add/remove) or change existing data.

Either we invoke indexing tasks whenever we code an action that changes data or we use an event model which listens to precise events.

1 – JPA event model

If you use JPA as a persistence mechanism you can take advantage of its elegant mechanism. You can register an entity’s listeners either at class level or at method level via annotations.
One can annotate an entity method as a listner to an event. The method is the will be executed when the event defined by the annotation occurs.
If this solution seems too intrusive or too specific, one can externalize this behaviour in a class and annotate the entity.

Add search features to your application, try Elasticsearch part 2 : start small

Hi,

This article is part of a whole which aims to describe how one could integrate Elasticsearch. The previous post discussed the concepts: why using a search engine?.

No people learn the same way. I usually need to understand the theory and I need to start small.
So I usually start by books. If there is no book I look for a blog that explains the philosophy. Finally I look for pieces of codes (official documentation, blogs, tutorials, github, etc). Starting small makes me confident and allows me to increase complexity gradually.

Depending on what you want to know about Elasticsearch you should read different section of the guide :

- SETUP section : describes how to install and run Elasticsearch (run as a service).
- API : this is the REST API which seems more complete than the others. Describes how to inter-operate with nodes : search, index, check clusters status.
- Query DSL : the query API is quite rich. You have explanations about the syntax and semantics of queries and filters.
- Mapping : mapping configures elasticsearch when indexing/searching on a particular type of document. Mapping is an important part which deserves special care.
- Modules : presents the technical architecture with low level services like discovery or http.
- Index modules : low level configuration on indices like sharding and logging.
- River : the river concept is the ability to feed your index from another datasource (pull data every X ms).
- Java and groovy API : if your software already runs in a jvm you can benefit from that and control elastic search via this API.