bits and pieces

September 30, 2011

Grails, Cassandra: Giving each test a clean DB to work with

Filed under: Uncategorized — Tags: , — roshandawrani @ 1:26 pm

If you are using Grails, you must be used to the ease with which your integration tests do not have to bother about what other tests are doing in the application database. Grails achieves it by starting a new transaction before each test and rolls it back when the test gets over – whether it passes or fails. So, the actual changes made to data are never written to the database and you are happy to have this isolation between tests, as it allows you to focus one each test independently instead of worrying about side-effects.

But what if the database you used did not support transactions? How do you achieve the same isolation then?

For such a database – Cassandra – it can be achieved by cleaning up after each integration test the data from all the Column Families (tables) that the Keyspace (schema) had (Ability to hook into Grails events made it quite easy). Not every test makes too many changes in the database, so it can work out quite OK – obviously, not as fast as Hibernate + RDBMS combo, but acceptable.

The steps I used:

  • Describe the keyspace and find out all the column families in it
  • For each column family you want to clean-up data from:
    • Do range queries on the column family, skipping any phantom records found (Use Pagination, if needed)
    • Collect all valid row keys
    • Do a batched mutation for all the row keys to clean-up the column family data

So, all worked well, but it seemed like too much clean-up work. I looked around and found that Cassandra supported a “truncate” operation, which was sadly missing in Hector API, so I promptly requested for the API gap to be filled. The “truncate” operation was soon made available in Hector API and it cut down our clean-up code from 20-lines of doing range-scans-and-picking-up-keys-to-delete to one-liner “truncate” call.

Over time, the schema grew (more column families, indexes) as well as the application (more integration tests needing more clean-up cycles) and the integration tests that used to finish in seconds started taking many minutes. For some time I doubted that maybe Cassandra upgrades have introduced some new configuration that should be tweaked, but then I measured. To my utmost surprise, it showed that each invocation of the one-liner “truncate” call had started spending 4-5 seconds kind of time in Hector + Cassandra, and in an overall integration tests cycle of 5.5 minutes, 5 minutes were in clean-up and 0.5 in actual tests :-)

We immediately switched back to olden ways of doing range-scans-and-batch-mutations and here is the difference in overall clean-up time (nearly 75 invocations in the whole integration test phase): 5 minutes vs 0.05 minutes! So, we are happily back to our tests taking 0.55 minute instead of 5.5 minutes!

Please, please do not use Hector + Cassandra’s “truncate” calls frequently to provide your tests a clean DB slate in your Grails / Java applications! Its performance is bad!

Also, sometimes 20 lines of code can be better than a one-liner. :-)

September 13, 2011

Better serialization of Groovy objects using XStream

Filed under: Uncategorized — Tags: — roshandawrani @ 1:24 pm

This blog post is to resolve a little disconnect between Groovy and XStream serialization library:

  • Groovy makes use of synthetic members quite a bit, and sometimes it also adds synthetic fields to classes it compiles.
  • XStream skips only static and transient data members (as of the latest version 1.4.1), but not synthetic ones, resulting in a little inconvenience that Groovy’s compiler-provided synthetic members also show up in serialized form.

Here is an example, where we try to serialize a Groovy object into JSON format:

import groovy.transform.Immutable
import com.thoughtworks.xstream.XStream
import com.thoughtworks.xstream.io.json.JsonHierarchicalStreamDriver

@Grab(group='com.thoughtworks.xstream', module='xstream', version='1.4.1')
XStream xstream = new XStream(new JsonHierarchicalStreamDriver())

def person = new Person(firstName: 'roshan', lastName: 'dawrani')
println xstream.toXML(person)

@Immutable
class Person {
	String firstName
	String lastName
}

The above code outputs the following because AST transformation done by compiler for @Immutable adds some extra fields for its internal use, such as “$print$names” and “$hash$code”, and it can be a little confusing or annoying to see these unknown data members mixed up with your regular ones.

{"Person": {
  "firstName": "roshan",
  "lastName": "dawrani",
  "$print$names": true,
  "$hash$code": 0
}}

In the following code, we try to remove the disconnect between Groovy and XStream by introducing a custom converter that enhances the filter applied by XStream on data members and excludes the synthetic ones as well:

import groovy.transform.Immutable

import java.lang.reflect.Field
import java.lang.reflect.Modifier

import com.thoughtworks.xstream.XStream
import com.thoughtworks.xstream.io.json.JsonHierarchicalStreamDriver
import com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider
import com.thoughtworks.xstream.converters.reflection.ReflectionConverter
import com.thoughtworks.xstream.mapper.Mapper

@Grab(group='com.thoughtworks.xstream', module='xstream', version='1.4.1')
XStream xstream = new XStream(new JsonHierarchicalStreamDriver())
xstream.registerConverter(new GroovyObjectConverter(xstream.mapper))

def person = new Person(firstName: 'roshan', lastName: 'dawrani')
println xstream.toXML(person)

@Immutable
class Person {
    String firstName
    String lastName
}

class GroovyObjectConverter extends ReflectionConverter {
    GroovyObjectConverter(Mapper mapper) {
        super(mapper, new GroovyObjectReflectionProvider())
    }

    boolean canConvert(Class type) {
        GroovyObject.class.isAssignableFrom(type)
    }

}

class GroovyObjectReflectionProvider extends PureJavaReflectionProvider {
    protected boolean fieldModifiersSupported(Field field) {
        int modifiers = field.getModifiers()
        super.fieldModifiersSupported(field) && !Modifier.isSynthetic(modifiers)
    }
}

With this technique the output is correctly shown as:

{"Person": {
  "firstName": "roshan",
  "lastName": "dawrani"
}}

Hope the technique is useful to some of you.

September 4, 2011

Making a Hudson /Jenkins build number available to a Grails application

Filed under: Uncategorized — Tags: , , , — roshandawrani @ 1:29 pm

It’s a quick note on how to make a Hudson / Jenkins build number available to a Grails app.

  • Jenkins / Hudson make the current build number available through the environment variable “BUILD_NUMBER”.
  • Grails exposes at runtime the application meta-data from the file “application.properties”
  • Grails fires a pre-WAR-creation event called “CreateWarStart” that can be utilized to make the running Jenkins/Hudson build number available through Grails application level meta-data.

Below is the Grails event handler that connects the dots above when you build your Grails app’s WAR (scripts/_Events.groovy):

eventCreateWarStart = { warName, stagingDir ->
    def buildNumber = System.getenv('BUILD_NUMBER')
    if(buildNumber) {
        ant.propertyfile(file:"${stagingDir}/WEB-INF/classes/application.properties") {
            entry(key:'build.number', value: buildNumber)
        }
    }
}

This build number can be retrieved at run-time using any of the following:

  • In GSP pages
  • <g:meta name="build.number"/>
    
  • In Groovy code
  • def buildNumber = grailsApplication.metadata['build.number']
    

You can learn more about hooking into Grails events here.

The Silver is the New Black Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: