Better serialization of Groovy objects using XStream

This blog post is to resolve a little disconnect between Groovy and XStream serialization library:

• Groovy makes use of synthetic members quite a bit, and sometimes it also adds synthetic fields to classes it compiles.
• XStream skips only static and transient data members (as of the latest version 1.4.1), but not synthetic ones, resulting in a little inconvenience that Groovy’s compiler-provided synthetic members also show up in serialized form.

Here is an example, where we try to serialize a Groovy object into JSON format:

import groovy.transform.Immutable
import com.thoughtworks.xstream.XStream
import com.thoughtworks.xstream.io.json.JsonHierarchicalStreamDriver

@Grab(group='com.thoughtworks.xstream', module='xstream', version='1.4.1')
XStream xstream = new XStream(new JsonHierarchicalStreamDriver())

def person = new Person(firstName: 'roshan', lastName: 'dawrani')
println xstream.toXML(person)

@Immutable
class Person {
	String firstName
	String lastName
}

The above code outputs the following because AST transformation done by compiler for @Immutable adds some extra fields for its internal use, such as “$print$names” and “$hash$code”, and it can be a little confusing or annoying to see these unknown data members mixed up with your regular ones.

{"Person": {
  "firstName": "roshan",
  "lastName": "dawrani",
  "$print$names": true,
  "$hash$code": 0
}}

In the following code, we try to remove the disconnect between Groovy and XStream by introducing a custom converter that enhances the filter applied by XStream on data members and excludes the synthetic ones as well:

import groovy.transform.Immutable

import java.lang.reflect.Field
import java.lang.reflect.Modifier

import com.thoughtworks.xstream.XStream
import com.thoughtworks.xstream.io.json.JsonHierarchicalStreamDriver
import com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider
import com.thoughtworks.xstream.converters.reflection.ReflectionConverter
import com.thoughtworks.xstream.mapper.Mapper

@Grab(group='com.thoughtworks.xstream', module='xstream', version='1.4.1')
XStream xstream = new XStream(new JsonHierarchicalStreamDriver())
xstream.registerConverter(new GroovyObjectConverter(xstream.mapper))

def person = new Person(firstName: 'roshan', lastName: 'dawrani')
println xstream.toXML(person)

@Immutable
class Person {
    String firstName
    String lastName
}

class GroovyObjectConverter extends ReflectionConverter {
    GroovyObjectConverter(Mapper mapper) {
        super(mapper, new GroovyObjectReflectionProvider())
    }

    boolean canConvert(Class type) {
        GroovyObject.class.isAssignableFrom(type)
    }

}

class GroovyObjectReflectionProvider extends PureJavaReflectionProvider {
    protected boolean fieldModifiersSupported(Field field) {
        int modifiers = field.getModifiers()
        super.fieldModifiersSupported(field) && !Modifier.isSynthetic(modifiers)
    }
}

With this technique the output is correctly shown as:

{"Person": {
  "firstName": "roshan",
  "lastName": "dawrani"
}}

Hope the technique is useful to some of you.

Making a Hudson /Jenkins build number available to a Grails application

It’s a quick note on how to make a Hudson / Jenkins build number available to a Grails app.

• Jenkins / Hudson make the current build number available through the environment variable “BUILD_NUMBER”.
• Grails exposes at runtime the application meta-data from the file “application.properties”
• Grails fires a pre-WAR-creation event called “CreateWarStart” that can be utilized to make the running Jenkins/Hudson build number available through Grails application level meta-data.

Below is the Grails event handler that connects the dots above when you build your Grails app’s WAR (scripts/_Events.groovy):

eventCreateWarStart = { warName, stagingDir ->
    def buildNumber = System.getenv('BUILD_NUMBER')
    if(buildNumber) {
        ant.propertyfile(file:"${stagingDir}/WEB-INF/classes/application.properties") {
            entry(key:'build.number', value: buildNumber)
        }
    }
}

This build number can be retrieved at run-time using any of the following:

• In GSP pages

<g:meta name="build.number"/>

• In Groovy code

def buildNumber = grailsApplication.metadata['build.number']

You can learn more about hooking into Grails events here.

Groovy Tip: Accessing Pattern Matcher instance Used in Switch Case Statements

Groovy allows you to use regular expression patterns in the switch case statements

doMatch '2'
doMatch 'a1'
 
def doMatch(s) {
    switch(s) {
        case ~/\d/:
            println 'a digit'
            break
        case ~/.*\d/:
            println 'ends with a digit'
    }
}

But what if you needed access to the underlying matcher instance that was used to do the pattern matching. Is there any magic symbol (‘$’, ‘#’, ‘$$’, ‘##’, ….) that gives you handle to that matcher in Groovy?

The short answer is: No.

The longer answer is: Yes, and here is how you can do it:

import java.util.regex.Matcher
 
def s = 'tracking_level_9'
 
switch(s) {
    case ~/tracking_level_(\d)/:
        def m = Matcher.lastMatcher
        println m[0][1]
        assert '9' == m[0][1]
}

Hope it helps some time.

Grails: Functional Testing a File Upload using HTTPBuilder / Spock

Let’s continue a bit more on the previous post and see how can we do functional testing of the same file-upload feature using HTTPBuilder and Spock libraries.

It needs to be a functional test and not an integration test, because for Grails integration tests, there is no real HTTP layer that comes in the picture to separate the HTTP client requests from your Grails application on the server. Functional tests allow us to hit the application to do the file upload, as normal users of the application would – going through the various layers of the application, verifying on the way various other things like security, URL mappings, etc, etc. How much more real and useful, the testing can get!

Although HTTPBuilder usually makes such testing a piece of cake, when it comes to making multipart/form-data requests, there is no direct support in it I could find. After searching here and there, I found that it is, eventually, fairly easy to plug-in a multipart entity in a HTTPBuilder post request, as shown in the example below.

Note: You need to additionally use the HTTPMime module of Apache HTTPComponents project.

import static groovyx.net.http.Method.POST

import groovyx.net.http.HTTPBuilder

import javax.servlet.http.HttpServletResponse

import org.apache.http.entity.mime.MultipartEntity
import org.apache.http.entity.mime.content.FileBody

import spock.lang.Specification

class FileUploadSpec extends Specification
{
    def baseUrl = 'http://localhost:8080/<yourapp>' // your app URL
    
    def 'upload image'()
    {
        when: 'uploading image'
        def imageType = 'image/jpeg'
        def imgFile = new File('<path to some existing image file>')
        assert imgFile.exists()

        def cbFile = new FileBody(imgFile, imageType)

        def http = new HTTPBuilder(baseUrl)
        
        http.auth.basic '<login id>', '<password>' // login credentials
        
        resp = http.request(POST) { req ->
            uri.path = 'user' // controller that recieves the file upload post request
            
            def mpEntity = new MultipartEntity()
            mpEntity.addPart("profileImg", cbFile)
            
            req.entity = mpEntity
        }
        
        then: 'there was no problem faced'
        resp.status == HttpServletResponse.SC_OK
        ...
        ...
    }
}

That’s it. Simple, isn’t it?

Grails: Mock Testing a File Upload

Here is a quick note on how to mock test a file upload feature in a Grails application (credits: Luke Daley). A little trick is needed here because by default in the unit tests, Grails will only give you an instance of MockHttpServletRequest as the request and that does not have any support for multipart/form-data.

But, hey, you are using the ultra-dynamic Groovy language underneath. How difficult could it be to turn Grails’ arm a bit to make it give you a multipart http request instead (MockMultipartHttpServletRequest)!

Here is how you can do it:

import org.springframework.mock.web.MockMultipartFile
import org.springframework.mock.web.MockMultipartHttpServletRequest

class FileUploadControllerTests extends grails.test.ControllerUnitTestCase
{
    ....
    ....
    void testFileUpload()
    {
        def imgContentType = 'image/jpeg'
        def imgContentBytes = '123' as byte[]
        ....
        ....
        controller.metaClass.request = new MockMultipartHttpServletRequest()
        controller.request.addFile(
            new MockMultipartFile('image', 'myImage.jpg', imgContentType, imgContentBytes)
        )
        ....
        controller.save()

        assertEquals HttpServletResponse.SC_OK, controller.response.status
        ....
    }
}

Grails: Custom Clean-up During Testing Using Build Events

For integration and functional tests, Grails ensures that each test gets a clean database and is not affected by the data changes made in other tests. Grails achieves it by executing each test within a transaction and rolling it back after the test is executed – so no commits actually make way to the database, effectively giving every test a clean slate.

But, what if your requirements differ from those in a standard Grails app – what if the database you want to use in your Grails application does not support transactions, or what if you want to do some non-DB clean-up before each test – then Grails mechanism does not work and it becomes your responsibility to do the clean-up, as you want.

Coming up with clean-up logic is fine, but the real question is, how to plug it in?

1) One way would be to change each of your test case and pollute it as below:

package foo

class MyControllerIntTests extends GroovyTestCase
{
    void setUp()
    {
        super.setUp()
        MyDataCleaner.resetData() /* apply the data clean-up logic */
    }
    
    void testA() { .... }       

    void testB() { .... }       
}

The above approach works, but soon becomes a maintenance headache, because for each new test case, you now have to remember to make this customary clean-up call.

2) Another approach would be to introduce a base class and make the clean-up call in its setUp(), as below:

package foo

class MyBaseIntTests extends GroovyTestCase
{
    void setUp()
    {
        super.setUp()
        MyDataCleaner.resetData() /* apply the data clean-up logic */
    }
    ...
    ...
}

class MyControllerIntTests extends MyBaseIntTests
{
    void setUp()
    {
        super.setUp()
    }
    
    void testA() { .... }       

    void testB() { .... }       
}

But this approach comes with its own issues as your tests may have to extend other special classes like grails.plugin.spock.IntegrationSpec, if your integration tests are Spock based or grails.plugin.geb.GebSpec, if you are functional tests are Geb based – you can’t easily introduce a common base class if your tests have a useful mix of such frameworks.

3) A much cleaner option is to hook into build events fired by Grails framework. Grails fires following testing related events that can be utilized for our clean-up purposes:

• TestPhaseStart
• TestCaseStart
• TestStart
• TestEnd
• TestCaseEnd
• TestPhaseEnd

The following piece of code can be put in _Events.groovy to hook into the events of interest and invoke the data clean-up logic.

boolean inIntegrationTestPhase = false

eventTestPhaseStart = { name ->
    inIntegrationTestPhase = (name == 'integration')
}

// reset the data in app DB
eventTestStart = { name ->
    if(inIntegrationTestPhase) {
        println "------------ Cleaning data before running the test '$name' ------------"
        MyDataCleaner.resetData()
    }
}

Now no need to maintain individual data clean-up calls from each test case, or to pollute the test cases’ inheritance hierarchy – just pick and choose when you want your data to be cleaned and do it in a transparent manner.

easyb BDD framework – isA(“simple Groovy DSL”)

Came to know y’day that easyb – a BDD (Behaviour Driven Development) framework – is groovy based and got eager to see how it is implemented, what groovy techniques is its DSL based on (does it also use AST transformation like Spock?), how does it make the groovy compiler support its *.story scripts, how does it support the following structure in its tests, etc?

scenario "any kind of scenario description", {
    given "describe the initial test state", { ... }
    when "actions that change things", { ... }
    then "post action verification", { ... }
}

The actual implementation, in terms of groovy features it uses, turned out to be simple-ish – no fancy AST transforms used, no tricky configuration of groovy compiler to make it understand that a *.story file contains a groovy script.

Easyb supports *Story.groovy and *.story files as the BDD scripts. When you run the easyb tests as below, it picks up all files ending with *Story.groovy and *.story from the list of scipts it is fed, reads their text and feeds that text to the groovy compiler as plain String. So as far as groovy compiler is concerned, it sees no stories.

java -cp classes;easyb-0.9.8.jar;lib\commons-cli-1.2.jar;lib\groovy-all-1.7.5.jar org.easyb.BehaviorRunner <somepath>\AccountsTest.story <somepath>\TransactionStory.groovy

Then comes the DSL part of it.

Nothing fancy here as well. Every story file that easyb processes gets a new Binding with a number of closures in the binding, for ex, scenario, given, when, then, and, but, before_each, after_each, etc. So, when you execute the following story:

scenario "money is withdrawn from an account 1 containing sufficient funds", {
    given "an account with 100 dollars", {
        account1 = new Account(100);
    }
    and "another account with 200 dollars", {
        account2 = new Account(200);
    }
    when "account holder withdrawls 20 dollars from account 1 and 40 dollars from account 2", {
        account1.withdrawl 20
        account2.withdrawl 40
    }
    then "account 1 should have 80 dollars left", {
        account1.balance.shouldBe 80
    }
    and "account 2 should have 160 dollars left", {
        account2.balance.shouldBe 160
    }
}
scenario "money is withdrawn from an account with insufficient funds", {
    given "an account with 10 dollars", {
        account = new Account(10);
    }
    when "withdraw more than the account has", {
        withdrawl = {account.withdrawl 20}
    }
    then "account holder should be notified about the insufficient funds", {
        ensureThrows(RuntimeException){
            withdrawl()
        }
    }
    and "then account should still have 10 dollars", {
        account.balance.shouldBe 10
        account.balance.shouldBeA(Integer)
    }
}

then easyb records the above test as 2 main test steps –

• Scenario 1 closure – having 5 child steps [2-given, 1-when, 2-then closures], and
• Scenario 2 closure – having 4 child steps [1-given, 1-when, 2-then closures]

Execution of the above story script results in all the closures getting stored in the script binding and then the same are played back (scenario closures are executed in sequence with their child closures executed in sequence). Data defined in “given” steps gets stored as script binding variables and is accessible to further steps like “when”, “then”, and “and”.

Last bit that remains is the following

account.balance.shouldBe 10
account.balance.shouldBeAn Integer

This is simply achieved through the use of groovy categories feature. All the story steps are executed in the scope of BehaviorCategory, which enhances java.lang.Object with the operations like following: shouldBe, shouldBeGreaterThan, shouldBeLessThan, shouldBeEqual, shouldntBe, shouldBeA, shouldNotBeA, shouldNotHave, etc, which means that in easyb stories, these operations can be called on any type of object whatsoever.

Just some regular groovy features seem to be enough for this BDD framework – script bindings, closures, categories, regular scripts compilation.

easy(b), isn’t it?

Groovy (New feature): Closures can now memo(r)ize their results

Groovy recently added a new feature to its closures – memoization – a simple yet very useful feature that can drastically improve the performance when applicable.

It was a shame that I didn’t know the meaning of this term that was coined way back in 1968 by Donald Mitchie (ref: http://en.wikipedia.org/wiki/Memoization) [But now that I have memoized it, next time I am asked what it is, I will reply faster! ]

So, all this new feature means is that a groovy closure can now remember the result of its invocation for a specific set of inputs – by internally caching the result. So, the first invocation for a given set of inputs does the actual work, then the result gets cached behind the scenes and any further invocations with the same inputs simply return the cached result, making the invocation as fast as it can be. Of-course, this feature should be used only when closure processing is such that a set of inputs will ALWAYS result in the same output – it MUST be side-effect-free.

Here is an example to illustrate the use. Let’s first see it without memoization:

cl = {a, b ->
    sleep(3000) // simulate some time consuming processing
    a + b
}

def callClosure(a, b) {
    def start = System.currentTimeMillis()

    cl(a, b)

    println "Inputs(a = $a, b = $b) - took ${System.currentTimeMillis() - start} msecs."
}

callClosure(1, 2)
callClosure(1, 2)
callClosure(2, 3)
callClosure(2, 3)
callClosure(3, 4)
callClosure(3, 4)

Here is the output showing that everytime the time cost of closure invocation in incurred:

Inputs(a = 1, b = 2) – took 3041 msecs.
Inputs(a = 1, b = 2) – took 3000 msecs.
Inputs(a = 2, b = 3) – took 3000 msecs.
Inputs(a = 2, b = 3) – took 3000 msecs.
Inputs(a = 3, b = 4) – took 3000 msecs.
Inputs(a = 3, b = 4) – took 3000 msecs.

Here is the memoized version:

cl = {a, b ->
    sleep(3000) // simulate some time consuming processing
    a + b
}
mem = cl.memoize()

def callClosure(a, b) {
    def start = System.currentTimeMillis()

    mem(a, b)

    println "Inputs(a = $a, b = $b) - took ${System.currentTimeMillis() - start} msecs."
}

callClosure(1, 2)
callClosure(1, 2)
callClosure(2, 3)
callClosure(2, 3)
callClosure(3, 4)
callClosure(3, 4)

callClosure(1, 2)
callClosure(2, 3)
callClosure(3, 4)

Here is the output showing how only the first invocation incurs the processing cost and after that the calls with the same inputs are almost free:

Inputs(a = 1, b = 2) – took 3054 msecs.
Inputs(a = 1, b = 2) – took 0 msecs.
Inputs(a = 2, b = 3) – took 3001 msecs.
Inputs(a = 2, b = 3) – took 0 msecs.
Inputs(a = 3, b = 4) – took 3000 msecs.
Inputs(a = 3, b = 4) – took 0 msecs.
Inputs(a = 1, b = 2) – took 0 msecs.
Inputs(a = 2, b = 3) – took 0 msecs.
Inputs(a = 3, b = 4) – took 0 msecs.

The trade-off involved is that you pay for time saving in terms of extra memory consumed by the internally cached inputs/outputs mappings. If you want to have more control over how many results you want to cache, other variants to memoize() are – memoizeAtMost(n), memoizeAtLeast(n), memoizeBetween(m, n)

To quickly see how memoizeAtMost(n) works, in the previous example, let’s replace mem = cl.memoize() by mem = cl.memoizeAtMost(2) – and the output changes as below (now only 2 results are cached, with the LRU ones evicted):

Inputs(a = 1, b = 2) – took 3051 msecs.
Inputs(a = 1, b = 2) – took 0 msecs. /* 1st result – cached! */
Inputs(a = 2, b = 3) – took 3000 msecs.
Inputs(a = 2, b = 3) – took 0 msecs. /* 2nd result – cached !! */
Inputs(a = 3, b = 4) – took 3001 msecs.
/* 3rd result – cached – but now (a = 1, b = 2) result evicted – on LRU basis */
Inputs(a = 3, b = 4) – took 0 msecs.
/* (a=1, b=2) got evicted, processing happens again and result cached, now evicting (a=2, b=3) result */
Inputs(a = 1, b = 2) – took 3000 msecs.
/* (a=2, b=3) got evicted, processing happens again and result cached, now evicting (a=3, b=4) result */
Inputs(a = 2, b = 3) – took 3000 msecs.
/* (a=3, b=4) got evicted, processing happens again */
Inputs(a = 3, b = 4) – took 3000 msecs.

Good feature, isn’t it?

Note: The new feature is only available from groovy version 1.8-beta-3 onwards.

Groovy++ Goodies (1) – Closure Level Fields

Groovy++ allows you to declare closure level fields using its new @Field annotation.

In core groovy, if you need to remember some state across multiple calls to the same closure instance, you need to do it using variable declared outside the closure scope. So, I see this Groovy++ feature as a useful goodie.

Below is some sample code to demonstrate the use of closure level fields:

@Typed void squares() {
    (1..20).each {
        @Field boolean lastOneSqaure = false
        @Field int which = 0
        
        if(lastOneSqaure) println "$it) Last number($which) was a square"
        
        double d = Math.sqrt(it)
        if((d - ((int) d) == 0)) {
            lastOneSqaure = true
            which = it
        } else {
            lastOneSqaure = false
        }
    }
}

squares()

The above code correctly outputs

2) Last number(1) was a square
5) Last number(4) was a square
10) Last number(9) was a square
17) Last number(16) was a square

The groovy++ compiler makes sure that the variables marked with @Field are declared as fields in the closure class and not as local variables – allowing you to preserve state across multiple invocations to a particular closure instance without needing any out-of-closure-scope assistance.

Added support to Groovy Compiler to allow source files with extensions of your choice

I am happy to finally push the change to groovy compiler so it allows users to mix and match the groovy source with multiple extensions of their choice – http://tinyurl.com/376f6mj.

For example, Groovy++ has *.gpp and *.grunit files. You may want to have *.g or *.gscript, or *.gdsl or *.gspec, or whatever else – and have a mix of any number of them.

The feature is currently available in 1.7.5 and 1.8.0-beta-2 snapshots, which are available from here –

• http://snapshots.repository.codehaus.org/org/codehaus/groovy/groovy-all/1.7.5-SNAPSHOT/.
• http://snapshots.repository.codehaus.org/org/codehaus/groovy/groovy-all/1.8.0-beta-2-SNAPSHOT/

All that needs to be done to use this feature is to have your extensions listed down in the file META-INF/services/org.codehaus.groovy.source.Extensions and make it available on the classpath and the groovy compiler will do the rest.

Here is a sample of how the file may look like:

# Format: one extension on each line without the leading “*.”
gpp
gscript
gdsl

Feedback welcome.