Friday, September 13, 2013

More clever builds and tests ?

Hey,

when you have already tried to build RHQ and run all the tests you probably have seen that this may take a huge amount of time, which is fine for the first build, but later when you e.g. change a typo in the as4-plugin, you don’t want GWT being compiled again.
Of course as a human developer it is relatively easy to just go into the right folder and only run the build there.

Now on build automation like Jenkins this is less easy, which is why I am writing
this post.

What I want to have is similar to

  • if a class in GWT has changed, only re-build GWT

  • if a class in a plugin has changed, only rebuild and test that plugin
    (and perhaps dependent plugins like hibernate on top of jmx)

  • if a class in enterprise changes, only build and test enterprise

  • if a class in server/jar changes, only rebuild that and run server-itests

  • if a class in core changes, rebuild everything (there may be more fine grained rules as e.g. a change in core/plugin-container does not require compiling GWT again)

This is probably a bit abbreviated, but you get the idea.

What I can imagine is that we compile the whole project (actually we may even do incremental compiles to get build times further down and may also only go into a specific module (and deps) and just build those).
And then instead of running mvn install we run mvn install -DskipTests
and afterwards analyze what has changed, throw the above rules
at it and only run the tests in the respective module(s).

We could perhaps have a little DSL like (which would live in the root of the project tree and be in git)

    rules: {
rule1: {
if :modules/enterprise/,
then: {
compile: modules/enterprise,
skip: modules/enterprise/gui/coregui,
test: ["modules/enterprise"]
}
},
rule2: {
if: modules/plugins/as7-plugin,
then: {
compile:[ modules/plugins/as7-plugin,
modules/integration-tests/as7-plugin],
test: [ modules/plugins/as7-plugin,
modules/integration-tests/as7-plugin]
}
}

And have them evaluated in a clever way by some maven build helper
that parses that file and also the list of changes since the last build to
figure out what needs testing.

We can still run a full build with everything to make sure that we don’t loose coverage
by those abbreviations

There may be build systems like gradle that have this capability built in; I think for maven
this requires some additional tooling

QUESTIONS:

  • Are there any "canned" solutions available?

  • Has anyone already done something like "partial tests" (and how)?

  • Anyone knows of maven plugins that can help here?

  • Anyone interested in getting that going? This is for sure not only interesting for RHQ but for a lot of projects

Having such an infrastructure will also help us in the future to better integrate
external patches, as faster builds / tests can allow to automatically test those and
"pre-approve" them.




4 comments:

Lukáš Krejčí said...

We could reuse maven deps for the rules on when to recompile/test something.

A change in core/util would force a recompile and retest of virtually everything, while a change in AS7 plugin would cause no other modules to recompile.

I.e. if there is a change in a module then recompile and retest all the modules that depend on it.

freon said...

During my 2011 GSoC I worked on this project and they had their itests divided into 4 different mvn modules (see the link above) and it was possible to turn on/off the "i-test sets" by mvn profiles.

I remember, some profiles tested SSL authentication and there was no need to run them if I did a change somewhere else.

It may work for us as well.. to prepare a set of scenarios like:
* change in coregui
* change in domain
* change in server jar
etc..

And control the build by activating particular mvn profiles.

Thomas Heute said...

I agree with freon, the easiest to maintain is to have a few Maven profiles, it's not perfect but that goes a long way (+ modularize the platform).

Heiko Rupp said...

I am sure the plugins can be used to determine sets of what is to be compiled tested.

That still leaves the question who decides what profile to use in Jenkins? That still requires logic to see "oh core/domain was changed, I need to rebuild all". Or "oh only a typo in the UI was fixed, so I do not need to compile/test the server again".

Moving plugins into their own repo would at least make this solve for commits to plugins, but is only part of the equation.