I’ve been asked recently to do a short overview of the automation and testing tools I’ve used on my current project. I’ve gotten a lot of time to play around with various test tools and configurations, as well as ways to automate the setup and configuration of a build system. We’ll be taking a quick look at what each tool provides us and list any pros/cons that I’ve run into.
Databases and tools
Flyway
What is it?
Flyway is a database versioning and migration tool. It gives us a standard way to build our database schema in a reproducible manner and thus helps us easily create reproducible integration tests. We supply it a list of folders with our scripts, and flyway executes the scripts for us and manages a “schema_version” table with the metadata about the scripts that have already run. It uses a specific file naming to achieve the script execution order, VXXXXX__script_name.sql where XXXXX is a number. Once new scripts are added, you run the migrate command and all scripts with a version number higher than the last executed script will run. It also has a maven plugin which allows us to easily manually trigger migrations and easy hook into a Jenkins build/deployment.
How we’ve used it
We’ve used Flyway to handle migrations on our DEV/SIT environments and use it to determine which scripts need to be packaged for our CAT/PROD environments since we don’t have permission to run database scripts in those. This multi env setup is made easy since flyway allows you to supply a list of folders and it’ll combine the scripts in them and run in order; we just supply different maven properties/profiles to change the list of folders (ie db-all,db-dev for dev, db-all,db-prod for prod) and database connection.
We’ve also used it to help maintain database state during integration tests. By combining, flyway, spring test, and Flyway Test Extensions we are able to clean and rebuild our database so that each test has a clean schema to run against. Dbunit also comes into play here for setting up our test data after flyway has cleaned and migrated the schema.
Pros
- Simple and easy to use
- Reproducible schemas
- Database scripts committed to version control and versioned in the database
- Scripts written in SQL
- Good support for spring, testing, and maven
Cons
- Can’t use in our upper envs (not a tool issue)
- Possible issues with multiple users creating/committing scripts due to versioning nature.
- If someone uses the same version number as you did there will be conflicts (this is why we use timestamps to reduce this chance).
- Also for example someone commits a script with a higher version than mine and the migration runs, and then I commit my script. My script version is now lower than the latest in schema_version table. Developers will just need to be aware of this, and update their script versions if necessary. Another option is that flyway does have an ability to run scripts out of order, so that it will go back and pickup scripts with lower version that were added after the higher version ran, but then you do run into the chance of non reproducible schemas. Whether you enable this option or not it is something that you should be aware of when using flyway.
Other options
Oracle/Postgresql/Mysql/Etc
What is it?
In this case, everyone should be familiar with at least one of these databases as they tend to be the most used. Most applications will connect to a database to persist data collected. We’ll mostly be talking about how we’ve used this database for our integration tests.
How we’ve used it
Our project uses the Oracle database, and we originally setup our tests to run against an oracle xe install. Our tests use flyway to connect to and manage the lifecycle of the database for our integration tests.
Pros
- Same database your application will connect to in a live env
- Allows for database specific features to be used (plsql, triggers, etc)
Cons
- Additional setup required
- Need to host database externally from tests
- Slower than in memory database
- Potentially slower depending on network connection
- Database must be up and accessible to run tests
H2 Database
What is it?
H2 is a fast, lightweight sql database written in Java. It supports embedded and server modes and can be run completely in memory. It also supports syntax compatibility for a number of other databases.
How we’ve used it
Our integration test were originally setup to run against an oracle instance. This became an issue when automating our build due to poor network connection to our dev database test schema. To remedy this we updated our sql scripts to be compatible in both H2 and Oracle (this was minimal since we weren’t using any Oracle specific features and enabled H2s compatibility mode). We then added a maven profile that would use a maven plugin – inmemdb to start H2 in server mode and change the connection properties to this new H2 server instead of oracle. This way our test can be easily run against a full Oracle install or H2 with a single command line property.
We’ve also created a POC in a different branch of the code which uses Arquillian (more on that later) and H2 in embedded mode. I’ll be exploring setting up the H2 server as a spring bean instead of a maven plugin in the future.
Pros
- Quick to setup using spring
- No need to externally host
- Initialized from spring/test (embedded)
- Initialized during pre-integration maven phase or spring (server)
- In memory mode very fast for integration tests that need to reset database often
Cons
- Limited support for database compatibility (no plsql support and other database specific features)
- Additional memory required in your jvm
- Embedded mode only accessible from within same jvm its launched (remedied by running in server mode)
Testing
JUnit
What is it?
Again, this is another one most people are familiar with. JUnit is a framework for writing test code in Java. It provides the foundation of our tests, providing test lifecycle hooks, assertions, maven support, 3rd party extensions and more.
How we’ve used it
All our tests, unit and integration start off with a JUnit test and hook in the appropriate runner depending on the type of test we perform.
Our unit tests tend to use vanilla JUnit or with Mockito to allow for quick and easy dependency mocking.
We’ll go more into our integration tests setups shortly.
While JUnit has been a great tool, we’ll be looking at testNG in the future due to having better ways to parameterize tests.
Pros
- Heavy used in the Java community and simple to use
- Lots of 3rd party extensions
- Maven support
Cons
- Tends to behind most other frameworks in features
Other Options
Spring Test
What is it?
Spring Test is a spring module meant to provide support and ease of testing spring enabled applications. It includes test runners for JUnit/TestNG, takes a spring java config/xml to build the application context, and supports spring profiles and more. It also provides extended support of testing Spring MVC by providing a way to test routing and assert various parts of the response (ie view return, status code returned, model attributes, etc)
How we’ve used it
For our database integration tests we’ve wired our test using the Spring test runner and loaded the entire core module of our project (services/repos/etc) with a spring defined datasource (normally the app uses a container supplied datasource through jndi). We then use spring profiles to ignore the jndi datasource and pickup our test datasource. We then use the flyway extension to execute flyway clean/migrate and dbunit to setup data to put our database in a known state and enable completely reproducible tests.
Pros
- Actively supported and developed by the Spring Source teams
- Test using spring, thus allowing the creation of reusable test beans
- Provides additional information to tests that is not available when unit testing (mvc test features)
- Allows for testing all spring components
Cons
- Still not inside our servlet container
- No access to container resources
- Cannot test UI
- Can’t test things controlled by the container (filters, web.xml mappings, etc)
Selenium Webdriver
What is it?
Selenium Webdriver is a tool which allows us to develop browser tests using code from just about any language. It uses the browsers internal API to find and interact with the browser; this is different from the old selenium where mouse and keyboard recording were used to interact with the browser. It also supports a wide variety of browsers, including mobile. See the selenium webdriver page for more.
How we’ve used it
We use selenium in a few different configurations so that we can tests as much as possible in our build process without deploying to a live environment to catch as many issues as possible before any deployment happens. We can also control our data using properties. We’ll discuss these configurations here.
First off, our configurations have very little duplication thanks to using Spring to wire our tests together. This allows us to create a common base class that setups spring/selenium and then creates and logs a user in for both environments. Let’s take a look at how each is setup.
Our dev environment is setup to mock out any external APIs, using database tables, to allow rapid prototyping and allow us to test the majority of the application without connecting to test environments of systems we don’t control. By doing this we can use DBUnit to populate the external data our system relies on and then run tests against. This environment configuration is controlled by Spring Profiles to enable swap out the real API calls with database writes and enable a web interface for creating mock data.
Now that you understand our dev environment we can talk about how selenium tests are run against it. As we talked about earlier you can back these tests with either an external database or H2 running in server mode since our application will be deployed into a container running in its own VM. For actually application deployment we have two options; First is that you have the application already deployed in a container running in eclipse, connected to the same database your tests will be connected to, this is general how you will develop new tests. Second is that we use maven to deploy our war during the pre-integration phase of our build. In this case we have a maven plugin that spawns a websphere liberty profile server and deploys the app before running our integration tests (our client uses websphere, but you can easily do this for other containers as well). Our spring profile then uses a NOOP implementation of the create user call in our setup (dbunit takes care of this data) and logs in using our dev panel (spring security database auth).
Now you understand how the tests are setup and executed, lets take a look at what we can and can’t test in this configuration. First we can test all UI features now as well as the majority of our functional tests (data creation, validations, etc). What we can’t test in this case are environment specific things, like connectivity between systems (firewalls/ports), webserver request modifications.
Next is our SIT configuration. In this configuration our app is deployed to our test servers (after dev integration test run/pass) and is run in its production configuration using http calls to external APIs. By the time we reach this point, the majority of our actual application testing is already covered. Since most of our integration test have already run during the build we’ll want to mainly test the happy paths to make sure all our api calls properly go through and we know the application is working properly. Again, for these tests we change our Spring profile to pickup different properties/test beans. In this profile our test is NOT connected to any database since all the data will be created through the application/external systems. So this time our create user instead of being NOOP we have it create user in the external system and then our login code is NOOP since the user is automatically logged in after creation.
We’ll discuss the test flow more when we talk about our Jenkins configuration.
Pros
- Most of the test executed during build, errors caught fast
- Using browser APIs allows for test closer to a real user experience
- Tests are fully reproducible in the dev configuration due to database lifecycle control
Cons
- Test are not run in container, and don’t have access to the container datasource and thus need to define their own that connects to the same as the container
- Server must be started up outside of the test (IDE/Maven spawned/etc)
Arquillian
What is it?
Arquillian is a new tool in development by JBoss to allow for in-container tests. Arquillian allows for you to define deployments from within our JUnit/TestNG test and have it deployed to a servlet container. It has a lot of extension support for spring, persistence, various servlet containers and extensions that add abstraction on top of selenium. Arquillian itself is still in early development stages with extensions slightly behind the core module.
How we’ve used it
We’ve used Arquillian to develop a proof of concept to allow all our integration tests to be run full self contained. We use the Tomcat container, spring extension, and persistence extension. We added a @Deployment to our selenium test base class, which packages up our war, and changed the spring runner to the arquillian runner. Combined with a embedded h2 database we are then able to run test fully self contained (in our development profile) without having to have any external resources already started.
The flow of a test in this configuration is as so.
- Arquillian/Spring extenion packages up our app with all maven dependencies (including test/provided but this is configurable)
- Arquillian extracts an embedded Tomcat server and starts it up
- Arquillian deploys the war to the embedded tomcat server
- Spring starts up and connects to the database
- Test starts, uses the same Spring context that is running in your tomcat deployment, has access to everything running on tomcat including jndi resources
- Test runs, passes and everything shutsdown
- Repeat
As you can see the main benefit of this setup is that you just hit run and arquillian takes care of the rest. This reduces a lot of extra configuration and container lifecycle management on our part and lets test run as a 100% real test inside our container.
The only downside at this point is that each test class is going to redeploy the tomcat/application which greatly increases our test times (and the main reason we haven’t merged this POC into trunk). Luckily the Arquillian developers are already aware of this issue and are planning to allow for test suite support to reduce these runtimes ( see https://issues.jboss.org/browse/ARQ-567 ).
Pros
- Tests completely control their own container/test lifecycles
- No need for external resources to be in place before tests run
- Full access to spring context running on serve and inside tests
- Test have access to jndi and other container resources
- Supported by JBoss
- Lots of additional support through extensions
Cons
- Slow test times due to container redeployment per test class
- Early stages of development, thus API changes common
- Increased memory on the JVM running tests with embedded containers
Build Environment
Vagrant
What is it?
Vagrant is a tool, written in Ruby, that provides a useful abstraction for multiple VM providers (VirtualBox, VMWare, EC2, etc) and allows for easy VM configuration and startup from the commandline. It also provides hooks for provisioners (such as Chef) and other extensions in the form of plugins. It also allows us to define VMs in code allowing us to check them into version control.
How we’ve used it
We’ve used Vagrant with to setup an Ubuntu 13.04 server, mount some shared folders, and configure networking. We used the vagrant-omnibus plugin to manage our Chef install and the vagrant-berkshelf plugin to manage copying our chef dependencies to the VM before provisioning. This is all configured in a Vagrantfile and started up by running the “vagrant up” command.
After this initial setup I developed the chef cookbook which we’ll talk about in a few. This allowed me to easily develop chef scripts iteratively by adding new functionality and testing it with “vagrant provision” to rerun the cookbook. We’ll discuss this development flow more when we talk about Chef.
Pros
- Simplifies VM configuration and provisioning
- VMs defined as code
- Easy development and testing of Chef cookbooks
- Good community support
Cons
- Evolvoing API and tools
Chef
What is it?
Chef is a tool with a focus on IT infrastructure automation and management. Chef comes in two flavors, first a full server/client setup with the intent of fully managing your IT and a standalone version called chef-solo that focuses on executing chef cookbooks with out server management side. Chef provides abstractions for installing and executing scripts and other common OS operations. Its written in Ruby, and can execute Ruby code which give’s it a large additional library of useful scripting tools.
You develop Chef Cookbooks which define a idempotent procedural way of installing a piece of software. This basically means you develop cookbooks that can be rerun over and over and your system will always end up in the same state without error. This development style, combined with Vagrant, allows for us to quickly and easily develop cookbooks in small focused chunks that we can rerun over and over as we add features. This also allows you to easily deploy an update to a large number of servers at once with minimal chance of error.
Chef cookbooks can be linted, unit tested, and integration tested to verify working code. Combined with Vagrant and Jenkins you can setup a continuous integration server for chef cookbooks.
Chef also comes with the benefit that it is being used as the base of amazon web services opsworks platform so that you can execute custom chef scripts easily.
How we’ve used it
We used Chef, along with Vagrant and berkshelf (chef cookbook dependency manager), to develop a Jenkins build box for our app. The build box installs the following software to support the build/test configurations we’ve been talking about.
- install/update apt
- install java 7
- setup dns-search domains and restart networking
- install jenkins server
- install git and svn
- setup svn proxy
- download and store svn certs
- install maven
- install chrome
- install xvfb (headless selenium tests)
- install chromedriver for selenium test
- configure jenkins server
Now most of the software above has a Chef cookbook already developed that can run against multiple OSes (for example java cookbook supports debian/rhel/window with multiple versions 6/7/ibm/oracle/etc). And all cookbooks can be parameterized, allowing for high reusability.
And since everything is defined in code, we can of course check it into version control.
Pros
- Infrastructure as Code/Version Control
- Automate installation and configuration of machines
- Reduce error due to repetition
- Continuous integration/Testable
- Quickly get up identical environments test/prod mirror/etc
- Much smaller than VMs
- Large base cookbook library
- AWS Opworks support
- Strong community support
- Ruby Libraries
Cons
- Provisioning time when starting from clean VM (more of issue for things like aws autoscaling)
Jenkins
What is it?
Jenkins is a tool written in Java that provides a continuous integration environment for projects. It has a very large plugin library and provides scripting support via the Groovy language. Due to this Jenkins is one of the most popular CI software. As with most things, there is a lot more the Jenkins than what we’ll be discussing here.
The goal of Jenkins is to provide an interface for developing build/deployment pipelines, task automation, reporting and more. It has support for most major build tools including maven which we’ll discuss. Jobs can be trigger manually, scheduled, off a version control hooks, polling a version control for updates, off of over builds completing, and more.
How we’ve used it
As we talked about before we used Chef to develop our Jenkins server. This required that we figure out how Jenkins was managing its configuration. Jenkins has a number of xml/json/text files that it and its plugins use to persist configuration changes. First I inited a git repo in my Jenkins home folder and then proceeded to commit changes as I updated the Jenkins configuration. This allowed me to track and add all the configuration updates to my cookbook. This itself seems like a large hassle due to just the sheer amount of configuration files and eventually will most likely need to be changed over to using the scripting console to configure and maintain the server.
By default Jenkins comes with a plugin to perform maven builds. We took advantage of this since the project is a maven project. We have two jobs configured in jenkins.
The first job performs CI for our trunk branch. This job polls for changes in svn. Once a change is picked up, Jenkins checks out the source code and performs out maven build command to run our full (dev) integration test suite. We either end up with a successful build (all tests passed), unstable build (all unit tests passed but integration test failures), or a failed build (compilation errors or unit test failures).
The second job, uses the buildresult-trigger plugin. The goal of this job is to tag and deploy the app to our test servers. This job triggers twice a day if it sees the first job with a new build in a successful state. If so this job will handle database migration using maven/flyway, handle websphere deployment using the websphere admin clinet, and then execute our happy path integration tests to make sure everything deployed correct. If the first job is not in success status, then this job will not run and thus prevent a build with integration test errors being deployed.
Pros
- Mature CI platform
- Tons of Community support
- Support for just about anything through plugins
- Scriptable
Cons
- Text file configuration hell (most likely solved by using scripting console)
- Doesn’t seem to be a ton of info on scripting outside of the javadocs
Naming Stuff is Hard 