One of the most beneficial things you can do when developing production quality applications is apply automated testing in a few different layers. In my most recent projects we’ve successfully built and deployed applications with no dedicated QA and minimal amounts of manual testing. First lets take a look at our layers.
- Unit Tests
- Spring Database/MVC Integration tests
- Pre-deployment Selenium Integration Tests
- Post-deployment Selenium Integration Tests
and then how we need to configure our tests for execution during
- development
- build
Unit Tests
Unit testing give us a few advantages. To put it bluntly, it forces you to design and code loosely coupled objects like we’ve always been taught, but not always practiced. This isn’t to say that other code isn’t testable, but it becomes a huge nightmare when you are testing code that does 100 different things.
Next, Unit Testing also documents our code. By writing a unit test I am telling other developers who end up working on the code “this is how it should work in these situations” and “these are my assumptions for this code”. Unit test combined with verbose self documenting code helps eliminate our need for large amounts of javadocs and other written documentations (not to say you should write no documentation, but you should understand when and where it is necessary).
The main thing to understand here is that these test do not determine if your system actually works. They just make sure the assumptions of the initial developer hold true for a given unit (class in this case).
Let’s take a look at an example unit test where we isolate our class under test using mockito to mock out its dependencies.
@RunWith(MockitoJUnitRunner.class)
public class ManifestServiceImplTest {
private static final long FILEID = -1L;
@InjectMocks
private ManifestService manifestServiceImpl = new ManifestServiceImpl();
@Mock
private UserService userService;
@Mock
private MidService midService;
@Mock
private ManifestRepository manifestRepository;
private Manifest manifest;
private User user;
private final String username = "abc";
@Captor
private ArgumentCaptor<Pageable> manifestPageCaptor;
@Before
public void setup() {
user = new User();
user.setUsername(username);
when(userService.find(username)).thenReturn(user);
manifest = new Manifest();
when(manifestRepository.findOne(FILEID)).thenReturn(manifest);
when(manifestRepository.save(manifest)).thenReturn(manifest);
}
@Test
public void getAvailableEnvNone() {
when(midService.hasCompletedMidCertificationStatus(username))
.thenReturn(false);
when(midService.hasIncompletedMidCertificationStatus(username))
.thenReturn(false);
assertTrue("no manifestEnvs should be returned if user has no mid",
manifestServiceImpl.getAvailableManifestEnvs(username)
.isEmpty());
}
@Test
public void getAvailableEnvCompleteOnly() {
when(midService.hasCompletedMidCertificationStatus(username))
.thenReturn(true);
when(midService.hasIncompletedMidCertificationStatus(username))
.thenReturn(false);
Set<ManifestEnv> manifestEnvs = manifestServiceImpl
.getAvailableManifestEnvs(username);
assertEquals(
"manifestEnvs should have 2 entries when user only has completed mid cert",
2, manifestEnvs.size());
assertTrue("manifestEnvs should contain all ManifestEnv enums",
manifestEnvs.containsAll(Arrays.asList(ManifestEnv.values())));
}
@
Test
public void getAvailableEnvIncompletOnly() {
when(midService.hasCompletedMidCertificationStatus(username))
.thenReturn(false);
when(midService.hasIncompletedMidCertificationStatus(username))
.thenReturn(true);
Set<ManifestEnv> manifestEnvs = manifestServiceImpl
.getAvailableManifestEnvs(username);
assertEquals(
"manifestEnvs hsould only have 1 entry when user has only incomplete mid cert",
1, manifestEnvs.size());
assertTrue("mainfestEnvs should only contain TEM",
manifestEnvs.contains(ManifestEnv.TEM));
}
@Test
public void getAvailableEnvBoth() {
when(midService.hasCompletedMidCertificationStatus(username))
.thenReturn(true);
when(midService.hasIncompletedMidCertificationStatus(username))
.thenReturn(true);
Set<ManifestEnv> manifestEnvs = manifestServiceImpl
.getAvailableManifestEnvs(username);
assertEquals(
"manifestEnvs should have 2 entries when user only has completed mid cert",
2, manifestEnvs.size());
assertTrue("manifestEnvs should contain all ManifestEnv enums",
manifestEnvs.containsAll(Arrays.asList(ManifestEnv.values())));
}
@Test
public void find() {
when(manifestRepository.findOne(FILEID)).thenReturn(manifest);
final Manifest returnedManifest = manifestServiceImpl.find(FILEID);
verify(manifestRepository).findOne(FILEID);
assertEquals("manifest should be returned when found by FILEID",
manifest, returnedManifest);
}
@Test
public void findNotFound() {
when(manifestRepository.findOne(FILEID)).thenReturn(null);
final Manifest returnedManifest = manifestServiceImpl.find(FILEID);
verify(manifestRepository).findOne(FILEID);
assertEquals(
"null should be returned when a manifest file is not found",
null, returnedManifest);
}
@Test
public void findUserManifestHistory() {
final Page<Manifest> page = new PageImpl<Manifest>(
Lists.newArrayList(manifest));
when(
manifestRepository.findByUserUsernameOrderByCreatedTimeDesc(
eq("abc"), isA(Pageable.class))).thenReturn(page);
manifestServiceImpl.findUserManifestHistory("abc");
verify(manifestRepository).findByUserUsernameOrderByCreatedTimeDesc(
eq("abc"), manifestPageCaptor.capture());
assertEquals("user manifest histroy should be max 7 for page size", 7,
manifestPageCaptor.getValue().getPageSize());
assertEquals(
"user manifest histroy should always return the first page", 0,
manifestPageCaptor.getValue().getPageNumber());
}
@Test
public void create() {
final Manifest returnedManifest = manifestServiceImpl.create(manifest,
username);
assertEquals("user should be set to manifest when creating", user,
returnedManifest.getUser());
verify(manifestRepository).save(manifest);
}
}
and our class under test
@Service
public class ManifestServiceImpl implements ManifestService {
@Autowired
private ManifestRepository manifestRepository;
@Autowired
private UserService userService;
@Autowired
private MidService midService;
@Override
public Manifest create(final Manifest manifest, final String username) {
Validate.notNull(manifest);
Validate.notBlank(username);
final User user = userService.find(username);
Validate.notNull(user);
manifest.setUser(user);
return manifestRepository.save(manifest);
}
@Override
public Set<ManifestEnv> getAvailableManifestEnvs(final String username) {
Validate.notBlank(username);
final Set<ManifestEnv> envs = Sets.newHashSet();
if (midService.hasCompletedMidCertificationStatus(username)) {
envs.add(ManifestEnv.PROD);
envs.add(ManifestEnv.TEM);
return envs;
}
if (midService.hasIncompletedMidCertificationStatus(username)) {
envs.add(ManifestEnv.TEM);
}
return envs;
}
@Override
@PostAuthorize("returnObject == null or returnObject.user.username == principal.username")
public Manifest find(final long id) {
return manifestRepository.findOne(id);
}
@Override
public Page<Manifest> findUserManifestHistory(final String username) {
Validate.notBlank(username);
final Pageable pageable = new PageRequest(0, 7);
return manifestRepository.findByUserUsernameOrderByCreatedTimeDesc(
username, pageable);
}
}
I mainly want to look at the find method so we can demonstrate what unit tests do and do not do for us.
The find method does nothing except delegate to a spring data repository interface. So we have nothing really to test except that the method got called. But even so we actually have two test methods for this method. The question is why? We can’t test much functionality here since our method isn’t doing much on its own, and code coverage would be 100% with only a single test method verifying the mocked out object had its findOne method called. But we have a found and notFound test, and this is because our interface can return a null or non null object. So here we aren’t really testing anything, but we are documenting that our interface will return null if nothing is found by the repository.
That being said, lets move on to some test that actually test our system (paritally) as a whole.
Spring Database and MVC Integration Tests
This is our first layer of integration tests. We use the spring-test framework for building these tests that focus on our spring container and down. These test spin up their own spring context and are not deployed to a container. Our test also have full access to our spring context so we can inject beans into our test class.
First lets look at a spring database integration test for our MainfestServiceImpl class to compare it with the unit tests we created.
@FlywayTest
@DBUnitSupport(loadFilesForRun = { "CLEAN_INSERT", "/dbunit/dbunit.base.xml",
"CLEAN_INSERT", "/dbunit/dbunit.dev.base.xml", "CLEAN_INSERT",
"/dbunit/dbunit.manifest.xml", "CLEAN_INSERT", "/dbunit/dbunit.mid.xml" })
public class ManifestServiceImplSpringDatabaseITest extends
AbstractSpringDatabaseTest {
@Autowired
private ManifestService manifestService;
@Autowired
private UserRepository userRepository;
private User user;
@Before
public void setup() {
user = userRepository.findOne(-1L);
}
@Test
@DBUnitSupport(loadFilesForRun = { "CLEAN_INSERT",
"/dbunit/dbunit.mid.cert.incomplete.xml" })
public void getAvailableManifestEnvsTEMOnly() {
Set<ManifestEnv> manifestEnvs = manifestService
.getAvailableManifestEnvs(user.getUsername());
assertEquals(
"only one ManifestEnv should be returned when user only has incomplete mid",
1, manifestEnvs.size());
assertTrue("returned manifestEnvs should contain TEM",
manifestEnvs.contains(ManifestEnv.TEM));
}
@Test
@DBUnitSupport(loadFilesForRun = { "CLEAN_INSERT",
"/dbunit/dbunit.mid.cert.complete.xml" })
public void getAvailableManifestEnvsTEMAndPROD() {
Set<ManifestEnv> manifestEnvs = manifestService
.getAvailableManifestEnvs(user.getUsername());
assertEquals(
"only TEM and PROD should be returned when user only has complete mid",
2, manifestEnvs.size());
assertTrue("returned manifestEnvs should contain TEM and PROD",
manifestEnvs.containsAll(Arrays.asList(ManifestEnv.values())));
}
@Test
public void findFound() {
assertNotNull(manifestService.find(-1L));
}
@Test
public void findNotFound() {
assertNull("null should be returned when file not found",
manifestService.find(-10L));
}
@Test
@DBUnitSupport(loadFilesForRun = { "CLEAN_INSERT",
"/dbunit/dbunit.manifest.xml" })
public void findUserManifestHistory() {
assertEquals("user should have 2 manifest in their history", 2,
manifestService.findUserManifestHistory(user.getUsername())
.getNumberOfElements());
}
@Test
public void create() throws IOException {
final Manifest manifest = new Manifest();
manifest.setEnvironment(ManifestEnv.PROD);
byte[] data = "hello".getBytes();
final MultipartFile multipartFile = new MockMultipartFile("somefile",
"file.txt", null, new ByteArrayInputStream(data));
manifest.setMultipartFile(multipartFile);
final Manifest returnManifest = manifestService.create(manifest,
user.getUsername());
assertTrue(returnManifest.getFileSystemResource().exists());
assertNotNull("id should be set for saved manifest", manifest.getId());
assertNotNull("createdTime should be set on manifest",
manifest.getCreatedTime());
assertNotNull("path should be set on manifest", manifest.getPath());
assertNotNull("filename should be set on manifest",
manifest.getFilename());
assertEquals("file should be saved when manifest is saved",
data.length, IOUtils.toByteArray(manifest
.getFileSystemResource().getInputStream()).length);
}
}
Again, similar test methods, but this time our intentions are different. For integration test we are now actually testing the application as it would be run in a live environment. We don’t mock anything here, but we do need to setup test data in our database so that our tests are reproducible.
Let’s look at the create method this time. Since our object is a hibernate entity, we expect hibernate to perform some operations for us. In this case, our entity has a prePersist method that writes a file to the file system before saving our entity to the database. That method also sets up the state of our entity by storing the path to the file, its original filename, time it was created, and hibernate assigns an id.
The @Flyway annotation handles our database lifecycle. It can be placed on the method/class level and will clean and rebuild the database. This combined with the @DBUnitSupport annotation lets us fully control the state of our database for each test. See https://github.com/flyway/flyway-test-extensions for more information.
That being said, lets take a look at the AbstractSpringDatabaseTest class that we extend so we can see how everything is configured for these tests.
@RunWith(SpringJUnit4ClassRunner.class)
@Category(IntegrationTest.class)
@ContextConfiguration(classes = { SpringTestConfig.class })
@ActiveProfiles({ "DEV_SERVICES", "TEST_DB", "DEV_SECURITY" })
@TestExecutionListeners({ DependencyInjectionTestExecutionListener.class,
FlywayDBUnitTestExecutionListener.class,
TransactionalTestExecutionListener.class })
@TransactionConfiguration(defaultRollback = false)
@Transactional("transactionManager")
public class AbstractSpringDatabaseTest {
}
So few things here. First is the @RunWith and @ContextConfiguration annotations setup our spring context and @ActiveProfiles sets the spring profiles we want to use while running these tests. The @TestExecutionListeners lets us register listeners that spring-test provides hooks for in our tests. DependencyInjectionTestExecutionListener allows us to inject beans directly into our test, FlywayDBUnitTestExecutionListener handles @Flyway and @DbUnitSupport annotations and TransactionalTestExecutionListener makes our tests transnational so hibernate has transactions to work within. Next for transactional support we have @TransactionConfiguration which allows us to configure our transactions and @Transactional(“transactionManager”) which actually wraps our test methods in transactions (you most likely have seen this annotation when writing transnational code).
Next we need to take a look at the SpringTestConfig class
@Configuration
@Import({ DatabaseTestConfig.class })
@ComponentScan(value = "com.company.artifact.app")
@PropertySource("classpath:test.properties")
public class SpringTestConfig {
@Bean
public static PropertySourcesPlaceholderConfigurer propertyPlaceholder() {
final PropertySourcesPlaceholderConfigurer placeholder = new PropertySourcesPlaceholderConfigurer();
placeholder.setIgnoreUnresolvablePlaceholders(true);
return placeholder;
}
Again, this class isn’t doing too much. It tells spring to scan our base class for beans and imports another configuration class. It also imports some properties for our test.
@Configuration
@Profile("TEST_DB")
@PropertySource({ "classpath:flyway.properties" })
public class DatabaseTestConfig {
@Value("${flyway.user}")
private String user;
@Value("${flyway.password}")
private String password;
@Value("${flyway.url}")
private String url;
@Value("${flyway.locations}")
private String locations;
@Value("${flyway.placeholdersPrefix}")
private String prefix;
@Value("${flyway.placeholderSuffix}")
private String suffix;
@Bean(destroyMethod = "close")
public DataSource dataSource() {
final BasicDataSource basicDataSource = new BasicDataSource();
basicDataSource.setUsername(user);
basicDataSource.setPassword(password);
basicDataSource.setUrl(url);
basicDataSource.setMaxActive(-1);
return basicDataSource;
}
@Bean
public FlywayHelperFactory flywayHelperFactory() {
final FlywayHelperFactory factory = new FlywayHelperFactory();
final Properties flywayProperites = new Properties();
flywayProperites.setProperty("flyway.user", user);
flywayProperites.setProperty("flyway.password", password);
flywayProperites.setProperty("flyway.url", url);
flywayProperites.setProperty("flyway.locations", locations);
flywayProperites.setProperty("flyway.placeholderPrefix", prefix);
flywayProperites.setProperty("flyway.placeholderSuffix", suffix);
factory.setFlywayProperties(flywayProperites);
return factory;
}
@Bean
public Flyway flyway() {
final Flyway flyway = flywayHelperFactory().createFlyway();
flyway.setDataSource(dataSource());
return flyway;
}
@Bean
@Qualifier("userNumber")
public String userNumber() {
return "";
}
}
We need to setup our own datasource for our tests since they won’t be run from within our container, and thus won’t have access to any jndi resources. Also here we configure flyway to use that datasource as well. flyway.properties is actually populated by defaults in our parent maven pom and can be overridden during a test run if needed. We’ll see later when we talk about the maven build how we use these properties to run against either oracle or h2 database.
Ignore the userNumber bean for now, we’ll get to that when we talk about pre and post deployment selenium tests.
Next lets look at how we extend these database test to support using spring-test for testing Spring MVC.
@WebAppConfiguration
public class AbstractSpringMvcTest extends AbstractSpringDatabaseTest {
@Autowired
protected WebApplicationContext webApplicationContext;
@Autowired
private FilterChainProxy springSecurityFilterChain;
protected MockMvc mockMvc;
@Before
public void setup() {
mockMvc =
MockMvcBuilders.webAppContextSetup(webApplicationContext)
.addFilter(springSecurityFilterChain).build();
}
protected UserDetailsRequestPostProcessor custregUser(final String username) {
return SecurityRequestPostProcessors.userDeatilsService(username).userDetailsServiceBeanId(
"custregUserDetailsService");
}
}
Here we are extending our database test functionality and adding in some spring mvc test configuration. The code here is setting up springs mockMvc for us and setting a username and userDetailsService for our security so we don’t need to mock out our user. Also we annotate our configuration with @WebAppConfiguration so that we have a WebApplicationContext created for us.
Currently Spring MVC test does not support spring security, but there is an example at https://github.com/spring-projects/spring-test-mvc/blob/master/src/test/java/org/springframework/test/web/server/samples/context/SecurityRequestPostProcessors.java that we borrow from. This let’s use create some request post processors and add in our spring security information before tests run. This makes it so we don’t need to mock out our SecurityContextHolder wrapper bean since we’ll set an actual authentication object on it. This feature will most likely be added in a later version of spring test.
There’s not much configuration here outside what we’ve covered during our database tests so lets take a look at an example using Spring mvc test.
@FlywayTest
@DBUnitSupport(loadFilesForRun = { "CLEAN_INSERT", "/dbunit/dbunit.base.xml",
"CLEAN_INSERT", "/dbunit/dbunit.dev.base.xml", "CLEAN_INSERT",
"/dbunit/dbunit.mid.xml", "CLEAN_INSERT",
"/dbunit/dbunit.mid.cert.complete.xml", "CLEAN_INSERT",
"/dbunit/dbunit.manifest.xml", })
public class ManifestControllerSpringMvcITest extends AbstractSpringMvcTest {
private User user;
@Autowired
private UserRepository userRepository;
private MockMultipartFile file;
@Before
public void setupData() {
user = userRepository.findOne(-1L);
file = new MockMultipartFile("multipartFile", "orig", null,
"bar".getBytes());
}
@Test
public void index() throws Exception {
mockMvc.perform(get("/manifests").with(custregUser(user.getUsername())))
.andExpect(view().name(is("manifest/index")))
.andExpect(
model().attributeExists("manifestHistory",
"manifestEnvironments"));
}
@Test
public void uploadAjaxEnvironmentValidationErrors() throws Exception {
mockMvc.perform(doFileUpload(file).accept(MediaType.APPLICATION_JSON))
.andExpect(status().isBadRequest())
.andExpect(
jsonPath("$.fieldErrors[0].field", is("environment")))
.andExpect(
jsonPath("$.fieldErrors[0].error",
is("This field cannot be null.")));
}
@Test
public void uploadAjaxFileEmtpyValidationErrors() throws Exception {
mockMvc.perform(
doFileUpload(
new MockMultipartFile("multipartFile", "orig", null,
new byte[0]))
.accept(MediaType.APPLICATION_JSON).param(
"environment", "PROD"))
.andExpect(content().contentType(MediaType.APPLICATION_JSON))
.andExpect(status().isBadRequest())
.andExpect(
jsonPath("$.fieldErrors[0].field", is("multipartFile")))
.andExpect(
jsonPath("$.fieldErrors[0].error",
is("Please select a valid file to upload.")));
}
@Test
public void uploadAjaxSuccess() throws Exception {
mockMvc.perform(
doFileUpload(file).param("environment", "PROD").accept(
MediaType.APPLICATION_JSON)).andExpect(status().isOk())
.andExpect(content().contentType(MediaType.APPLICATION_JSON));
}
@Test
public void uploadEnvironmentValidationErrors() throws Exception {
mockMvc.perform(doFileUpload(file))
.andExpect(status().isOk())
.andExpect(model().hasErrors())
.andExpect(
model().attributeHasFieldErrors("manifest",
"environment"));
}
@Test
public void uploadEmptyFileValidationErrors() throws Exception {
mockMvc.perform(
doFileUpload(new MockMultipartFile("multipartFile", "orig",
null, new byte[0])))
.andExpect(status().isOk())
.andExpect(model().hasErrors())
.andExpect(
model().attributeHasFieldErrors("manifest",
"multipartFile"));
}
@Test
public void uploadSuccess() throws Exception {
mockMvc.perform(doFileUpload(file).param("environment", "PROD"))
.andExpect(redirectedUrl("/manifests"))
.andExpect(model().hasNoErrors());
}
private MockHttpServletRequestBuilder doFileUpload(
final MockMultipartFile file) {
return fileUpload("/manifests").file(file).with(
custregUser(user.getUsername()));
}
}
and the controller under test
@Controller
public class ManifestController {
@Autowired
private ManifestService manifestService;
@Autowired
private SecurityHolder securityHolder;
@RequestMapping(value = "manifests", method = RequestMethod.GET)
public String index(@ModelAttribute final Manifest manifest,
final Model model) {
setupManifestModel(model);
return "manifest/index";
}
private void setupManifestModel(final Model model) {
model.addAttribute("manifestHistory", manifestService
.findUserManifestHistory(securityHolder.getName()));
model.addAttribute("manifestEnvironments", manifestService
.getAvailableManifestEnvs(securityHolder.getName()));
}
@RequestMapping(value = { "manifests", "api/manifests" }, method = RequestMethod.POST, produces = MediaType.APPLICATION_JSON_VALUE)
public @ResponseBody
Manifest uploadAjax(@Valid @ModelAttribute final Manifest manifest,
final BindingResult bindingResult)
throws MethodArgumentNotValidException {
if (!manifestService.getAvailableManifestEnvs(securityHolder.getName())
.contains(manifest.getEnvironment())) {
bindingResult.rejectValue("environment", "invalid.manifest.env");
}
if (bindingResult.hasErrors()) {
throw new MethodArgumentNotValidException(null, bindingResult);
}
return manifestService.create(manifest, securityHolder.getName());
}
@RequestMapping(value = "manifests", method = RequestMethod.POST)
public String upload(@Valid @ModelAttribute final Manifest manifest,
final BindingResult bindingResult, final Model model,
final RedirectAttributes redirectAttributes) {
if (!manifestService.getAvailableManifestEnvs(securityHolder.getName())
.contains(manifest.getEnvironment())) {
bindingResult.rejectValue("environment", "invalid.manifest.env");
}
if (bindingResult.hasErrors()) {
setupManifestModel(model);
return "manifest/index";
}
manifestService.create(manifest, securityHolder.getName());
redirectAttributes.addFlashAttribute("flashMessage",
"manifest.upload.success");
return "redirect:/manifests";
}
@RequestMapping(value = { "manifests/{id}", "api/manifests/{id}" }, method = RequestMethod.GET, produces = MediaType.APPLICATION_OCTET_STREAM_VALUE)
public @ResponseBody
FileSystemResource download(@PathVariable final Long id,
final HttpServletResponse httpServletResponse)
throws FileNotFoundException, IOException {
final Manifest manifest = manifestService.find(id);
if (manifest == null)
throw new NotFoundException();
httpServletResponse.setHeader("Content-Disposition",
"attachment; filename=" + manifest.getFilename());
return manifest.getFileSystemResource();
}
@Autowired
private MessageSource messageSource;
@ExceptionHandler(MethodArgumentNotValidException.class)
@ResponseStatus(value = HttpStatus.BAD_REQUEST)
public @ResponseBody
HttpValidationMessage validation(final MethodArgumentNotValidException e,
final Locale locale) {
return new HttpValidationMessage(e.getBindingResult(), messageSource,
locale);
}
}
So what are we and aren’t we testing here? First off we are not testing the UI and servlet container features. What we are testing is our HTTP API that we’ve created in spring, along with all the services and other objects involved. You’re spring code will be executed as if it had received a real requests from the servlet container.
Spring test provides us with a nice builder pattern api for creating mock http requests and having them run through spring mvc. We can easily include things like request params, content type, and more. Then spring give us access to the http response, such as content type and response codes as well as other headers, along with any spring mvc features like model and views.
Decoupling External Systems
Before we get into selenium testing, we need to talk about our application profiles. We rely on external systems for all our user/company data and security and sometimes we even create/modify data in other systems when necessary which would need to be reset. To allow for easily reproducible and fast selenium tests we need to decouple our system from these other systems. To achieve this we used spring profiles to provide database implementations of our api calls. So instead of a class using springs restOperations to make a http call, we instead just have the interface backed by a hibernate object. These database implementations are activated by our DEV_SERVICES profile which you have seen in our test configurations. We have to do something similar with our security. Instead of using the custom filter provided by the external system we use spring security’s jdbc implementation and tie that to the DEV_SECURITY profile. With this done we can control all the data from the external systems using flyway and dbunit. We can then cover the missing api calls in post deployment selenium tests or in spring tests.
Selenium Integration Tests
Now that we can start talking about our selenium tests. The idea here is to split our test into 2 categories, Pre and Post deployment.
The purpose of pre-deployment test are to test the UI and functionality of the system in a reproducible manner so that they can be run by developers before committing code and during continuous integration builds to catch any issues before we deploy our application. If your system has a regression, or a database script error, or many of the other things that could go wrong should be caught here. We are testing the 90% of application at this point, include browser, server, and database interactions. We are not testing our external services since they are backed by the database and we are not testing any production server settings/features/etc.
Now post-deployment tests are less about verifying the application features/validations work properly and more about testing that the application deployed correctly. These test will need to setup user/company data in the external systems before they run and use the custom authentication provided by them. They’ll test the happy paths of the functionality to verify that all the external apis, database connections, etc are working properly. Also you can test web server configuration here like making sure you redirect all http to https and any other type of url rewrites/proxying/whatever that would be configured in your production env but not your developer ones.
Let’s start with our ManifestController pre-deployment selenium test
/**
*
* @author stephen.garlick
* @author lindsei.berman
*
*/
@DBUnitSupport(loadFilesForRun = { "CLEAN_INSERT", "/dbunit/dbunit.base.xml",
"CLEAN_INSERT", "/dbunit/dbunit.dev.base.xml", "CLEAN_INSERT",
"/dbunit/dbunit.mid.xml", "CLEAN_INSERT",
"/dbunit/dbunit.mid.cert.incomplete.xml" })
@FlywayTest
public class ManifestControllerSeleniumITest extends AbstractDevSeleniumTest {
@Value("${selenium.base.url}")
private String baseUrl;
@Autowired
private ManifestPage manifestPage;
@Autowired
private ResourceLoader resourceLoader;
@Autowired
private SeleniumElementVisibilityTester seleniumElementVisibilityTester;
@Before
@Override
public void setup() throws Exception {
super.setup();
getWebDriver().get(baseUrl + "/manifests");
}
@Test
@DBUnitSupport(loadFilesForRun = { "CLEAN_INSERT",
"/dbunit/dbunit.mid.cert.incomplete.xml" })
public void fileSizeCannotBeZero() throws IOException {
manifestPage
.selectFile(getAbsoluteFilePath("classpath:files/manifest-empty-test-data"));
assertTrue(manifestPage.isFileErrorDisplayed());
}
@Test
public void successfulUpload() throws IOException {
manifestPage
.selectFile(
getAbsoluteFilePath("classpath:files/manifest-not-empty-test-data"))
.submit();
assertTrue(manifestPage.getManifestHistorySize() >= 1);
}
/**
* When Client's certification is incomplete he/she should be able to view
* only the Pre-Production option in the Environment selection drop down box
*
* @throws IOException
*/
@Test
@DBUnitSupport(loadFilesForRun = { "CLEAN_INSERT",
"/dbunit/dbunit.mid.cert.incomplete.xml" })
public void userIncompleteCertificationOnlyViewPreProduction()
throws IOException {
assertEquals("Pre-Production", manifestPage.getTEMEnvironmentText());
}
/**
* When Client is Certified to upload manifests he/she should be able to
* view both Pre-Production and Production options in the Environment
* selection drop down box
*
* @throws IOException
*/
@Test
@DBUnitSupport(loadFilesForRun = { "CLEAN_INSERT",
"/dbunit/dbunit.mid.cert.complete.xml" })
public void userCertifiedViewBothPreProductionAndProduction()
throws IOException {
assertEquals("user should see both prod and preprod options", 2,
manifestPage.getNumberOfEnvironmentOptions());
}
/**
*
* @throws IOException
*
* when the user picks a manifest using the manifest select
* button. The manifest name should be displayed beside the
* cancel and upload button. Then once the cancel button is
* pressed the name should no longer be displayed and the
* file-select should be displayed
**/
@Test
public void manifestCancelSuccessful() throws IOException {
int before = manifestPage.getManifestHistorySize();
manifestPage
.selectFile(
getAbsoluteFilePath("classpath:files/manifest-not-empty-test-data"))
.cancel();
assertTrue(manifestPage.isFileSelectDisplayed());
int after = manifestPage.getManifestHistorySize();
assertEquals(before, after);
}
/**
* After manifest select button is pressed and a file is chosen
* successfully(not too small) then the upload and cancel button should be
* visible
*
* @throws IOException
*/
@Test
public void manifestClickandFileChoosenUploadandCancelDisplayed()
throws IOException {
manifestPage
.selectFile(getAbsoluteFilePath("classpath:files/manifest-not-empty-test-data"));
List<String> buttons = Lists.newArrayList("upload-file-button",
"cancel-file-button");
seleniumElementVisibilityTester.testElementsDisplayedAndEnabled(
getWebDriver(), buttons);
}
private String getAbsoluteFilePath(final String resource)
throws IOException {
return resourceLoader.getResource(resource).getFile().getAbsolutePath();
}
}
Again you’ll see we are controlling the database with flyway and dbunit. One thing you might realize is that we require that we have a server started up to run this test. We solve this later with maven for our builds, but for development we need to have our server up when running our tests. This is solved by Arquillian which is quickly approaching production readiness. We won’t be going into that today, but look for a future post.
If you’ve done browser work you’ll notice a lot of familiar things in the above code like css selectors. Here we are able to test that specific elements on our page are visible, enabled, and anything else you could determine from within a browser. This is because of seleniums webdriver, which interacts with an actual api for each browser directly; turn on debugging and you can see http calls for each interaction you perform within the test.
Lets go in a little deeper and start looking at our base classes.
@Category(IntegrationTest.class)
@TestExecutionListeners({DependencyInjectionTestExecutionListener.class,
FlywayDBUnitTestExecutionListener.class})
@ActiveProfiles({"TEST_DB", "TEST_DEV"})
@ContextConfiguration(classes = {DatabaseTestConfig.class, SeleniumTestConfig.class})
public class AbstractDevSeleniumTest extends AbstractSeleniumTest {
}
Most of this should look familiar. We have a new profile TEST_DEV which will discuss in a moment. Also we see a new configuration class
@Configuration
@ComponentScan("com.company.artifact.test.selenium")
@PropertySource({ "classpath:selenium/selenium.properties" })
public class SeleniumTestConfig {
@Bean(destroyMethod = "stop", initMethod = "start")
public ChromeDriverService chromeDriverService() throws IOException {
final ChromeDriverService chromeDriverService = new ChromeDriverService.Builder()
.usingDriverExecutable(
new File(System.getProperty("user.home")
+ "/chromedriver")).usingAnyFreePort().build();
return chromeDriverService;
}
@Bean(destroyMethod = "quit")
public ChromeDriver chromeDriver() throws IOException {
final ChromeDriver chromeDriver = new ChromeDriver(
chromeDriverService());
return chromeDriver;
}
/**
* Configuration for integration tests the run during the build process.
*
* @author stephen.garlick
*
*/
@Configuration
@Profile("TEST_DEV")
@PropertySource("classpath:selenium/selenium-build.properties")
static class BuildSeleniumConfig {
}
/**
* Configuration for integration tests that run post deployment.
*
* @author stephen.garlick
*
*/
@Configuration
@Profile("TEST_SIT")
@PropertySource("classpath:selenium/selenium-sit.properties")
static class SitSeleniumConfig {
}
@Bean
public static PropertySourcesPlaceholderConfigurer propertyPlaceholder() {
final PropertySourcesPlaceholderConfigurer placeholder = new PropertySourcesPlaceholderConfigurer();
placeholder.setIgnoreUnresolvablePlaceholders(true);
return placeholder;
}
}
Here we setup our chromeDriverService which expects the chromedriver executable and then the chromeDriver bean itself which we’ll be using to interact with the browser. Then we component scan for our reusable selenium beans and pulling in some properties.
Next let’s take a look at our base test class
@RunWith(SpringJUnit4ClassRunner.class)
public abstract class AbstractSeleniumTest {
@Value("${bcg.user.name}")
private String bcgUserName;
private String userNumber;
private String username;
@Autowired
@Qualifier("userNumber")
private Provider<String> userNumberProvider;
@Before
public void setup() throws Exception {
userNumber = userNumberProvider.get();
username = bcgUserName + userNumber;
createUserTester.createUser(webDriver, username);
loginTester.loginIn(webDriver, username);
}
@After
public void tearDown() throws Exception {
webDriver.manage().deleteAllCookies();
}
@Autowired
private WebDriver webDriver;
public WebDriver getWebDriver() {
return webDriver;
}
@Autowired
private SeleniumLoginTester loginTester;
@Autowired
private SeleniumCreateUserTester createUserTester;
}
This is where a lot of the work is going on for our tests, so lets break it down.
First the setup method. This method will be run for our pre and post deployment tests but will do different things based on the TEST_DEV or TEST_SIT profiles. If you are in a TEST_DEV (pre-deployment) test then it takes the bcgUserName property adds empty string to it and then uses that as the username for our test. Next it does a createUser call, which in this case is an empty implementation since dbunit will take care of this during database setup. Next it’ll login using our dev login page we discussed earlier. Now for the TEST_SIT profile userNumber will actually pull a number from a database sequence, which we’ll see when we look at the post deployment configuration, and creatUser will actually create a user in the external system and login does nothing because we are already logged in after creating the user.
The only thing we do after each test is clear the cookies, and thus authentication info, from the webDriver. We do this instead of instantiating a new webDriver so that we can create its bean as a singleton and reduce the amount of time we spend creating/tearing down our browser.
Next lets look at our post-deployment configuration.
@Category(SitIntegrationTest.class)
@TestExecutionListeners({DependencyInjectionTestExecutionListener.class})
@ActiveProfiles({"TEST_SIT"})
@ContextConfiguration(classes = {SitIntegrationTestConfig.class, SeleniumTestConfig.class})
public class AbstractSitSeleniumTest extends AbstractSeleniumTest {
}
Again similar to our pre-deployment except for different spring profiles and configuration classes. And this time we aren’t dealing with setting up the database so all of that configuration is gone.
Lets take a look at the new configuration class
@Configuration
@Profile("TEST_SIT")
public class SitIntegrationTestConfig {
@Bean(destroyMethod = "close")
public DataSource dataSource() {
final BasicDataSource basicDataSource = new BasicDataSource();
basicDataSource.setDriverClassName("oracle.jdbc.OracleDriver");
basicDataSource.setUsername("user");
basicDataSource.setPassword("password");
basicDataSource.setUrl("url");
basicDataSource.setMaxActive(-1);
return basicDataSource;
}
@Bean
public JdbcOperations jdbcOperations() {
return new JdbcTemplate(dataSource());
}
@Bean
@Qualifier("userNumber")
@Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE)
public String userNumber() {
return jdbcOperations().queryForObject("select sit_user_seq.nextval from dual", String.class);
}
}
Here we setup a connection to a database where we have a sequence object created. Before each test our base class will pull a new userNumber bean, which will return the next number form the sequence, and add it to our username so that we can create new users in the live system for our tests without needing to update the username everytime the tests are run.
Finally remember, that the setup method on the base class can be overriden to change this default creating/loging in/etc behavior in our setup() method. This can be useful when creating helper scripts that create X amount of users in the external system for manual testing and other things.
Page Pattern Support
It’s a recommend practice to use the page pattern for selenium. I won’t be going into the pattern itself, but see Page Objects for an explanation from the selenium developers.
We are going to look at a small bit of code used to support this pattern within our tests. You may have seen this object in our selenium test earlier
/**
*
* @author stephen.garlick
* @author linsei.berman
*
*/
@Page
public class ManifestPage {
@FindBy(id = "upload-file-button")
private WebElement submitButton;
@FindBy(id = "file-select")
private WebElement fileSelect;
@FindBy(id = "fileinput")
private WebElement fileUpload;
@FindBy(id = "cancel-file-button")
private WebElement cancelButton;
@FindBy(id = "file-error")
private WebElement fileError;
@FindBys(value = { @FindBy(id = "history-body"), @FindBy(tagName = "tr") })
private List<WebElement> manifestHistory;
@FindBy(xpath = "//*[@id='environment']/option[1]")
private WebElement temEnvironmentOption;
@FindBy(xpath = "//*[@id='environment']")
private WebElement environmentOptions;
public boolean isFileSelectDisplayed() {
return fileSelect.isDisplayed();
}
public ManifestPage selectFile(final String filePath) {
fileUpload.sendKeys(filePath);
return this;
}
public ManifestPage submit() {
submitButton.click();
return this;
}
public int getNumberOfEnvironmentOptions() {
return new Select(environmentOptions).getOptions().size();
}
public ManifestPage cancel() {
cancelButton.click();
return this;
}
public boolean isFileErrorDisplayed() {
return fileError.isDisplayed();
}
public int getManifestHistorySize() {
return manifestHistory.size();
}
public String getTEMEnvironmentText() {
return temEnvironmentOption.getText();
}
}
and the Page annotation which uses the spring @Component annotation so we can register them as beans by classpath scanning
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Component
public @interface Page {
}
and the PageBeanPostProcessor where we check for the annotation on beans created and call PageFactory.initElements to configure the bean’s selenium annotated fields with our webDriver.
@Component
public class PageBeanPostProcessor implements BeanPostProcessor {
@Autowired
private WebDriver webDriver;
@Override
public Object postProcessBeforeInitialization(Object bean, String beanName)
throws BeansException {
if (bean.getClass().isAnnotationPresent(Page.class)) {
PageFactory.initElements(webDriver, bean);
}
return bean;
}
@Override
public Object postProcessAfterInitialization(Object bean, String beanName)
throws BeansException {
return bean;
}
}
Now we don’t need to worry about initializing our page beans in each test they are used and can inject them with spring.
Decoupling the Database
By default we develop against an oracle database. But this means that our test will need an oracle instance setup in advance before our test runs. To remove this need we use an in memory database called h2 which allows for oracle syntax. While h2 is fine for projects using ORMs like hibernate, it might not be the best option if you are using a lot of vendor specific features that h2 does not have compatibility for. Just remember that when making the decision to use h2 or not.
We use maven to spawn our h2 database as a tcp server so that our websphere instance and tests can both connect to it while running in different jvms. Let’s take a look at our parent pom.
<profile> <id>h2-database</id> <activation> <property> <name>db</name> <value>h2</value> </property> </activation> <properties> <flyway.url>jdbc:h2:tcp://localhost:8082/mem:test;MODE=Oracle;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</flyway.url> <flyway.user>sa</flyway.user> <flyway.password>sa</flyway.password> <database.datasource.class>org.h2.jdbcx.JdbcDataSource</database.datasource.class> <database.driver.jar>h2-1.3.175.jar</database.driver.jar> </properties> </profile>
First we have a profile that changes all the flyway connection settings to that of our h2 database.
<plugin>
<groupId>com.btmatthews.maven.plugins.inmemdb</groupId>
<artifactId>inmemdb-maven-plugin</artifactId>
<version>1.4.2</version>
<configuration>
<monitorPort>11527</monitorPort>
<monitorKey>inmemdb</monitorKey>
<daemon>true</daemon>
<type>h2</type>
<port>8082</port>
<database>test</database>
<username>${flyway.user}</username>
<password>${flyway.password}</password>
</configuration>
<dependencies>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>${h2.version}</version>
</dependency>
</dependencies>
<executions>
<execution>
<id>start-db</id>
<goals>
<goal>run</goal>
</goals>
<phase>pre-integration-test</phase>
</execution>
<execution>
<id>stop</id>
<goals>
<goal>stop</goal>
</goals>
<phase>post-integration-test</phase>
</execution>
</executions>
</plugin>
Here is our plugin configuration for spawning the h2 database on our pre-integration-test phase and stopping it in our post-ingration-test phase.
And finally with our h2 database spawned we can use the flyway plugin to do the initial migration
<plugin>
<groupId>com.googlecode.flyway</groupId>
<artifactId>flyway-maven-plugin</artifactId>
<version>${plugin-version.flyway}</version>
<executions>
<execution>
<phase>pre-integration-test</phase>
<goals>
<goal>clean</goal>
<goal>init</goal>
<goal>migrate</goal>
</goals>
<configuration>
</configuration>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc6</artifactId>
<version>${ojdbc6.version}</version>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>${h2.version}</version>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>db</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
<configuration>
</configuration>
</plugin>
Now our database is up and schema migrated and we are ready to deploy to our websphere server and start running our selenium integration tests via the maven failsafe plugin.
Bootstrapping the Websphere Liberty Profile Server
Now that we’ve gotten a database up and migrated we need a way to setup our test server, in this case websphere liberty profile, so that we can deploy the application and let our selenium tests run.
Again we are going to our pom.xml to
<plugin>
<groupId>com.ibm.websphere.wlp.maven.plugins</groupId>
<artifactId>liberty-maven-plugin</artifactId>
<version>1.0</version>
<executions>
<execution>
<id>pre-integration-setup</id>
<phase>pre-integration-test</phase>
<goals>
<goal>start-server</goal>
<goal>deploy</goal>
</goals>
</execution>
<execution>
<id>post-integration-setup</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop-server</goal>
</goals>
</execution>
</executions>
<configuration>
<assemblyArtifact>
<groupId>com.company</groupId>
<artifactId>wlp-test-server</artifactId>
<version>1.1</version>
<type>zip</type>
</assemblyArtifact>
<configFile>${project.build.directory}/test-classes/server.xml</configFile>
<appArchive>${project.build.directory}/webapp.war</appArchive>
<timeout>60</timeout>
<verifyTimeout>60</verifyTimeout>
</configuration>
</plugin>
This plugin allows us to use start up websphere liberty profile server and deploy our war file automatically from maven. We’ve packaged up the server as a maven artifact and deploy it to a private repo; This server includes any necessary provided dependencies in its /lib folder before being zipped up.
Next the plugin allows us to use a server.xml file which is a configuration file for websphere. We have the following server.xml template that gets processed by maven during our build to set the correct database (h2 or oracle).
<server description="new server">
<!-- Enable features -->
<webContainer deferServletLoad="false" />
<featureManager>
<feature>jsp-2.2</feature>
<feature>servlet-3.0</feature>
<feature>localConnector-1.0</feature>
<feature>jdbc-4.0</feature>
<feature>jndi-1.0</feature>
<feature>beanValidation-1.0</feature>
<feature>jpa-2.0</feature>
</featureManager>
<httpEndpoint host="0.0.0.0" httpPort="9080" httpsPort="9443"
id="defaultHttpEndpoint" />
<applicationMonitor updateTrigger="mbean" />
<dataSource id="db" jndiName="jdbc/datasource">
<jdbcDriver libraryRef="driverLib" javax.sql.DataSource="${database.datasource.class}"/>
<properties URL="${flyway.url}"
password="${flyway.user}" user="${flyway.user}" />
</dataSource>
<library id="driverLib">
<fileset dir="${wlp.install.dir}/lib" includes="${database.driver.jar}" />
</library>
<jndiEntry id="profile" jndiName="spring.profiles.active"
value="DEV_SECURITY,DEV_SERVICES,CONTAINER_DB" />
</server>
You’ll notice properites like database.datasource.class which were defined in our pom.xml.
Now all of our tests can be run during our build and we have no need to manually setup any databases or web servers to run our integration tests on.
Closing Comments
Now we have ways to easily developer each layer of our tests and have them run during our maven build with a single command. From here we could easily create CI jobs in jenkins to handle testing, reporting and deployments for us so that we can focus on developing our app and tests.
Naming Stuff is Hard 