Continuous Build, Deploy & Test

By Srikanth Ramamurthy

Introduction

Its 9:00PM today. Yet another project’s release day; the application needs to be released to the customer. Build engineer is struggling to make a clean build of the application. Programmers are doing last minute fixes. Testing group is waiting for a proper build so that they can certify the application before the release. Project manager is in deep thoughts thinking what to say to the client if the release doesn't happen.

Finally the release happens at mid-night. Come next day, the first things you see in your mailbox are the bad feedback from the customer about the release you made last night. The feedback could vary from missing features to bad fixes to regression with previous release to missing database scripts to wrong build and what not.

This is a typical scenario which is seen / heard across projects, across companies.

Now how about the below scenario:

Programmers do a code freeze the previous day. The build engineer gives a proper build to the testing group at 9:00AM on the release day. The testing group certifies the application for release by 2:00PM. The project manager compiles the release notes and ships the application by 3:00PM to the customer. The entire team goes for an evening movie show at 5:00PM. Come next day, the first things you see in your mailbox are the positive feedback from the customer about the release you made the previous day.

You may wonder whether this is a dream or a reality. But yes, this is a reality at Trigent except that we don’t go to movies on every release we make.

Trigent achieves this by using one of its best practices “Continuous build, deploy & test”. This paper talks about the practice we follow, the tools & methodologies we use in making this dream come true.

Test First Code Next

Before writing a single line of code, programmer designs and writes test cases to test his yet to be developed program. He wears a tester's cap which gives an out side view to the functionality to be implemented. This helps the programmer in understanding the feature better and making the logic foolproof.

These test cases helps in identification of pre-state data to be used for the creation of test database and also results in generation of test data to be used to test a particular program.

The test data are represented in xml format. These test xml are defined according to Trigent's proprietary xml schema. The test xml also identifies the test pass and test failure conditions.

The main goal of this activity is not only to come up with good test cases but also in making these test cases as regression test suites. This enables the projects to run the tests on the application as many times as required and helps in identification of any regression in the functionality.

Automatic Code Review

Programmers have their own way of coding. Similarly code reviewers have their own way of reviewing the code. On top of that, some coding issues could be trivial and may not be caught in the code reviews.  Manually enforcing a good coding practice is a nightmare and almost impossible when the code base becomes very large.

Trigent achieves uniformity in the code by following automatic code review process. This process prevents syntactically bad code getting into the source repository. To enable this, Trigent uses below open source tools:

Hammurapi

This tool from http://www.hammurapi.biz is an open source code inspection tool. This tool checks the source code for compliance with a pre-defined set of rules or best practices.
Trigent has selectively chosen some of the rule sets provided by this tool.

FindBugs

This tool from http://findbugs.sourceforge.net is a tool to find bugs in Java programs. It looks for instances of "bug patterns" --- code instances that are likely to be errors. FindBugs uses static analysis to look for general code quality problems.

EMMA

This tool from http://emma.sourceforge.net is an open-source toolkit for measuring and reporting the Code coverage. Trigent uses this tool to measure the effectiveness of the test cases. If the code coverage is below 60% then the test cases are enhanced to increase the coverage.

Trigent uses these tools in two ways:

1. Integration with the build process. In this case, these tools are run against the code during the build process and helps in early detection of the code problems.
2. Self-review. In this case, each programmer runs these tools against his code to find and fix the problems earlier on before putting his code into the source repository.

Automatic Build & Test

Trigent follows an automatic build & test process. The whole process executes without manual intervention. This enables the projects to have ready to ship code that are 100% reviewed & tested at any given point of time. This process helps in continuous code integration resulting in reliable builds of the application.

This diagram depicts Trigent’s automatic build process:

Tools used

Below are the tools used in the automatic build process:

1. CruiseControl (http://cruisecontrol.sourceforge.net/) is a framework for continuous build process.
2. Apache ANT (http://ant.apache.org/) as a build tool
3. Trigent's proprietary Test Driver to execute the test cases

Build Process

The build process is explained below:

1. Build engineer schedules the application build using CruiseControl. Usually the builds will be done during evening times.
2. At the scheduled time, the build process starts.
3. Latest copy of the source code will be obtained from the source repositories like CVS or Visual source safe.
4. Automatic code reviews will be done on the source code.
5. If any errors, the results will be emailed to the development team for further actions and the build process will fail.
6. On no errors, the application will be built.
7. The test database will be created using the pre-state data scripts.
8. The application gets deployed on the test application server.
9. The latest copy of the regression test suite xml files will be obtained from the source repository.
10. The test driver will be invoked and the test xml will be executed against the application deployed.
11. The test results will be logged to a log file.
12. The test result log file will be emailed to the testing & development teams for further actions.
13. The source code gets labeled appropriately in the source repository.

Test Process

Trigent uses a homegrown test driver. This driver is generic and can be used test the business and data access layer code. The test driver is implemented on top of jTestCase from http://jtestcase.sourceforge.net/

This diagram depicts Trigent's test process:

The test process is explained below:

1. Process starts with the translation of business use cases into test cases. Tester translates the use cases to appropriate test cases.
2. Tester converts these test cases to:
1. Pre-state data required for the test database creation
2. Test data in terms of test xml
3. Test xml and pre-state data will be put into the source repository
4. Tests will be executed by:
1. Programmers to test their modules
2. Testing group to system test the application
3. Automatic build process to validate the build
5. Tester will verify the test logs for errors.
6. Bugs will be logged into the bug tracker.
7. Development team will take further actions on the bugs found.

Conclusion

With "Continuous build, deploy and test" practice, Trigent has seen overall reduction in risk with respect to project delivery. Release process is streamlined. It has brought complete transparency into the build and testing process.

Programmers are disciplined, as the entire team will know if the build fails and the culprit behind that failure. The team is careful and makes sure that bad code does not get into source repository.

Code is integrated continuously and bugs are discovered earlier on instead of waiting till the last minute. Regression test suites are enhanced leading to reliable application resulting in timely delivery making the customer happy.

And of course, you get to go home early on the project release date with a smile on your face.