Automated UI Testing with Jenkins CI

For years, Jenkins has been the software of choice for companies offering Continuous Integration (CI) servers. Simplicity, a wide range available plugins, and a free software license make Jenkins a very appealing choice.

Jenkins is by no means a new offer on the market, that’s why its capabilities can be somewhat surprising. Although we’ve been using CI servers in Macoscope for quite some time now, using them for automating UI tests allowed us to uncover a whole new set of their abilities.

jenkins

Running Tests on Devices Using Jenkins CI

After writing a set of UI tests, we decided to run them directly from the level of the CI server. When we tried to run the tests using Travis CI we were kind of disappointed — the instructions were executed at a very slow pace, some of the view elements were not found, so we decided to give another server a shot. We needed a fast CI that would allow us to run tests either on physical devices or fast simulators which we’ve already used over the course of writing the tests. After deliberating for a while, we went with Jenkins. The same tests that Travis CI executed in a very unstable manner were handled very quickly and in a very stable manner after being migrated to Jenkins, while their results were located directly in Jenkins.

The configuration of a project in CI has to allow for building a project, installing an application directly on the device, and reporting of test results.

Building an Android Project and Installing It on a Device

To build the project and install the application directly on the device you need to put the command to uninstall and reinstall the application in the execute shell field. The command has to be typed in separately for the production and test versions of the app. As you can see, the command calling Gradle contains instructions to execute unit tests. We decided against separating them because then Jenkins will bring any irregularities to our attention much quicker in the event of an app failing unit tests.

To set the build command, follow these instructions:

  1. In the Configure perspective, find the Build section and then locate the Execute shell field
  2. In the Command field, type in the commands that will reinstall the application and execute the test suite on the device

    ~/Library/Android/sdk/platform-tools/adb uninstall com.package.beta || true
    ~/Library/Android/sdk/platform-tools/adb uninstall com.package.beta.test || true
    ./gradlew testBetaDebugUnitTest connectedBetaDebugAndroidTest

image02

Improving Test Report Clarity Using the Jenkins JUnit Plugin

Manual inspection of test results was inconvenient and laborious, so we decided to automate result reporting.

Automatic tests are launched using the JUnit library. If you want to visualize the results in Jenkins CI, you need to use the JUnit Plugin.

To add the capability to create test reports and configure it properly, follow these instructions:

  1. In Jenkins’ main perspective, locate and click the Manage Jenkins button
  2. In the administration panel, locate and open the Manage Plugins section
  3. From the list of available plugins, select and install JUnit Plugin
  4. Open the Configure perspective of the project
  5. In the Post-build Actions field, add the Publish JUnit test result report action
  6. In the Test report XMLs field, type in the path to the location of the test results file

    app/build/outputs/androidTest-results/connected/flavors/BETA/*.xml
    
image01

A detailed report allows you to check which tests were executed and what their results were. In case any of them fails, the reports will let you bring up individual errors and establish their causes. image04

The JUnit Plugin creates a visualization of data from previous test sessions and enables us to analyze how the results changed over time. image06

Staying Up-to-Date with Slack Notifications

Automated result reporting allows us to expedite the process of removing bugs from an application. We decided to integrate Jenkins with Slack and it quickly turned out that it was a very convenient solution. The integration reduces the possibility of us missing the failure of one or more tests.

To integrate Jenkins with Slack, follow these instructions:

  1. In Jenkins’ main perspective, locate and click the Manage Jenkins button
  2. In the administration panel, locate and open the Manage Plugins section
  3. From the list of available plugins, select and install the Slack Notification Plugin
  4. Open the Configure perspective of the project
  5. In the Post-build Actions field, add the Slack Notifications action
  6. In the upper part of the Configure perspective, locate plugin settings and select the types of events that will trigger posting to the appropriate Slack channel
  7. Open Advanced…
  8. Select the Include Custom Message field and type in the link to the test report Android Report:
    http://bartek.local:8080/job/package%20UI%20Tests/ws/app/build/reports/androidTests/connected/flavors/BETA/index.html
  9. Open the desktop or web version of the Slack app and select the Customize Slack option
  10. From the side panel choose Customize Apps
  11. Locate and add the Jenkins CI plugin
  12. Open the plugin administration view and select Add Configuration
  13. In the Post to Channel field, select the name of the channel that will display the notifications and confirm the selection using the Add Jenkins CI integration
  14. Copy the value of the generated Token
  15. Move to the Slack Notifications section in the Edit View in Jenkins
  16. In the Team Domain field, type in your Slack domain name
  17. In the Integration Token field, paste the value of the Token generated earlier
  18. Check whether the integration works as intended using the Test Connection button
image05

The Slack notification quickly informs the user about the status of the performed tests and presents a link to a test report which contains detailed data on the execution of each individual test. image03

Periodic Triggering of UI Tests

UI testing is a time-consuming process and the tests themselves can be unstable sometimes. We might consider giving up running UI tests for every build and switching to running them separately at a predetermined time. Running UI tests during the build process would unnecessarily draw out the build time by a couple of minutes.

To create a rule establishing the timeframes for test execution, follow these instructions:

  1. Launch project view
  2. Open the Configure section
  3. Locate the Build Triggers section and the Schedule field
  4. In the Schedule field, type in the rule for triggering the execution of the tests, e.g. 53 10 * * *
image00

Conclusion

The configurations described in this post allow us to examine test results at varying levels of detail. We demonstrated how you can improve your own QA process by launching tests automatically from the level of a Continuous Integration server. Using Jenkins for launching automatic tests has allowed us to improve the overall quality of the products we ended up shipping. This post does in no way exhaust the topic of using Jenkins CI for automatic testing, but provides a basic framework that will allow readers to start testing Android apps on their own. If you have any ideas on how to improve automatic testing, don’t hesitate and leave a comment below.

Got inspired? E-mail us and we’ll get in touch to find out how our design and development services can drive business value for you.

Display Calendar Events on a Watch Face with Android Wear
Swap Your Old-Fashioned Keychain for a Location-Aware App