Testing Philosophy

Hi All! I'm attaching here a draft testing philosophy, which was the output of part of the ODK summit here in Seattle this week. Questions, comments and any feedback is welcome! If we like this, perhaps we can codify it in the repo itself in a .md? Next steps would be to formalize the manual testing pre-release procedures and a testing philosophy for the other apps, as well as whether cross-app testing is desired. I'm also working on an example connectedAndroidTest for end-to-end verification of the xml-to-android-UI-component pipeline.

Draft Testing philosophy

When we introduce new code or functionality, we should strive to add the best and fastest test coverage we can, while being pragmatic about cost/benefit.

"Best and fastest" can be defined by the testing pyramid definition here: https://martinfowler.com/bliki/TestPyramid.html

With the addition of manual tests at the very top, above UI tests (slowest, most expensive)

The below detailed draft philosophy was discussed only in the context of collect...further discussions are necessary for the other ODK applications!

Unit Tests

Overall goal
The goal shouldn’t be 100% unit test coverage, but rather a pragmatic approach to test those atomic pieces of complex functionality where there may be many edge cases

When to unit test
As an example, a “util” package full of small static methods should have 100% coverage with a battery of test cases covering any edge cases we can come up with

When not to unit test
We especially shouldn’t worry about unit tests where we know code will be refactored or removed

Robolectric Tests

A testing framework for running “android-like” tests without requiring a real or virtual android device to be connected

Overall goal
Where version discrepancies are unlikely to cause inconsistent behavior and test results, but where unit testing is not possible (the component to be tested needs Android libraries), robolectric tests are necessary

Espresso Tests

Automated testing that requires a connected real or virtual device to be connected

Overall goal
These are integration tests, and at a minimum should cover main happy path interactions:

  • Form parsing/rendering
  • Form discovery
  • Form submission

Additionally, where known regressions have occurred in the past, adding tests is useful (e.g., a button is no longer visible in a release, add a test to very the button is visible for subsequent releases).

Manual Tests

Overall goal
We should codify a manual testing plan (steps, expected outcome) to be run before each release