Hi All! I'm attaching here a draft testing philosophy, which was the output of part of the ODK summit here in Seattle this week. Questions, comments and any feedback is welcome! If we like this, perhaps we can codify it in the repo itself in a .md? Next steps would be to formalize the manual testing pre-release procedures and a testing philosophy for the other apps, as well as whether cross-app testing is desired. I'm also working on an example connectedAndroidTest for end-to-end verification of the xml-to-android-UI-component pipeline.
Draft Testing philosophy
When we introduce new code or functionality, we should strive to add the best and fastest test coverage we can, while being pragmatic about cost/benefit.
"Best and fastest" can be defined by the testing pyramid definition here: https://martinfowler.com/bliki/TestPyramid.html
With the addition of manual tests at the very top, above UI tests (slowest, most expensive)
The below detailed draft philosophy was discussed only in the context of collect...further discussions are necessary for the other ODK applications!
Unit Tests
Overall goal
The goal shouldn’t be 100% unit test coverage, but rather a pragmatic approach to test those atomic pieces of complex functionality where there may be many edge cases
When to unit test
As an example, a “util” package full of small static methods should have 100% coverage with a battery of test cases covering any edge cases we can come up with
When not to unit test
We especially shouldn’t worry about unit tests where we know code will be refactored or removed
Robolectric Tests
www.robolectric.org
A testing framework for running “android-like” tests without requiring a real or virtual android device to be connected
Overall goal
Where version discrepancies are unlikely to cause inconsistent behavior and test results, but where unit testing is not possible (the component to be tested needs Android libraries), robolectric tests are necessary
Espresso Tests
Automated testing that requires a connected real or virtual device to be connected
Overall goal
These are integration tests, and at a minimum should cover main happy path interactions:
- Form parsing/rendering
- Form discovery
- Form submission
Additionally, where known regressions have occurred in the past, adding tests is useful (e.g., a button is no longer visible in a release, add a test to very the button is visible for subsequent releases).
Manual Tests
Overall goal
We should codify a manual testing plan (steps, expected outcome) to be run before each release