Testing framework for Collect

Hi all,

Now that we have some basic CI in place, it seems like the next step
is to get a testing framework in place so we can start running tests
against pull requests.

Here again, I think it'd be great to get some feedback from the
community. Is Robolectric what most people use?

Thanks,

Yaw

I am a big fan of Robolectric for unit tests. I use Robolectric for unit
tests and I use Espresso for integration tests or when I actually need
tests to run on a real device (or emulator). This is a relatively common
setup. At Google, for instance, many Android projects use Robolectric for
unit tests and then also have integration tests.

For those not familiar with Robolectric, it allows Android code to be run
in the JVM. It does this by running against real AOSP source code and it
exposes more state than native Android to support more assertions. There
are some tradeoffs with this approach. Sometimes things run on Robolectric
but not Android or vice versa. I encountered a bug testing
PreferenceFragments, for example, where Android accepted a raw int value or
an org.odk.R resource value, but Robolectric accepted only the resource
value. (I can't remember now why this mattered, but it did.) There is also
generally a delay in getting support for the latest Android versions. The
Robolectric shadows for Nougat aren't out yet, for example. In general
these are soluble problems and for me the benefits far outweigh the
drawbacks.

The Robolectric homepage http://robolectric.org/ gives some examples of
what these tests look like, and their github
https://github.com/robolectric/robolectric has more information. My
Collect repackaging has Robolectric tests
https://github.com/srsudar/MamasDelRioAndroid/tree/master/app/src/test/java/org/mamasdelrio/android
for the components I added and demonstrates what it would look like to
incorporate it into a Collect build.

ยทยทยท On Thu, Sep 8, 2016 at 3:26 AM, Yaw Anokwa wrote:

Hi all,

Now that we have some basic CI in place, it seems like the next step
is to get a testing framework in place so we can start running tests
against pull requests.

Here again, I think it'd be great to get some feedback from the
community. Is Robolectric what most people use?

Thanks,

Yaw

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi Yaw,

We use Espresso, UI Automator, and the Android JUnit 4 framework for our
unit and integration tests. As a result, our tests must be run on a device
or an emulator.

Clarice

ยทยทยท On Thu, Sep 8, 2016 at 7:47 AM, Sam Sudar wrote:

I am a big fan of Robolectric for unit tests. I use Robolectric for unit
tests and I use Espresso for integration tests or when I actually need
tests to run on a real device (or emulator). This is a relatively common
setup. At Google, for instance, many Android projects use Robolectric for
unit tests and then also have integration tests.

For those not familiar with Robolectric, it allows Android code to be run
in the JVM. It does this by running against real AOSP source code and it
exposes more state than native Android to support more assertions. There
are some tradeoffs with this approach. Sometimes things run on Robolectric
but not Android or vice versa. I encountered a bug testing
PreferenceFragments, for example, where Android accepted a raw int value or
an org.odk.R resource value, but Robolectric accepted only the resource
value. (I can't remember now why this mattered, but it did.) There is also
generally a delay in getting support for the latest Android versions. The
Robolectric shadows for Nougat aren't out yet, for example. In general
these are soluble problems and for me the benefits far outweigh the
drawbacks.

The Robolectric homepage http://robolectric.org/ gives some examples of
what these tests look like, and their github
https://github.com/robolectric/robolectric has more information. My
Collect repackaging has Robolectric tests
https://github.com/srsudar/MamasDelRioAndroid/tree/master/app/src/test/java/org/mamasdelrio/android
for the components I added and demonstrates what it would look like to
incorporate it into a Collect build.

On Thu, Sep 8, 2016 at 3:26 AM, Yaw Anokwa yanokwa@nafundi.com wrote:

Hi all,

Now that we have some basic CI in place, it seems like the next step
is to get a testing framework in place so we can start running tests
against pull requests.

Here again, I think it'd be great to get some feedback from the
community. Is Robolectric what most people use?

Thanks,

Yaw

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

1 Like

I posted this response to Yaw in an email, but it occurs to me that he likely was hoping we'd send it out to the greater community!

We have kind of 4 pronged testing approach for our code currently:

  1. Pure Java Tests [JQuery, running on Jenkins] - For the code that runs on Android as well as on our REPL tools, server tools, etc. This covers most XForm/JavaRosa engine tests, as well as the CommCare engine tests, and run super fast.
  2. Android Mock Tests [Robolectric, running on Jenkins] - These are tests for Android functionality that are tested against "Android Code" through mocks/shadow objects provded by http://robolectric.org/ . These run pretty fast overall, but involve a lot of setup/teardown.
  3. "On Device" Regression Tests [Calabash, running on Amazon Device Cloud] - These are tests that are running the actual binary code against an actual device in real time with user-input level events through the calabash-android system. The device cloud lets us run them against real devices in a well managed way (that eliminates finicky ADB bullshit). The tests are sandboxed, so they don't have access to, say, the camera, and can only run against our APK (can't test integrations). These are slow as fuck (As you can guess).
  4. "On Device" Integration Tests [Calabash, running on a server in our closet] - These are similar to the previous tests, but since we are running on our own hardware we can do things like pull down the notifications bar, or execute ADB commands to perform management actions, or check the state on the server.

Robolectric is really great, but I will say it is super annoying how fast it turns over new versions, and how much time we spend re-connecting stuff when either it, or the android SDK's get big changes.

We are still fully rolling out our Callabash strategy, and it's a fantastic stand-in for manual QA/Regression/Integration testing, but we also haven't yet quite crossed the threshold where we make big changes and have to face the burden of updating the scripts a bunch to reflect changes in the UI, so it's hard to say how much we'll love it in a few years.

I will say that these tools are great, and much more useful than they were 2 years ago, but they still feel quite immature to me (the actual calabash scripting documentation is in the "if you want to know what you can do, inspect the source code!" state). This feels like a place where shops treat their internal testing stack as a proprietary advantage against other firms and don't want to document or build communities around them. There are enough people using them that I think they will still be around in a few years, but I honestly am not quite convinced their developers won't disappear in 9 months and never come back.

-Clayton

ยทยทยท On Thursday, September 8, 2016 at 6:26:48 AM UTC-4, Yaw Anokwa wrote: > Hi all, > > Now that we have some basic CI in place, it seems like the next step > is to get a testing framework in place so we can start running tests > against pull requests. > > Here again, I think it'd be great to get some feedback from the > community. Is Robolectric what most people use? > > Thanks, > > Yaw

Hello Everyone,

I am Lokesh Nandanwar, a second year undergraduate student at National
Institute of Technology and Science, Durgapur, India. I found the idea:
"ODK Collect - Adding system/operational testing" For GSoC'17 very
interesting.

I have a 1.5 years of experience in developing android apps. I have already
developed 3 Android applications. My recent app was Weather++, which is a
my version of weather forecast app. This app features Sync location,
Notification alert, Maps, etc. The app is available on the play store and
can be found here:Play Store
https://play.google.com/store/apps/details?id=com.lokeshkvn.android.weather.
The repository of the app can be found on github :Weatherpp
https://github.com/lokeshkvn/Weatherpp.

I also has little experience in Functional Testing(Unit testing) of Android
Applications as I have implemented it in my application Github repo :
Sunshine <https://github.com/lokeshkvn/Sunshine-Version .>

I have also worked in front end Web Development for bulding website for my
Institute's Robotics Club. Github Repo :Robocell
https://github.com/lokeshkvn/MY-ROBOCELL

My Github profile : https://github.com/lokeshkvn

Should I start discussing the idea ?

Thanks & Regards,

Lokesh Nandanwar,
National Institute of Technology, Durgapur, India

ยทยทยท > >

Clarice,

Any reason why you haven't added Roboelectric to the mix?

Yaw

ยทยทยท On Mon, Sep 12, 2016 at 5:44 AM, clarice larson wrote: > Hi Yaw, > > We use Espresso, UI Automator, and the Android JUnit 4 framework for our > unit and integration tests. As a result, our tests must be run on a device > or an emulator. > > Clarice > > On Thu, Sep 8, 2016 at 7:47 AM, Sam Sudar wrote: >> >> I am a big fan of Robolectric for unit tests. I use Robolectric for unit >> tests and I use Espresso for integration tests or when I actually need tests >> to run on a real device (or emulator). This is a relatively common setup. At >> Google, for instance, many Android projects use Robolectric for unit tests >> and then also have integration tests. >> >> For those not familiar with Robolectric, it allows Android code to be run >> in the JVM. It does this by running against real AOSP source code and it >> exposes more state than native Android to support more assertions. There are >> some tradeoffs with this approach. Sometimes things run on Robolectric but >> not Android or vice versa. I encountered a bug testing PreferenceFragments, >> for example, where Android accepted a raw int value or an org.odk.R resource >> value, but Robolectric accepted only the resource value. (I can't remember >> now why this mattered, but it did.) There is also generally a delay in >> getting support for the latest Android versions. The Robolectric shadows for >> Nougat aren't out yet, for example. In general these are soluble problems >> and for me the benefits far outweigh the drawbacks. >> >> The Robolectric homepage gives some examples of what these tests look >> like, and their github has more information. My Collect repackaging has >> Robolectric tests for the components I added and demonstrates what it would >> look like to incorporate it into a Collect build. >> >> >> >> >> >> On Thu, Sep 8, 2016 at 3:26 AM, Yaw Anokwa wrote: >>> >>> Hi all, >>> >>> Now that we have some basic CI in place, it seems like the next step >>> is to get a testing framework in place so we can start running tests >>> against pull requests. >>> >>> Here again, I think it'd be great to get some feedback from the >>> community. Is Robolectric what most people use? >>> >>> Thanks, >>> >>> Yaw >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "ODK Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to opendatakit-developers+unsubscribe@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "ODK Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit-developers+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "ODK Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to opendatakit-developers+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.

A recent Collect PR adding tests (woohoo!) would benefit from a decision on
whether or not to use Robolectric or just instrumentation tests:

Specifically look at the TextUtils.java file there to see the discussion.

I would vote to use Robolectric. If that is agreed as as reasonable
decision, I can put together a PR adding it with a basic example.

If not, we'll have to choose something else to allow the PR author to
proceed.

ยทยทยท On Fri, Sep 16, 2016 at 12:53 PM, wrote:

I posted this response to Yaw in an email, but it occurs to me that he
likely was hoping we'd send it out to the greater community!

We have kind of 4 pronged testing approach for our code currently:

  1. Pure Java Tests [JQuery, running on Jenkins] - For the code that runs
    on Android as well as on our REPL tools, server tools, etc. This covers
    most XForm/JavaRosa engine tests, as well as the CommCare engine tests, and
    run super fast.
  2. Android Mock Tests [Robolectric, running on Jenkins] - These are tests
    for Android functionality that are tested against "Android Code" through
    mocks/shadow objects provded by http://robolectric.org/ . These run
    pretty fast overall, but involve a lot of setup/teardown.
  3. "On Device" Regression Tests [Calabash, running on Amazon Device Cloud]
  • These are tests that are running the actual binary code against an actual
    device in real time with user-input level events through the
    calabash-android system. The device cloud lets us run them against real
    devices in a well managed way (that eliminates finicky ADB bullshit). The
    tests are sandboxed, so they don't have access to, say, the camera, and can
    only run against our APK (can't test integrations). These are slow as fuck
    (As you can guess).
  1. "On Device" Integration Tests [Calabash, running on a server in our
    closet] - These are similar to the previous tests, but since we are running
    on our own hardware we can do things like pull down the notifications bar,
    or execute ADB commands to perform management actions, or check the state
    on the server.

Robolectric is really great, but I will say it is super annoying how fast
it turns over new versions, and how much time we spend re-connecting stuff
when either it, or the android SDK's get big changes.

We are still fully rolling out our Callabash strategy, and it's a
fantastic stand-in for manual QA/Regression/Integration testing, but we
also haven't yet quite crossed the threshold where we make big changes and
have to face the burden of updating the scripts a bunch to reflect changes
in the UI, so it's hard to say how much we'll love it in a few years.

I will say that these tools are great, and much more useful than they were
2 years ago, but they still feel quite immature to me (the actual calabash
scripting documentation is in the "if you want to know what you can do,
inspect the source code!" state). This feels like a place where shops treat
their internal testing stack as a proprietary advantage against other firms
and don't want to document or build communities around them. There are
enough people using them that I think they will still be around in a few
years, but I honestly am not quite convinced their developers won't
disappear in 9 months and never come back.

-Clayton

On Thursday, September 8, 2016 at 6:26:48 AM UTC-4, Yaw Anokwa wrote:

Hi all,

Now that we have some basic CI in place, it seems like the next step
is to get a testing framework in place so we can start running tests
against pull requests.

Here again, I think it'd be great to get some feedback from the
community. Is Robolectric what most people use?

Thanks,

Yaw

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

1 Like

Hi Lokesh,

Join the #gsoc channel in the Slack (http://slack.opendatakit.org) and
meet your fellow students there!

As to the project, please put some of your ideas for testing at
https://github.com/opendatakit/collect/issues/392.

Thanks,

Yaw

ยทยทยท On Sat, Mar 4, 2017 at 3:54 AM, Lokesh Nandanwar wrote: > Hello Everyone, > > I am Lokesh Nandanwar, a second year undergraduate student at National > Institute of Technology and Science, Durgapur, India. I found the idea: "ODK > Collect - Adding system/operational testing" For GSoC'17 very interesting. > > I have a 1.5 years of experience in developing android apps. I have already > developed 3 Android applications. My recent app was Weather++, which is a my > version of weather forecast app. This app features Sync location, > Notification alert, Maps, etc. The app is available on the play store and > can be found here:Play Store. The repository of the app can be found on > github :Weatherpp. > > I also has little experience in Functional Testing(Unit testing) of Android > Applications as I have implemented it in my application Github repo : > Sunshine > > I have also worked in front end Web Development for bulding website for my > Institute's Robotics Club. Github Repo :Robocell > > My Github profile : https://github.com/lokeshkvn > > Should I start discussing the idea ? > > Thanks & Regards, > > Lokesh Nandanwar, > National Institute of Technology, Durgapur, India >> >> > > -- > You received this message because you are subscribed to the Google Groups > "ODK Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to opendatakit-developers+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.

The vanilla JUnit tests are fine for simpler unit tests at the bottom
layers of an app (utility classes).

Roboelectric and Mockito are better for subsystem testing in the middle
layers of an app, where you have to mock interfaces below the subsystem.
However, because we have a high rate of change in our interfaces,
maintaining subsystem tests for the middle layers is very expensive and not
currently an efficient use of developer time.

Instead, most of our tests involve some type of integration or end-to-end
testing, which is ideal for detecting regressions as we shift and adjust
the interfaces within. Running androidTest suites solves that
integration-testing and end-to-end regression testing need.

e.g., for ODK Tables and ODK Survey, much of the functionality relies on a
calls into the services layer provided by ODK Services.

ยทยทยท On Fri, Sep 16, 2016 at 6:23 AM, Yaw Anokwa wrote:

Clarice,

Any reason why you haven't added Roboelectric to the mix?

Yaw

On Mon, Sep 12, 2016 at 5:44 AM, clarice larson clarlars@gmail.com wrote:

Hi Yaw,

We use Espresso, UI Automator, and the Android JUnit 4 framework for our
unit and integration tests. As a result, our tests must be run on a
device
or an emulator.

Clarice

On Thu, Sep 8, 2016 at 7:47 AM, Sam Sudar sudar.sam@gmail.com wrote:

I am a big fan of Robolectric for unit tests. I use Robolectric for unit
tests and I use Espresso for integration tests or when I actually need
tests
to run on a real device (or emulator). This is a relatively common
setup. At
Google, for instance, many Android projects use Robolectric for unit
tests
and then also have integration tests.

For those not familiar with Robolectric, it allows Android code to be
run
in the JVM. It does this by running against real AOSP source code and it
exposes more state than native Android to support more assertions.
There are
some tradeoffs with this approach. Sometimes things run on Robolectric
but
not Android or vice versa. I encountered a bug testing
PreferenceFragments,
for example, where Android accepted a raw int value or an org.odk.R
resource
value, but Robolectric accepted only the resource value. (I can't
remember
now why this mattered, but it did.) There is also generally a delay in
getting support for the latest Android versions. The Robolectric
shadows for
Nougat aren't out yet, for example. In general these are soluble
problems
and for me the benefits far outweigh the drawbacks.

The Robolectric homepage gives some examples of what these tests look
like, and their github has more information. My Collect repackaging has
Robolectric tests for the components I added and demonstrates what it
would
look like to incorporate it into a Collect build.

On Thu, Sep 8, 2016 at 3:26 AM, Yaw Anokwa yanokwa@nafundi.com wrote:

Hi all,

Now that we have some basic CI in place, it seems like the next step
is to get a testing framework in place so we can start running tests
against pull requests.

Here again, I think it'd be great to get some feedback from the
community. Is Robolectric what most people use?

Thanks,

Yaw

--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com