Ona has been hired by a client to add support for push notifications into ODK Collect. We already have a list of requirements from the client that need to be implemented before a deadline. Since our intention is to make this feature available to everyone else, we'd appreciate feedback from the community.
Initial Requirements
Whenever a form schema, or any of its media files is updated on ODK Aggregate, a notification of the change should be sent to all devices with ODK Collect that have previous versions of the form. The notification message should be shown on the devices' notification centres.
There should be an "Auto-download form changes" toggle in ODK Collect's settings:
When the toggle is enabled, form schemas and media files will be auto-downloaded whenever a form update notification is received, and it's safe to update the form (the form is currently not opened).
When the toggle is disabled, the form schema and media files are not auto-download whenever a form update notification is received. However, a "New form version available" alert message is displayed whenever a user opens the form from the "Fill Blank Form" screen. The alert message should be dismissable.
An aggregate user should be able to send a free-form message to all devices that have a form downloaded in ODK Collect. The notification message should be shown on the device's notification centre, and should open up an activity with all received free-form messages arranged chronologically.
Proposed Implementation
We will initially add support for pushing notifications from Ona Data[7], rather than ODK Aggregate.
In order to deliver this feature to our client in good time, we've considered three approaches of adding the features to Collect. This is potentially temporary, as we wait for the community to give the necessary feedback:
Build a separate Android app that handles push notifications, and that 'talks' to Collect using broadcast messages. Depending on what the community decides, we will continue to maintain this as a separate app, or move all functionality to Collect.
Create a feature branch in ODK Collect's repo on GitHub. Continue working from there as the community fully specs out the feature for use by everyone else.
Implement the feature in our fork of ODK Collect[10]. Push the code to ODK Collect when the community fully specs out the feature for use by everyone else.
Firebase/GCM, Amazon SNS, and Message Queue Telemetry Transport (MQTT) have been considered as possible technologies/protocols for the push notification system. We propose to use MQTT. Key points on the protocol:
MQTT is an open protocol (vendor lock-in not possible).
MQTT is payload agnostic allowing clients to send JSON payloads, for instance. In comparison, similar messaging protocols like XMPP would require wrapping the JSON inside XML[6].
There are various client (including Android[1], Java[2], and Python[3]), and broker[4] implementations available.
MQTT client libraries are not considered "first-class citizens" on Android. For instance, MQTT client libraries are unable to wake the device from sleep. In comparison, Firebase is able to wake the device from sleep if a high-priority message is received[5].
Though not finalized, there's an issue on GitHub suggesting the structure of push notification payloads[8].
Key points from a discussion[9] we had on this feature request during the ODK Transition Planning workshop in Seattle:
Push notification payloads for form updates contain the form manifest, and not the update diff.
Should the "auto-download form update" toggle be a server level setting?
It'd be great to add a "release notes" field in the form manifest pushed when a form update is published.
Timeline
Implementation is set to begin in the second week of July 2017, and should take around five weeks to implement (based on the agreed upon timeline with the client).
Community Feedback
Some areas we are looking out for feedback from the community are:
What other requirements should we consider (but not necessarily implement immediately).
What technology to use for push notifications.
Help in beefing up the push notification spec for form updates, and free form messages.
Suggestions on what the community thinks is the best way of adding the feature to ODK Collect.
Great to see some movement on this again, thanks @adam.butler!
I am in strong agreement for polling to detect form updates. The polling frequency can be made configurable with a reasonable default that balances data usage and usefulness. As Adam mentions in his spec above, @Grzesiek2010 has already started a proof of concept of this at https://github.com/opendatakit/collect/pull/1700.
How to notify users of new versions
The spec has an open question regarding user notification of form updates with the following options:
Download the new version without informing the ODK Collect user
Showing an alert to inform the surveyor that a new version is available
Displaying a mark (a red •, for example) next to the form to indicate that a newer version is available
In "Get Blank Forms", forms with updates are already identified as of Collect v1.12. That sounds like 3. I would be in favor of also making both 1 and 2 possible. Automatic downloads could be configurable and off by default and notifications could be on by default. I don't think the notifications would be too disruptive and I would tend not to make them configurable.
Additionally, opening a form should trigger a version check if the device is online.
What to do with outdated versions of forms
Another piece that needs to be addressed is what to do with outdated forms. Currently, all unique formId and version combinations remain on the device and are displayed to the user. This can be quite confusing. I see two options:
only show the most recently added formId and version pair in the UI. Leave others on the device but hide them from users. Possibly add a configuration option to show them all (match current behavior).
remove outdated ones from the device. The big downside here is that filled forms connected to that formId and version combination could no longer be opened.
I prefer 1 for its simplicity and that it avoids bad states that 2 might make possible.
This does not address what to do when the formId and version stay the same but the form itself changes. This is allowed by certain servers that allow adding or removing questions without changing the version or can happen in Aggregate if a user deletes a form and then uploads a different one with the same formId and version combination. I think we should handle that case separately at a later date. This could be addressed by using form hash to uniquely identify forms rather than the formId and version combination but that's a deeper change.
On the form updates without version increment question, I would be tempted to do nothing. If admins are not changing version numbers, that's their lookout. Adding a form hash is duplicating functionality, and would probably lead to both user confusion and technical debt in the long term.
If no-one else chimes in, I'll update the issue with your additions.
Cool. @LN, @adam.butler I guess polling makes sense, especially since deploying the extra broker will definitely not work for some setups. I had a few questions though:
At what point do you think a device should stop polling the /formList endpoint (if that's what will be polled)? Or is it a good idea to constantly increase the time between polls after each request. I'm trying to avoid a situation where devices that were previously used in a survey are adding unnecessary load on the server.
Do you think letting RSS (or a similar protocol) inform on whether forms have updates is an overkill? I say RSS because it will still involve polling and can be baked into Aggregate (won't need setups to deploy an extra service).
I tend to think that a server should be able to cope with the load of sporadic polling: it's not like these requests require a lot of network traffic or disk writes. It would be good to get some real world numbers, but let's consider (what I would assume to be) a very popular form that is installed on 1000 instances of ODK Collect. Even if those instances are running Collect non-stop (which is of course unrealistic), we would be talking about 1.666 requests per second (assuming the proposed interval of 600 seconds). Each of those requests would have a response of a few (single figure) KB. Given that the average size of a webpage these days has ballooned to multiple MB, this seems to me to be a comparatively reasonable load.
Let me start by stating on record that I totally RSS. But in this case, I don't think there's much to gain from it. RSS readers pull feeds from servers, so using RSS would just mean polling for some XML, which is the same as what we would be doing by polling /formList directly. And using RSS would require additions to Aggregate (and maybe even the XForm spec) in order to serve the RSS-compliant XML files.
(One optimisation that did just occur to me while typing this: it would be good for the polling process on Collect to group forms by server. That way, if I have 5 forms from the same instance of Aggregate, Collect could hit /formList once and find out the latest versions of all forms on the server, instead of having to make 5 separate requests.)
Please do correct me if I'm wrong or have misunderstood anything, or if you have any more thoughts on this!
I'm a bit concerned because of deployments like ona.io where we expect tens of thousands of devices online and polling, but I guess we should test first and see if the load is worth worrying about. I've been going with the assumption that Collect will be polling even when in the background.
Cool. Yeah, it might be an overkill for now. As per the user story presented on this post though collect will also be checking for free-form messages. Figured it would have been nice to let a RSS handle both types of updates and not worry about having to poll the server twice for the two kinds of updates.
I’m a bit concerned because of deployments like ona.io where we expect tens of thousands of devices online and polling, but I guess we should test first and see if the load is worth worrying about.
Right, good point, I didn't consider multi-tenanted cloud installations. Would you be OK with running some ballpark stress tests? Maybe aggressive caching of formList could help?
I’ve been going with the assumption that Collect will be polling even when in the background.
I was actually thinking that, at least as an initial MVP (but also as a way of attenuating potential server load), it would be enough for polling to happen while the app is foregrounded. This is an imperfect implementation, but it might be "good enough" to begin with, and it saves having to send notifications while in a background state, which might be complicated? (I honestly don't know, offhand).
I was kind of hoping that background polling could be Notifications 2.0 functionality...
As per the user story presented on this post though collect will also be checking for free-form messages.
I think I'm missing something here - what kind of free-form messages do you mean?
In reducing what gets returned and only returning what is necessary, should we do a polling which only needs to result in new or updated forms? I understand this could mean an addition to the spec. ODK Collect and related clients would need to cache this information probably using ETag headers.
The freeform message could be, "A retraining session is scheduled for the 2nd ... at the local market. Please remember to bring the mobile power banks".
The idea here is to communicate with enumerators in the field who have downloaded a specific form. It could be updated instructions on how to approach a particular question.
That's a very interesting idea @Ukang_a_Dickson. But I feel like that would be a separate piece of functionality, that would need to take into account a lot of different issues (how to display the message, how to send the message on the server, how to designate the desired recipients for a message, etc.)
It might end up using a similar implementation to this, but I think it would be easier to treat it as a separate proposal.
I think the hope was that we could use the same functionality to tackle both, it could be a matter of deciding what event type does Collect need to be aware of, and how does it react in those events. One aspect of this would have been, if a form has been deleted or made inactive on the server, does Collect need to remove it from the device? Does the user get notified of that fact?
If the polling happens when the device is online and Collect is in the foreground, I agree with @adam.butler that this would provide desired behavior without significant load.
In fact, doesn't Enketo already poll periodically for updates? See @martijnr's comment above:
Clients could cache the formList XML and corresponding manifest XML and use either ETag or Last-Modified headers when making the polling requests. My preference would be to start simple to guarantee it works with existing servers and add optimizations if needed.
I would personally really like to focus on the form update case initially and then to expand iteratively from there. I think what @adam.butler has written out is tightly scoped and provides clear user benefit. Once it's in place and necessary optimizations are added then natural next questions would be:
what should happen if a form that used to be in the formList is no longer in the formList?
can/should polling be used for arbitrary text notifications?
We had two open issues from the TSC call that I wanted to discuss...
I propose we allow for background polling with off as the default and options for 15, 30, and 60 minutes. We should add some $RANDOM time internally so we don't hammer a campaign's servers when they have thousands of phones sitting in a room and the clock strikes 12.
Nice to have: Have Collect check for an update when a connection goes from unavailable to available. This is what we do with auto-send and it would be really nice here because a form could then be auto-downloaded as an enumerator comes into a connection. Again, we'll have to add some $RANDOM time to this trigger.
I propose that users who enable background checking and don't have automatic form downloads get one (and only one) Android notification informing them that a new version is available.
Nice to have: Auto-send shows an Android notification when a form is sent in the background. We do the same for when a form is downloaded in the background.
As of Collect v1.15.0, there is a setting in Form Management to turn on automatic checks on form version. If an update is available, a notification will be shown and if automatic downloads are turned on, the new version will be downloaded
Furthermore, only the latest version of a form is now shown in the user interface. To show every version, uncheck the "Hide old form versions" option shown above.
RSS. But in this case, I don't think there's much to gain from it. RSS readers pull feeds from servers, so using RSS would just mean polling for some XML, which is the same as what we would be doing by polling 