Collect messaging push notifications

[Jason_Rogena]

Edit: form update notifications are now provided by Collect through polling. See Notifications about form updates. A broader system for messaging push notifications may still be desired.

Ona has been hired by a client to add support for push notifications into ODK Collect. We already have a list of requirements from the client that need to be implemented before a deadline. Since our intention is to make this feature available to everyone else, we'd appreciate feedback from the community.

Initial Requirements

• Whenever a form schema, or any of its media files is updated on ODK Aggregate, a notification of the change should be sent to all devices with ODK Collect that have previous versions of the form. The notification message should be shown on the devices' notification centres.
• There should be an "Auto-download form changes" toggle in ODK Collect's settings:
  • When the toggle is enabled, form schemas and media files will be auto-downloaded whenever a form update notification is received, and it's safe to update the form (the form is currently not opened).
  • When the toggle is disabled, the form schema and media files are not auto-download whenever a form update notification is received. However, a "New form version available" alert message is displayed whenever a user opens the form from the "Fill Blank Form" screen. The alert message should be dismissable.
• An aggregate user should be able to send a free-form message to all devices that have a form downloaded in ODK Collect. The notification message should be shown on the device's notification centre, and should open up an activity with all received free-form messages arranged chronologically.

Proposed Implementation

We will initially add support for pushing notifications from Ona Data[7], rather than ODK Aggregate.

In order to deliver this feature to our client in good time, we've considered three approaches of adding the features to Collect. This is potentially temporary, as we wait for the community to give the necessary feedback:

1. Build a separate Android app that handles push notifications, and that 'talks' to Collect using broadcast messages. Depending on what the community decides, we will continue to maintain this as a separate app, or move all functionality to Collect.
2. Create a feature branch in ODK Collect's repo on GitHub. Continue working from there as the community fully specs out the feature for use by everyone else.
3. Implement the feature in our fork of ODK Collect[10]. Push the code to ODK Collect when the community fully specs out the feature for use by everyone else.

Firebase/GCM, Amazon SNS, and Message Queue Telemetry Transport (MQTT) have been considered as possible technologies/protocols for the push notification system. We propose to use MQTT. Key points on the protocol:

• MQTT is an open protocol (vendor lock-in not possible).
• MQTT is payload agnostic allowing clients to send JSON payloads, for instance. In comparison, similar messaging protocols like XMPP would require wrapping the JSON inside XML[6].
• There are various client (including Android[1], Java[2], and Python[3]), and broker[4] implementations available.
• MQTT client libraries are not considered "first-class citizens" on Android. For instance, MQTT client libraries are unable to wake the device from sleep. In comparison, Firebase is able to wake the device from sleep if a high-priority message is received[5].

Though not finalized, there's an issue on GitHub suggesting the structure of push notification payloads[8].

Key points from a discussion[9] we had on this feature request during the ODK Transition Planning workshop in Seattle:

• Push notification payloads for form updates contain the form manifest, and not the update diff.
• Should the "auto-download form update" toggle be a server level setting?
• It'd be great to add a "release notes" field in the form manifest pushed when a form update is published.

Timeline

Implementation is set to begin in the second week of July 2017, and should take around five weeks to implement (based on the agreed upon timeline with the client).

Community Feedback

Some areas we are looking out for feedback from the community are:

1. What other requirements should we consider (but not necessarily implement immediately).
2. What technology to use for push notifications.
3. Help in beefing up the push notification spec for form updates, and free form messages.
4. Suggestions on what the community thinks is the best way of adding the feature to ODK Collect.

References

1. https://github.com/eclipse/paho.mqtt.android
2. https://github.com/eclipse/paho.mqtt.java
3. https://github.com/eclipse/paho.mqtt.python
4. https://mosquitto.org/
5. https://developer.android.com/training/monitoring-device-state/doze-standby.html#using_fcm
6. https://xmpp.org/rfcs/rfc3921.html#stanzas
7. https://github.com/onaio/onadata
8. https://github.com/onaio/onadata/issues/866
9. https://docs.google.com/document/d/1uFiyGqijAoQ02k1rSmM7NyD4ikv_E4yXwEXwTSJAhuo/pub
10. https://github.com/onaio/collect

[LN]

Better support for form updates is a very common request (here for example) and I think it'd be great to have an ecosystem-wide answer to how they are communicated to enumerators and how the update process on Collect can be made easier than it currently is.

###How forms get updated
Before getting to the communications aspect which is what your post is mostly about, I'd like to also discuss the update mechanism. I know Ona allows more significant updates to forms than Aggregate does. Aggregate currently requires a different formID for any change that affects database schema (adding fields, renaming fields, etc).

For Ona users who use ODK Collect to fill out forms, how do you currently tell them to update when there's a major change to a form with the same ID? Do you tell them to delete the form and then redownload it? Do you give them any guidance around partially filled out instances that may still be on the device? In Collect, those partially filled instances would no longer be openable if the form definition changes. @martijnr I believe Enketo has a strategy for handling the situation where there are drafts and the form definition changes. Is this something that it would make sense for the ODK ecosystem to align on?

@Jason_Rogena, am I correct in saying that these things need to be figured out regardless of the way that Collect learns about form updates? In other words, even if you end up building an external app, you'll need to tell Collect to update the form and so Collect has to know what that means?

By the way, I tried to verify how all this currently works in my Ona demo account. I was able to replace my form in Settings > Edit Form Settings > Replace form without any errors. However, the new form doesn't show up in either Enketo or Collect.

###Push notifications
As I've been thinking about this, it seems like the push notifications story is much simpler in the context of a hosted service such as Ona than it is in the context of ad-hoc server installs like what happens with Aggregate. It makes sense for Ona, Kobo, etc to run message brokers but I don't really know how that would work for organizations that self-host. With self-hosting, polling may be more appropriate rather than pushing. Additionally/alternately, although I appreciate your point about vendor lock-in, relying on a hosted solution like Firebase Cloud Messaging might be more realistic in a self-host context. I'd like to be wrong on this so if anyone has a good sense of how Aggregate instances could be set up for pushing notifications and form updates using a protocol like MQTT, please share.

An additional question which I think is relevant to your implementation as well is when registration and de-registration for messages happens. For example, let's say I download form1 with Ona credentials sally/password. Presumably Collect would then register for any updates to that form. Let's say I then change the Collect Ona credentials to bob/password1. Do I get registered for notifications for bob's form1 and stop receiving notifications for sally's form1?

Is anyone else in the ecosystem interested in push notifications? Would it make sense for Enketo to receive them, @martijnr? Is it something Kobo @dorey or ELMO @Tom_Smyth would want to add?

I hope some of these questions help in your design. I don't yet know what the best approach is to adding the feature to Collect and I think the answers to the questions will help guide that.

[Jason_Rogena]

@LN

For Ona users who use ODK Collect to fill out forms, how do you currently tell them to update when there's a major change to a form with the same ID? Do you tell them to delete the form and then redownload it?

Ona Data supports updating a form's schema without changing the form id. The users don't have to delete the previous version of the form. Both the old and new form will appear on the list of available blank forms. We, however, recommend the user deletes the previous version only if all form submissions have been finalised (to avoid confusion when filling in a blank form since both the old and the new form versions will likely have the same name).

@LN

am I correct in saying that these things need to be figured out regardless of the way that Collect learns about form updates? In other words, even if you end up building an external app, you'll need to tell Collect to update the form and so Collect has to know what that means?

Yes you are.

@LN

By the way, I tried to verify how all this currently works in my Ona demo account. I was able to replace my form in Settings > Edit Form Settings > Replace form without any errors. However, the new form doesn't show up in either Enketo or Collect.

@LN sorry to hear that. Did the new form have the same form name. If so, it probably showed up on the list of downloadable forms, but with the same name as the old form. It's hard to distinguish if a form has been updated if you don't change the form name.

@LN

It makes sense for Ona, Kobo, etc to run message brokers but I don't really know how that would work for organizations that self-host. With self-hosting, polling may be more appropriate rather than pushing. Additionally/alternately, although I appreciate your point about vendor lock-in, relying on a hosted solution like Firebase Cloud Messaging might be more realistic in a self-host context.

Yeah, I think so too. One potential way of handling this is allowing Collect to handle both (similar to how form submissions can be done to Aggregate, or Google Drive).

@LN

An additional question which I think is relevant to your implementation as well is when registration and de-registration for messages happens.

Whenever a form is downloaded, Collect will subscribe to the MQTT topic corresponding to the form, and listen out for form update and free form messages on that topic.

If the ODK user is updated, Collect will unsubscribe from all the form topics that were previously subscribed to, but that the new user doesn't have access to.

MQTT allows for the broker to preserve messages if Collect ever goes offline, and send the messages (in the order the broker received them) whenever Collect comes back. This should take care of scenarios where Collect is used in setups with intermittent data connections.

Glad to answer more questions.

[martijnr]

@LN

Would it make sense for Enketo to receive them, @martijnr?

Not for form updates. I think the current polling-for-updates that Enketo does (by retrieving the hash published in the form list upon form load and at regular intervals) works well (if the server has some sensible policies about how a form can be updated wrt changes in the data structure).

[issa]

I'm glad someone is tackling this problem!

So, there is a part of me that would strongly prefer Firebase as it is the officially anointed answer from Google, with all the support and efficiency benefits that entails. On the other hand, Google's "officially anointed" answer on this subject has changed so many times I find it hard to trust them. If MQTT isn't a concern, I think it's a reasonable pick. About four years ago a very excited John Cohn (of IBM) told me of their use of MQTT for sensor communications in their IoT SmartCities initiative and told me to check it out. Given that endorsement and from what I know about it, independent of Android-contextual factors it addresses most of the problems we'd want it to.

My primary reservation is your note that "MQTT client libraries are unable to wake the device from sleep":

• Does this imply that the underlying TCP connection is severed sometime during device sleep? If so, would connection reëstablishment in unstable network conditions have a negative impact on experience relative to a more-native solution?
• Does your choice of MQTT regardless imply that you've done some research and found that people don't mind their messages not arriving until Collect itself is opened?
• Does this preclude scenarios wherein eg with auto-download on the device could essentially self-navigate a intermittent network situation without babysitting? That seems like a valuable benefit to me.

I'd be very interested to see a slightly more detailed look at whatever protocol+format you intend to lay on top of the transport service, MQTT or otherwise.

It would also be nice to brainstorm other future possible uses of a pubsub communication protocol and make sure we leave the door open for those.

On the ODK side, we'll have to figure out how we might work this feature in before we solve our own form-updates headache.

[Jason_Rogena]

@issa

Does this imply that the underlying TCP connection is severed sometime during device sleep? If so, would connection reëstablishment in unstable network conditions have a negative impact on experience relative to a more-native solution?

Yes, the device going to sleep means that the connection breaks. In such a scenario, MQTT pushes are only delivered when the device wakes from sleep. From what I've seen, the device is able to quickly reconnect to the broker, even with an unstable network connection.

@issa

Does your choice of MQTT regardless imply that you've done some research and found that people don't mind their messages not arriving until Collect itself is opened?

The MQTT library on Android uses a background service to connect to the broker. Collect therefore doesn't need to be opened for messages to be delivered.

@issa

Does this preclude scenarios wherein eg with auto-download on the device could essentially self-navigate a intermittent network situation without babysitting? That seems like a valuable benefit to me.

Message can be set to be resent by the broker at least once (QoS level 1), or exactly once (QoS level 2) when the client reconnects to the broker. If handled well, no user intervention will be required when in an intermittent network situation.

@issa

I'd be very interested to see a slightly more detailed look at whatever protocol+format you intend to lay on top of the transport service, MQTT or otherwise.

It'd be nice if you could take a look at the message structure proposed here.

[issa]

Sounds like you've got great answers for everything!

I took a look at the message structure; my only proposal is to add a toplevel version field on the message itself (eg not in the payload) in case that format changes. Otherwise it seems like having a toplevel fully qualified type and a freeform payload blob enables unlimited flexibility, which seems great. Hopefully we use it wisely.

[issa]

Any need for encryption?

[LN]

@issa Beyond the specific tech choices, what are your thoughts regarding @Jason_Rogena's question on where to do this development? I think that if there's a good way to do this without Ona needing to maintain a Collect fork, that would be great for everyone. Do you think there is a path for self-hosted servers to provide push notifications this way? Is that even an important consideration? As long as the feature doesn't add overhead to users who can't use it and doesn't dramatically increase the apk size, it seems it would be good to have in the app for servers that can support it.

I haven't thought about this deeply but my sense is that an external app would be impractical. It's unfortunate because I do like that it would force strict separation of concerns and allow the functionality to be pulled in to core at a later date if some refinement is needed. Some of my specific concerns are how it would know what user has successfully authenticated in Collect and what forms are available at any given moment.

@Jason_Rogena

@LN sorry to hear that. Did the new form have the same form name. If so, it probably showed up on the list of downloadable forms, but with the same name as the old form. It's hard to distinguish if a form has been updated if you don't change the form name.

I just tried again and it turns out I hadn't noticed the Upload button that is to the left of the filename so I was just clicking on Save. The upload is just spinning and spinning right now but I'll try again later and hopefully third time will be a charm.

[issa]

Well, if the service is on its own protocol and infrastructure (ie MQTT) there's no reason it wouldn't work on self-hosted servers provided they're on the net. So, any scenario in which a remote form update would work in the first place should work, yes? So I think yes, it's a valuable consideration but I think we have the answer to it.

As for fork or no-fork, no-fork is of course preferable. But I don't know enough about skeleton burial grounds in this code to really comment on the practicality and feasibility of such an approach. Separate app is probably asking more trouble than it's worth.

[LN]

Ah, for some reason I was thinking of the message broker as a separate entity from "the server" but that doesn't have to be the case.

[yanokwa]

This is a fantastic discussion and I hope my feedback can help move this forward!

My guess, based on similar feature requests that I've seen over time, is that your client wants forms that automatically update and also wants a way to message enumerators (about forms or about whatever). If my guess is correct, then we should be careful not to conflate the two.

My argument for a separate app

It seems very useful to have arbitrary messaging to an ODK Messages app and allow those messages to fire of specific-intents to Collect or whatever app you have installed. For example, a reminder message for all enumerators come to Site X on a certain date, could have an associated payload that opens up a calendar. You can't really do this with SMS or WhatsApp (although my gut says this has to exist somewhere).

And if that approach of a standalone app seems reasonable, then what we'd need on the Collect side is broadcast receivers that can fire off background tasks. One of those tasks could be to start a download of the new form or flag a form as outdated. This aligns very nicely with the work we have to do to clean up Collect's integration points anyway.

I don't know how this two app approach will work with ONA's auth, but given that Facebook and Facebook Messenger can share auth, doesn't seem impossible.

Changes to Collect to support form updates

For the download of a new form, we can build some safe defaults here for users. If there is a v2 of the form, you can download and stage it, force all finalized v1 submissions to be sent (or wait until that is done), delete v1 form, then present v2 to the user.

Also, independent of push notifications for updates, I do think Collect itself should poll for updates for those people who want updates but don't want to have messaging infrastructure.

MQTT vs GCM
I don't know these technologies well-enough to provide thoughtful feedback. If you need control, it seems like MQTT is great. That said, a lot of people use GCM (even if it's just to wake up an app to then poll), because message delivery is hard. I'd bias for the choice that is easiest to undo.

[LN]

@Jason_Rogena when I heard you talking about this, you mentioned that you thought implementing a separate app would not be that difficult. Given the conversation we've been having, do you still believe that to be the case? If so, that may be a really good way forward that lets you move quickly and that lets this conversation continue in parallel.

How does this conversation relate to this Ona Collect issue? Do you already have a prototype implementation in your fork? If so, would you be willing to share it? I think we could offer a different kind of feedback when we understood exactly what you are considering a form update, what the MQTT client integration actually looks like, etc.

[Trung]

I understand that if MQTT is implemented other than GCM (now it's FCM), there would be an em tra payload as you have to maintain a listener besides the system one.

[issa]

The major downside to multiple apps (besides user experience) is that it's yet one more thing which may demand coördinated/synchronized releases in the future. Do we want to add yet more to our burden on that front?

[Jason_Rogena]

We're planning on using TLS to encrypt messages between the broker and clients. Mosquitto, the broker we plan to use, allows us to accept connections from certificates signed by a specific CA. We could possibly use an internally created CA to sign certificates for Collect, Ona Data, and Aggregate.

[Absolomon_Kihara]

@Jason_Rogena

Some areas we are looking out for feedback from the community are:
What other requirements should we consider (but not necessarily implement immediately).

It will be good to have the auto update feature be included as part of individual form settings which would enable an individual form notifications to be muted.

[Greg_Pommen]

As I get more users using ODK, I don't think this feature can be implemented fast enough! I don't know the technologies involved but my wishlist would be:

-auto-sending of any remaining data
-deleting the old form
-push update of the new form without any user involvement necessary.

I know this isn't about push updates and your notification system @Jason_Rogena would certainly be a big step forward. However, I'm willing to be that the users would still be inclined to ignore, put off, swipe away and forget, etc. etc. any message notifications. The simpler and less error prone for users we can make it the better.

[Mic]

Is there Progress in implementing this?
It has been a while since July...
It would really help, i see some of our data-collectors using old forms and i have to text them all the time to update...

Hope this Feature Comes to ODK Collect

[LN]

You can see @Jason_Rogena's progress on an external team management app at https://github.com/onaio/collect-team-management.

The server component will initially only be available for Ona.