Collect messaging push notifications

Sounds like you've got great answers for everything!

I took a look at the message structure; my only proposal is to add a toplevel version field on the message itself (eg not in the payload) in case that format changes. Otherwise it seems like having a toplevel fully qualified type and a freeform payload blob enables unlimited flexibility, which seems great. Hopefully we use it wisely. :slight_smile:

Any need for encryption?

@issa Beyond the specific tech choices, what are your thoughts regarding @Jason_Rogena's question on where to do this development? I think that if there's a good way to do this without Ona needing to maintain a Collect fork, that would be great for everyone. Do you think there is a path for self-hosted servers to provide push notifications this way? Is that even an important consideration? As long as the feature doesn't add overhead to users who can't use it and doesn't dramatically increase the apk size, it seems it would be good to have in the app for servers that can support it.

I haven't thought about this deeply but my sense is that an external app would be impractical. It's unfortunate because I do like that it would force strict separation of concerns and allow the functionality to be pulled in to core at a later date if some refinement is needed. Some of my specific concerns are how it would know what user has successfully authenticated in Collect and what forms are available at any given moment.

I just tried again and it turns out I hadn't noticed the Upload button that is to the left of the filename so I was just clicking on Save. :flushed: The upload is just spinning and spinning right now but I'll try again later and hopefully third time will be a charm.

Well, if the service is on its own protocol and infrastructure (ie MQTT) there's no reason it wouldn't work on self-hosted servers provided they're on the net. So, any scenario in which a remote form update would work in the first place should work, yes? So I think yes, it's a valuable consideration but I think we have the answer to it.

As for fork or no-fork, no-fork is of course preferable. But I don't know enough about skeleton burial grounds in this code to really comment on the practicality and feasibility of such an approach. Separate app is probably asking more trouble than it's worth.

Ah, for some reason I was thinking of the message broker as a separate entity from "the server" but that doesn't have to be the case.

This is a fantastic discussion and I hope my feedback can help move this forward!

My guess, based on similar feature requests that I've seen over time, is that your client wants forms that automatically update and also wants a way to message enumerators (about forms or about whatever). If my guess is correct, then we should be careful not to conflate the two.

My argument for a separate app

It seems very useful to have arbitrary messaging to an ODK Messages app and allow those messages to fire of specific-intents to Collect or whatever app you have installed. For example, a reminder message for all enumerators come to Site X on a certain date, could have an associated payload that opens up a calendar. You can't really do this with SMS or WhatsApp (although my gut says this has to exist somewhere).

And if that approach of a standalone app seems reasonable, then what we'd need on the Collect side is broadcast receivers that can fire off background tasks. One of those tasks could be to start a download of the new form or flag a form as outdated. This aligns very nicely with the work we have to do to clean up Collect's integration points anyway.

I don't know how this two app approach will work with ONA's auth, but given that Facebook and Facebook Messenger can share auth, doesn't seem impossible.

Changes to Collect to support form updates

For the download of a new form, we can build some safe defaults here for users. If there is a v2 of the form, you can download and stage it, force all finalized v1 submissions to be sent (or wait until that is done), delete v1 form, then present v2 to the user.

Also, independent of push notifications for updates, I do think Collect itself should poll for updates for those people who want updates but don't want to have messaging infrastructure.

MQTT vs GCM
I don't know these technologies well-enough to provide thoughtful feedback. If you need control, it seems like MQTT is great. That said, a lot of people use GCM (even if it's just to wake up an app to then poll), because message delivery is hard. I'd bias for the choice that is easiest to undo.

1 Like

@Jason_Rogena when I heard you talking about this, you mentioned that you thought implementing a separate app would not be that difficult. Given the conversation we've been having, do you still believe that to be the case? If so, that may be a really good way forward that lets you move quickly and that lets this conversation continue in parallel.

How does this conversation relate to this Ona Collect issue? Do you already have a prototype implementation in your fork? If so, would you be willing to share it? I think we could offer a different kind of feedback when we understood exactly what you are considering a form update, what the MQTT client integration actually looks like, etc.

I understand that if MQTT is implemented other than GCM (now it's FCM), there would be an em tra payload as you have to maintain a listener besides the system one.

The major downside to multiple apps (besides user experience) is that it's yet one more thing which may demand coördinated/synchronized releases in the future. Do we want to add yet more to our burden on that front?

2 Likes

We're planning on using TLS to encrypt messages between the broker and clients. Mosquitto, the broker we plan to use, allows us to accept connections from certificates signed by a specific CA. We could possibly use an internally created CA to sign certificates for Collect, Ona Data, and Aggregate.

It will be good to have the auto update feature be included as part of individual form settings which would enable an individual form notifications to be muted.

1 Like

As I get more users using ODK, I don't think this feature can be implemented fast enough! I don't know the technologies involved but my wishlist would be:

-auto-sending of any remaining data
-deleting the old form
-push update of the new form without any user involvement necessary.

I know this isn't about push updates and your notification system @Jason_Rogena would certainly be a big step forward. However, I'm willing to be that the users would still be inclined to ignore, put off, swipe away and forget, etc. etc. any message notifications. The simpler and less error prone for users we can make it the better.

4 Likes

Is there Progress in implementing this?
It has been a while since July...
It would really help, i see some of our data-collectors using old forms and i have to text them all the time to update...

Hope this Feature Comes to ODK Collect :slight_smile:

You can see @Jason_Rogena's progress on an external team management app at https://github.com/onaio/collect-team-management.

The server component will initially only be available for Ona.

Push notifications have been added to the public Smap Server https://sg.smap.com.au. Video here https://www.youtube.com/watch?v=Hbq1-_fcSCg

I'm using firebase cloud messaging so the fieldTask app won't be compatible with the odk implementation in this regard.

We've implemented a polling-based form updater in our project's fork of Collect as well as added GCM-based messaging up- and downstream. The advantage of using GCM or Firebase messaging is that the message broker is hosted by Google. As mentioned before in the thread, though, Google has switched the recommended messaging service and may stop hosting GCM some time in the future.

Is the server side message broker going to be a part of the Ona Data server in https://github.com/onaio/onadata? It seems that the chosen solution is to build a separate app to handle the push notifications. Does the Collect team management app have other planned features?

Hello. Has there been progress implementing 'add[ing] support for push notifications into ODK Collect'?

Other purposes for push notifications are being assessed?
For instance to notify users that their data package is ready. To notify that there are new features... To make instant communication on their selections (editable within questionnaires). Anyone followed up with this?

Can you say more about what this means? If you mean if a form attachment with CSV data is updated, Collect can auto-update a form definition when that happens.

Agreed these are good use cases to consider.

Hi LN! Thank you for your response. Well, we collect data and after that people would like to know the status of their survey, what is happening to their data.Can we have notifications to say, "your data is on the way", "your data has been received and will be processed soon", "your results are ready"