Aggregate JSON Publisher Error - Datastore table is missing a reference part OR has extra copies

What is the problem? Please be detailed.

We have a relatively large form in ODK Aggregate (i.e. s4w_nepal_v1.01) with roughly 9000 submissions. We are using JSON publisher to move data into a MongoDB for a new web app that we are developing. Other JSON publishers for other forms are working perfectly, but the JSON publisher for this form consistently hangs after 32 dictionaries. We also receive a Stack Driver notification from background 1 service in the Google App Engine stating the following:

org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
_uploadAllSubmissions: org.opendatakit.aggregate.exception.ODKExternalServiceException: org.opendatakit.common.datamodel.ODKEnumeratedElementException: SELECT * FROM S4W_NEPAL_V1_01_TEMP_IMAGE_REF WHERE _TOP_LEVEL_AURI = uuid:feb2c339-26d6-4ac4-ab07-805993648dc4 AND DOM_AURI = uuid:edc3e01e-c12a-4787-a9c4-91e222a1c2ec is missing a reference part OR has extra copies.

When we view the records matching these criteria in the Datastore we noticed that there are three records, and specifically two for "Part 2." From a quick review, we see that all the other records only have one "Part 1" and one "Part 2."

What ODK tool and version are you using? And on what device and operating system version?

ODK Aggregate v1.4.15 on the Google Cloud.

What steps can we take to reproduce the problem?

You can login to our Datastore and take a look at the problems using the IDs provided above.

What you have you tried to fix the problem?

Our intuition is to delete the "duplicate" "Part 2" record in the database, but we aren't sure if this is the correct approach to solve this issue, and we also don't know what other problems this might create.

Anything else we should know or have? If you have a test form or screenshots or logs, attach here.

I Jeff! I'll take a look but it seems your diagnostics are correct. Let me get back to you with more info.

Thanks @ggalmazor! I look forward to figuring out what you find.

This is the same form that we were having troubles with earlier with the Google Sheet publisher, so I'm wondering if these issues are somehow related? It seems that somehow the Datastore tables associated with this form are a bit "buggy." We have released a few new versions of the form and are now using.

Hi, @Jeff_Davids! Sorry for the long delay on my answer.

I've reviewed your DataStore and I think we should remove these entities which appear to be duplicates:

These entities correspond to the oldest duplicate of the same part of an image attachment. If you want, I can proceed to remove them.

Hi @ggalmazor! Now it is my turn to appologize for the sluggish response!

I deleted tese two entities recently and I'll keep you posted on how this seems to impact the performance of the publishers for the S4W-Nepal v1.01 form.

Thanks again for your help with this!

1 Like

Hi, @Jeff_Davids! I'm glad you got it working again