Include non-relevant groups and fields in ODK Central API responses

Development
1

Dear ODK community,

we recently encountered some unexpected behaviour for ODK Central's OData and SubmissionXML APIs. Depending on whether relevant was false for a group or field when filling in the form, they will be included or not included in inconsistent ways in the API responses. See below for the different cases.

grafik
grafik732×204 6.39 KB

The submission XML appearantly never includes groups or fields when they were not relevant. However, the OData submission response includes non-relevant fields with null-values but does not include non-relevant groups and their inner-fields.

What behaviour should be expected? From the OData API, we'd expect that all fields are always given, even if they were not relevant. This means that non-relevant groups would be included in responses, just like non-relevant normal fields already are.

Below you can find sample responses and an XlsForm to reproduce.

Thanks for you thoughts!

Best,
Vitus

Addition: We noticed the issue because the behaviour was different on ODK Aggregate, if I remember correctly. In ODK Aggregate's JSON publishing mechanism, all fields were always included, independent of whether they were relevant or not.

OData responses with different relevant-options:

{
	"value": [
		{
			"__id": "uuid:38c44ef9-1b4f-4305-8b06-9a525520b604",
			"show_group": "yes",
			"show_field": "yes",
			"number": null,
			"__system": {},
			"group": {
				"inner_number": null
			},
			"meta": {
				"instanceID": "uuid:38c44ef9-1b4f-4305-8b06-9a525520b604"
			}
		},
		{
			"__id": "uuid:f77aa888-8cfd-4695-b261-292333477106",
			"show_group": "yes",
			"show_field": "no",
			"number": null,
			"__system": {},
			"group": {
				"inner_number": null
			},
			"meta": {
				"instanceID": "uuid:f77aa888-8cfd-4695-b261-292333477106"
			}
		},
		{
			"__id": "uuid:30f81702-0967-4e6f-bf97-383f7e4221c8",
			"show_group": "no",
			"show_field": "no",
			"number": null,
			"__system": {},
			"meta": {
				"instanceID": "uuid:30f81702-0967-4e6f-bf97-383f7e4221c8"
			}
		}
	],
	"@odata.context": "https://example.com/v1/projects/4/forms/field_conditions_minimal_dv2.svc/$metadata#Submissions",
	"@odata.count": 3
}

Different Submission XMLs:

<!-- group is relevant, field is not-->
<data xmlns:jr="http://openrosa.org/javarosa" xmlns:orx="http://openrosa.org/xforms" id="field_conditions_minimal_dv2" version="1">
    <show_group>yes</show_group>
    <show_field>no</show_field>
    <group>
        <inner_number/>
    </group>
    <meta>
        <instanceID>uuid:f77aa888-8cfd-4695-b261-292333477106</instanceID>
    </meta>
</data>
<!-- group and field are not relevant-->
<data xmlns:jr="http://openrosa.org/javarosa" xmlns:orx="http://openrosa.org/xforms" id="field_conditions_minimal_dv2" version="1">
    <show_group>no</show_group>
    <show_field>no</show_field>
    <meta>
        <instanceID>uuid:30f81702-0967-4e6f-bf97-383f7e4221c8</instanceID>
    </meta>
</data>
<!-- group and field are relevant-->
<data xmlns:jr="http://openrosa.org/javarosa" xmlns:orx="http://openrosa.org/xforms" id="field_conditions_minimal_dv2" version="1">
    <show_group>yes</show_group>
    <show_field>yes</show_field>
    <group>
        <inner_number/>
    </group>
    <number/>
    <meta>
        <instanceID>uuid:38c44ef9-1b4f-4305-8b06-9a525520b604</instanceID>
    </meta>
</data>

XlsForm:
central_nullvalues.xlsx (12.4 KB)

2 Likes
2

Hi @vlehn, and welcome to the ODK forum! When you have a chance, I'd encourage you to introduce yourself here. Doing so helps build community!

Thank you for this report! This sounds similar to the issue discussed here. We'll be changing that behavior as part of v1.2, so the OData should include non-relevant fields. We're hoping to release v1.2 later this month.

3

And now I'm realizing you were probably part of that discussion on GitHub! Have you had a chance to see whether you encounter this same behavior for the latest commit to central-backend? As far as I know, ODK Central doesn't consider relevant when generating the OData, so I'm hoping it's the same issue.

4

Hi Matthew, thanks for your quick reaction!

Indeed, I also reported the other issue. I tested the recent case with and without that commit, and the behaviour is the same. So the commit did not solve that one, I'm afraid.

Depending on what responses are expected, I can also file a bug report for it.

P.S: Yes, will introduce myself soon!

1 Like
5

@issa, just wanted to make sure you saw this topic. What do you think is the expected behavior in this case?

2 Likes
6

Hi @Matthew_White and @issa

So it seems like developing a position regarding this issue is not as simple. :wink: Still, is the topic something that you'd consider part of the current development roadmap (independent of what the outcome might be)? Could it even find its way into v1.2?

As stated, we would expect to always receive the full field set via OData JSON API, as then there would be fewer "surprises" in those individual submissions.

Would really appreciate if you can give me some perspective.

Thank you in any case!
Vitus

7

Hi to all,
using my central2pg set of PostgreSQL function with new forms, I am facing the same problem.
I am using the api result to automatically generate database tables and I realized that questions are missing. It would be great to get all fields, including nulls in submission data.

As a temporary workaround, is it possible to query the API to get all fields for each table instead of the whole form ?
The given json doe not help to know wich table is concerned by each attribute?

Here is the form (sorry it is not the simplest one :
suivis_mc_costieres_V2.xls (27.5 KB)) and here a submission for the "releves_indicateurs" table

{"__id":"9f397e9780e26dedf9e63c7313bd7f66ee4b9375",
"__Submissions-id":"uuid:2b633df3-63c5-4f61-8625-7ea0e78585d9",
"infos_generales_cen_occitanie":{"suivi_controle":"controle_simple",
"espece":null,
"saison":null,
"maitrise_fonciere":"convention_mae"},
"infos_releve_ug":{"now_releve_ug":"2021-05-06T09:14:29.142+02:00",
"date_time_releve_ug":"2021-05-05T09:05:00.000+02:00",
"id_ug":2305},
"conformite":{"controle_interne_conformite":"conforme",
"controle_interne_description":null},
"previsionnel":{"previsionnel_a_faire":null,
"previsionnel_autre":null},
"observations":{"indices_presence":"non",
"presence_individus":"non",
"remarques_mae":null,
"remarques_ppte":null}
}

Thanks a lot :slight_smile:

2 Likes
8

okay, once i wrapped my head back around all this stuff i saw a way to fix it. should be coming in the next release.

3 Likes
9

As of v1.3, we now include all groups in OData, even if they are not relevant. Thank you for reporting this issue!

3 Likes