Null request body on JSON publish

Submissions with characters with accents, when published with JSON, appear
to result in a malformed JSON post(null body). Is this a known issue, or
perhaps something is wrong with my setup?

Regards,

Sam

Are you using ODK Aggregate 1.4.9 ?

Do you have an example form definition and submission that I can test with?

If you are reading the POST body within Java, note that you need to specify
UTF-8 when you construct your Reader. Or use a conversion library that can
directly process the byte stream.

··· On Thu, May 26, 2016 at 5:38 AM, Sam wrote:

Submissions with characters with accents, when published with JSON, appear
to result in a malformed JSON post(null body). Is this a known issue, or
perhaps something is wrong with my setup?

Regards,

Sam

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Thank you for your reply. After seeing your message, I upgraded to 1.4.9,
but with same result. The form definition is attached. The JSON body,
when properly recieved, looks like the JSON below.

As a test, I submitted the JSON POST below from the FireFox RESTful Client
plugin, with an accented character, just to see if it was failing when read
by the server.

The server that is receiving the POST is a .NET Api controller that looks
something like this. I do not have to construct a reader. As mentioned, I
tested the POST using a different REST client just to see if an accented
character would go through, and it did, so I am still inclined to think it
is an issue with the POST coming out of Aggregate.

public HttpResponseMessage Post([FromBody] Newtonsoft.Json.Linq.JToken json)
{
if (json == null)
{
...logs the error to a log file...

            return new HttpResponseMessage(HttpStatusCode.OK);
        }
      ... processes the result

}

-------Example normal JSON Body------------------------------
{
"token": "",
"content": "record",
"formId": "identification-nouv",
"formVersion": "1",
"data": [
{
"meta-instance-id": "uuid:fc9d56c9-7df4-4f7e-a9a9-bebc19696850",
"meta-model-version": 1,
"meta-ui-version": null,
"meta-submission-date": "2016-05-26T22:40:48Z",
"meta-is-complete": true,
"meta-date-marked-as-complete": "2016-05-26T22:40:48Z",
"agence": "",
"user": "",
"groupe": "",
"client": "new",
"username": "",
"nom": "Sám",
"postnom": null,
"prenom": null,
"sexe": "M",
"lieu_naissance": ".",
"addresse": null,
"logement": null,
"nom_conjoint": null,
"telephone": null,
"email": null,
"document": null,
"lieu_delivrance_opt": null,
"lieu_delivrance_other": null,
"lieu_delivrance": null,
"num_document": null,
"date_delivrance": "2016-05-26",
"date_naissance": "2016-05-26",
"etat_civil": null,
"secteur": null,
"personnes_en_charges": null,
"photo": null,
"piece_identite": {
"filename": "1464302442639.jpg",
"type": "image/jpeg",
"url":
"http://10.0.0.139:8080/view/binaryData?blobKey=identification-nouv[%40version%3Dnull+and+%40uiVersion%3Dnull]%2Fidentification-new-members[%40key%3Duuid%3Afc9d56c9-7df4-4f7e-a9a9-bebc19696854]%2Fpiece_identite"
},
"start": "2016-05-26T22:39:55Z",
"end": "2016-05-26T22:40:46Z",
"today": "2016-05-26",
"deviceid": "99000064206307",
"subscriberid": "311480165257343",
"simserial": "89148000001635930547",
"phonenumber": "9176737474",
"instanceID": "uuid:fc9d56c9-7df4-4f7e-a9a9-bebc19696854"
}
]
}

identification.xml (13.3 KB)

··· On Thursday, May 26, 2016 at 3:03:20 PM UTC-4, Mitch wrote: > > Are you using ODK Aggregate 1.4.9 ? > > Do you have an example form definition and submission that I can test with? > > If you are reading the POST body within Java, note that you need to > specify UTF-8 when you construct your Reader. Or use a conversion library > that can directly process the byte stream. > >

Different JSON parsers have different expectations and strictness on their
handling of whitespace in a JSON serialization.

e.g., some do not accept pretty-printed JSON serializations.

I suspect this is the issue.

Try reading the entity body as a UTF-8 character stream (or string), then
run the JSON parser on that string object to see if you can get a more
detailed error message (i.e., a character offset).

If you get a character offset, and write the string to a file, then we can
begin to find a solution.

··· On Thu, May 26, 2016 at 4:13 PM, Sam wrote:

Thank you for your reply. After seeing your message, I upgraded to 1.4.9,
but with same result. The form definition is attached. The JSON body,
when properly recieved, looks like the JSON below.

As a test, I submitted the JSON POST below from the FireFox RESTful Client
plugin, with an accented character, just to see if it was failing when read
by the server.

The server that is receiving the POST is a .NET Api controller that looks
something like this. I do not have to construct a reader. As mentioned, I
tested the POST using a different REST client just to see if an accented
character would go through, and it did, so I am still inclined to think it
is an issue with the POST coming out of Aggregate.

public HttpResponseMessage Post([FromBody] Newtonsoft.Json.Linq.JToken
json)
{
if (json == null)
{
...logs the error to a log file...

            return new HttpResponseMessage(HttpStatusCode.OK);
        }
      ... processes the result

}

-------Example normal JSON Body------------------------------
{
"token": "",
"content": "record",
"formId": "identification-nouv",
"formVersion": "1",
"data": [
{
"meta-instance-id": "uuid:fc9d56c9-7df4-4f7e-a9a9-bebc19696850",
"meta-model-version": 1,
"meta-ui-version": null,
"meta-submission-date": "2016-05-26T22:40:48Z",
"meta-is-complete": true,
"meta-date-marked-as-complete": "2016-05-26T22:40:48Z",
"agence": "",
"user": "",
"groupe": "",
"client": "new",
"username": "",
"nom": "Sám",
"postnom": null,
"prenom": null,
"sexe": "M",
"lieu_naissance": ".",
"addresse": null,
"logement": null,
"nom_conjoint": null,
"telephone": null,
"email": null,
"document": null,
"lieu_delivrance_opt": null,
"lieu_delivrance_other": null,
"lieu_delivrance": null,
"num_document": null,
"date_delivrance": "2016-05-26",
"date_naissance": "2016-05-26",
"etat_civil": null,
"secteur": null,
"personnes_en_charges": null,
"photo": null,
"piece_identite": {
"filename": "1464302442639.jpg",
"type": "image/jpeg",
"url": "
http://10.0.0.139:8080/view/binaryData?blobKey=identification-nouv[%40version%3Dnull+and+%40uiVersion%3Dnull]%2Fidentification-new-members[%40key%3Duuid%3Afc9d56c9-7df4-4f7e-a9a9-bebc19696854]%2Fpiece_identite
"
},
"start": "2016-05-26T22:39:55Z",
"end": "2016-05-26T22:40:46Z",
"today": "2016-05-26",
"deviceid": "99000064206307",
"subscriberid": "311480165257343",
"simserial": "89148000001635930547",
"phonenumber": "9176737474",
"instanceID": "uuid:fc9d56c9-7df4-4f7e-a9a9-bebc19696854"
}
]
}

On Thursday, May 26, 2016 at 3:03:20 PM UTC-4, Mitch wrote:

Are you using ODK Aggregate 1.4.9 ?

Do you have an example form definition and submission that I can test
with?

If you are reading the POST body within Java, note that you need to
specify UTF-8 when you construct your Reader. Or use a conversion library
that can directly process the byte stream.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

You can also try https://odk-post-receiver.appspot.com to see if that works.

Yaw

··· -- Need ODK consultants? Nafundi provides form design, server setup, in-field training, and software development for ODK. Go to https://nafundi.com to get started.

On Fri, May 27, 2016 at 10:01 PM, Mitch Sundt mitchellsundt@gmail.com wrote:

Different JSON parsers have different expectations and strictness on their
handling of whitespace in a JSON serialization.

e.g., some do not accept pretty-printed JSON serializations.

I suspect this is the issue.

Try reading the entity body as a UTF-8 character stream (or string), then
run the JSON parser on that string object to see if you can get a more
detailed error message (i.e., a character offset).

If you get a character offset, and write the string to a file, then we can
begin to find a solution.

On Thu, May 26, 2016 at 4:13 PM, Sam fergusondsamuel@gmail.com wrote:

Thank you for your reply. After seeing your message, I upgraded to 1.4.9,
but with same result. The form definition is attached. The JSON body, when
properly recieved, looks like the JSON below.

As a test, I submitted the JSON POST below from the FireFox RESTful Client
plugin, with an accented character, just to see if it was failing when read
by the server.

The server that is receiving the POST is a .NET Api controller that looks
something like this. I do not have to construct a reader. As mentioned, I
tested the POST using a different REST client just to see if an accented
character would go through, and it did, so I am still inclined to think it
is an issue with the POST coming out of Aggregate.

public HttpResponseMessage Post([FromBody] Newtonsoft.Json.Linq.JToken
json)
{
if (json == null)
{
...logs the error to a log file...

            return new HttpResponseMessage(HttpStatusCode.OK);
        }
      ... processes the result

}

-------Example normal JSON Body------------------------------
{
"token": "",
"content": "record",
"formId": "identification-nouv",
"formVersion": "1",
"data": [
{
"meta-instance-id": "uuid:fc9d56c9-7df4-4f7e-a9a9-bebc19696850",
"meta-model-version": 1,
"meta-ui-version": null,
"meta-submission-date": "2016-05-26T22:40:48Z",
"meta-is-complete": true,
"meta-date-marked-as-complete": "2016-05-26T22:40:48Z",
"agence": "",
"user": "",
"groupe": "",
"client": "new",
"username": "",
"nom": "Sám",
"postnom": null,
"prenom": null,
"sexe": "M",
"lieu_naissance": ".",
"addresse": null,
"logement": null,
"nom_conjoint": null,
"telephone": null,
"email": null,
"document": null,
"lieu_delivrance_opt": null,
"lieu_delivrance_other": null,
"lieu_delivrance": null,
"num_document": null,
"date_delivrance": "2016-05-26",
"date_naissance": "2016-05-26",
"etat_civil": null,
"secteur": null,
"personnes_en_charges": null,
"photo": null,
"piece_identite": {
"filename": "1464302442639.jpg",
"type": "image/jpeg",
"url":
"http://10.0.0.139:8080/view/binaryData?blobKey=identification-nouv[%40version%3Dnull+and+%40uiVersion%3Dnull]%2Fidentification-new-members[%40key%3Duuid%3Afc9d56c9-7df4-4f7e-a9a9-bebc19696854]%2Fpiece_identite"
},
"start": "2016-05-26T22:39:55Z",
"end": "2016-05-26T22:40:46Z",
"today": "2016-05-26",
"deviceid": "99000064206307",
"subscriberid": "311480165257343",
"simserial": "89148000001635930547",
"phonenumber": "9176737474",
"instanceID": "uuid:fc9d56c9-7df4-4f7e-a9a9-bebc19696854"
}
]
}

On Thursday, May 26, 2016 at 3:03:20 PM UTC-4, Mitch wrote:

Are you using ODK Aggregate 1.4.9 ?

Do you have an example form definition and submission that I can test
with?

If you are reading the POST body within Java, note that you need to
specify UTF-8 when you construct your Reader. Or use a conversion library
that can directly process the byte stream.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Mitch,

The default .net JsonMediaTypeFormatter was resulting in a null parameter being passed into the MVC controller whenever the body of the post had accented chars. I implemented a custom MediaTypeFormatter to capture the results as a string and parse the JSON in my own code as you suggest below, and that is working.

Thank you for pointing me in the right direction.

Yaw, thank you for the suggestion, I was unable to use the test server, because my Aggregate instance is on a private network behind a VPN.

Sam

··· On Friday, May 27, 2016 at 3:01:25 PM UTC-4, Mitch wrote: > > Different JSON parsers have different expectations and strictness on their > handling of whitespace in a JSON serialization. > > e.g., some do not accept pretty-printed JSON serializations. > > I suspect this is the issue. > > Try reading the entity body as a UTF-8 character stream (or string), then > run the JSON parser on that string object to see if you can get a more > detailed error message (i.e., a character offset). > > If you get a character offset, and write the string to a file, then we can > begin to find a solution. > > > >

Fascinating.

The w3C standard explicitly states that the application/json mime type is
UTF-8.

For a JSON parser to not handle the full range of characters is a very
significant failure.

··· On Sat, May 28, 2016 at 11:19 AM, Sam wrote:

Mitch,

The default .net JsonMediaTypeFormatter was resulting in a null parameter being passed into the MVC controller whenever the body of the post had accented chars. I implemented a custom MediaTypeFormatter to capture the results as a string and parse the JSON in my own code as you suggest below, and that is working.

Thank you for pointing me in the right direction.

Yaw, thank you for the suggestion, I was unable to use the test server, because my Aggregate instance is on a private network behind a VPN.

Sam

On Friday, May 27, 2016 at 3:01:25 PM UTC-4, Mitch wrote:

Different JSON parsers have different expectations and strictness on
their handling of whitespace in a JSON serialization.

e.g., some do not accept pretty-printed JSON serializations.

I suspect this is the issue.

Try reading the entity body as a UTF-8 character stream (or string), then
run the JSON parser on that string object to see if you can get a more
detailed error message (i.e., a character offset).

If you get a character offset, and write the string to a file, then we
can begin to find a solution.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com