Cannot send ODK forms to server with Images

Dear ODK Developer Group,

We are unable to send ODK forms with multiple images to the server due to some unavoidable reasons.When I send ODK forms with images from android App ODK Collect it is in processing state. After sometimes the connection time out has been occurred. PLease any one give solution to this.

A few questions:

(1) is your ODK Aggregate server running on Google AppEngine or elsewhere?

(2) what is the quality of the wifi / cellular connection to the server?

··· ---------------- If the cellular network is 2G or 3G, you generally cannot send images. Satellite uplinks also generally prevent sending images.

This is due to the extremely slow bandwidth of the transmissions.

3G uplink data transmission speeds can be at or below 0.4 Mbit / s.

If your image is 2MB in size (16 Mbits), at this worst-case transmission
rate, it takes 40 seconds to send one image.

A good quality 3G network could have an uplink speed of 22 Mbit / s. This
same image would take less than a second to upload.

============
This is important because a submission cannot be processed until all the
data is received by the ODK Aggregate software.

BUT the Google AppEngine infrastructure (and the Tomcat server) start a
request-duration timer upon receipt of the first byte of the submission and
both of those will reject, ignore or instruct ODK Aggregate to abandon
processing of a submission after some limited amount of time.

On Google AppEngine, this time is 60 seconds.

So if the transmission of your submission consumes 40 seconds over a poor
connection, then this leaves only 20 seconds for ODK Aggregate to process
the submission before it is terminated by Google AppEngine infrastructure.

In practice, ODK Aggregate, if it is active, takes less than 2 seconds to
process a submission.

However, if it needs to be started (because the server has no running
instances on Google AppEngine infrastructure), this can take upwards of 20
seconds before it can begin processing the submission. Start-up is not an
issue if you are running your own Tomcat server.

And if you have a heavily utilized server, times can be improved by
increasing the size of your Google AppEngine server or Tomcat JVM.

Mitch

On Wed, Sep 28, 2016 at 12:43 AM, anirudha@sdrc.co.in wrote:

Dear ODK Developer Group,

We are unable to send ODK forms with multiple images to the server due to
some unavoidable reasons.When I send ODK forms with images from android App
ODK Collect it is in processing state. After sometimes the connection time
out has been occurred. PLease any one give solution to this.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Hello Mitch,

I have already tested with 4G network and with high speed. But when forms with multiple images uploaded in server it will take so much times and then it failed with connection timeout error.
We are using tomcat server and MySql as database server. I have also allocated around 6 GB of memory to tomcat JVM and around 1.5 GB of memory to MYSQL buffer size. I have also made some changes in tomcat server.xml file like connection timeout,max httpheadersize, and also max_allowed packet, max user connection limit in MYSQL also.

But there is no change in submission point of view. Forms with large image size(min 1MB) stuck in the way of upload.

Is there anything can I change in ODK Application point of view. Now we are in big trouble why this is happening.

Hope your support will help us to resolve the issue soon.

Regards,
Anirudha,
System Admin,
SDRC (P) Ltd.
email:anirudha@sdrc.co.in

This sounds like a networking or firewall issue.

Download the ODK Form Uploader tool (
https://opendatakit.org/downloads/download-category/other/ )

And see https://opendatakit.org/use/form-uploader/

The media files don't have to be in any way referred to by the content of
the form definition file. Try using this tool to upload a form definition
(e.g., the sample form https://opendatakit.org/use/xlsform/ ) with a large
set of image media files (e.g., from your example submission). This tool,
like ODK Collect, will batch the upload into approximately 10MB chunks.

Try:

  • running FormUploader on the server you have configured,
  • running it on a computer within your network
  • running it from a nearby WiFi hotspot similar to one you might use for
    ODK Collect submissions.

And, of course, you will need to delete the form or change the form Id of
it for each run. Deleting a form will take a little bit of time because it
will need to delete all the media files.

I would expect that the nearby WiFi hotspot will fail in the same way as
ODK Collect does.

If it works, then it would indicate that it is not a network or firewall
issue.

··· On Thu, Sep 29, 2016 at 8:31 AM, wrote:

Hello Mitch,

I have already tested with 4G network and with high speed. But when forms
with multiple images uploaded in server it will take so much times and then
it failed with connection timeout error.
We are using tomcat server and MySql as database server. I have also
allocated around 6 GB of memory to tomcat JVM and around 1.5 GB of memory
to MYSQL buffer size. I have also made some changes in tomcat server.xml
file like connection timeout,max httpheadersize, and also max_allowed
packet, max user connection limit in MYSQL also.

But there is no change in submission point of view. Forms with large image
size(min 1MB) stuck in the way of upload.

Is there anything can I change in ODK Application point of view. Now we
are in big trouble why this is happening.

Hope your support will help us to resolve the issue soon.

Regards,
Anirudha,
System Admin,
SDRC (P) Ltd.
email:anirudha@sdrc.co.in

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Hello Mitch,

As per your instruction I have done the steps to upload files in the server
having protected with firewall from external network and yes it will work
and the form has been successfully uploaded in the server through
FormUploader.

So I think there is no firewall issue in the process.

Regards,
Anirudha

··· -------------------------------------------- Anirudha Rath Infra Manager, SDRC (P) Ltd. M:-+91:7682883110

On Sat, Oct 1, 2016 at 12:05 AM, Mitch Sundt mitchellsundt@gmail.com wrote:

This sounds like a networking or firewall issue.

Download the ODK Form Uploader tool ( https://opendatakit.org/
downloads/download-category/other/ )

And see https://opendatakit.org/use/form-uploader/

The media files don't have to be in any way referred to by the content of
the form definition file. Try using this tool to upload a form definition
(e.g., the sample form https://opendatakit.org/use/xlsform/ ) with a
large set of image media files (e.g., from your example submission). This
tool, like ODK Collect, will batch the upload into approximately 10MB
chunks.

Try:

  • running FormUploader on the server you have configured,
  • running it on a computer within your network
  • running it from a nearby WiFi hotspot similar to one you might use for
    ODK Collect submissions.

And, of course, you will need to delete the form or change the form Id of
it for each run. Deleting a form will take a little bit of time because it
will need to delete all the media files.

I would expect that the nearby WiFi hotspot will fail in the same way as
ODK Collect does.

If it works, then it would indicate that it is not a network or firewall
issue.

On Thu, Sep 29, 2016 at 8:31 AM, anirudha@sdrc.co.in wrote:

Hello Mitch,

I have already tested with 4G network and with high speed. But when forms
with multiple images uploaded in server it will take so much times and then
it failed with connection timeout error.
We are using tomcat server and MySql as database server. I have also
allocated around 6 GB of memory to tomcat JVM and around 1.5 GB of memory
to MYSQL buffer size. I have also made some changes in tomcat server.xml
file like connection timeout,max httpheadersize, and also max_allowed
packet, max user connection limit in MYSQL also.

But there is no change in submission point of view. Forms with large
image size(min 1MB) stuck in the way of upload.

Is there anything can I change in ODK Application point of view. Now we
are in big trouble why this is happening.

Hope your support will help us to resolve the issue soon.

Regards,
Anirudha,
System Admin,
SDRC (P) Ltd.
email:anirudha@sdrc.co.in

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/opendatakit-developers/3DI94rSldp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Did you try this from a wifi hotspot (outside of your network, if
applicable) that ODK Collect might use?

Can you successfully submit forms with images from a different model device
or phone?

i.e., is this associated with a particular make / model of device?

··· On Mon, Oct 3, 2016 at 8:33 AM, Anirudha Rath wrote:

Hello Mitch,

As per your instruction I have done the steps to upload files in the
server having protected with firewall from external network and yes it will
work and the form has been successfully uploaded in the server through
FormUploader.

So I think there is no firewall issue in the process.

Regards,
Anirudha

Anirudha Rath
Infra Manager, SDRC (P) Ltd.
M:-+91:7682883110

On Sat, Oct 1, 2016 at 12:05 AM, Mitch Sundt mitchellsundt@gmail.com wrote:

This sounds like a networking or firewall issue.

Download the ODK Form Uploader tool ( https://opendatakit.org/down
loads/download-category/other/ )

And see https://opendatakit.org/use/form-uploader/

The media files don't have to be in any way referred to by the content of
the form definition file. Try using this tool to upload a form definition
(e.g., the sample form https://opendatakit.org/use/xlsform/ ) with a
large set of image media files (e.g., from your example submission). This
tool, like ODK Collect, will batch the upload into approximately 10MB
chunks.

Try:

  • running FormUploader on the server you have configured,
  • running it on a computer within your network
  • running it from a nearby WiFi hotspot similar to one you might use for
    ODK Collect submissions.

And, of course, you will need to delete the form or change the form Id of
it for each run. Deleting a form will take a little bit of time because it
will need to delete all the media files.

I would expect that the nearby WiFi hotspot will fail in the same way as
ODK Collect does.

If it works, then it would indicate that it is not a network or firewall
issue.

On Thu, Sep 29, 2016 at 8:31 AM, anirudha@sdrc.co.in wrote:

Hello Mitch,

I have already tested with 4G network and with high speed. But when
forms with multiple images uploaded in server it will take so much times
and then it failed with connection timeout error.
We are using tomcat server and MySql as database server. I have also
allocated around 6 GB of memory to tomcat JVM and around 1.5 GB of memory
to MYSQL buffer size. I have also made some changes in tomcat server.xml
file like connection timeout,max httpheadersize, and also max_allowed
packet, max user connection limit in MYSQL also.

But there is no change in submission point of view. Forms with large
image size(min 1MB) stuck in the way of upload.

Is there anything can I change in ODK Application point of view. Now we
are in big trouble why this is happening.

Hope your support will help us to resolve the issue soon.

Regards,
Anirudha,
System Admin,
SDRC (P) Ltd.
email:anirudha@sdrc.co.in

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/to
pic/opendatakit-developers/3DI94rSldp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Dear Mitch,
Apologies for wrongly typed as it will work instead of it is working.

··· > Regards, > Anirudha > -------------------------------------------- > Anirudha Rath > Infra Manager, SDRC (P) Ltd. > M:-+91:7682883110 > > > On Sat, Oct 1, 2016 at 12:05 AM, Mitch Sundt wrote: > >> This sounds like a networking or firewall issue. >> >> Download the ODK Form Uploader tool ( https://opendatakit.org/down >> loads/download-category/other/ ) >> >> And see https://opendatakit.org/use/form-uploader/ >> >> The media files don't have to be in any way referred to by the content of >> the form definition file. Try using this tool to upload a form definition >> (e.g., the sample form https://opendatakit.org/use/xlsform/ ) with a >> large set of image media files (e.g., from your example submission). This >> tool, like ODK Collect, will batch the upload into approximately 10MB >> chunks. >> >> Try: >> - running FormUploader on the server you have configured, >> - running it on a computer within your network >> - running it from a nearby WiFi hotspot similar to one you might use for >> ODK Collect submissions. >> >> And, of course, you will need to delete the form or change the form Id of >> it for each run. Deleting a form will take a little bit of time because it >> will need to delete all the media files. >> >> I would expect that the nearby WiFi hotspot will fail in the same way as >> ODK Collect does. >> >> If it works, then it would indicate that it is not a network or firewall >> issue. >> >> >> On Thu, Sep 29, 2016 at 8:31 AM, wrote: >> >>> Hello Mitch, >>> >>> I have already tested with 4G network and with high speed. But when >>> forms with multiple images uploaded in server it will take so much times >>> and then it failed with connection timeout error. >>> We are using tomcat server and MySql as database server. I have also >>> allocated around 6 GB of memory to tomcat JVM and around 1.5 GB of memory >>> to MYSQL buffer size. I have also made some changes in tomcat server.xml >>> file like connection timeout,max httpheadersize, and also max_allowed >>> packet, max user connection limit in MYSQL also. >>> >>> But there is no change in submission point of view. Forms with large >>> image size(min 1MB) stuck in the way of upload. >>> >>> Is there anything can I change in ODK Application point of view. Now we >>> are in big trouble why this is happening. >>> >>> Hope your support will help us to resolve the issue soon. >>> >>> Regards, >>> Anirudha, >>> System Admin, >>> SDRC (P) Ltd. >>> email:anirudha@sdrc.co.in >>> >>> >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "ODK Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to opendatakit-developers+unsubscribe@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Mitch Sundt >> Software Engineer >> University of Washington >> mitchellsundt@gmail.com >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "ODK Developers" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/to >> pic/opendatakit-developers/3DI94rSldp8/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> opendatakit-developers+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > >

[image: Inline image 1]

[image: Inline image 2]

please find the image above for server application

Regards,
Anirudha

··· -------------------------------------------- Anirudha Rath Infra Manager, SDRC (P) Ltd. M:-+91:7682883110

On Tue, Oct 4, 2016 at 10:30 PM, Mitch Sundt mitchellsundt@gmail.com wrote:

Did you try this from a wifi hotspot (outside of your network, if
applicable) that ODK Collect might use?

Can you successfully submit forms with images from a different model
device or phone?

i.e., is this associated with a particular make / model of device?

On Mon, Oct 3, 2016 at 8:33 AM, Anirudha Rath anirudha@sdrc.co.in wrote:

Hello Mitch,

As per your instruction I have done the steps to upload files in the
server having protected with firewall from external network and yes it will
work and the form has been successfully uploaded in the server through
FormUploader.

So I think there is no firewall issue in the process.

Regards,
Anirudha

Anirudha Rath
Infra Manager, SDRC (P) Ltd.
M:-+91:7682883110

On Sat, Oct 1, 2016 at 12:05 AM, Mitch Sundt mitchellsundt@gmail.com wrote:

This sounds like a networking or firewall issue.

Download the ODK Form Uploader tool ( https://opendatakit.org/down
loads/download-category/other/ )

And see https://opendatakit.org/use/form-uploader/

The media files don't have to be in any way referred to by the content
of the form definition file. Try using this tool to upload a form
definition (e.g., the sample form https://opendatakit.org/use/xlsform/ )
with a large set of image media files (e.g., from your example
submission). This tool, like ODK Collect, will batch the upload into
approximately 10MB chunks.

Try:

  • running FormUploader on the server you have configured,
  • running it on a computer within your network
  • running it from a nearby WiFi hotspot similar to one you might use for
    ODK Collect submissions.

And, of course, you will need to delete the form or change the form Id
of it for each run. Deleting a form will take a little bit of time because
it will need to delete all the media files.

I would expect that the nearby WiFi hotspot will fail in the same way as
ODK Collect does.

If it works, then it would indicate that it is not a network or firewall
issue.

On Thu, Sep 29, 2016 at 8:31 AM, anirudha@sdrc.co.in wrote:

Hello Mitch,

I have already tested with 4G network and with high speed. But when
forms with multiple images uploaded in server it will take so much times
and then it failed with connection timeout error.
We are using tomcat server and MySql as database server. I have also
allocated around 6 GB of memory to tomcat JVM and around 1.5 GB of memory
to MYSQL buffer size. I have also made some changes in tomcat server.xml
file like connection timeout,max httpheadersize, and also max_allowed
packet, max user connection limit in MYSQL also.

But there is no change in submission point of view. Forms with large
image size(min 1MB) stuck in the way of upload.

Is there anything can I change in ODK Application point of view. Now we
are in big trouble why this is happening.

Hope your support will help us to resolve the issue soon.

Regards,
Anirudha,
System Admin,
SDRC (P) Ltd.
email:anirudha@sdrc.co.in

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/to
pic/opendatakit-developers/3DI94rSldp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/opendatakit-developers/3DI94rSldp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

adding rohit in loop

Regards,
Anirudha

··· -------------------------------------------- Anirudha Rath Infra Manager, SDRC (P) Ltd. M:-+91:7682883110

On Wed, Oct 5, 2016 at 4:01 PM, Anirudha Rath anirudha@sdrc.co.in wrote:

[image: Inline image 1]

[image: Inline image 2]

please find the image above for server application

Regards,
Anirudha

Anirudha Rath
Infra Manager, SDRC (P) Ltd.
M:-+91:7682883110

On Tue, Oct 4, 2016 at 10:30 PM, Mitch Sundt mitchellsundt@gmail.com wrote:

Did you try this from a wifi hotspot (outside of your network, if
applicable) that ODK Collect might use?

Can you successfully submit forms with images from a different model
device or phone?

i.e., is this associated with a particular make / model of device?

On Mon, Oct 3, 2016 at 8:33 AM, Anirudha Rath anirudha@sdrc.co.in wrote:

Hello Mitch,

As per your instruction I have done the steps to upload files in the
server having protected with firewall from external network and yes it will
work and the form has been successfully uploaded in the server through
FormUploader.

So I think there is no firewall issue in the process.

Regards,
Anirudha

Anirudha Rath
Infra Manager, SDRC (P) Ltd.
M:-+91:7682883110

On Sat, Oct 1, 2016 at 12:05 AM, Mitch Sundt mitchellsundt@gmail.com wrote:

This sounds like a networking or firewall issue.

Download the ODK Form Uploader tool ( https://opendatakit.org/down
loads/download-category/other/ )

And see https://opendatakit.org/use/form-uploader/

The media files don't have to be in any way referred to by the content
of the form definition file. Try using this tool to upload a form
definition (e.g., the sample form https://opendatakit.org/use/xlsform/ )
with a large set of image media files (e.g., from your example
submission). This tool, like ODK Collect, will batch the upload into
approximately 10MB chunks.

Try:

  • running FormUploader on the server you have configured,
  • running it on a computer within your network
  • running it from a nearby WiFi hotspot similar to one you might use
    for ODK Collect submissions.

And, of course, you will need to delete the form or change the form Id
of it for each run. Deleting a form will take a little bit of time because
it will need to delete all the media files.

I would expect that the nearby WiFi hotspot will fail in the same way
as ODK Collect does.

If it works, then it would indicate that it is not a network or
firewall issue.

On Thu, Sep 29, 2016 at 8:31 AM, anirudha@sdrc.co.in wrote:

Hello Mitch,

I have already tested with 4G network and with high speed. But when
forms with multiple images uploaded in server it will take so much times
and then it failed with connection timeout error.
We are using tomcat server and MySql as database server. I have also
allocated around 6 GB of memory to tomcat JVM and around 1.5 GB of memory
to MYSQL buffer size. I have also made some changes in tomcat server.xml
file like connection timeout,max httpheadersize, and also max_allowed
packet, max user connection limit in MYSQL also.

But there is no change in submission point of view. Forms with large
image size(min 1MB) stuck in the way of upload.

Is there anything can I change in ODK Application point of view. Now
we are in big trouble why this is happening.

Hope your support will help us to resolve the issue soon.

Regards,
Anirudha,
System Admin,
SDRC (P) Ltd.
email:anirudha@sdrc.co.in

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/to
pic/opendatakit-developers/3DI94rSldp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/to
pic/opendatakit-developers/3DI94rSldp8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.