Best way to handle a very large form

Hi All

I want to enquire about the best way to handle a very large form, if all
the skips are removed the questionnaire holds about 9000 questions.

I am in the process of finalising the form in the next few days.

I do not want to run into any issues once the form goes live so I am asking
would it be better to split the form into smaller forms, however I have a
single section that is about 7000 questions long and am unable to split
this.

Will i run into any type of limitation.

Also what is the best way to convert the form as sometime the online
converter gives time out errors when trying to convert the form.

Also what would be the best in terms of hosting the form, tomcat or app
engine?

The survey plans to get to a sample size of 3000, do you foresee any issues
with exporting to csv?

I await your responses eagerly

Bashir

··· -- Bashir Jahed - Director The OSi LAB (PTY) Ltd www.osilab.net | 083 414 0453 | bashir@osilab.net

Bashir,

At SurveyCTO, we have worked to steadily improve support for very long
forms -- both in terms of device performance, and in terms of the memory
required on the device. Unfortunately, we have not yet worked with the ODK
team to merge those changes back into the standard ODK Collect. However, I
will say that 9,000 fields may be beyond what even our much-optimized
SurveyCTO Collect can handle.

Ultimately, it will depend on your device's speed and memory. And when it
comes to memory, it is not simply the total memory on the device, but the
total amount the device will allow any single app to utilize. Thus, there
is no fixed answer regarding what will or won't work. As with an
unfortunate post just a day or two ago, you can even have a form that works
much of the time -- but if it is close to the memory limit, adding a repeat
group can push it over the edge and result in a crash.

You can always try ODK Collect and SurveyCTO Collect. If your form works on
the latter but not the former, then there are two choices: either you use
SurveyCTO, or you convince us and somebody on the core team to merge our
improvements. (We are quite open to contributing our improvements and have
just been discussing it today, in fact, but it will take some work both by
us and by somebody on the core team... and it is a matter of prioritizing
that work.)

Best,

Chris

··· On Thu, Jan 23, 2014 at 5:06 AM, Bashir Jahed wrote:

Hi All

I want to enquire about the best way to handle a very large form, if all
the skips are removed the questionnaire holds about 9000 questions.

I am in the process of finalising the form in the next few days.

I do not want to run into any issues once the form goes live so I am
asking would it be better to split the form into smaller forms, however I
have a single section that is about 7000 questions long and am unable to
split this.

Will i run into any type of limitation.

Also what is the best way to convert the form as sometime the online
converter gives time out errors when trying to convert the form.

Also what would be the best in terms of hosting the form, tomcat or app
engine?

The survey plans to get to a sample size of 3000, do you foresee any
issues with exporting to csv?

I await your responses eagerly

Bashir

--
Bashir Jahed - Director
The OSi LAB (PTY) Ltd
www.osilab.net | 083 414 0453 | bashir@osilab.net

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Agreed with Chris that you are probably asking for trouble. I'd look for
more ways to split it. If you aren't using repeats, might be time to try
them.

I'd probably host on a Tomcat server so you can optimize there too. Maybe
get a beefy Linode?

As far as running into timeouts on the form conversion, try installing
https://github.com/SEL-Columbia/pyxform and running it offline.

Oh, and test with a pilot deployment so you know what to expect when it
goes live.

Yaw

··· -- Need ODK services? http://nafundi.com provides form design, server setup, professional support, and software development for ODK.

On Thu, Jan 23, 2014 at 2:18 AM, Christopher Robert crobert@surveycto.comwrote:

Bashir,

At SurveyCTO, we have worked to steadily improve support for very long
forms -- both in terms of device performance, and in terms of the memory
required on the device. Unfortunately, we have not yet worked with the ODK
team to merge those changes back into the standard ODK Collect. However, I
will say that 9,000 fields may be beyond what even our much-optimized
SurveyCTO Collect can handle.

Ultimately, it will depend on your device's speed and memory. And when it
comes to memory, it is not simply the total memory on the device, but the
total amount the device will allow any single app to utilize. Thus, there
is no fixed answer regarding what will or won't work. As with an
unfortunate post just a day or two ago, you can even have a form that works
much of the time -- but if it is close to the memory limit, adding a repeat
group can push it over the edge and result in a crash.

You can always try ODK Collect and SurveyCTO Collect. If your form works
on the latter but not the former, then there are two choices: either you
use SurveyCTO, or you convince us and somebody on the core team to merge
our improvements. (We are quite open to contributing our improvements and
have just been discussing it today, in fact, but it will take some work
both by us and by somebody on the core team... and it is a matter of
prioritizing that work.)

Best,

Chris

On Thu, Jan 23, 2014 at 5:06 AM, Bashir Jahed admin@osilab.net wrote:

Hi All

I want to enquire about the best way to handle a very large form, if all
the skips are removed the questionnaire holds about 9000 questions.

I am in the process of finalising the form in the next few days.

I do not want to run into any issues once the form goes live so I am
asking would it be better to split the form into smaller forms, however I
have a single section that is about 7000 questions long and am unable to
split this.

Will i run into any type of limitation.

Also what is the best way to convert the form as sometime the online
converter gives time out errors when trying to convert the form.

Also what would be the best in terms of hosting the form, tomcat or app
engine?

The survey plans to get to a sample size of 3000, do you foresee any
issues with exporting to csv?

I await your responses eagerly

Bashir

--
Bashir Jahed - Director
The OSi LAB (PTY) Ltd
www.osilab.net | 083 414 0453 | bashir@osilab.net

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks for the responses, unfortunately repeat groups will not work as the
survey is very specific and has loads of loops and specifies.

I will post the survey for view once we are done with it and maybe someone
can advise if there is a way to optimise it

Bashir

··· On 23 Jan 2014 20:52, "Yaw Anokwa" wrote:

Agreed with Chris that you are probably asking for trouble. I'd look for
more ways to split it. If you aren't using repeats, might be time to try
them.

I'd probably host on a Tomcat server so you can optimize there too. Maybe
get a beefy Linode?

As far as running into timeouts on the form conversion, try installing
https://github.com/SEL-Columbia/pyxform and running it offline.

Oh, and test with a pilot deployment so you know what to expect when it
goes live.

Yaw

Need ODK services? http://nafundi.com provides form design, server setup,
professional support, and software development for ODK.

On Thu, Jan 23, 2014 at 2:18 AM, Christopher Robert <crobert@surveycto.com wrote:

Bashir,

At SurveyCTO, we have worked to steadily improve support for very long
forms -- both in terms of device performance, and in terms of the memory
required on the device. Unfortunately, we have not yet worked with the ODK
team to merge those changes back into the standard ODK Collect. However, I
will say that 9,000 fields may be beyond what even our much-optimized
SurveyCTO Collect can handle.

Ultimately, it will depend on your device's speed and memory. And when it
comes to memory, it is not simply the total memory on the device, but the
total amount the device will allow any single app to utilize. Thus, there
is no fixed answer regarding what will or won't work. As with an
unfortunate post just a day or two ago, you can even have a form that works
much of the time -- but if it is close to the memory limit, adding a repeat
group can push it over the edge and result in a crash.

You can always try ODK Collect and SurveyCTO Collect. If your form works
on the latter but not the former, then there are two choices: either you
use SurveyCTO, or you convince us and somebody on the core team to merge
our improvements. (We are quite open to contributing our improvements and
have just been discussing it today, in fact, but it will take some work
both by us and by somebody on the core team... and it is a matter of
prioritizing that work.)

Best,

Chris

On Thu, Jan 23, 2014 at 5:06 AM, Bashir Jahed admin@osilab.net wrote:

Hi All

I want to enquire about the best way to handle a very large form, if all
the skips are removed the questionnaire holds about 9000 questions.

I am in the process of finalising the form in the next few days.

I do not want to run into any issues once the form goes live so I am
asking would it be better to split the form into smaller forms, however I
have a single section that is about 7000 questions long and am unable to
split this.

Will i run into any type of limitation.

Also what is the best way to convert the form as sometime the online
converter gives time out errors when trying to convert the form.

Also what would be the best in terms of hosting the form, tomcat or app
engine?

The survey plans to get to a sample size of 3000, do you foresee any
issues with exporting to csv?

I await your responses eagerly

Bashir

--
Bashir Jahed - Director
The OSi LAB (PTY) Ltd
www.osilab.net | 083 414 0453 | bashir@osilab.net

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.