Updating a form during Ongoing Survey

Dear all,
I am about to start using an ODK form in the field for health data
collection. I get the feeling that during usage there may be changes
required of the form (even though testing has been done extensively,
but there is no substitute for the real thing). How do these changes
during the study affect the final quality of my data? How easy is it
to reconcile data collected with two different version of the same
form (the differences may just be the addition of a further question,
etc, not major surgery)? What are the pitfalls, what should I avoid
doing, etc.

Any advice would be greatly appreciated,
Tumaini

Tumaini,

Currently, any change of a form is a new form. If you change the
wording, it's a new form. If you change the structure (i.e., add or
remove a prompt), it's a new form. The implications of any forms
changes are thus the same as adding new form to Aggregate.

You can make the old form not downloadable (essentially unpublishing
it), and then make the new form downloadable (maybe add a v2.0 to the
title). Enumerators will have to download this new form, and when you
export from Aggregate, you will get a separate file.

You can also merge the exported files (either using Excel or perhaps
in Fusion Tables) after the fact. If the change is minor it should be
straightforward. Either way, Aggregate will not help you automatically
merge two versions of the form.

There is ongoing work in Aggregate to support non-structural changes
to forms without creating a new form, but no ETA on that. See
https://groups.google.com/group/opendatakit-developers/browse_thread/thread/ae6d2ffb9cc07a52
for more. There are other workarounds/hacks, but I can't seem to find
the relevant thread...

Hope that helps,

Yaw

ยทยทยท On Sun, May 20, 2012 at 3:31 AM, Tumaini Kilimba wrote: > Dear all, > I am about to start using an ODK form in the field for health data > collection. I get the feeling that during usage there may be changes > required of the form (even though testing has been done extensively, > but there is no substitute for the real thing). How do these changes > during the study affect the final quality of my data? How easy is it > to reconcile data collected with two different version of the same > form (the differences may just be the addition of a further question, > etc, not major surgery)? What are the pitfalls, what should I avoid > doing, etc. > > Any advice would be greatly appreciated, > Tumaini > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en
1 Like

Thanks Yaw, your answer seems clear enough so forgive me if I seem to
be asking the obvious; What if am not adding or removing a question on
a form, all am doing is adding an option on a question, e.g a
question may initially have the options YES or NO, then at a
later date it is decided it also needs the option of MAYBE. So am not
making any structural changes to the database or instance (I dont
think), am just adding another response option that can go into an
existing column. Would this still be considered as a new form?

Apologies again if its a redundant question considering the extensive
explanation you have given.
Tumaini

ยทยทยท On May 20, 11:00 pm, Yaw Anokwa wrote: > Tumaini, > > Currently, any change of a form is a new form. If you change the > wording, it's a new form. If you change the structure (i.e., add or > remove a prompt), it's a new form. The implications of any forms > changes are thus the same as adding new form to Aggregate. > > You can make the old form not downloadable (essentially unpublishing > it), and then make the new form downloadable (maybe add a v2.0 to the > title). Enumerators will have to download this new form, and when you > export from Aggregate, you will get a separate file. > > You can also merge the exported files (either using Excel or perhaps > in Fusion Tables) after the fact. If the change is minor it should be > straightforward. Either way, Aggregate will not help you automatically > merge two versions of the form. > > There is ongoing work in Aggregate to support non-structural changes > to forms without creating a new form, but no ETA on that. Seehttps://groups.google.com/group/opendatakit-developers/browse_thread/... > for more. There are other workarounds/hacks, but I can't seem to find > the relevant thread... > > Hope that helps, > > Yaw > > > > > > > > On Sun, May 20, 2012 at 3:31 AM, Tumaini Kilimba wrote: > > Dear all, > > I am about to start using an ODK form in the field for health data > > collection. I get the feeling that during usage there may be changes > > required of the form (even though testing has been done extensively, > > but there is no substitute for the real thing). How do these changes > > during the study affect the final quality of my data? How easy is it > > to reconcile data collected with two different version of the same > > form (the differences may just be the addition of a further question, > > etc, not major surgery)? What are the pitfalls, what should I avoid > > doing, etc. > > > Any advice would be greatly appreciated, > > Tumaini > > > -- > > Post: opendatakit@googlegroups.com > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > Options:http://groups.google.com/group/opendatakit?hl=en

Hello,
in my situation, I have two separate instances of Aggregate running on
Tomcat, each with a survey form with very slight differences (one ore two
added columns, some text changes, data type changes). I want to merge the
data from the first instance into the second instance via Excel. My
questions in regards to merging are:

  1. The tables which hold the survey data (all those whose table names DO
    NOT start with an underscore in Postgres) are the only tables I need to
    export to CSV for merging. Is this correct?
  2. if the merge is successful, onto the new Aggregate instance which
    already has data, when I navigate to my instance and go to view
    submissions, will this data that I recently added via excel also be visible?
  3. How will the change in data types affect the eventual database (in
    the original form, there were some elements that had a datatype
    of 'int' rather than 'string')

Thanks in advance,
Tumaini

ยทยทยท On Sunday, May 20, 2012 11:00:49 PM UTC+3, Yaw Anokwa wrote: > > Tumaini, > > Currently, any change of a form is a new form. If you change the > wording, it's a new form. If you change the structure (i.e., add or > remove a prompt), it's a new form. The implications of any forms > changes are thus the same as adding new form to Aggregate. > > You can make the old form not downloadable (essentially unpublishing > it), and then make the new form downloadable (maybe add a v2.0 to the > title). Enumerators will have to download this new form, and when you > export from Aggregate, you will get a separate file. > > You can also merge the exported files (either using Excel or perhaps > in Fusion Tables) after the fact. If the change is minor it should be > straightforward. Either way, Aggregate will not help you automatically > merge two versions of the form. > > There is ongoing work in Aggregate to support non-structural changes > to forms without creating a new form, but no ETA on that. See > > https://groups.google.com/group/opendatakit-developers/browse_thread/thread/ae6d2ffb9cc07a52 > for more. There are other workarounds/hacks, but I can't seem to find > the relevant thread... > > Hope that helps, > > Yaw > > On Sun, May 20, 2012 at 3:31 AM, Tumaini Kilimba wrote: > > Dear all, > > I am about to start using an ODK form in the field for health data > > collection. I get the feeling that during usage there may be changes > > required of the form (even though testing has been done extensively, > > but there is no substitute for the real thing). How do these changes > > during the study affect the final quality of my data? How easy is it > > to reconcile data collected with two different version of the same > > form (the differences may just be the addition of a further question, > > etc, not major surgery)? What are the pitfalls, what should I avoid > > doing, etc. > > > > Any advice would be greatly appreciated, > > Tumaini > > > > -- > > Post: opendatakit@googlegroups.com > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en >

Any change (even changing one letter in the text) makes it a new form.
In the future, it might not be, but right now it is.

ยทยทยท On Mon, May 21, 2012 at 1:18 AM, Tumaini Kilimba wrote: > Thanks Yaw, your answer seems clear enough so forgive me if I seem to > be asking the obvious; What if am not adding or removing a question on > a form, all am doing is adding an option on a question, e.g a > question may initially have the options YES or NO, then at a > later date it is decided it also needs the option of MAYBE. So am not > making any structural changes to the database or instance (I dont > think), am just adding another response option that can go into an > existing column. Would this still be considered as a new form? > > Apologies again if its a redundant question considering the extensive > explanation you have given. > Tumaini > > On May 20, 11:00 pm, Yaw Anokwa wrote: >> Tumaini, >> >> Currently, any change of a form is a new form. If you change the >> wording, it's a new form. If you change the structure (i.e., add or >> remove a prompt), it's a new form. The implications of any forms >> changes are thus the same as adding new form to Aggregate. >> >> You can make the old form not downloadable (essentially unpublishing >> it), and then make the new form downloadable (maybe add a v2.0 to the >> title). Enumerators will have to download this new form, and when you >> export from Aggregate, you will get a separate file. >> >> You can also merge the exported files (either using Excel or perhaps >> in Fusion Tables) after the fact. If the change is minor it should be >> straightforward. Either way, Aggregate will not help you automatically >> merge two versions of the form. >> >> There is ongoing work in Aggregate to support non-structural changes >> to forms without creating a new form, but no ETA on that. Seehttps://groups.google.com/group/opendatakit-developers/browse_thread/... >> for more. There are other workarounds/hacks, but I can't seem to find >> the relevant thread... >> >> Hope that helps, >> >> Yaw >> >> >> >> >> >> >> >> On Sun, May 20, 2012 at 3:31 AM, Tumaini Kilimba wrote: >> > Dear all, >> > I am about to start using an ODK form in the field for health data >> > collection. I get the feeling that during usage there may be changes >> > required of the form (even though testing has been done extensively, >> > but there is no substitute for the real thing). How do these changes >> > during the study affect the final quality of my data? How easy is it >> > to reconcile data collected with two different version of the same >> > form (the differences may just be the addition of a further question, >> > etc, not major surgery)? What are the pitfalls, what should I avoid >> > doing, etc. >> >> > Any advice would be greatly appreciated, >> > Tumaini >> >> > -- >> > Post: opendatakit@googlegroups.com >> > Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> > Options:http://groups.google.com/group/opendatakit?hl=en > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en

(1) Yes, the survey data is only in the non-leading-underscore tables.

(2) & (3) this is not supported by ODK Aggregate and would be a
do-at-your-own-risk operation. The database will generally make reasonable
conversions between integer and string values. In general, if you were to
do this, you would typically write a SQL script and avoid all potential
issues with CSV representation of the data. (e.g., INSERT INTO tablename
(...) SELECT ... )

If the data appears in the table, it will be displayed. You don't need to
do anything special. If you add columns to a table, they will not be
displayed (only the columns in the _FORM_DATA_MODEL are considered part of
a form and displayable). Because that table defines the datatype of a
column, changing the datatype of a column will generally break unless you
fiddle with the definition in that table. The one case where this will
work is if you increase the length of a string or blob column; that is
always permitted and the change will be picked up the next time ODK
Aggregate is started.

Mitch

ยทยทยท On Thu, Jun 28, 2012 at 3:09 AM, Tumaini Kilimba wrote:

Hello,
in my situation, I have two separate instances of Aggregate running on
Tomcat, each with a survey form with very slight differences (one ore two
added columns, some text changes, data type changes). I want to merge the
data from the first instance into the second instance via Excel. My
questions in regards to merging are:

  1. The tables which hold the survey data (all those whose table names
    DO NOT start with an underscore in Postgres) are the only tables I need to
    export to CSV for merging. Is this correct?
  2. if the merge is successful, onto the new Aggregate instance which
    already has data, when I navigate to my instance and go to view
    submissions, will this data that I recently added via excel also be visible?
  3. How will the change in data types affect the eventual database (in
    the original form, there were some elements that had a datatype
    of 'int' rather than 'string')

Thanks in advance,
Tumaini

On Sunday, May 20, 2012 11:00:49 PM UTC+3, Yaw Anokwa wrote:

Tumaini,

Currently, any change of a form is a new form. If you change the
wording, it's a new form. If you change the structure (i.e., add or
remove a prompt), it's a new form. The implications of any forms
changes are thus the same as adding new form to Aggregate.

You can make the old form not downloadable (essentially unpublishing
it), and then make the new form downloadable (maybe add a v2.0 to the
title). Enumerators will have to download this new form, and when you
export from Aggregate, you will get a separate file.

You can also merge the exported files (either using Excel or perhaps
in Fusion Tables) after the fact. If the change is minor it should be
straightforward. Either way, Aggregate will not help you automatically
merge two versions of the form.

There is ongoing work in Aggregate to support non-structural changes
to forms without creating a new form, but no ETA on that. See
https://groups.google.com/group/opendatakit-developers/
browse_thread/thread/**ae6d2ffb9cc07a52https://groups.google.com/group/opendatakit-developers/browse_thread/thread/ae6d2ffb9cc07a52
for more. There are other workarounds/hacks, but I can't seem to find
the relevant thread...

Hope that helps,

Yaw

On Sun, May 20, 2012 at 3:31 AM, Tumaini Kilimba tkilimba@ihi.or.tz wrote:

Dear all,
I am about to start using an ODK form in the field for health data
collection. I get the feeling that during usage there may be changes
required of the form (even though testing has been done extensively,
but there is no substitute for the real thing). How do these changes
during the study affect the final quality of my data? How easy is it
to reconcile data collected with two different version of the same
form (the differences may just be the addition of a further question,
etc, not major surgery)? What are the pitfalls, what should I avoid
doing, etc.

Any advice would be greatly appreciated,
Tumaini

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@**googlegroups.comopendatakit%2Bunsubscribe@googlegroups.com
Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Thanks Mitch,
Also i was wondering. In the first incarnation of the form, there were
questions such as "what Antenatal Care Treatment have you had?" followed by
element which offers options such as PMTCT,ANAEMIA,OTHER. if what
was chosen is the option OTHER, then the next question would be "WHAT OTHER
TREATMENT" which would provide an element allowing them to specify
just ONE other treatment. It was decided we needed to give the option of
specifying more than one OTHER in case respondent had more than one answer.
So this other was put in a repeating group whivh would loop to ask if there
was any other response. About 30 questions have this repeating OTHER
option,whereas the initial form did not. What would be the best way of
merging the old single response to OTHER which goes on the same FORM_CORE
table wit h the new data which has these OTHER responses in separate tables
(as all repeating groups/loops go in separate tables)?

Thanks,
Tumaini

ยทยทยท On 28 Jun 2012 20:17, "Mitch S" wrote:

(1) Yes, the survey data is only in the non-leading-underscore tables.

(2) & (3) this is not supported by ODK Aggregate and would be a
do-at-your-own-risk operation. The database will generally make reasonable
conversions between integer and string values. In general, if you were to
do this, you would typically write a SQL script and avoid all potential
issues with CSV representation of the data. (e.g., INSERT INTO tablename
(...) SELECT ... )

If the data appears in the table, it will be displayed. You don't need to
do anything special. If you add columns to a table, they will not be
displayed (only the columns in the _FORM_DATA_MODEL are considered part of
a form and displayable). Because that table defines the datatype of a
column, changing the datatype of a column will generally break unless you
fiddle with the definition in that table. The one case where this will
work is if you increase the length of a string or blob column; that is
always permitted and the change will be picked up the next time ODK
Aggregate is started.

Mitch

On Thu, Jun 28, 2012 at 3:09 AM, Tumaini Kilimba tkilimba@ihi.or.tzwrote:

Hello,
in my situation, I have two separate instances of Aggregate running on
Tomcat, each with a survey form with very slight differences (one ore two
added columns, some text changes, data type changes). I want to merge the
data from the first instance into the second instance via Excel. My
questions in regards to merging are:

  1. The tables which hold the survey data (all those whose table names
    DO NOT start with an underscore in Postgres) are the only tables I need to
    export to CSV for merging. Is this correct?
  2. if the merge is successful, onto the new Aggregate instance which
    already has data, when I navigate to my instance and go to view
    submissions, will this data that I recently added via excel also be visible?
  3. How will the change in data types affect the eventual database (in
    the original form, there were some elements that had a datatype
    of 'int' rather than 'string')

Thanks in advance,
Tumaini

On Sunday, May 20, 2012 11:00:49 PM UTC+3, Yaw Anokwa wrote:

Tumaini,

Currently, any change of a form is a new form. If you change the
wording, it's a new form. If you change the structure (i.e., add or
remove a prompt), it's a new form. The implications of any forms
changes are thus the same as adding new form to Aggregate.

You can make the old form not downloadable (essentially unpublishing
it), and then make the new form downloadable (maybe add a v2.0 to the
title). Enumerators will have to download this new form, and when you
export from Aggregate, you will get a separate file.

You can also merge the exported files (either using Excel or perhaps
in Fusion Tables) after the fact. If the change is minor it should be
straightforward. Either way, Aggregate will not help you automatically
merge two versions of the form.

There is ongoing work in Aggregate to support non-structural changes
to forms without creating a new form, but no ETA on that. See
https://groups.google.com/group/opendatakit-developers/
browse_thread/thread/**ae6d2ffb9cc07a52https://groups.google.com/group/opendatakit-developers/browse_thread/thread/ae6d2ffb9cc07a52
for more. There are other workarounds/hacks, but I can't seem to find
the relevant thread...

Hope that helps,

Yaw

On Sun, May 20, 2012 at 3:31 AM, Tumaini Kilimba tkilimba@ihi.or.tz wrote:

Dear all,
I am about to start using an ODK form in the field for health data
collection. I get the feeling that during usage there may be changes
required of the form (even though testing has been done extensively,
but there is no substitute for the real thing). How do these changes
during the study affect the final quality of my data? How easy is it
to reconcile data collected with two different version of the same
form (the differences may just be the addition of a further question,
etc, not major surgery)? What are the pitfalls, what should I avoid
doing, etc.

Any advice would be greatly appreciated,
Tumaini

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@**googlegroups.comopendatakit%2Bunsubscribe@googlegroups.com
Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

Note: This is becoming an opendatakit-developers@ e-mail, as you're getting
into the internals of the datastore.

You should be able to create records in the repeat-group table for each of
the original non-null "OTHER" entries. The table structure is pretty easy:

_TOP_LEVEL_AURI == _URI of record in top-level (_CORE) table of which this
is a part
_PARENT_AURI == _URI of record in enclosing parent table (would be the same
as _TOP_LEVEL_AURI unless this is a repeat group nested within another
repeat group).
_ORDINAL_NUMBER == 1

Again, I'd use SQL scripts for this.

Mitch

ยทยทยท On Thu, Jun 28, 2012 at 10:59 AM, Tumaini Kilimba wrote:

Thanks Mitch,
Also i was wondering. In the first incarnation of the form, there were
questions such as "what Antenatal Care Treatment have you had?" followed by
element which offers options such as PMTCT,ANAEMIA,OTHER. if what
was chosen is the option OTHER, then the next question would be "WHAT OTHER
TREATMENT" which would provide an element allowing them to specify
just ONE other treatment. It was decided we needed to give the option of
specifying more than one OTHER in case respondent had more than one answer.
So this other was put in a repeating group whivh would loop to ask if there
was any other response. About 30 questions have this repeating OTHER
option,whereas the initial form did not. What would be the best way of
merging the old single response to OTHER which goes on the same FORM_CORE
table wit h the new data which has these OTHER responses in separate tables
(as all repeating groups/loops go in separate tables)?

Thanks,
Tumaini
On 28 Jun 2012 20:17, "Mitch S" mitchellsundt@gmail.com wrote:

(1) Yes, the survey data is only in the non-leading-underscore tables.

(2) & (3) this is not supported by ODK Aggregate and would be a
do-at-your-own-risk operation. The database will generally make reasonable
conversions between integer and string values. In general, if you were to
do this, you would typically write a SQL script and avoid all potential
issues with CSV representation of the data. (e.g., INSERT INTO tablename
(...) SELECT ... )

If the data appears in the table, it will be displayed. You don't need
to do anything special. If you add columns to a table, they will not be
displayed (only the columns in the _FORM_DATA_MODEL are considered part of
a form and displayable). Because that table defines the datatype of a
column, changing the datatype of a column will generally break unless you
fiddle with the definition in that table. The one case where this will
work is if you increase the length of a string or blob column; that is
always permitted and the change will be picked up the next time ODK
Aggregate is started.

Mitch

On Thu, Jun 28, 2012 at 3:09 AM, Tumaini Kilimba tkilimba@ihi.or.tzwrote:

Hello,
in my situation, I have two separate instances of Aggregate running on
Tomcat, each with a survey form with very slight differences (one ore two
added columns, some text changes, data type changes). I want to merge the
data from the first instance into the second instance via Excel. My
questions in regards to merging are:

  1. The tables which hold the survey data (all those whose table
    names DO NOT start with an underscore in Postgres) are the only tables I
    need to export to CSV for merging. Is this correct?
  2. if the merge is successful, onto the new Aggregate instance which
    already has data, when I navigate to my instance and go to view
    submissions, will this data that I recently added via excel also be visible?
  3. How will the change in data types affect the eventual database
    (in the original form, there were some elements that had a
    datatype of 'int' rather than 'string')

Thanks in advance,
Tumaini

On Sunday, May 20, 2012 11:00:49 PM UTC+3, Yaw Anokwa wrote:

Tumaini,

Currently, any change of a form is a new form. If you change the
wording, it's a new form. If you change the structure (i.e., add or
remove a prompt), it's a new form. The implications of any forms
changes are thus the same as adding new form to Aggregate.

You can make the old form not downloadable (essentially unpublishing
it), and then make the new form downloadable (maybe add a v2.0 to the
title). Enumerators will have to download this new form, and when you
export from Aggregate, you will get a separate file.

You can also merge the exported files (either using Excel or perhaps
in Fusion Tables) after the fact. If the change is minor it should be
straightforward. Either way, Aggregate will not help you automatically
merge two versions of the form.

There is ongoing work in Aggregate to support non-structural changes
to forms without creating a new form, but no ETA on that. See
https://groups.google.com/group/opendatakit-developers/
browse_thread/thread/**ae6d2ffb9cc07a52https://groups.google.com/group/opendatakit-developers/browse_thread/thread/ae6d2ffb9cc07a52
for more. There are other workarounds/hacks, but I can't seem to find
the relevant thread...

Hope that helps,

Yaw

On Sun, May 20, 2012 at 3:31 AM, Tumaini Kilimba tkilimba@ihi.or.tz wrote:

Dear all,
I am about to start using an ODK form in the field for health data
collection. I get the feeling that during usage there may be changes
required of the form (even though testing has been done extensively,
but there is no substitute for the real thing). How do these changes
during the study affect the final quality of my data? How easy is it
to reconcile data collected with two different version of the same
form (the differences may just be the addition of a further question,
etc, not major surgery)? What are the pitfalls, what should I avoid
doing, etc.

Any advice would be greatly appreciated,
Tumaini

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@**googlegroups.comopendatakit%2Bunsubscribe@googlegroups.com
Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Dear Yaw,

Is there any feature available in the current versions, that allows merging data of ongoing survey in case any changes are made to the form and re-upload it to the Aggregate.

Regards,
Imran