Data Spontaneously Deleted From Devices

Dear all,
We have an Hybrid Android application which uses ODK Collect to do
Demographic Surveillance surveys while the Android app does some custom
functions including pre-filling ODK Forms with data stored on the device
(User Data, Household Data, etc). For reasons I am yet to understand, ODK
stores Finalized Forms after a household visit, and the data can be seen
when one goes to Edit Saved Form. However, after a few forms have been
filled (no specific number) forms erratically start to lose data, and going
into Edit Saved Forms all you see is the form fields but the form is now
blank. This is quite erratic, in the sense that one does not know when
exactly it deletes data (some fieldworkers claim after 5 visits, others
claim after seven visits, etc) but also erratic in the sense that the same
form, e.g a pregnancy-outcome form during the course of an interviewers day
will produce some instances where there is no data deletion and other
instances where data has been deleted.

Could anyone guide me as to where would be the first place to look, in
terms of trying to debug this?

Appreciate any help,
Tumaini

Hi Tumaini,

You have to be careful about relevance when you restore or save a
form. If you open a form, change relevance, and save that form,
JavaROSA will discard any data that isn't relevant. I think the engine
will also throw away empty nodes. I'd start there...

Yaw

··· -- Need ODK services? Go to http://nafundi.com for form design, data cleaning, custom reporting, and software development for ODK.

On Fri, Aug 16, 2013 at 8:23 AM, Tumaini Kilimba tkilimba@ihi.or.tz wrote:

Dear all,
We have an Hybrid Android application which uses ODK Collect to do
Demographic Surveillance surveys while the Android app does some custom
functions including pre-filling ODK Forms with data stored on the device
(User Data, Household Data, etc). For reasons I am yet to understand, ODK
stores Finalized Forms after a household visit, and the data can be seen
when one goes to Edit Saved Form. However, after a few forms have been
filled (no specific number) forms erratically start to lose data, and going
into Edit Saved Forms all you see is the form fields but the form is now
blank. This is quite erratic, in the sense that one does not know when
exactly it deletes data (some fieldworkers claim after 5 visits, others
claim after seven visits, etc) but also erratic in the sense that the same
form, e.g a pregnancy-outcome form during the course of an interviewers day
will produce some instances where there is no data deletion and other
instances where data has been deleted.

Could anyone guide me as to where would be the first place to look, in terms
of trying to debug this?

Appreciate any help,
Tumaini

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hello Yaw, thanks for the response. Could you elaborate a little more
please, when you say "open a form and change its relevance" do you mean
opening the forms xml file in an editor and modifying it?

Also, wrt throwing empty nodes, you mean throwing nodes with no data?
Because in our case, all the data is lost from a saved form.

Appreciate the help,
Tumaini

··· On 17 Aug 2013 01:58, "Yaw Anokwa" wrote:

Hi Tumaini,

You have to be careful about relevance when you restore or save a
form. If you open a form, change relevance, and save that form,
JavaROSA will discard any data that isn't relevant. I think the engine
will also throw away empty nodes. I'd start there...

Yaw

Need ODK services? Go to http://nafundi.com for form design, data
cleaning, custom reporting, and software development for ODK.

On Fri, Aug 16, 2013 at 8:23 AM, Tumaini Kilimba tkilimba@ihi.or.tz wrote:

Dear all,
We have an Hybrid Android application which uses ODK Collect to do
Demographic Surveillance surveys while the Android app does some custom
functions including pre-filling ODK Forms with data stored on the device
(User Data, Household Data, etc). For reasons I am yet to understand, ODK
stores Finalized Forms after a household visit, and the data can be seen
when one goes to Edit Saved Form. However, after a few forms have been
filled (no specific number) forms erratically start to lose data, and
going
into Edit Saved Forms all you see is the form fields but the form is now
blank. This is quite erratic, in the sense that one does not know when
exactly it deletes data (some fieldworkers claim after 5 visits, others
claim after seven visits, etc) but also erratic in the sense that the
same
form, e.g a pregnancy-outcome form during the course of an interviewers
day
will produce some instances where there is no data deletion and other
instances where data has been deleted.

Could anyone guide me as to where would be the first place to look, in
terms
of trying to debug this?

Appreciate any help,
Tumaini

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Tumaini,

I mean editing a saved form on the phone and re-saving it. Let's say
you have a saved form where gender is female and you save relevant
questions (e.g., about pregnancy). Then you edit that saved form in
Collect, switch gender to male, and save the form. All the
non-relevant questions (e.g., about pregnancy) will be discarded.

Empty nodes are nodes like or where you
don't have any data stored inside the node, but the node is present.
If you are generating forms via code, you might think that those nodes
get read by JavaROSA, but that's not necessarily true.

If you are losing all the data in the form, then something else is
probably going on. I'd look at the SQLite DBs and see if there is a
mismatch between the metadata and the XML files. I'd also check to
make sure nothing weird is going on with your SD card (especially if
the problems happen only on some phones).

Yaw

··· -- Need ODK services? Go to http://nafundi.com for form design, data cleaning, custom reporting, and software development for ODK.

On Sat, Aug 17, 2013 at 1:42 AM, Tumaini Kilimba tkilimba@ihi.or.tz wrote:

Hello Yaw, thanks for the response. Could you elaborate a little more
please, when you say "open a form and change its relevance" do you mean
opening the forms xml file in an editor and modifying it?

Also, wrt throwing empty nodes, you mean throwing nodes with no data?
Because in our case, all the data is lost from a saved form.

Appreciate the help,
Tumaini

On 17 Aug 2013 01:58, "Yaw Anokwa" yanokwa@nafundi.com wrote:

Hi Tumaini,

You have to be careful about relevance when you restore or save a
form. If you open a form, change relevance, and save that form,
JavaROSA will discard any data that isn't relevant. I think the engine
will also throw away empty nodes. I'd start there...

Yaw

Need ODK services? Go to http://nafundi.com for form design, data
cleaning, custom reporting, and software development for ODK.

On Fri, Aug 16, 2013 at 8:23 AM, Tumaini Kilimba tkilimba@ihi.or.tz wrote:

Dear all,
We have an Hybrid Android application which uses ODK Collect to do
Demographic Surveillance surveys while the Android app does some custom
functions including pre-filling ODK Forms with data stored on the device
(User Data, Household Data, etc). For reasons I am yet to understand,
ODK
stores Finalized Forms after a household visit, and the data can be seen
when one goes to Edit Saved Form. However, after a few forms have been
filled (no specific number) forms erratically start to lose data, and
going
into Edit Saved Forms all you see is the form fields but the form is now
blank. This is quite erratic, in the sense that one does not know when
exactly it deletes data (some fieldworkers claim after 5 visits, others
claim after seven visits, etc) but also erratic in the sense that the
same
form, e.g a pregnancy-outcome form during the course of an interviewers
day
will produce some instances where there is no data deletion and other
instances where data has been deleted.

Could anyone guide me as to where would be the first place to look, in
terms
of trying to debug this?

Appreciate any help,
Tumaini

--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks againYaw,
whenu say "see if theres a mismatch between the metadata and xml", what do
you mean? Also, within the same device you can have 5 instances of
"pregnancy_outcome" form, yet only two will have no data, and the other
three would be fine. Would a mismatch affect ALL instances of a form, or
could such random selectiveness occur?

Thanks,
Tumaini

Tumaini,

I mean editing a saved form on the phone and re-saving it. Let's say
you have a saved form where gender is female and you save relevant
questions (e.g., about pregnancy). Then you edit that saved form in
Collect, switch gender to male, and save the form. All the
non-relevant questions (e.g., about pregnancy) will be discarded.

Empty nodes are nodes like or where you
don't have any data stored inside the node, but the node is present.
If you are generating forms via code, you might think that those nodes
get read by JavaROSA, but that's not necessarily true.

If you are losing all the data in the form, then something else is
probably going on. I'd look at the SQLite DBs and see if there is a
mismatch between the metadata and the XML files. I'd also check to
make sure nothing weird is going on with your SD card (especially if
the problems happen only on some phones).

Yaw

Need ODK services? Go to http://nafundi.com for form design, data
cleaning, custom reporting, and software development for ODK.

Hello Yaw, thanks for the response. Could you elaborate a little more
please, when you say "open a form and change its relevance" do you mean
opening the forms xml file in an editor and modifying it?

Also, wrt throwing empty nodes, you mean throwing nodes with no data?
Because in our case, all the data is lost from a saved form.

Appreciate the help,
Tumaini

Hi Tumaini,

You have to be careful about relevance when you restore or save a
form. If you open a form, change relevance, and save that form,
JavaROSA will discard any data that isn't relevant. I think the engine
will also throw away empty nodes. I'd start there...

Yaw

Need ODK services? Go to http://nafundi.com for form design, data
cleaning, custom reporting, and software development for ODK.

Dear all,
We have an Hybrid Android application which uses ODK Collect to do
Demographic Surveillance surveys while the Android app does some
custom

functions including pre-filling ODK Forms with data stored on the
device

(User Data, Household Data, etc). For reasons I am yet to understand,
ODK
stores Finalized Forms after a household visit, and the data can be
seen

when one goes to Edit Saved Form. However, after a few forms have
been

filled (no specific number) forms erratically start to lose data, and
going
into Edit Saved Forms all you see is the form fields but the form is
now

blank. This is quite erratic, in the sense that one does not know
when

exactly it deletes data (some fieldworkers claim after 5 visits,
others

claim after seven visits, etc) but also erratic in the sense that the
same
form, e.g a pregnancy-outcome form during the course of an
interviewers

day
will produce some instances where there is no data deletion and other
instances where data has been deleted.

Could anyone guide me as to where would be the first place to look,
in

terms
of trying to debug this?

Appreciate any help,
Tumaini

--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send

an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups

"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an

email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups

"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an

email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.

··· On 17 Aug 2013 15:51, "Yaw Anokwa" wrote: > On Sat, Aug 17, 2013 at 1:42 AM, Tumaini Kilimba wrote: > > On 17 Aug 2013 01:58, "Yaw Anokwa" wrote: > >> On Fri, Aug 16, 2013 at 8:23 AM, Tumaini Kilimba wrote: > For more options, visit https://groups.google.com/groups/opt_out.

If you have a DB entry but no XML or vice versa, then that's a place
to start. Or if the timestamps of when the DB said you updated a file
and the file itself differ by a lot, then that's a place to start. My
intuition is that if you have some sort of DB or filesystem issue, it
could be random. Anyway, all this is speculation. Best thing to do is
start from the code you changed, look through the /odk folder of the
good devices and the bad, add logging, test, etc.

··· On Sat, Aug 17, 2013 at 6:02 AM, Tumaini Kilimba wrote: > Thanks againYaw, > whenu say "see if theres a mismatch between the metadata and xml", what do > you mean? Also, within the same device you can have 5 instances of > "pregnancy_outcome" form, yet only two will have no data, and the other > three would be fine. Would a mismatch affect ALL instances of a form, or > could such random selectiveness occur? > > Thanks, > Tumaini > > > On 17 Aug 2013 15:51, "Yaw Anokwa" wrote: >> >> Tumaini, >> >> I mean editing a saved form on the phone and re-saving it. Let's say >> you have a saved form where gender is female and you save relevant >> questions (e.g., about pregnancy). Then you edit that saved form in >> Collect, switch gender to male, and save the form. All the >> non-relevant questions (e.g., about pregnancy) will be discarded. >> >> Empty nodes are nodes like or where you >> don't have any data stored inside the node, but the node is present. >> If you are generating forms via code, you might think that those nodes >> get read by JavaROSA, but that's not necessarily true. >> >> If you are losing all the data in the form, then something else is >> probably going on. I'd look at the SQLite DBs and see if there is a >> mismatch between the metadata and the XML files. I'd also check to >> make sure nothing weird is going on with your SD card (especially if >> the problems happen only on some phones). >> >> Yaw >> -- >> Need ODK services? Go to http://nafundi.com for form design, data >> cleaning, custom reporting, and software development for ODK. >> >> On Sat, Aug 17, 2013 at 1:42 AM, Tumaini Kilimba wrote: >> > Hello Yaw, thanks for the response. Could you elaborate a little more >> > please, when you say "open a form and change its relevance" do you mean >> > opening the forms xml file in an editor and modifying it? >> > >> > Also, wrt throwing empty nodes, you mean throwing nodes with no data? >> > Because in our case, all the data is lost from a saved form. >> > >> > Appreciate the help, >> > Tumaini >> > >> > On 17 Aug 2013 01:58, "Yaw Anokwa" wrote: >> >> >> >> Hi Tumaini, >> >> >> >> You have to be careful about relevance when you restore or save a >> >> form. If you open a form, change relevance, and save that form, >> >> JavaROSA will discard any data that isn't relevant. I think the engine >> >> will also throw away empty nodes. I'd start there... >> >> >> >> Yaw >> >> -- >> >> Need ODK services? Go to http://nafundi.com for form design, data >> >> cleaning, custom reporting, and software development for ODK. >> >> >> >> >> >> On Fri, Aug 16, 2013 at 8:23 AM, Tumaini Kilimba wrote: >> >> > Dear all, >> >> > We have an Hybrid Android application which uses ODK Collect to do >> >> > Demographic Surveillance surveys while the Android app does some >> >> > custom >> >> > functions including pre-filling ODK Forms with data stored on the >> >> > device >> >> > (User Data, Household Data, etc). For reasons I am yet to understand, >> >> > ODK >> >> > stores Finalized Forms after a household visit, and the data can be >> >> > seen >> >> > when one goes to Edit Saved Form. However, after a few forms have >> >> > been >> >> > filled (no specific number) forms erratically start to lose data, and >> >> > going >> >> > into Edit Saved Forms all you see is the form fields but the form is >> >> > now >> >> > blank. This is quite erratic, in the sense that one does not know >> >> > when >> >> > exactly it deletes data (some fieldworkers claim after 5 visits, >> >> > others >> >> > claim after seven visits, etc) but also erratic in the sense that the >> >> > same >> >> > form, e.g a pregnancy-outcome form during the course of an >> >> > interviewers >> >> > day >> >> > will produce some instances where there is no data deletion and other >> >> > instances where data has been deleted. >> >> > >> >> > Could anyone guide me as to where would be the first place to look, >> >> > in >> >> > terms >> >> > of trying to debug this? >> >> > >> >> > Appreciate any help, >> >> > Tumaini >> >> > >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups >> >> > "ODK Developers" group. >> >> > To unsubscribe from this group and stop receiving emails from it, >> >> > send >> >> > an >> >> > email to opendatakit-developers+unsubscribe@googlegroups.com. >> >> > For more options, visit https://groups.google.com/groups/opt_out. >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups >> >> "ODK Developers" group. >> >> To unsubscribe from this group and stop receiving emails from it, send >> >> an >> >> email to opendatakit-developers+unsubscribe@googlegroups.com. >> >> For more options, visit https://groups.google.com/groups/opt_out. >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "ODK Developers" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to opendatakit-developers+unsubscribe@googlegroups.com. >> > For more options, visit https://groups.google.com/groups/opt_out. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "ODK Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit-developers+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/groups/opt_out. > > -- > You received this message because you are subscribed to the Google Groups > "ODK Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to opendatakit-developers+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out.