Cannot export to CSV

Hi

I set up the ODK Aggregate with Google App Engine as the back end server in 2013. The ODK aggregate was 1.3.4 version. I upgraded it upto 1.4.1. Now, ODK Aggregate cannot export the forms to CSV neither it is publishing the forms as Google Spreadsheets or Fusion tables.
I was told to upgrade the ODK Aggregate to the latest version but since I am not able to back up the data (one of the questionnaires has more than 15,000 entries), is it safe to upgrade the Aggregate without backing up the data?
I tried using ODK Briefcase to pull the data but it failed after fetching 1400 instances.
And when I click the publish button in the Aggregate, the following error pops up;
Error: Problem persisting data or accessing data (Somehow DB entities for publisher got into problem state)

Please help!

Thanks

Poornima

Did this problem start happening immediately after the upgrade, or were you
collecting data for a period of time, and then it started happening?

If it happened immediately after the upgrade, please invite me (
mitchellsundt@gmail.com ) as a developer so that I can confirm that the
AppEngine instance is still in good condition.

Assuming I determine that everything is OK, or if this started happening
after collecting data after the upgrade, you should then upgrade to ODK
Aggregate 1.4.5.

The error message may indicate that there is a data corruption issue in one
of your submissions. The ODK Aggregate 1.4.5 release provides better
logging of the details of that problem.

··· On Wed, Apr 8, 2015 at 10:45 AM, wrote:

Hi

I set up the ODK Aggregate with Google App Engine as the back end server
in 2013. The ODK aggregate was 1.3.4 version. I upgraded it upto 1.4.1.
Now, ODK Aggregate cannot export the forms to CSV neither it is publishing
the forms as Google Spreadsheets or Fusion tables.
I was told to upgrade the ODK Aggregate to the latest version but since I
am not able to back up the data (one of the questionnaires has more than
15,000 entries), is it safe to upgrade the Aggregate without backing up the
data?
I tried using ODK Briefcase to pull the data but it failed after fetching
1400 instances.
And when I click the publish button in the Aggregate, the following error
pops up;
Error: Problem persisting data or accessing data (Somehow DB entities for
publisher got into problem state)

Please help!

Thanks

Poornima

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Did this problem start happening immediately after the upgrade, or were you collecting data for a period of time, and then it started happening?

If it happened immediately after the upgrade, please invite me ( mitche...@gmail.com ) as a developer so that I can confirm that the AppEngine instance is still in good condition.

Assuming I determine that everything is OK, or if this started happening after collecting data after the upgrade, you should then upgrade to ODK Aggregate 1.4.5.

The error message may indicate that there is a data corruption issue in one of your submissions. The ODK Aggregate 1.4.5 release provides better logging of the details of that problem.

Hi

I set up the ODK Aggregate with Google App Engine as the back end server in 2013. The ODK aggregate was 1.3.4 version. I upgraded it upto 1.4.1. Now, ODK Aggregate cannot export the forms to CSV neither it is publishing the forms as Google Spreadsheets or Fusion tables.

I was told to upgrade the ODK Aggregate to the latest version but since I am not able to back up the data (one of the questionnaires has more than 15,000 entries), is it safe to upgrade the Aggregate without backing up the data?

I tried using ODK Briefcase to pull the data but it failed after fetching 1400 instances.

And when I click the publish button in the Aggregate, the following error pops up;

Error: Problem persisting data or accessing data (Somehow DB entities for publisher got into problem state)

Please help!

Thanks

Poornima

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

Hello Mitch

Thank You for your quick response.
We were collecting data for a long time now. The problem of exporting to CSV or publishing the data started before upgrading the Aggregate.
I could upgrade 2 versions up (currently we are using version 1.4.1). We don't want to take risk of upgrading to the latest versions without backing up the data.
Is there any way to store the data before proceeding to the upgrading of the Aggregate?

And also is there any way to find out the corrupted data in the Submissions?

Thanks

Poornima

··· On Wednesday, April 8, 2015 at 3:16:48 PM UTC-4, Mitch Sundt wrote: > On Wed, Apr 8, 2015 at 10:45 AM, wrote:

Our data-backup mechanism is to use ODK Briefcase. To do the remaining
upgrades more safely (logging was only added in 1.4.5 to address the
problem you are currently having, so we need to upgrade to 1.4.5 to fix it):

(1) uncheck "Accept Submissions" from the Forms Management page for all of
your forms. This prevents your users from submitting data into your forms.
(2) Run ODK Briefcase. Pull every form down to your laptop. This should
succeed for every form except the one(s) that have this data corruption
issue. You may need to pull a form at a time into ODK Briefcase, but I
think you can just check and pull everything.

Because there is the problem with a submission in one or more of your
forms, you do have the potential to lose all data submitted for that formId
after that problematic row was submitted (data submitted into other forms
is not impacted, and will not be lose).

(3) Upgrade to ODK Aggregate 1.4.5 (I would upgrade to 1.4.2, log in,
upgrade to 1.4.3, log in, upgrade to 1.4.4, log in, and then upgrade to
1.4.5). Doing it one at a time, and performing the manual upgrade steps
described in the release notes will reduce the likelihood of problems with
the upgrade.
https://code.google.com/p/opendatakit/wiki/AggregateReleaseNotes

··· ---------------------- Once you have upgraded, and you have logged in, the publishers will resume and try to publish to the server. These will fail due to the bad submission, and there will be a log entry on the AppEngine logs that identifies the formId and the instanceId of the bad submission, and the table with the bad fields.

Using that, you can figure out what the problems are with that submission
and which extra records need to be deleted.

On Wed, Apr 8, 2015 at 12:52 PM, poornimak20@gmail.com wrote:

On Wednesday, April 8, 2015 at 3:16:48 PM UTC-4, Mitch Sundt wrote:

Did this problem start happening immediately after the upgrade, or were
you collecting data for a period of time, and then it started happening?

If it happened immediately after the upgrade, please invite me (
mitche...@gmail.com ) as a developer so that I can confirm that the
AppEngine instance is still in good condition.

Assuming I determine that everything is OK, or if this started happening
after collecting data after the upgrade, you should then upgrade to ODK
Aggregate 1.4.5.

The error message may indicate that there is a data corruption issue in
one of your submissions. The ODK Aggregate 1.4.5 release provides better
logging of the details of that problem.

On Wed, Apr 8, 2015 at 10:45 AM, poorn...@gmail.com wrote:
Hi

I set up the ODK Aggregate with Google App Engine as the back end server
in 2013. The ODK aggregate was 1.3.4 version. I upgraded it upto 1.4.1.
Now, ODK Aggregate cannot export the forms to CSV neither it is publishing
the forms as Google Spreadsheets or Fusion tables.

I was told to upgrade the ODK Aggregate to the latest version but since
I am not able to back up the data (one of the questionnaires has more than
15,000 entries), is it safe to upgrade the Aggregate without backing up the
data?

I tried using ODK Briefcase to pull the data but it failed after
fetching 1400 instances.

And when I click the publish button in the Aggregate, the following
error pops up;

Error: Problem persisting data or accessing data (Somehow DB entities
for publisher got into problem state)

Please help!

Thanks

Poornima

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

Hello Mitch

Thank You for your quick response.
We were collecting data for a long time now. The problem of exporting to
CSV or publishing the data started before upgrading the Aggregate.
I could upgrade 2 versions up (currently we are using version 1.4.1). We
don't want to take risk of upgrading to the latest versions without backing
up the data.
Is there any way to store the data before proceeding to the upgrading of
the Aggregate?

And also is there any way to find out the corrupted data in the
Submissions?

Thanks

Poornima

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

And again, to do that, it is perhaps best to invite me as a developer
https://code.google.com/p/opendatakit/wiki/AppEngineTroubleshooting

If you have software and database expertise, I can provide more detail and
you can try it yourself. But I would generally not recommend doing so.

··· On Wed, Apr 8, 2015 at 1:17 PM, Mitch Sundt wrote:

Our data-backup mechanism is to use ODK Briefcase. To do the remaining
upgrades more safely (logging was only added in 1.4.5 to address the
problem you are currently having, so we need to upgrade to 1.4.5 to fix it):

(1) uncheck "Accept Submissions" from the Forms Management page for all of
your forms. This prevents your users from submitting data into your forms.
(2) Run ODK Briefcase. Pull every form down to your laptop. This should
succeed for every form except the one(s) that have this data corruption
issue. You may need to pull a form at a time into ODK Briefcase, but I
think you can just check and pull everything.

Because there is the problem with a submission in one or more of your
forms, you do have the potential to lose all data submitted for that formId
after that problematic row was submitted (data submitted into other forms
is not impacted, and will not be lose).

(3) Upgrade to ODK Aggregate 1.4.5 (I would upgrade to 1.4.2, log in,
upgrade to 1.4.3, log in, upgrade to 1.4.4, log in, and then upgrade to
1.4.5). Doing it one at a time, and performing the manual upgrade steps
described in the release notes will reduce the likelihood of problems with
the upgrade.
https://code.google.com/p/opendatakit/wiki/AggregateReleaseNotes


Once you have upgraded, and you have logged in, the publishers will resume
and try to publish to the server. These will fail due to the bad
submission, and there will be a log entry on the AppEngine logs that
identifies the formId and the instanceId of the bad submission, and the
table with the bad fields.

Using that, you can figure out what the problems are with that submission
and which extra records need to be deleted.

On Wed, Apr 8, 2015 at 12:52 PM, poornimak20@gmail.com wrote:

On Wednesday, April 8, 2015 at 3:16:48 PM UTC-4, Mitch Sundt wrote:

Did this problem start happening immediately after the upgrade, or were
you collecting data for a period of time, and then it started happening?

If it happened immediately after the upgrade, please invite me (
mitche...@gmail.com ) as a developer so that I can confirm that the
AppEngine instance is still in good condition.

Assuming I determine that everything is OK, or if this started
happening after collecting data after the upgrade, you should then upgrade
to ODK Aggregate 1.4.5.

The error message may indicate that there is a data corruption issue in
one of your submissions. The ODK Aggregate 1.4.5 release provides better
logging of the details of that problem.

On Wed, Apr 8, 2015 at 10:45 AM, poorn...@gmail.com wrote:
Hi

I set up the ODK Aggregate with Google App Engine as the back end
server in 2013. The ODK aggregate was 1.3.4 version. I upgraded it upto
1.4.1. Now, ODK Aggregate cannot export the forms to CSV neither it is
publishing the forms as Google Spreadsheets or Fusion tables.

I was told to upgrade the ODK Aggregate to the latest version but since
I am not able to back up the data (one of the questionnaires has more than
15,000 entries), is it safe to upgrade the Aggregate without backing up the
data?

I tried using ODK Briefcase to pull the data but it failed after
fetching 1400 instances.

And when I click the publish button in the Aggregate, the following
error pops up;

Error: Problem persisting data or accessing data (Somehow DB entities
for publisher got into problem state)

Please help!

Thanks

Poornima

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

Hello Mitch

Thank You for your quick response.
We were collecting data for a long time now. The problem of exporting to
CSV or publishing the data started before upgrading the Aggregate.
I could upgrade 2 versions up (currently we are using version 1.4.1). We
don't want to take risk of upgrading to the latest versions without backing
up the data.
Is there any way to store the data before proceeding to the upgrading of
the Aggregate?

And also is there any way to find out the corrupted data in the
Submissions?

Thanks

Poornima

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

In the worst case, the data in the problematic submission(s) can be lost.
Data collected before or after the submission will not be impacted.

Once the bad submission is fixed, any publishers set up on that form should
resume publishing (and publish the fixed submission entry, and all
subsequent ones).

··· On Wed, Apr 8, 2015 at 1:19 PM, Mitch Sundt wrote:

And again, to do that, it is perhaps best to invite me as a developer
https://code.google.com/p/opendatakit/wiki/AppEngineTroubleshooting

If you have software and database expertise, I can provide more detail and
you can try it yourself. But I would generally not recommend doing so.

On Wed, Apr 8, 2015 at 1:17 PM, Mitch Sundt mitchellsundt@gmail.com wrote:

Our data-backup mechanism is to use ODK Briefcase. To do the remaining
upgrades more safely (logging was only added in 1.4.5 to address the
problem you are currently having, so we need to upgrade to 1.4.5 to fix it):

(1) uncheck "Accept Submissions" from the Forms Management page for all
of your forms. This prevents your users from submitting data into your
forms.
(2) Run ODK Briefcase. Pull every form down to your laptop. This should
succeed for every form except the one(s) that have this data corruption
issue. You may need to pull a form at a time into ODK Briefcase, but I
think you can just check and pull everything.

Because there is the problem with a submission in one or more of your
forms, you do have the potential to lose all data submitted for that formId
after that problematic row was submitted (data submitted into other forms
is not impacted, and will not be lose).

(3) Upgrade to ODK Aggregate 1.4.5 (I would upgrade to 1.4.2, log in,
upgrade to 1.4.3, log in, upgrade to 1.4.4, log in, and then upgrade to
1.4.5). Doing it one at a time, and performing the manual upgrade steps
described in the release notes will reduce the likelihood of problems with
the upgrade.
https://code.google.com/p/opendatakit/wiki/AggregateReleaseNotes


Once you have upgraded, and you have logged in, the publishers will
resume and try to publish to the server. These will fail due to the bad
submission, and there will be a log entry on the AppEngine logs that
identifies the formId and the instanceId of the bad submission, and the
table with the bad fields.

Using that, you can figure out what the problems are with that submission
and which extra records need to be deleted.

On Wed, Apr 8, 2015 at 12:52 PM, poornimak20@gmail.com wrote:

On Wednesday, April 8, 2015 at 3:16:48 PM UTC-4, Mitch Sundt wrote:

Did this problem start happening immediately after the upgrade, or
were you collecting data for a period of time, and then it started
happening?

If it happened immediately after the upgrade, please invite me (
mitche...@gmail.com ) as a developer so that I can confirm that the
AppEngine instance is still in good condition.

Assuming I determine that everything is OK, or if this started
happening after collecting data after the upgrade, you should then upgrade
to ODK Aggregate 1.4.5.

The error message may indicate that there is a data corruption issue
in one of your submissions. The ODK Aggregate 1.4.5 release provides better
logging of the details of that problem.

On Wed, Apr 8, 2015 at 10:45 AM, poorn...@gmail.com wrote:
Hi

I set up the ODK Aggregate with Google App Engine as the back end
server in 2013. The ODK aggregate was 1.3.4 version. I upgraded it upto
1.4.1. Now, ODK Aggregate cannot export the forms to CSV neither it is
publishing the forms as Google Spreadsheets or Fusion tables.

I was told to upgrade the ODK Aggregate to the latest version but
since I am not able to back up the data (one of the questionnaires has
more than 15,000 entries), is it safe to upgrade the Aggregate without
backing up the data?

I tried using ODK Briefcase to pull the data but it failed after
fetching 1400 instances.

And when I click the publish button in the Aggregate, the following
error pops up;

Error: Problem persisting data or accessing data (Somehow DB entities
for publisher got into problem state)

Please help!

Thanks

Poornima

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

Hello Mitch

Thank You for your quick response.
We were collecting data for a long time now. The problem of exporting to
CSV or publishing the data started before upgrading the Aggregate.
I could upgrade 2 versions up (currently we are using version 1.4.1). We
don't want to take risk of upgrading to the latest versions without backing
up the data.
Is there any way to store the data before proceeding to the upgrading of
the Aggregate?

And also is there any way to find out the corrupted data in the
Submissions?

Thanks

Poornima

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com