Aggregate Google API Credentials affecting data access

Hi

We are running Aggr.v1.4.11 on a paid App Engine account. Yesterday I
tried to establish a data flow to a Google (Spread)sheet and am now
experiencing inconsistent responses from Aggregate with the following
message:

Error: Problem persisting data or accessing data (Somehow DB entities for
publisher got into problem state)My first question is: Can changing the
Google API credentials under Site Admin Preferences affect Aggregate's
performance?
* Currently the page refers to the API key as recommended
and the OAuth2 creds as *necessary for publishing to Drive

In my Google Cloud Platform service account credentials GUIs/dialogues I
did not always find the references/experience referred to in the ODK
knowledge bases. It is quite possible that I got confused between .p12, API
keys, Client IDs and Service Account descriptors although I was careful to
retain what had been present. The game has changed since
http://opendatakit.org/use/aggregate/oauth2-service-account/. I have
attached a couple of scrshts of the new dialogue but there is no clarity as
to which APIs we need to create what credentials for or what data flows we
have to cater for.

I have looked at my error logs in reference to


and it reminded me of learning to knit. Inconsistent in Aggregate at
present is that some commands work and some don't, some of the time. An
export successfully downloads and then the next time I select an export
just takes me back to the submissions page. Instances can be submitted
sporadically.

My second question is: Is there a clear and simple way forward?

Regards,
Chris

image

You need an API Key and a Service account key

This documentation is still accurate in what Google APIs you need to enable
and what credentials you need to obtain. The screen navigation and
interactions have, however, changed.

https://opendatakit.org/use/aggregate/oauth2-service-account/

Yes, the screen shots and workflow are likely different. After all, we last
updated them 5 months ago (April 2016).

However, the names for what need to be created have not changed -- service
account (step 11), and API key (step 17). And the names for what Google
APIs you need to enable have also not changed (step 7).

··· ================ The web UI refreshes its screen every 3-6 seconds. If you are on the Submissions page and it refreshes, if you have very large forms, you might be able to navigate off to a different screen only to have the fetch of that large dataset returned to your browser, triggering a redraw of the Submissions page. (throwing you back to that screen).

This generally indicates that you should increase the size of your server.

https://opendatakit.org/use/aggregate/#Changing_Server_Size

Also, please be aware that as your data set grows, it will eventually be
unable to be Exported (increasing the server size only works up to a
point). Instead, you will need to use ODK Briefcase to pull the data set
down to your local computer and use the export-to-csv functionality within
ODK Briefcase to generate CSVs (slightly different format) locally.

On Sat, Sep 24, 2016 at 1:33 AM, Chris Berens chris@mapland.co.za wrote:

Hi

We are running Aggr.v1.4.11 on a paid App Engine account. Yesterday I
tried to establish a data flow to a Google (Spread)sheet and am now
experiencing inconsistent responses from Aggregate with the following
message:

Error: Problem persisting data or accessing data (Somehow DB entities for
publisher got into problem state)My first question is: Can changing the
Google API credentials under Site Admin Preferences affect Aggregate's
performance?
* Currently the page refers to the API key as recommended
and the OAuth2 creds as *necessary for publishing to Drive

In my Google Cloud Platform service account credentials GUIs/dialogues I
did not always find the references/experience referred to in the ODK
knowledge bases. It is quite possible that I got confused between .p12, API
keys, Client IDs and Service Account descriptors although I was careful to
retain what had been present. The game has changed since
http://opendatakit.org/use/aggregate/oauth2-service-account/. I have
attached a couple of scrshts of the new dialogue but there is no clarity as
to which APIs we need to create what credentials for or what data flows we
have to cater for.

I have looked at my error logs in reference to https://github.com/
opendatakit/opendatakit/wiki/Aggregate-AppEngine-
Troubleshooting#repairing-a-filled-in-form-submission and it reminded me
of learning to knit. Inconsistent in Aggregate at present is that some
commands work and some don't, some of the time. An export successfully
downloads and then the next time I select an export just takes me back to
the submissions page. Instances can be submitted sporadically.

My second question is: Is there a clear and simple way forward?

Regards,
Chris

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Thanks Mitch, for your time and thought.

I will re-run through these steps next week.

Regards,
Chris

··· On 30 September 2016 at 18:18, Mitch Sundt wrote:

You need an API Key and a Service account key

This documentation is still accurate in what Google APIs you need to
enable and what credentials you need to obtain. The screen navigation and
interactions have, however, changed.

https://opendatakit.org/use/aggregate/oauth2-service-account/

Yes, the screen shots and workflow are likely different. After all, we
last updated them 5 months ago (April 2016).

However, the names for what need to be created have not changed -- service
account (step 11), and API key (step 17). And the names for what Google
APIs you need to enable have also not changed (step 7).

================
The web UI refreshes its screen every 3-6 seconds. If you are on the
Submissions page and it refreshes, if you have very large forms, you might
be able to navigate off to a different screen only to have the fetch of
that large dataset returned to your browser, triggering a redraw of the
Submissions page. (throwing you back to that screen).

This generally indicates that you should increase the size of your server.

https://opendatakit.org/use/aggregate/#Changing_Server_Size

Also, please be aware that as your data set grows, it will eventually be
unable to be Exported (increasing the server size only works up to a
point). Instead, you will need to use ODK Briefcase to pull the data set
down to your local computer and use the export-to-csv functionality within
ODK Briefcase to generate CSVs (slightly different format) locally.

On Sat, Sep 24, 2016 at 1:33 AM, Chris Berens chris@mapland.co.za wrote:

Hi

We are running Aggr.v1.4.11 on a paid App Engine account. Yesterday I
tried to establish a data flow to a Google (Spread)sheet and am now
experiencing inconsistent responses from Aggregate with the following
message:

Error: Problem persisting data or accessing data (Somehow DB entities
for publisher got into problem state)My first question is: Can
changing the Google API credentials under Site Admin Preferences affect
Aggregate's performance?
* Currently the page refers to the API key as *
recommended* and the OAuth2 creds as *necessary for publishing to Drive

In my Google Cloud Platform service account credentials GUIs/dialogues I
did not always find the references/experience referred to in the ODK
knowledge bases. It is quite possible that I got confused between .p12, API
keys, Client IDs and Service Account descriptors although I was careful to
retain what had been present. The game has changed since
http://opendatakit.org/use/aggregate/oauth2-service-account/. I have
attached a couple of scrshts of the new dialogue but there is no clarity as
to which APIs we need to create what credentials for or what data flows we
have to cater for.

I have looked at my error logs in reference to
https://github.com/opendatakit/opendatakit/wiki/Aggregate-
AppEngine-Troubleshooting#repairing-a-filled-in-form-submission and it
reminded me of learning to knit. Inconsistent in Aggregate at present is
that some commands work and some don't, some of the time. An export
successfully downloads and then the next time I select an export just takes
me back to the submissions page. Instances can be submitted sporadically.

My second question is: Is there a clear and simple way forward?

Regards,
Chris

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/opendatakit/-I61BJAPoZA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Chris Berens
Pr GISc (SA) PLATO#1308
+27 (0)82 567 9322
www.mapland.co.za