This appears to be a change in behavior by Google.
Testing this on opendatakit.appspot.com, I see this error in the log:
Unable to insert data into spreadsheet test_big_upload exception: 429
TOO_MANY_REQUESTS (GoogleSpreadsheet.java:652
https://console.cloud.google.com/debug/fromlog?appModule=background&appVersion=1&file=GoogleSpreadsheet.java&line=652&logInsertId=58e3cc700000b535c8dee181&logNanos=1491323995996138000&nestedLogIndex=13&project=opendatakit-hrd&authuser=1)
{ "code" : 429, "errors" : [ { "domain" : "global", "message" :
"Insufficient tokens for quota 'WriteGroup' and limit 'USER-100s' of
service 'sheets.googleapis.com' for consumer
'project_number:322300403941'.", "reason" : "rateLimitExceeded" } ],
"message" : "Insufficient tokens for quota 'WriteGroup' and limit
'USER-100s' of service 'sheets.googleapis.com' for consumer
'project_number:322300403941'.", "status" : "RESOURCE_EXHAUSTED" }
There is now a per-second and per-day quota for publishing into Google
Sheets. To see your quota status, navigate to the Google Cloud Platform
console, the API Manager, the Library page, select Google Sheets and then
select the quotas tab.
Or, click on this:
https://console.cloud.google.com/apis/api/sheets.googleapis.com/quotas
Then sign into your appropriate gmail account (or change to the appropriate
one like you can do in gmail), and select the project in which you created
your server's Oauth2 credentials.
*There is a 100 writes per second per user default quota limit. *
This limit is causing the publishing to enter a fail-and-retry sequence,
which cycles into an Active_pause state and/or an active_retry and
eventually a paused state (which adds an even longer delay between retries).
The quickest fix is to apply for an increased writes per second per user
quota limit (click the pencil to the right of that value). I am curious
to hear how the Google team will respond.
I will make a code change to reduce the size of the batch of rows that ODK
Aggregate submits to Google Sheets, but this default quota limit is very
very low, and there is little likelihood that we can stay below it,
especially if you are publishing several different forms, or forms that
have many fields.
Pushing Google to allow greater publisher rates will be part of the
long-term solution to this issue.
···
On Tue, Apr 4, 2017 at 1:43 AM, wrote:
Hi, I am running into a similar issue but with google spreadsheet option.
All of the publishers created before seem to be working fine, but the new
one created on 03/04/2017 initially has status of ACTIVE_PAUSE, then it's
changing to ACTIVE_RETRY for a while to become Paused at the end. This
process is being repeated whenever I click "Restart Publisher Paused".
Strangely enough spreadsheet created by aggregate server is blank (only
headers) despite of choosing "BOTH Upload Existing & Stream New Submission
Data" option.
I'm using the newest version of ODK Collect (v1.5.1) and ODK Aggregate
v1.4.13.
Thanks,
Rafal
On Thursday, 28 April 2016 00:18:32 UTC+1, sama...@verasolutions.org wrote:
Hi all, having some trouble publishing Aggregate v1.4.8 submissions to a
JSON server. I've outlined setup steps and error logs below - is anyone
running into a similar issue?
I've tested a few different forms, all run through ODK Validate without
errors. Submissions have been made using the latest version of Collect.
Under Form Mgmt > Forms List > Publish, publisher settings for each form
are as follows:
- Publish to: Z-ALPHA JSON Server
- Data to publish: BOTH Upload Existing & Stream New Submission Data
(have tested all three options unsuccessfully)
- Url to publish to: OpenFn inbox URL
- Authorization Token: [none]
- Include Media as: Links(URLs) to Media
Under Form Mgmt > Published Data, the STATUS of each submission is
either paused, active_retry, or active_paused; nothing is being published.
App engine log files
- The following message is repeatedly thrown when attempting to publish
data to the OpenFn inbox URL:
org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl:
uploadAllSubmissions: org.opendatakit.aggregate.exception.ODKExternalServiceException:
java.lang.IllegalStateException: Unrecognized Http request method
- More detail on the error found under 'View request logs':
0.1.0.2 - - [27/Apr/2016:19:36:03 +0530] "GET /gae/uploadSubmissionsTask?
fscUri=uuid%3A65d3eab1-f706-416f-9fb6-a8987eec9dea HTTP/1.1" 500 476
https://building-markets-1292.appspot.com/aggregateui/servicesadminservice
"AppEngine-Google; (+http://code.google.com/appengine)" "
1.background.building-markets-1292.appspot.com" ms=394 cpu_ms=1619
cpm_usd=5.3196999999999997e-8 instance=00c61b117c8d9c5acb01ac6e5e9eb03290fbfaeea1a536abeb
app_engine_release=1.9.36 trace_id=bdc066555e45b7ced14d218ca3a01339
I've attached some screenshots here as well. Any troubleshooting advice
is much appreciated.
Best,
Sam
--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com
