Please upgrade to 1.0.3.
The Upload Submissions queue manages the publishing of data to the external
services. You should not purge entries in this queue, as that will cause
publishing to not occur. Once I resolve issues relating to logins on
Tomcat, I will be investigating issue:
http://code.google.com/p/opendatakit/issues/detail?id=484 reporting failure
of streaming publishers to publish new submissions to external services.
You may be having this problem, but the publishing of already-submitted
data should not be failing.
v1.0.3 moves the upload-submissions activity to the foreground thread, and
fixes some issues with the watchdog being overly-active, which should bring
your background instance usage back underneath the 9-hour quota. If you
are submitting data more frequently than once every 6 seconds (or if you
are uploading multiple submissions from a given phone at one go), then
1.0.3 will be much better than 1.0.2 w.r.t. read quota and performance.
Improvements are particularly notable when using ODK Briefcase.
When you have your website open, if you view or refresh the "Submissions"
sub-tab (and that sub-tab auto-refreshes every 10 seconds), please keep in
mind that this executes a database read operation for at least 100
completed form submissions. Depending upon the structure of your form, the
number of reads to accomplish this can be quite high:
For a given form, on AppEngine, displaying a single completed form
submission costs a minimum of:
ReadsPerCompletedSubmission =
(1 read +
(1+'n') reads per multiple-choice question (i.e., ) +
3 x (number of image, audio, or video captures) +
RepeatGroupCost)
x
(1 + (number of image captures in main (non-repeat-group) portion of
form) )
Where 'n' is the average number of choices checked for that multiple-choice
question.
Repeat groups add the following to the total:
RepeatGroupCost =
(2 read +
(1+'n') reads per multiple-choice question within the repeat group +
3 x (number of image, audio, or video captures within the repeat group) )
x
'm'
Where 'm' is the average number of repeats per submission for this repeat
group.
So if you have a form with 6 multiple-choice questions, each averaging 3
responses, plus
one 1 repeat group, itself with a single multiple-choice question averaging
2 responses, and
the repeat group averages 4 completed groups per submission, and you also
have 2 audio
clips, 2 images, and 3 videos within the form (not the repeat group), then
the total costs are:
(1 + 6 x (1+3) + 3 x (2 + 2 + 3) + 4 x (2 + 1 x (1+2)) )
x
(1 + 2)
yielding:
198 reads / submission.
Anytime a submission is read, such as during publishing, this will be the
cost.
In particular, refreshing the 'Submissions' tab (which shows 100 records),
will then cost:
19,800 reads (.02 M reads)
This only counts the reads needed to access the completed submission.
There are other read operations needed to track what forms are available,
what rights the user has for accessing the system, and what has or has not
been published to the external services. From 1.0.2 onward, those are all
heavily cached, but also add into the total.
···
-------
Mitch
On Mon, Jan 2, 2012 at 10:35 AM, Asim Fayaz asimfayaz@gmail.com wrote:
Any idea what's going on? I can post logs etc if you tell me what you
want to see.
Happy New Year!
On Dec 31 2011, 4:02 pm, Asim Fayaz asimfa...@gmail.com wrote:
Aggregate v1.0.2
On Dec 30, 9:55 pm, Yaw Anokwa yano...@gmail.com wrote:
Asim,
What version of Aggregate you using?
Yaw
On Fri, Dec 30, 2011 at 08:49, Asim Fayaz asimfa...@gmail.com wrote:
I'm having trouble understanding the two.
I have four forms in aggregate. Submissions to all four are working
fine. I've set all four to publish both existing and new entries to
fusion tables. For some reason, it takes a long time for them to
publish. I just posted entries in 3 of them and all show fine in
Aggregate. The problem it seems is the upload-submission-queue. I
don't know what that is but there have been cases where purging the
queue has resulted in new entries showing up in the relevant fusion
tables.
I'm also having trouble understanding read operations. The value
currently stands at 3.75 million operations in the last 9 hours. I
just refreshed again and now it's 3.77 million operations. And now
it's at 3.79 million. I know for sure that nobody else is using the
application right now. What's causing these read operations? I've set
my daily quota to $5.00 and read operations have already used $2.62.
The remaining is spent on write operations and on backend instance
hours. Where are they used and how can I ensure that I'm not wasting
them? As of now, I'm at 0.89 million write operations and 17.24
backend instance hours.
Thanks in advance!
Asim
--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options:http://groups.google.com/group/opendatakit?hl=en
--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com