Help! ODK Collect stopped sending data after 1000 instances, how can i rebuild?

hey ryan,

i'm cc'ing the mailing list so the other developers can chip in.

how do you know you are only getting 1000 instances? are you exporting
to csv/kml? what happens when you try to view more data in the web
interface?

yaw

··· On Wed, Jun 22, 2011 at 14:15, Ryan E wrote: > Hi, > > I really love ODK and recently we used it on a large project > collecting info about streetlights. Everything seemed to be working in > the field, but the tech failed to tell me about serious slowness > issues as he finished the project. Once we went in and checked > Aggregate (should have done this everyday) we were horrified to see > only 1000 instances had actually been updated. It seems like the > SQLlite database will only hold that many rows? > > It appears that at this point I could either manually upload 900+ > instances to aggregate or try to recreate a SQL Lite database with the > instances that didnt make it in. Obviously I'd like to do the latter > but I don't know the particulars of generating the metadata. Can you > help? > > I have 1000 instances (xml files) that I basically want to import into > Aggregate, hwo can I do this efficiently? > > Thank you, > Ryan >

Thank you for the quick reply.

Our technician was able to collect ~2300 instances (.xml files inside
a directory) on his android. He claims to have sent the data to
Aggregate without any errors, however, when we logged in they were not
all there.

I just realized that the 1000 row limit i saw was a result of the
SQLite database browser I was using, the metadata/data table does
indeed has all 2003 records (I had go to another page of results).
Sorry bout that.

At this point I have about 1000 instances I need imported into
aggregate. They show up in Collect under the Data tab, but I can't do
anything with them (Collect seems to think they were already
uploaded). Can I change the 'files' table status column to say
something besides submitted so they will appear in the "Send Finished
Data" screen?

Ryan

On his device I opened the metadata/data file and saw only 1000 rows
in the files database

··· On Wed, Jun 22, 2011 at 11:33 AM, Yaw Anokwa wrote: > hey ryan, > > i'm cc'ing the mailing list so the other developers can chip in. > > how do you know you are only getting 1000 instances? are you exporting > to csv/kml? what happens when you try to view more data in the web > interface? > > yaw > > On Wed, Jun 22, 2011 at 14:15, Ryan E wrote: >> Hi, >> >> I really love ODK and recently we used it on a large project >> collecting info about streetlights. Everything seemed to be working in >> the field, but the tech failed to tell me about serious slowness >> issues as he finished the project. Once we went in and checked >> Aggregate (should have done this everyday) we were horrified to see >> only 1000 instances had actually been updated. It seems like the >> SQLlite database will only hold that many rows? >> >> It appears that at this point I could either manually upload 900+ >> instances to aggregate or try to recreate a SQL Lite database with the >> instances that didnt make it in. Obviously I'd like to do the latter >> but I don't know the particulars of generating the metadata. Can you >> help? >> >> I have 1000 instances (xml files) that I basically want to import into >> Aggregate, hwo can I do this efficiently? >> >> Thank you, >> Ryan >> >

I'm assuming you're using 1.1.5? If you update the database status column
to say "complete" then they should all appear in the upload list again.

··· On Wed, Jun 22, 2011 at 11:43 AM, Ryan E wrote:

Thank you for the quick reply.

Our technician was able to collect ~2300 instances (.xml files inside
a directory) on his android. He claims to have sent the data to
Aggregate without any errors, however, when we logged in they were not
all there.

I just realized that the 1000 row limit i saw was a result of the
SQLite database browser I was using, the metadata/data table does
indeed has all 2003 records (I had go to another page of results).
Sorry bout that.

At this point I have about 1000 instances I need imported into
aggregate. They show up in Collect under the Data tab, but I can't do
anything with them (Collect seems to think they were already
uploaded). Can I change the 'files' table status column to say
something besides submitted so they will appear in the "Send Finished
Data" screen?

Ryan

On his device I opened the metadata/data file and saw only 1000 rows
in the files database

On Wed, Jun 22, 2011 at 11:33 AM, Yaw Anokwa yanokwa@gmail.com wrote:

hey ryan,

i'm cc'ing the mailing list so the other developers can chip in.

how do you know you are only getting 1000 instances? are you exporting
to csv/kml? what happens when you try to view more data in the web
interface?

yaw

On Wed, Jun 22, 2011 at 14:15, Ryan E ryan253@gmail.com wrote:

Hi,

I really love ODK and recently we used it on a large project
collecting info about streetlights. Everything seemed to be working in
the field, but the tech failed to tell me about serious slowness
issues as he finished the project. Once we went in and checked
Aggregate (should have done this everyday) we were horrified to see
only 1000 instances had actually been updated. It seems like the
SQLlite database will only hold that many rows?

It appears that at this point I could either manually upload 900+
instances to aggregate or try to recreate a SQL Lite database with the
instances that didnt make it in. Obviously I'd like to do the latter
but I don't know the particulars of generating the metadata. Can you
help?

I have 1000 instances (xml files) that I basically want to import into
Aggregate, hwo can I do this efficiently?

Thank you,
Ryan

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

agreed. also, before uploading, i'd recommend you setup a google
fusion tables forwarding for the incoming data. it'll make your life a
little easier.

··· On Wed, Jun 22, 2011 at 14:55, Carl Hartung wrote: > I'm assuming you're using 1.1.5? If you update the database status column > to say "complete" then they should all appear in the upload list again. > > > On Wed, Jun 22, 2011 at 11:43 AM, Ryan E wrote: >> >> Thank you for the quick reply. >> >> Our technician was able to collect ~2300 instances (.xml files inside >> a directory) on his android. He claims to have sent the data to >> Aggregate without any errors, however, when we logged in they were not >> all there. >> >> I just realized that the 1000 row limit i saw was a result of the >> SQLite database browser I was using, the metadata/data table does >> indeed has all 2003 records (I had go to another page of results). >> Sorry bout that. >> >> At this point I have about 1000 instances I need imported into >> aggregate. They show up in Collect under the Data tab, but I can't do >> anything with them (Collect seems to think they were already >> uploaded). Can I change the 'files' table status column to say >> something besides submitted so they will appear in the "Send Finished >> Data" screen? >> >> Ryan >> >> >> On his device I opened the metadata/data file and saw only 1000 rows >> in the files database >> >> On Wed, Jun 22, 2011 at 11:33 AM, Yaw Anokwa wrote: >> > hey ryan, >> > >> > i'm cc'ing the mailing list so the other developers can chip in. >> > >> > how do you know you are only getting 1000 instances? are you exporting >> > to csv/kml? what happens when you try to view more data in the web >> > interface? >> > >> > yaw >> > >> > On Wed, Jun 22, 2011 at 14:15, Ryan E wrote: >> >> Hi, >> >> >> >> I really love ODK and recently we used it on a large project >> >> collecting info about streetlights. Everything seemed to be working in >> >> the field, but the tech failed to tell me about serious slowness >> >> issues as he finished the project. Once we went in and checked >> >> Aggregate (should have done this everyday) we were horrified to see >> >> only 1000 instances had actually been updated. It seems like the >> >> SQLlite database will only hold that many rows? >> >> >> >> It appears that at this point I could either manually upload 900+ >> >> instances to aggregate or try to recreate a SQL Lite database with the >> >> instances that didnt make it in. Obviously I'd like to do the latter >> >> but I don't know the particulars of generating the metadata. Can you >> >> help? >> >> >> >> I have 1000 instances (xml files) that I basically want to import into >> >> Aggregate, hwo can I do this efficiently? >> >> >> >> Thank you, >> >> Ryan >> >> >> > >> >> -- >> Post: opendatakit@googlegroups.com >> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en > >

Aggregate 0.9.x limits the number of rows it will show to about 1000 records
so that it won't time-out on the server without producing any results.
Perhaps you have the data up on the server but are running into that limit?

You can use Briefcase to download the full set of data from your server.

Mitch

··· On Wed, Jun 22, 2011 at 11:57 AM, Yaw Anokwa wrote:

agreed. also, before uploading, i'd recommend you setup a google
fusion tables forwarding for the incoming data. it'll make your life a
little easier.

On Wed, Jun 22, 2011 at 14:55, Carl Hartung carlhartung@gmail.com wrote:

I'm assuming you're using 1.1.5? If you update the database status
column
to say "complete" then they should all appear in the upload list again.

On Wed, Jun 22, 2011 at 11:43 AM, Ryan E ryan253@gmail.com wrote:

Thank you for the quick reply.

Our technician was able to collect ~2300 instances (.xml files inside
a directory) on his android. He claims to have sent the data to
Aggregate without any errors, however, when we logged in they were not
all there.

I just realized that the 1000 row limit i saw was a result of the
SQLite database browser I was using, the metadata/data table does
indeed has all 2003 records (I had go to another page of results).
Sorry bout that.

At this point I have about 1000 instances I need imported into
aggregate. They show up in Collect under the Data tab, but I can't do
anything with them (Collect seems to think they were already
uploaded). Can I change the 'files' table status column to say
something besides submitted so they will appear in the "Send Finished
Data" screen?

Ryan

On his device I opened the metadata/data file and saw only 1000 rows
in the files database

On Wed, Jun 22, 2011 at 11:33 AM, Yaw Anokwa yanokwa@gmail.com wrote:

hey ryan,

i'm cc'ing the mailing list so the other developers can chip in.

how do you know you are only getting 1000 instances? are you exporting
to csv/kml? what happens when you try to view more data in the web
interface?

yaw

On Wed, Jun 22, 2011 at 14:15, Ryan E ryan253@gmail.com wrote:

Hi,

I really love ODK and recently we used it on a large project
collecting info about streetlights. Everything seemed to be working
in
the field, but the tech failed to tell me about serious slowness
issues as he finished the project. Once we went in and checked
Aggregate (should have done this everyday) we were horrified to see
only 1000 instances had actually been updated. It seems like the
SQLlite database will only hold that many rows?

It appears that at this point I could either manually upload 900+
instances to aggregate or try to recreate a SQL Lite database with
the
instances that didnt make it in. Obviously I'd like to do the latter
but I don't know the particulars of generating the metadata. Can you
help?

I have 1000 instances (xml files) that I basically want to import
into
Aggregate, hwo can I do this efficiently?

Thank you,
Ryan

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitchellsundt@gmail.com