Subsequent database updates

Hello,

I am wondering about how (and if) people have updated their databases built
up with ODK data. Is there an easy way to later capture data and link it
back to the original records, in some cases replacing them, or even
removing entries from the map (when a fruitstall is no longer there, for
example)?

How does this play out in the database?

So far it seems to me like ODK is a great tool for gathering data, but not
as well suited to updating existing datasets, although ODK Clinic is able
to access and modify records as you are working.

Thank you for any help!

I've just realised I didn't explain the problem properly.

We are identifying our stalls or shops by their name, location, address,
and business registration number. The problem comes from the fact that
addresses are very uncertain here, with streets often being known by
several names and the numbering being haphazard. The name and business
registration cannot be used as keys because if the business changes then so
do those two attributes. Rather, those are the attributes we need to know
how to change.

Finally, geolocation isn't accurate enough, so no each subsequent visit
we'll get a slightly different reading.

So the problem, then, what absolute attribute could we use to identify our
locations, which we could use to find them in the database to make changes?

··· On Tuesday, 29 May 2012 13:40:43 UTC+8, Christian Mondorf wrote: > > Hello, > > I am wondering about how (and if) people have updated their databases > built up with ODK data. Is there an easy way to later capture data and link > it back to the original records, in some cases replacing them, or even > removing entries from the map (when a fruitstall is no longer there, for > example)? > > How does this play out in the database? > > So far it seems to me like ODK is a great tool for gathering data, but not > as well suited to updating existing datasets, although ODK Clinic is able > to access and modify records as you are working. > > Thank you for any help! >

It is best to think of ODK Aggregate as a "repository of record" for your
data collection efforts. I.e., it is not intended for dynamic updates to
existing records, but, rather, to hold all data as collected.

There is a limited ability to delete individual collected records and to
delete records older than a certain date or all records that have already
been published to an external service (e.g., Google Spreadsheets or Fusion
Tables).

For your problem, it seems like the business registration number ties the
records for a given business to each other. What you need is a way to
reconcile and consolidate the set of records sharing the same registration
number into a single record. This "data cleansing" operation is not
currently supported by ODK Aggregate. I haven't thought much about the
details, but it would probably be costly to implement this on Google
AppEngine; longer term, once Google's MySQL cloud service moves off of beta
and we know what the costs are, we will likely move onto it, giving us much
greater flexibility to support data cleansing operations on all deployments.

If you are maintaining your own MySQL or PostgreSQL database, there are
various 3rd party data visualization, statistical, and Business
Intelligence tools, that can access MySQL or PostgreSQL databases directly
(or even the administration consoles of pgAdmin or MySQL Workbench), and
you can use those to define your own reports and screens to aid you in
consolidating these records. With respect to ODK Aggregate, as long as the
top-level record in the ..._CORE table is deleted, the record disappears
from view; this will create orphan data records in the other tables used
for that form (e.g., unreachable images, for example). Depending upon how
rapidly your data is updated, and how large any media files might be that
are attached to it, it may be acceptable to run for a year or two without
cleaning up these orphan data records. By that time, there will likely be
a new version of ODK Aggregate that will require a full dump of data into
Briefcase and a reload of the datastore, at which time those orphan data
records will be automatically purged.

Mitch

··· On Tue, May 29, 2012 at 1:15 AM, Christian Mondorf wrote:

I've just realised I didn't explain the problem properly.

We are identifying our stalls or shops by their name, location, address,
and business registration number. The problem comes from the fact that
addresses are very uncertain here, with streets often being known by
several names and the numbering being haphazard. The name and business
registration cannot be used as keys because if the business changes then so
do those two attributes. Rather, those are the attributes we need to know
how to change.

Finally, geolocation isn't accurate enough, so no each subsequent visit
we'll get a slightly different reading.

So the problem, then, what absolute attribute could we use to identify our
locations, which we could use to find them in the database to make changes?

On Tuesday, 29 May 2012 13:40:43 UTC+8, Christian Mondorf wrote:

Hello,

I am wondering about how (and if) people have updated their databases
built up with ODK data. Is there an easy way to later capture data and link
it back to the original records, in some cases replacing them, or even
removing entries from the map (when a fruitstall is no longer there, for
example)?

How does this play out in the database?

So far it seems to me like ODK is a great tool for gathering data, but
not as well suited to updating existing datasets, although ODK Clinic is
able to access and modify records as you are working.

Thank you for any help!

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

I would assume that the business registration number is unique to each
enterprise. It's outside the scope of ODK, but in your database, you could
consolidate your records based on that field (or some other uuid).

··· On Tuesday, May 29, 2012, Christian Mondorf wrote:

I've just realised I didn't explain the problem properly.

We are identifying our stalls or shops by their name, location, address,
and business registration number. The problem comes from the fact that
addresses are very uncertain here, with streets often being known by
several names and the numbering being haphazard. The name and business
registration cannot be used as keys because if the business changes then so
do those two attributes. Rather, those are the attributes we need to know
how to change.

Finally, geolocation isn't accurate enough, so no each subsequent visit
we'll get a slightly different reading.

So the problem, then, what absolute attribute could we use to identify our
locations, which we could use to find them in the database to make changes?

On Tuesday, 29 May 2012 13:40:43 UTC+8, Christian Mondorf wrote:

Hello,

I am wondering about how (and if) people have updated their databases
built up with ODK data. Is there an easy way to later capture data and link
it back to the original records, in some cases replacing them, or even
removing entries from the map (when a fruitstall is no longer there, for
example)?

How does this play out in the database?

So far it seems to me like ODK is a great tool for gathering data, but
not as well suited to updating existing datasets, although ODK Clinic is
able to access and modify records as you are working.

Thank you for any help!

--
Post: opendatakit@googlegroups.com <javascript:_e({}, 'cvml',
'opendatakit@googlegroups.com');>
Unsubscribe: opendatakit+unsubscribe@googlegroups.com <javascript:_e({},
'cvml', 'opendatakit%2Bunsubscribe@googlegroups.com');>
Options: http://groups.google.com/group/opendatakit?hl=en

Thanks to both of you, you've both made some very interesting points.

Unfortunately the business registration number doesn't really solve our
problem since a change in the business would result in the new business
occupying the address having a new business number.

I know the obvious solution is to use the address as a unique identifier
but this project was started as a way to circumvent the problems with
working with addresses in Malaysia outside of the capital..

··· On Wednesday, 30 May 2012 10:36:08 UTC+8, ニコノコ wrote: > > I would assume that the business registration number is unique to each > enterprise. It's outside the scope of ODK, but in your database, you could > consolidate your records based on that field (or some other uuid). > > On Tuesday, May 29, 2012, Christian Mondorf wrote: > >> I've just realised I didn't explain the problem properly. >> >> We are identifying our stalls or shops by their name, location, address, >> and business registration number. The problem comes from the fact that >> addresses are very uncertain here, with streets often being known by >> several names and the numbering being haphazard. The name and business >> registration cannot be used as keys because if the business changes then so >> do those two attributes. Rather, those are the attributes we need to know >> how to change. >> >> Finally, geolocation isn't accurate enough, so no each subsequent visit >> we'll get a slightly different reading. >> >> So the problem, then, what absolute attribute could we use to identify >> our locations, which we could use to find them in the database to make >> changes? >> >> On Tuesday, 29 May 2012 13:40:43 UTC+8, Christian Mondorf wrote: >>> >>> Hello, >>> >>> I am wondering about how (and if) people have updated their databases >>> built up with ODK data. Is there an easy way to later capture data and link >>> it back to the original records, in some cases replacing them, or even >>> removing entries from the map (when a fruitstall is no longer there, for >>> example)? >>> >>> How does this play out in the database? >>> >>> So far it seems to me like ODK is a great tool for gathering data, but >>> not as well suited to updating existing datasets, although ODK Clinic is >>> able to access and modify records as you are working. >>> >>> Thank you for any help! >>> >> -- >> Post: opendatakit@googlegroups.com >> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> >

What are you tracking? Or, what's your objective here?

Personally, I don't consider an address as ideal unique identifier (i.e.
several corporations could share the same physical address but are
separate juridical entities.) For tracking repeated surveys of same
respondents, the ODK case study I've read assigns a unique ID for each
respondent (read through a QRcode).

··· On Wednesday, May 30, 2012, Christian Mondorf wrote:

Thanks to both of you, you've both made some very interesting points.

Unfortunately the business registration number doesn't really solve our
problem since a change in the business would result in the new business
occupying the address having a new business number.

I know the obvious solution is to use the address as a unique identifier
but this project was started as a way to circumvent the problems with
working with addresses in Malaysia outside of the capital..

On Wednesday, 30 May 2012 10:36:08 UTC+8, ニコノコ wrote:

I would assume that the business registration number is unique to each
enterprise. It's outside the scope of ODK, but in your database, you could
consolidate your records based on that field (or some other uuid).

On Tuesday, May 29, 2012, Christian Mondorf wrote:

I've just realised I didn't explain the problem properly.

We are identifying our stalls or shops by their name, location, address,
and business registration number. The problem comes from the fact that
addresses are very uncertain here, with streets often being known by
several names and the numbering being haphazard. The name and business
registration cannot be used as keys because if the business changes then so
do those two attributes. Rather, those are the attributes we need to know
how to change.

Finally, geolocation isn't accurate enough, so no each subsequent visit
we'll get a slightly different reading.

So the problem, then, what absolute attribute could we use to identify
our locations, which we could use to find them in the database to make
changes?

On Tuesday, 29 May 2012 13:40:43 UTC+8, Christian Mondorf wrote:

Hello,

I am wondering about how (and if) people have updated their databases
built up with ODK data. Is there an easy way to later capture data and link
it back to the original records, in some cases replacing them, or even
removing entries from the map (when a fruitstall is no longer there, for
example)?

How does this play out in the database?

So far it seems to me like ODK is a great tool for gathering data, but
not as well suited to updating existing datasets, although ODK Clinic is
able to access and modify records as you are working.

Thank you for any help!

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@**googlegroups.com
Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com <javascript:_e({}, 'cvml',
'opendatakit@googlegroups.com');>
Unsubscribe: opendatakit+unsubscribe@googlegroups.com <javascript:_e({},
'cvml', 'opendatakit%2Bunsubscribe@googlegroups.com');>
Options: http://groups.google.com/group/opendatakit?hl=en

We're trying to have a continually updated map of traditional trade
outlets, so mostly small "mom and pop" shops, and have them geocoded
because addresses here are a bit of an uncertain science sometimes.

We've thought of giving people barcodes which could be scanned on following
visits, but this runs into two problems. The first is that the person we
speak to on the first visit may be gone or have lost the code when we
return. We've considered stickers but worry they may not be as permanent as
one hopes.

The second problem, of course is that such shops close down or are
replaced, and handing over the barcode to the new tenants is probably the
last thing on their mind.

··· On Wednesday, 30 May 2012 18:28:34 UTC+8, ニコノコ wrote: > > What are you tracking? Or, what's your objective here? > > Personally, I don't consider an address as ideal unique identifier (i.e. > several corporations could share the same physical address but are > separate juridical entities.) For tracking repeated surveys of same > respondents, the ODK case study I've read assigns a unique ID for each > respondent (read through a QRcode). > > > > > On Wednesday, May 30, 2012, Christian Mondorf wrote: > >> Thanks to both of you, you've both made some very interesting points. >> >> Unfortunately the business registration number doesn't really solve our >> problem since a change in the business would result in the new business >> occupying the address having a new business number. >> >> I know the obvious solution is to use the address as a unique identifier >> but this project was started as a way to circumvent the problems with >> working with addresses in Malaysia outside of the capital.. >> >> On Wednesday, 30 May 2012 10:36:08 UTC+8, ニコノコ wrote: >>> >>> I would assume that the business registration number is unique to each >>> enterprise. It's outside the scope of ODK, but in your database, you could >>> consolidate your records based on that field (or some other uuid). >>> >>> On Tuesday, May 29, 2012, Christian Mondorf wrote: >>> >>>> I've just realised I didn't explain the problem properly. >>>> >>>> We are identifying our stalls or shops by their name, location, >>>> address, and business registration number. The problem comes from the fact >>>> that addresses are very uncertain here, with streets often being known by >>>> several names and the numbering being haphazard. The name and business >>>> registration cannot be used as keys because if the business changes then so >>>> do those two attributes. Rather, those are the attributes we need to know >>>> how to change. >>>> >>>> Finally, geolocation isn't accurate enough, so no each subsequent visit >>>> we'll get a slightly different reading. >>>> >>>> So the problem, then, what absolute attribute could we use to identify >>>> our locations, which we could use to find them in the database to make >>>> changes? >>>> >>>> On Tuesday, 29 May 2012 13:40:43 UTC+8, Christian Mondorf wrote: >>>>> >>>>> Hello, >>>>> >>>>> I am wondering about how (and if) people have updated their databases >>>>> built up with ODK data. Is there an easy way to later capture data and link >>>>> it back to the original records, in some cases replacing them, or even >>>>> removing entries from the map (when a fruitstall is no longer there, for >>>>> example)? >>>>> >>>>> How does this play out in the database? >>>>> >>>>> So far it seems to me like ODK is a great tool for gathering data, but >>>>> not as well suited to updating existing datasets, although ODK Clinic is >>>>> able to access and modify records as you are working. >>>>> >>>>> Thank you for any help! >>>>> >>>> -- >>>> Post: opendatakit@googlegroups.com >>>> Unsubscribe: opendatakit+unsubscribe@**googlegroups.com >>>> Options: http://groups.google.com/**group/opendatakit?hl=en >>>> >>> -- >> Post: opendatakit@googlegroups.com >> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> >

Seems like you're delving more and more towards a full blown GIS. Maybe
also look at uDIG or qGIS as some open source GIS/data collection
applications.

Possibly geo-code the businesses with a "geo-id". Using your GPS lat-long
to create a unique Id for each location that could always link back to the
location and northern actual business in that location.

Could a "point" be located on the map by using structure footprints prior
to field collection or are there quality aerial photos available for this
location? If so field personal could navigate to each point, which would
have a unique id, and populate it with the business data.

Several US 911 emergency address management organizations use a point to
represent structures and continually maintain new residents and businesses
using the above scenario.

··· On May 30, 2012 9:21 PM, "Christian Mondorf" wrote:

We're trying to have a continually updated map of traditional trade
outlets, so mostly small "mom and pop" shops, and have them geocoded
because addresses here are a bit of an uncertain science sometimes.

We've thought of giving people barcodes which could be scanned on
following visits, but this runs into two problems. The first is that the
person we speak to on the first visit may be gone or have lost the code
when we return. We've considered stickers but worry they may not be as
permanent as one hopes.

The second problem, of course is that such shops close down or are
replaced, and handing over the barcode to the new tenants is probably the
last thing on their mind.

On Wednesday, 30 May 2012 18:28:34 UTC+8, ニコノコ wrote:

What are you tracking? Or, what's your objective here?

Personally, I don't consider an address as ideal unique identifier (i.e.
several corporations could share the same physical address but are
separate juridical entities.) For tracking repeated surveys of same
respondents, the ODK case study I've read assigns a unique ID for each
respondent (read through a QRcode).

On Wednesday, May 30, 2012, Christian Mondorf wrote:

Thanks to both of you, you've both made some very interesting points.

Unfortunately the business registration number doesn't really solve our
problem since a change in the business would result in the new business
occupying the address having a new business number.

I know the obvious solution is to use the address as a unique identifier
but this project was started as a way to circumvent the problems with
working with addresses in Malaysia outside of the capital..

On Wednesday, 30 May 2012 10:36:08 UTC+8, ニコノコ wrote:

I would assume that the business registration number is unique to each
enterprise. It's outside the scope of ODK, but in your database, you could
consolidate your records based on that field (or some other uuid).

On Tuesday, May 29, 2012, Christian Mondorf wrote:

I've just realised I didn't explain the problem properly.

We are identifying our stalls or shops by their name, location,
address, and business registration number. The problem comes from the fact
that addresses are very uncertain here, with streets often being known by
several names and the numbering being haphazard. The name and business
registration cannot be used as keys because if the business changes then so
do those two attributes. Rather, those are the attributes we need to know
how to change.

Finally, geolocation isn't accurate enough, so no each subsequent
visit we'll get a slightly different reading.

So the problem, then, what absolute attribute could we use to identify
our locations, which we could use to find them in the database to make
changes?

On Tuesday, 29 May 2012 13:40:43 UTC+8, Christian Mondorf wrote:

Hello,

I am wondering about how (and if) people have updated their databases
built up with ODK data. Is there an easy way to later capture data and link
it back to the original records, in some cases replacing them, or even
removing entries from the map (when a fruitstall is no longer there, for
example)?

How does this play out in the database?

So far it seems to me like ODK is a great tool for gathering data,
but not as well suited to updating existing datasets, although ODK Clinic
is able to access and modify records as you are working.

Thank you for any help!

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@**googlegroups.com
Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

I suppose, just like in most cities, you'd have shops that are required to
have business licenses to operate. That'd be their UUID. It's extremely
unlikely for these not to be unique.

Your idea of passing barcodes to the next tenant is strange, unless your
tracking what's found in a particular geopoint rather than what business is
at that particular geopoint.

If it's impossible to assign them a unique identifier assigned to each
respondent, I'll play around with the geopoint location to compute the
distance and then see if they share the business name within a certain
radius to check if they've been surveyed before. Again, this is not within
the scope of ODK, which is a "data collection tool".

@Tapan
Could you elaborate a bit on "local ground". I haven't heard of this
before.

··· On Thursday, May 31, 2012, Christian Mondorf wrote:

We're trying to have a continually updated map of traditional trade
outlets, so mostly small "mom and pop" shops, and have them geocoded
because addresses here are a bit of an uncertain science sometimes.

We've thought of giving people barcodes which could be scanned on
following visits, but this runs into two problems. The first is that the
person we speak to on the first visit may be gone or have lost the code
when we return. We've considered stickers but worry they may not be as
permanent as one hopes.

The second problem, of course is that such shops close down or are
replaced, and handing over the barcode to the new tenants is probably the
last thing on their mind.

On Wednesday, 30 May 2012 18:28:34 UTC+8, ニコノコ wrote:

What are you tracking? Or, what's your objective here?

Personally, I don't consider an address as ideal unique identifier (i.e.
several corporations could share the same physical address but are
separate juridical entities.) For tracking repeated surveys of same
respondents, the ODK case study I've read assigns a unique ID for each
respondent (read through a QRcode).

On Wednesday, May 30, 2012, Christian Mondorf wrote:

Thanks to both of you, you've both made some very interesting points.

Unfortunately the business registration number doesn't really solve our
problem since a change in the business would result in the new business
occupying the address having a new business number.

I know the obvious solution is to use the address as a unique identifier
but this project was started as a way to circumvent the problems with
working with addresses in Malaysia outside of the capital..

On Wednesday, 30 May 2012 10:36:08 UTC+8, ニコノコ wrote:

I would assume that the business registration number is unique to each
enterprise. It's outside the scope of ODK, but in your database, you could
consolidate your records based on that field (or some other uuid).

On Tuesday, May 29, 2012, Christian Mondorf wrote:

I've just realised I didn't explain the problem properly.

We are identifying our stalls or shops by their name, location,
address, and business registration number. The problem comes from the fact
that addresses are very uncertain here, with streets often being known by
several names and the numbering being haphazard. The name and business
registration cannot be used as keys because if the business changes then so
do those two attributes. Rather, those are the attributes we need to know
how to change.

Finally, geolocation isn't accurate enough, so no each subsequent
visit we'll get a slightly different reading.

So the problem, then, what absolute attribute could we use to identify
our locations, which we could use to find them in the database to make
changes?

On Tuesday, 29 May 2012 13:40:43 UTC+8, Christian Mondorf wrote:

Hello,

I am wondering about how (and if) people have updated their databases
built up with ODK data. Is there an easy way to later capture data and link
it back to the original records, in some cases replacing them, or even
removing entries from the map (when a fruitstall is no longer there, for
example)?

How does this play out in the database?

So far it seems to me like ODK is a great tool for gathering data,
but not as well suited to updating existing datasets, although ODK Clinic
is able to access and modify records as you are working.

Thank you for any help!

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@**googlegroups.com
Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

The next version of Local Ground will also support some aspects of
what you are describing, albeit in a different way. Alpha release by
end of June!

··· On Wed, May 30, 2012 at 6:36 PM, Eric Meadows wrote: > Seems like you're delving more and more towards a full blown GIS. Maybe > also look at uDIG or qGIS as some open source GIS/data collection > applications. > > Possibly geo-code the businesses with a "geo-id". Using your GPS lat-long to > create a unique Id for each location that could always link back to the > location and northern actual business in that location. > > Could a "point" be located on the map by using structure footprints prior to > field collection or are there quality aerial photos available for this > location? If so field personal could navigate to each point, which would > have a unique id, and populate it with the business data. > > Several US 911 emergency address management organizations use a point to > represent structures and continually maintain new residents and businesses > using the above scenario. > > On May 30, 2012 9:21 PM, "Christian Mondorf" wrote: >> >> We're trying to have a continually updated map of traditional trade >> outlets, so mostly small "mom and pop" shops, and have them geocoded because >> addresses here are a bit of an uncertain science sometimes. >> >> We've thought of giving people barcodes which could be scanned on >> following visits, but this runs into two problems. The first is that the >> person we speak to on the first visit may be gone or have lost the code when >> we return. We've considered stickers but worry they may not be as permanent >> as one hopes. >> >> The second problem, of course is that such shops close down or are >> replaced, and handing over the barcode to the new tenants is probably the >> last thing on their mind. >> >> On Wednesday, 30 May 2012 18:28:34 UTC+8, ニコノコ wrote: >>> >>> What are you tracking? Or, what's your objective here? >>> >>> Personally, I don't consider an address as ideal unique identifier (i.e. >>> several corporations could share the same physical address but are separate >>> juridical entities.) For tracking repeated surveys of same respondents, the >>> ODK case study I've read assigns a unique ID for each respondent (read >>> through a QRcode). >>> >>> >>> >>> >>> On Wednesday, May 30, 2012, Christian Mondorf wrote: >>>> >>>> Thanks to both of you, you've both made some very interesting points. >>>> >>>> Unfortunately the business registration number doesn't really solve our >>>> problem since a change in the business would result in the new business >>>> occupying the address having a new business number. >>>> >>>> I know the obvious solution is to use the address as a unique identifier >>>> but this project was started as a way to circumvent the problems with >>>> working with addresses in Malaysia outside of the capital.. >>>> >>>> On Wednesday, 30 May 2012 10:36:08 UTC+8, ニコノコ wrote: >>>>> >>>>> I would assume that the business registration number is unique to each >>>>> enterprise. It's outside the scope of ODK, but in your database, you could >>>>> consolidate your records based on that field (or some other uuid). >>>>> >>>>> On Tuesday, May 29, 2012, Christian Mondorf wrote: >>>>>> >>>>>> I've just realised I didn't explain the problem properly. >>>>>> >>>>>> We are identifying our stalls or shops by their name, location, >>>>>> address, and business registration number. The problem comes from the fact >>>>>> that addresses are very uncertain here, with streets often being known by >>>>>> several names and the numbering being haphazard. The name and business >>>>>> registration cannot be used as keys because if the business changes then so >>>>>> do those two attributes. Rather, those are the attributes we need to know >>>>>> how to change. >>>>>> >>>>>> Finally, geolocation isn't accurate enough, so no each subsequent >>>>>> visit we'll get a slightly different reading. >>>>>> >>>>>> So the problem, then, what absolute attribute could we use to identify >>>>>> our locations, which we could use to find them in the database to make >>>>>> changes? >>>>>> >>>>>> On Tuesday, 29 May 2012 13:40:43 UTC+8, Christian Mondorf wrote: >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I am wondering about how (and if) people have updated their databases >>>>>>> built up with ODK data. Is there an easy way to later capture data and link >>>>>>> it back to the original records, in some cases replacing them, or even >>>>>>> removing entries from the map (when a fruitstall is no longer there, for >>>>>>> example)? >>>>>>> >>>>>>> How does this play out in the database? >>>>>>> >>>>>>> So far it seems to me like ODK is a great tool for gathering data, >>>>>>> but not as well suited to updating existing datasets, although ODK Clinic is >>>>>>> able to access and modify records as you are working. >>>>>>> >>>>>>> Thank you for any help! >>>>>> >>>>>> -- >>>>>> Post: opendatakit@googlegroups.com >>>>>> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >>>>>> Options: http://groups.google.com/group/opendatakit?hl=en >>>> >>>> -- >>>> Post: opendatakit@googlegroups.com >>>> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >>>> Options: http://groups.google.com/group/opendatakit?hl=en >> >> -- >> Post: opendatakit@googlegroups.com >> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en

Actually, that's exactly it. The idea of using barcodes to track locations
was to have a trigger which would signal that a given location was owned by
a new shop, so the record for the previous shop should be purged from the
database.

You are right, otherwise, that business registration is a good UUID.

I've been thinking for the time being we may capture business registrations
along with the rest of the data. Then, on following runs, we would contrast
the new data with the older version. If a business ID repeats that shop
would be considered to have a history in the database. If it doesn't it
would be a new one. If it was there before but wasn't encountered in this
second run, it would be a sign the shop is gone and we should deactivate it.

I realise this solution is not very elegant and I'm worried about how
robust the resulting data actually is.

@Eric, I also think it seems more and more like we're finding ourselves
moving down the road to using a fuller GIS solution. I've been looking at
qGIS.

@Tapan, I'm also intrigued by your comment. I looked
at http://localground.org/ but for now there isn't a lot of info there.

··· On Thursday, 31 May 2012 11:45:55 UTC+8, ニコノコ wrote: > > I suppose, just like in most cities, you'd have shops that are required to > have business licenses to operate. That'd be their UUID. It's extremely > unlikely for these not to be unique. > > Your idea of passing barcodes to the next tenant is strange, unless your > tracking what's found in a particular geopoint rather than what business is > at that particular geopoint. > > If it's *impossible* to assign them a unique identifier assigned to each > respondent, I'll play around with the geopoint location to compute the > distance and then see if they share the business name within a certain > radius to check if they've been surveyed before. Again, this is not within > the scope of ODK, which is a "data collection tool". > > @Tapan > Could you elaborate a bit on "local ground". I haven't heard of this > before. > > On Thursday, May 31, 2012, Christian Mondorf wrote: > >> We're trying to have a continually updated map of traditional trade >> outlets, so mostly small "mom and pop" shops, and have them geocoded >> because addresses here are a bit of an uncertain science sometimes. >> >> We've thought of giving people barcodes which could be scanned on >> following visits, but this runs into two problems. The first is that the >> person we speak to on the first visit may be gone or have lost the code >> when we return. We've considered stickers but worry they may not be as >> permanent as one hopes. >> >> The second problem, of course is that such shops close down or are >> replaced, and handing over the barcode to the new tenants is probably the >> last thing on their mind. >> >> On Wednesday, 30 May 2012 18:28:34 UTC+8, ニコノコ wrote: >>> >>> What are you tracking? Or, what's your objective here? >>> >>> Personally, I don't consider an address as ideal unique identifier (i.e. >>> several corporations could share the same physical address but are >>> separate juridical entities.) For tracking repeated surveys of same >>> respondents, the ODK case study I've read assigns a unique ID for each >>> respondent (read through a QRcode). >>> >>> >>> >>> >>> On Wednesday, May 30, 2012, Christian Mondorf wrote: >>> >>>> Thanks to both of you, you've both made some very interesting points. >>>> >>>> Unfortunately the business registration number doesn't really solve our >>>> problem since a change in the business would result in the new business >>>> occupying the address having a new business number. >>>> >>>> I know the obvious solution is to use the address as a unique >>>> identifier but this project was started as a way to circumvent the problems >>>> with working with addresses in Malaysia outside of the capital.. >>>> >>>> On Wednesday, 30 May 2012 10:36:08 UTC+8, ニコノコ wrote: >>>>> >>>>> I would assume that the business registration number is unique to each >>>>> enterprise. It's outside the scope of ODK, but in your database, you could >>>>> consolidate your records based on that field (or some other uuid). >>>>> >>>>> On Tuesday, May 29, 2012, Christian Mondorf wrote: >>>>> >>>>>> I've just realised I didn't explain the problem properly. >>>>>> >>>>>> We are identifying our stalls or shops by their name, location, >>>>>> address, and business registration number. The problem comes from the fact >>>>>> that addresses are very uncertain here, with streets often being known by >>>>>> several names and the numbering being haphazard. The name and business >>>>>> registration cannot be used as keys because if the business changes then so >>>>>> do those two attributes. Rather, those are the attributes we need to know >>>>>> how to change. >>>>>> >>>>>> Finally, geolocation isn't accurate enough, so no each subsequent >>>>>> visit we'll get a slightly different reading. >>>>>> >>>>>> So the problem, then, what absolute attribute could we use to >>>>>> identify our locations, which we could use to find them in the database to >>>>>> make changes? >>>>>> >>>>>> On Tuesday, 29 May 2012 13:40:43 UTC+8, Christian Mondorf wrote: >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I am wondering about how (and if) people have updated their >>>>>>> databases built up with ODK data. Is there an easy way to later capture >>>>>>> data and link it back to the original records, in some cases replacing >>>>>>> them, or even removing entries from the map (when a fruitstall is no longer >>>>>>> there, for example)? >>>>>>> >>>>>>> How does this play out in the database? >>>>>>> >>>>>>> So far it seems to me like ODK is a great tool for gathering data, >>>>>>> but not as well suited to updating existing datasets, although ODK Clinic >>>>>>> is able to access and modify records as you are working. >>>>>>> >>>>>>> Thank you for any help! >>>>>>> >>>>>> -- >>>>>> Post: opendatakit@googlegroups.com >>>>>> Unsubscribe: opendatakit+unsubscribe@**google**groups.com >>>>>> Options: http://groups.google.com/**group**/opendatakit?hl=en >>>>>> >>>>> -- >>>> Post: opendatakit@googlegroups.com >>>> Unsubscribe: opendatakit+unsubscribe@**googlegroups.com >>>> Options: http://groups.google.com/**group/opendatakit?hl=en >>>> >>> -- >> Post: opendatakit@googlegroups.com >> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> >

I wouldn't be too hasty about purging the data. The business may have
simply moved down the street (may only require an amendment of the business
license, but most likely to have the same reference number.), or the same
business is now operated by new owners/management.

Well, if you really want an elegant solution, I suggest that you look into
cooperating with the local government office in charge of overseeing
business licensing. You're most likely to have most of the data you want,
maybe except for geographic coordinates, and depending on local
regulations, businesses are actually required to inform the government if
they're retiring the business / closing shop.

QGIS is nice. Check out this Malaysian guy who's actively promoting its use
in Malaysian government offices: http://qgismalaysia.blogspot.com

Apparently, the Local Ground project was inspired by another interesting
project: Walking Papers http://walking-papers.org.

··· On Thursday, May 31, 2012, Christian Mondorf wrote:

Actually, that's exactly it. The idea of using barcodes to track locations
was to have a trigger which would signal that a given location was owned by
a new shop, so the record for the previous shop should be purged from the
database.

You are right, otherwise, that business registration is a good UUID.

I've been thinking for the time being we may capture business
registrations along with the rest of the data. Then, on following runs, we
would contrast the new data with the older version. If a business ID
repeats that shop would be considered to have a history in the database. If
it doesn't it would be a new one. If it was there before but wasn't
encountered in this second run, it would be a sign the shop is gone and we
should deactivate it.

I realise this solution is not very elegant and I'm worried about how
robust the resulting data actually is.

@Eric, I also think it seems more and more like we're finding ourselves
moving down the road to using a fuller GIS solution. I've been looking at
qGIS.

@Tapan, I'm also intrigued by your comment. I looked at
http://localground.org/ but for now there isn't a lot of info there.

On Thursday, 31 May 2012 11:45:55 UTC+8, ニコノコ wrote:

I suppose, just like in most cities, you'd have shops that are required to
have business licenses to operate. That'd be their UUID. It's extremely
unlikely for these not to be unique.

Your idea of passing barcodes to the next tenant is strange, unless your
tracking what's found in a particular geopoint rather than what business is
at that particular geopoint.

If it's impossible to assign them a unique identifier assigned to each
respondent, I'll play around with the geopoint location to compute the
distance and then see if they share the business name within a certain
radius to check if they've been surveyed before. Again, this is not within
the scope of ODK, which is a "data collection tool".

@Tapan
Could you elaborate a bit on "local ground". I haven't heard of this
before.

On Thursday, May 31, 2012, Christian Mondorf wrote:

We're trying to have a continually updated map of traditional trade
outlets, so mostly small "mom and pop" shops, and have them geocoded
because addresses here are a bit of an uncertain science sometimes.

We've thought of giving people barcodes which could be scanned on
following visits, but this runs into two problems. The first is that the
person we speak to on the first visit may be gone or have lost the code
when we return. We've considered stickers but worry they may not be as
permanent as one hopes.

The second problem, of course is that such shops close down or are
replaced, and handing over the barcode to the new tenants is probably the
last thing on their mind.

On Wednesday, 30 May 2012 18:28:34 UTC+8, ニコノコ wrote:

What are you tracking? Or, what's your objective here?

Personally, I don't consider an address as ideal unique identifier (i.e.
several corporations could share the same physical address but are
separate juridical entities.) For tracking repeated surveys of same
respondents, the ODK case study I've read assigns a unique ID for each
respondent (read through a QRcode).

On Wednesday, May 30, 2012, Christian Mondorf wrote:

Thanks to both of you, you've both made some very interesting points.

Unfortunately the business registration number doesn't really solve our
problem since a change in the business would result in the new business
occupying the address having a new business number.

I know the obvious solution is to use the address as a unique identifier
but this project was started as a way to circumvent the problems with
working with addresses in Malaysia outside of the capital..

On Wednesday, 30 May 2012 10:36:08 UTC+8, ニコノコ wrote:

I would assume that the business registration number is unique to each
enterprise. It's outside the scope of ODK, but in your database, you could
consolidate your records based on that field (or some other uuid).

On Tuesday, May 29, 2012, Christian Mondorf wrote:

I've just realised I didn't explain the problem properly.

We are identifying our stalls or shops by their name, location, address,
and business registration number. The problem comes from the fact that
addresses are very uncertain here, with streets often being known by
several names and the numbering being haphazard. The name and business
registration cannot be used as keys because if the business changes then so
do those two attributes. Rather, those are the attributes we need to know
how to change.

Finally, geolocation isn't accurate enough, so no each subsequent visit
we'll get a slightly different reading.

So the problem, then, what absolute attribute could we use to identify our
locations, which we could use to find them in the database to make changes?

On Tuesday, 29 May 2012 13:40:43 UTC+8, Christian Mondorf wrote:

Hello,

I am wondering about how (and if) people have updated their databases bu

--
Post: opendatakit@googlegroups.com <javascript:_e({}, 'cvml',
'opendatakit@googlegroups.com');>
Unsubscribe: opendatakit+unsubscribe@googlegroups.com <javascript:_e({},
'cvml', 'opendatakit%2Bunsubscribe@googlegroups.com');>
Options: http://groups.google.com/group/opendatakit?hl=en