Cycle through records from pulled data

We have a sample of say 1000 individuals randomly sampled from a country that we will interview. The names of these individuals are uploaded in the ODK app. For each individual, we know the names of shops in their neighborhood (this can also be uploaded in the ODK app).

We want to collect information on the shopping experience of the individuals in all the shops in their neighborhood. The first question would simply be: have you ever shopped in name of the shop and will also show a picture of the shop. Answer options would be yes/no and if no we proceed to the next shop in the neighborhood and if yes we ask a range of questions (eg. did you like the music that was playing in the shop,...). After all questions are answered we go to the second shop, etc until all shops in the neighborhood are exhausted.

Note that some individuals will have only one or two shops in their neighborhood, others may have up to 20 shops.

Hi @bjvca
We're glad you're here. When you get a chance, please introduce yourself on this forum thread. I'd also encourage you to add a real picture as your avatar because it helps build community!

On this forum we help users, we provide sample forms and fix the forms attached by users but I doubt anyone is going to build a complex form for you from scratch especially that you didn't even attach the csv file that you want to use so we don't know what the structure is.
Have you tried it on your own? Could you attach anything that would be a good starting point to help you?

Hi Grzegorz,
Thank you very much for your response. I fully understand that nobody is going to build a complex form for us. We are more interested in knowing if this is technically possible with ODK before starting to build the entire form.

I attach an example of the kind of data we would be able to upload or use in some way. In the first column are the names of the persons we need to interview. These will need to be uploaded in the from and selected using a select one widget at the start of the interview. Then, depending on the person chosen to be interviewed, the app should ask for every shop indicated in the remaining rows if the person liked the music in that shop (yes/no/dont know).

So if the enumerator selects "shopper_4", the next question should be:
did you like the music in shoprite? (yes/no/DK)
did you like the music in walmart? (yes/no/DK)
and then it should ask to save the record and the interview ends.

if the enumerator selects shopper_3, the next question should be:
did you like the music in marks_spencer? (yes/no/DK)
and then it should ask to save the record and the interview ends.

if the enumerator selects shopper_5, the next question should be:
did you like the music in walmart? (yes/no/DK)
did you like the music in costco? (yes/no/DK)
did you like the music in safeway? (yes/no/DK)
and then it should ask to save the record and the interview ends.

etc...
Thank you so much for any advice!

example_csv.csv (212 Bytes)

It is possible but your csv file would need to have a different structure. Ideally, it would be separated into two csv files like:
shoppers.csv (427 Bytes) shops.csv (256 Bytes)

Then you could build a form like:
shopsForm.xlsx (6.3 KB)

1 Like

Hi Grzegorz,
Thanks again for this! This is extremely helpful. But I was wondering if it is possible to get rid of the select_multiple shops line and directly enter in the loop? We know the shops that each shopper has to rate, and they have to rate all of them.

An alternative would be that all shops are pre-selected such that the enumerator can just swipe to the first shop in his/her list.

Best,
Bjorn

Yeah I know it would be better to get rid of that select_multiple question but I doubt it's possible...
Maybe @LN knows something more?

Thanks,
If it is impossible to get rid of that select_multiple, perhaps I can pre-select all (sometimes there are up to 20 shops so it is really a pain to have to manually select all).

I have looked into > this post but can not immediately see how to make this work for all the options instead of only for the first position as below:

once(instance('options')/root/item[position()=1]/name)

This answer could be helpful Loop over pre-established list - #6 by LN but I think it doesn't wok with external csv files.
Do you really need external files? Or maybe your data is not that big and could be added to the choices sheet?

Thanks for your continuing support. I tried running the example form of @LN on ona.io but it gave me this error when I tried to upload:

DK Validate Errors: >> Something broke the parser. See below for a hint. Error evaluating field 'nom_masque' (${masque}[1]/nom_masque[1]): null Caused by: java.lang.NullPointerException ... 10 more The following files failed validation: ${tmpl2324wjx} Result: Invalid

Also, my knowledge of ODK is not sufficient to get an idea of what the form does and then to

pattern match on the structure without a full understanding of the details.

Bjorn

I played with that form some time ago and despite the fact that at first glance it may look like magic it works but as I said with the internal choice list (not from external csv files) so that's why I asked you whether your dataset is really that big that you need external csv files?

I don't know what is the limits are to an internal choice list. Currently I have the data in your shoppers.csv as a choice list and they are about 3,400 records. In my final version of the shops.csv, there would be about 18,000 records because each shopper would rate on average about 5 to 6 shops.

But as I mentioned before, I get an error running example form in the post you refer to so I don't really know what the magic looks like (and I don't want to start messing with the partial solution I have now before I know why I can not run the example form).

I've updated the sample form and put a comment on the nom_masque field.

The form converter used to always generate translations even for single-language forms which made the nom_masque expression a little more complicated. Now, there are no translations generated for a single-language form so attempting to look up translated text fails.

The actual performance will depend a lot on the devices you use and on how many columns your dataset has. We verify that 50k records with three columns can be queried instantly on most devices from 2019+.

There is no difference in performance or what can be done with internal datasets and CSVs attached using the recommended approach (NOT what @Grzesiek2010 shared). That means you could use the same approach from the form I shared. Unless you're using older devices, that will likely work well.

If that performance isn't acceptable, only then would I recommend looking at a solution using search/pulldata. The concept would be the same but the syntax is different.

2 Likes

Hi @LN

Thanks for updating the sample form. It works now. Still, I am not sure this is actually what I am looking for.

I have tried to incorporate the idea in the sample form that @Grzesiek2010 proposed. Recall that this did what I wanted, but it would be better to get rid of the "select_multiple shops" question. I tried to accomplish this with the approach of @LN but the thing is that I need to filter the choice list by shopper and that does not seem to work (see shopsForm_2.xlsx (8.0 KB) that now has shops.csv incorporated in the choice list but shoppers.csv (427 Bytes) still needs to be uploaded as media file).

Any additional clues would be immensely appreciated!

Maybe something like this:
shops.xlsx (15.3 KB)

1 Like

Yes, that is exactly what I was looking for! Thank you so much!