1. What is the issue? Please be detailed.
I am dipping my toes into the entity waters and would like to make it clear, for an enumerator, any questions where they have changed the existing entity data (i.e. previous survey). For context, the form has 80+ questions, probably c20 are unlikely to need changing (fixed attributes of the entity), 40 that COULD have changed, and the remaining 20 are questions about the current state of the entity (and in this case best not to influence the response by showing the previous)
I have two scenarios:
1: If there is an error in the original entity value(s) and the enumerator changes a value - I would like them to be aware that they have changed it when checking prior to submission
Example might be the angle of slope (which I hope hasn't changed!)
2: An aspect of the entity has changed so the original value needs to be updated - I want to pre-populate the form with some entity data and leave some fields empty so that this does not bias the repeat data.
Examples for scenario 2:
a) count the number of gates or length of ditch (there might be new ones since the last survey, but useful for the enumerator to know how many were there previously and highlight if the field has changed);
b) qualitative score of current condition (best not to influence that, so an empty response - no need to have a colour highlight / change, but it would be subject to validation to check it's not empty)
It would be really helpful for the enumerator to be able to tell the difference so that if they change an existing value this is highlighted, compared with entering new values. A kind of visual validation in the field.
I can't see anything in the docs section on form styling but wondered if I'm looking in the wrong place... I was hoping that a constraint could be used, but this appears only to allow a message (that could be useful for scenario 1, but probably not too helpful for scenario 2).
Intending to come to the next Entity Office Hour - be warned @LN
Once again this might fall into the 'ideas' category, so please move as appropriate. Hopefully not...
I'm looking forward to it! And to anyone wondering what this is, it's part of ODK Insiders!
For the first scenario, I don't think you'd want to use constraints because those would prevent you from making changes. You won't be able to apply a style to the field itself but you can have a styled message that only appears if the value has changed. You can also do things like share the old value and ask whether it's still accurate. Here are a few ideas: https://docs.google.com/spreadsheets/d/17rqStJer5SaDUazxTgpp3FnJpBDOKtnndwhgwE9mpek/edit#gid=1068911091
I used an in-form list to make the example easy to try but everything should work exactly the same with a select_one_from_file and an entity list named trees.
For scenario 2, it sounds like you want to capture a new value and make the field required. If you need to save those values to the Entity, you would use the save_to column. Does that sound right?
Thanks @LN for your usual extra-ordinarily helpful response!
Good point about constraint. That fails my requirement.
The suggestion in the spreadsheet is a neat idea - I think I would have a very complex form to manage all the variables that might change, but it could definitely have it's place.
What I was hoping for, in terms of a check in pseudo code: If(${value}!=${entity_value},changeTextColour,asYouWere) so your response confirms that it doesn't exist.
I had tried to include the entity value in the hint column but got errors from Central when trying to upload the xlsx, and the errors disappeared when I removed these instance... inclusions. So I assumed they wouldn't work (my usual problem of it's me, not you)! I'll try that again now I've seen your example. It might be a way of seeing which variables are different, rather than changing the style, just not quite as visual - probably better than pre-populating for some fields though.
I had reached similar conclusions about scenario 2 - thanks for confirming.
And that brings me to a more 'philosophical' or data management issue I'm trying to resolve - and will probably raise at the Office Hours: how to best manage entities and submissions in terms of fixed properties and those that might change. My Old Skool data relationship head thinks I should store only 'fixed' properties within an entity table, then 'state' properties in a submission (that either contains the fixed properties or has a KEY) - trying to keep the overhead of the entity list minimal, but then having the previous / most recent state can be advantageous, so that points towards all fields being within the entity table.
Old Dog, New Tricks - struggling to get my head around how the data will unfurl over time and be traceable without becoming a Data Monster. I know this is my problem not technically an ODK one, but it's a stumbling block for me, and a reason I've been reluctant to open Pandora's Entity Box!
Can you say a little bit more about what your concern with this is? Is it tedious to create? Are you worried about the extra blank note fields that will need to be submitted? Form performance? Something else?
If I were doing something like you describe, I would pick one of the user experience options from the sample spreadsheet, write out the XLSForm for one property, test and refine that, and then copy and paste the ~3 rows for however many properties I want to possibly get an update for. The logic in any of these is simple and it's isolated so you should be able to do hundreds, likely thousands, and quite probably tens of thousands without issue (for a small total number of Entities).
I think you're running into this pyxform issue which we hope to address soon.
You can put instance expressions directly in note fields. If you want to provide some kind of visual feedback, though, you'll need to use a separate calculate to compute the comparison. You can do something like define a calculate with an expression like
if(${value} != ${entity_value}, "⚠️ Value changed from ${entity_value} to ${value}", "")
And then reference that calculate from the label of the field. Currently in Collect that will only update after leaving the screen and coming back. It will be visible in the summary/hierarchy/jump screen. So that could be useful for review but I'm not convinced it's all that much simpler than the example I showed with a second question and relevance.
That is an art and certainly is going to be very project-specific. Adding Entities certainly adds more interesting design considerations. I think that will be a great question to dig into when we get together for office hours later this week.
I have tried again with the entity values in labels and found it to now work - I had used double quote, but changed it to single quote (after seeing your example) and now it works:
shows on the collect form:
Condition - previous: 4 (including on the jump form)
success!!
Regarding the 'relevant' questions - the main concern is the extraneous data columns in the output when using Notes - I won't find it helpful to have a CSV that is 130 columns wide when I have 80 data columns. But for smaller / shorter forms I think the technique has value - and the 'code block' repeat method of adding the note is a method that I already use - thus avoiding the tedium of construction, but it is still what I would call 'clunky' to pull together at scale. From a user perspective it slows data entry to have to navigate past each of the displayed notes (so in my case maybe an extra 40 swipes?)
I was hoping that entities were not going to be a 'dark art'
I think we forgot that double quotes are allowed and only detect instance expressions with single quotes currently. Will try to fix that for the next Central release.
@ahblake I wonder whether this might be why you were seeing some expressions that worked as expected and others not.
That's fair. You could do something like use a note_ prefix and filter those out automatically in your analysis. We'll likely add an option like that at some point.
Yes, I would probably put each pair in a field-list so they're clearly grouped.
Here's a brief summary of the highlights from the Insiders Office Hours:
We discussed one of @seewhy's needs around footpaths. There's an in-depth assessment of these paths every 5 years and then some ongoing maintenance or other todos related to individual paths in between.
If there was only the assessment on a 5 year schedule, I would probably recommend using a manually-attached CSV to get data from the last assessment into the current assessment.
Because there is also the possibility of ongoing work to do with paths, it's likely worth it to have an Entity List representing those paths. That way any form that does path-related work can use that same list with up-to-date data. Certain forms can update its properties like maybe a "last visited date".
Path Entities likely can have a small number of properties representing unchanging aspects of the paths. Their role is to help identify the right path in a form and to connect form submissions about the same Entity.
For the 5-year assessment, there's a desire to make past data accessible. My suggestion is to use a separate Entity List for that. It could be named something like path_assessments.
Each path_assessment Entity would have a path property that stores the id of the path that the assessment is about. It would also have a property representing the assessment year. That would make it possible to write expressions like instance('path_assessment')/root/item[path=${path} and year=${year}] to access an assessment about a specific path in a specific year.
The 5-year assessment analysis would likely be done on data from the path_assessment Entity List and the corresponding submissions could be ignored. In contrast, ongoing maintenance data would probably be accessed from submissions. You can mix and match as appropriate, keeping in mind that once an Entity is created from a Submission, the two lead separate existences (updating one doesn't update the other).
One limitation/downside to consider is that currently a form submission can only update one Entity List. That means if the assessment creates an Entity in the path_assessment list, it can't also update a path Entity to do something like mark the Entity as visited in the current year. Instead, you have to write more complex expressions to compute things like completed status.
Hopefully these high-level notes are helpful to someone. I'd like to eventually put together a worked example for this. @seewhy is there any chance that your data is or could be made public? It would be very cool to build a demo for some real paths that people could potentially go visit. No pressure, though, I can always generate data!
I think we forgot that double quotes are allowed and only detect