Proposal: specify a fixed image to be annotated or signed

[LN]

Sign or annotate a fixed image specified in the form definition describes a feature that would allow form designers to specify an image to be annotated or signed.

There have been two different approaches proposed for adding support for this in the ODK XForms spec.

Use defaults

One idea brought up by @Grzesiek2010 in https://github.com/opendatakit/collect/pull/2428 is to use the standard default value mechanism. This doesn't currently work for binary files (as described below) but it could be a way to support annotation/signature of fixed images without adding any form spec changes.

As stated in the specification, "Any value inside a primary instance is considered a default value for that question. If that node has a corresponding input element that value will be displayed to the user when the question is rendered." However, this is not currently true for questions of type binary. This is because clients typically (always?) see the following as different and saves them in different places:

• media attached to the form that can be used to, for example, illustrate a question or lookup a value
• media attached to a filled form that will be sent to the server with a submission

That means that if a form's primary instance contains the following: <my_image>my_face.jpg</my_image>, the client won't find my_face.jpg in media attached to the current filled form and so the default value won't be recognized. This is true even if the user uploads a media file with filename my_face.jpg alongside the form definition. However, if the user selects an image called my_face.jpg from their client, <my_image>my_face.jpg</my_image> will now be a valid part of the primary instance.

It's weird that defaults work for all types except for media (binary). One way to make this more consistent would be for clients, when they see a question of binary type with a filename value, to look in the directory for the current filled form for that image (current behavior) and if it's not found, look in the directory for media attached to the corresponding blank form and make a copy to the filled form directory.

One question about this is whether defaults should also be allowed for audio, video, raw images, etc. I think so for consistency even though it's unlikely to be all that useful.

Use the image label or introduce a new attribute

As described by Mitch in https://github.com/opendatakit/collect/issues/84. This would more closely match how media files are added to a form for illustration purposes (images, audio, video).

Another related option would be to introduce an appearance such as markup that combines with annotate or signature and hides the buttons to select or capture a new image while also using the label image for the annotation.

No matter the spec option we choose, we'll probably want an appearance to hide the images for taking and choosing images.

[martijnr]

Thanks for this @ln! It is very helpful to me to see this written out.

I think the default-value-with-image-file approach is possible but to avoid magic to pass image files with an empty form, we would need to introduce a new datatype (e.g. anyURI or base64Binary). For the former, I think we could use the jr:// scheme to retrieve the file from the manifest and submit as base64Binary. The latter would be a little easier to implement on the client. Orbeon supports [something like that]((some example here).

The challenge is probably to make this easy to use for users when designing and uploading a form and analyzing the submitted data (edited images). It would probably require some fancy processing in pyxform and especially in Aggregate/Central/Ona/KoBoToolbox.

Otherwise, to keep things simple for pyxform, Aggregate, Ona, KoBo, I'm in favor of a more hacky but not terrible approach using a new appearance and question label (tbd exactly how).

[LN]

Always good to dive into these tricky spec issues with you, @martijnr!

@martijnr

I think the default-value-with-image-file approach is possible but to avoid magic to pass image files with an empty form, we would need to introduce a new datatype (e.g. anyURI or base64Binary). For the former, I think we could use the jr:// scheme to retrieve the file from the manifest and submit as base64Binary.

To give a concrete example, the form definition would have a node that looks like <my_image>jr://images/my_face.jpg</my_image>, the question widget would copy that file, allow the user to modify it, and then save <my_image>my_face.jpg</my_image> (or perhaps <my_image>new_filename_generated_by_widget.jpg</my_image>)? I don't find this terribly offensive but I'm still not convinced it's better than what I was describing.

The base64Binary option seems like a convenience option not to have to separate the form and its media. I think we can consider that separately.

@martijnr

avoid magic

Agreed this is generally an important thing to aim for. That said, if a blank form's instance has <my_image>my_face.jpg</my_image> and its corresponding manifest contains a my_face.jpg file, that doesn't seem like magic to me. I find it rather consistent, again given a slight change in client rules in where media described in an instance is fetched from. That is, the client would look first in the instance media storage location with a fallback to the form media storage location.

I see that you want to maintain the conceptual distinction between media attached with the blank form and media attached with a filled form but it's not clear to me why it's such a critical distinction. It doesn't seem consistent with the way other defaults work and it seems that with the approach you've described, a binary node with a default would look different from a binary node with a value set by the client. This feels odd (though not terrible).

What am I missing?!

[LN]

@issa, it would be great to get your thoughts on this since you recently worked on parsing ODK XForms to extract a list of required media files for Central's media attachment feature. Do you have a sense of what would seem most consistent and least magical to you?

[issa]

My impression is that the default values approach feels very aesthetically nice from a math perspective, but I'm not honestly sure what problem it solves besides enabling this case. What else would default binary values be used for? Why would we do the work of downloading all that data just to replace it?

I'm also not very clear on the suggested datatype, or how the filesystem on this thing actual works—is it shared between form-world and submission-world? Or would the default value be special-cased to look at form media and applied values look to submission media?

Are there other combinatorics to worry about by introducing default values for binary fields? I don't know a lot about XForms but I know it's ridiculously interconnected.

From the perspective you were after, either way it's a new detection case. As much as it seems like the messy all-of-the-things-all-over option, it's truly not offensive from a detection perspective to just hunt down some particular binding or input attribute for a filename.

[Xiphware]

@LN

I see that you want to maintain the conceptual distinction between media attached with the blank form and media attached with a filled form but it's not clear to me why it's such a critical distinction.

Agreed.

I dont quite see why this cant be accomplished by simply 'prefilling' the form with defaults - in this case an image - and having some (appearance?) flag to indicate that instead of replacing the (image) 'value', instead you want to mark-it-up/edit the existing 'value'. Indeed, you probably want to support basically the same usecase anyway, to allow users to markup a photo after they've taken it [although I agree that marking up a previously captured signature doesn't make a lof of sense... But in this case the previous 'signature' would actually be a background picture].

[martijnr]

@LN

It doesn't seem consistent with the way other defaults work and it seems that with the approach you've described, a binary node with a default would look different from a binary node with a value set by the client. This feels odd (though not terrible).

What am I missing?!

I agree that would be odd. With the new datatype I was suggesting to store the new value in a new way as well. The new way would be as a URI or base64Binary. I think anyURI, would cover both jr://image/path/to/image.jpg as well as base64 (but am not sure).

[LN]

@martijnr

I was suggesting to store the new value in a new way as well. The new way would be as a URI or base64Binary.

Could you please share an example of what the XML would look like? I need something concrete to sink my teeth into!

@issa

What else would default binary values be used for? Why would we do the work of downloading all that data just to replace it?

A user would have to explicitly decide to set a default value and attach an image so I don't think there would ever be a case where downloading the data would be a "waste." Are you thinking of a scenario?

@issa

is it shared between form-world and submission-world? Or would the default value be special-cased to look at form media and applied values look to submission media?

I'm imagining a change in client lookup rules when a node with a binary type binding in the instance has a value. Right now clients only look in the submission world. The change I propose is for them to still look in the submission world first but if they don't find something with that filename there, they look in the form world and copy the asset to the submission world. This maps to my mental model of what a default is.

@issa

Are there other combinatorics to worry about by introducing default values for binary fields?

I can't think of anything but perhaps you can give me an example of something like that you've seen before?

@Xiphware

and having some (appearance?) flag to indicate that instead of replacing the (image) 'value', instead you want to mark-it-up/edit the existing 'value'.

Can you say more about why that flag would be necessary? The way I see it, we'd just exactly match what happens now if you open an existing filled form and it has a binary file associated with it. That is, the question would be populated with the contents of that file and the user could either continue marking that up or replace it with a new "canvas" to mark up.

[Xiphware]

@LN

Can you say more about why that flag would be necessary? The way I see it, we'd just exactly match what happens now if you open an existing filled form and it has a binary file associated with it. That is, the question would be populated with the contents of that file and the user could either continue marking that up or replace it with a new "canvas" to mark up.

For a signature, if there's no background image then you when you 'edit' the signature you basically want to replace the image with a new signature image (not markup the previous signature, which would be strange). But if there is a background image then you do want to markup the image (with your signature). Without a flag or something I dont quite see how you can determine whether any existing image is a background (to markup) or a previous signature (to replace)?

@LN

the question would be populated with the contents of that file and the user could either continue marking that up or replace it with a new "canvas" to mark up.

Alternatively, every time you open up an existing image (signature or otherwise) you have to prompt whether the replace or markup, before proceeding, which sounds like what's happening now.

[LN]

@Xiphware

For a signature, if there's no background image then you when you 'edit' the signature you basically want to replace the image with a new signature image (not markup the previous signature, which would be strange).

Currently signature and markup have the same behavior: when you edit an existing value, you modify what's already there. This is consistent with how paper or other kinds of digital signature interfaces work -- if you're like me and often lose the bar on the t in your last name in your signature, you can always go back and add it in. You can also erase what's there and start over.

[Xiphware]

@LN

Currently signature and markup have the same behavior: when you edit an existing value, you modify what's already there.

Seems a bit strange to me to 'markup' a signature, but for simplicity (and consistency) it probably no biggie. As you said, you can always hit "Clear" to explicitly replace anything already there

[martijnr]

@LN

The change I propose is for them to still look in the submission world first but if they don't find something with that filename there, they look in the form world and copy the asset to the submission world. This maps to my mental model of what a default is.

I think this may make more sense in Collect than in Enketo. All I want to change in the proposal is to clearly indicate where the file is (@issa's submission-world and form-world distinction is helpful). Since we already have a way to point to media labels and external data, a jr:// URL makes sense to me to point to a file that was uploaded as part of media labels and external data.

I am realizing now that our <bind type="binary"> is completely custom so that gives us some flexibility. Our ODK spec says:

String ID (with binary file attached to submission)

The String ID is in fact a file name with extension and the client/server is supposed to find a file with that name in the attachments that came with the XML record.

I wonder if we can see <image>tree.jpg</image> as shorthand for <image>./tree.jpg</image> or <image>file://tree.jpg</image> and tweak/clarify our spec to something wider like:

URI pointing to binary file. For local files attached to a submission, the filename with extension should be used.

This would facilitate loading a default value <image>jr://image/tree.jpg</image> (for any binary question type - though we could restrict to annotate).

I think this would mean that bind type="binary" should be seen as a subset of the W3C type="anyURI".

[LN]

Thanks for reviving this, @martijnr! I think it will be very useful functionality for users so it would be great to find an approach we can agree on.

@martijnr

I think this may make more sense in Collect than in Enketo.

Ah! That's very possible. My mental model is that clients pick some location to store the image files that a specific form definition can refer to. When clients see jr://image, they replace it with the appropriate location and use the path that follows it to identify a specific file. This is the form-world.

In records, only filenames with extensions are used. Systems can choose where they store media for any given submission and interpret paths in the records as relative to that. This is the submission-world.

Systems may or may not store actual files. For example, they may instead store binary blobs in databases. The distinction between the two worlds is that binary "files" in form-world are shared across records for that form definition whereas files in submission-world belong to a specific record.

Does Enketo do something different?

@martijnr

All I want to change in the proposal is to clearly indicate where the file is (@issa's submission-world and form-world distinction is helpful). Since we already have a way to point to media labels and external data, a jr:// URI makes sense to me to point to a file that was uploaded as part of media labels and external data.

I see what you're driving at and it makes sense to me in the form definition but I'm just not sure what should happen with submissions. It wouldn't make sense for clients to send records to servers with jr:// URIs, would it? Currently, filling out a blank form means taking the primary instance block from the form def as-is, and letting the user set/modify its values. It seems like allowing jr:// URIs as defaults in the instance would break that. (EDIT: unless we do allow jr:// URIs in submissions but that would require the receiving entity to use the original form definition media to reconstruct records. Granted, it's not likely that users would want to submit the original, unmodified, default media but it seems that it should be possible.)

That is, everything is fine if the user modifies the value because clients can look up the media file using the jr:// URI, allow for some modification on it and then save back just the filename. But what if the user doesn't want to change the default for some reason? It seems like we'd have to do something like say that form recomputation results in any binary files referred to by a jr:// URI being copied to the submission-world and the URI in the instance being modified.

If I'm understanding correctly, I would be ok with that but I still have a slight preference for instead allowing unprefixed filenames in the form definition's primary instance and modifying the rules for finding those as I have described (if it doesn't exist in submission-world, copy from form-world without any URI change needed in the record).

Either way, I imagine that the XLSForm user experience would be the same -- users would just put a filename with extension in the default column, right?

@martijnr

URI pointing to binary file. For local files attached to a submission, the filename with extension should be used.

This seems like it more accurately represents current reality so I'm on board. Perhaps we could be even more explicit and say something like "For local files attached to a submission, only the filename with extension should be used without a scheme or subdirectories." Would those changes capture your intent?

[martijnr]

Thank you for your feedback, and flexibility!

@LN

Does Enketo do something different?

Yes, records and form resources are separate. In any case, I see this default file as a form resource and to be able to locate it (sanely), some explicit "form resource" reference is really important in my opinion.

@LN

But what if the user doesn't want to change the default for some reason? It seems like we'd have to do something like say that form recomputation results in any binary files referred to by a jr:// URI being copied to the submission-world and the URI in the instance being modified.

The way I see it, we don't have to do anything on the client side in that case. It's for the server to deal with. A user sees the jr:// reference if the server does nothing special (but the server could replace that with the word "default" or something, or replace the jr: reference with a link to the default image that is present with the form resources.)

@LN

Either way, I imagine that the XLSForm user experience would be the same -- users would just put a filename with extension in the default column, right?

Yes, definitely. The fancy stuff will be hidden in pyxform.

@LN

Perhaps we could be even more explicit and say something like

Yes, that sounds great, and maybe avoid any confusion with unchanged default values by adding "user-uploaded" so we'd end up with:

"URI pointing to binary file. For user-uploaded files attached to a submission, only the filename with extension should be used without a scheme or subdirectories."

[LN]

@martijnr

but the server could replace that with the word "default" or something, or replace the jr: reference with a link to the default image that is present with the form resources

I was hoping to avoid additional complexity on the server side but given that it will be unusual to see jr:// references in submissions, I think it's an ok compromise.

So I think that the ODK XForms spec changes would be:

• The modification to the binary row in the data types section that you've suggested above. ("URI pointing to binary file. For user-uploaded files attached to a submission, only the filename with extension should be used without a scheme or subdirectories.")
• A new paragraph in the primary instance section after "any value inside a primary instance..." that says something like "For nodes of type binary, defaults use file endpoint URIs."

The second is not strictly necessary but I think it would be helpful to make explicit.

@Xiphware, any hard questions or do you want to join our consensus party?

@tomsmyth, you may want to also review. A healthcare usecase would be letting a clinician circle pain zones on a default body image. That would be similar to what the image map question type enables but wouldn't require creating an svg and allows for more qualitative/analog input.

[Xiphware]

It seems like we should be able to get away with just adding something to say:

1. values in the primary instance for binary files can either be the existing filename.ext format (which the client currently uses for new photos, signatures, files, ...) - or an jr:// URI endpoint (which would indicate an annotatable default).
2. any jr:// file references in the primary instance should either be stripped by the client upon submission (eg the default wasn't changed), or will be ignored by the server upon receipt.
3. client should replace any jr:// with a new filename should the image be annotated/signed (as opposed to updating the form's original).

Have I got that right?

[martijnr]

There is a slight difference in my understanding. A filename.ext refers to a file attached to the submission. Any URL points to a file elsewhere. So your point number 2, any scheme should not be removed if the default remains unchanged, because that would require the client to attach the default file to the submission (which seems unnecessary).

[LN]

Thanks for jumping in, @Xiphware!

@Xiphware

be stripped by the client upon submission (eg the default wasn't changed)

I don't think we'd want to do that for the reasons that @martijnr describes but also because we want to be able to represent "the default was maintained" differently from "the default was cleared."

@Xiphware

client should replace any jr:// with a new filename should the image be annotated/signed (as opposed to updating the form's original)

Do you think this needs to be said explicitly? If so, where does it go? Maybe in the new paragraph I described after "any value inside a primary instance..." that describes binary defaults?

[Xiphware]

@LN

I don't think we'd want to do that for the reasons that @martijnr describes but also because we want to be able to represent "the default was maintained" differently from "the default was cleared."

Yup. Good point. Server can ignore a jr:// image reference in the instance XML (although that might preclude 'submitting' an image in the future via specifying a URL to somewhere else - eg DropBox. - instead of including binary data in the POST. But that's another can of worms...)

@LN

Do you think this needs to be said explicitly? If so, where does it go?

Yeah probably superfluous. Client should be doing this anyway whenever saving an image.

I guess that mean's I've joined you consensus party, huh?

[danbjoseph]

implemented and pending pyxform https://github.com/XLSForm/pyxform/issues/405 and docs https://github.com/getodk/docs/issues/1184