ODK Collect v2024.2 Beta: prompt to recover, select from map line and shape styling, offline Entities prototype

[LN]

ODK Collect betas are an opportunity to get community feedback on upcoming releases. If you have an ongoing data collection campaign, we recommend quickly verifying your form on a test device. The release will be delayed until all reported issues are fixed.

This beta includes an experimental prototype of offline Entities! You may find it useful to try if you are planning future workflows that would benefit from creating or updating Entities directly on device. For now, offline Entities are opt-in, see the end of the post to learn how. We are currently planning for production-ready offline Entities in v2024.3 which we aim to release in early fall.

Joining the beta program
To join the beta program, find ODK Collect in the Play Store on your device (not in the web browser) and scroll all the way down. Please don't join the beta with a device or account actively used for data collection! In particular, note that joining the beta is account-based. If you use the same Google account across multiple devices, do not join the beta with that account.

Leaving the beta program
You can leave the beta program from the bottom of the Play Store at any time. Once you leave, you will get the next production update when it is released. If you need to go back to the previous production release, uninstall and reinstall the app. Your settings will be reset but your forms will remain (though backups are always recommended).

What to check in this release

• If Collect closes unexpectedly because of a crash or going out of memory, when you next open the form you were working on at that time, you will see a dialog asking whether you want to recover work that was automatically saved. Previously, recovery was done automatically without any indication that it had happened. You can try this by swiping up to kill Collect while you're filling out a form.
  • It is possible for Collect to go out of memory while a form is being filled and then receive a form update when it is re-opened. Previously, the change in form version would mean your prior work could not be recovered. You will now be given the option to recover saved work even if the form has since been updated. Recovered data will be opened with the updated form version. See this issue reported by @mathieubossaert
• Autosend should now retry periodically at increasing time intervals after a failure. We recommend testing out autosend with your expected network configuration if you use "cellular only" or "WiFi only" autosend options.
• Dynamic relevance expressions should now update within a field list. See this issue reported by @wroos
• Styling of lines and shapes in select from map. In addition to the special properties documented here, you can now set stroke and fill to colors and stroke-width to an integer.
• Text and number questions now support a new masked appearance requested by @TobiasMcNulty. When this is present, the data being input will be hidden (like in a password field).
• Read-only geoshape and geotrace questions should not allow edits. See this issue reported by @ahblake
• Beautiful icons have been added to many of the capture buttons (audio, video, picture, date, time, barcode, and more). See this screenshot.

Offline Entities prototype

Offline Entities in this release are considered a prototype because we will be changing the way Entities are stored to improve performance. This means offline Entities will be erased after upgrading to a future Collect version. This prototype does not handle multiple offline updates well yet, especially when there are conflicts across multiple devices/data collectors.

We encourage those using Entities in offline environments to try this prototype with their workflows and let us know how they plan to use it so we can make sure we have considered all common needs.

To try local Entities available in follow-up forms even when offline:

1. Explicitly tell Collect which Entity List(s) to manage locally. From the Collect main screen, tap the project icon in the upper right corner and then tap the Settings button. Scroll down to Experimental and tap on it, then tap Entities, Add entity list, and type in the name of an Entity List to manage locally. You’ll then need to resynchronize or redownload forms so that the local entity lists can be updated.
2. Toggle on Enable local entities so that follow-up forms actually use those local Entities.

In future versions of Collect, neither of these steps will be required.

Additional testing
You can find a full list of changes in the release notes:

• https://github.com/getodk/collect/releases/tag/v2024.2.0-beta.0

Thanks to all testers for your help!

[ahblake]

@LN

• Styling of lines and shapes in select from map . In addition to the special properties documented here , you can now set stroke and fill to colors and stroke-width to an integer.

This is so good. On implementation;

• you can't have a purely transparent colour for a polygon fill can you? I left some values blank and they appeared in the default red.
• is the fill rendered a lighter/ partially transparent version of the colour selected? It appears to be from a quick test but I'm not sure.
• what are the default values for unstyled polygons?

I need to build out a palette to use now...

Edit: styling only applies to select from map doesn't it? No way to affect view/edit/create geoshape/trace? I have painted myself into a corner with a library of tilesets that have some items shown in red, and overlaying a selected geoshape is now invisible as it's also red and without vertice indicators.

@LN

• Read-only geoshape and geotrace questions should not allow edits. See this issue reported by @ahblake

Super, tried this and a read only geoshape has no visible vertices to move anymore as expected.

[mathieubossaert]

@ahblake

This is so good

So good ! Many thanks to the team !

@ahblake

is the fill rendered a lighter/ partially transparent version of the color selected? It appears to be from a quick test but I'm not sure.

From my own test, if you don't set the fill property, the polygon is filled with the default "red" transparent.

First test here :
I achieve to manage showing visited/to visit places on the same select map

But not in a dynamic way (I mean not with a test that defines the style to apply). Anyway this is a first workaround and the end user don't care about how it's done

• I duplicated the all the json objects with another style.
• I also add a property call "visited" set to 'true' or 'false'
• And I use choice filter to show the green version (visited) or the orange version (to visit).

It will be used on the field next week

The form (in french) :
presence_absence_par_maille_pour_forum.xlsx (451,4 Ko)
The geojson :
mailles_4326_presence_absence.geojson.zip (29,4 Ko)

[mathieubossaert]

@mathieubossaert

It will be used on the field next week

I forgot we are in beta... so it will be used on the field as soon as 2024.2 will be released.

[mathieubossaert]

@LN

If Collect closes unexpectedly because of a crash or going out of memory, when you next open the form you were working on at that time, you will see a dialog asking whether you want to recover work that was automatically saved.

Second test. I started a form an then closed Collect without saving the form.
When I restart the app and try to fill the same form (or another one), I get this error :

I've just corrected the translation error (précédant -> précédent)

[LN]

@ahblake

you can't have a purely transparent colour for a polygon fill can you?

That's currently the case. Our thinking is that red is a more useful default. Is there a case in which you'd really want full transparency?

@ahblake

is the fill rendered a lighter/ partially transparent version of the colour selected?

Correct. We'll make sure to document that clearly.

@ahblake

what are the default values for unstyled polygons?

The stroke and fill are pure red (#ff0000) with opacity of 68% and stroke width 8.

@ahblake

styling only applies to select from map doesn't it?

Yes, that's right. Unfortunately we'd have to come up with a different spec to style geoshape/trace fields. I don't have a great workaround for you but will think about it.

@mathieubossaert

I achieve to manage showing visited/to visit places on the same select map

Looks great!

@mathieubossaert

not in a dynamic way (I mean not with a test that defines the style to apply).

Yes, that would be nice, wouldn't it? It's a concept we're interested in but the spec will be quite different from anything we have now.

I get this error

I think it should be fixed in beta 1! There was a problem with the date string translation.

I've just corrected the translation error (précédant -> précédent)

[ahblake]

@LN

Is there a case in which you'd really want full transparency?

As they are always 'filled' from a user interaction POV (i.e. tapping on edge or interior) perhaps not. pseudo transparency could be achieved by using a very pale fill colour perhaps, that would be then made 68% opacity by Collect.

@LN

I don't have a great workaroud for you but will think about it.

It would probably be easier if I just regenerated my tiles with not-red details!

Thanks for all the other clarifications!

Edit: Happy birthday @LN !

[ahblake]

@mathieubossaert

But not in a dynamic way (I mean not with a test that defines the style to apply). Anyway this is a first workaround and the end user don't care about how it's done

You could utilise entities to write a new colour to the stroke/fill fields upon completing a record? The beta offline entities functionality would reflect this on device while disconnected, and on other devices once synced. Perhaps it goes from colour A (visited = false) to colour B (visited = true, but data not reviewed/approved) to colour C (visited = true).

[LN]

@ahblake

Happy birthday @LN !

Aw, thanks! I feel very lucky to know people from around the world -- I get an extra long birthday!

@ahblake

You could utilise entities to write a new colour to the stroke/fill fields upon completing a record?

Absolutely. I really do hope that one day we can come up with a reasonable way to define the condition-to-color mapping somewhere so that only one property needs to be updated. Ideally we could do this for tables of entities as well. Lots of exciting challenges ahead of us!

[mathieubossaert]

@LN

Yes, that would be nice, wouldn't it? It's a concept we're interested in but the spec will be quite different from anything we have now.

Oh yes it would be so nice !

[seewhy]

An almost unnecessary test of the 'Stroke' style - because others have managed to get it working.

Confirming that I can use this feature from a csv with the column 'stroke'

I have just (this week) trained a whole bunch of staff with the promise that this feature is coming soon! It is REALLY useful to be able to see different lines clearly, and defined by (stroke) colour.

Thanks for implementing this feature - in the current version we just have red lines on the map and lots of head scratching!

I had been hoping the Offline layers loading feature might be included in this release - that same bunch of people can't load their base maps because their devices are locked... But then, there's always something to look forward to next time...

Running on an old Sony Z5 with Android 7.1 - a test device composting in my desk drawer

[LN]

It's great to see a screenshot of how you're planning to use the new functionality, thank you!

@seewhy

I had been hoping the Offline layers loading feature might be included in this release

That's the plan. We wanted to get this first beta out to get initial feedback and catch bugs like the issue @mathieubossaert identified with French dates and will follow up shortly with another beta including the mbtiles selection.

@Grzesiek2010 and @seadowg, I want to make sure you've seen the excitement in this thread!

[LN]

We've just released a new beta that adds the ability to import mbtiles from device!

We had one surprise along the way which is that we can't easily filter by file extension so we can't show only mbtiles files in the file picker. For now, all files can be selected but any that are not mbtiles will be ignored without feedback. We'll work on adding that feedback.

You should be able to add multiple mbtiles at once but the file picker interface to do so will depend on your device. Generally you will need to tap the icon for the file, not the filename.

We haven't added deletion yet and will be working on that next week. If you have any other feedback on the functionality, we'd love to hear it.

[ahblake]

Testing this imminently!

Edit: Tested!

I opened a project, went into Settings / Maps, tapped on Layer (toast is mostly hidden at bottom of screen in landscape mode, have to drag up, reported similar UI issue previously). (I also created a submission and opened a geowidget then used the layer picker, process was the same.)

Scrolling to the bottom gave me the 'Add layer' button, and on a Galaxy Tab S8+ / Android 14 / One UI 6.1 the system picker let me navigate to the location (SD card in a USB dock), open the folder, use the ⋮ button to select all, then tap Select and they were added.

If you don't select a layer to add, you have to use the < button repeatedly to go up to the top level and then the last tap will return to the select toast. I can't see a quicker way to bail out.

Separate to adding layers, after selecting a layer when the list of layers is too long to fit on screen, it jumps to the top, so you have to scroll down again to check that it's selected.

Also, layers now appear to be sorted in order of date added, definitely not A-Z anymore, which makes finding the layer tricky when there are many and they are added at different times.

Added layers can be for project only or all projects. This is a really nice UX improvement, and is handy to quarantine tilesets. Was this function always available (the layers folder under projects\projectID\) - it's not mentioned in the docs for mbtiles?

[Grzesiek2010]

Hi @ahblake
Thanks for testing and sharing your thoughts!

@ahblake

I opened a project, went into Settings / Maps, tapped on Layer (toast is mostly hidden at bottom of screen in landscape mode, have to drag up, reported similar UI issue previously). (I also created a submission and opened a geowidget then used the layer picker, process was the same.)

This is the default android behavior. As you have noticed it works in a similar way in other parts of the app as well. We will consider changing it.

@ahblake

If you don't select a layer to add, you have to use the < button repeatedly to go up to the top level and then the last tap will return to the select toast. I can't see a quicker way to bail out.

You are describing your experience with the Android file picker I guess? It's the built-in file picker (not our implementation). It might work in a different way depending on your Android version but it's not something we can configure.

@ahblake

Separate to adding layers, after selecting a layer when the list of layers is too long to fit on screen, it jumps to the top, so you have to scroll down again to check that it's selected.

Yeah, it shouldn't work like that. We will try to improve it.

@ahblake

Also, layers now appear to be sorted in order of date added, definitely not A-Z anymore, which makes finding the layer tricky when there are many and they are added at different times.

Good catch we will fix this.

@ahblake

Was this function always available (the layers folder under projects\projectID\)

It has been available for over 2 years now. You are right it looks like it's not described in the docs. We will fix this.

[seewhy]

@ahblake

Edit: Tested!

What he said!

I've tested this on Android 7 using Shelter App (work profile with a separate Google account) - it is capable of loading tiles from the personal profile folders, which is a bonus

Being able to order the tiles by date or name would be nice (best not to assume we are using the most recently added, so A-Z might the best default)

Also confirming that the dialogue is accessible and works from Project Settings and within the map layer button of a geo-widget.

This is a fantastic step forward for locked devices. Thank you for your hard work on this, great for it to nearly be at my fingertips!

[ahblake]

Just noticed something when I couldn't get an added layer.

When selecting a layer via the map widget, only the first 99 tilesets are shown in the list to select from, plus 'none' at the top, so a limit of 100 choices. The two most recent ones that I added were not available (due to sort by date added).

When selecting a layer from the Project Settings, they are there, but there are only 86 choices (plus 'none'), and a bunch of tiles that are visible via the map widget aren't present here.

If I select my desired tileset here it is used in the map view, but if I change tiles I have to go back to project settings to select it again.

Using the Files app and looking in the ODK layers folder (i.e. available to all projects) and the project layers folder (i.e. available to this project only), there are 83 common and 3 in the project... or 86 total, which matches the Project Settings list.

Looking in a different project folder ("Project A"), there are another 16x tilesets, so 83 common + 16 = the 99 that I can see via the map widget in Project B, and the 3x in the Project B folder aren't visible.

Going into Project A, the 3x layers in Project B's layers folder aren't visible via widget layer selector or project settings selector, but this might also be due to the 100x item cap excluding them.

I think there's possibly a couple of issues here

1. Project A has tilesets in its layers folder but these are accessible to Project B via the map widget. (But not via Project Settings)
2. There is a limit to the number of tilesets that can be displayed to select from,100 total, or 'none' plus 99x.

Edit: another issue that just appeared, in the non beta release (haven't replicated it on beta yet), I copied over a large tileset (324mb), it was working, then a little later the layer disappeared, when trying to select it, it wasn't present anymore. Looking in the files app, the extension had .corrupt appended to it. I replaced the file and the same thing happened again.

[LN]

@ahblake

Project A has tilesets in its layers folder but these are accessible to Project B via the map widget. (But not via Project Settings)

This sounds like a bug, we'll look into it.

@ahblake

There is a limit to the number of tilesets that can be displayed to select from,100 total, or 'none' plus 99x.

Is using more than 100 offline layers an actual requirement that you have? I assume so giving that you're trying it out. We did decide to target a smaller number of files at least initially to reduce some complexity.

[seadowg]

@ahblake

Project A has tilesets in its layers folder but these are accessible to Project B via the map widget. (But not via Project Settings)

That's fixed and should be in the next beta.

@ahblake

There is a limit to the number of tilesets that can be displayed to select from,100 total, or 'none' plus 99x.

There's not an intentional limit (although it'd still be good to hear if you're actually going to be dealing with that many). We've made some changes to that screen however, so let us know if it's still a problem in the next beta.