Use pyODK to upload and publish media files?

seewhy · March 7, 2023, 3:55pm

I realise that I'm trying to fly before I can crawl, but I'm wondering if there is a way of updating and publishing media files using pyODK and/or via the API? I can't see anything documented, but it would be a great way for me to be able to automate a work-flow.

In my case I have a select_one_from_file (external CSV dataset) and the list frequently needs to be updated - it relates to 'live' projects where we are using the same form for different sites / clients. Having a consistent name for the project / site between enumerators helps to filter the dataset!

At present may work flow looks something like: update / create the csv (which I actually export from our projects database), log into Central, navigate to the form, create a new draft, drink coffee, upload the csv and publish the draft. Central then propagates the new version to the devices... doesn't sound much, but being able to do that more simply / quickly would be very welcome - and save the complaints about the list not being up to date!

I think this might also be useful for updating entities that have not been created by ODK? We often get given datasets and are asked to populate / audit: at present I'm using geopoints and basemaps to link data through select_one from map and when traces and polygons functions come on stream (no pressure ) that will expand the possibilities. I am trying to avoid duplicated forms on Central, so will be looking for new ways of streamlining data collection... so an easy way of updating the external dataset would be a benefit again - hopefully not just to me...

Thanks...

aurdipas · March 9, 2023, 1:20pm

github.com/getodk/pyodk

Add client.forms.update to publish form updates

getodk:master ← lognaturel:update-ln

opened 11:12PM - 15 Feb 23 UTC

lognaturel

+460 -3

Based on #25. It just had a couple of changes left to make and I'm eager to get …this out and published. Noteworthy changes this makes: - Removes the ability to do an update that only bumps the version. There's no reason to do this so it points to an application error and an exception is helpful. - Removes `version_updater`. Maybe it's a nice affordance but we can add it if it comes up. Maybe just specifying a version as opposed to a transformation function would be preferred. Maybe nobody cares. - Removes the version return. It's not clear what it would be used for. To make it complete we'd want to get the version from the response for the case where only the form definition is updated. It seems like complexity without clear payoff. We can add a return value if we see a need for it. - Ignore XLSForm warnings. At that point they're not actionable. Warnings can always be safely ignored. #### What has been done to verify that this works as intended? Added tests of the high-level flow of what update does. I think those effectively document what is intentional about the implementation and cover most interesting logic. #### Why is this the best possible solution? Were any other approaches considered? I think this addresses user needs in the most minimal way leaving various options to expand functionality as needed. #### How does this change affect users? Describe intentional changes to behavior and behavior that could have accidentally been affected by code changes. In other words, what are the regression risks? No regression risk, all new. #### Do we need any specific form for testing your changes? If so, please attach one. No. #### Does this change require updates to documentation? If so, please file an issue [here]( https://github.com/getodk/docs/issues/new) and include the link below. Readme updated. #### Before submitting this PR, please make sure you have: - [x] included test cases for core behavior and edge cases in `tests` - [x] run `nosetests` and verified all tests pass - [x] run `python bin/pre_commit.py` to format / lint code - [x] verified that any code or assets from external sources are properly credited in comments

client.forms.update is what you are asking.

github.com

getodk/pyodk/blob/master/pyodk/_endpoints/forms.py

import logging
from datetime import datetime
from typing import Any, Callable, Dict, Iterable, List, Optional

from pyodk._endpoints import bases
from pyodk._endpoints.form_draft_attachments import FormDraftAttachmentService
from pyodk._endpoints.form_drafts import FormDraftService
from pyodk._utils import validators as pv
from pyodk._utils.session import Session
from pyodk.errors import PyODKError

log = logging.getLogger(__name__)


# TODO: actual response has undocumented fields: enketoOnceId, sha, sha256, draftToken


class Form(bases.Model):
    projectId: int
    xmlFormId: str

This file has been truncated. show original

seewhy · March 9, 2023, 2:01pm

Oh wow! Thank you for interpreting my suggestion, and making the link to how it's being worked on.

I do feel a bit like some parallel evolution has gone on here, looking @LN 's comments within the pull request:

"Script a form attachment update. This is useful for synchronizing with an external data source or updating entities based on form submissions that have come in. In that case the form version generally doesn't matter (the definition isn't changing anyway)."

Honestly, I hadn't read that (and wouldn't have posted here if I had!)

So, note to self - looks like github is the place to look when thinking about pyODK. I missed the single line change in README.md that is the key to this: "update: Create a new version of an existing Form" - I'm still not sure I would have made the intellectual leap, but its really obvious now you mention it

As always, thanks for your tolerance to stupid questions / ideas, and for pointing me in the right direction.

I'll just need to work out how to use it now...

seewhy · March 22, 2023, 10:29am

I thought it might be helpful for folk trawling the forum to have the example of the solution - so it's all in one place...

Thanks to the signposting by @aurdipas I was able to create a python script to update a particular form with a csv file based on the information in pyodk docs at GitHub... I had to update my version of pyodk so that it has the latest functions ( pip install --upgrade pyodk )

Note that I've referenced one project on the server - it seemed to be a sensible thing to do, but maybe that's not necessary. Happy to be given feedback on the script if it isn't particularly elegant!

from pyodk.client import Client

with Client(project_id=2) as client:
    client.forms.update(
        form_id="myformID",
        attachments=["\Path\to\my\file\ODKprojects.csv"])

I can now set this as a script to run in a single click - so much more likely to have the latest project list on the server. Unfortunately, it probably means fewer excuses to drink coffee...

Thanks for your help and for the work to make this happen.