Dear all,
With reference to the last Insiders Call where I had the chance to present the approach of CABOZ in managing agricultural supply chains by using entities, I'm happy to share more details especially with reference to how this approach helps us to generate reliable data for regulations such the EUDR (European Union Deforestation Regulation).
EUDR is a major driver for digitalisation in many affected supply chains, but reliable base data about farmers, farms and product traceability are often not available. ODK can play an important role for farmer allied organisations to get compliant with this and upcoming regulations.
Project design
We conceptualise the forms and the role of entities as a link between them, similar to the design of a relational database, and try to normalise information using entities to avoid redundancies as much as possible. This way, we achieve a coherent data system. Below is an early sketch of the different areas we wanted to cover.
We create forms tailored to our needs to collect information in the following areas:
- Geography: countries, regions, sections, villages, …
- Organisation: personnel, logistics, warehouses, prices, …
- Agronomy: farm, fields, products, interventions, …
- Traceability: Purchase, transport, stock, sales, …
- Social: household members, workers, …
If the data is rather static (like countries or regions), we use attached CSV files. If the data depends dynamically on the local context, we use forms to create and update entities (like farmers, fields, personnel, sections, villages) to give partner organisations the flexibility they need for their daily work.
At the centre of this form hierarchy are the forms to create and update farms (with their code and basic information about the farmer) and the fields (with their code and geospatial information). Based on these basic forms, we can now attach multiple other forms to cover areas like household surveys, on-farm agronomic interventions, product traceability, …
Use existing data
In most cases we work with farmer allied organisation such as cooperatives who already work with an excel based dataset of farmers and fields. To ensure that the digitalisation process is well received by employees and to avoid duplicate work, it is important to develop forms able to retrieve data from this source as attached CSV files.
We generally also integrate existing code systems to uniquely identify farmers and fields. In the interest of a positive perception of the digitalisation of workflows users should feel “at home”.
Avoid duplicates
It is important to integrate extensive checks for duplicates such as internal codes or checking if, as an example, a person with the same name and living in the same village is not already registered with another code.
Avoid hard coding
We try to avoid hard-coded forms as much as possible to enable our partners, who have no knowledge of XLS forms, to customise basic data as much as possible to their needs. An example is a dedicated form to register villages of a given area. This ensures a uniform spelling and easier data analyses.
Forms should give enumerators feedback
We often pull data from entities to let enumerators see, what data is already available. Per example if an enumerator wants to register a new field of a given farmer, the form will display all the already registered fields with some relevant information. If enumerators want to update entities such as a field, we give them the possibility to chose the field either by its code or on a map. All this helps to have enumerators generate quality data on an informed basis.
Make life easy for enumerators and office staff
As an example, our forms calculate the productivity of fields based on different parameters directly in the form. Also the value of purchases are calculated, depending on price, weight and quality assessments. This generates an immediate feedback and simplifies the lives of the users. It also enables them to inform partners like farmers about the findings to support them to take informed decisions.
Making geospatial data compatible with EUDR requirements
When geospatial information such as polygons and coordinates are collected, we often run into technical issues when feeding the raw data into platforms to check for technical conformity and deforestation activities. Most common issues are:
- 6 decimal places for all geospatial data
- no self-intersecting polygons
- no overlapping polygons
- no duplicates
Since ODK isn't capable yet to resolve such issues and since it would be tedious to clean up all these data by hand in QGIS or similar applications for thousands of datasets, we developed two scripts for QGIS to 1) clean up geometric errors in polygons (mainly self-intersections) and 2) get rid of overlaps between polygons below a user defined percentage as threshold, but leave larger overlaps as they are, assuming that fundamental issues have to get resolved by hand (such as duplicates) or to be verified in real life (such as land-ownership issues).
After modifying the polygons the script recalculates the surface and its geographical centre.
We update the dataset (the entities) on ODK Central with the corrected CSV file with pyodk to make the new geography available to enumerators next time they visit the same field.
Below is a screen video of the process in QGIS. It takes just a few minutes to clean about 1500 polygons.
Develop forms that help to resolve real life challenges
Often our enumerators are faced with real life challenges such as changing ownership of fields. We developed forms able to not only change the farm linked to a field, but also adapt its internal code to this new ownership while maintaining its UID to preserve data integrity and historisation. The changes are also registered as an entity to have enumerators see the changes made in the past.
Track product flow with ODK
We track product traceability with different forms to track incoming products and movements between warehouses and finally clients.
Incoming products are first registered in repeat groups when the are bags weighed together on a balance. A picture of the display of the balance is taken for countercheck. In a second phase all corresponding paper receipts of purchases with farmers are registered in the same form. The difference in the weighed weight in combination with the quality assessment and the total weight and value of the purchased weight is automatically calculated and displayed to the user so that they can make informed decisions immediately.
For those interested I attach some screen videos:
Farm field registration and modification
As an example I hope this form for the farm-field registration gives some further technical insights.
a1_unites_agricoles.xlsx (510.1 KB)
Thanks to the great and supportive ODK team and to all who took the effort to answer my questions during the last few months on the forum.
Just reach out if you need further information or support.
Best, Daniel