# Use AI to do (species) recognition on photos **URL:** \ **Category:** Ideas\ **Tags:** odk-central\ **Created:** [September 4, 2026, 2:43pm UTC](https://forum.getodk.org/t/use-ai-to-do-species-recognition-on-photos/58540 "2026-09-04T14:43:45Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Edmonds](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/edmonds/32/31942_2.png) [@Edmonds](https://forum.getodk.org/u/Edmonds)\ **Post date:** [September 4, 2026, 2:43pm UTC](https://forum.getodk.org/t/use-ai-to-do-species-recognition-on-photos/58540/1 "2026-09-04T14:43:45Z")
Could we use AI to recognize for example species on photos? ObsIdentify is a great example. It however works online only (and is only for the Benelux countries). Especially for the landuse sector running AI on a photo database could help to map biodiversity. It could run on ODK Central and give back a/some results for every photo it scans(?). ---
**Author:** ![yanokwa](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/yanokwa/32/1695_2.png) [@yanokwa](https://forum.getodk.org/u/yanokwa)\ **Post date:** [September 4, 2026, 5:57pm UTC](https://forum.getodk.org/t/use-ai-to-do-species-recognition-on-photos/58540/2 "2026-09-04T17:57:45Z")
@LN shared an approach to do something similar with audio, where you use an API integration to process submissions with recorded conversations and you update those submissions with the transcriptions. You can do something very similar with photos... > [@Python script to update submissions of a form with an AI-generated transcript of its audio attachment(s)](https://forum.getodk.org/t/python-script-to-update-submissions-of-a-form-with-an-ai-generated-transcript-of-its-audio-attachment-s/41338): > > I've been playing a bit with OpenAI's [Whisper transcription library](https://github.com/openai/whisper). The open source library is relatively small and can be run on simple hardware. It's highly accurate for English even with relatively small models. It can also do translation and there's good information on how likely it is to be correct for each language on the Github page. I put together a small script that can be used to update submissions of a form with a transcript of an audio file from that submission. This approach would… ---
**Author:** ![Edmonds](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/edmonds/32/31942_2.png) [@Edmonds](https://forum.getodk.org/u/Edmonds)\ **Post date:** [September 4, 2026, 7:36pm UTC](https://forum.getodk.org/t/use-ai-to-do-species-recognition-on-photos/58540/3 "2026-09-04T19:36:45Z")
@yanokwa , interesting. Thanks for the link. ---
**Author:** ![Edmonds](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/edmonds/32/31942_2.png) [@Edmonds](https://forum.getodk.org/u/Edmonds)\ **Post date:** [September 4, 2026, 7:39pm UTC](https://forum.getodk.org/t/use-ai-to-do-species-recognition-on-photos/58540/4 "2026-09-04T19:39:50Z")
@yanokwa , see also Merlin Bird ID app to recognize bird sounds. Also AI based. Works also extremely good. Interesting to explore these options with AI to label sounds and imagery on a database. ---
**Author:** ![punkch](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/punkch/32/32944_2.png) [@punkch](https://forum.getodk.org/u/punkch)\ **Post date:** [September 11, 2026, 8:32pm UTC](https://forum.getodk.org/t/use-ai-to-do-species-recognition-on-photos/58540/5 "2026-09-11T20:32:44Z")
Totally doable offline if you want to shed some blood, sweat and tears over the solution. 🤘 Fair warning: what follows is a theory, not something I have shipped to the field. It is how I would attack it, and I think the pieces exist, but expect to hit snags along the way. The server-side route above works well when submissions reach Central. If you need the answer _in the form, in the field, with no signal_, you could do the recognition on the phone. The trick would be not to put the model into Collect at all, but to build a small **sidecar app** and let Collect call it. **Why not "every species on Earth"?** There are about 2.1 million described species (roughly 1.5 million of them animals), and estimates of the true number run to 8 million or more. No offline model on a phone is going to tell that many apart. A classifier with millions of labels would be huge, slow, and mostly wrong, because thousands of species look near-identical in a phone photo. Even the big online services (ObsIdentify, iNaturalist) narrow by region for this reason. So I would aim for a **specialized model** : one that only has to choose between the 50 to 500 species that can actually occur in your survey area. That is a much easier question, and it is the one the field worker is really asking. **The good news: specializing might just be naming the species** With the model I have in mind, specialization would not be training. It would be a text file. You list the species you expect to see, the model turns each name into a vector once, and from then on it only compares photos against those names. Change the list, re-run one script, done. No dataset, no GPU, no retraining. That is the part that makes me think this is feasible for a small team. **How it could fit together** 1. Collect already has the `ex:` appearance for text questions. It launches another app by intent and takes back whatever that app returns. 2. The sidecar app would open the camera, run the model on the photo, show the top 5 guesses, and let the field worker tap one. 3. That name comes back into the ODK question. A second field could carry the confidence score. The XLSForm side would be one line: | type | name | label | appearance | | --- | --- | --- | --- | | text | species | Species | `ex:org.example.speciesid.CLASSIFY(score=/data/score)` | | text | score | Confidence | | Collect could even pass the list per form: `ex:...CLASSIFY(species=/data/list)` where `list` is a calculate built from a "which site?" question, so one app would serve several regions. **The model I would try first** [BioCLIP 2](https://huggingface.co/imageomics/bioclip-2) (MIT licence, trained on ~214M organism photos). It has two halves: an image encoder and a text encoder. Both turn their input into a 768-number vector, and a photo matches the species name whose vector is closest. - The text encoder would run **once, on a laptop** , for every species on your list. The result is a tiny file (3 KB per species). - Only the image encoder would go on the phone, exported to ONNX and quantized to int8. Roughly 300 MB, probably 2–5 s per photo on a mid-range phone, likely needs 4 GB RAM. ONNX Runtime would do the inference. Accuracy should follow the list size. On the model card's own benchmarks with thousands of species it is ~55–75% top-1 depending on the group. With a regional list of a few hundred I would expect something in the 85–95% range, and a top-5 picker should catch most of the rest. That last number is my guess, not a measurement. If you have your own photos (camera traps, previous surveys), there could be an optional fine-tuning step on top: compute embeddings of your photos, nudge the name vectors toward them, and with 150+ photos per species let the last few layers of the encoder adjust too. Same output files, same app. If 300 MB is too heavy, the original BioCLIP (ViT-B/16) would drop to ~90 MB and under 1 s per photo at some accuracy cost. **Honest caveats** - It is a research model. It will be wrong sometimes. Keep the human in the loop. - Collect cannot hand its photo to another app (private storage), so the sidecar would take its own photo. Keep a normal `image` question next to it for the archive copy. I am not an Android developer so there could be a better solution for this. - The model file would have to be copied to the phone once by USB; it is too big for a normal APK. - Nobody has proven the 2–5 s figure or the accuracy on real survey photos. Treat both as hypotheses to test on one phone before planning around them. I generated [this step-by-step](https://claude.ai/public/artifacts/7e204fee-5a95-438d-9901-47af63a82c1d) guide (species list → export the model in Colab → Kotlin sidecar app → ODK form → optional fine-tuning) for people with no Android background and to explain the theory in more detail, but again, untested end to end. ---
**Author:** ![Edmonds](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/edmonds/32/31942_2.png) [@Edmonds](https://forum.getodk.org/u/Edmonds)\ **Post date:** [September 21, 2026, 9:17am UTC](https://forum.getodk.org/t/use-ai-to-do-species-recognition-on-photos/58540/6 "2026-09-21T09:17:29Z")
Hi @punkch , Thanks for. sharing these insights. I recently participated in a hackathon with this idea and one of the outcomes was that we process our photos on the server with an API to a model (Bioclip/@Planet) running on a seperate server. The Bioclip model indeed showed the best results, however, far from flawless, what was also caused by the quality of the photos. Moreover, these models need more context, for example, region, landscape, etc to increase accuracy. I will check and consider your solutions too. Thanks again.