# pyODK: Using cursors to efficiently pull new data only

**URL:** <https://forum.getodk.org/t/pyodk-using-cursors-to-efficiently-pull-new-data-only/39750>\
**Category:** Development\
**Created:** [December 1, 2022, 2:56pm UTC](https://forum.getodk.org/t/pyodk-using-cursors-to-efficiently-pull-new-data-only/39750 "2022-12-01T14:56:32Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![chrislrobert](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/chrislrobert/32/1613_2.png) [@chrislrobert](https://forum.getodk.org/u/chrislrobert)\
**Post date:** [December 1, 2022, 2:56pm UTC](https://forum.getodk.org/t/pyodk-using-cursors-to-efficiently-pull-new-data-only/39750/1 "2022-12-01T14:56:32Z")

</div>

Thanks for releasing pyODK to facilitate Python-based workflows, for the helpful webinar, etc. I'd like to integrate ODK support into the Python [_surveydata_ package](https://github.com/orangechairlabs/py-surveydata/), and have a quick question about the best approach to cursors: if I want to pull only new or updated submissions for a form, would the best approach be to use SubmissionService.get\_table()'s _filter_ parameter with _createdAt_ and/or _updatedAt_? Are there any examples out there I might look at? Other approaches you'd recommend?

This _surveydata_ package is new and evolving, but the idea is to provide some abstraction between survey platforms and storage systems, and to allow for efficient synchronization across the two. Any advice on the best way to integrate ODK would be welcome, but I think it should be pretty easy to start with cursor-based data sync.

Thanks very much,

Chris

---

<div class="post-metadata">

**Author:** ![LN](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/ln/32/1605_2.png) [@LN](https://forum.getodk.org/u/LN)\
**Post date:** [December 5, 2022, 5:06am UTC](https://forum.getodk.org/t/pyodk-using-cursors-to-efficiently-pull-new-data-only/39750/2 "2022-12-05T05:06:24Z")

</div>

> [@chrislrobert](#):
>
> use SubmissionService.get\_table()'s _filter_ parameter with _createdAt_ and/or _updatedAt_

Yes, that’s likely your best bet. There is one issue to be aware of: currently saving a timestamp and using it in a subsequent query with gt (greater than) will sometimes include duplicates that make it look like Central is doing a greater than or equal comparison. That’s because the database is storing higher precision timestamps than used in OData. This probably isn’t an issue if you’re dealing with submission updates but I did want to mention it because it’s surprising. We’ll likely have this patched by the end of January.

If you’d prefer to have data in a format other than JSON, that would be helpful to know. The CSV endpoint uses the same filtering. That would have to be accessed using pyodk’s raw HTTP verb methods. Central doesn’t currently provide filtering on the submission list endpoint but you could use the OData response to get submission ids and then request the raw XML for each submission if you wanted.

---

<div class="post-metadata">

**Author:** ![chrislrobert](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/chrislrobert/32/1613_2.png) [@chrislrobert](https://forum.getodk.org/u/chrislrobert)\
**Post date:** [December 6, 2022, 12:50pm UTC](https://forum.getodk.org/t/pyodk-using-cursors-to-efficiently-pull-new-data-only/39750/3 "2022-12-06T12:50:37Z")

</div>

Okay, great, thanks! And I appreciate the warning re: potential duplicates, but I plan to use greater-or-equal with explicit duplicate handling anyway, out of an abundance of caution (in case the previous request happened to end with one but not all submissions with matching timestamps). JSON format also works well for the storage systems and pulling into Pandas, etc.

Thanks again,

Chris