Payload string too long error in Central with central-webhook installed

[renato_cava]

Hi ODK team and community,

1. What is the issue? Please be detailed.

After a network outage that lasted around 6 hours, our ODK Central server entered into a loop with this recurring error:

!! unexpected worker loop error: error: payload string too long
at Parser.parseErrorMessage (/usr/odk/node_modules/pg-protocol/...)
at ... new_audit_log() line 40 at PERFORM pg_notify('odk-events', js::text)>

The submissions are still being saved correctly in the database, but entities are not being created or updated, likely because the internal event system is blocked by an oversized payload.

We don’t have unusually large forms.

This seems like a reasonable workaround, but we would like to clarify a few things before attempting any manual intervention:

Has anyone else experienced this exact error after extended downtime or high submission activity?

Is there a safe way to flush the pending queue of events manually in PostgreSQL or ODK Central?

Is it recommended to patch the new_audit_log trigger or wait for a core-level fix?

Any guidance or shared experience would be greatly appreciated!

Thanks in advance.

2. What steps can we take to reproduce this issue?
It's hard to reproduce precisely, but this is what happened in our case:

The ODK Central server experienced a network outage (or firewall block) for approximately 6 hours.

During this time, submissions were still made by field teams and stored locally by ODK Collect.
Once the connection was restored, a batch of submissions was sent in quick succession to the Central server.
After this, the Central service started logging !! unexpected worker loop error: payload string too long repeatedly.
Submissions are stored in the PostgreSQL database, but entities are no longer being created or updated.

3. What have you tried to fix the issue?

Restarted all services using docker compose.

Removed orphan containers and verified all volumes and container states.

Confirmed the issue is in pg_notify inside the new_audit_log() function.

Reviewed logs to confirm that the error persists even after restart.

Checked that form submissions are saved correctly but entity creation is not triggered.

Avoided altering the new_audit_log trigger for now, pending guidance from the community.

4. Upload any forms or screenshots you can share publicly below.

[yanokwa]

From the error log, it looks like an issue with central-webhook. That's a tool made by @spwoodcock and I've tagged him so he can help.

[renato_cava]

Thanks @yanokwa for the response,

You're right that the error occurred while central-webhook was active, but I’d like to clarify that the issue doesn't seem to originate exclusively from the webhook service.

What we observed is that the error happens after a temporary disconnection of network or and database outage. Once connectivity is restored, a large number of accumulated events (like submission.create) are processed rapid succession (maybe one batch). During that burst, we saw failures related to the new_audit_log_trigger.

Disabling the new_audit_log_trigger temporarily resolved the issue, and everything worked as expected afterward.

So...

It’s important to note that central-webhook itself doesn’t create entities, so these events are generated by ODK Central, and the problem seems to occur when many of them are processed at once. Our current hypothesis is that the audit trigger fails under high load or rapid event execution.

We’ll keep investigating and would appreciate any advice from @spwoodcock or others on how to handle this kind of scenario without turning off the audit logging.

Thanks again for your support.

[spwoodcock]

Hey @renato_cava - thanks for describing the issue in so much detail.

This is definitely related to central-webhook (an unofficial webhook service I made), so I'm not sure it belongs on the ODK forum.

I'm sure its related to https://github.com/hotosm/central-webhook/issues/9

So we have 3 audit events that cause the postgres trigger to NOTIFY. By default all three trigger the processing function, but it would be much nicer to constuct the trigger events used based on the ones enabled by the user instead.

The submission.create event was a bit untested and tacked on at the end. Unfortunately it causes the payload size issues linked in the issue you made on the repo & here.

What we need is:

• Me to have more time to work on this and cover more scenarios / fix the submission.create event for large forms (can't make any promises here)
• Another contributor to raise some PRs to fix this

Lets continue any discussions via the github repo issues