Files larger than ~256MiB are failing to upload to s3, stopping the whole process

Support
1

1. What is the issue? Please be detailed.
When attachments larger than 250MiB are uploaded to Central, and later then s3 upload scheduled jobs runs, it stops with the following error: Cannot create a string longer than 0x1fffffe8 characters

2. What steps can we take to reproduce this issue?

  1. Set up self hosted ODK Central with S3 for file storage
  2. Raise the default nginx body limit from 100m to e.g. 300m
  3. Upload large files (via ODK Collect, Enketo won't allow you)
  4. Either wait for the cron job, or run node lib/bin/s3.js upload-pending yourself

3. What have you tried to fix the issue?
We ended up chunking files and sending chunks in separate form fields.

4. Upload any forms or screenshots you can share publicly below.
Anything with file input.

More details:

  1. the encoding is utf-8 : https://github.com/brianc/node-postgres/blob/master/packages/pg-protocol/src/buffer-reader.ts#L7
  2. I was confused how can it be utf-8 given that it's reading blobs
  3. Turns out this lib is using postgres wire protocol in text mode
  4. Looks like postgres encodes binary data in text format as 2 hexadecimal digits per byte
    0x1fffffe8 is 536870888 in decimal, this is max characters supported by node's buf.toString
  5. We need two characters to encode one byte, so 536870888 / 2 = 268435444 B ~= 256MiB
1 Like
2

Proposed solution: Proposal: better large attachments handling via presigned upload urls for S3