Using external AWS S3 bucket in region other than us-east-1

Development
1

Preamble

  • Central requires config to upload blob media to an external S3 bucket:
    S3_SERVER=https://service.com
S3_ACCESS_KEY=MY_ACCESS_KEY
S3_SECRET_KEY=MY_SECRET
S3_BUCKET_NAME=MY_BUCKET
  • This mode of authentication, using an access key / secret key pair is not security best practice.
  • I have configured IAM role-based auth from my EC2 instance to the S3 as follows.
  • Understandably, directly connecting like this does not work with the Minio SDK (requires AWS CLI). But it's possible to connect using temporary credentials generated on connection.
  • As ODK Central cannot use S3 without having S3_ACCESS_KEY and S3_SECRET_KEY set, this approach would require adding dummy values, then using a middleware Lambda to generate temporary credentials on connection, relaying the S3 response back to Central.
  • It's a bit too complex to do this for my use case (it's not extremely sensitive media uploads), so I just decided to go against best practice and generate access/secret key credentials for an IAM user.

Issue

  • With the above out the way, I have configured an S3 bucket in region af-south-1 for connecting to Central, with the Central EC2 instance in the same region.

  • Upon connection (using the non region specific URL https://s3.amazonaws.com as advised in the docs), I get an error:

    S3Error: The af-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.
    at parseError (/usr/odk/node_modules/minio/dist/main/internal/xml-parser.js:35:13)
    at Object.parseResponseError (/usr/odk/node_modules/minio/dist/main/internal/xml-parser.js:82:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Client.makeRequestStreamAsync (/usr/odk/node_modules/minio/dist/main/internal/client.js:476:19) {
  code: 'IllegalLocationConstraintException',
  requestid: 'D4MEBFK1GXE8J12Y',
  hostid: '7AkjYY7VYZH9A2yKAFO6Ne1EL1/AKcU80CL7FGOvReOp+ReDy3AlXRiZs9BCHTQMUjPq0Z0iLOc=',
  amzRequestid: 'D4MEBFK1GXE8J12Y',
  amzId2: '7AkjYY7VYZH9A2yKAFO6Ne1EL1/AKcU80CL7FGOvReOp+ReDy3AlXRiZs9BCHTQMUjPq0Z0iLOc=',
  amzBucketRegion: 'af-south-1'
}

  • If I attempt the region specific URL https://s3.sf-south-1-amazonaws.com, then I get a different error:

    S3Error: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'af-south-1'
    at parseError (/usr/odk/node_modules/minio/dist/main/internal/xml-parser.js:35:13)
    at Object.parseResponseError (/usr/odk/node_modules/minio/dist/main/internal/xml-parser.js:82:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Client.makeRequestStreamAsync (/usr/odk/node_modules/minio/dist/main/internal/client.js:476:19)
    at async Client.getBucketRegionAsync (/usr/odk/node_modules/minio/dist/main/internal/client.js:526:19)
    at async Client.makeRequestStreamAsync (/usr/odk/node_modules/minio/dist/main/internal/client.js:447:25) {
  code: 'AuthorizationHeaderMalformed',
  region: 'af-south-1',
  requestid: '8S4ZCAANYKKKJEVW',
  hostid: 'tpcMWQj6s0rTzOXt1GifWdGLYp6U/t0GQSak6TU5J890VHXC4weJw5uwQd6t1/9L6CoUJ8YodCw=',
  amzRequestid: '8S4ZCAANYKKKJEVW',
  amzId2: 'tpcMWQj6s0rTzOXt1GifWdGLYp6U/t0GQSak6TU5J890VHXC4weJw5uwQd6t1/9L6CoUJ8YodCw=',
  amzBucketRegion: undefined
}node:internal/process/promises:391
    triggerUncaughtException(err, true /* fromPromise */);

  • I attempted modifying the code to add the region param 'region': 'sf-south-1' to no avail.

Question

Is there a config option I am missing to specify the region?

Is disallowing any region other than us-east-1 for AWS expected behaviour?

Offer To Assist

  • If other people also face this issue, please comment to let us know!
  • If this is a genuine bug, and it would help me debugging further to find the cause, I am more than happy to PR.
  • For now, I plan to work around this by simply creating an S3 bucket in us-east-1.
2

One of our goals was to support non-AWS blob storage and the tradeoff seems reasonable for most users of ODK. The important thing is then to make sure that access to your server is strictly limited.

AWS writes in when to create a user instead of a role:

Third-party AWS clients – If you are using tools that don’t support access with IAM Identity Center, such as third-party AWS clients or vendors that aren't hosted on AWS, use IAM user long-term access keys.

minio detects the region and sets it for subsequent calls and we have tested this with other regions. What I think is going on is that it uses the global endpoint which isn't supported by regions launched after 2019. We were hoping we wouldn't have to have an explicit configuration item for region but it looks like we may have to for those regions. We will look into it and let you know what we find.

1 Like