OIDC issue when logging in

brian_252 · April 29, 2024, 8:03pm

git clone from today (4/29/2024)

1. What is the issue? Please be detailed.

OIDC authentication issue after the redirect from the identity provider.

2. What steps can we take to reproduce this issue?

Fresh install of ODK. Running it behind Traefik. Central username/password authentication worked for my first account which I promoted to an admin. Login worked. I created a project.

I set up a new application in Authentik and enabled an OIDC provider. I supplied the redirect URL according to Central Docs.

I took the OIDC parameters and populated the environment variables as well.

On login, I click the Continue button and I'm sent to Authentik. There I have MFA set up and authentication works fine. Metrics even show the authn was successful. I'm then redirected back to ODK. Here I get this error message.

Something went wrong during login. Please contact your server administrator.

I'm using Firefox. It also failed with Chrome. The SAML Extension in Chrome didn't reveal any glaring errors, except the state property appeared to be empty . Shrug.

My SSO account email address matches the email address of my first user in ODK.

3. What have you tried to fix the issue?

Checked SAML Extension of Chrome and server logs.

Logs show 303 redirects on /v1/oidc/callback.

Logs show 404 not found on /v1/sessions/restore.

How can I enable a higher level of logging?

4. Upload any forms or screenshots you can share publicly below.

I have altered the code parameter below.

central-nginx-1               | 172.24.0.3 - - [29/Apr/2024:19:37:36 +0000] "GET /version.txt HTTP/1.1" 304 0 "https://domain.town/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" "70.91.90.38"
central-nginx-1               | 172.24.0.3 - - [29/Apr/2024:19:37:40 +0000] "GET /v1/oidc/login HTTP/1.1" 307 805 "https://domain.town/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" "70.91.90.38"
central-service-1             | ::ffff:172.23.0.7 - - [29/Apr/2024:19:37:40 +0000] "GET /v1/oidc/login HTTP/1.0" 307 805
central-nginx-1               | 172.24.0.3 - - [29/Apr/2024:19:37:42 +0000] "GET /v1/oidc/callback?code=f827d43237f74ba4b147ab507cc9809a&state= HTTP/1.1" 303 186 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" "70.91.90.38"
central-nginx-1               | 172.24.0.3 - - [29/Apr/2024:19:37:42 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" "70.91.90.38"
central-service-1             | ::ffff:172.23.0.7 - - [29/Apr/2024:19:37:42 +0000] "GET /v1/oidc/callback?code=f827d43237f74ba4b147ab507cc9809a&state= HTTP/1.0" 303 186
central-nginx-1               | 172.24.0.3 - - [29/Apr/2024:19:37:42 +0000] "GET /css/app.fef7c193.css HTTP/1.1" 304 0 "https://domain.town/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" "70.91.90.38"
central-nginx-1               | 172.24.0.3 - - [29/Apr/2024:19:37:42 +0000] "GET /js/app.c08fa6fa.js HTTP/1.1" 304 0 "https://domain.town/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" "70.91.90.38"
central-nginx-1               | 172.24.0.3 - - [29/Apr/2024:19:37:42 +0000] "GET /js/chunk-vendors.5190219a.js HTTP/1.1" 304 0 "https://domain.town/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" "70.91.90.38"
central-nginx-1               | 172.24.0.3 - - [29/Apr/2024:19:37:42 +0000] "GET /css/component-analytics-introduction.4f6ad8ae.css HTTP/1.1" 304 0 "https://domain.town/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" "70.91.90.38"
central-nginx-1               | 172.24.0.3 - - [29/Apr/2024:19:37:42 +0000] "GET /js/component-analytics-introduction.44c7f1f6.js HTTP/1.1" 304 0 "https://domain.town/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" "70.91.90.38"
central-nginx-1               | 172.24.0.3 - - [29/Apr/2024:19:37:42 +0000] "GET /v1/sessions/restore HTTP/1.1" 404 76 "https://domain.town/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" "70.91.90.38"
central-service-1             | ::ffff:172.23.0.7 - - [29/Apr/2024:19:37:42 +0000] "GET /v1/sessions/restore HTTP/1.0" 404 76
central-nginx-1               | 172.24.0.3 - - [29/Apr/2024:19:37:57 +0000] "GET /version.txt HTTP/1.1" 304 0 "https://domain.town/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" "70.91.90.38"

I have both ODK and Authentik running as docker containers. I'm not sure that direct communication from ODK will reach Authentik.

brian_252 · April 29, 2024, 8:17pm

Yes. I docker compose exec'd into the service container and attempted to access the IDP via wget. It worked. I did not attempt and authentication. Hmmm, I wonder if I could set up a command line app to simulate ODK.

brian_252 · April 29, 2024, 9:45pm

Found an exception being thrown here by console logging and rebuilding.

github.com

getodk/central-backend/blob/b4754cf52bfa64b1ca841bc9ccb64a38726398e8/lib/resources/oidc.js#L140


      
          service.get('/oidc/callback', endpoint.html(async (container, _, req, res) => {
            try {
              const code_verifier = req.cookies[CODE_VERIFIER_COOKIE]; // eslint-disable-line camelcase
              const next          = req.cookies[NEXT_COOKIE];          // eslint-disable-line no-multi-spaces
              res.clearCookie(CODE_VERIFIER_COOKIE, callbackCookieProps);
              res.clearCookie(NEXT_COOKIE,          callbackCookieProps); // eslint-disable-line no-multi-spaces
          
              const client = await getClient();
          
              const params = client.callbackParams(req);
              const tokenSet = await client.callback(getRedirectUri(), params, { code_verifier });
          
              const { access_token } = tokenSet;
          
              const userinfo = await client.userinfo(access_token);
          
              const { email, email_verified } = userinfo;
              if (!email) {
                container.Sentry.captureException(new Error(`Required claim not provided in UserInfo Response: 'email'`));
                return errorToFrontend(req, res, 'email-claim-not-provided');
              }

Going down this rabbit hole.

ktuite · April 29, 2024, 11:00pm

Can you confirm which version of Central you are using? We just released a new version 2024.1 a few hours ago, right around the time you originally posted, so I really can't tell! You can check <your odk url>/version.txt.

brian_252 · April 30, 2024, 12:06am

versions:
20dcbf46703ded595b82bda713c624673a3faff9 (v2023.5.1)
ab0c8ecbf837c7e433b20c7d7d1d2955cc8df1c6 client (v2023.5.0)
983ec81e69793fdb589ffdc346a16ef977489be4 server (v2023.5.0)

brian_252 · April 30, 2024, 12:08am

I printed the exception immediately after it was caught.

central-service-1             | RPError: state mismatch, expected undefined, got: 
central-service-1             |     at Client.callback (/usr/odk/node_modules/openid-client/lib/client.js:399:13)
central-service-1             |     at /usr/odk/lib/resources/oidc.js:148:37
central-service-1             |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
central-service-1             |   checks: { code_verifier: 'eEWu1mh5DgAEz9-15x64xCQxEnSrR-vT_nxqaqwjXoc' },
central-service-1             |   params: { code: 'e39ae330ecf04cc7afe7c5c46ec2bbc2', state: '' }
central-service-1             | }
central-service-1             | Trace
central-service-1             |     at /usr/odk/lib/resources/oidc.js:183:15
central-service-1             |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

(line numbers are off due to debugging)

Evidently it's due to that state query string paramater from OIDC.

Reading up a little, it appears that "state" is a nonce.

brian_252 · April 30, 2024, 12:10am

And from git...

# git submodule 
 ab0c8ecbf837c7e433b20c7d7d1d2955cc8df1c6 client (v2023.5.0)
 983ec81e69793fdb589ffdc346a16ef977489be4 server (v2023.5.0)

Oh that matches above

brian_252 · April 30, 2024, 12:16am

That's pretty funny! I'll upgrade tomorrow.

ktuite · April 30, 2024, 3:32am

Please let us know if it still a problem in the new version (it very well could be) and we will look into it on our end as well!

brian_252 · April 30, 2024, 3:15pm

With 2024.1, I get the same issue. I confirmed the upgrade by checking version.txt.

I'm gonna review the configuration of this OIDC provider in Authentik compared to the other 2 providers which are known to be working.

brian_252 · April 30, 2024, 3:35pm

The configuration of all 3 providers in Authentik are the same.

But now I have a theory. Based on the exception. I'm logging this right after it's caught in lib/resources/oidc.js.

central-service-1             | RPError: state mismatch, expected undefined, got: 
central-service-1             |     at Client.callback (/usr/odk/node_modules/openid-client/lib/client.js:399:13)
central-service-1             |     at /usr/odk/lib/resources/oidc.js:148:37
central-service-1             |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
central-service-1             |   checks: { code_verifier: 'GtLmC_eR4-hJnDSXIuHuyCesINjkMHouOMzzuvzFeUM' },
central-service-1             |   params: { code: '7917e5a005784e71b59a88cbfb9e0f5d', state: '' }
central-service-1             | }
central-service-1             | Trace
central-service-1             |     at /usr/odk/lib/resources/oidc.js:182:15
central-service-1             |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

It looks like ODK is expecting state to be undefined. Whereas Authentik is providing an empty string. In Chrome DevTools, I see the call back looks like this:

https://domain.town/v1/oidc/callback?code=blah&state=

Maybe what ODK is expecting is something like this:

https://domain.town/v1/oidc/callback?code=blah

brian_252 · April 30, 2024, 3:49pm

Yeah, it's a good lead. Looking at openid-client (v5.4.3) on line 398 it's doing the check. https://github.com/panva/node-openid-client/blob/v5.4.3/lib/client.js#L398

So in data like this (from above):

   checks: { code_verifier: 'GtLmC_eR4-hJnDSXIuHuyCesINjkMHouOMzzuvzFeUM' },
   params: { code: '7917e5a005784e71b59a88cbfb9e0f5d', state: '' }

either we need to add state: '' to checks, or maybe remove state: '' from params.

brian_252 · April 30, 2024, 5:38pm

ODK does not care to check state. Notice only code_verifier is specified.

const tokenSet = await client.callback(getRedirectUri(), params, { code_verifier });

Authentik sends state -- I guess -- no matter what. I can dig deeper into that.

node-openid-client has a similar bug report https://github.com/panva/node-openid-client/issues/377 TLDR; if state is provided in params (query string), it must be checked.

brian_252 · May 1, 2024, 2:15pm

Still learning how OIDC works. Found that PKCS is a preferred alternative to the state nonce for preventing CSRF attacks. ODK is already doing this via code_verifier.

See https://datatracker.ietf.org/doc/html/draft-ietf-oauth-security-topics for reference.

brian_252 · May 1, 2024, 7:50pm

I don't always test, but when I do, I test in production.

I'm adding the state parameter. It's always been optional. It provided resistance to CSRF attacks. The Oauth spec recently added code_challenge (https://datatracker.ietf.org/doc/html/rfc7636), and this also provides resistance to CSRF attacks, but it did not deprecate state. It remains optional.

I found in the code where Authentik sets state to the empty string. Not sure why it's doing this. Maybe for legacy reasons.

query_params["state"] = [str(self.params.state) if self.params.state else ""]

Sweet, my build is done.... And that works. I'll check if the ODK testing suite passes. Looks like the oidc-provider used in testing also will pass along this state parameter just fine.

If and so, I'll submit a PR if that's ok with you.

brian_252 · May 1, 2024, 7:55pm

Had an issue with make dev-oidc. Can GH test this?

❯ make dev-oidc
node lib/bin/enforce-node-version.js
node lib/bin/run-migrations.js
Error: password authentication failed for user "jubilant"
make: *** [Makefile:35: migrations] Error 1

brian_252 · May 1, 2024, 8:07pm

github.com/getodk/central-backend

improve: Also pass along a state (nonce) paramater.

getodk:master ← openbrian:oidc-state-parameter

opened 08:07PM - 01 May 24 UTC

openbrian

+8 -2

bug: #1134 It's not needed for CSRF protection because code_challenge does th…is, but the Authentik IDP will return &state= in the query string. This will trigger node-openid-client to fail a check. Closes #1134  #### What has been done to verify that this works as intended? #### Why is this the best possible solution? Were any other approaches considered? #### How does this change affect users? Describe intentional changes to behavior and behavior that could have accidentally been affected by code changes. In other words, what are the regression risks? #### Does this change require updates to the API documentation? If so, please update docs/api.yaml as part of this PR. #### Before submitting this PR, please make sure you have: - [ ] run `make test` and confirmed all checks still pass OR confirm CircleCI build passes - [ ] verified that any code from external sources are properly credited in comments or that everything is internally sourced Testing the oidc way failed for me. I mean the tests would not kick off.

alxndrsn · May 3, 2024, 6:22am

brian_252:

Had an issue with make dev-oidc. Can GH test this?

❯ make dev-oidc
node lib/bin/enforce-node-version.js
node lib/bin/run-migrations.js
Error: password authentication failed for user "jubilant"
make: *** [Makefile:35: migrations] Error 1

Hi @brian_252, thanks for digging so deeply on this issue.

The error above is a failure to authenticate with PostgreSQL. I suspect you will see the same error with make dev and make run, neither of which run OIDC-related code.

If you are running Postgres with docker, you may be able to resolve this by calling node lib/bin/create-docker-databases.js.

Otherwise, stop any Postgres instance running on port 5432 and call make run-docker-postgres.

Meanwhile, I'll take a look at the state/nonce issue

LN · December 5, 2024, 4:53pm

We tried to get this addressed on the Authentik side but did not get any response there -- https://github.com/goauthentik/authentik/pull/9735

We've now merged https://github.com/getodk/central-backend/pull/1327 which should address the issue. It should be released in the next couple of weeks.