Unable to login getting "Something went wrong: error code 500"

Hi,
I have recently upgraded from 1.3 to 1.4.1 and started getting logging issues. Which gets fixed for a while after restarting containers. Please guide.
Screenshot of error and log file (tail 200) is attached.


logs.txt (200.9 KB)

Hi,

I don't have a solution but trying to get things started.

In the logs I see some SQL queries going haywire:

e[36;1mpostgres_1            |e[0m 2022-03-14T18:12:10.295688657Z 	select path, value from form_field_values
e[36;1mpostgres_1            |e[0m 2022-03-14T18:12:10.295694603Z 	inner join
e[36;1mpostgres_1            |e[0m 2022-03-14T18:12:10.295699323Z 	  (select id, "submissionId" from submission_defs
e[36;1mpostgres_1            |e[0m 2022-03-14T18:12:10.295704498Z 	    where current=true and "localKey" is null) as defs
e[36;1mpostgres_1            |e[0m 2022-03-14T18:12:10.295744674Z 	  on defs.id=form_field_values."submissionDefId"
e[36;1mpostgres_1            |e[0m 2022-03-14T18:12:10.295751168Z 	inner join
e[36;1mpostgres_1            |e[0m 2022-03-14T18:12:10.295755580Z 	  (select * from submissions
e[36;1mpostgres_1            |e[0m 2022-03-14T18:12:10.295760613Z 	    where "deletedAt" is null and draft=$1) as subs
e[36;1mpostgres_1            |e[0m 2022-03-14T18:12:10.295765367Z 	  on subs.id=defs."submissionId"
e[36;1mpostgres_1            |e[0m 2022-03-14T18:12:10.295770531Z 	where
e[36;1mpostgres_1            |e[0m 2022-03-14T18:12:10.295775246Z 	  form_field_values."formId"=$2 and
e[36;1mpostgres_1            |e[0m 2022-03-14T18:12:10.295780450Z 	  (("submissions"."createdAt" >= $3) and ("submissions"."createdAt" <= $4))
e[36;1mpostgres_1            |e[0m 2022-03-14T18:12:10.295786051Z 	group by path, value
e[36;1mpostgres_1            |e[0m 2022-03-14T18:12:10.295790970Z 	order by path asc, value asc
e[36;1mpostgres_1            |e[0m 2022-03-15T06:45:38.382480609Z LOG:  unexpected EOF on client connection with an open transaction
e[36;1mpostgres_1            |e[0m 2022-03-15T06:45:38.579936934Z LOG:  received fast shutdown request
e[36;1mpostgres_1            |e[0m 2022-03-15T06:45:38.579967426Z LOG:  aborting any active transactions
e[36;1mpostgres_1            |e[0m 2022-03-15T06:45:38.580020435Z LOG:  autovacuum launcher shutting down
e[36;1mpostgres_1            |e[0m 2022-03-15T06:45:38.584740787Z LOG:  shutting down
e[36;1mpostgres_1            |e[0m 2022-03-15T06:45:38.602915989Z LOG:  database system is shut down
e[36;1mpostgres_1            |e[0m 2022-03-16T10:03:37.934439970Z ERROR:  missing FROM-clause entry for table "submissions" at character 380
e[36;1mpostgres_1            |e[0m 2022-03-16T10:03:37.934494379Z STATEMENT:  
e[36;1mpostgres_1            |e[0m 2022-03-16T10:03:37.934499725Z 	select path, value from form_field_values
e[36;1mpostgres_1            |e[0m 2022-03-16T10:03:37.934504030Z 	inner join
e[36;1mpostgres_1            |e[0m 2022-03-16T10:03:37.934507595Z 	  (select id, "submissionId" from submission_defs
e[32menketo_redis_main_1   |e[0m 2022-03-20T18:59:17.022569243Z 1:M 20 Mar 2022 18:59:17.021 * 1 changes in 300 seconds. Saving...
e[36;1mpostgres_1            |e[0m 2022-03-16T10:03:37.934511527Z 	    where current=true and "localKey" is null) as defs
e[35mmail_1                |e[0m 2022-02-08T07:45:20.828028911Z 	s/^#?(dc_other_hostnames)=.*/\1='\'''\''/;

After a while the database connection pool seems exhausted, which explains why a restart fixes things for a while.
This is an issue that likely only the core team will be able to help you with.

Can you tell whether the database migrations have been run successfully after the upgrade?
Can you tell whether you've upgraded to the latest version, so there's not a version mismatch in the code?

Hey Florin,
Thank you for replying,

Can you guide me how to check if database upgrade was successful?

I have upgraded to
versions:
4f2c9678ee717c4b55ef41d947da1d6a7c817a6c (v1.4.1-2-g4f2c967)
461c1037997f9fb88075b75612b83cea297ced42 client (v1.4.0)
3462cd5d84c4f886a2b9c4a314227ea6da193de0 server (v1.4.1)

It's probably not a migration issue because v1.4.1 won't run if migrations haven't finished successfully. One thing you can try is run the docker-compose up command without the -d so you see what the output is. You will be able to confirm if migrations have indeed completed.

My recommendation is to try the upgrade process again to get to v1.4.2. There is a fix on v1.4.2 for a memory usage on form upload that you might be running into.

Then since this issue seems predictable, monitor your server's RAM and CPU to see if there is anything strange going on after the upgrade. I like htop and docker stats for monitoring.

Are you using the included PostgreSQL DB or an external or custom DB?

login into your instance and run the below command in your docker
df -hT
Might be you are running out of space.

Best,
@iamnarendrasingh

I pasted some more details that happened to me, in the following thread: Severe issue on multiple Central installations with images - Error 500