Please a friend has a similar issue, showing two central_nginx_1, one of them was showing an error when we restarted using docker-compose restart.
We then ran docker-compose build
After this was done, all the projects were gone.
I saw this link from my search:
https://docs.getodk.org/central-troubleshooting/#database-disappeared-after-running-docker-commands
After tracing all the volume names, we are still unable to restore the database.
Please any help is sincerely appreciated.
So Central is still installed but you don't see any of the old projects? Did you have to recreate the admin account needed to log into Central?
Yes, central is still installed. We recreated the admin account to log in. All users previously created also didn't exist anymore. We suspect it's a data persistence issue.
In the instructions for recovering Projects at https://docs.getodk.org/central-troubleshooting/#database-disappeared-after-running-docker-commands, there are no pg_hba.conf files in /var/lib/docker/volumes/{letters and numbers}/_data. However, there are several pg_hba.conf files that are located in sub-directories of those {letters and numbers} directories:
root@odk-central:/var/lib/docker/volumes# find . -name "pg_hba.conf"
./postgres.data.bak/6073a4d12442bba9f7612f91d17c9ef3428a62e52e3a3b3dc00b5c675025d7af/_data/pg_hba.conf
./postgres.data.bak/a30e45983767d2850110adc5dc40f5ee012b1c3b01e98b16b2c97a16b717de21/_data/pg_hba.conf
./postgres.data.bak/_data/pg_hba.conf
./postgres.data.bak/aaa50f0fea2c1fd545a05edc55a38bd1dfac69c2c9648f95e396c13830f5ccb1/_data/pg_hba.conf
./6073a4d12442bba9f7612f91d17c9ef3428a62e52e3a3b3dc00b5c675025d7af/8082929abb108e5873cc4aa369fad6716145a722a7e9c99c2ee017794671436e/_data/pg_hba.conf
./6073a4d12442bba9f7612f91d17c9ef3428a62e52e3a3b3dc00b5c675025d7af/353255342fa57f624abf53a847608a8e57a9b234dddaa5b16d681755281acc52/_data/pg_hba.conf
./a30e45983767d2850110adc5dc40f5ee012b1c3b01e98b16b2c97a16b717de21/fbc54d0c8c234850aa8a81fab94fe6c3a4880fb5092a0ae115b08107e46c2f95/_data/pg_hba.conf
./a30e45983767d2850110adc5dc40f5ee012b1c3b01e98b16b2c97a16b717de21/78a69a124bf34db2e66c409721740649935a8db47d0dd5fd88ddcaddc2d3ddb7/_data/pg_hba.conf
./a30e45983767d2850110adc5dc40f5ee012b1c3b01e98b16b2c97a16b717de21/_data/pg_hba.conf
I work with @snillok4u so I can add some context on this issue. We resized our digitalocean droplet and realized that nginx wasnt starting because its port was used by something else. We thought it was best to then upgrade our ODK central installation hoping the issue might be solved, and it did, only for us to no longer be able to login as all of our data were simply GONE; projects, submissions, EVERYTHING. I understand its a data persistence issue, but still open to what ever solution anyone can provide on this platform. We are desperate at this point, this data has to be recovered. Thankyou in advance.
First of all, sorry about the data loss. Before you do anything further, it'd be good to pause to take stock of where you are so you don't make a bad situation worse.
Did you have any backups turned on? Either the Google Drive backups or Digital Ocean's snapshots? If you don't, now would be a good time to turn on snapshots so you can rollback.
Look through your shell history to see what commands you ran to see what went wrong. My guess is that you either brought the container down rather than stopping it. You can confirm by running grep "docker-compose down" ~/.bash_history to see if you did indeed run the command.
okay, i'll give that a go now.
We did run this command & sadly we don't have snapshots to rollback to.
@yanokwa we need your help on this badly.
I reached out privately to @johndotcom to reattach the database volume to the database container.
As for next steps, I'll update our docs to strongly recommend automated backups for users who are self-hosting. I'll also simplify the reattaching directions in the troubleshooting guide. We'll continue to work towards making sure bringing down containers don't make it seem like you've lost data.
Thank you so much @yanokwa for this, we are forever grateful. I am speechless as to what to write here. God bless you real good.
Backup was negelected, as important as it is. I have learned the importance now, the hard way though.
Thank you @kdreger1 for responding when I reached out, for taking so much of your time, for the late-night effort, and for the more information added here. Thank you so much.
We are super grateful.
This comes late I know, but I continue to enjoy the solution shared on this platform to our issue till today. It was a great learning opportunity for us all. We are grateful to have both ODK and such an incredible community behind it.