Dear team, i pray you are well. I upgraded our ODK central instance to that version starting at 1am. I read that it would take hours so I started the upgrade at 1 a.m. It ran until 6 AM and still had not stopped. I terminated it. Of course, it rolled back. I repeated the process, it has now run for 5 hours. There are not error logs, so nothing bad is happening. and now here. Everyone is stuck in the field, we have an EBOLA outbreak. am stranded. Did the documentation mean it would take all these hours?I need some guidance on how to go about it. I have tried to restart the upgrade and i still get the same delay. However, there are no error logs
Sorry you’ve run into problems, @pwaiswa.
The server is most likely migrating data from the old version to the new version. Depending on the number of submissions you have and the performance of your server, this can take quite a while, sometimes several hours. If the process is interrupted, it will start over the next time you try.
A few questions that will help us troubleshoot:
- What version are you upgrading from? The older the version, the more migrations need to be run.
- Are you able to temporarily increase the RAM or CPU available to your machine or external database?
Based on what you’ve shared so far, my recommendation is to let the migration finish. That’s the safest option. Unfortunately, it’s difficult to estimate how long it will take because the duration depends heavily on the performance of your server.
Unfortunately, i thought there was a problem so i stopped it. I do not have many resources so i have set it to <<'SQL'
ALTER SYSTEM SET shared_buffers = '4GB';
ALTER SYSTEM SET effective_cache_size = '12GB';
ALTER SYSTEM SET maintenance_work_mem = '2GB';
ALTER SYSTEM SET work_mem = '128MB';
ALTER SYSTEM SET wal_buffers = '64MB';
ALTER SYSTEM SET max_wal_size = '16GB';
ALTER SYSTEM SET min_wal_size = '2GB';
ALTER SYSTEM SET checkpoint_timeout = '30min';
ALTER SYSTEM SET checkpoint_completion_target = '0.9';
SQL.
I have 15371333 records, 25GB dump
Am upgrading from Central v2025.4
I have just restarted it. In case i fail, is it possible to roll back to my previous version so people do not loose tomorrow too?
At this stage, I would strongly recommend you stop making changes, wait until the migration finishes, and then we can talk about steps you can take to avoiding this kind of issue.
Noted with thanks. Am off the server for now, letting it run its migrations in peace. I just hope and pray all is well by morning. It takes a lot of time though
I have left this to run till time X. For now, I want to install Central v2025.4 and restore the dump i have at least by morning, on another VM so people can continue using the system. My challenge is, the old approach
git clone https://github.com/getodk/central;
cd central;
git checkout v2025.4;
git submodule update --init;
docker-compose build;
seems to be off. I keep getting "error: pathspec 'v2025.4' did not match any file(s) known to git"
Could i be missing something?
Hello all. Service has been restored. Thanks for bearing with me. All i had to do was to be patient.
To the dev team, is there a way this can be solved? Am not even sure i have a better suggestion but the time it takes is long. Am just lucky i had those records, People should be having tons of them. You could also give a picture of roughly how many hours for a range of storage or records, such that people do not think it is the usual few hours they have waited for on previous upgrades
Otherwise thanks team. Thanks @yanokwa for always coming through to support
I'm so glad you were able to let the migration done. I know it does not feel good to have the server down with minimal feedback.
We usually have a log message at the beginning of each database migration which helps users get a sense of progress but we forgot to do it in this case. We will make sure this is a non-negotiable for future releases. We will also try to echo out a percent completion in the future.
You saw the notice at https://docs.getodk.org/central-upgrade/#central-upgrade-2026-1 about the lengthy migration, right?
That's about as specific as we're willing to be because total time taken depends on:
- the number of affected resources (e.g. submissions, Entities, submission attachments...) on a server
- the amount of CPU and RAM available to the application server
- the amount of CPU and RAM available to the database server
- the amount of swap available, the speed of the storage volume
- other activity on the server
We recommend performing the update on a clone of your production environment if you work in an environment where you need high availability. Performing an upgrade on a clone gives you an opportunity to see how long the upgrade will take and plan accordingly.
It can also give you confidence that there is nothing unusual about the upgrade process and how it interacts with your specific environment. We do extensive verification on clones for our Cloud service. If we see that a migration results in downtime, we adjust database resources to make them go faster.
If your server is resource-constrained, you could consider deleting forms that are no longer in active use. This will purge all associated submissions and mean there is less work to do for any future migrations involving submissions or forms.