ODK Aggregate Keeps Refreshing

Hi Nader,

Do you mean the first part of the url? if so, it is tbsap-mapping.

Thanks,
Imran

no,
when you install aggregate to google app engine

Hi Nader,

I got your point. To be honest, I do not remember or did not taken a note of it. However, there are few chances: TBSAP mapping, TBSAP_mapping, TBSAP Mapping, TBSAP_Mapping or Aggregate.

Thanks,
Imran

Is there any chance that, I can retrieve this instance name from Google console?

Hi IMRAN,

ok

Best Rgards

1 Like

Hi folks. I wanted to jump in here and provide a very clear warning...

Deleting anything from App Engine's data store is very dangerous. You can lose all your data and corrupt your install.

Before any of this proceeds, there must be backups of all forms and all data. Then, the person who is doing these changes should have previous experience following the instructions at https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting.

@A.N.M_AL-IMRAN you have many forms and a live data collection campaign. I do have access to your server, but I did not receive a notification or your email that you had given me access and so did not know you were having problems. Do you have complete backups of your forms and data? Are you aware that these changes we are discussing could further corrupt your data store?

@Nader Do you have previous experience fixing these kinds of problems on App Engine installs?

3 Likes

+1 ... as a former IT operations person, please please please never experiment on a "live" or "production" system. Always make a full backup first, and if possible, test your changes on a non-production system first, creating good documentation, with a plan to "roll back" your work to the state it was in before -- just in case any problems come up!

Be careful out there!

2 Likes

Hi Downey,

You are right.

all what i am going to do is simulating IMRAN issues,

i did create a project in google app engine and i will upload aggregate.

no experiment on a “live” data.

Best Regards

2 Likes

Cool. Let us know if you are able to reproduce the problem on your test instance! :nerd_face:

1 Like

Yep, i am not going to do any thing

because if i want to do some thing i have to get a permission from the STARS.

Best Regards

1 Like

this is the most difficult step.

:thinking:

2 Likes

Dear Yaw,

Thanks for your response. I was trying to connect using ODK briefcase. However, the connection did not establish. I sent you access to your Nufundi email. I am not sure, whether google allows sending to another email other than Gmail. I am resending it right now. Please let me know if you receive that. If you don't receive that please send me your Gmail if possible.

Thanks,
Imran

Hi IMRN,

i am sorry for my late respond.

in fact i couldn't find solution for your issue,
so i don't have anything to do in your google account.

Best Regards

1 Like

I was able to fix this issue and I wanted to detail what I did in this particular case so others can benefit. Much of the credit for this goes to @Mitch_S for doing a great job documenting the underlying problem and the solution.

None of the steps below are hard, but you really need to be systematic and detail-oriented. If you can't be systematic and detail-oriented, it's probably best to find someone technical you trust to make these changes.

Before attempting repairs

  1. Take a moment to understand that playing around with App Engine's data store is very dangerous. You can lose all your data and corrupt your install. Let the people who own the data know this.
  2. Get admin access to the server on Google App Engine. The person who installed the server on App Engine will have administrative access.
  3. Make sure the server is on unlimited billing plan. This is a good idea because you didn't want to run out of quota while making changes to the database.
  4. Make sure no one is sending data to the server while you are making changes to the database.
  5. Read through (but do not act until reading this whole post) the excellent instructions on repairing the form definition table that Mitch put together.

How I solved this particular problem

  1. I looked at the server's logs on App Engine (Stackdriver -> Logging) and saw lots of errors about a form listing. This confirmed that this was likely due to a form issue.

  2. Imran had noted that he had uploaded a form twice and had deleted a duplicate form from the database. The exact description he sent me was..

    select * from opendatakit._form_info_submission_association where SUBMISSION_FORM_ID = “chart_review”. This query returned two rows. I deleted those rows by selecting the check boxes and deleted them. Those forms were empty.

    This confirmed that the root of the problem was probably a double form upload. Imran was correct in deleting the two entries from _form_info_submission_association, but he also needed to delete the various references to the form in the _form-* tables.

  3. Once I confirmed from Imran that there were no submissions/data for both forms, I proceeded with the instructions in repairing the form definition table. I deviated from the above instructions in a few ways

    1. I backed up all the _form_* tables first. To do that, you'll need to create a bucket for that data (Storage -> Browser), then initiate the back up (Datastore -> Admin).
    2. I walked through the steps without deleting anything and made sure all the various UUIDs matched per the instructions.
    3. For the real deletions, I confirmed the UUIDS visually and also sorted by creation date to make sure I was deleting the correct rows. Creation date is a nice confirmation because duplicate forms happen at basically the same time.
    4. The forms had some images associated and so I also deleted those in the _form_manifest_* tables. I confirmed that these entries could be deleted by reading this Aggregate Database Structure document.
    5. Once I confirmed everything worked, then I deleted the backups.

How to prevent this problem from happening

  1. Do regular back ups with ODK Briefcase. The command line interface makes this easier to automate.
  2. Make sure you are running ODK Aggregate v1.4.11 or later. These sorts of problems are less likely with later versions. Read ODK Aggregate Release Notes for how to upgrade.
  3. If you have a lot of forms and data, increase the size of the App Engine server to make it more performant.
4 Likes

Grete job Yaw and Mich

so IMRN is busy pulling data.

Best Regards

Dear Yaw,

Missed this complete description. Thanks for posting the life-saving guideline. Thanks for your support in recovering the Aggregate and the data in it. Thanks to @Nader and @Mitch_S.

Regards,
Imran

Hi Imran,

in fact I didn't do any thing,

Yaw and Mich our hero who recover your data.

Best Regards

1 Like

Hello Yaw,
I am having similar issue, the difference is that I did not use app engine. I configured aggregate to use Mysql on my local machine and I have not uploaded form yet. The continue refreshing happens when I log into aggregate.

I would be grateful for your help.

Oscar

Hi Agyei,

if your local server didn't have a form yet this may happen from either corrupted configuration files of server or cookies

try with clear your browser cookies or use new one

if the issue still exist then reinstall MySQL and Apache tomcat

Best Regards

A post was split to a new topic: Aggregate keeps refreshing