Build down?

Last night I was using Build (http://build.opendatakit.org/),
everything was working fine.

I let the page stay stagnate for a few hours and came back and needed
to log back in.

It would not accept my login information.

I reset my password; received the new password in my email, and tried
it - it still gave me the error: "Could not log you in with those
credentials. Please try again", I reset my password a second time -
same thing happened.

I tried again this morning. Still getting the same error message.

I tried to report a bug via Help... it took me to a page with a 404
error (https://github.com/clint-tseng/odkmaker/issues)

Matt

It's possible the support team managing the server was messing with it. Let me check with them and make sure things are fine.

—Clint

··· On Feb 22, 2011, at 5:47 AM, Matt Basinger wrote:

Last night I was using Build (http://build.opendatakit.org/),
everything was working fine.

I let the page stay stagnate for a few hours and came back and needed
to log back in.

It would not accept my login information.

I reset my password; received the new password in my email, and tried
it - it still gave me the error: "Could not log you in with those
credentials. Please try again", I reset my password a second time -
same thing happened.

I tried again this morning. Still getting the same error message.

I tried to report a bug via Help... it took me to a page with a 404
error (https://github.com/clint-tseng/odkmaker/issues)

Matt

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

Ok - this is getting silly.

I created a NEW build username...

(my original was "mattbasinger", now I used "mlb2133" for this one)...

First thing I did once I was signed in with my new username was click
on file, and immediately it asked me to sign in again... and again it
didn't accept my new username/password - same error message as before.

Is something wrong with the user database? I'm running out of email
addresses to create new usernames with.

Clint - any ideas?

Matt

··· On Feb 22, 8:47 am, Matt Basinger wrote: > Last night I was using Build (http://build.opendatakit.org/), > everything was working fine. > > I let the page stay stagnate for a few hours and came back and needed > to log back in. > > It would not accept my login information. > > I reset my password; received the new password in my email, and tried > it - it still gave me the error: "Could not log you in with those > credentials. Please try again", I reset my password a second time - > same thing happened. > > I tried again this morning. Still getting the same error message. > > I tried to report a bug via Help... it took me to a page with a 404 > error (https://github.com/clint-tseng/odkmaker/issues) > > Matt

I started having similar problems about 48 hours ago. It seems clear there
is an issue with the server.

··· On Wed, Feb 23, 2011 at 11:55 AM, Matt Basinger wrote:

Ok - this is getting silly.

I created a NEW build username...

(my original was "mattbasinger", now I used "mlb2133" for this one)...

First thing I did once I was signed in with my new username was click
on file, and immediately it asked me to sign in again... and again it
didn't accept my new username/password - same error message as before.

Is something wrong with the user database? I'm running out of email
addresses to create new usernames with.

Clint - any ideas?

Matt

On Feb 22, 8:47 am, Matt Basinger mattbasin...@gmail.com wrote:

Last night I was using Build (http://build.opendatakit.org/),
everything was working fine.

I let the page stay stagnate for a few hours and came back and needed
to log back in.

It would not accept my login information.

I reset my password; received the new password in my email, and tried
it - it still gave me the error: "Could not log you in with those
credentials. Please try again", I reset my password a second time -
same thing happened.

I tried again this morning. Still getting the same error message.

I tried to report a bug via Help... it took me to a page with a 404
error (https://github.com/clint-tseng/odkmaker/issues)

Matt

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

Just so everyone knows, I'm homing in on the issue. It has something to do with a change I made last week. Expect a fix by later tonight.

Thanks,
—Clint

··· On Feb 22, 2011, at 5:08 PM, Neil Penman wrote:

I started having similar problems about 48 hours ago. It seems clear there is an issue with the server.

On Wed, Feb 23, 2011 at 11:55 AM, Matt Basinger mattbasinger@gmail.com wrote:
Ok - this is getting silly.

I created a NEW build username...

(my original was "mattbasinger", now I used "mlb2133" for this one)...

First thing I did once I was signed in with my new username was click
on file, and immediately it asked me to sign in again... and again it
didn't accept my new username/password - same error message as before.

Is something wrong with the user database? I'm running out of email
addresses to create new usernames with.

Clint - any ideas?

Matt

On Feb 22, 8:47 am, Matt Basinger mattbasin...@gmail.com wrote:

Last night I was using Build (http://build.opendatakit.org/),
everything was working fine.

I let the page stay stagnate for a few hours and came back and needed
to log back in.

It would not accept my login information.

I reset my password; received the new password in my email, and tried
it - it still gave me the error: "Could not log you in with those
credentials. Please try again", I reset my password a second time -
same thing happened.

I tried again this morning. Still getting the same error message.

I tried to report a bug via Help... it took me to a page with a 404
error (https://github.com/clint-tseng/odkmaker/issues)

Matt

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

This should be fixed now. I'd like to apologize profusely for the breakage and inconvenience; I assure you that no data has been lost.

Next steps for those who were broken:

  1. Go back to the account you normally use. Given the nature of the bug, I'm assuming you got nothing useful done on the many new accounts you created today.
  2. Either use the last password you were issued, or request a new one.
  3. Log in.
  4. Change your password back to something humane.
  5. That's it—things should be fixed now.

Stuff to avoid this happening again:

  1. I already have a test script that I manually run through each time I have a new candidate for release (you can see the test instance at http://staging.build.opendatakit.org). I need to add some new scenarios to it to catch some cases I had missed before, including this one.
  2. I've updated the messaging when you log in and auth fails. This will prevent the confusion with people appearing to be logged out (more on that in a second).
  3. Ideally, a volunteer or two that uses Build frequently can run on the staging version of the site. This will be easier once I add the requested import/export from/to local file feature.

Details for the curious and technically minded (in hopefully nontechnical terms):
What went wrong is essentially that a bug appeared in the code that accepts new forms and puts them into the database; it was no longer in sync with the code that saves existing forms. This is a serious issue, but it was made more serious by the fact that it started causing errors when we pulled forms back out of the database to show to the user. That is also a serious issue, but it too was made more serious: when you attempt to log in, we pull basic information about you and all your forms out from the database and pass it back to display. This means that when you attempted to log in, the server was throwing an error.

Since the ODK Build Server API conforms heavily to REST, when a user's credentials are not valid, we throw an error 401 "not authorized" back to the web browser. The application then tells the user their credentials failed, and to try again. When an unexpected error happens in the server, as was the case described above, we throw an error 500 "internal server error". Unfortunately, we were treating all errors equally in the browser application, and so even though the server was actually complaining that it was having serious issues, the application simply assumed that the user mistyped their password and cheerily prompted them to try again. In fact, the user was logged in just fine after each attempt—if they refreshed their browser they would have skipped the login prompt entirely.

··· ==========

Thanks for coming immediately to the mailing list to report the issue. It helps me diagnose things when reports come in like this.

As a side effect of all this, I've pushed several new features for Build out to the production site. Those of you on slow connections should see a markedly improved load time, and a number of smaller bugs have been fixed as well.

Best,
—Clint Tseng

On Feb 22, 2011, at 6:17 PM, Clint Tseng wrote:

Just so everyone knows, I'm homing in on the issue. It has something to do with a change I made last week. Expect a fix by later tonight.

Thanks,
—Clint

On Feb 22, 2011, at 5:08 PM, Neil Penman wrote:

I started having similar problems about 48 hours ago. It seems clear there is an issue with the server.

On Wed, Feb 23, 2011 at 11:55 AM, Matt Basinger mattbasinger@gmail.com wrote:
Ok - this is getting silly.

I created a NEW build username...

(my original was "mattbasinger", now I used "mlb2133" for this one)...

First thing I did once I was signed in with my new username was click
on file, and immediately it asked me to sign in again... and again it
didn't accept my new username/password - same error message as before.

Is something wrong with the user database? I'm running out of email
addresses to create new usernames with.

Clint - any ideas?

Matt

On Feb 22, 8:47 am, Matt Basinger mattbasin...@gmail.com wrote:

Last night I was using Build (http://build.opendatakit.org/),
everything was working fine.

I let the page stay stagnate for a few hours and came back and needed
to log back in.

It would not accept my login information.

I reset my password; received the new password in my email, and tried
it - it still gave me the error: "Could not log you in with those
credentials. Please try again", I reset my password a second time -
same thing happened.

I tried again this morning. Still getting the same error message.

I tried to report a bug via Help... it took me to a page with a 404
error (https://github.com/clint-tseng/odkmaker/issues)

Matt

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

Clint - great! I logged onto Build successfully - thanks! (it took me a
while to figure out that to change my password I click on my username -
maybe this is me being slow... but you might consider adding a dedicated
"preferences" button, or something that let's a user know how to get to the
prompt to change their password)

Very excited to see new features: 1) import/export file, 2) questions with
image prompts

When you push these features to the staging sight let me know, I'd love to
take them for a test run!

Thanks again for the fast response and resolving these login issues.

Matt

··· On Wed, Feb 23, 2011 at 5:16 AM, Clint Tseng wrote:

This should be fixed now. I'd like to apologize profusely for the breakage
and inconvenience; I assure you that no data has been lost.

Next steps for those who were broken:

  1. Go back to the account you normally use. Given the nature of the bug,
    I'm assuming you got nothing useful done on the many new accounts you
    created today.
  2. Either use the last password you were issued, or request a new one.
  3. Log in.
  4. Change your password back to something humane.
  5. That's it—things should be fixed now.

Stuff to avoid this happening again:

  1. I already have a test script that I manually run through each time I
    have a new candidate for release (you can see the test instance at
    http://staging.build.opendatakit.org). I need to add some new scenarios to
    it to catch some cases I had missed before, including this one.
  2. I've updated the messaging when you log in and auth fails. This will
    prevent the confusion with people appearing to be logged out (more on that
    in a second).
  3. Ideally, a volunteer or two that uses Build frequently can run on the
    staging version of the site. This will be easier once I add the requested
    import/export from/to local file feature.

Details for the curious and technically minded (in hopefully nontechnical
terms):

What went wrong is essentially that a bug appeared in the code that accepts
new forms and puts them into the database; it was no longer in sync with the
code that saves existing forms. This is a serious issue, but it was made
more serious by the fact that it started causing errors when we pulled forms
back out of the database to show to the user. That is also a serious
issue, but it too was made more serious: when you attempt to log in, we pull
basic information about you and all your forms out from the database and
pass it back to display. This means that when you attempted to log in, the
server was throwing an error.

Since the ODK Build Server API conforms heavily to REST, when a user's
credentials are not valid, we throw an error 401 "not authorized" back to
the web browser. The application then tells the user their credentials
failed, and to try again. When an unexpected error happens in the server, as
was the case described above, we throw an error 500 "internal server error".
Unfortunately, we were treating all errors equally in the browser
application, and so even though the server was actually complaining that it
was having serious issues, the application simply assumed that the user
mistyped their password and cheerily prompted them to try again. In fact,
the user was logged in just fine after each attempt—if they refreshed their
browser they would have skipped the login prompt entirely.

==========

Thanks for coming immediately to the mailing list to report the issue. It
helps me diagnose things when reports come in like this.

As a side effect of all this, I've pushed several new features for Build
out to the production site. Those of you on slow connections should see a
markedly improved load time, and a number of smaller bugs have been fixed as
well.

Best,
—Clint Tseng

On Feb 22, 2011, at 6:17 PM, Clint Tseng wrote:

Just so everyone knows, I'm homing in on the issue. It has something to do
with a change I made last week. Expect a fix by later tonight.

Thanks,
—Clint

On Feb 22, 2011, at 5:08 PM, Neil Penman wrote:

I started having similar problems about 48 hours ago. It seems clear there
is an issue with the server.

On Wed, Feb 23, 2011 at 11:55 AM, Matt Basinger mattbasinger@gmail.comwrote:

Ok - this is getting silly.

I created a NEW build username...

(my original was "mattbasinger", now I used "mlb2133" for this one)...

First thing I did once I was signed in with my new username was click
on file, and immediately it asked me to sign in again... and again it
didn't accept my new username/password - same error message as before.

Is something wrong with the user database? I'm running out of email
addresses to create new usernames with.

Clint - any ideas?

Matt

On Feb 22, 8:47 am, Matt Basinger mattbasin...@gmail.com wrote:

Last night I was using Build (http://build.opendatakit.org/),
everything was working fine.

I let the page stay stagnate for a few hours and came back and needed
to log back in.

It would not accept my login information.

I reset my password; received the new password in my email, and tried
it - it still gave me the error: "Could not log you in with those
credentials. Please try again", I reset my password a second time -
same thing happened.

I tried again this morning. Still getting the same error message.

I tried to report a bug via Help... it took me to a page with a 404
error (https://github.com/clint-tseng/odkmaker/issues)

Matt

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Matt Basinger, MS EngScD
Research Scientist, Post-Doc
The Earth Institute, Columbia University
mlb2133@columbia.edu

  • 1 212 854 7306 [office]
  • 1 646 290 4152 [mobile]