Problem using Bi-directionnal Sync on local Aggregate 1.4.4 instance

Hi all,

I have a Aggregate 1.4.4 instance, running on a laptop (Tomcat 6, Windows 7 install) in a LAN network.

I have been able to:

  • Use ODK 1.X mechanism with Survey
  • Use the Legacy XML mechanism to upload forms (with the -media process) to the server and download them on a phone via Survey again
  • Send finalize forms from Survey to my Aggregate Instance.

Where I can't get things to work is when I try anything that has to do with Sync / Reset App Server, so basically the 2.0 mechanism (unless Legacy XML uses it as well).

Here's what I have done so far Server-side:

  • Allows ODK Tables Synchronization
  • Gave all access rights (possible) for Anonymous users (its a test LAN so don't care)
  • Gave all all rights to a Google Account
  • Gave all all rights to a ODK Account

If I start Survey as a standalone (NOT in the context of Tables using the "Changing App Name"), and try to either Sync/Reset App serer, I get:

Error
Account Re-Authorization required

Before getting this message thought, I do see in the notices:

Starting
Retrieving list of Tables from server

It stays in this state for a while. Perhaps it times out.

Some questions:

If I use Sync, does it matter what I write under Survey's settings regarding Username & password? I tried both leaving blank & entering the Google Account (with all rights) and an ODK account (again with all rights granted).

If the URL entered for 1.X mechanisme worked (entered in the settings of SURVEY), then that same URL should work as well for Sync right? Or is there a chance that my problem would by that my phone doesn't actually connects to the server and thus times out? But then shouldn't it mention a failure to connect?

Another option is that my Aggregate server is the problem, though if it works fine for any 1.X operation it seems to me it should work for 2.0 (provided the settings mentionned above).

If it works for ODK 1.x, then if it is properly configured for what you are
trying to do with 2.0, it should also work.

The username/password settings in Survey are purely for the legacy pathway
and the user e-mail setting within Survey is currently not piped through to
Sync, so neither of these affect the 2.0 pipeline.

"Rest App Server" requires* Administer Tables* privileges.

"Sync Now" requires Synchronize Tables privileges

With respect to server configuration:

If you grant the Anonymous User the above privileges, when you connect to
the server, the server never actually challenges the client for their
identity, and they are simply granted access to the server (in this case,
the user you select on the Android device is irrelevant).

Only if the Anonymous User does not have the needed access does the server
authenticate the Android device using the user account you configured for
it.

Remember to "Save Changes" on the Permissions tab to get the changes to be
applied (and to take effect).

--------------why you always get the re-authorization error--------------

I hate the client library we are using to make the synchronization calls to
the server. It is very difficult to get error information out of it; and it
is Android-specific. We want to use a more generic library (Apache Wink
seems likely) so we can share the synchronization logic across platforms.
But we couldn't just drop Wink in (we tried). In an effort to get
everything assembled for the release, I decided to report every error as a
"Re-Authorization Required" error.

So that is why everything is a failing at 'Authorization'. It is the only
failure that is ever reported. This will be fixed as we switch to a better
client library with better error-reporting mechanisms.

------------what is likely going on------------

From your description, it sounds like you are trying to Sync from ODK
Survey?

This requires an ODK Aggregate settings change.

(1) Go to Site Admin / Preferences
(2) Click on "Change ODK 2.0 App Name"
(3) enter
survey
(4) Click on "Change ODK 2.0 App Name" button to complete the change.

This changes ODK Aggregate to work with the "survey" application --
allowing sync of all data under /sdcard/opendatakit/survey

By default, ODK Aggregate is configured to work with the "tables"
application.

Note that if you launch ODK Sync directly, it launches as the "tables"
application. It is only when you launch ODK Sync from the cloud icon within
ODK Survey that it launches as the "survey" application and syncs the data
in the /sdcard/opendatakit/survey directory.

Longer term, we will probably change the UI when directly launching ODK
Sync to require the user to explicitly choose an ODK 2.0 App Name (
sub-directory under /sdcard/opendatakit ) to launch into.

The aim here is to enable multiple independent 'applications' built upon
our tools to co-exist on a single Android device without interference.

So you could have

/sdcard/opendatakit/rebuildingSupplies
/sdcard/opendatakit/
sanitation
/sdcard/opendatakit/

medicalSupplies
as 3 different applications, that a data collector could launch into and
switch between as needed. Each application submits data into a different
ODK Aggregate server and that data is processed by a different support
staff.

Eventually there would be an APK that you create for your particular
application, and it then launches into Survey or Tables or Sync as needed.

By setting everything as anonymous

··· On Thu, Sep 11, 2014 at 11:34 AM, wrote:

Hi all,

I have a Aggregate 1.4.4 instance, running on a laptop (Tomcat 6, Windows
7 install) in a LAN network.

I have been able to:

  • Use ODK 1.X mechanism with Survey
  • Use the Legacy XML mechanism to upload forms (with the -media process)
    to the server and download them on a phone via Survey again
  • Send finalize forms from Survey to my Aggregate Instance.

Where I can't get things to work is when I try anything that has to do
with Sync / Reset App Server, so basically the 2.0 mechanism (unless Legacy
XML uses it as well).

Here's what I have done so far Server-side:

  • Allows ODK Tables Synchronization
  • Gave all access rights (possible) for Anonymous users (its a test LAN so
    don't care)
  • Gave all all rights to a Google Account
  • Gave all all rights to a ODK Account

If I start Survey as a standalone (NOT in the context of Tables using the
"Changing App Name"), and try to either Sync/Reset App serer, I get:

Error
Account Re-Authorization required

Before getting this message thought, I do see in the notices:

Starting
Retrieving list of Tables from server

It stays in this state for a while. Perhaps it times out.

Some questions:

If I use Sync, does it matter what I write under Survey's settings
regarding Username & password? I tried both leaving blank & entering the
Google Account (with all rights) and an ODK account (again with all rights
granted).

If the URL entered for 1.X mechanisme worked (entered in the settings of
SURVEY), then that same URL should work as well for Sync right? Or is there
a chance that my problem would by that my phone doesn't actually connects
to the server and thus times out? But then shouldn't it mention a failure
to connect?

Another option is that my Aggregate server is the problem, though if it
works fine for any 1.X operation it seems to me it should work for 2.0
(provided the settings mentionned above).

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Thanks for the quick and detailed reply.

So it turns out I wasn't actually on a LAN, I set myself up on my actual connection. It doesn't really change anything, or so I thought.

So then I tried on a offline network, and now for some reason I can't get passed the Authentication when on Sync: I get " Authentication failed: Java.OI.oiException" that's an Android message though.

I can still upload/download forms the 1.X way as usual.

I should still be able to authenticate I would think while offline. Unfortunately I can't try it online now as I am in a very slow hotel connection behind a proxy....

Update: So I can Authenticate (without getting the OI Java exception) only IF I tap "Authenticate Account" when I am connect on an online network with my phone.

If I actually am connected to a LAN, it always throws up the exception (see attched screenshot).

I trying syncing my account (the one I choose when using Survey) just before but it didn't change the outcome.

I'm still unable to use sync and get past this Java OI exception. I tried re-installing OI File manager but didn't change anything....

Should it matter what OI version is used? I'm using 2.0.2

Might be my S3, I'll have access to an Xperia pro tomorrow so I'll try that and give any update if that works....

I get the following, on both an instance running on a LAN which I have set up and an instance hosted on a server (own server, not app engine or amazon) (see attached png).

Since I was set up to use survey as the App Name, I replace the first "tables" in the link you provided with "survey" (which gave the exact same result as the one attached). To cover all bases, I also change the App Name on the server back to "tables", and then tried the link your provided as is (just replacing the instance name).

In all cases, exactly the same results.

I have another question:

When I upload forms on the server through the form uploader, do I need the definition.csv & properties.csv even though I am not using tables, but survey? At first reading the documentation it seemed to me that is needed for tables, not for survey. I tried nonetheless to joing them in the -media file that contains the forms.zip (outside the .zip, just as was shown in the documentation about using the Legacy XML mechanism).

Even with the definition & properties I got the same result as without it.

Francis

Oh and attached the tomcat log for the LAN instance. Unfortunately the one I really care about is the one hosted on a server, but since they give same result this might be of help.

THe last error entries on september 22nd between 6am to 6:20 am would be me trying to access /odktables/tables/tables

tomcat6-stdout.2014-09-20.log (698 KB)

Thanks for the very detailed reply Mitch, as always.

Unfortunately I have really crappy internet access right now so I'll have to wait before I can really test this out, it gets hard to tell if it fails because it times out, the connection drops or something else.

Also I can't try it on my LAN because I can't authenticate if my phone isn't online. To be more precise: When I tap "Authorize account" while connected to a LAN, I get the Java OI Exception mentionned earlier, and can't even access the Sync Now / Reset APP Server buttons.

I'll be back within realm of decent internet in a bit more than a week, so more details then....

Hmm, first let's try a sanity check on the server.

Since the anonymous user is set up on your server, try navigating to the
following url, replacing the "opendatakit-tablesdemo" with your own
instance name:

https://opendatakit-tablesdemo.appspot.com/odktables/tables/tables

This asks the server for a list of all the tables for the app with the id
"tables", which is currently the name of the default app. Granted, this is
a confusing name.

Click the above link as-is, without replacing anything, for an example of
the sort of thing you should expect to see. XML is the default encoding
type of the response, so you should see xml. If you do, then the server is
configured correctly, at least for this phase.

Does this work?

··· On Sat, Sep 20, 2014 at 3:14 PM, wrote:

I'm still unable to use sync and get past this Java OI exception. I tried
re-installing OI File manager but didn't change anything....

Should it matter what OI version is used? I'm using 2.0.2

Might be my S3, I'll have access to an Xperia pro tomorrow so I'll try
that and give any update if that works....

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

That XML is the correct response in this case -- it indicates that your
server has no ODK 2.0 tables uploaded to it. It also confirms that the
anonymous user should be able to access the server for retrieving tables
and syncing data.

··· ---

Try granting the anonymousUser the "Administer Tables" permission. (be sure
to Save Changes).

And then try authenticating and clicking "Reset App Server" to create the
tables on the server and configure the server with exactly the same
settings as exist on the device.

This should succeed, and you should see the tables on your ODK Aggregate
server under the ODK Tables tab.

Once that works, we know that the URL and the AppName settings on the
server and device are correct for creating the tables on the server.

Then, delete one of your tables (or an App-Level file) from ODK Aggregate.

And remove the "Administer Tables" permission from the anonymous user.

Grant "Administer Tables" to the gmail account you are attempting to use.

Confirm that the gmail account name is spelled exactly as Google reports it
by logging onto Gmail and clicking on the person image/icon on the Gmail
page (in the upper right corner). The capitalization and punctuation must
exactly match.

Save changes

Now return to the device, ensure that the gmail account you are trying to
use exactly matches the capitalization and punctuation as reported by the
Gmail page.

And now click Authenticate and "Reset App Server"

Did that succeed and create the table (or restore the file) on the server?


If it did work, then something must have been corrected during the process
of getting the anonymousUser to work.

If it did not work, we now know there is an issue with the authentication
of the account and the communication of the account name to the ODK
Aggregate server.

There have been a number of changes by Google looming with their OpenID and
OAuth2 authentication mechanisms. If it was working, but now doesn't, that
might also be a factor.

Mitch

On Sun, Sep 21, 2014 at 10:28 PM, francisvachon.it@gmail.com wrote:

Oh and attached the tomcat log for the LAN instance. Unfortunately the one
I really care about is the one hosted on a server, but since they give same
result this might be of help.

THe last error entries on september 22nd between 6am to 6:20 am would be
me trying to access /odktables/tables/tables

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

And the 304 (Not Modified) exception in the log is to be expected.

If you refresh a page and the source file has not changed, it will return a
304 error to indicate that the browser should keep using the copy of the
file it already has. (this saves bandwidth).

··· On Mon, Sep 22, 2014 at 11:04 AM, Mitch Sundt wrote:

That XML is the correct response in this case -- it indicates that your
server has no ODK 2.0 tables uploaded to it. It also confirms that the
anonymous user should be able to access the server for retrieving tables
and syncing data.


Try granting the anonymousUser the "Administer Tables" permission. (be
sure to Save Changes).

And then try authenticating and clicking "Reset App Server" to create the
tables on the server and configure the server with exactly the same
settings as exist on the device.

This should succeed, and you should see the tables on your ODK Aggregate
server under the ODK Tables tab.

Once that works, we know that the URL and the AppName settings on the
server and device are correct for creating the tables on the server.

Then, delete one of your tables (or an App-Level file) from ODK
Aggregate.

And remove the "Administer Tables" permission from the anonymous user.

Grant "Administer Tables" to the gmail account you are attempting to use.

Confirm that the gmail account name is spelled exactly as Google reports
it by logging onto Gmail and clicking on the person image/icon on the Gmail
page (in the upper right corner). The capitalization and punctuation
must exactly match.

Save changes

Now return to the device, ensure that the gmail account you are trying to
use exactly matches the capitalization and punctuation as reported by the
Gmail page.

And now click Authenticate and "Reset App Server"

Did that succeed and create the table (or restore the file) on the server?


If it did work, then something must have been corrected during the process
of getting the anonymousUser to work.

If it did not work, we now know there is an issue with the authentication
of the account and the communication of the account name to the ODK
Aggregate server.

There have been a number of changes by Google looming with their OpenID
and OAuth2 authentication mechanisms. If it was working, but now doesn't,
that might also be a factor.

Mitch

On Sun, Sep 21, 2014 at 10:28 PM, francisvachon.it@gmail.com wrote:

Oh and attached the tomcat log for the LAN instance. Unfortunately the
one I really care about is the one hosted on a server, but since they give
same result this might be of help.

THe last error entries on september 22nd between 6am to 6:20 am would be
me trying to access /odktables/tables/tables

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com