ODK collect text field

Hello all,

I was wondering about word limit for Text fields in ODK collect. So far I
noticed that maximum 45 words as limit and what ever the data which was
entered there after, was just vanished.
Is it normal that text fields has a limit in words? Any kind of help is
appreciated.

Thanks in advance

Joseph

I think there's a 255-character limit default per field, then it gets
truncated. You may use length binding to set a bigger size.

http://opendatakit.org/help/form-design/binding/

/r
Nik

··· On Tuesday, February 12, 2013, Joseph wrote:

Hello all,

I was wondering about word limit for Text fields in ODK collect. So far I
noticed that maximum 45 words as limit and what ever the data which was
entered there after, was just vanished.
Is it normal that text fields has a limit in words? Any kind of help is
appreciated.

Thanks in advance

Joseph

--

Post: opendatakit@googlegroups.com <javascript:_e({}, 'cvml',
'opendatakit@googlegroups.com');>
Unsubscribe: opendatakit+unsubscribe@googlegroups.com <javascript:_e({},
'cvml', 'opendatakit%2Bunsubscribe@googlegroups.com');>
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com <javascript:_e({},
'cvml', 'opendatakit%2Bunsubscribe@googlegroups.com');>.
For more options, visit https://groups.google.com/groups/opt_out.

Hey Nik & Mitch,

Thanks a lot for instant replies. I appreciate it and I will try
them straight away

Cheers
Joseph

··· On Tuesday, February 12, 2013 11:54:55 AM UTC+5:30, Joseph wrote: > > Hello all, > > I was wondering about word limit for Text fields in ODK collect. So far I > noticed that maximum 45 words as limit and what ever the data which was > entered there after, was just vanished. > Is it normal that text fields has a limit in words? Any kind of help is > appreciated. > > Thanks in advance > > Joseph > >

Wrong link -- see: http://opendatakit.org/help/form-design/guidelines/
Scroll down to Datastore String Length

The 255-character limit that can be changed.

Databases need to know the maximum amount of space to reserve for data.
We chose 255 as it is about a screenful of text (on a phone) and because it
is also the default used by most database software.

Mitch

··· On Tue, Feb 12, 2013 at 3:24 AM, ゴー・ニコライ wrote:

I think there's a 255-character limit default per field, then it gets
truncated. You may use length binding to set a bigger size.

http://opendatakit.org/help/form-design/binding/

/r
Nik

On Tuesday, February 12, 2013, Joseph wrote:

Hello all,

I was wondering about word limit for Text fields in ODK collect. So far I
noticed that maximum 45 words as limit and what ever the data which was
entered there after, was just vanished.
Is it normal that text fields has a limit in words? Any kind of help is
appreciated.

Thanks in advance

Joseph

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Hi Mitch,

I have question regarding worst data storage efficiency within data store,
if I increase string length more than 255 char. Could you please explain a
bit about what you mean by storage efficiency? Why can't you assign 16000
UTF-8 characters as default, this certainly helps surveys based on
qualitative (narrative) data?

Thank you
Joseph

··· On Wednesday, February 13, 2013 12:58:55 AM UTC+5:30, Mitch wrote: > > Wrong link -- see: http://opendatakit.org/help/form-design/guidelines/ > Scroll down to Datastore String Length > > The 255-character limit that can be changed. > > Databases need to know the maximum amount of space to reserve for data. > We chose 255 as it is about a screenful of text (on a phone) and because > it is also the default used by most database software. > > Mitch > > On Tue, Feb 12, 2013 at 3:24 AM, ゴー・ニコライ <nikol...@gmail.com wrote: > >> I think there's a 255-character limit default per field, then it gets >> truncated. You may use length binding to set a bigger size. >> >> http://opendatakit.org/help/form-design/binding/ >> >> /r >> Nik >> >> >> On Tuesday, February 12, 2013, Joseph wrote: >> >>> Hello all, >>> >>> I was wondering about word limit for Text fields in ODK collect. So far >>> I noticed that maximum 45 words as limit and what ever the data which was >>> entered there after, was just vanished. >>> Is it normal that text fields has a limit in words? Any kind of help is >>> appreciated. >>> >>> Thanks in advance >>> >>> Joseph >>> >>> -- >>> -- >>> Post: opendatakit@googlegroups.com >>> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >>> Options: http://groups.google.com/group/opendatakit?hl=en >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "ODK Community" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to opendatakit+unsubscribe@googlegroups.com. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> >> -- >> -- >> Post: opend...@googlegroups.com >> Unsubscribe: opendatakit...@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "ODK Community" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit...@googlegroups.com . >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com

The maximum field width you set does not affect storage efficiency all that
much (less than 2% overhead beyond what is required to represent the actual
data value).

I believe pretty much all modern databases, when they store string fields,
store them as length-delimited fields, so they use only a few bytes more
than the number of bytes required to store the string itself.

That said, there are some key limitations to understand:

(1) Google AppEngine's BigTable only stores strings up to 500 unicode
characters long. If you specify a string of 16000 UTF-8 characters, we
create a shadow Text datatype to store the entire string (for up to 1
megabyte of text). This is all transparent to the user, but impacts
efficiency. In fact, this 500-character limit was increased from its
original size, so we actually do this dual-storage if the maximum length of
the string field is defined to be longer than 255 characters (so even if
you store a 1-character string in a field defined to be 256 characters
long, we will store it twice, once under the string, once in a Text field
(which is stored separately from the row contents)).

(2) Google AppEngine's BigTable datastore has a 1,000,000 byte limit to the
maximum length of any row in any data table.

(3) MySQL databases have a 65536-byte limit to the maximum length of any
row in any data table.

(4) PostgreSQL databases have a 1-megabyte limit to the maximum length of
any row in any data table.

Because of (2), (3), and (4), when ODK Aggregate creates database tables in
which to store a form, it needs to count the maximum number of characters
and bytes that may be needed to store the value of each field (column in
the table), and use that maximum length to split the fields across as many
backing database tables as required by these limitations.

Furthermore, each database table has a few columns of metadata -- a maximum
of about 800 bytes.

For MySQL, the impact of this is dramatic: If you defined every string
field to be 16,000 bytes long, that means that MySQL databases would only
be able to store the answers to 4 string fields per database table (because
of the 65,536 byte limit on each row). So if you had 100 questions, you
would end up having 25 database tables to store all those answers. If you
only had 255 bytes per string field, then you would only need 1 table.
(Google AppEngine and PostgreSQL databases would require 2 tables vs 1
table).

And the more tables you have, the more queries need to be issued across all
those tables to retrieve your data.

Finally, because filtering logic across tables is nearly impossible to do
with Google AppEngine (you have to do all the join logic within your
application layer), there is a limitation within ODK Aggregate that when we
define value filters, only the filters on the 1st backing database table
are applied.

Mitch

··· On Tue, Feb 19, 2013 at 1:39 AM, Joseph wrote:

Hi Mitch,

I have question regarding worst data storage efficiency within data store,
if I increase string length more than 255 char. Could you please explain a
bit about what you mean by storage efficiency? Why can't you assign 16000
UTF-8 characters as default, this certainly helps surveys based on
qualitative (narrative) data?

Thank you
Joseph

On Wednesday, February 13, 2013 12:58:55 AM UTC+5:30, Mitch wrote:

Wrong link -- see: http://opendatakit.org/help/**form-design/guidelines/http://opendatakit.org/help/form-design/guidelines/

Scroll down to Datastore String Length

The 255-character limit that can be changed.

Databases need to know the maximum amount of space to reserve for data.
We chose 255 as it is about a screenful of text (on a phone) and because
it is also the default used by most database software.

Mitch

On Tue, Feb 12, 2013 at 3:24 AM, ゴー・ニコライ nikol...@gmail.com wrote:

I think there's a 255-character limit default per field, then it gets
truncated. You may use length binding to set a bigger size.

http://opendatakit.org/help/**form-design/binding/http://opendatakit.org/help/form-design/binding/

/r
Nik

On Tuesday, February 12, 2013, Joseph wrote:

Hello all,

I was wondering about word limit for Text fields in ODK collect. So far
I noticed that maximum 45 words as limit and what ever the data which was
entered there after, was just vanished.
Is it normal that text fields has a limit in words? Any kind of help is
appreciated.

Thanks in advance

Joseph

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@**googlegroups.com
Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@**googlegroups.com

Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Hey Mitch,

Thanks a lot mate for this information. I appreciate it.

Joseph

Hi Mitch,

Is there a trick to add string size in XLSform without editing the XML form?

Best,
piqo

··· On Tuesday, February 19, 2013 at 7:44:35 PM UTC+1, Mitch Sundt wrote: > > The maximum field width you set does not affect storage efficiency all > that much (less than 2% overhead beyond what is required to represent the > actual data value). > > I believe pretty much all modern databases, when they store string fields, > store them as length-delimited fields, so they use only a few bytes more > than the number of bytes required to store the string itself. > > That said, there are some key limitations to understand: > > (1) Google AppEngine's BigTable only stores strings up to 500 unicode > characters long. If you specify a string of 16000 UTF-8 characters, we > create a shadow Text datatype to store the entire string (for up to 1 > megabyte of text). This is all transparent to the user, but impacts > efficiency. In fact, this 500-character limit was increased from its > original size, so we actually do this dual-storage if the maximum length of > the string field is defined to be longer than 255 characters (so even if > you store a 1-character string in a field defined to be 256 characters > long, we will store it twice, once under the string, once in a Text field > (which is stored separately from the row contents)). > > (2) Google AppEngine's BigTable datastore has a 1,000,000 byte limit to > the maximum length of any row in any data table. > > (3) MySQL databases have a 65536-byte limit to the maximum length of any > row in any data table. > > (4) PostgreSQL databases have a 1-megabyte limit to the maximum length of > any row in any data table. > > Because of (2), (3), and (4), when ODK Aggregate creates database tables > in which to store a form, it needs to count the maximum number of > characters and bytes that may be needed to store the value of each field > (column in the table), and use that maximum length to split the fields > across as many backing database tables as required by these limitations. > > Furthermore, each database table has a few columns of metadata -- a > maximum of about 800 bytes. > > For MySQL, the impact of this is dramatic: If you defined every string > field to be 16,000 bytes long, that means that MySQL databases would only > be able to store the answers to 4 string fields per database table (because > of the 65,536 byte limit on each row). So if you had 100 questions, you > would end up having *25 database tables* to store all those answers. If > you only had 255 bytes per string field, then you would only need *1 > table.* (Google AppEngine and PostgreSQL databases would require 2 > tables vs 1 table). > > And the more tables you have, the more queries need to be issued across > all those tables to retrieve your data. > > Finally, because filtering logic across tables is nearly impossible to do > with Google AppEngine (you have to do all the join logic within your > application layer), there is a limitation within ODK Aggregate that when we > define value filters, only the filters on the 1st backing database table > are applied. > > Mitch > > On Tue, Feb 19, 2013 at 1:39 AM, Joseph <gnano...@gmail.com > wrote: > >> Hi Mitch, >> >> I have question regarding worst data storage efficiency within data >> store, if I increase string length more than 255 char. Could you please >> explain a bit about what you mean by storage efficiency? Why can't you >> assign 16000 UTF-8 characters as default, this certainly helps surveys >> based on qualitative (narrative) data? >> >> Thank you >> Joseph >> >> >> >> On Wednesday, February 13, 2013 12:58:55 AM UTC+5:30, Mitch wrote: >> >>> Wrong link -- see: http://opendatakit.org/help/form-design/guidelines/ >>> Scroll down to Datastore String Length >>> >>> The 255-character limit that can be changed. >>> >>> Databases need to know the maximum amount of space to reserve for data. >>> We chose 255 as it is about a screenful of text (on a phone) and because >>> it is also the default used by most database software. >>> >>> Mitch >>> >>> On Tue, Feb 12, 2013 at 3:24 AM, ゴー・ニコライ wrote: >>> >>>> I think there's a 255-character limit default per field, then it gets >>>> truncated. You may use length binding to set a bigger size. >>>> >>>> http://opendatakit.org/help/form-design/binding/ >>>> >>>> /r >>>> Nik >>>> >>>> >>>> On Tuesday, February 12, 2013, Joseph wrote: >>>> >>>>> Hello all, >>>>> >>>>> I was wondering about word limit for Text fields in ODK collect. So >>>>> far I noticed that maximum 45 words as limit and what ever the data which >>>>> was entered there after, was just vanished. >>>>> Is it normal that text fields has a limit in words? Any kind of help >>>>> is appreciated. >>>>> >>>>> Thanks in advance >>>>> >>>>> Joseph >>>>> >>>>> -- >>>>> -- >>>>> Post: opendatakit@googlegroups.com >>>>> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >>>>> Options: http://groups.google.com/group/opendatakit?hl=en >>>>> >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "ODK Community" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to opendatakit+unsubscribe@googlegroups.com. >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>>> >>>>> >>>> -- >>>> -- >>>> Post: opend...@googlegroups.com >>>> Unsubscribe: opendatakit...@googlegroups.com >>>> >>>> Options: http://groups.google.com/group/opendatakit?hl=en >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "ODK Community" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to opendatakit...@googlegroups.com. >>>> >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>>> >>>> >>> >>> >>> >>> -- >>> Mitch Sundt >>> Software Engineer >>> University of Washington >>> mitche...@gmail.com >> >> -- >> -- >> Post: opend...@googlegroups.com >> Unsubscribe: opendatakit...@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "ODK Community" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit...@googlegroups.com . >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com

To my knowledge, that feature has not been added to XLSForm.

Editing the XML is the only way to do it.

··· On Tue, May 12, 2015 at 1:15 AM, El Piqo wrote:

Hi Mitch,

Is there a trick to add string size in XLSform without editing the XML
form?

Best,
piqo

On Tuesday, February 19, 2013 at 7:44:35 PM UTC+1, Mitch Sundt wrote:

The maximum field width you set does not affect storage efficiency all
that much (less than 2% overhead beyond what is required to represent the
actual data value).

I believe pretty much all modern databases, when they store string
fields, store them as length-delimited fields, so they use only a few bytes
more than the number of bytes required to store the string itself.

That said, there are some key limitations to understand:

(1) Google AppEngine's BigTable only stores strings up to 500 unicode
characters long. If you specify a string of 16000 UTF-8 characters, we
create a shadow Text datatype to store the entire string (for up to 1
megabyte of text). This is all transparent to the user, but impacts
efficiency. In fact, this 500-character limit was increased from its
original size, so we actually do this dual-storage if the maximum length of
the string field is defined to be longer than 255 characters (so even if
you store a 1-character string in a field defined to be 256 characters
long, we will store it twice, once under the string, once in a Text field
(which is stored separately from the row contents)).

(2) Google AppEngine's BigTable datastore has a 1,000,000 byte limit to
the maximum length of any row in any data table.

(3) MySQL databases have a 65536-byte limit to the maximum length of any
row in any data table.

(4) PostgreSQL databases have a 1-megabyte limit to the maximum length of
any row in any data table.

Because of (2), (3), and (4), when ODK Aggregate creates database tables
in which to store a form, it needs to count the maximum number of
characters and bytes that may be needed to store the value of each field
(column in the table), and use that maximum length to split the fields
across as many backing database tables as required by these limitations.

Furthermore, each database table has a few columns of metadata -- a
maximum of about 800 bytes.

For MySQL, the impact of this is dramatic: If you defined every string
field to be 16,000 bytes long, that means that MySQL databases would only
be able to store the answers to 4 string fields per database table (because
of the 65,536 byte limit on each row). So if you had 100 questions, you
would end up having 25 database tables to store all those answers. If
you only had 255 bytes per string field, then you would only need 1
table.
(Google AppEngine and PostgreSQL databases would require 2
tables vs 1 table).

And the more tables you have, the more queries need to be issued across
all those tables to retrieve your data.

Finally, because filtering logic across tables is nearly impossible to do
with Google AppEngine (you have to do all the join logic within your
application layer), there is a limitation within ODK Aggregate that when we
define value filters, only the filters on the 1st backing database table
are applied.

Mitch

On Tue, Feb 19, 2013 at 1:39 AM, Joseph gnano...@gmail.com wrote:

Hi Mitch,

I have question regarding worst data storage efficiency within data
store, if I increase string length more than 255 char. Could you please
explain a bit about what you mean by storage efficiency? Why can't you
assign 16000 UTF-8 characters as default, this certainly helps surveys
based on qualitative (narrative) data?

Thank you
Joseph

On Wednesday, February 13, 2013 12:58:55 AM UTC+5:30, Mitch wrote:

Wrong link -- see: http://opendatakit.org/help/form-design/guidelines/

Scroll down to Datastore String Length

The 255-character limit that can be changed.

Databases need to know the maximum amount of space to reserve for data.
We chose 255 as it is about a screenful of text (on a phone) and
because it is also the default used by most database software.

Mitch

On Tue, Feb 12, 2013 at 3:24 AM, ゴー・ニコライ nikol...@gmail.com wrote:

I think there's a 255-character limit default per field, then it gets
truncated. You may use length binding to set a bigger size.

http://opendatakit.org/help/form-design/binding/

/r
Nik

On Tuesday, February 12, 2013, Joseph wrote:

Hello all,

I was wondering about word limit for Text fields in ODK collect. So
far I noticed that maximum 45 words as limit and what ever the data which
was entered there after, was just vanished.
Is it normal that text fields has a limit in words? Any kind of help
is appreciated.

Thanks in advance

Joseph

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Hi everyone ! is there a way to recover truncated text ?
Thanks

Hi, has any of this info changed? Any way to edit field length in XLS Form?
All of our forms are written in XLS, I don't know XML and I'd love an
alternative to learning it and recreating it all just to adjust the limit
of one field...!

Thanks!

··· On Tuesday, May 12, 2015 at 9:19:56 PM UTC+5:30, Mitch Sundt wrote: > > To my knowledge, that feature has not been added to XLSForm. > > Editing the XML is the only way to do it. > > > On Tue, May 12, 2015 at 1:15 AM, El Piqo <piq...@gmail.com > wrote: > >> Hi Mitch, >> >> Is there a trick to add string size in XLSform without editing the XML >> form? >> >> Best, >> piqo >> >> On Tuesday, February 19, 2013 at 7:44:35 PM UTC+1, Mitch Sundt wrote: >>> >>> The maximum field width you set does not affect storage efficiency all >>> that much (less than 2% overhead beyond what is required to represent the >>> actual data value). >>> >>> I believe pretty much all modern databases, when they store string >>> fields, store them as length-delimited fields, so they use only a few bytes >>> more than the number of bytes required to store the string itself. >>> >>> That said, there are some key limitations to understand: >>> >>> (1) Google AppEngine's BigTable only stores strings up to 500 unicode >>> characters long. If you specify a string of 16000 UTF-8 characters, we >>> create a shadow Text datatype to store the entire string (for up to 1 >>> megabyte of text). This is all transparent to the user, but impacts >>> efficiency. In fact, this 500-character limit was increased from its >>> original size, so we actually do this dual-storage if the maximum length of >>> the string field is defined to be longer than 255 characters (so even if >>> you store a 1-character string in a field defined to be 256 characters >>> long, we will store it twice, once under the string, once in a Text field >>> (which is stored separately from the row contents)). >>> >>> (2) Google AppEngine's BigTable datastore has a 1,000,000 byte limit to >>> the maximum length of any row in any data table. >>> >>> (3) MySQL databases have a 65536-byte limit to the maximum length of any >>> row in any data table. >>> >>> (4) PostgreSQL databases have a 1-megabyte limit to the maximum length >>> of any row in any data table. >>> >>> Because of (2), (3), and (4), when ODK Aggregate creates database tables >>> in which to store a form, it needs to count the maximum number of >>> characters and bytes that may be needed to store the value of each field >>> (column in the table), and use that maximum length to split the fields >>> across as many backing database tables as required by these limitations. >>> >>> Furthermore, each database table has a few columns of metadata -- a >>> maximum of about 800 bytes. >>> >>> For MySQL, the impact of this is dramatic: If you defined every string >>> field to be 16,000 bytes long, that means that MySQL databases would only >>> be able to store the answers to 4 string fields per database table (because >>> of the 65,536 byte limit on each row). So if you had 100 questions, you >>> would end up having *25 database tables* to store all those answers. If >>> you only had 255 bytes per string field, then you would only need *1 >>> table.* (Google AppEngine and PostgreSQL databases would require 2 >>> tables vs 1 table). >>> >>> And the more tables you have, the more queries need to be issued across >>> all those tables to retrieve your data. >>> >>> Finally, because filtering logic across tables is nearly impossible to >>> do with Google AppEngine (you have to do all the join logic within your >>> application layer), there is a limitation within ODK Aggregate that when we >>> define value filters, only the filters on the 1st backing database table >>> are applied. >>> >>> Mitch >>> >>> On Tue, Feb 19, 2013 at 1:39 AM, Joseph wrote: >>> >>>> Hi Mitch, >>>> >>>> I have question regarding worst data storage efficiency within data >>>> store, if I increase string length more than 255 char. Could you please >>>> explain a bit about what you mean by storage efficiency? Why can't you >>>> assign 16000 UTF-8 characters as default, this certainly helps surveys >>>> based on qualitative (narrative) data? >>>> >>>> Thank you >>>> Joseph >>>> >>>> >>>> >>>> On Wednesday, February 13, 2013 12:58:55 AM UTC+5:30, Mitch wrote: >>>> >>>>> Wrong link -- see: http://opendatakit.org/help/form-design/guidelines/ >>>>> >>>>> Scroll down to Datastore String Length >>>>> >>>>> The 255-character limit that can be changed. >>>>> >>>>> Databases need to know the maximum amount of space to reserve for >>>>> data. >>>>> We chose 255 as it is about a screenful of text (on a phone) and >>>>> because it is also the default used by most database software. >>>>> >>>>> Mitch >>>>> >>>>> On Tue, Feb 12, 2013 at 3:24 AM, ゴー・ニコライ wrote: >>>>> >>>>>> I think there's a 255-character limit default per field, then it gets >>>>>> truncated. You may use length binding to set a bigger size. >>>>>> >>>>>> http://opendatakit.org/help/form-design/binding/ >>>>>> >>>>>> /r >>>>>> Nik >>>>>> >>>>>> >>>>>> On Tuesday, February 12, 2013, Joseph wrote: >>>>>> >>>>>>> Hello all, >>>>>>> >>>>>>> I was wondering about word limit for Text fields in ODK collect. So >>>>>>> far I noticed that maximum 45 words as limit and what ever the data which >>>>>>> was entered there after, was just vanished. >>>>>>> Is it normal that text fields has a limit in words? Any kind of help >>>>>>> is appreciated. >>>>>>> >>>>>>> Thanks in advance >>>>>>> >>>>>>> Joseph >>>>>>> >>>>>>> -- >>>>>>> -- >>>>>>> Post: opendatakit@googlegroups.com >>>>>>> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >>>>>>> Options: http://groups.google.com/group/opendatakit?hl=en >>>>>>> >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "ODK Community" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to opendatakit+unsubscribe@googlegroups.com. >>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>>> -- >>>>>> Post: opend...@googlegroups.com >>>>>> Unsubscribe: opendatakit...@googlegroups.com >>>>>> >>>>>> Options: http://groups.google.com/group/opendatakit?hl=en >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "ODK Community" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to opendatakit...@googlegroups.com. >>>>>> >>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Mitch Sundt >>>>> Software Engineer >>>>> University of Washington >>>>> mitche...@gmail.com >>>> >>>> -- >>>> -- >>>> Post: opend...@googlegroups.com >>>> Unsubscribe: opendatakit...@googlegroups.com >>>> Options: http://groups.google.com/group/opendatakit?hl=en >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "ODK Community" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to opendatakit...@googlegroups.com. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>>> >>>> >>> >>> >>> >>> -- >>> Mitch Sundt >>> Software Engineer >>> University of Washington >>> mitche...@gmail.com >>> >> -- >> -- >> Post: opend...@googlegroups.com >> Unsubscribe: opendatakit...@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "ODK Community" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit...@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com >

Unless you have deleted the submitted forms on your devices, each device
retains all the data that was sent to the server.

You can recover your data from your devices.

e.g., using ODK Briefcase or if you create a new formVersion with the
longer length fields, you can upload that to a new server (the longer field
lengths will not be applied to an existing data table). And then redirect
all your clients to point to that new server and long-press on the Send
Forms screen to show all sent and unsent forms, then select all and send to
that new server.

For either mechanism, it is probably not something you want to trust to
your data collectors.

··· On Thu, Sep 10, 2015 at 3:20 PM, wrote:

Hi everyone ! is there a way to recover truncated text ?
Thanks

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

To my knowledge, the XLSForm generator has not been modified to allow you
to specify field length.

··· On Wed, Dec 9, 2015 at 1:54 AM, Jordan Levinson wrote:

Hi, has any of this info changed? Any way to edit field length in XLS
Form? All of our forms are written in XLS, I don't know XML and I'd love an
alternative to learning it and recreating it all just to adjust the limit
of one field...!

Thanks!

On Tuesday, May 12, 2015 at 9:19:56 PM UTC+5:30, Mitch Sundt wrote:

To my knowledge, that feature has not been added to XLSForm.

Editing the XML is the only way to do it.

On Tue, May 12, 2015 at 1:15 AM, El Piqo piq...@gmail.com wrote:

Hi Mitch,

Is there a trick to add string size in XLSform without editing the XML
form?

Best,
piqo

On Tuesday, February 19, 2013 at 7:44:35 PM UTC+1, Mitch Sundt wrote:

The maximum field width you set does not affect storage efficiency all
that much (less than 2% overhead beyond what is required to represent the
actual data value).

I believe pretty much all modern databases, when they store string
fields, store them as length-delimited fields, so they use only a few bytes
more than the number of bytes required to store the string itself.

That said, there are some key limitations to understand:

(1) Google AppEngine's BigTable only stores strings up to 500 unicode
characters long. If you specify a string of 16000 UTF-8 characters, we
create a shadow Text datatype to store the entire string (for up to 1
megabyte of text). This is all transparent to the user, but impacts
efficiency. In fact, this 500-character limit was increased from its
original size, so we actually do this dual-storage if the maximum length of
the string field is defined to be longer than 255 characters (so even if
you store a 1-character string in a field defined to be 256 characters
long, we will store it twice, once under the string, once in a Text field
(which is stored separately from the row contents)).

(2) Google AppEngine's BigTable datastore has a 1,000,000 byte limit to
the maximum length of any row in any data table.

(3) MySQL databases have a 65536-byte limit to the maximum length of
any row in any data table.

(4) PostgreSQL databases have a 1-megabyte limit to the maximum length
of any row in any data table.

Because of (2), (3), and (4), when ODK Aggregate creates database
tables in which to store a form, it needs to count the maximum number of
characters and bytes that may be needed to store the value of each field
(column in the table), and use that maximum length to split the fields
across as many backing database tables as required by these limitations.

Furthermore, each database table has a few columns of metadata -- a
maximum of about 800 bytes.

For MySQL, the impact of this is dramatic: If you defined every string
field to be 16,000 bytes long, that means that MySQL databases would only
be able to store the answers to 4 string fields per database table (because
of the 65,536 byte limit on each row). So if you had 100 questions, you
would end up having 25 database tables to store all those answers.
If you only had 255 bytes per string field, then you would only need 1
table.
(Google AppEngine and PostgreSQL databases would require 2
tables vs 1 table).

And the more tables you have, the more queries need to be issued across
all those tables to retrieve your data.

Finally, because filtering logic across tables is nearly impossible to
do with Google AppEngine (you have to do all the join logic within your
application layer), there is a limitation within ODK Aggregate that when we
define value filters, only the filters on the 1st backing database table
are applied.

Mitch

On Tue, Feb 19, 2013 at 1:39 AM, Joseph gnano...@gmail.com wrote:

Hi Mitch,

I have question regarding worst data storage efficiency within data
store, if I increase string length more than 255 char. Could you please
explain a bit about what you mean by storage efficiency? Why can't you
assign 16000 UTF-8 characters as default, this certainly helps surveys
based on qualitative (narrative) data?

Thank you
Joseph

On Wednesday, February 13, 2013 12:58:55 AM UTC+5:30, Mitch wrote:

Wrong link -- see: http://opendatakit.org/help/
form-design/guidelines/
Scroll down to Datastore String Length

The 255-character limit that can be changed.

Databases need to know the maximum amount of space to reserve for
data.
We chose 255 as it is about a screenful of text (on a phone) and
because it is also the default used by most database software.

Mitch

On Tue, Feb 12, 2013 at 3:24 AM, ゴー・ニコライ nikol...@gmail.com wrote:

I think there's a 255-character limit default per field, then it
gets truncated. You may use length binding to set a bigger size.

http://opendatakit.org/help/form-design/binding/

/r
Nik

On Tuesday, February 12, 2013, Joseph wrote:

Hello all,

I was wondering about word limit for Text fields in ODK collect. So
far I noticed that maximum 45 words as limit and what ever the data which
was entered there after, was just vanished.
Is it normal that text fields has a limit in words? Any kind of
help is appreciated.

Thanks in advance

Joseph

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Jordan,

You can add the following columns to your XLSForm instance::attribute,
bind::attribute, body::attribute. These three allow you to pass config
information from the XLSForm to the XML.

In this case, add a "bind::attribute" column and in the row of the
question, add "odk:length=600". When you convert, the length
information should be in the XML. I haven't tried this myself, so test
first.

Yaw

··· -- Need ODK consultants? Nafundi provides form design, server setup, in-field training, and software development for ODK. Go to https://nafundi.com to get started.

On Wed, Dec 9, 2015 at 7:06 PM, Mitch Sundt mitchellsundt@gmail.com wrote:

To my knowledge, the XLSForm generator has not been modified to allow you to
specify field length.

On Wed, Dec 9, 2015 at 1:54 AM, Jordan Levinson jord.levinson@gmail.com wrote:

Hi, has any of this info changed? Any way to edit field length in XLS
Form? All of our forms are written in XLS, I don't know XML and I'd love an
alternative to learning it and recreating it all just to adjust the limit of
one field...!

Thanks!

On Tuesday, May 12, 2015 at 9:19:56 PM UTC+5:30, Mitch Sundt wrote:

To my knowledge, that feature has not been added to XLSForm.

Editing the XML is the only way to do it.

On Tue, May 12, 2015 at 1:15 AM, El Piqo piq...@gmail.com wrote:

Hi Mitch,

Is there a trick to add string size in XLSform without editing the XML
form?

Best,
piqo

On Tuesday, February 19, 2013 at 7:44:35 PM UTC+1, Mitch Sundt wrote:

The maximum field width you set does not affect storage efficiency all
that much (less than 2% overhead beyond what is required to represent the
actual data value).

I believe pretty much all modern databases, when they store string
fields, store them as length-delimited fields, so they use only a few bytes
more than the number of bytes required to store the string itself.

That said, there are some key limitations to understand:

(1) Google AppEngine's BigTable only stores strings up to 500 unicode
characters long. If you specify a string of 16000 UTF-8 characters, we
create a shadow Text datatype to store the entire string (for up to 1
megabyte of text). This is all transparent to the user, but impacts
efficiency. In fact, this 500-character limit was increased from its
original size, so we actually do this dual-storage if the maximum length of
the string field is defined to be longer than 255 characters (so even if
you store a 1-character string in a field defined to be 256 characters long,
we will store it twice, once under the string, once in a Text field (which
is stored separately from the row contents)).

(2) Google AppEngine's BigTable datastore has a 1,000,000 byte limit to
the maximum length of any row in any data table.

(3) MySQL databases have a 65536-byte limit to the maximum length of
any row in any data table.

(4) PostgreSQL databases have a 1-megabyte limit to the maximum length
of any row in any data table.

Because of (2), (3), and (4), when ODK Aggregate creates database
tables in which to store a form, it needs to count the maximum number of
characters and bytes that may be needed to store the value of each field
(column in the table), and use that maximum length to split the fields
across as many backing database tables as required by these limitations.

Furthermore, each database table has a few columns of metadata -- a
maximum of about 800 bytes.

For MySQL, the impact of this is dramatic: If you defined every string
field to be 16,000 bytes long, that means that MySQL databases would only be
able to store the answers to 4 string fields per database table (because of
the 65,536 byte limit on each row). So if you had 100 questions, you would
end up having 25 database tables to store all those answers. If you only had
255 bytes per string field, then you would only need 1 table. (Google
AppEngine and PostgreSQL databases would require 2 tables vs 1 table).

And the more tables you have, the more queries need to be issued across
all those tables to retrieve your data.

Finally, because filtering logic across tables is nearly impossible to
do with Google AppEngine (you have to do all the join logic within your
application layer), there is a limitation within ODK Aggregate that when we
define value filters, only the filters on the 1st backing database table are
applied.

Mitch

On Tue, Feb 19, 2013 at 1:39 AM, Joseph gnano...@gmail.com wrote:

Hi Mitch,

I have question regarding worst data storage efficiency within data
store, if I increase string length more than 255 char. Could you please
explain a bit about what you mean by storage efficiency? Why can't you
assign 16000 UTF-8 characters as default, this certainly helps surveys based
on qualitative (narrative) data?

Thank you
Joseph

On Wednesday, February 13, 2013 12:58:55 AM UTC+5:30, Mitch wrote:

Wrong link -- see:
http://opendatakit.org/help/form-design/guidelines/
Scroll down to Datastore String Length

The 255-character limit that can be changed.

Databases need to know the maximum amount of space to reserve for
data.
We chose 255 as it is about a screenful of text (on a phone) and
because it is also the default used by most database software.

Mitch

On Tue, Feb 12, 2013 at 3:24 AM, ゴー・ニコライ nikol...@gmail.com wrote:

I think there's a 255-character limit default per field, then it
gets truncated. You may use length binding to set a bigger size.

http://opendatakit.org/help/form-design/binding/

/r
Nik

On Tuesday, February 12, 2013, Joseph wrote:

Hello all,

I was wondering about word limit for Text fields in ODK collect. So
far I noticed that maximum 45 words as limit and what ever the data which
was entered there after, was just vanished.
Is it normal that text fields has a limit in words? Any kind of
help is appreciated.

Thanks in advance

Joseph

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I forgot about this recent addition to XLSForm.

Unfortunately, it still requires a final hand-editing step, but it is
considerably easier.

The steps are detailed here:

https://opendatakit.org/help/form-design/guidelines/#datastore_string_length

THANKS Mitch and Yaw!!! So helpful.

Best,

··· On Saturday, December 12, 2015 at 1:26:45 AM UTC+5:30, Mitch Sundt wrote: > > I forgot about this recent addition to XLSForm. > > Unfortunately, it still requires a final hand-editing step, but it is > considerably easier. > > The steps are detailed here: > > > https://opendatakit.org/help/form-design/guidelines/#datastore_string_length > > >

i am searching about almost same issue , for text field the limitation 256 character only above that wont be catch
but i am searching for that if we can have unlimited character for textfield

@Khalil you can increase the limit but you can't specify unlimited lengths. See ODK collect text field for some context on why that is.

tnx great info and well explanation

1 Like

tnx @LN much appreciated