Formdef.json spec / documentation

Where is the documentation for the current surveyJSON / formdef.json spec?
I can't find it anywhere on the Github page.

We don't currently have documentation for the formDef.json. Ideally, we
would like people to focus on the creation of the Excel spreadsheet and use
the XLSXConverter documentation (
https://opendatakit.org/use/2_0_tools/odk-application-designer-2-0-rev126/odk-xlsxconverter-2-0-rev126/)
as we may still make changes to the structure of the formDef.json.

Clarice

··· On Fri, Oct 9, 2015 at 6:23 PM, Gregor MacLennan < gmaclennan@digital-democracy.org> wrote:

Where is the documentation for the current surveyJSON / formdef.json spec?
I can't find it anywhere on the Github page.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

It would be useful to have some kind of open discussion about the
formDef.json spec. I'm working on form design and data collection tools and
would like to use the same/similar form definition language / not repeat
mistakes / work towards compatibility / collaborate on thinking through the
best structure.

Gregor

··· On Monday, October 12, 2015 at 11:50:19 AM UTC-6, clarice larson wrote: > > We don't currently have documentation for the formDef.json. Ideally, we > would like people to focus on the creation of the Excel spreadsheet and use > the XLSXConverter documentation ( > https://opendatakit.org/use/2_0_tools/odk-application-designer-2-0-rev126/odk-xlsxconverter-2-0-rev126/) > as we may still make changes to the structure of the formDef.json. > > Clarice > > On Fri, Oct 9, 2015 at 6:23 PM, Gregor MacLennan < gmacl...@digital-democracy.org > wrote: > >> Where is the documentation for the current surveyJSON / formdef.json >> spec? I can't find it anywhere on the Github page. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "ODK Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit-developers+unsubscribe@googlegroups.com >> . >> For more options, visit https://groups.google.com/d/optout. >> > >

Below is a brief summary of the content of the formDef.json.

At the moment, the sample forms and code are the reference implementation.
As we need to make structural changes, we re-generate all of the
formDef.json files from the XLSX files.

Hence why we refer people to the XLSX file format documentation.

i.e., don't trust that what follows will exist as-is into the release
version.

··· ================================================ formDef.json generated by XLSXConverter contains an object with 2 entries:

{ xlsx: {...},
specification: {...} }

The xlsx object holds a summary of the contents of the xlsx file such that
you could reconstruct that features of that file from this section of the
formDef.json.

This currently consists of a key for each sheet in the original XLSX, and
each of those being a list of objects, one for each non-empty row in that
sheet, with the keys being the column headings on that sheet.

We record this for trace-ability back to the original XLSX file content.
And to avoid the "I've got the XML but lost my XLSX file; do you have a
tool to reconstruct the XLSX file from that?" problem.

Presumably, if users created a form definition using some other tool e.g.,
"Build 2.0", the file would contain a 'build' key with some sort of source
description for that form creation tool that could allow reconstruction of
that source format.


The interpreter of the formDef.json file ONLY pays attention to the
'specification' key.

It should be considered improper for the runtime engine to look outside of
the 'specification' section of this file.


The 'specification' key has this object structure:

{ 'column_types': { },
'settings' : {},
'choices' : {},
'queries' : {},
'calculates' : {},
'model' : {},
'section_names' : ["initial", "survey", ...],
'sections' : { initial: {}, survey: {}, ... }
}

'column_types' identifies the object keys within a prompt or screen
object that have special interpretations by the runtime environment.

e.g., "required": "formula"

Specifies that when the runtime environment finds that key, its value
should be wrapped and converted to a function() {} object such that it can
be evaluated when executing the form and the evaluation context contains
some helper functions, the calculate expressions, etc.

We have others, such as formula(context) which would wrap the value in a
function(context) {}, etc.

The list and meanings of these is subject to change.

The intent is that a user could add a 'column_types' sheet to their XLSX
file and declare the treatment for a given column heading, and the runtime
environment would apply that treatment. So if you wanted a
'extraConstraint' that was a formula, you could declare that on the XLSX
sheet, then supply content under that heading in any section sheet, and it
would pass through and become available in the runtime environment.

i.e., the XLSX format is extensible without changing the converter or the
runtime.

'settings' is a restructuring of the content of the settings XLSX sheet,
with omitted entries generated from other entries that were provided. I.e.,
you don't need to specify tableId in the XLSX file; the converter will
generate it if it wasn't specified, and it will always be present in this
settings object.

'choices' is a restructuring of the choices sheet.

'queries' is a restructuring of the queries sheet.

'calculates' is a restructuring of the calculates sheet.

'model' is being deprecated. It is a synthesis of all the field names
used in the survey and subsection sheets, and the content of the
'prompt_types' and 'model' sheets (if present). It defines the storage
model of the form.

When the formId and tableId are identical, the 'model' sheet is written out
into the definition.csv and the properties.csv (along with other
information).

We are striving to remove 'model' from the 'specification' and rely on
those other files to fully describe the storage model. This and other
changes should significantly simplify the underlying code of the JS runtime
environment.

'section_names' is a list of the sections in the form. All forms contain
an 'initial' and a 'survey' section. The form launches into the 'initial'
section which then launches into the 'survey' section. The XLSXconverter
generates the 'initial' section if the XLSX file does not provide one.

'sections' is an object with one key for each section name. Each object
describes the content of that corresponding section.

The structure of a section object is likely to change. We want to perform
syntax and semantic tests in the XLSXconverter and provide auto-generated
content where possible.

The current structure is:

{ "section_name": name,
"nested_sections": {},
"reachable_sections" : {},
"prompts": ,
"validation_tag_map": {},
"operations": ,
"branch_label_map": {},
}

*nested_sections *are the sections that are directly called via 'do
section'.

*reachable_sections *is the recursive closure of nested_sections. It is an
error for reachable_sections to contain this section (any possible means to
invoke 'do section' within this section or any section is can call is not
allowed).

prompts is a list of the prompts in the section

*validation_tag_map *tracks the prompts that have constraints linked to a
given validation tag

operations is a list of processing steps that are executed to traverse
this section. It is the 'assembly code' generated by the XLSX sheet.

*branch_label_map *is a map of the branch labels into the operations list.

=================================
I am probably missing some other object keys needed to support custom
prompts and other features.

Below this level, it is best to look at the content of a formDef.json

We are likely to be changing some of this as we refine the functionality
that the runtime JS environment implements.

On Wed, Oct 21, 2015 at 2:48 PM, Gregor MacLennan < gmaclennan@digital-democracy.org> wrote:

It would be useful to have some kind of open discussion about the
formDef.json spec. I'm working on form design and data collection tools and
would like to use the same/similar form definition language / not repeat
mistakes / work towards compatibility / collaborate on thinking through the
best structure.

Gregor

On Monday, October 12, 2015 at 11:50:19 AM UTC-6, clarice larson wrote:

We don't currently have documentation for the formDef.json. Ideally, we
would like people to focus on the creation of the Excel spreadsheet and use
the XLSXConverter documentation (
https://opendatakit.org/use/2_0_tools/odk-application-designer-2-0-rev126/odk-xlsxconverter-2-0-rev126/)
as we may still make changes to the structure of the formDef.json.

Clarice

On Fri, Oct 9, 2015 at 6:23 PM, Gregor MacLennan < gmacl...@digital-democracy.org> wrote:

Where is the documentation for the current surveyJSON / formdef.json
spec? I can't find it anywhere on the Github page.

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Now on the wiki if folks want to make edits.

Yaw

··· -- Need ODK consultants? https://nafundi.com provides form design, server setup, in-field training, and software development for ODK.

On Thu, Oct 22, 2015 at 8:38 AM, Mitch Sundt mitchellsundt@gmail.com wrote:

Below is a brief summary of the content of the formDef.json.

At the moment, the sample forms and code are the reference implementation.
As we need to make structural changes, we re-generate all of the
formDef.json files from the XLSX files.

Hence why we refer people to the XLSX file format documentation.

i.e., don't trust that what follows will exist as-is into the release
version.

================================================
formDef.json generated by XLSXConverter contains an object with 2 entries:

{ xlsx: {...},
specification: {...} }

The xlsx object holds a summary of the contents of the xlsx file such that
you could reconstruct that features of that file from this section of the
formDef.json.

This currently consists of a key for each sheet in the original XLSX, and
each of those being a list of objects, one for each non-empty row in that
sheet, with the keys being the column headings on that sheet.

We record this for trace-ability back to the original XLSX file content. And
to avoid the "I've got the XML but lost my XLSX file; do you have a tool to
reconstruct the XLSX file from that?" problem.

Presumably, if users created a form definition using some other tool e.g.,
"Build 2.0", the file would contain a 'build' key with some sort of source
description for that form creation tool that could allow reconstruction of
that source format.


The interpreter of the formDef.json file ONLY pays attention to the
'specification' key.

It should be considered improper for the runtime engine to look outside of
the 'specification' section of this file.


The 'specification' key has this object structure:

{ 'column_types': { },
'settings' : {},
'choices' : {},
'queries' : {},
'calculates' : {},
'model' : {},
'section_names' : ["initial", "survey", ...],
'sections' : { initial: {}, survey: {}, ... }
}

'column_types' identifies the object keys within a prompt or screen object
that have special interpretations by the runtime environment.

e.g., "required": "formula"

Specifies that when the runtime environment finds that key, its value should
be wrapped and converted to a function() {} object such that it can be
evaluated when executing the form and the evaluation context contains some
helper functions, the calculate expressions, etc.

We have others, such as formula(context) which would wrap the value in a
function(context) {}, etc.

The list and meanings of these is subject to change.

The intent is that a user could add a 'column_types' sheet to their XLSX
file and declare the treatment for a given column heading, and the runtime
environment would apply that treatment. So if you wanted a 'extraConstraint'
that was a formula, you could declare that on the XLSX sheet, then supply
content under that heading in any section sheet, and it would pass through
and become available in the runtime environment.

i.e., the XLSX format is extensible without changing the converter or the
runtime.

'settings' is a restructuring of the content of the settings XLSX sheet,
with omitted entries generated from other entries that were provided. I.e.,
you don't need to specify tableId in the XLSX file; the converter will
generate it if it wasn't specified, and it will always be present in this
settings object.

'choices' is a restructuring of the choices sheet.

'queries' is a restructuring of the queries sheet.

'calculates' is a restructuring of the calculates sheet.

'model' is being deprecated. It is a synthesis of all the field names used
in the survey and subsection sheets, and the content of the 'prompt_types'
and 'model' sheets (if present). It defines the storage model of the form.

When the formId and tableId are identical, the 'model' sheet is written out
into the definition.csv and the properties.csv (along with other
information).

We are striving to remove 'model' from the 'specification' and rely on those
other files to fully describe the storage model. This and other changes
should significantly simplify the underlying code of the JS runtime
environment.

'section_names' is a list of the sections in the form. All forms contain an
'initial' and a 'survey' section. The form launches into the 'initial'
section which then launches into the 'survey' section. The XLSXconverter
generates the 'initial' section if the XLSX file does not provide one.

'sections' is an object with one key for each section name. Each object
describes the content of that corresponding section.

The structure of a section object is likely to change. We want to perform
syntax and semantic tests in the XLSXconverter and provide auto-generated
content where possible.

The current structure is:

{ "section_name": name,
"nested_sections": {},
"reachable_sections" : {},
"prompts": ,
"validation_tag_map": {},
"operations": ,
"branch_label_map": {},
}

nested_sections are the sections that are directly called via 'do section'.

reachable_sections is the recursive closure of nested_sections. It is an
error for reachable_sections to contain this section (any possible means to
invoke 'do section' within this section or any section is can call is not
allowed).

prompts is a list of the prompts in the section

validation_tag_map tracks the prompts that have constraints linked to a
given validation tag

operations is a list of processing steps that are executed to traverse this
section. It is the 'assembly code' generated by the XLSX sheet.

branch_label_map is a map of the branch labels into the operations list.

=================================
I am probably missing some other object keys needed to support custom
prompts and other features.

Below this level, it is best to look at the content of a formDef.json

We are likely to be changing some of this as we refine the functionality
that the runtime JS environment implements.

On Wed, Oct 21, 2015 at 2:48 PM, Gregor MacLennan gmaclennan@digital-democracy.org wrote:

It would be useful to have some kind of open discussion about the
formDef.json spec. I'm working on form design and data collection tools and
would like to use the same/similar form definition language / not repeat
mistakes / work towards compatibility / collaborate on thinking through the
best structure.

Gregor

On Monday, October 12, 2015 at 11:50:19 AM UTC-6, clarice larson wrote:

We don't currently have documentation for the formDef.json. Ideally, we
would like people to focus on the creation of the Excel spreadsheet and use
the XLSXConverter documentation
(https://opendatakit.org/use/2_0_tools/odk-application-designer-2-0-rev126/odk-xlsxconverter-2-0-rev126/)
as we may still make changes to the structure of the formDef.json.

Clarice

On Fri, Oct 9, 2015 at 6:23 PM, Gregor MacLennan gmacl...@digital-democracy.org wrote:

Where is the documentation for the current surveyJSON / formdef.json
spec? I can't find it anywhere on the Github page.

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Thanks so much for this Yaw and Mitch, this is super useful.

··· On Wednesday, October 21, 2015 at 10:51:14 PM UTC-6, Yaw Anokwa wrote: > > Now on the wiki if folks want to make edits. > > Yaw > -- > Need ODK consultants? https://nafundi.com provides form design, server > setup, in-field training, and software development for ODK. > > On Thu, Oct 22, 2015 at 8:38 AM, Mitch Sundt <mitche...@gmail.com > wrote: > > Below is a brief summary of the content of the formDef.json. > > > > At the moment, the sample forms and code are the reference > implementation. > > As we need to make structural changes, we re-generate all of the > > formDef.json files from the XLSX files. > > > > Hence why we refer people to the XLSX file format documentation. > > > > i.e., don't trust that what follows will exist as-is into the release > > version. > > > > ================================================ > > formDef.json generated by XLSXConverter contains an object with 2 > entries: > > > > { xlsx: {...}, > > specification: {...} } > > > > The xlsx object holds a summary of the contents of the xlsx file such > that > > you could reconstruct that features of that file from this section of > the > > formDef.json. > > > > This currently consists of a key for each sheet in the original XLSX, > and > > each of those being a list of objects, one for each non-empty row in > that > > sheet, with the keys being the column headings on that sheet. > > > > We record this for trace-ability back to the original XLSX file content. > And > > to avoid the "I've got the XML but lost my XLSX file; do you have a tool > to > > reconstruct the XLSX file from that?" problem. > > > > Presumably, if users created a form definition using some other tool > e.g., > > "Build 2.0", the file would contain a 'build' key with some sort of > source > > description for that form creation tool that could allow reconstruction > of > > that source format. > > > > ------------------- > > > > The interpreter of the formDef.json file ONLY pays attention to the > > 'specification' key. > > > > It should be considered improper for the runtime engine to look outside > of > > the 'specification' section of this file. > > > > ------------------- > > The 'specification' key has this object structure: > > > > { 'column_types': { }, > > 'settings' : {}, > > 'choices' : {}, > > 'queries' : {}, > > 'calculates' : {}, > > 'model' : {}, > > 'section_names' : ["initial", "survey", ...], > > 'sections' : { initial: {}, survey: {}, ... } > > } > > > > 'column_types' identifies the object keys within a prompt or screen > object > > that have special interpretations by the runtime environment. > > > > e.g., "required": "formula" > > > > Specifies that when the runtime environment finds that key, its value > should > > be wrapped and converted to a function() {} object such that it can be > > evaluated when executing the form and the evaluation context contains > some > > helper functions, the calculate expressions, etc. > > > > We have others, such as formula(context) which would wrap the value in a > > function(context) {}, etc. > > > > The list and meanings of these is subject to change. > > > > The intent is that a user could add a 'column_types' sheet to their XLSX > > file and declare the treatment for a given column heading, and the > runtime > > environment would apply that treatment. So if you wanted a > 'extraConstraint' > > that was a formula, you could declare that on the XLSX sheet, then > supply > > content under that heading in any section sheet, and it would pass > through > > and become available in the runtime environment. > > > > i.e., the XLSX format is extensible without changing the converter or > the > > runtime. > > > > 'settings' is a restructuring of the content of the settings XLSX sheet, > > with omitted entries generated from other entries that were provided. > I.e., > > you don't need to specify tableId in the XLSX file; the converter will > > generate it if it wasn't specified, and it will always be present in > this > > settings object. > > > > 'choices' is a restructuring of the choices sheet. > > > > 'queries' is a restructuring of the queries sheet. > > > > 'calculates' is a restructuring of the calculates sheet. > > > > 'model' is being deprecated. It is a synthesis of all the field names > used > > in the survey and subsection sheets, and the content of the > 'prompt_types' > > and 'model' sheets (if present). It defines the storage model of the > form. > > > > When the formId and tableId are identical, the 'model' sheet is written > out > > into the definition.csv and the properties.csv (along with other > > information). > > > > We are striving to remove 'model' from the 'specification' and rely on > those > > other files to fully describe the storage model. This and other changes > > should significantly simplify the underlying code of the JS runtime > > environment. > > > > 'section_names' is a list of the sections in the form. All forms > contain an > > 'initial' and a 'survey' section. The form launches into the 'initial' > > section which then launches into the 'survey' section. The XLSXconverter > > generates the 'initial' section if the XLSX file does not provide one. > > > > 'sections' is an object with one key for each section name. Each object > > describes the content of that corresponding section. > > > > The structure of a section object is likely to change. We want to > perform > > syntax and semantic tests in the XLSXconverter and provide > auto-generated > > content where possible. > > > > The current structure is: > > > > { "section_name": name, > > "nested_sections": {}, > > "reachable_sections" : {}, > > "prompts": [], > > "validation_tag_map": {}, > > "operations": [], > > "branch_label_map": {}, > > } > > > > nested_sections are the sections that are directly called via 'do > section'. > > > > reachable_sections is the recursive closure of nested_sections. It is an > > error for reachable_sections to contain this section (any possible means > to > > invoke 'do section' within this section or any section is can call is > not > > allowed). > > > > prompts is a list of the prompts in the section > > > > validation_tag_map tracks the prompts that have constraints linked to a > > given validation tag > > > > operations is a list of processing steps that are executed to traverse > this > > section. It is the 'assembly code' generated by the XLSX sheet. > > > > branch_label_map is a map of the branch labels into the operations list. > > > > ================================= > > I am probably missing some other object keys needed to support custom > > prompts and other features. > > > > Below this level, it is best to look at the content of a formDef.json > > > > We are likely to be changing some of this as we refine the functionality > > that the runtime JS environment implements. > > > > > > On Wed, Oct 21, 2015 at 2:48 PM, Gregor MacLennan <gmacl...@digital-democracy.org > wrote: > >> > >> It would be useful to have some kind of open discussion about the > >> formDef.json spec. I'm working on form design and data collection tools > and > >> would like to use the same/similar form definition language / not > repeat > >> mistakes / work towards compatibility / collaborate on thinking through > the > >> best structure. > >> > >> Gregor > >> > >> > >> On Monday, October 12, 2015 at 11:50:19 AM UTC-6, clarice larson wrote: > >>> > >>> We don't currently have documentation for the formDef.json. Ideally, > we > >>> would like people to focus on the creation of the Excel spreadsheet > and use > >>> the XLSXConverter documentation > >>> ( > https://opendatakit.org/use/2_0_tools/odk-application-designer-2-0-rev126/odk-xlsxconverter-2-0-rev126/) > > >>> as we may still make changes to the structure of the formDef.json. > >>> > >>> Clarice > >>> > >>> On Fri, Oct 9, 2015 at 6:23 PM, Gregor MacLennan wrote: > >>>> > >>>> Where is the documentation for the current surveyJSON / formdef.json > >>>> spec? I can't find it anywhere on the Github page. > >>>> > >>>> -- > >>>> You received this message because you are subscribed to the Google > >>>> Groups "ODK Developers" group. > >>>> To unsubscribe from this group and stop receiving emails from it, > send > >>>> an email to opendatakit-developers+unsubscribe@googlegroups.com > . > >>>> For more options, visit https://groups.google.com/d/optout. > >>> > >>> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "ODK Developers" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to opendatakit-developers+unsubscribe@googlegroups.com > . > >> For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > -- > > Mitch Sundt > > Software Engineer > > University of Washington > > mitche...@gmail.com > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "ODK Developers" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to opendatakit-developers+unsubscribe@googlegroups.com > . > > For more options, visit https://groups.google.com/d/optout. >

One final question:
is https://github.com/opendatakit/app-designer/tree/master/xlsxconverter
the active repo for development of xlsxconverter?
Gregor

··· On Thursday, October 29, 2015 at 3:51:14 PM UTC-6, Gregor MacLennan wrote: > > Thanks so much for this Yaw and Mitch, this is super useful. > > On Wednesday, October 21, 2015 at 10:51:14 PM UTC-6, Yaw Anokwa wrote: >> >> Now on the wiki if folks want to make edits. >> >> Yaw >> -- >> Need ODK consultants? https://nafundi.com provides form design, server >> setup, in-field training, and software development for ODK. >> >> On Thu, Oct 22, 2015 at 8:38 AM, Mitch Sundt wrote: >> > Below is a brief summary of the content of the formDef.json. >> > >> > At the moment, the sample forms and code are the reference >> implementation. >> > As we need to make structural changes, we re-generate all of the >> > formDef.json files from the XLSX files. >> > >> > Hence why we refer people to the XLSX file format documentation. >> > >> > i.e., don't trust that what follows will exist as-is into the release >> > version. >> > >> > ================================================ >> > formDef.json generated by XLSXConverter contains an object with 2 >> entries: >> > >> > { xlsx: {...}, >> > specification: {...} } >> > >> > The xlsx object holds a summary of the contents of the xlsx file such >> that >> > you could reconstruct that features of that file from this section of >> the >> > formDef.json. >> > >> > This currently consists of a key for each sheet in the original XLSX, >> and >> > each of those being a list of objects, one for each non-empty row in >> that >> > sheet, with the keys being the column headings on that sheet. >> > >> > We record this for trace-ability back to the original XLSX file >> content. And >> > to avoid the "I've got the XML but lost my XLSX file; do you have a >> tool to >> > reconstruct the XLSX file from that?" problem. >> > >> > Presumably, if users created a form definition using some other tool >> e.g., >> > "Build 2.0", the file would contain a 'build' key with some sort of >> source >> > description for that form creation tool that could allow reconstruction >> of >> > that source format. >> > >> > ------------------- >> > >> > The interpreter of the formDef.json file ONLY pays attention to the >> > 'specification' key. >> > >> > It should be considered improper for the runtime engine to look outside >> of >> > the 'specification' section of this file. >> > >> > ------------------- >> > The 'specification' key has this object structure: >> > >> > { 'column_types': { }, >> > 'settings' : {}, >> > 'choices' : {}, >> > 'queries' : {}, >> > 'calculates' : {}, >> > 'model' : {}, >> > 'section_names' : ["initial", "survey", ...], >> > 'sections' : { initial: {}, survey: {}, ... } >> > } >> > >> > 'column_types' identifies the object keys within a prompt or screen >> object >> > that have special interpretations by the runtime environment. >> > >> > e.g., "required": "formula" >> > >> > Specifies that when the runtime environment finds that key, its value >> should >> > be wrapped and converted to a function() {} object such that it can be >> > evaluated when executing the form and the evaluation context contains >> some >> > helper functions, the calculate expressions, etc. >> > >> > We have others, such as formula(context) which would wrap the value in >> a >> > function(context) {}, etc. >> > >> > The list and meanings of these is subject to change. >> > >> > The intent is that a user could add a 'column_types' sheet to their >> XLSX >> > file and declare the treatment for a given column heading, and the >> runtime >> > environment would apply that treatment. So if you wanted a >> 'extraConstraint' >> > that was a formula, you could declare that on the XLSX sheet, then >> supply >> > content under that heading in any section sheet, and it would pass >> through >> > and become available in the runtime environment. >> > >> > i.e., the XLSX format is extensible without changing the converter or >> the >> > runtime. >> > >> > 'settings' is a restructuring of the content of the settings XLSX >> sheet, >> > with omitted entries generated from other entries that were provided. >> I.e., >> > you don't need to specify tableId in the XLSX file; the converter will >> > generate it if it wasn't specified, and it will always be present in >> this >> > settings object. >> > >> > 'choices' is a restructuring of the choices sheet. >> > >> > 'queries' is a restructuring of the queries sheet. >> > >> > 'calculates' is a restructuring of the calculates sheet. >> > >> > 'model' is being deprecated. It is a synthesis of all the field names >> used >> > in the survey and subsection sheets, and the content of the >> 'prompt_types' >> > and 'model' sheets (if present). It defines the storage model of the >> form. >> > >> > When the formId and tableId are identical, the 'model' sheet is written >> out >> > into the definition.csv and the properties.csv (along with other >> > information). >> > >> > We are striving to remove 'model' from the 'specification' and rely on >> those >> > other files to fully describe the storage model. This and other >> changes >> > should significantly simplify the underlying code of the JS runtime >> > environment. >> > >> > 'section_names' is a list of the sections in the form. All forms >> contain an >> > 'initial' and a 'survey' section. The form launches into the 'initial' >> > section which then launches into the 'survey' section. The >> XLSXconverter >> > generates the 'initial' section if the XLSX file does not provide one. >> > >> > 'sections' is an object with one key for each section name. Each object >> > describes the content of that corresponding section. >> > >> > The structure of a section object is likely to change. We want to >> perform >> > syntax and semantic tests in the XLSXconverter and provide >> auto-generated >> > content where possible. >> > >> > The current structure is: >> > >> > { "section_name": name, >> > "nested_sections": {}, >> > "reachable_sections" : {}, >> > "prompts": [], >> > "validation_tag_map": {}, >> > "operations": [], >> > "branch_label_map": {}, >> > } >> > >> > nested_sections are the sections that are directly called via 'do >> section'. >> > >> > reachable_sections is the recursive closure of nested_sections. It is >> an >> > error for reachable_sections to contain this section (any possible >> means to >> > invoke 'do section' within this section or any section is can call is >> not >> > allowed). >> > >> > prompts is a list of the prompts in the section >> > >> > validation_tag_map tracks the prompts that have constraints linked to a >> > given validation tag >> > >> > operations is a list of processing steps that are executed to traverse >> this >> > section. It is the 'assembly code' generated by the XLSX sheet. >> > >> > branch_label_map is a map of the branch labels into the operations >> list. >> > >> > ================================= >> > I am probably missing some other object keys needed to support custom >> > prompts and other features. >> > >> > Below this level, it is best to look at the content of a formDef.json >> > >> > We are likely to be changing some of this as we refine the >> functionality >> > that the runtime JS environment implements. >> > >> > >> > On Wed, Oct 21, 2015 at 2:48 PM, Gregor MacLennan wrote: >> >> >> >> It would be useful to have some kind of open discussion about the >> >> formDef.json spec. I'm working on form design and data collection >> tools and >> >> would like to use the same/similar form definition language / not >> repeat >> >> mistakes / work towards compatibility / collaborate on thinking >> through the >> >> best structure. >> >> >> >> Gregor >> >> >> >> >> >> On Monday, October 12, 2015 at 11:50:19 AM UTC-6, clarice larson wrote: >> >>> >> >>> We don't currently have documentation for the formDef.json. Ideally, >> we >> >>> would like people to focus on the creation of the Excel spreadsheet >> and use >> >>> the XLSXConverter documentation >> >>> ( >> https://opendatakit.org/use/2_0_tools/odk-application-designer-2-0-rev126/odk-xlsxconverter-2-0-rev126/) >> >> >>> as we may still make changes to the structure of the formDef.json. >> >>> >> >>> Clarice >> >>> >> >>> On Fri, Oct 9, 2015 at 6:23 PM, Gregor MacLennan wrote: >> >>>> >> >>>> Where is the documentation for the current surveyJSON / formdef.json >> >>>> spec? I can't find it anywhere on the Github page. >> >>>> >> >>>> -- >> >>>> You received this message because you are subscribed to the Google >> >>>> Groups "ODK Developers" group. >> >>>> To unsubscribe from this group and stop receiving emails from it, >> send >> >>>> an email to opendatakit-developers+unsubscribe@googlegroups.com. >> >>>> For more options, visit https://groups.google.com/d/optout. >> >>> >> >>> >> >> -- >> >> You received this message because you are subscribed to the Google >> Groups >> >> "ODK Developers" group. >> >> To unsubscribe from this group and stop receiving emails from it, send >> an >> >> email to opendatakit-developers+unsubscribe@googlegroups.com. >> >> For more options, visit https://groups.google.com/d/optout. >> > >> > >> > >> > >> > -- >> > Mitch Sundt >> > Software Engineer >> > University of Washington >> > mitche...@gmail.com >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "ODK Developers" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an >> > email to opendatakit-developers+unsubscribe@googlegroups.com. >> > For more options, visit https://groups.google.com/d/optout. >> >

Gregor,

I believe so. Or rather, there is no other public repo :slight_smile:

Yaw

··· -- Need ODK consultants? https://nafundi.com provides form design, server setup, in-field training, and software development for ODK.

On Mon, Nov 2, 2015 at 2:23 AM, Gregor MacLennan gmaclennan@digital-democracy.org wrote:

One final question: is
https://github.com/opendatakit/app-designer/tree/master/xlsxconverter the
active repo for development of xlsxconverter?
Gregor

On Thursday, October 29, 2015 at 3:51:14 PM UTC-6, Gregor MacLennan wrote:

Thanks so much for this Yaw and Mitch, this is super useful.

On Wednesday, October 21, 2015 at 10:51:14 PM UTC-6, Yaw Anokwa wrote:

Now on the wiki if folks want to make edits.

Yaw

Need ODK consultants? https://nafundi.com provides form design, server
setup, in-field training, and software development for ODK.

On Thu, Oct 22, 2015 at 8:38 AM, Mitch Sundt mitche...@gmail.com wrote:

Below is a brief summary of the content of the formDef.json.

At the moment, the sample forms and code are the reference
implementation.
As we need to make structural changes, we re-generate all of the
formDef.json files from the XLSX files.

Hence why we refer people to the XLSX file format documentation.

i.e., don't trust that what follows will exist as-is into the release
version.

================================================
formDef.json generated by XLSXConverter contains an object with 2
entries:

{ xlsx: {...},
specification: {...} }

The xlsx object holds a summary of the contents of the xlsx file such
that
you could reconstruct that features of that file from this section of
the
formDef.json.

This currently consists of a key for each sheet in the original XLSX,
and
each of those being a list of objects, one for each non-empty row in
that
sheet, with the keys being the column headings on that sheet.

We record this for trace-ability back to the original XLSX file
content. And
to avoid the "I've got the XML but lost my XLSX file; do you have a
tool to
reconstruct the XLSX file from that?" problem.

Presumably, if users created a form definition using some other tool
e.g.,
"Build 2.0", the file would contain a 'build' key with some sort of
source
description for that form creation tool that could allow reconstruction
of
that source format.


The interpreter of the formDef.json file ONLY pays attention to the
'specification' key.

It should be considered improper for the runtime engine to look outside
of
the 'specification' section of this file.


The 'specification' key has this object structure:

{ 'column_types': { },
'settings' : {},
'choices' : {},
'queries' : {},
'calculates' : {},
'model' : {},
'section_names' : ["initial", "survey", ...],
'sections' : { initial: {}, survey: {}, ... }
}

'column_types' identifies the object keys within a prompt or screen
object
that have special interpretations by the runtime environment.

e.g., "required": "formula"

Specifies that when the runtime environment finds that key, its value
should
be wrapped and converted to a function() {} object such that it can be
evaluated when executing the form and the evaluation context contains
some
helper functions, the calculate expressions, etc.

We have others, such as formula(context) which would wrap the value in
a
function(context) {}, etc.

The list and meanings of these is subject to change.

The intent is that a user could add a 'column_types' sheet to their
XLSX
file and declare the treatment for a given column heading, and the
runtime
environment would apply that treatment. So if you wanted a
'extraConstraint'
that was a formula, you could declare that on the XLSX sheet, then
supply
content under that heading in any section sheet, and it would pass
through
and become available in the runtime environment.

i.e., the XLSX format is extensible without changing the converter or
the
runtime.

'settings' is a restructuring of the content of the settings XLSX
sheet,
with omitted entries generated from other entries that were provided.
I.e.,
you don't need to specify tableId in the XLSX file; the converter will
generate it if it wasn't specified, and it will always be present in
this
settings object.

'choices' is a restructuring of the choices sheet.

'queries' is a restructuring of the queries sheet.

'calculates' is a restructuring of the calculates sheet.

'model' is being deprecated. It is a synthesis of all the field names
used
in the survey and subsection sheets, and the content of the
'prompt_types'
and 'model' sheets (if present). It defines the storage model of the
form.

When the formId and tableId are identical, the 'model' sheet is written
out
into the definition.csv and the properties.csv (along with other
information).

We are striving to remove 'model' from the 'specification' and rely on
those
other files to fully describe the storage model. This and other
changes
should significantly simplify the underlying code of the JS runtime
environment.

'section_names' is a list of the sections in the form. All forms
contain an
'initial' and a 'survey' section. The form launches into the 'initial'
section which then launches into the 'survey' section. The
XLSXconverter
generates the 'initial' section if the XLSX file does not provide one.

'sections' is an object with one key for each section name. Each object
describes the content of that corresponding section.

The structure of a section object is likely to change. We want to
perform
syntax and semantic tests in the XLSXconverter and provide
auto-generated
content where possible.

The current structure is:

{ "section_name": name,
"nested_sections": {},
"reachable_sections" : {},
"prompts": ,
"validation_tag_map": {},
"operations": ,
"branch_label_map": {},
}

nested_sections are the sections that are directly called via 'do
section'.

reachable_sections is the recursive closure of nested_sections. It is
an
error for reachable_sections to contain this section (any possible
means to
invoke 'do section' within this section or any section is can call is
not
allowed).

prompts is a list of the prompts in the section

validation_tag_map tracks the prompts that have constraints linked to a
given validation tag

operations is a list of processing steps that are executed to traverse
this
section. It is the 'assembly code' generated by the XLSX sheet.

branch_label_map is a map of the branch labels into the operations
list.

=================================
I am probably missing some other object keys needed to support custom
prompts and other features.

Below this level, it is best to look at the content of a formDef.json

We are likely to be changing some of this as we refine the
functionality
that the runtime JS environment implements.

On Wed, Oct 21, 2015 at 2:48 PM, Gregor MacLennan gmacl...@digital-democracy.org wrote:

It would be useful to have some kind of open discussion about the
formDef.json spec. I'm working on form design and data collection
tools and
would like to use the same/similar form definition language / not
repeat
mistakes / work towards compatibility / collaborate on thinking
through the
best structure.

Gregor

On Monday, October 12, 2015 at 11:50:19 AM UTC-6, clarice larson wrote:

We don't currently have documentation for the formDef.json. Ideally,
we
would like people to focus on the creation of the Excel spreadsheet
and use
the XLSXConverter documentation

(https://opendatakit.org/use/2_0_tools/odk-application-designer-2-0-rev126/odk-xlsxconverter-2-0-rev126/)
as we may still make changes to the structure of the formDef.json.

Clarice

On Fri, Oct 9, 2015 at 6:23 PM, Gregor MacLennan gmacl...@digital-democracy.org wrote:

Where is the documentation for the current surveyJSON / formdef.json
spec? I can't find it anywhere on the Github page.

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

The active tip of our ODK 2.0 GitHub repos is the 'development' branch.

We are trying to reserve 'master' as the stable tip.

At the moment, the development branch is using bootstrap and is in
transition on the functionality and implementation of the ODK Tables
javascript APIs, while the master tip is using jQueryMobile.

··· On Sun, Nov 1, 2015 at 7:09 PM, Yaw Anokwa wrote:

Gregor,

I believe so. Or rather, there is no other public repo :slight_smile:

Yaw

Need ODK consultants? https://nafundi.com provides form design, server
setup, in-field training, and software development for ODK.

On Mon, Nov 2, 2015 at 2:23 AM, Gregor MacLennan gmaclennan@digital-democracy.org wrote:

One final question: is
https://github.com/opendatakit/app-designer/tree/master/xlsxconverter
the
active repo for development of xlsxconverter?
Gregor

On Thursday, October 29, 2015 at 3:51:14 PM UTC-6, Gregor MacLennan wrote:

Thanks so much for this Yaw and Mitch, this is super useful.

On Wednesday, October 21, 2015 at 10:51:14 PM UTC-6, Yaw Anokwa wrote:

Now on the wiki if folks want to make edits.

Yaw

Need ODK consultants? https://nafundi.com provides form design, server
setup, in-field training, and software development for ODK.

On Thu, Oct 22, 2015 at 8:38 AM, Mitch Sundt mitche...@gmail.com wrote:

Below is a brief summary of the content of the formDef.json.

At the moment, the sample forms and code are the reference
implementation.
As we need to make structural changes, we re-generate all of the
formDef.json files from the XLSX files.

Hence why we refer people to the XLSX file format documentation.

i.e., don't trust that what follows will exist as-is into the release
version.

================================================
formDef.json generated by XLSXConverter contains an object with 2
entries:

{ xlsx: {...},
specification: {...} }

The xlsx object holds a summary of the contents of the xlsx file such
that
you could reconstruct that features of that file from this section of
the
formDef.json.

This currently consists of a key for each sheet in the original XLSX,
and
each of those being a list of objects, one for each non-empty row in
that
sheet, with the keys being the column headings on that sheet.

We record this for trace-ability back to the original XLSX file
content. And
to avoid the "I've got the XML but lost my XLSX file; do you have a
tool to
reconstruct the XLSX file from that?" problem.

Presumably, if users created a form definition using some other tool
e.g.,
"Build 2.0", the file would contain a 'build' key with some sort of
source
description for that form creation tool that could allow
reconstruction
of
that source format.


The interpreter of the formDef.json file ONLY pays attention to the
'specification' key.

It should be considered improper for the runtime engine to look
outside
of
the 'specification' section of this file.


The 'specification' key has this object structure:

{ 'column_types': { },
'settings' : {},
'choices' : {},
'queries' : {},
'calculates' : {},
'model' : {},
'section_names' : ["initial", "survey", ...],
'sections' : { initial: {}, survey: {}, ... }
}

'column_types' identifies the object keys within a prompt or screen
object
that have special interpretations by the runtime environment.

e.g., "required": "formula"

Specifies that when the runtime environment finds that key, its value
should
be wrapped and converted to a function() {} object such that it can
be
evaluated when executing the form and the evaluation context contains
some
helper functions, the calculate expressions, etc.

We have others, such as formula(context) which would wrap the value
in
a
function(context) {}, etc.

The list and meanings of these is subject to change.

The intent is that a user could add a 'column_types' sheet to their
XLSX
file and declare the treatment for a given column heading, and the
runtime
environment would apply that treatment. So if you wanted a
'extraConstraint'
that was a formula, you could declare that on the XLSX sheet, then
supply
content under that heading in any section sheet, and it would pass
through
and become available in the runtime environment.

i.e., the XLSX format is extensible without changing the converter or
the
runtime.

'settings' is a restructuring of the content of the settings XLSX
sheet,
with omitted entries generated from other entries that were provided.
I.e.,
you don't need to specify tableId in the XLSX file; the converter
will
generate it if it wasn't specified, and it will always be present in
this
settings object.

'choices' is a restructuring of the choices sheet.

'queries' is a restructuring of the queries sheet.

'calculates' is a restructuring of the calculates sheet.

'model' is being deprecated. It is a synthesis of all the field names
used
in the survey and subsection sheets, and the content of the
'prompt_types'
and 'model' sheets (if present). It defines the storage model of the
form.

When the formId and tableId are identical, the 'model' sheet is
written
out
into the definition.csv and the properties.csv (along with other
information).

We are striving to remove 'model' from the 'specification' and rely
on
those
other files to fully describe the storage model. This and other
changes
should significantly simplify the underlying code of the JS runtime
environment.

'section_names' is a list of the sections in the form. All forms
contain an
'initial' and a 'survey' section. The form launches into the
'initial'
section which then launches into the 'survey' section. The
XLSXconverter
generates the 'initial' section if the XLSX file does not provide
one.

'sections' is an object with one key for each section name. Each
object
describes the content of that corresponding section.

The structure of a section object is likely to change. We want to
perform
syntax and semantic tests in the XLSXconverter and provide
auto-generated
content where possible.

The current structure is:

{ "section_name": name,
"nested_sections": {},
"reachable_sections" : {},
"prompts": ,
"validation_tag_map": {},
"operations": ,
"branch_label_map": {},
}

nested_sections are the sections that are directly called via 'do
section'.

reachable_sections is the recursive closure of nested_sections. It is
an
error for reachable_sections to contain this section (any possible
means to
invoke 'do section' within this section or any section is can call is
not
allowed).

prompts is a list of the prompts in the section

validation_tag_map tracks the prompts that have constraints linked
to a
given validation tag

operations is a list of processing steps that are executed to
traverse
this
section. It is the 'assembly code' generated by the XLSX sheet.

branch_label_map is a map of the branch labels into the operations
list.

=================================
I am probably missing some other object keys needed to support custom
prompts and other features.

Below this level, it is best to look at the content of a formDef.json

We are likely to be changing some of this as we refine the
functionality
that the runtime JS environment implements.

On Wed, Oct 21, 2015 at 2:48 PM, Gregor MacLennan gmacl...@digital-democracy.org wrote:

It would be useful to have some kind of open discussion about the
formDef.json spec. I'm working on form design and data collection
tools and
would like to use the same/similar form definition language / not
repeat
mistakes / work towards compatibility / collaborate on thinking
through the
best structure.

Gregor

On Monday, October 12, 2015 at 11:50:19 AM UTC-6, clarice larson wrote:

We don't currently have documentation for the formDef.json.
Ideally,
we
would like people to focus on the creation of the Excel spreadsheet
and use
the XLSXConverter documentation

(
https://opendatakit.org/use/2_0_tools/odk-application-designer-2-0-rev126/odk-xlsxconverter-2-0-rev126/
)
as we may still make changes to the structure of the formDef.json.

Clarice

On Fri, Oct 9, 2015 at 6:23 PM, Gregor MacLennan gmacl...@digital-democracy.org wrote:

Where is the documentation for the current surveyJSON /
formdef.json
spec? I can't find it anywhere on the Github page.

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send
an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send
an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com