Validate issue for decimal-date-time

What is the problem? Please be detailed.
This post is linked to the form on the Showcase channel

If I convert the form with XLSForm (online/offline) it gives a validation error.
The sysntax looks correct and if I use the offline converter without validation, it converts the form that works correctly in Collect.
What breaks the validator is the calculation
decimal-date-time(concat(${year}, "-01-01"))
where ${year} is the year extracted from a date.
Another issue is that because there's a variable ${weekofyear} that has the same name of the file, this leads to another error.

What ODK tool and version are you using? And on what device and operating system version?
XLSForm Offline 1.8.1 (Windows 7) and Online 1.3.1, ODK Collect 1.19.0
What steps can we take to reproduce the problem?
Just try to convert the attached form with XLSForm online or offline.

What you have you tried to fix the problem?
Convert with offline without validation works.
Renaming the variable ${weekofyear} and removing the calculation decimal-date-time(concat(${year}, "-01-01")) pass xls passes the validation test.

Anything else we should know or have? If you have a test form or screenshots or logs, attach here.
weekofyear.xls (9.5 KB)


So this has been quite tricky to nail down...

So I've created a test form with:

year1 = format-date("2019-03-08","%Y")
newdate = concat(${year1},'-01-01')

This passes Validate fine.

However, when I change just the year calculation (!?) to take its date from an user input, rather than literal, ie

date: "Enter date"
year2 = format-date(${date},"%Y")
newdate = concat(${year2},'-01-01')

then Validate throws the error "The problem was located in calculate expression for ${result}\nXPath evaluation: type mismatch \nconverting to date\nCaused by: org.javarosa.xpath.XPathTypeMismatchException: The problem was located in calculate expression for ${result}\nXPath evaluation: type mismatch \nconverting to date\n\t... 10 more\n\nResult: Invalid"

note, in both cases the decimal-date-time() XPath calculation itself is identical, referencing the result of another calculation stored in nodeset /data/newdate. Further, in both cases this newdate XPath calculation is also basically identical, concat()'ing the result stored in yet another nodeset /data/yearX with a literal string. The only difference is in one case the year is being calculated from yet-yet-another XPath calculation format-date(); if the argument to this function is a literal we are OK, but if its a nodeset it fails.

I can only imagine what Validate is doing here... it seems to be following all the dependencies when checking the argument type of decimal-date-time(), and if its finds a calculation, it then requires all nested dependencies to be ultimately literals only. I'm at a bit of a loss - perhaps someone more intimately familiar with how Validate is performing its dependency checking knows exactly what's going on here? [Unfortunately, in my week-of-year demo form there's no getting around needing to generate a date calculated from a user input, so there's no workaround to this Validate bug I can think of].

I'm going to move this thread to Development, since its probably way too esoteric now for regular users.

(I have a test form for above, but Discourse doesnt appear to be letting me attach it... sigh)

Here is my test form (Discourse behaving better today I guess):

decimal-date-time-bug.xls (19 KB)

As given the form will cause Validate to throw an error when you try to load it in XLSForm online. If you change the newdate calculation to instead use year1 it loads fine.

Any insight is welcome. As soon as I can better narrow down what Validate/javaRosa is actually doing wrong I'll open a github issue, but right now its rather got me stumped.

1 Like

OK, I think I may have nailed it...

With the following form:

date:	year, default=2019
result0	= decimal-date-time("")	
result1	= decimal-date-time(${year})	
result2 = decimal-date-time(concat(${year},""))	
result3 = decimal-date-time(concat(${year},"2019-01-01"))	
result4	= decimal-date-time(concat(${year},"-01-01"))	

Validate barfs on result4! I think what is happening is that, as part of validating function argument type checking, Validate is actually attempting to dynamically evaluate the calculate XPath expressions (!) in an attempt to determine whether the date function's (string) argument is in fact a valid date string. Unfortunately, this is before the form has actually started executing, so $(year) is still basically null, (even though I explicitly gave it a default value, although that's not the root of the problem). Which is why result3 is actually deemed to be OK (wrong!) - because concat(null,"2019-01-01") -> "2019-01-01" which is a valid date; but concat(null,"-01-01") -> "-01-01" is not a valid date. So Validate incorrectly determines there a type mismatch in the latter, and throws a error "XPathTypeMismatchException: The problem was located in calculate expression for ${result4} XPath evaluation: type mismatch converting to date".

I believe the root of the problem is that Validate probably should not attempt to dynamically evaluate XPath expressions in order to guess the type of (string) operands when checking XPath function arguments. The string value of these arguments can only be reliably determined at actual runtime, for all the reasons described above. There is fundamentally a limited ability to perform XPath type checking during a static analysis of a form's XPath expressions, so I think in this case we should instead leave it to the runtime javaRosa XPath evaluator (in Collect) throw a NaN/NULL/whatever when it fails to convert (string) arguments to their expected type.

But this seems like it could be a not insignificant change in javaRosa/Validate behavior, so I'm not sure how best to proceed. Thoughts anyone? @yanokwa, @ggalmazor, @martijnr, @aurdipas ? [I can move over to Slack, and/or open a github issue to continue the technical discussion if preferred]

Thanks for studying this in such detail, @Xiphware!

I think this has hints of a bug. We should file an issue in JR with some detailed scenario that we could reproduce in a junit test to start working out some solutions.

K. I'll open a JR github issue to track. Opened

The simplest testcase I was able to come up with exhibiting the behavior is the above, namely

date:	year, default=2019
result = decimal-date-time(concat(${year},"-01-01"))

which is valid and runs fine in Collect, but causes a form type-mismatch error with Validate. I suspect this may only apply to date/time functions type-checking, but until we can isolate it in the code I cant be 100% certain this wont pop up in other non-date related cases.

1 Like