So I ran across this today, and it struck me as a bit odd... Strictly speaking, it appears legitimate - at least in terms of XForms/XLSForm spec - to indeed have a repeat group inside of a group with a field-list appearance. Specifically, the XLSForm spec says nothing about excluding such, merely...
In your form, you can use the field-list appearance on a group to display multiple questions on the same screen.
That is to say, the field-list appearance is basically just a UI preference to the XForm client to display everything on the same 'screen' [sic]. Indeed, the Enketo web client supports it quite happily; eg the following (XLSForm) form contains a field-list group with two repeat groups inside it, and it is rendered in the web UI as:
RepeatFieldList.xlsx (18.5 KB)
However, as noted previously, ODK Collect will throw a runtime warning if you attempt to use this form:
Indeed, the docs specifically state this isnt permissable, with a warning:
You can apply the field-list appearance to a repeat to make all of the repeated questions go on a single screen. However, you can't have a repeat inside of another group with the field-list appearance.
However, strictly speaking, this is only the case for the Collect app (again, probably for the reasons @Grzesiek2010 alluded to).
What's rather interesting is that if you do actually attempt to do this, as shown in the above Collect snapshot, Collect actually proceeds to happily present the second nested repeat group's question(s) in the field list group (!?), completely skipping the first nested repeat group; that seems very odd indeed...
I think if Collect cant reasonably support repeat groups within field-lists then ideally we probably shouldn't let the user even start the form, or preferably even throw an error when they try to download such a form. I dont think exhibiting odd behavior when Collect eventually hits a repeat group nested within a field-list is desirable.
We should also probably relax the documentation to indicate this is Collect UI-specific behavior, which doesn't affetc Enketo web clients.