@Raj_Pravat the documentation has an XLSForm example. There's another one linked from my post above. What @johngodlee is trying to do is not currently available in XLSForm.
This is also not currently supported by the pyxform implementation so you'll definitely need to edit the XML for the time being.
First, your pos expression needs to be adjusted. You want the position of the repeat so you'll need to use position(../../) from within the nested group. Your itemset declaration would look something like:
<itemset nodeset="/data/pet[position() != current()/../pos and pet_group/animal_type != '']">
<value ref="pet_group/animal_type"/>
<label ref="pet_group/animal_type"/>
</itemset>
The itemset is always built directly on a repeat, not on any nested groups. So you need to start with /data/pet. The group nesting is accounted for in the value and label reference declarations. Those say "once you've found the repeat instances to include, here's how you get the right field to use as value and label".
For the second part of the repeat matching test that filters out any repeats in which animal_type is empty, you also need to account for the nested group. So the expression needs to be pet_group/animal_type != ''.