That's our general approach that works in the same way for all widgets. The rule is: if question is read-only and there is no answer hide the whole answer container (if you added default answer it should be displayed). In case of selects with image-map appearance - that container holds the svg.
We had multiple problems in the past with different behaviors across different read-only questions and finally we were able to work out one common approach (the one I described) so I think we don't want to change it.
It looks like you don't need to display hat svg as a part of other widgets (different question types like selects text widgets etc) like we do with images/audio/videos that we can add to every question type. You just want to have a question type that only displays svg and nothing else (something like the note question). Am I right?