If you have multiple languages, I think you need a media column for each language, even if the video is the same. For example have both a media:video::English (en) and a media:video::French (fr).
See this post dealing with what seems to be a similar issue with images: