I live in Vietnam and speak Vietnamese all day. It's extremely uncommon for foreigners to learn Vietnamese, even if they have lived here for decades, so people don't expect it.

My vocabulary is more than adequate and my grasp of idiom and pronunciation are very good, and Vietnamese is a tonal language

But.

I can barely understand spoken Vietnamese at all. It goes by too fast, there is too much variation in pronunciation e.g. using a G sound for R. Ususally when I don't understand there was not one word not in my vocabulary.

I've never had this problem in any other language, not even in Cantonese.