There are some immigrants to the country who don't realize English is an official language and generally people speak that, more so than their mother tongue.
So they will just automatically start conversations in that language. I kinda get it if you're a senior citizen (though common sense would be the Caucasian person you're talking to probably doesn't understand you) but when you're a middle-aged or young person, you should realize that and have some basic understanding of English.
For example, I was shoveling snow and my Italian neighbor was also shoveling snow next door. This other neighbor had gotten his car stuck in the snow and abandoned it so when another car pulled up behind him, he couldn't get around that stuck car. The Chinese guy in the second car rolls down his window and yells out at my neighbour and I, and in Chinese asks where's the driver of the car in front? My Italian neighbour is like WTF and doesn't understand, obviously.
While some may say that being a tourist and assuming everyone understands English is kind of a dick move, at least English is one of the languages more commonly spoken in various countries. It's a dick move, however, to assume that everyone is of the same dialect otherwise.