People here say that casually. I kept hearing a friend say it for days, then one night I was out getting pizza after many drinks, and when the guy behind the counter (not sure what nationality he was, but not terribly dark and not caucasian) gave me my pizza I said, thanks boss. He said, excuse me? And glared at me angrily. So... I don't know if thats a cultural no no or something, but I never said it again.
I think it's meant to be out of respect, much like sir, or mam but people take offense to this for whatever reason. I don't really like when people call me mam but I don't mind that much because the person is just usually trying to be polite . I don't get it that much though I think because people usually think I'm a lot younger.
This thread makes me think of my grandpa though who is a character and I remember we went to get coffee one time and the much younger barista said, "what can I get for you, boss?" My grandpa turned to me and said, "would you look at that? He called me boss. Ha...ha. You see there, this guy really respects me." He got a kick out of it. Maybe an older generation appreciates acknowledgment like this, I don't know.
I get called Boss because I work like that, can't help it but when BF started calling me Boss I asked, not because I am controlling or anything?" He said "no, because with your business you are the boss but I get to say it anyway" I told him not to expect it back in case anyone were to hear me say it and think he had some kind of upper hand, he laughed and still says "Yes Boss" with a smirk. He has a few other fave nic names he calls me, we won't go there, hahahaha I can't remember if anyone in Europe called me anything odd. Never paid attention I guess but everyone was nice in most countries I was in. France was a little odd because they have a thing with Americans even still for some reason and if you speak any kind of English they assume you are American.
While the use has morphed a bit because of the whole 'Like a Boss' phenomena, it is indeed a poke at the power dynamic of slavery/servitude. And maybe you don't subscribe to colonialism. I'm pretty sure the people of the occupied countries didn't subscribe to colonialism either but they didn't have a choice and neither do you in this day and age. Great Britain is going through a 'karma is a bitch' phase. Keep Calm and Check Your Privilege.