I don't consider myself a liberal, but I'll take a shot:
Neither my American Heritage Dictionary of the English Language nor an online search turns up the word racist.
Some words transition to an 'ist' definitiion. The word science can transition to scientist: A person who has expert knowledge of one or more sciences. There are many examples of this.
It appears that race does not [officially] transition to racist in the same way that science does to scientist. At least not as far back as 1971. Since new words are invented yearly, I may need a new dictionary. Certainly the word racist is out there and pervasive, but not found in my dictionary.
While race has a number of definitions in my fine dictionary, none point to the closest word applicable to what I suppose is Boilerman's question. That closest word IMO might be racism which is defined in my dictionary. Is Boilerman practicing racism when he tells his friends to go back to where they came from?
Racism: The notion that one's own ethnic stock is superior. This dictionary does not elaborate regarding behaviors now generally assigned in our time to racism.
So, unless Boilerman's instructions to his friends are based on a notion that they, and "where they came from" are ethnically inferior to him and where he came from, I would not say that he spoke from a position of racism. His friends might disagree with me; it is even likely they would disagree. But their disagreement does not mean they are correct. Only Boilerman knows the truth of it. My guess is that these friends were (or would in theory be) complaining about conditions in the United States.
Though it is certainly dependent on context, asking anyone to "go back to where you came from" almost guarantees that the hearer will take offense. Anyone willing to opine in such a way, within this minefield of political correctness we live in, can unfortunately be accused of racism, any negative ism, intended or not. Kind of an "in the eye of the beholder" thing. So, Boilerman probably can't win such a debate whether or not his intent is pure.