Am I really being a freeloader?

You are not a freeloader, you are using the system. He is jealous of the advantages you are getting that he did not have access to. In a way, I agree. White people are the new minority