Yours is only one way to define elitist.

Another definition of elite that doesn't tie it to social class:

a group of persons exercising the major share of authority or influence within a larger group