So either it won’t grow to mainstream size because of that, or it will and the demographic will shift more to the center which will be off-putting to the original users.

No the admins would ban those communities or defederate other instances that allow those communities.