How I Feel About Your Policies Depends On How Much Power You Have

There's various kinds of policy preferences where my actual opinion about them is something like "if this is a tiny niche belief then I'm glad it exists, it's a useful input to ultimate collective negotiation, but if it gained significant power in society I think that would be pretty bad."

This seems pretty banal when I write it, but I don't feel like I've heard it discussed much and I don't know what to do with it on a practical level.

Should I pre-emptively oppose a group who are still quite small because I think that if they got (significantly) bigger that would be bad? Or should I support the group's existence now, and if they get bigger deal with the consequences later? Neither of these feel great.

I suspect I have this problem more than many people do because I'm philosophically Millian, I really do think that even wrong arguments can often help other people improve their thinking, and also that "banning" views (to the extent that's ever possible) can often backfire. So I suspect a lot of people are much less likely to see a philosophy and think "I'm ok with some of that but don't want a ton of it", they either want it everywhere or nowhere?, I don't know if this is true.

Variety is the spice of life, but there's many spices that I'm happy to eat if they're <1% of a dish and yet not excited to gulp whole spoonfuls of. How am I supposed to relate to the people who are trying to fill my soup with their one spice, if I support them at their current level but oppose them if they got more successful?