There's actually a term for exactly what you're describing! It's called nomasexual: attraction to everyone aside from men.
There's also trixensexual, which is attraction to women and nonbinary people. The label doesn't specify which noninary identities are included under this term, however.