On the question of inner alternation/conjunction (TestCategory=A|B instead of TestCategory=A|TestCategory=B):

Short answer: no, not for v1. I've added this as Open Question #9 in the RFC and propose deferring it.

Three approaches were considered:

1. Outer parser splits |/& inside values. Rejected. It forces every kind into the same RHS grammar. Trait=name=value already uses = inside its value, and a Query= value can legitimately contain | (e.g. Query=/asm/test|other/). Making | mean "OR within value" universally breaks both kinds.
2. Per-kind opt-in AllowInnerAlternation flag on TestNodeFilterKind. Workable but adds API surface to v1 and forces a precedence decision — does TestCategory=A|B&Priority=1 mean (TestCategory=A | TestCategory=B) & Priority=1, or TestCategory=A | (TestCategory=B & Priority=1)? Either choice will surprise some users.
3. A separate values-list syntax (e.g. TestCategory=[A,B,C] or TestCategory in (A,B,C)) added later. Non-breaking: the new punctuation is invalid in today's grammar, so we can adopt it later with zero migration cost.

For v1 I'd rather ship the smaller, unambiguous surface and match VSTest exactly (VSTest itself does not support inner | — it requires the verbose TestCategory=A|TestCategory=B form, so we lose no parity). If user feedback shows the verbose form is a real pain point, option 3 is the path I'd take in a follow-up RFC.

See Open Question #9 in 14b11e4 for the same rationale captured in the RFC.

@bradwilson @OsirisTerje @thomhurst — pinging you as test framework owners (xUnit / NUnit / TUnit). This RFC proposes a new --filter CLI option for Microsoft.Testing.Platform with a kind-based grammar (Kind=Value | Kind=Value) and an extensibility hook so each framework can register its own filter kinds and the DefaultFilterMatchTextProperty used for bare matches. Would love your feedback on the surface area, the extensibility model, and anything that wouldn't fit your framework's semantics.

@MarcoRossignoli @Youssef1313 — would also appreciate your eyes on the design before we move toward implementation.

A big thing to think about for this I think is llms. For TUnit, they get confused and try to use old vstest expressions a lot. I think good error messages should help them with this though.

Also are we opening up custom built extensions at all? I'd be happy to keep the existing treenode filter command line since users already use it, but would rather do it cleanly with actual filter types in the code, than workarounds

What about the source based --filter support package that you've already provided to us? Doesn't that conflict with this proposal? Does this mean we shouldn't ship with it?

A big thing to think about for this I think is llms. For TUnit, they get confused and try to use old vstest expressions a lot. I think good error messages should help them with this though.

I think you're just going to anger users if your --filter isn't compatible with the one that VSTest ships with, whether that knowledge and muscle memory comes from the developer themselves or via LLM.

The goal is to merge the various options all together with a small caveat for when no prefix is specified. Let's take as example --filter SomeTest which in VSTest case means TestCase.FQN contains "SomeTest", but we don't have the same FQN in MTP. We could "construct" it based of TestMethodIdentifierProperty (namespace.class.method) but I know this FQN implementation is already framework specific and doesn't always match this pattern.

Should anything be changed for xUnit.net v3 4.0 in the face of this proposal? Should I remove the --filter option?

I'm concerned not only with whether this new version of --filter actually serves the user's needs of a purely VSTest-compatible filter, but also whether some new version of .NET SDK is going to be broken with xUnit.net because we also bring a --filter option.

Reading through the whole RFC, it appears that --filter will be live whether the test framework opts in or not (because of Built-in Kinds).

If this is true, I need to remove --filter immediately from 4.0.

The goal is to merge the various options all together with a small caveat for when no prefix is specified.

I think before trying to smush everything together, we should support extensible filters and allowing extension authors to define their own.

I thought MTP was meant to be quite agnostic and I opinionated, but in terms of filters, it offers no extensibility and is completely opinionated currently.

Being opinionated isn't necessarily bad imo because it keeps test frameworks having similar expected filters. But I think not allowing us to offer more custom options is a real extensibility blocker

Should --treenode-filter keep working forever rather than being deprecated? Cost is small; "purity" is the only argument for removal.

I really dislike that MTP doesn't have a way to have a "hidden" option. We use this in xUnit.net frequently when transitioning from one syntax to another to preserve backward compatibility without listing the explicit back-compat option.

Example: when reporters became user configurable, and we switched all reporters from their short form (e.g., -json) to the longer form (e.g., -reporter json). We still support all the original built-in short form names for backward compatibility but they don't show up in the help. https://github.com/xunit/xunit/blob/1250120334242dfd4bca417c6317fb24147c6cf9/src/xunit.v3.runner.common/Parsers/CommandLineParserBase.cs#L235-L240

Being opinionated isn't necessarily bad imo because it keeps test frameworks having similar expected filters. But I think not allowing us to offer more custom options is a real extensibility blocker

This is explicitly called out in the Open Questions section (and deferred to be addressed later):

8. Per-project inconsistency in mixed-framework solutions. Different frameworks register different kind sets; running one filter expression across a solution may match in one project and error in another ("unknown kind"). Proposed mitigation: at solution entry, downgrade "unknown kind" from an error to a warning only when the kind is known by at least one project in the solution. Costly to implement; defer to a follow-up RFC.

What that potential implementation probably means is that things won't necessarily work when they're expected to, and the user is expected to see and interpret the warning to understand why. To be fair, there's no good answer here: Do you "hope for the best" and just run things, hoping the user knows that some of the filter is inapplicable to some of the test projects? Or do you refuse to run, making is hard or impossible for users to craft together a single query for solutions which target multiple test frameworks? Really only Microsoft knows (or could know) how often people are in such situations; I don't really field many questions at all about mixing xUnit.net with other test frameworks.

Should anything be changed for xUnit.net v3 4.0 in the face of this proposal? Should I remove the --filter option?

I haven't look at xUnit v4, I suppose you register --filter yourself (not through the bridge). Thinking about it, it may already be a problem for the bridge because if core platform "claims" --filter and vstest bridge does too it would cause issue. I can obviously update the bridge but if someone uses older version of bridge with newer core platform this would break.

I'll sleep on it and think about what I could do.

I really dislike that MTP doesn't have a way to have a "hidden" option. We use this in xUnit.net frequently when transitioning from one syntax to another to preserve backward compatibility without listing the explicit back-compat option.

Nothing is set in stone yet but I have tried to work on #8501 to help with that. I also did #7113 which we could probably combine. I am not against us all discussing and designing a different solution.

but in terms of filters, it offers no extensibility and is completely opinionated currently.

In this RFC, I am discussing the way to add prefix and the related handler logic which should in-part cover this need. I'll get back to your various open tickets around filter extensibility to see how to open it more.

To be fair, there's no good answer here

Mixed framework is more of a niche than mass thing but it's something that exist (especially for more mature products). I know this is a problem we have with the fact we don't have enough common filters. When we started design of MTP, I was hoping to the new filter to be "enforced" on all frameworks so we could have a filter covering most of the scenario for all frameworks but sadly because of the various internal pushbacks we are in some situation.

Maybe we should have claimed some more general options to cause less issues. Again I was naively hoping we would be allowed more time and we could all work together on the growing specs rather than us all moving on our side (not a blame, more of a reality check).

Overall maybe this design is interesting but would need to be applied only to a next breaking change version of MTP and for now we only move with adding more --filter-xxx options. I'll re-read it all (including your comments) tomorrow with fresh view to see if there is anything to move forward.

The goal is to merge the various options all together with a small caveat for when no prefix is specified. Let's take as example --filter SomeTest which in VSTest case means TestCase.FQN contains "SomeTest", but we don't have the same FQN in MTP. We could "construct" it based of TestMethodIdentifierProperty (namespace.class.method) but I know this FQN implementation is already framework specific and doesn't always match this pattern.

MTP doesn't need to have the knowledge of FQN. It's test framework responsibility to filter.

The platform cannot do much about filtering anyways. It's up to test frameworks to do the filtering because the framework is where the knowledge of running the tests is.

I don't think that's necessarily true. In the filtering draft pr I did last year, I proposed a way to make filters self aware on how to filter, based on being passed TestNodes and using the information available on them.

I think that's actually quite a good way to keep behaviour consistent across frameworks, while also allowing 'extensible' filtering plugins to just work in a plug-n-play way, just by installing them, without having to have the test framework aware of them. As long as the test framework is correctly populating things like test display names, test metadata property etc.

Do you see any downsides to that approach?

I think this is an important question to resolve:

Regardless of the actual implementation (and test framework requirements) required to put --filter into MTP itself, what the plan is to support older MTP projects if newer builds of MTP bring a --filter option in the box, without them being broken? A new major version of MTP due to this being a breaking change?