Possibly another approach is to instead produce InterpolationStart and InterpolationEnd tokens, and then avoid any token nesting from the lexer.

...which would be how we currently produce parentheses (but not how we produce strings — those return a Literal::String, rather than StringStart and StringEnd)

I tried this implementation strategy when rewriting from Pest to chumsky, but my bottom line was the same: the parser expects a flat stream of tokens and not a tree.

This PR is a good effort, but I don't know how to continue from here. The from_nested might work, but even if it does, having nested token streams is quite uncommon.

I like your idea of InterpolationStart and InterpolationEnd much more!

So the following:

a = f"hello {"Mr. " + name}, I'm {age} years old"

... would be lexed as:

Ident: "a"
Ctrl: =
StringQualifier: "f"
String: "hello "
InterpolationStart <--- this means {
String: "Mr. "
Ctrl: +
Ident: "name"
InterpolationEnd
String: ", I'm "
InterpolationStart
Ident: age
InterpolationEnd
String: " years old"
StringQualifier: ""

... or maybe like this:

Ident: "a"
Ctrl: =
InterpolationStart: "f" <--- this means f"
InterpolationString: "hello "
String: "Mr. "
Ctrl: +
Ident: "name"
InterpolationString: ", I'm "
Ident: age
InterpolationString: " years old"
InterpolationEnd

Meh, are nested string interpolation even something that is useful?
Sounds like something that needlessly complicates the parser and that nobody would ever actually use or care about.

Meh, are nested string interpolation even something that is useful?
Sounds like something that needlessly complicates the parser and that nobody would ever actually use or care about.

This is a good question!

It depends what form of nesting you're asking about. From the linked issue, this seems helpful, and arguably confusing that we don't allow it:

from invoices
derive x = s"{foo - bar} + 5" # currently fails

OTOH, having s & f strings inside other s & f strings I agree isn't that helpful.

OTOH, having s & f strings inside other s & f strings I agree isn't that helpful.

I do agree, although there are cases where it would be useful:

...
f"my items are { std.array.join this.items "," }"

I worry about language consistency around interpolation:

• if we support only idents, this is not ergonomic, but a reasonable limitation,
• if we support all expressions, this is very ergonomic, but hard to pull off,
• if we support all expressions except string literals and tuples, it gets 90% of ergonomics, but is also quite arbitrary rule that is hard to remember.

I'm not against this incremental improvement, but I would like to have a long-term plan on what we plan on supporting.

Ok, I'm impressed with Pablo's work - Python is able to parse this:

>>> f"My dict: { {'a': 1} }"
"My dict: {'a': 1}"

I've thought this was hard, not I think this is harder. :D

Python is able to parse this:

But not this!

f"{lambda x: x+2}"