Supersedes https://github.com/ziglang/zig/pull/24487
AstSmith
This generates zig ASTs from testing.Smith and is based off the langref's PEG.
The choice to not build the Ast while generating and instead parsing it afterwards makes the smith more versatile by not being tied to a single implementation at a cost of efficiency.
Additionally, a new function boolWeighted was added to Smith due to its frequent use in AstSmith.
PEG / Parser Changes
All the changes made here are to places where the PEG was more permissive than the parser. Changes to the parser make it more permissive and changes to the PEG make it more strict. When choosing between these two options for discrepancies, I opted for the choice that was more natural and increased code readability.
Changes to the Parser
- Tuple types can now be
inline and extern (e.g. extern struct).
- Break labels are now only consumed if both the colon and identifier are present instead of failing if there is only a colon.
- Labeled blocks are no longer parsed in PrimaryExpr (so they are now allowed to have CurlySuffixExpr) as in the PEG.
- While expressions can now be grouped on the same line.
- Added distinction in error messages for "a multiline string literal" so places where only single string literals are allowed do not give "expected 'a string literal', found 'a string literal'".
Changes to the PEG
- Made it so extern functions cannot have a body
- Made it so ... can be only the last function argument
- Made it so many item pointers can't have bit alignment
- Made it so asm inputs / outputs can not be multiline string literals
- Added distinction between block-level statements and regular statements
Pointer Qualifier Order
The PEG allowed for duplicated qualifiers, which the parser did not. The simplest fix for this was to make each be allowed zero or one times which required giving them a order similar to how FnProto already works. The chosen order is the same as used by zig fmt. The parser still accepts them in any order similar to functions.
Backtracking
Made it so several places could not backtrack in the PEG. A common pattern for this was (A / !A).
!ExprSuffix
Expressions ending with expressions now have !ExprSuffix after. This change prevents expressions such as if (a) T else U{} being be parsable as (if (a) T else U){}. It also stops some backtracking, take for example:
if (a) for (b) |c| d else |e| f
It may seem at first that the else clause belongs to the for, however
it actually belongs to the if because for else-clauses cannot have a
payload. This is fixed by a new KEYWORD_else / !KEYWORD_else, however
this alone does not fix more complex cases such as:
if (a) for (b) |c| d() else |e| f
The PEG would first attempt to parse it as expected but fail due to the new guard. It will then backtrack to
if (a) (for (b) |c| d)() else |e| f
which is surprising but avoids the new guard. So, !ExprSuffix is required to disallow this type of backtracking.
!LabelableExpr
For identifiers, excluding labels is necessary despite ordered choice due to pointer bit alignment. For example *align(a : b: for (c) e) T could backtrack to *align(a : b : (for (c) e)) T.
!SinglePtrTypeStart
Prevents expressions like break * break which is parsed as break (*break) backtracking to (break) * (break)
!BlockExpr
Prevents expressions like test { {} = a; } being backtracked to and parsed as test { ({} = a); } (the parenthesis are just for demonstration, that expression is not legal either)
!ExprStatement
In addition to splitting up block level statements, statements that are also parsable as expressions are now part of ExprStatement to disallow backtracking.
zig fmt
The fuzz test for zig fmt has been updated to use AstSmith and has found several new bugs which have been fixed.
Supersedes https://github.com/ziglang/zig/pull/24487
# AstSmith
This generates zig ASTs from `testing.Smith` and is based off the langref's PEG.
The choice to not build the Ast while generating and instead parsing it afterwards makes the smith more versatile by not being tied to a single implementation at a cost of efficiency.
Additionally, a new function `boolWeighted` was added to `Smith` due to its frequent use in `AstSmith`.
# PEG / Parser Changes
All the changes made here are to places where the PEG was more permissive than the parser. Changes to the parser make it more permissive and changes to the PEG make it more strict. When choosing between these two options for discrepancies, I opted for the choice that was more natural and increased code readability.
Changes to the Parser
* Tuple types can now be `inline` and `extern` (e.g. `extern struct`).
* Break labels are now only consumed if both the colon and identifier are present instead of failing if there is only a colon.
* Labeled blocks are no longer parsed in PrimaryExpr (so they are now allowed to have CurlySuffixExpr) as in the PEG.
* While expressions can now be grouped on the same line.
* Added distinction in error messages for "a multiline string literal" so places where only single string literals are allowed do not give "expected 'a string literal', found 'a string literal'".
Changes to the PEG
* Made it so extern functions cannot have a body
* Made it so ... can be only the last function argument
* Made it so many item pointers can't have bit alignment
* Made it so asm inputs / outputs can not be multiline string literals
* Added distinction between block-level statements and regular statements
## Pointer Qualifier Order
The PEG allowed for duplicated qualifiers, which the parser did not. The simplest fix for this was to make each be allowed zero or one times which required giving them a order similar to how FnProto already works. The chosen order is the same as used by zig fmt. The parser still accepts them in any order similar to functions.
## Backtracking
Made it so several places could not backtrack in the PEG. A common pattern for this was (A / !A).
### !ExprSuffix
Expressions ending with expressions now have !ExprSuffix after. This change prevents expressions such as `if (a) T else U{}` being be parsable as `(if (a) T else U){}`. It also stops some backtracking, take for example:
`if (a) for (b) |c| d else |e| f`
It may seem at first that the else clause belongs to the `for`, however
it actually belongs to the `if` because for else-clauses cannot have a
payload. This is fixed by a new `KEYWORD_else / !KEYWORD_else`, however
this alone does not fix more complex cases such as:
`if (a) for (b) |c| d() else |e| f`
The PEG would first attempt to parse it as expected but fail due to the new guard. It will then backtrack to
`if (a) (for (b) |c| d)() else |e| f`
which is surprising but avoids the new guard. So, !ExprSuffix is required to disallow this type of backtracking.
### !LabelableExpr
For identifiers, excluding labels is necessary despite ordered choice due to pointer bit alignment. For example `*align(a : b: for (c) e) T` could backtrack to `*align(a : b : (for (c) e)) T`.
### !SinglePtrTypeStart
Prevents expressions like `break * break` which is parsed as `break (*break)` backtracking to `(break) * (break)`
### !BlockExpr
Prevents expressions like `test { {} = a; }` being backtracked to and parsed as `test { ({} = a); }` (the parenthesis are just for demonstration, that expression is not legal either)
### !ExprStatement
In addition to splitting up block level statements, statements that are also parsable as expressions are now part of ExprStatement to disallow backtracking.
# zig fmt
The fuzz test for zig fmt has been updated to use AstSmith and has found several new bugs which have been fixed.