grammar clarifications regarding tabs and carriage returns #38

andrewrk commented 2021年07月08日 03:33:33 +02:00 (Migrated from github.com)

NL = New Line (0x0a)
CR = Carriage Return (0x0d)
TAB = Tab (0x09)

Inside Line Comments and Documentation Comments

• Any TAB is rejected by the grammar since it is ambiguous how it should be rendered.
• CR directly preceding NL is unambiguously part of the newline sequence. It is accepted by the grammar and removed by zig fmt, leaving only NL.
• CR anywhere else is rejected by the grammar.

Inside Multi-Line String Literals

• zig fmt may not mangle multi line string literals, and therefore the control character TAB are rejected by the grammar inside multi-line string literals.
• CR inside the multiline string literal is also rejected for the same reason
• However CR directly before NL is interpreted as only a newline and not part of the multiline string. zig fmt will delete the CR.

For string literals that want to include CR, TAB, or any other control sequences, they will need to use regular string literals and the ++ operator, or @embedFile.

Whitespace

• TAB used as whitespace is unambiguous. It is accepted by the grammar and replaced by the canonical whitespace by zig fmt.
• CR used as whitespace, whether directly preceding NL or stray, is still unambiguously whitespace. It is accepted by the grammar and replaced by the canonical whitespace by zig fmt.

astronaute-meme

NL = New Line (0x0a) CR = Carriage Return (0x0d) TAB = Tab (0x09) ## Inside Line Comments and Documentation Comments * Any TAB is rejected by the grammar since it is ambiguous how it should be rendered. * CR directly preceding NL is unambiguously part of the newline sequence. It is accepted by the grammar and removed by `zig fmt`, leaving only NL. * CR anywhere else is rejected by the grammar. ## Inside Multi-Line String Literals * `zig fmt` may not mangle multi line string literals, and therefore the control character TAB are rejected by the grammar inside multi-line string literals. * CR inside the multiline string literal is also rejected for the same reason * However CR directly before NL is interpreted as only a newline and not part of the multiline string. zig fmt will delete the CR. For string literals that want to include CR, TAB, or any other control sequences, they will need to use regular string literals and the `++` operator, or `@embedFile`. ## Whitespace * TAB used as whitespace is unambiguous. It is accepted by the grammar and replaced by the canonical whitespace by `zig fmt`. * CR used as whitespace, whether directly preceding NL or stray, is still unambiguously whitespace. It is accepted by the grammar and replaced by the canonical whitespace by `zig fmt`. 

vrischmann commented 2021年07月15日 12:12:48 +02:00 (Migrated from github.com)

This

zig fmt may not mangle multi line string literals, and therefore the control characters TAB and CR are rejected by the grammar inside multi-line string literals.

causes a problem when using git on windows, unless I'm missing something.

If you clone this gist (which contains no CR) on Windows with the default git configuration, it produces this:

PS C:\Users\vincent\dev\tmp\zig-invalid-bytes2> xxd .\main.zig
00000000: 666e 2063 7265 6174 6554 6573 7454 6162 fn createTestTab
00000010: 6c65 7328 2920 2176 6f69 6420 7b0d 0a20 les() !void {..
00000020: 2020 2063 6f6e 7374 2062 6172 203d 2026 const bar = &
00000030: 5b5f 5d5b 5d63 6f6e 7374 2075 387b 0d0a [_][]const u8{..
00000040: 2020 2020 2020 2020 5c5c 666f 6f62 6172 \\foobar
00000050: 0d0a 2020 2020 7d3b 0d0a 2020 2020 5f20 .. };.. _
00000060: 3d20 6261 723b 0d0a 7d = bar;..}

which does not compile:

PS C:\Users\vincent\dev\tmp\zig-invalid-bytes2> zig build-exe .\main.zig
.\main.zig:3:17: error: expected ',', found invalid bytes
 \\foobar
 ^
.\main.zig:3:18: note: invalid byte: '\n'
 \\foobar
 ^

it is the intended behaviour but IMO it's surprising. Currently as far as I can see you need to either configure git with core.autocrlf=false or use a gitattributes files.

This > zig fmt may not mangle multi line string literals, and therefore the control characters TAB and CR are rejected by the grammar inside multi-line string literals. causes a problem when using git on windows, unless I'm missing something. If you clone [this gist](https://gist.github.com/f6bd051870769d26c44722f17f637690.git) (which contains no CR) on Windows with the default git configuration, it produces this: ``` PS C:\Users\vincent\dev\tmp\zig-invalid-bytes2> xxd .\main.zig 00000000: 666e 2063 7265 6174 6554 6573 7454 6162 fn createTestTab 00000010: 6c65 7328 2920 2176 6f69 6420 7b0d 0a20 les() !void {.. 00000020: 2020 2063 6f6e 7374 2062 6172 203d 2026 const bar = & 00000030: 5b5f 5d5b 5d63 6f6e 7374 2075 387b 0d0a [_][]const u8{.. 00000040: 2020 2020 2020 2020 5c5c 666f 6f62 6172 \\foobar 00000050: 0d0a 2020 2020 7d3b 0d0a 2020 2020 5f20 .. };.. _ 00000060: 3d20 6261 723b 0d0a 7d = bar;..} ``` which does not compile: ``` PS C:\Users\vincent\dev\tmp\zig-invalid-bytes2> zig build-exe .\main.zig .\main.zig:3:17: error: expected ',', found invalid bytes \\foobar ^ .\main.zig:3:18: note: invalid byte: '\n' \\foobar ^ ``` it is the intended behaviour but IMO it's surprising. Currently as far as I can see you need to either configure git with `core.autocrlf=false` or use a [gitattributes files](https://www.git-scm.com/docs/gitattributes#_eol).

thejoshwolfe commented 2021年07月15日 15:58:15 +02:00 (Migrated from github.com)

It still surprises me that Git's default configuration is to give you the wrong bytes on windows. Can you clarify which client you're using? is it provided by GitHub? by git-scm.com? by a cygwin package manager (probably not this one)?

As much as I'd like to say that Git is wrong (which i kinda did in the last paragraph), the conflict is arising from the concept of a "text file", which many people think is an array of lines rather than a single byte array. Sometimes discussions of character encoding get tangled up in the concept of a "text file" too.

But Zig is not interested in all the complexity of "text files", and specifies always UTF-8, always spaces over tabs, always LF line endings. There's some room for accepting and canonicalizing (in zig fmt) unambiguous alternatives for everyone's convenience, but CRLF line endings are officially the wrong way to end lines in Zig (along with NEL, LS, and PS). While it bothers me emotionally that some Git cient on Windows is wasting our time here talking about CRLF line endings, it seems important to get this right to avoid even more wasted time for everyone in the future.

With that preamble ramble out of the way: i think we should definitely canonicalize CRLF line endings in multiline string literals to LF line endings.

It still surprises me that Git's default configuration is to give you the wrong bytes on windows. Can you clarify which client you're using? is it provided by GitHub? by git-scm.com? by a cygwin package manager (probably not this one)? As much as I'd like to say that Git is wrong (which i kinda did in the last paragraph), the conflict is arising from the concept of a "text file", which many people think is an array of lines rather than a single byte array. Sometimes discussions of character encoding get tangled up in the concept of a "text file" too. But Zig is not interested in all the complexity of "text files", and specifies always UTF-8, always spaces over tabs, always LF line endings. There's some room for accepting and canonicalizing (in zig fmt) unambiguous alternatives for everyone's convenience, but CRLF line endings are officially the wrong way to end lines in Zig (along with NEL, LS, and PS). While it bothers me emotionally that some Git cient on Windows is wasting our time here talking about CRLF line endings, it seems important to get this right to avoid even more wasted time for everyone in the future. With that preamble ramble out of the way: i think we should definitely canonicalize CRLF line endings in multiline string literals to LF line endings.

vrischmann commented 2021年07月15日 16:31:29 +02:00 (Migrated from github.com)

I'm using the installer from git-scm.

I'm using the installer from [git-scm](https://git-scm.com/download/win).

Vexu commented 2021年07月15日 16:53:06 +02:00

Couldn't we just add an error like file uses tabs/carriage returns; run 'zig fmt {path-to-file}' to convert it to the canonical form. It would allow us to completely ban tabs/CRLF while also providing a pretty decent user experience.

If mangling the files is an issue, the feature can be put behind a --convert-whitespace flag.

Couldn't we just add an error like `file uses tabs/carriage returns; run 'zig fmt {path-to-file}' to convert it to the canonical form`. It would allow us to completely ban tabs/CRLF while also providing a pretty decent user experience. If mangling the files is an issue, the feature can be put behind a `--convert-whitespace` flag.

nektro commented 2022年04月29日 01:48:11 +02:00 (Migrated from github.com)

could even make zig build call zig fmt on the whole directory before running, or otherwise tell them to run it on the whole folder since its likely not the one file thats mangled

could even make `zig build` call `zig fmt` on the whole directory before running, or otherwise tell them to run it on the whole folder since its likely not the one file thats mangled

andrewrk commented 2022年10月31日 01:24:41 +01:00 (Migrated from github.com)

Couldn't we just add an error like file uses tabs/carriage returns; run 'zig fmt {path-to-file}' to convert it to the canonical form. It would allow us to completely ban tabs/CRLF while also providing a pretty decent user experience.

Yes, I'm thinking along the same lines. Make it a hard error, just like unused locals, and then have --autofix (as well as zig fmt) make the compilation "just work" while modifying the source files to comply.

> Couldn't we just add an error like `file uses tabs/carriage returns; run 'zig fmt {path-to-file}' to convert it to the canonical form`. It would allow us to completely ban tabs/CRLF while also providing a pretty decent user experience. Yes, I'm thinking along the same lines. Make it a hard error, just like unused locals, and then have `--autofix` (as well as `zig fmt`) make the compilation "just work" while modifying the source files to comply.

rockorager commented 2024年08月03日 03:14:21 +02:00 (Migrated from github.com)

For string literals that want to include [...] any other control sequences, they will need to use regular string literals and the ++ operator, or @embedFile.

This is specific to multi-line string literals, right? I can still have const seq = "\x1b[31m";? Or potentially any other string literal that contains 0x00...0x1F, except for CR, NL, and Tab?

> For string literals that want to include [...] any other control sequences, they will need to use regular string literals and the ++ operator, or @embedFile. This is specific to *multi-line* string literals, right? I can still have `const seq = "\x1b[31m";`? Or potentially any other string literal that contains `0x00`...`0x1F`, *except* for `CR`, `NL`, and `Tab`?

castholm commented 2024年08月05日 23:50:37 +02:00

This is specific to multi-line string literals, right? I can still have const seq = "\x1b[31m";? Or potentially any other string literal that contains 0x00...0x1F, except for CR, NL, and Tab?

Yes, this clarification is specifically for multi-line string literals. Regular string literals can of course still use escape sequences to encode characters/byte sequences like control characters or invalid UTF-8 sequences that are illegal in Zig source code. But multi-line string literals don't support escape sequences and thus have no way to encode HT, LF, CR or other illegal characters, so for those rare cases you are need to concatenate the multi-line string literal with a regular string literal containing the necessary escape sequences, or put the text data in a separate file and import it using @embedfile.

> This is specific to _multi-line_ string literals, right? I can still have `const seq = "\x1b[31m";`? Or potentially any other string literal that contains `0x00`...`0x1F`, _except_ for `CR`, `NL`, and `Tab`? Yes, this clarification is specifically for multi-line string literals. Regular string literals can of course still use escape sequences to encode characters/byte sequences like control characters or invalid UTF-8 sequences that are illegal in Zig source code. But multi-line string literals don't support escape sequences and thus have no way to encode HT, LF, CR or other illegal characters, so for those rare cases you are need to concatenate the multi-line string literal with a regular string literal containing the necessary escape sequences, or put the text data in a separate file and import it using `@embedfile`.

PanSashko commented 2024年09月08日 13:09:23 +02:00 (Migrated from github.com)

The restriction to use tabs in comments and multi line strings is frustrating.

What is "rendering" and why does it matter?

It is a really annoying change for any project where there are comments or multi line strings with tabs. But what are real benefits for others who do not care?

The restriction to use tabs in comments and multi line strings is frustrating. What is "rendering" and why does it matter? It is a really annoying change for any project where there are comments or multi line strings with tabs. But what are real benefits for others who do not care?

thejoshwolfe commented 2024年09月08日 15:46:07 +02:00 (Migrated from github.com)

@PanSashko is your project where you want tabs in comments and strings something you can link to? It might help better understand the pros and cons to see a real example.

@PanSashko is your project where you want tabs in comments and strings something you can link to? It might help better understand the pros and cons to see a real example.

PanSashko commented 2024年09月08日 17:11:59 +02:00 (Migrated from github.com)

@thejoshwolfe unfortunately I cannot publish it yet. But I can describe 2 examples.

Imagine a pet language that is indent-based like Python. Let's say it supports tabs or spaces for indentation. Then, e.g., parser tests should cover both cases. Some small tests are using multi line strings for embedding source code under test.
An obvious workaround for that case is @embedFile, but why?
With the restriction not all UTF-8 strings can be represented in Zig's multi line strings.

Other example. Imagine a Zig project and 2 developers that work on it. One of them prefers 4-space indents, other one 2-space indents. The easiest option is to use tabs, and that's allowed by Zig, according to this issue. But the comments restriction disables an ability to temporarily comment out a piece of code, as it does not compile.
So the convenient code commenting is broken for any Zig project that uses tabs. As I understand, that is not justified, because regular comments are ignored by compiler anyway.

I think before this change Zig coders from both sides of the "spaces vs tabs war" were happy. Also everyone who did not care was happy. This why I really wonder what is the advantage of this change?

@thejoshwolfe unfortunately I cannot publish it yet. But I can describe 2 examples. Imagine a pet language that is indent-based like Python. Let's say it supports tabs or spaces for indentation. Then, e.g., parser tests should cover both cases. Some small tests are using multi line strings for embedding source code under test. An obvious workaround for that case is `@embedFile`, but why? With the restriction not all UTF-8 strings can be represented in Zig's multi line strings. Other example. Imagine a Zig project and 2 developers that work on it. One of them prefers 4-space indents, other one 2-space indents. The easiest option is to use tabs, and that's allowed by Zig, according to this issue. But the comments restriction disables an ability to temporarily comment out a piece of code, as it does not compile. So the convenient code commenting is broken for any Zig project that uses tabs. As I understand, that is not justified, because regular comments are ignored by compiler anyway. I think before this change Zig coders from both sides of the "spaces vs tabs war" were happy. Also everyone who did not care was happy. This why I really wonder what is the advantage of this change?

castholm commented 2024年09月08日 18:37:52 +02:00

One problem with rejecting tabs in comments is that you can't easily comment out code. If you are forced to write Zig with tabs for whatever reason, you will need to find a way to make your editor replace the commented out tabs with spaces (and then bring them back when uncommenting). Just selecting lines and prepending them with // won't work if they contain indentation.

One problem with rejecting tabs in comments is that you can't easily comment out code. If you are forced to write Zig with tabs for whatever reason, you will need to find a way to make your editor replace the commented out tabs with spaces (and then bring them back when uncommenting). Just selecting lines and prepending them with `//` won't work if they contain indentation.

thejoshwolfe commented 2024年09月08日 21:35:21 +02:00 (Migrated from github.com)

i certainly agree that it's confusing to get an error when commenting out working code. the original design of zig was to consistently disallow tabs (and all this is true about carriage returns as well), but in order for zig fmt to fix tabs, it had to support parsing them without error. once the default implementation of zig switched to the self hosted one, the main compiler got this tolerance as well.

however, there is still an intention to disallow hard tabs in zig source, whether in a comment, a string literal, or whitespace around tokens. see andrewrk's comment above. this would avoid the confusion when commenting out code, because the error would be much sooner.

What is "rendering" and why does it matter?

the relevant aspect of rendering in this context is the vertical alignment of grapheme clusters across multiple lines when using a monospace font. different editor implementations and configurations are also relevant. i have a hard time explaining why it matters in general, but certainly there are plenty of specific cases where the text is intended to align in a particular way independent of editor configuration.

Imagine a pet language that is indent-based like Python. Let's say it supports tabs or spaces for indentation. Then, e.g., parser tests should cover both cases. Some small tests are using multi line strings for embedding source code under test.

I don't have an easy way to search the zig codebase at the moment, but I'm pretty sure zig's own tests do this. the workaround is to use ++ "\t" ++ style string concatenation to get the invalid characters into the test case, or at least that's an option i would reach for. pretty awkward, but it seems appropriate for how unusual the use case is.

Imagine a Zig project and 2 developers that work on it. One of them prefers 4-space indents, other one 2-space indents. The easiest option is to use tabs,

The intent is that you use zig fmt, and then your preferences yield to the community standard. allowing different preferences for indentation is an explicit antifeature of zig fmt.

I think before this change Zig coders from both sides of the "spaces vs tabs war" were happy

ideally, we get to a point where is no spaces vs tabs war among zig authors, because there will be no tabs in zig. that's what i hope for at least.

If you are forced to write Zig with tabs for whatever reason

@castholm are you in that situation? how did that happen?

i certainly agree that it's confusing to get an error when commenting out working code. the original design of zig was to consistently disallow tabs (and all this is true about carriage returns as well), but in order for `zig fmt` to fix tabs, it had to support parsing them without error. once the default implementation of `zig` switched to the self hosted one, the main compiler got this tolerance as well. however, there is still an intention to disallow hard tabs in zig source, whether in a comment, a string literal, or whitespace around tokens. see andrewrk's comment above. this would avoid the confusion when commenting out code, because the error would be much sooner. > What is "rendering" and why does it matter? the relevant aspect of rendering in this context is the vertical alignment of grapheme clusters across multiple lines when using a monospace font. different editor implementations and configurations are also relevant. i have a hard time explaining why it matters in general, but certainly there are plenty of specific cases where the text is intended to align in a particular way independent of editor configuration. > Imagine a pet language that is indent-based like Python. Let's say it supports tabs or spaces for indentation. Then, e.g., parser tests should cover both cases. Some small tests are using multi line strings for embedding source code under test. I don't have an easy way to search the zig codebase at the moment, but I'm pretty sure zig's own tests do this. the workaround is to use `++ "\t" ++` style string concatenation to get the invalid characters into the test case, or at least that's an option i would reach for. pretty awkward, but it seems appropriate for how unusual the use case is. > Imagine a Zig project and 2 developers that work on it. One of them prefers 4-space indents, other one 2-space indents. The easiest option is to use tabs, The intent is that you use `zig fmt`, and then your preferences yield to the community standard. allowing different preferences for indentation is an explicit antifeature of `zig fmt`. > I think before this change Zig coders from both sides of the "spaces vs tabs war" were happy ideally, we get to a point where is no spaces vs tabs war among zig authors, because there will be no tabs in zig. that's what i hope for at least. > If you are forced to write Zig with tabs for whatever reason @castholm are you in that situation? how did that happen?

castholm commented 2024年09月08日 22:07:28 +02:00

are you in that situation? how did that happen?

I'm not (but I recall from earlier discussion from back when tabs were hard banned that some people suggested they need them for accessibility) and fwiw I'm strictly on the spaces side. I was mainly just trying to make a point that allowing them in source but not comments is the worst of both worlds.
If the end goal is to begrudgingly support tabs but as a second-class citizen (i.e. still have the canonical zig fmt format favor spaces), they should also be supported in regular comments.
If the end goal is to make tabs a compile error, they shouldn't be supported anywhere. But as you mention, since zig fmt is currently the mechanism that normalizes source encoding, for zig fmt to even be able to parse and fix up the code they would need to be allowed.

> are you in that situation? how did that happen? I'm not (but I recall from earlier discussion from back when tabs were hard banned that some people suggested they need them for accessibility) and fwiw I'm strictly on the spaces side. I was mainly just trying to make a point that allowing them in source but not comments is the worst of both worlds. If the end goal is to begrudgingly support tabs but as a second-class citizen (i.e. still have the canonical `zig fmt` format favor spaces), they should also be supported in regular comments. If the end goal is to make tabs a compile error, they shouldn't be supported anywhere. But as you mention, since `zig fmt` is currently the mechanism that normalizes source encoding, for `zig fmt` to even be able to parse and fix up the code they would need to be allowed.

PanSashko commented 2024年09月09日 00:19:28 +02:00 (Migrated from github.com)

pretty awkward, but it seems appropriate for how unusual the use case is.

I do not see any point in awkwardness, when it is so easy to fix. But in this case it was working, but now it is intentionally broken.

On other hand, "unambiguous source formatting" looks to me like a fictional benefit. I would be happy to hear at least one practical example when it is required or useful, because I cannot find any.

Recommending zig fmt and "community standards" for everyone is OK. But in this case it causes annoyance for some with no benefits for others.

> pretty awkward, but it seems appropriate for how unusual the use case is. I do not see any point in awkwardness, when it is so easy to fix. But in this case it was working, but now it is intentionally broken. On other hand, "unambiguous source formatting" looks to me like a fictional benefit. I would be happy to hear at least one practical example when it is required or useful, because I cannot find any. Recommending `zig fmt` and "community standards" for everyone is OK. But in this case it causes annoyance for some with no benefits for others.

thejoshwolfe commented 2024年09月28日 17:22:52 +02:00 (Migrated from github.com)

On other hand, "unambiguous source formatting" looks to me like a fictional benefit. I would be happy to hear at least one practical example when it is required or useful, because I cannot find any.

Back in Java 1.5, before openjdk, the source for java.lang.String and other classes was formatted using a mix of tabs and spaces, where the intent was that a tab stood in for 8 spaces (to save disk space or something probably). I can't find the source code for jdk pre openjdk 7 (without creating an oracle account?), but i can recreate it as an example. Starting with this modern excerpt: github.com/openjdk/jdk@73ebb848fd/src/java.base/share/classes/java/lang/String.java (L552-L581)

then replacing 8 spaces with tabs to recreate the style of jdk 1.5, it looks like this in github web/mobile app/whatever you're using, which for me renders the intended way:

@SuppressWarnings("removal")privateString(Charsetcharset,byte[]bytes,intoffset,intlength){if(length==0){this.value="".value;this.coder="".coder;}elseif(charset==UTF_8.INSTANCE){if(COMPACT_STRINGS){intdp=StringCoding.countPositives(bytes,offset,length);if(dp==length){this.value=Arrays.copyOfRange(bytes,offset,offset+length);this.coder=LATIN1;return;}// Decode with a stable copy, to be the result if the decoded length is the samebyte[]latin1=Arrays.copyOfRange(bytes,offset,offset+length);intsp=dp;// first dp bytes are already in the copywhile(sp<length){intb1=latin1[sp++];if(b1>=0){latin1[dp++]=(byte)b1;continue;}if((b1&0xfe)==0xc2&&sp<length){// b1 either 0xc2 or 0xc3intb2=latin1[sp];if(b2<-64){// continuation bytes are always negative values in the range -128 to -65latin1[dp++]=(byte)decode2(b1,b2);sp++;continue;}}

And if you set tabs to appear as 2 spaces, a configuration i've seen commonly in web development contexts, it looks like this:

image

I would call that simply incorrect behavior. For the indentation to appear to go backwards when it should go forwards is a failure between the source code author and the editor configuration. The original authors (according to my story, which i've simulated in this discussion) were vigilant about formatting the indentation consistently; it wasn't sloppiness; it was a strategy that was prone to ambiguous rendering.

In countless modern scenarios, mostly in closed-source code authored by my colleagues at work, I've encountered a few stray tabs here and there where someone didn't notice that their editor was defaulting to inserting tabs where there were already spaces or visa versa. This typically happens in code bases without auto formatters, linters, or other modern comforts, such as in shell scripts for devops use cases. It looked correct with their editor configuration, which is often 4-space tab width, but then in the github PR review web interface, tabs are typically rendered 8-spaces wide. This causes incorrect indentation rendering, which is a failure between the author and the renderer configuration.

These types of failure are the main casualties in the tabs vs spaces war. There are many solutions, of course, and an auto formatter is involved in many of them. Zig has an autoformatter, and it's generally recommended to use it in order to avoid ambiguous and incorrect appearing indentation like these examples.

> On other hand, "unambiguous source formatting" looks to me like a fictional benefit. I would be happy to hear at least one practical example when it is required or useful, because I cannot find any. Back in Java 1.5, before openjdk, the source for `java.lang.String` and other classes was formatted using a mix of tabs and spaces, where the intent was that a tab stood in for 8 spaces (to save disk space or something probably). I can't find the source code for jdk pre openjdk 7 (without creating an oracle account?), but i can recreate it as an example. Starting with this modern excerpt: https://github.com/openjdk/jdk/blob/73ebb848fdb66861e912ea747c039ddd1f7a5f48/src/java.base/share/classes/java/lang/String.java#L552-L581 then replacing 8 spaces with tabs to recreate the style of jdk 1.5, it looks like this in github web/mobile app/whatever you're using, which for me renders the intended way: ```java @SuppressWarnings("removal") private String(Charset charset, byte[] bytes, int offset, int length) { if (length == 0) { this.value = "".value; this.coder = "".coder; } else if (charset == UTF_8.INSTANCE) { if (COMPACT_STRINGS) { int dp = StringCoding.countPositives(bytes, offset, length); if (dp == length) { this.value = Arrays.copyOfRange(bytes, offset, offset + length); this.coder = LATIN1; return; } // Decode with a stable copy, to be the result if the decoded length is the same byte[] latin1 = Arrays.copyOfRange(bytes, offset, offset + length); int sp = dp; // first dp bytes are already in the copy while (sp < length) { int b1 = latin1[sp++]; if (b1 >= 0) { latin1[dp++] = (byte)b1; continue; } if ((b1 & 0xfe) == 0xc2 && sp < length) { // b1 either 0xc2 or 0xc3 int b2 = latin1[sp]; if (b2 < -64) { // continuation bytes are always negative values in the range -128 to -65 latin1[dp++] = (byte)decode2(b1, b2); sp++; continue; } } ``` And if you set tabs to appear as 2 spaces, a configuration i've seen commonly in web development contexts, it looks like this:  I would call that simply incorrect behavior. For the indentation to appear to go backwards when it should go forwards is a failure between the source code author and the editor configuration. The original authors (according to my story, which i've simulated in this discussion) were vigilant about formatting the indentation consistently; it wasn't sloppiness; it was a strategy that was prone to ambiguous rendering. In countless modern scenarios, mostly in closed-source code authored by my colleagues at work, I've encountered a few stray tabs here and there where someone didn't notice that their editor was defaulting to inserting tabs where there were already spaces or visa versa. This typically happens in code bases without auto formatters, linters, or other modern comforts, such as in shell scripts for devops use cases. It looked correct with their editor configuration, which is often 4-space tab width, but then in the github PR review web interface, tabs are typically rendered 8-spaces wide. This causes incorrect indentation rendering, which is a failure between the author and the renderer configuration. These types of failure are the main casualties in the tabs vs spaces war. There are many solutions, of course, and an auto formatter is involved in many of them. Zig has an autoformatter, and it's generally recommended to use it in order to avoid ambiguous and incorrect appearing indentation like these examples.

PanSashko commented 2024年09月30日 09:32:39 +02:00 (Migrated from github.com)

@thejoshwolfe you described a "problem" that Zig still has too. It allows mixed indentation, so Zig sources with mixed indentation will look like your screenshot for some tab widths.

On other hand, zig fmt solves all of that for anyone who cares.
I don't see the point of breaking developer experience for those who use tabs and do not use zig fmt.

@thejoshwolfe you described a "problem" that Zig still has too. It allows mixed indentation, so Zig sources with mixed indentation will look like your screenshot for some tab widths. On other hand, `zig fmt` solves all of that for anyone who cares. I don't see the point of breaking developer experience for those who use tabs and do not use `zig fmt`.

IntegratedQuantum commented 2024年10月03日 14:52:22 +02:00 (Migrated from github.com)

I just came across this because I wanted to comment out some code to take care of after testing the other changes that I made.
And I have to say, getting an error for commented code is a pretty bad user experience.

Sure there is a workaround for this: Only comment the code without the tabs in front of it.
This is quite easy to do with a multi-cursor capable editor, but it is non-obvious.

Do you really want to return to the old (stage1) days, where we (tab users) had to write an extra pre-processing step in the build.zig just to be able to conveniently use tabs in the source code?

I just came across this because I wanted to comment out some code to take care of after testing the other changes that I made. And I have to say, getting an error for commented code is a pretty bad user experience. Sure there is a workaround for this: Only comment the code without the tabs in front of it. This is quite easy to do with a multi-cursor capable editor, but it is non-obvious. Do you really want to return to the old (stage1) days, where we (tab users) had to write an extra pre-processing step in the build.zig just to be able to conveniently use tabs in the source code?

jibal commented 2024年11月09日 21:15:37 +01:00 (Migrated from github.com)

"CR used as whitespace, whether directly preceding NL or stray, is still unambiguously whitespace."

But it's not unambiguously rendered ... there's no telling how it will be displayed, and on some displays it will hide code. If you're concerned about rendering, then you should ban CRs that don't immediately precede NL ... unlike tabs, there's no reason at all for anyone to want a stray CR. The reference manual gets it right:

Each LF may be immediately preceded by a single CR (byte value 0x0d, code point U+000d, '\r') to form a Windows style line ending, but this is discouraged. Note that in multiline strings, CRLF sequences will be encoded as LF when compiled into a zig program. A CR in any other context is not allowed.

BTW, the reference manual has the wrong (example) regex for line ends (how has no one noticed? Or is feedback about such things broken? I like that the D documentation has a link on every page to file an error report): /\r\n?|[\n\u0085\u2028\u2029]/

"CR used as whitespace, whether directly preceding NL or stray, is still unambiguously whitespace." But it's not unambiguously _rendered_ ... there's no telling how it will be displayed, and on some displays it will hide code. If you're concerned about rendering, then you should ban CRs that don't immediately precede NL ... unlike tabs, there's no reason at all for anyone to want a stray CR. The reference manual gets it right: > Each LF may be immediately preceded by a single CR (byte value 0x0d, code point U+000d, '\r') to form a Windows style line ending, but this is discouraged. Note that in multiline strings, CRLF sequences will be encoded as LF when compiled into a zig program. **A CR in any other context is not allowed.** BTW, the reference manual has the wrong (example) regex for line ends (how has no one noticed? Or is feedback about such things broken? I like that the D documentation has a link on every page to file an error report): /\r\n?|[\n\u0085\u2028\u2029]/

jibal commented 2024年11月09日 21:42:51 +01:00 (Migrated from github.com)

Inside Line Comments [...]
Any TAB is rejected by the grammar since it is ambiguous how it should be rendered.

But who cares? I suggest that inside of non-doc comments anything at all should be allowed, including ctrl chars and chars with the high bit set (so arbitrary unicode). Since non-doc comments are completely elided, it's none of the compiler's (or language designers') damn business what is in them.

> Inside Line Comments [...] > Any TAB is rejected by the grammar since it is ambiguous how it should be rendered. But who cares? I suggest that inside of non-doc comments anything at all should be allowed, including ctrl chars and chars with the high bit set (so arbitrary unicode). Since non-doc comments are completely elided, it's none of the compiler's (or language designers') damn business what is in them.

gamebox commented 2025年05月12日 20:28:05 +02:00 (Migrated from github.com)

I'm going to speak up and say that this is currently biting us on the Roc language team as well. In our rewrite, I've moved to implement our decision to use TAB for indentation in all code formatted by roc fmt and in doing so ran into a ton of our parser tests being virtually unimplementable without moving them to separate files.

I think it's OK for us to do this long term, but this change (in multiline string specifically) is taking a very simple change I could do on a lunch break and turning it into a complete refactoring of the parser tests - for what seems like minimal benefit. The choice Zig has made with regards to tabs in general is fine, but restricting the user's ability to use them in places the language should fundamentally care nothing or very little about seems a bit much.

I'd love to see an example where this would "save someone's bacon" in the context of a multiline string literal.

We love Zig but this just feels to me personally like an unnecessary speedbump.

I'm going to speak up and say that this is currently biting us on the Roc language team as well. In our rewrite, I've moved to implement our decision to use TAB for indentation in all code formatted by `roc fmt` and in doing so ran into a ton of our parser tests being virtually unimplementable without moving them to separate files. I think it's OK for us to do this long term, but this change (in multiline string specifically) is taking a very simple change I could do on a lunch break and turning it into a complete refactoring of the parser tests - for what seems like minimal benefit. The choice Zig has made with regards to tabs in general is fine, but restricting the user's ability to use them in places the language should fundamentally care nothing or very little about seems a bit much. I'd love to see an example where this would "save someone's bacon" in the context of a multiline string literal. We love Zig but this just feels to me personally like an unnecessary speedbump.

abvee commented 2025年06月21日 08:15:56 +02:00 (Migrated from github.com)

the relevant aspect of rendering in this context is the vertical alignment of grapheme clusters across multiple lines when using a monospace font. different editor implementations and configurations are also relevant. i have a hard time explaining why it matters in general, but certainly there are plenty of specific cases where the text is intended to align in a particular way independent of editor configuration.

Why should a language compiler care about how text is rendered, and what font is being used ?? I admit that these are problems, but they should be solved by the terminal emulator or the editor. Either that or you should expect every single program that outputs to a terminal to care about these things.

Forcing no tabs in comments also doesn't make sense. Wouldn't I have the same rendering problems with regular tabs ? I don't know too much about rendering, but why would adding // in the beginning cause rendering issues when tabs are used ??

>the relevant aspect of rendering in this context is the vertical alignment of grapheme clusters across multiple lines when using a monospace font. different editor implementations and configurations are also relevant. i have a hard time explaining why it matters in general, but certainly there are plenty of specific cases where the text is intended to align in a particular way independent of editor configuration. Why should a language compiler care about how text is rendered, and what font is being used ?? I admit that these are problems, but they should be solved by the terminal emulator or the editor. Either that or you should expect every single program that outputs to a terminal to care about these things. Forcing no tabs in comments also doesn't make sense. Wouldn't I have the same rendering problems with regular tabs ? I don't know too much about rendering, but why would adding `//` in the beginning cause rendering issues when tabs are used ??

KilianHanich commented 2025年07月28日 07:31:15 +02:00 (Migrated from github.com)

What is "rendering" and why does it matter?

the relevant aspect of rendering in this context is the vertical alignment of grapheme clusters across multiple lines when using a monospace font. different editor implementations and configurations are also relevant. i have a hard time explaining why it matters in general, but certainly there are plenty of specific cases where the text is intended to align in a particular way independent of editor configuration.

There are many different kinds of visual disabilities (not just blindness or bad eyesight or colour weakness, or colour blindness) and for some setting the font size to 120 with 1 indent = 1 character and for others it can be font size 16 with 1 indent = 30 characters, and there are many others. Using tabs, this is easy. Using spaces, it's not. Here's a small saying I heard from a documentary many years ago: "What may be convenience to normal people can be a necessity for disabled people".

zig fmt does deliver on ensuring consistency, which is actually important for screen readers. This can lead to the interesting fact that in e.g. Germany inconsistently formatted source code can lead to fines for companies since it messes with the ability for screen readers to properly understand the code.

But since it does so via spaces among other things, it makes things more difficult for certain disabilities since they would constantly need to reformat before/after e.g. committing to source control if they used it (in some languages this already happens, leading sometimes to interesting bugs).

This would become a major problem if the compiler would start to care about it too.

Following from that, I don't think the compiler should care about formatting or rendering, only zig fmt should, since enforcing it in the compiler too would make it more difficult to read source code from other people for certain people without patching or creating alternative compilers which I don't think is a worthwhile hill to die on (besides the fact that it's not nice towards them).

> > What is "rendering" and why does it matter? > > the relevant aspect of rendering in this context is the vertical alignment of grapheme clusters across multiple lines when using a monospace font. different editor implementations and configurations are also relevant. i have a hard time explaining why it matters in general, but certainly there are plenty of specific cases where the text is intended to align in a particular way independent of editor configuration. There are many different kinds of visual disabilities (not just blindness or bad eyesight or colour weakness, or colour blindness) and for some setting the font size to 120 with 1 indent = 1 character and for others it can be font size 16 with 1 indent = 30 characters, and there are many others. Using tabs, this is easy. Using spaces, it's not. Here's a small saying I heard from a documentary many years ago: "What may be convenience to normal people can be a necessity for disabled people". `zig fmt` does deliver on ensuring consistency, which is actually important for screen readers. This can lead to the interesting fact that in e.g. Germany inconsistently formatted source code can lead to fines for companies since it messes with the ability for screen readers to properly understand the code. But since it does so via spaces among other things, it makes things more difficult for certain disabilities since they would constantly need to reformat before/after e.g. committing to source control if they used it (in some languages this already happens, leading sometimes to interesting bugs). This would become a major problem if the compiler would start to care about it too. Following from that, I don't think the **compiler** should care about formatting or rendering, only `zig fmt` should, since enforcing it in the compiler too would make it more difficult to **read** source code from other people for certain people without patching or creating alternative compilers which I don't think is a worthwhile hill to die on (besides the fact that it's not nice towards them).

mnemnion commented 2025年09月25日 17:51:45 +02:00 (Migrated from github.com)

I'll weigh in: this policy is fundamentally misguided when it comes to multiline strings.

I understand having no tabs in code, that makes sense to me, for all the reasons already covered. I agree with it in fact. A future where zig build* won't allow them, and zig fmt standardizes them to spaces, is a good one to aim for. That will also solve the main pain point with disallowing tabs in comments.

Comments in Zig are Markdown, and Markdown is indentation-sensitive. That's a good and sufficient reason to ban them there as well. Arguably it could also convert them to the same number of spaces as it does tab indents, I don't care either way.

Multiline strings though? Tell me, what business does zig fmt have in formatting the interior of a string, thus changing the data it represents into something else? None, it has no business doing that. So the fact that it can't turn those tabs into spaces is irrelevant, because it should not do so in the first place.

The question is should they be legal. My answer is yes.

One of the important use cases for multiline strings is embedding formatted textual data, and out in the wild and wooly real world, tabs remain widespread in data formats. Makefiles require tabs. Properly formatted Go code is lousy with them. Roc, as we just found out, uses tabs. It also uses Zig. The Zig is causing problems for the tabs. I don't think it's Zig's place to be opinionated about how Roc does indentation.

This is not a useful place to be purist about HT being haraam. What problem does it solve? How does it make Zig a better language, one I can confidently recommend to others without too much apologizing on its behalf? How does it serve the user to ban tabs in multiline strings?

Please reconsider. Thank you.

\r and \v

Some hypothetical person counters: what about CR and VT, those are classically considered whitespace, what about them?

To which I reply: LF is actually a delimiter in multiline strings, one which defines a closed interval (meaning the newline is included in the data), but a delimiter nonetheless. Allowing a CR anywhere other than directly before a newline would be perverse, no living operating system considers a bare CR to be whitespace, so in practice this would mean two forms of the delimiter, where the difference is invisible. Some lines ending in CRLF and others in just LF is a mistake, clearly, how should it be fixed? Answer: no CR.

Also it would result in a source file which mixes LF and CRLF delimiters, since Zig doesn't allow the latter anywhere else. And that is truly haraam, it would cause numerous problems to no benefit.

I say "no benefit" because DOSifying a UNIX text file is a solved problem. It's clear where the CRs go, after all. If they're absolutely needed in static memory, this can in fact be easily done at comptime, it's just not a practical issue. With tabs things are different; the tab ban causes actual problems and they're not fixable with a trivial filter.

Vertical tab is nearly extinct in the wild, fortunately, and remaining use cases always put it between newlines. So if someone did need it, no problem:

\\ /* This is a weird GNU source file++"\x0b\n"++// Vertical tab goes here\\ and now it has a vertical tab in it */

Tabs are very far from being extinct in the wild, and I don't expect it will happen in my lifetime, if ever. Rather, they are ubiquitous.

I end up answering a lot of Zig newcomer questions, and I don't like it when I have to fall back on something like "the core team just, really hates tabs". I don't think ensuring that rg '\r' comes back clean is worth it.

I'll weigh in: this policy is fundamentally misguided when it comes to multiline strings. I understand having no tabs in code, that makes sense to me, for all the reasons already covered. I agree with it in fact. A future where `zig build*` won't allow them, and `zig fmt` standardizes them to spaces, is a good one to aim for. That will also solve the main pain point with disallowing tabs in comments. Comments in Zig are Markdown, and Markdown is indentation-sensitive. That's a good and sufficient reason to ban them there as well. Arguably it could also convert them to the same number of spaces as it does tab indents, I don't care either way. Multiline strings though? Tell me, what business does `zig fmt` have in formatting the interior of a string, thus changing the data it represents into something else? None, it has no business doing that. So the fact that it can't turn those tabs into spaces is irrelevant, because it should not do so in the first place. The question is should they be **legal**. My answer is **yes**. One of the important use cases for multiline strings is _embedding formatted textual data_, and out in the wild and wooly real world, _tabs remain widespread_ in data formats. Makefiles require tabs. Properly formatted Go code is lousy with them. Roc, as we just found out, uses tabs. It also uses Zig. The Zig is causing problems for the tabs. I don't think it's Zig's place to be opinionated about how Roc does indentation. This is not a useful place to be purist about `HT` being _haraam_. What problem does it solve? How does it make Zig a better language, one I can confidently recommend to others without too much apologizing on its behalf? How does it _serve the user_ to ban tabs in multiline strings? Please reconsider. Thank you. ## `\r` and `\v` Some hypothetical person counters: what about `CR` and `VT`, those are classically considered whitespace, what about them? To which I reply: `LF` is actually a delimiter in multiline strings, one which defines a closed interval (meaning the newline is included in the data), but a delimiter nonetheless. Allowing a `CR` anywhere other than directly before a newline would be perverse, no living operating system considers a bare `CR` to be whitespace, so in practice this would mean two forms of the delimiter, where the difference is invisible. Some lines ending in `CRLF` and others in just `LF` is a mistake, clearly, how should it be fixed? Answer: no `CR`. Also it would result in a source file which mixes `LF` and `CRLF` delimiters, since Zig doesn't allow the latter anywhere else. And that is truly _haraam_, it would cause numerous problems to no benefit. I say "no benefit" because DOSifying a UNIX text file is a solved problem. It's clear where the `CR`s go, after all. If they're absolutely needed in static memory, this can in fact be easily done at comptime, it's just not a practical issue. With tabs things are different; the tab ban causes actual problems and they're not fixable with a trivial filter. Vertical tab is nearly extinct in the wild, fortunately, and remaining use cases always put it between newlines. So if someone did need it, no problem: ```zig \\ /* This is a weird GNU source file ++ "\x0b\n" ++ // Vertical tab goes here \\ and now it has a vertical tab in it */ ``` Tabs are very far from being extinct in the wild, and I don't expect it will happen in my lifetime, if ever. Rather, they are _ubiquitous_. I end up answering a lot of Zig newcomer questions, and I don't like it when I have to fall back on something like "the core team just, really hates tabs". I don't think ensuring that `rg '\r'` comes back clean is worth it.