#1210 Fixed highlighting of non-unicode chars in Output #1375

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

kittaakos merged 1 commit into main from #1210

Aug 31, 2022

Merged

#1210 Fixed highlighting of non-unicode chars in Output #1375

kittaakos merged 1 commit into main from #1210

Aug 31, 2022

Conversation

kittaakos

Copy link

Contributor

@kittaakos kittaakos commented Aug 31, 2022

Motivation

To avoid showing a warning in the Output if an unicode character is detected as

non-basic ASCII characters,
invisible characters, and (e.g 'NO-BREAK SPACE' (U+00A0))
ambiguous characters (e.g. 'EN DASH' (U+2013) or 'FULLWIDTH RIGHT PARENTHESIS' (U+FF09))

However, I could not develop a sketch that could reproduce the original issues reported in #1210.

unsigned long previousMillis;
void setup() {
 Serial.begin(9600);
}
void loop() {
 if (Serial.available() > 0) {
 while (Serial.available() > 0) {
 Serial.write(Serial.read());
 delay(10);
 }
 } else if (millis() - previousMillis >= 1000) {
 previousMillis = millis();
 Serial.println(u8"\U0001F60E"); // Unicode Character 'SMILING FACE WITH SUNGLASSES' (U+1F60E) https://www.fileformat.info/info/unicode/char/1f60e/index.htm
 Serial.println(u8"\u03BC"); // Unicode Character 'GREEK SMALL LETTER MU' (U+03BC) https://www.fileformat.info/info/unicode/char/03bc/index.htm
 Serial.println(u8"\u00A0"); // Unicode Character 'NO-BREAK SPACE' (U+00A0) https://www.fileformat.info/info/unicode/char/00a0/index.htm
 Serial.println(u8"\u2013"); // Unicode Character 'EN DASH' (U+2013) https://www.fileformat.info/info/unicode/char/2013/index.htm
 Serial.println(u8"\uFF09"); // Unicode Character 'FULLWIDTH RIGHT PARENTHESIS' (U+FF09) https://www.fileformat.info/info/unicode/char/ff09/index.htm
 Serial.write(0xEF);
 Serial.write(0xBC);
 Serial.write(0x89);
 Serial.println();
 }
}

If I modify the Output update code and forcefully print the following invalid characters (– )), I can see the problem:
Screen Shot 2022年08月31日 at 12 52 33

With the proposed changes, the problem goes away:
Screen Shot 2022年08月31日 at 12 54 53

Change description

Other information

Closes #1210

Reviewer checklist

PR addresses a single concern.
The PR has no duplicates (please search among the Pull Requests before creating one)
PR title and description are properly filled.
Docs have been added / updated (for bug fixes / features)


 Fixed highlighting of non-unicode chars in Output

32f3525

Closes #1210
Signed-off-by: Akos Kitta <a.kitta@arduino.cc>

@kittaakos kittaakos added topic: code type: imperfection topic: theia labels

Aug 31, 2022

@kittaakos kittaakos requested review from per1234 and davegarthsimpson

August 31, 2022 10:55

@kittaakos kittaakos mentioned this pull request

Aug 31, 2022

The output is treated as the source code for correction prompts #1210

Closed

3 tasks

davegarthsimpson

davegarthsimpson approved these changes

Aug 31, 2022

View reviewed changes

per1234

per1234 approved these changes

Aug 31, 2022

View reviewed changes

Copy link

Contributor

@per1234 per1234 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified this fixes #1210

Thanks Akos!

@per1234

Copy link

Contributor

per1234 commented Aug 31, 2022

I could not develop a sketch that could reproduce the original issues

A successful compilation of any sketch will produce the original issue if Arduino CLI is using the zh locale. The reason is these problematic characters are present in the sketch memory usage report that is printed at the end of every successful compilation. The English language version looks like this:

Sketch uses 662 bytes (0%) of program storage space. Maximum is 253952 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 8183 bytes for local variables. Maximum is 8192 bytes.

Unfortunately, this doesn't seem possible to achieve via the Arduino IDE 2.x "Language" preference because that sets the locale key of arduino-cli.yaml to zh-cn (which is the code used by the VS Code language pack) instead of zh.

But this sketch can also be used to reproduce it completely independently from locale: