C# Language Lexer

Question

Here is a Lexer for a programming language I am working on. Any feedback would be appreciated. I only started learning C# a couple of days ago, so please excuse my newbie code :)

namespace Sen
{
 enum TokenType
 {
 IDENTIFIER,
 NUMBER,
 STRING,
 SEMICOLON,
 PLUS,
 MINUS,
 STAR,
 SLASH,
 }
 class Token
 {
 public TokenType type;
 public string value;
 public Token(TokenType type, string value = "")
 {
 this.type = type;
 this.value = value;
 }
 }
 class Lexer
 {
 public readonly List<Token> tokens;
 private int charIdx;
 private readonly string sourceRaw;
 char CurrentChar
 {
 get { return sourceRaw[charIdx]; }
 }
 public Lexer(string sourceRaw)
 {
 this.sourceRaw = sourceRaw;
 tokens = new List<Token>();
 }
 bool IsEnd
 {
 get { return charIdx >= sourceRaw.Length; }
 }
 char? NextChar()
 {
 try {
 return sourceRaw[charIdx++];
 } catch (IndexOutOfRangeException) {
 return null;
 }
 }
 public void Lex()
 {
 while (!IsEnd) {
 switch (CurrentChar) {
 case ';': AddToken(TokenType.SEMICOLON); break;
 case ' ': break;
 case '\'':
 case '"':
 LexString();
 break;
 case '+': AddToken(TokenType.PLUS); break;
 case '-': AddToken(TokenType.MINUS); break;
 case '*': AddToken(TokenType.STAR); break;
 case '/': AddToken(TokenType.SLASH); break;
 default:
 if (char.IsLetter(CurrentChar)) {
 LexIdentifier();
 continue;
 } else if (char.IsNumber(CurrentChar)) {
 LexNumber();
 continue;
 }
 throw new UnexpectedCharacterException(CurrentChar);
 }
 NextChar();
 }
 }
 void AddToken(TokenType type, string value = "")
 {
 tokens.Add(new Token(type, value));
 }
 void LexIdentifier()
 {
 int startIdx = charIdx;
 int endIdx = startIdx;
 while (!IsEnd && CurrentChar != ' ' && CurrentChar != ';') {
 if (!char.IsLetterOrDigit(CurrentChar) && CurrentChar != '_')
 throw new UnexpectedCharacterException(CurrentChar);
 NextChar();
 endIdx++;
 }
 string value = sourceRaw[startIdx..endIdx];
 AddToken(TokenType.IDENTIFIER, value);
 }
 
 void LexNumber()
 {
 int startIdx = charIdx;
 int endIdx = startIdx;
 while (!IsEnd && CurrentChar != ' ' && CurrentChar != ';' && char.IsNumber(CurrentChar)) {
 NextChar();
 endIdx++;
 }
 string value = sourceRaw[startIdx..endIdx];
 AddToken(TokenType.NUMBER, value);
 }
 void LexString()
 {
 char opening = CurrentChar;
 int startIdx = charIdx + 1;
 int endIdx = startIdx - 1;
 NextChar();
 while (CurrentChar != opening) {
 if (IsEnd)
 throw new ExpectedCharacterException(opening);
 NextChar();
 endIdx++;
 }
 string value = sourceRaw[startIdx..endIdx];
 AddToken(TokenType.STRING, value);
 }
 }
}

Answer 1

Welcome to CR and to C#. First thing first, you should become familiar with C# Naming Conventions. A few that I choose to emphasize in regards to your post:

In Token class, the fields type and value should become properties named Type and Value. In general, fields are private unless they are constant or static. If you wish to expose a field as public, then it should be a property instead. Also, properties and methods should be named with Pascal casing.

Though not required, I personally prefer to decorate all properties, fields, and method with its access modifier, even if it is private. Granted, private is the default but I want to make sure that a beginner has given it thought and explicitly marked it so.

Regarding braces, there are 2 areas for improvement. One, the current thinkng with C# is that the open and close braces occur on their own line. And two, one-liners are frowned upon and should encorporate braces.

Taking that into consideration, this would be a rewrite of one method:

private char? NextChar()
{
 try
 {
 return sourceRaw[charIdx++];
 } 
 catch (IndexOutOfRangeException)
 {
 return null;
 }
}

Except that entire method can use a less expensive if rather than a try-catch block.

private char? NextChar() => (charIdx >= 0 && !IsEnd) 
 ? sourceRaw[charIdx++] 
 : null;

Why both to catch an exception if all you is ignore it? Especially when there is simple code that can easily work around it.

Back to braces, lines such as:

if (IsEnd)
 throw new ExpectedCharacterException(opening);

should be converted to:

if (IsEnd)
{
 throw new ExpectedCharacterException(opening);
}

There are a few properties or methods where you may consider using =>. Example:

private bool IsEnd => charIdx >= sourceRaw.Length;

You seem to use CurrentChar != ' ' && CurrentChar != ';' frequently. Apparently, these are delimiters between tokens and values. The DRY Principle (Don't Repeat Yourself) suggests this could become its own property:

private bool IsDelimiter => CurrentChar == ' ' || CurrentChar == ';'

Elsewhere in code you would replace CurrentChar != ' ' && CurrentChar != ';' with !IsDelimiter. The advantage here, besides readability, is that if you were ever to add a 3rd delimiter in the future, you would only have to change it in one spot.