Python parser Module tutorial

Question

I am writing an application which reads an input file that currently has its own grammar, which is processed by lex/yacc.

I'm looking to modify this so as to make this input file a Python script instead, and was wondering if someone can point me to a beginner's guide to using the parser module in Python. I'm fairly new to Python itself, but have worked through a fair chunk of the online tutorial.

From what I have researched, I know there are options (such as pyparsing) which can allow me to keep the existing grammar and use Pyparsing as a replacement for lex/yacc. However, I am curious to learn the Python parser module in more detail and explore its feasibility.

Thanks.

Answer 1

You mean the parser module? It's a parser for Python source code only, not a general purpose parser. You can't use it to parse anything else.

• endbegin: Yes, that's what I was referring to. I was thinking about having the input file to my application be a python file. So the grammar would be the python language, and this parser module would parse it. I was just wondering if there was a friendlier tutorial for this.
• Keith Gaughan: You might find the built-in ast module easier to use in that case.
• endbegin: Thanks for your replies. Can you recommend a beginner-level tutorial on using the AST module in Python? I've tried to search the web, but haven't seen anything concrete.

Answer 2

As Jochen said, the parser module is for parsing Python code. I think you're best off checking out Ned Batchelder's list of parsers. PyParsing does things pretty differently from Lex and Yacc, so I'm not sure why you think you could keep your existing grammar and lexer. A better bet might be David Beazley's PLY toolkit. It's solid and has excellent documentation.

• endbegin: I'd like to get away from the lex/yacc way in general. I just saw your recommendation on the ast module and will try and find a friendly tutorial on it. Thanks.

Answer 3

I recommend that you check out https://github.com/erezsh/lark

It's great for newcomers to parsing: It can parse ALL context-free grammars, it automatically builds an AST (with line & column numbers), and it accepts the grammar in EBNF format, which is considered the standard and is very easy to write.