User:DanielRenfro/Parser Notes

The goal of these notes are to:

1. understand parsing in general
2. thoroughly understand and document the MediaWiki parser

• Lexical analysis
• Parser

• Lex
• Yacc
• Flex lexical analyser
• GNU bison

• UTF-8

• syntax
• human-language
• namespaces
• magic words
• html-like tags
• templates (recursive)
• parser functions

1. Call helper function Parser::internalParse(), which in turns calls
   1. Parser::replaceVariables, which replaces magic variables, templates, and template arguments with the appropriate text.
      1. It calls Parser::preprocessToDom, which preprocesses some wikitext and returns the document tree.
      2. Next it creates a PPFrame_DOM object and calls its expand() method to do the actual template magic.
   2. Sanitizer::removeHTMLtags(), which cleans up HTML, removes dangerous tags and attributes, and removes HTML comments.
   3. Parser::doTableStuff, which handles and renders the wikitext for tables.
   4. Parser::doDoubleUnderscore, which removes valid double-underscore items, like __NOTOC__, and records them in array $Parser->mDoubleUnderscores.
   5. Parser::doHeadings, which parses and renders section headers.
   6. Parser::replaceInternalLinks, which processes internal links ([[ ]]) and stores them in $Parser->mLinkHolders (a LinkHolderArray object),
   7. Parser::doAllQuotes, which replaces single quotes with HTML markup (<i>, <b>, etc).
   8. Parser::replaceExternalLinks, which replaces and renders external links.
   9. Parser::doMagicLinks, which replaces special strings like "ISBN xxx" and "RFC xxx" with magic external links.
   10. Parser::formatHeadings, which:
       • auto numbers headings if that options is enabled,
       • adds an [edit] link to sections for users who have enabled the option and can edit the page,
       • adds a Table of contents on the top for users who have enabled the option, and
       • auto-anchors headings.
2. Next, parse() calls Parser::doBlockLevels, which renders lists from lines starting with ':', '*', '#', etc.
3. Parser::replaceLinkHolders is called, which calls LinkHolderArray::replace on $Parser->mLinkHolders to replace link placeholders with actual links, in the buffer Placeholders created in Skin::makeLinkObj()
4. Next, the text is language converted (when applicable) using the convert method of the appropriate Language object.
5. Parser::replaceTransparentTags is called, which replaces transparent tags with values which are provided by the callback functions in $Parser->mTransparentTagHooks. Transparent tag hooks are like regular XML-style tag hooks, except they operate late in the transformation sequence, on HTML instead of wikitext.
6. Sanitizer::normalizeCharReferences is called, which ensures that any entities and character references are legal for XML and XHTML specifically.
7. If HTML tidy is enabled, MWTidy::tidy is called to do the tidying.
8. Finally the rendered HTML result of the parse process is stored in the ParserOutput object $Parser->mOutput, which is returned to the caller of Parser::parse.

• Markup spec
• Alternative parsers
• Manual:Code
• Manual:ParserOptions.php

• The MediaWiki parser, uncovered