Documenting and detailing a single script based on the comments inside

Question

I am going to write a set of scripts, each independent from the others but with some similarities. The structure will most likely be the same for all the scripts and probably looks like:

# -*- coding: utf-8 -*-
"""
Small description and information
@author: Author
"""
# Imports
import numpy as np
import math
from scipy import signal
...
# Constant definition (always with variable in capital letters)
CONSTANT_1 = 5
CONSTANT_2 = 10
# Main class
class Test():
 def __init__(self, run_id, parameters):
 # Some stuff not too important
 
 def _run(self, parameters):
 # Main program returning a result object. 

For each script, I would like to write documentation and export it in PDF. I need a library/module/parser which reads the scripts, extracts the noted comment, code and puts it back together in the desired output format.

For instance, in the _run() method, there might be several steps detailed in the comments:

def _run(self, parameters):
 # Step 1: we start by doing this
 code to do it
 
 # Step 2: then we do this
 code to do it
 code 
 code # this code does that

Which library/parser could I use to analyze the python script and output a PDF? At first, I was thinking of sphinx, but it is not suited to my need as I would have to design a custom extension. Moreover, sphinx strength lies in the links and hierarchy between multiple scripts of a same or of different modules. In my case, I will only be documenting one script, one file at a time.

Then, my second idea is to use the RST format and RST2PDF to create the PDF. For the parser, I could then design a parser which reads the .py file and extract the commented/decorated lines or set of lines as proposed below, and then write the RST file.

#-description
## Title of something
# doing this here
#-
#-code
some code to extract and put in the doc
some more code
#-

Finally, I would also like to be able to execute some code and catch the result in order to put it in the output PDF file. For instance, I could run a python code to compute the SHA1 hash of the .py file content and include this as a reference in the PDF documentation.

• zxzak: Have considered using jupyter notebook? It can be converted to PDF.
• Mathieu: @zxzak I remember that I couldn't convert it into .pdf, I had something missing. What is required to convert it to .pdf? Can you also save/extract the code into a .py file? Moreover, I feel like Jupiter notebook is not as great as an IDE to develop code. For instance, with spyder, you have the code auto-completion, the variable explorer, the IPython console which are all gathered together and very easy to use.
• Mathieu: @zxzak And now that I think about it, to export into a .py file, a complete function will have to be placed into the same notebook cell, while in my case, to document and detail the function, I have to cut down the _run() method into multiple cells. This method can easily reach 200-300 lines and represents 90% of the code.
• Nerxis: @Mathieu There are several ways how to convert it into .pdf, check e.g. this suggestion. And to convert ntb into .py file is also not a problem. Check jupytext, but you should be able to do this directly with jupyter, e.g. jupyter nbconvert --to script YOUR_NOTEBOOK.ipynb.
• Chronial: Fyi: What you are you trying to do looks like literate programming. Maybe knowing that term will help you on your search. It led me to a similar SO Question

Answer 1

Docstrings instead of comments

In order to make things easier for yourself, you probably want to make use of docstrings rather than comments:

A docstring is a string literal that occurs as the first statement in a module, function, class, or method definition. Such a docstring becomes the __doc__ special attribute of that object.

This way, you can make use of the __doc__ attribute when parsing the scripts when generating documentation.

The three double quoted string placed immediately after the function/module definition that becomes the docstring is just syntactic sugaring. You can edit the __doc__ attribute programmatically as needed.

For instance, you can make use of decorators to make the creation of docstrings nicer in your specific case. For instance, to let you comment the steps inline, but still adding the comments to the docstring (programmed in browser, probably with errors):

def with_steps(func):
 def add_step(n, doc):
 func.__doc__ = func.__doc__ + "\nStep %d: %s" % (n, doc)
 func.add_step = add_step
@with_steps
def _run(self, parameters):
 """Initial description that is turned into the initial docstring"""
 _run.add_step(1, "we start by doing this")
 code to do it
 
 _run.add_step(2, "then we do this")
 code to do it
 code 

Which would create a docstring like this:

Initial description that is turned into the initial docstring
Step 1: we start by doing this
Step 2: then we do this

You get the idea.

Generating PDF from documented scripts

Sphinx

Personally, I'd just try the PDF-builders available for Sphinx, via the bundled LaTeXBuilder or using rinoh if you don't want to depend on LaTeX.

However, you would have to use a docstring format that Sphinx understands, such as reStructuredText or Google Style Docstrings.

AST

An alternative is to use ast to extract the docstrings. This is probably what the Sphinx autodoc extension uses internally to extract the documentation from the source files. There are a few examples out there on how to do this, like this gist or this blog post.

This way you can write a script that parses and outputs any formats you want. For instance, you can output Markdown or reST and convert it to PDF using pandoc.

You could write marked up text directly in the docstrings, which would give you a lot of flexibility. Let's say you wanted to write your documentation using markdown – just write markdown directly in your docstring.

def _run(self, parameters):
 """Example script
 ================
 This script does a, b, c
 1. Does something first
 2. Does something else next
 3. Returns something else
 Usage example:
 
 result = script(parameters)
 foo = [r.foo for r in results]
 """

This string can be extracted using ast and parsed/processed using whatever library you see fit.

• Mathieu: Thank you, I didn't know about the __doc__ attribute nor that you could edit it. It is closer to what I need, however it still lacks flexibility. For instance, what if I want to include a snippet of code in my document?
• Henrik: @Mathieu You can put anything you want in the docstring. IIRC Sphinx supports reST and Google Style docstrings, but if you go down the ast-route you can do whatever, Markdown for instance, or even LaTeX or HTML for that matter. I'll update the answer to give you an example.
• Mathieu: As this is the closest to what I need, I awarded here the bounty, especially since you taught me severals points I did not know. In the meantime, I did implement a tag system which I then retrieve with a text parser (tags for description blocks, tags for code blocks to include in PDF Doc, tags for header, ...). I then generate an RST doc from those tags and make the pdf with RST2PDF. I'm really surprise there are no libraries doing this in a more robust way than my implementation, but it will be enough for my needs. I'll keep in mind the __doc__ edit trick which I might use. Thanks!

Answer 2

Comments are not suitable for documentation, typically they are used to highlight specific aspects which are relevant to developers (not users) only. To achieve your goal, you can use __doc__ strings in various places:

• module-level
• class-level
• function-/method-level

In case your _run method is really long and you feel the doc-string is too far apart from the actual code then this is a strong sign that your function is too long anyway. It should be split into multiple smaller functions to improve clarity, each of which can have its doc-string. For example the Google style guide suggests that if a function exceeds 40 lines of code, it should be broken into smaller pieces.

Then you can use for example Sphinx to parse that documentation and convert if to PDF format.

Here's an example setup (using Google doc style):

# -*- coding: utf-8 -*-
"""
Small description and information.
@author: Author
Attributes:
 CONSTANT_1 (int): Some description.
 CONSTANT_2 (int): Some description.
"""
import numpy as np
import math
from scipy import signal
CONSTANT_1 = 5
CONSTANT_2 = 10
class Test():
 """Main class."""
 def __init__(self, run_id, parameters):
 """Some stuff not too important."""
 pass
 
 def _run(self, parameters):
 """Main program returning a result object.
 Uses `func1` to compute X and then `func2` to convert it to Y.
 Args:
 parameters (dict): Parameters for the computation
 Returns:
 result
 """
 X = self.func1(parameters)
 Y = self.func2(X)
 return Y
 def func1(self, p):
 """Information on this method."""
 pass
 def func2(self, x):
 """Information on this method."""
 pass

Then with Sphinx you can use the sphinx-quickstart command line utility to set up a sample project. In order to create documentation for the script you can use sphinx-apidoc. For that purpose you can create a separate directory scripts, add an empty __init__.py file and place all your scripts inside that directory. After running these steps the directory structure will look like the following (assuming you didn't separate build and source directories during sphinx-quickstart (which is the default)):

$ tree
.
├── _build
├── conf.py
├── index.rst
├── make.bat
├── Makefile
├── scripts
│ └── __init__.py
│  └── example.py
├── _static
└── _templates

For sphinx-apidoc to work, you need to enable the sphinx-autodoc extension. Depending on the doc-style you use, you might also need to enable a corresponding extension. The above example is using Google doc style, which is handled by the Napoleon extension. These extensions can be enabled in conf.py:

extensions = ['sphinx.ext.autodoc', 'sphinx.ext.napoleon']

Then you can run sphinx-apidoc as follows (-e puts every module/script on a separate page, -f overwrites existing doc files, -P documents private members (those starting with _)):

$ sphinx-apidoc -efPo api scripts/
Creating file api/scripts.rst.
Creating file api/scripts.example.rst.
Creating file api/modules.rst.

This command created the necessary instructions for the actual build command. In order for the build too to be able to import and correctly document your scripts, you also need to set the import path accordingly. This can be done by uncommenting the following three lines near the top in conf.py:

import os
import sys
sys.path.insert(0, os.path.abspath('.'))

To make your scripts' docs appear in the documentation you need to link them from within the main index.rst file:

Welcome to ExampleProject's documentation!
==========================================
.. toctree::
 :maxdepth: 2
 :caption: Contents:
 api/modules

Eventually you can run the build command:

$ make latexpdf

Then the resulting documentation can be found at _build/latex/<your-project-name>.pdf.

This is a screenshot of the resulting documentation:

Example APIdoc

Note that there are various themes available to change the look of your documentation. Sphinx also supports plenty of configuration options to customize the build of your documentation.

• Mathieu: So very quickly. I did look into sphinx, and tried it. I don't use at all the interconnectivity between class/functions/modules. Moreover, I agree that usually you want to split large function into multilple small ones. However, in this case it makes a lot of sense of having one big function _run(). Thus, sphinx isn't at all a solution for me as there is no way to catch comment which are inside the function describing certain steps, settings applied without having to write an extension like autodoc or napoleon. Thank you for your post, it's a far better start than most tutorial/doc on sphinx

Answer 3

Doxygen sounds suiable for this. It supports Python documentation strings and can also parse comment that start with ## as described here:

https://www.doxygen.nl/manual/docblocks.html#pythonblocks

To get the output in PDF format you need to install a LaTeX processor, such as MikTex. When you run Doxygen it will create a latex folder that includes a "make" shell script. Run the shell script and the PDF file will be generated,.

To include content that's generated elsewhere, e.g. the SHA1 hashes you mentioned, you could use the @include command within a comment. Note that Doxygen's @include commands will only work if you're using ## comments.

e.g.

## Documentation for a class.
#
# More details.
# @include PyClassSha1Hash.txt
class PyClass:

• Mathieu: It looks to me like Doxygen, just as sphinx, is only retrieving the documentation string for the functions/class; which in this case you can mark either with the classical double """ or with this ## syntax you show above. It can not retrieve doc which is within the function/class.
• Terry Ebdon: It can retrieve documentation from anywhere, as long as it's within an appropriate comment block. It will even process markdown files. It's incredibly versatile.