by Brandon Rhodes • Home

Why triple-quotes make PyFlakes hang Emacs

I need to refine the aspersions which I cast against PyFlakes last night in my response to Chris McDonough, answering his bounty against the bug that Emacs would hang in Flyspell mode when he typed triple-quotes. My email, I admit, was not entirely fair to PyFlakes; but you must remember that I was writing late at night, and in great haste, wanting to be the first to respond among however many dozens of Emacs LISP programmers were racing to converge on the solution to Chris's problem.

My mistake was to exaggerate somewhat the verbosity with which PyFlakes reports a syntax error. My email to Chris, in fact, made the following rather extravagant claim:

The problem is that … on a syntax error, PyFlakes prints out an error message, then the entire contents of the module that it cannot import, and finally a line that contains a number of spaces equal to the offset into the file of the syntax error …

Just glancing at the PyFlakes source code is enough to see that this accusation cannot be true:

    try:
        ...
    except (SyntaxError, IndentationError):
        value = sys.exc_info()[1]
        (lineno, offset, line) = value[1][1:]
        ...
        print >> sys.stderr, 'could not compile %r:%d:' \
            \% (filename, lineno)
        print >> sys.stderr, line
        print >> sys.stderr, " " * (offset-2), "^"

Clearly, this simply prints the line that the Python exception cites as having caused the problem, followed by a primitive attempt to position a ^ character at the location of the error. For simple syntax errors, this actually produces output which is identical to that of normal Python:

$ python error1.py
  File "error1.py", line 3
    return x y
             ^
SyntaxError: invalid syntax
$ pyflakes error1.py
could not compile 'error1.py':3:
    return x y
             ^

But the PyFlakes behavior is quite different from that of the standard interpreter if the syntax error happens in a Python statement that has been continued across several lines of source code. Imagine that there are two functions in a file, and that we have started typing a docstring for the first one but have not yet closed the triple-quote:

def square(x):
    """Returns the square of x.
    return x * x

def cube(x):
    """Returns the cube of x."""
    return x * x * x

Here, Python and PyFlakes give quite different reports:

$ python error2.py
  File "error2.py", line 6
    """Returns the cube of x."""
             ^
SyntaxError: invalid syntax
$ ~/.emacs.d/usr/bin/pyflakes error2.py
could not compile 'error2.py':6:
    """Returns the square of x.
    return x * x

def cube(x):
    """Returns the cube of x."""
 ...76 spaces... ^

Do you see what has happened? The unterminated triple-quoted string looks as though it ends several lines later, at what we ultimately intend to be the beginning of the next triple-quoted string. (If the file instead contained no further triple-quoted strings, then the new string would appear to extend all the way to the end of the file.)

This is no problem for the Python interpreter itself, which modestly displays only the final line of the multi-line syntax error. But PyFlakes, not checking for this possibility, prints out the entire triple-quoted string, followed by a ^ character that is indented the entire length of the triple-quoted string. In the example that I was testing last night, this produced a line of nearly four thousand spaces that then ended in the lone little caret character.

So while PyFlakes is certainly more verbose than standard Python, it is not being nearly as profligate as I claimed. It does not insist on printing out your whole module, but limits its output to lines that actually appear involved in the error. Either way, it is the following line — the one that starts with all of the spaces — that is really the problem, since too many spaces send one of the Emacs Flymake regular expressions spiralling into exponential oblivion.

Perhaps Flymake should support a command-line option that omits code snippets entirely, so that Emacs will have only error messages to process. Either way, Chris and I can be more productive now that we can safely integrate a patched version of this excellent tool into our coding sessions.

©2021