• Mathieu Serandour's avatar
    py/lexer: Change token position for new lines. · c85db052
    Mathieu Serandour authored
    
    
    Set the position of new line tokens as the end of the preceding line
    instead of the beginning of the next line.  This is done by first moving
    the pointer to the end of the current line to skip any whitespace, record
    the position for the token, then finaly skip any other line and whitespace.
    
    The previous behavior was to skip every new line and whitespace, including
    the indent of the next line, before recording the token position.
    
    (Note that both lex->emit_dent and lex->nested_bracket_level equal 0 if
    had_physical_newline == true, which allows simplifying the if-logic for
    MP_TOKEN_NEWLINE.)
    
    And update the cmd_parsetree.py test expected output, because the position
    of the new-line token has changed.
    
    Fixes issue #12792.
    Signed-off-by: default avatarMathieu Serandour <mathieu.serandour@numworks.fr>
    c85db052
lexer.c 33.7 KB