token_utils.py
- A collection of token-related functions¶
A couple of functions for dealing with tokens generated by the tokenize module.
-
pyminifier.token_utils.
listified_tokenizer
(source)[source]¶ Tokenizes source and returns the tokens as a list of lists.
-
pyminifier.token_utils.
untokenize
(tokens)[source]¶ Converts the output of tokenize.generate_tokens back into a human-readable string (that doesn’t contain oddly-placed whitespace everywhere).
Note: Unlike tokenize.untokenize(), this function requires the 3rd and 4th items in each token tuple (though we can use lists or tuples).