llvm.org GIT mirror llvm / cbdd389
[opt-viewer] Don't Decode HTML bytes for Python 2 Summary: D36624 added some python3 compatibility. But that fix has a problem.. With python2 (which is specified by `#!/usr/bin/env python2.7`), if the env variables do not specify the UTF8, and the source file is UTF8 (contains non-ASCII symbols), then the `.decode('utf-8')` causes the following exception: ``` Reading YAML files... Rendering HTML files... 8 of 41Traceback (most recent call last): File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 277, in <module> print_progress) File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 213, in generate_report should_print_progress) File "/build/llvm/tools/opt-viewer/optpmap.py", line 45, in pmap result = map(_wrapped_func, func_and_args, *args, **kwargs) File "/build/llvm/tools/opt-viewer/optpmap.py", line 25, in _wrapped_func return func(argument) File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 174, in _render_file SourceFileRenderer(source_dir, output_dir, filename).render(remarks) File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 125, in render self.render_source_lines(self.source_stream, line_remarks) File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 79, in render_source_lines </tr>'''.format(**locals()), file=self.stream) UnicodeEncodeError: 'ascii' codec can't encode character u'\xf4' in position 47: ordinal not in range(128) ``` This is similar to https://bugs.llvm.org/show_bug.cgi?id=33548, which was fixed by https://reviews.llvm.org/D37661 Unlike that fix, here, *removing* `.decode('utf-8')` actually fixes it. Since i assume that the original fix is needed, i simply made that fix conditional, since for python2 it actually breaks things. Reviewers: modocache, anemet Reviewed By: anemet Subscribers: fhahn, llvm-commits Differential Revision: https://reviews.llvm.org/D38289 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@315350 91177308-0d34-0410-b5e6-96231b3b80d8 Roman Lebedev 1 year, 10 months ago
1 changed file(s) with 6 addition(s) and 1 deletion(s). Raw diff Collapse all Expand all
99 import os.path
1010 import re
1111 import shutil
12 import sys
1314 from pygments import highlight
1415 from pygments.lexers.c_cpp import CppLexer
6162 html_highlighted = highlight(
6263 file_text,
6364 self.cpp_lexer,
64 self.html_formatter).decode('utf-8')
65 self.html_formatter)
67 # On Python 3, pygments.highlight() returns a bytes object, not a str.
68 if sys.version_info >= (3, 0):
69 html_highlighted = html_highlighted.decode('utf-8')
6671 # Take off the header and footer, these must be
6772 # reapplied line-wise, within the page structure