[Python-3000-checkins] r66209 - in python/branches/py3k: Lib/test/test_imp.py Misc/NEWS Parser/tokenizer.c

[brett.cannon]

Author: brett.cannon
Date: Thu Sep 4 07:04:25 2008
New Revision: 66209

Log:
PyTokenizer_FindEncoding() always failed because it set the tokenizer state
with only a file pointer when it called fp_setreadl() which expected a file
path. Changed fp_setreadl() to use either a file path or file descriptor
(derived from the file pointer) to fix the issue.

Closes issue 3594.
Reviewed by Antoine Pitrou and Benjamin Peterson.


Modified:
python/branches/py3k/Lib/test/test_imp.py
python/branches/py3k/Misc/NEWS
python/branches/py3k/Parser/tokenizer.c

Modified: python/branches/py3k/Lib/test/test_imp.py
==============================================================================
--- python/branches/py3k/Lib/test/test_imp.py (original)
+++ python/branches/py3k/Lib/test/test_imp.py Thu Sep 4 07:04:25 2008
@@ -1,4 +1,5 @@
import imp
+import sys
import unittest
from test import support

@@ -59,6 +60,21 @@
'"""Tokenization help for Python programs.\n')
fp.close()

+ def test_issue3594(self):
+ temp_mod_name = 'test_imp_helper'
+ sys.path.insert(0, '.')
+ try:
+ with open(temp_mod_name + '.py', 'w') as file:
+ file.write("# coding: cp1252\nu = 'test.test_imp'\n")
+ file, filename, info = imp.find_module(temp_mod_name)
+ file.close()
+ self.assertEquals(file.encoding, 'cp1252')
+ finally:
+ del sys.path[0]
+ support.unlink(temp_mod_name + '.py')
+ support.unlink(temp_mod_name + '.pyc')
+ support.unlink(temp_mod_name + '.pyo')
+
def test_reload(self):
import marshal
imp.reload(marshal)

Modified: python/branches/py3k/Misc/NEWS
==============================================================================
--- python/branches/py3k/Misc/NEWS (original)
+++ python/branches/py3k/Misc/NEWS Thu Sep 4 07:04:25 2008
@@ -12,6 +12,10 @@
Core and Builtins
-----------------

+- Issue 3594: Fix Parser/tokenizer.c:fp_setreadl() to open the file being
+ tokenized by either a file path or file pointer for the benefit of
+ PyTokenizer_FindEncoding().
+
- Issue #3696: Error parsing arguments on OpenBSD <= 4.4 and Cygwin. On
these systems, the mbstowcs() function is slightly buggy and must be
replaced with strlen() for the purpose of counting of number of wide

Modified: python/branches/py3k/Parser/tokenizer.c
==============================================================================
--- python/branches/py3k/Parser/tokenizer.c (original)
+++ python/branches/py3k/Parser/tokenizer.c Thu Sep 4 07:04:25 2008
@@ -448,8 +448,12 @@
if (io == NULL)
goto cleanup;

- stream = PyObject_CallMethod(io, "open", "ssis",
- tok->filename, "r", -1, enc);
+ if (tok->filename)
+ stream = PyObject_CallMethod(io, "open", "ssis",
+ tok->filename, "r", -1, enc);
+ else
+ stream = PyObject_CallMethod(io, "open", "isis",
+ fileno(tok->fp), "r", -1, enc);
if (stream == NULL)
goto cleanup;