<br><br><div><span class="gmail_quote">On 6/12/07, <b class="gmail_sendername">Giovanni Bajo</b> &lt;<a href="mailto:rasky@develer.com">rasky@develer.com</a>&gt; wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
On 6/12/2007 6:30 PM, Phillip J. Eby wrote:<br><br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;import imp, os, sys<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;from pkgutil import ImpImporter<br>&gt;&gt;<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;suffixes = set(ext for ext,mode,typ in imp.get_suffixes())
<br>&gt;&gt;<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;class CachedImporter(ImpImporter):<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;def __init__(self, path):<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if not os.path.isdir(path):<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;raise ImportError(&quot;Not an existing directory&quot;)
<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;super(CachedImporter, self).__init__(path)<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.refresh()<br>&gt;&gt;<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;def refresh(self):<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.cache = set()<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for fname in
os.listdir(path):<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;base, ext = os.path.splitext(fname)<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if ext in suffixes and &#39;.&#39; not in base:<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.cache.add(base)<br>&gt;&gt;
<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;def find_module(self, fullname, path=None):<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if fullname.split(&quot;.&quot;)[-1] not in self.cache:<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return None&nbsp;&nbsp;# no need to check further<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return super(CachedImporter, self).find_module(fullname,
<br>&gt;&gt; path)<br>&gt;&gt;<br>&gt;&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sys.path_hooks.append(CachedImporter)<br>&gt;<br>&gt; After a bit of reflection, it seems the refresh() method needs to be a<br>&gt; bit different:<br>&gt;<br>&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; def refresh(self):
<br>&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; cache = set()<br>&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; for fname in os.listdir(self.path):<br>&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; base, ext = os.path.splitext(fname)<br>&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if not ext or (ext in suffixes and &#39;.&#39; not in base):
<br>&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; cache.add(base)<br>&gt;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; self.cache = cache<br>&gt;<br>&gt; This version fixes two problems: first, a race condition could occur if<br>&gt; you called refresh() while an import was taking place in another
<br>&gt; thread.&nbsp;&nbsp;This version fixes that by only updating self.cache after the<br>&gt; new cache is completely built.<br>&gt;<br>&gt; Second, the old version didn&#39;t handle packages at all.&nbsp;&nbsp;This version<br>&gt; handles them by treating extension-less filenames as possible package
<br>&gt; directories.&nbsp;&nbsp;I originally thought this should check for a subdirectory<br>&gt; and __init__, but this could get very expensive if a sys.path directory<br>&gt; has a lot of subdirectories (whether or not they&#39;re packages).&nbsp;&nbsp;Having
<br>&gt; false positives in the cache (i.e. names that can&#39;t actually be<br>&gt; imported) could slow things down a bit, but *only* if those names match<br>&gt; something you&#39;re trying to import.&nbsp;&nbsp;Thus, it seems like a reasonable
<br>&gt; trade-off versus needing to scan every subdirectory at startup or even<br>&gt; to check whether all those names *are* subdirectories.<br><br>There is another couple of things I&#39;ll fix as soon as I try it. First is
<br>that I&#39;d call refresh() lazily on the first find_module because I don&#39;t<br>want to listdir() directories on sys.path that will never be accessed.<br><br>The idea of using sys.path_hooks is very clever (I hadn&#39;t thought of
<br>it... because I didn&#39;t know of path_hooks in the first place! It appears<br>to be undocumented and sparsely indexed by google as well), and it will<br>probably help me a lot in my task of fixing this problem in the
2.x serie.</blockquote><div><br><br>PEP 302 documents all of this, but unfortunately was never documented in the official docs.<br><br>I also have some pseudocode of how import (roughly) works at sandbox/trunk/import_in_py/pseudocode.py .
<br><br>-Brett</div></div>