homepage

This is an experimental patch that limits the frequency of stat() calls in _FileFinder.find_module(). It speeds up finding modules by 2x here, but unfortunately breaks some tests (which expect modules to appear immediately when created).

The patch uses the time module which is not a built-in, so it breaks bootstrapping by directly importing a module, and an extension at that. At best you could switch to a modified _FileFinder after importlib is initially running and able to import extension modules.

> The patch uses the time module which is not a built-in, so it breaks
> bootstrapping by directly importing a module, and an extension at
> that. At best you could switch to a modified _FileFinder after
> importlib is initially running and able to import extension modules.
Yup, as I said it's more of a proof-of-concept.
(I was actually surprised that time is not a built-in module)

On Mon, Feb 20, 2012 at 18:06, Antoine Pitrou <report@bugs.python.org>wrote:
>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> > The patch uses the time module which is not a built-in, so it breaks
> > bootstrapping by directly importing a module, and an extension at
> > that. At best you could switch to a modified _FileFinder after
> > importlib is initially running and able to import extension modules.
>
> Yup, as I said it's more of a proof-of-concept.
> (I was actually surprised that time is not a built-in module)
Yeah, I had to double-check myself.

FWIW, I doubt you'd get many objections if you ended up wanting to make time a builtin module and inject it into the bootstrapping namespace.
While I don't think the delay in noticing filesystem changes is reasonable as the default behaviour, it might be interesting to see how tricky it would be to create a custom importer (for either meta_path or path_hooks) that did this.

> While I don't think the delay in noticing filesystem changes is
> reasonable as the default behaviour, it might be interesting to see how 
> tricky it would be to create a custom importer (for either meta_path or 
> path_hooks) that did this.
Mmmh. Then I think it would be simpler to expose a global option to disable freshness checks. That said, it has to be activated as early as possible to be effective, which doesn't make it very practical as an opt-in setting.

As long as the interpreter knows about about files that *it* wrote, no repeat checks during startup seems utterly reasonable; sneaking in a new or changed file is inherently a race condition.
I think it would also be reasonable for general use, so long as there was also a way to say "for this particular directory, always check".

Well, we already have a delay in detecting new files in importlib as the directory mtime check already adds a delay of the granularity of the mtime value for a directory. So if you manage to mutate a directory, import, write a file, and then import again all within the mtime of a directory, you will miss the new file (which has already triggered sporadic failures in various tests because they were not blowing out importlib's cache). IOW there is already a freshness problem that if we don't have importlib will never get bootstrapped because people will complain about performance too much.

I wonder if the circumstances changed since this is an old issue? the method looks to have a deprecation warning and changed a bit too, I stand to be corrected.

Related : https://bugs.python.org/issue14067