I'm currently building a package that is installed with 'python setup.py develop'. But it seems the .pth files aren't processed when building sys.path.I am using anaconda, with the root environment. Importing true installs (like numpy, pandas) is working well.I actually discovered the issue after extracting a series of utility functions in a new sub-package of my project.

I dig into this as I can't seem to import other modules from the val_tool package I'm building (except if the module I'm trying to import is in the same sub-package than the module I'm importing it from). Even adding paths to pyxll.cfg doesn't help.

the .pth files should get processed when site.py is imported, and that's what added the extra paths the the python path.

You can check what site.py file is imported by setting the environment variable PYXLL_DEBUG_LOG=c:\temp\pyxll-debug-log.txt (for example). That will print out some more debug information before Python has been fully initialized, including where site.py is being imported from.

You can check what site.py is used in your normal python prompt by importing it and printing it, i.e.

If your package in documents\projects\val_tool, or documents\projects\val_tool\utils? You seem to have one in your normal python path and the other in your pyxll config. Maybe that's why you can't import it even though it's in your config file?

I actually have been using the PYXLL_DEBUG_LOG feature but the log doesn't show anything relevant (as far as I can see, I may miss it though). It's a bunch of environment variables, only a few of them being related to python or pyxll. Please let me know the key or variable I should be looking for.

I added the path to utils/ in pyxll.cfg only to be able to print the sys.path variable.

My ideal target is not to have to add anything (or manage it through the pyxll-utils extension loader)

What I have been doing before is to add the vtio/ and est/ paths in pyxll.cfg which are the sub-packages that contains the modules exposing functions/macros to Excel. But when doing that, these modules can only import from the same sub-package, not from the utils/ sub-package. Adding the path to utils/ doesn't help (and would only be a band-aid outside of development).

You may find it in the main log file instead of the pyxll-debug log file (but you still need the environment variable set). Towards the end of the pyxll-debug log file it should have a line about redirecting to the other log once it's read the config.

Great, that's the script that is supposed to set up your sys.path by reading the .pth files.

It should look in <python>/lib/site-packages where <python> is sys.prefix or sys.exec_prefix (see getsitepackages in site.py) and process any .pth files it finds in those folders (see addsitedir).

So, possibly one of two things is going wrong... either sys.prefix and or sys.exec_prefix is not giving the correct path, or the .pth file isn't where site.py is expecting it to be.

Could you try printing out sys.prefix and sys.exec_prefix to check they point to your Anaconda install please? If they don't you can use a pyxll-startup.py file in the same folder as pyxll.xll to modify them before site.py is imported, eg

So indeed sys.prefix and sys.exec_prefix are empty when the interpreter loads from pyxll.I added the pyxll-startup.py file as advised and even though the two variables are now correct, the path is still the same, so I guess it's about the .pth files...

I tried adding prints in site.py but those do not appear in any of pyxll logs and not in a cmd prompt when I launch excel from it. I'll try later printing to a file.

Ok, so here are the results of my investigations:The interpreter launched by PyXll loads the correct site.py but actually bypasses all its execution because it is launched with the no_site flag that site.py tests before execution:

As I couldn't override the sys.flag.no_site value (it is read-only), I circumvent the issue by importing a modified copy of site.py from pyxll-startup.py in which i disabled the no_site check. With this, my project now works as intended:

In pyxll.cfg, i specify the path to the modules that expose excel functions (I'm not totally sure why I still need to do that but I suspect this is due to PyXll binding more than sys.path issues so fair enough) and the associated modules

I also left the paths to the python distribution (as per the anaconda/pyxll installation instructions) but I suspect this is not necessary anymore

The modules exposing excel functions can import other modules of my project (independently of their location relative to the modules directly imported by PyXll)

But this is definitely an ugly hack.

From there, a few questions:

For which purpose does PyXll launch the interpreter with no_site enabled?

Any side effect to be expected from my hack?

Would it be possible to launch the interpreter without no_site? or leave that as a configuration flag to be specified somewhere?

Absent that, do you have any idea for a better solution than what I'm currently doing?

thanks very much for looking into this and your thorough explanation of what's going wrong.

no_site is an interpreter flag that tells Python to import site.py when the interpreter is initialized. When PyXLL is initialized the import of site.py is deferred to allow pyxll-startup to be imported first. This seems like an oversight in a newer version of site.py (previously, and in the version I have been testing against) this check was not there as it's not needed (since it's just supposed to tell the interpreter whether to import site.py or not, not change the behavior of site.py once it is imported). I will look at how PyXLL can be modified to handle this change to site.py.

Perhaps a cleaner work around for now would be to import site.py in your pyxll-startup.py and just call main?

Having said that I don't see any problem with your hack (but I will resolve this no_site issue so it's not necessary).

Now the python path is being set correctly you shouldn't need to add the path explicitly to your config file. When PyXLL looks for the modules listed in the config file it looks on the python path, and it shouldn't matter if that comes from the default path, or added by site.py, or is included in the config. If the modules you want it to import are part of a package you can use the full module name in the config (e.g. mypackage.mymodule).

Thanks again for your help diagnosing this.

Best regards,Tony

ps. yes, sorry about that Mix up with the hosting provider that will hopefully be resolved soon!