Importing packages with overlapping package hierarchies in Python

I started Python programming 4-5 years ago. However, I usually still use Java to write more complicated projects such as data processing platforms for many reasons. One reason is the that writing complicated projects where you have to separate one project into many related and dependent projects much easier.

Lately, I started writing a Python stub for an RPC project that I’m working on that would enable me and possibly my team to write Machine Learning code in various languages and execute these functions by using RPC from other languages.

However, as a more experienced Java developer some perks that are really obvious and easy to use in java are not as clear and obvious in Python. At least not to someone who is doing these things in Python for the same time. A particular case that I experienced in my project was importing packages and modules from different Python projects with same/similar package structure. This is a common practice for example you want to separate the common libraries, classes and codes for a project into a minimal project that will be imported by the other parts. In my case where I’m writing a RPC platform and Name Server I have the following structure in Java and Python:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

JRNS:

RPCCommon

pack1/

pack2/

common/

module1.py

module2.py

__init__.py

__init__.py

__init__.py

RPCStub

pack1/

pack2/

stub/

module3.py

__init__.py

__init__.py

__init__.py

However, when I’m trying to import module1 in module3.py file it fails run by throwing the following error:

Python

1

2

3

4

Traceback(most recent call last):

File"RPCStub/pack1/pack2/module3.py",linex,in<module>

frompack1.pack2.module1 importmodule1_class

ImportError:No module named pack1.pack2.module_class

The problem here is that the overlap of packages prevents the interpreter to find the right package name and module. In order to solve the problem we need to allow Python’s __path__ variable to be extended or overwritten. This can be done by adding the following code to the __init__.py of packages that may cause a conflict (This will include pack1, pack2 in my example above).