[Python-Dev] Usability of the limited API

With Python 3.5 shipping an embeddable copy of the interpreter onWindows, I thought I'd try out a simple embedded interpreter as anexperiment. I wanted to use the limited API, as I'd rather it wereeasy to upgrade the interpreter without recompiling the embedding app.

But the "Very high-level embedding" example in the docs doesn'tcompile with the limited API.

The Py_DecodeLocale/Py_SetProgramName/PyMem_RawFree bit can probablybe replaced by a Py_SetProgramName call specifying a static value,it's not exactly crucial. (Py_DecodeLocale appears to be defined as inthe limited API by the headers, but not exported from python3.dll, bythe way, which implies that something's out of sync).

I dumped out the exported symbols from python3.dll, which is thesimplest way I could think of finding out what is in the limited API(it's hardly user friendly, but never mind). And frustratingly, noneof the very high level PyRun_XXX APIs are available.

At this point, I think I'll probably just give up and use the fullAPI, but it does make me question whether the limited API is actuallyusable as it stands.

I was hoping to be able to suggest as an application bundling optionthat people could write a trivial wrapper script in C to fire up aPython script, and bundle that along with its dependencies and theembeddable Python distribution. Looks like that's doable, but onlyusing the full API, which makes upgrading the bundled Pythoninterpreter a bit messier. Ah, well, no huge loss :-(

But after this experiment, I do wonder - is the limited API really aviable option for embedders?

Search Discussions

Nick CoghlanI personally suspect the requirement to preserve source compatibility with Python 2 has meant that the limited ABI hasn't been exercised very well to date. As far as the high level embedding API goes, I expect it's just an oversight that it's missing from the stable ABI. There are some that *can't* be exposed (the ones that rely on FILE pointers), but the others should be OK. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

On 29 May 2015 at 00:11, Paul Moore wrote:I was hoping to be able to suggest as an application bundling optionthat people could write a trivial wrapper script in C to fire up aPython script, and bundle that along with its dependencies and theembeddable Python distribution. Looks like that's doable, but onlyusing the full API, which makes upgrading the bundled Pythoninterpreter a bit messier. Ah, well, no huge loss :-(

But after this experiment, I do wonder - is the limited API really aviable option for embedders?

I personally suspect the requirement to preserve source compatibilitywith Python 2 has meant that the limited ABI hasn't been exercisedvery well to date.

As far as the high level embedding API goes, I expect it's just anoversight that it's missing from the stable ABI. There are some that*can't* be exposed (the ones that rely on FILE pointers), but theothers should be OK.

Steve DowerZach has a patch to automate putting the right exports in python3.dll, which I'm strongly in favor of, but it was rejected because people may have added APIs that aren't meant to be stable. Right now, you can #include a number of prototypes that aren't actually available because there are two places to update and so one (in this case, the DLL) doesn't get updated. I think the current plan is to remove everything not currently in the DLL from the stable ABI and force people to add them back

Zach has a patch to automate putting the right exports in python3.dll, which I'm strongly in favor of, but it was rejected because people may have added APIs that aren't meant to be stable.

Right now, you can #include a number of prototypes that aren't actually available because there are two places to update and so one (in this case, the DLL) doesn't get updated.

I think the current plan is to remove everything not currently in the DLL from the stable ABI and force people to add them back manually. This way we can enable the generator without committing to a large set of new APIs.

I don't have the issue number handy, but it should be near the top of the recently modified list.

With Python 3.5 shipping an embeddable copy of the interpreter onWindows, I thought I'd try out a simple embedded interpreter as anexperiment. I wanted to use the limited API, as I'd rather it wereeasy to upgrade the interpreter without recompiling the embedding app.

But the "Very high-level embedding" example in the docs doesn'tcompile with the limited API.

The Py_DecodeLocale/Py_SetProgramName/PyMem_RawFree bit can probablybe replaced by a Py_SetProgramName call specifying a static value,it's not exactly crucial. (Py_DecodeLocale appears to be defined as inthe limited API by the headers, but not exported from python3.dll, bythe way, which implies that something's out of sync).

I dumped out the exported symbols from python3.dll, which is thesimplest way I could think of finding out what is in the limited API(it's hardly user friendly, but never mind). And frustratingly, noneof the very high level PyRun_XXX APIs are available.

At this point, I think I'll probably just give up and use the fullAPI, but it does make me question whether the limited API is actuallyusable as it stands.

I was hoping to be able to suggest as an application bundling optionthat people could write a trivial wrapper script in C to fire up aPython script, and bundle that along with its dependencies and theembeddable Python distribution. Looks like that's doable, but onlyusing the full API, which makes upgrading the bundled Pythoninterpreter a bit messier. Ah, well, no huge loss :-(

But after this experiment, I do wonder - is the limited API really aviable option for embedders?

Paul MooreI recall seeing that issue. I'm fine with that - getting the two in sync is obviously worth doing (and clearly in hand). I'm personally not sure whether automating the exposure of symbols is the correct approach, as I'm not sure people typically even consider the stable API when adding functions. Is the default (what you get if somebody just blindly adds a symbol with no thought for the stable API) to expose it or not? If the default is that it's not exposed, then automation seems reasonable,

On 28 May 2015 at 15:28, Steve Dower wrote:I don't have the issue number handy, but it should be near the top of therecently modified list.

I recall seeing that issue. I'm fine with that - getting the two insync is obviously worth doing (and clearly in hand). I'm personallynot sure whether automating the exposure of symbols is the correctapproach, as I'm not sure people typically even consider the stableAPI when adding functions. Is the default (what you get if somebodyjust blindly adds a symbol with no thought for the stable API) toexpose it or not? If the default is that it's not exposed, thenautomation seems reasonable, otherwise I'm not so sure.

The bigger issue for me is that it looks like the stable API doesn'tinclude functions that allow you to just run a script/file.

At a minimum, PyRun_SimpleString should be available. I'd also like tosee a variant of PyRun_SimpleFile that let you pass a filename (eithera .py file, or a zipfile, or a directory name - basically what you canpass to the Python interpreter directly). You can sort of do it via"import runpy; runpy.run_path(filename)", but you get into all sortsof fun with requoting filenames, etc.

With the fact that we're distributing an embeddable interpreter forWindows, I'd like to be able to promote it as a bundling option,easier to use than things like py2exe/cx_Freeze.

Steve DowerNow I'm at my desk, the issue is http://bugs.python.org/issue23903 I believe new symbols are considered stable by default, so perhaps we actually want a test that will generate a C file that imports everything "stable" and will break the buildbots if someone adds something new without explicitly adding it to the list of stable functions? That sounds like the only way to make the extra overhead of two lists work. I guess we could also invert all the logic in the header files so that symbols are

On 28 May 2015 at 15:28, Steve Dower wrote:I don't have the issue number handy, but it should be near the top ofthe recently modified list.

I recall seeing that issue. I'm fine with that - getting the two in sync isobviously worth doing (and clearly in hand). I'm personally not sure whetherautomating the exposure of symbols is the correct approach, as I'm not surepeople typically even consider the stable API when adding functions. Is thedefault (what you get if somebody just blindly adds a symbol with no thought forthe stable API) to expose it or not? If the default is that it's not exposed,then automation seems reasonable, otherwise I'm not so sure.

I believe new symbols are considered stable by default, so perhaps we actually want a test that will generate a C file that imports everything "stable" and will break the buildbots if someone adds something new without explicitly adding it to the list of stable functions? That sounds like the only way to make the extra overhead of two lists work. I guess we could also invert all the logic in the header files so that symbols are unstable by default.

The bigger issue for me is that it looks like the stable API doesn't includefunctions that allow you to just run a script/file.

I think a combination of Py_CompileString and PyEval_EvalCode is what you want here, though I can't think of any good reason not to have a stable helper method (or even a macro?) for this.

At a minimum, PyRun_SimpleString should be available. I'd also like to see avariant of PyRun_SimpleFile that let you pass a filename (either a .py file, ora zipfile, or a directory name - basically what you can pass to the Pythoninterpreter directly). You can sort of do it via "import runpy;runpy.run_path(filename)", but you get into all sorts of fun with requotingfilenames, etc.

With the fact that we're distributing an embeddable interpreter for Windows, I'dlike to be able to promote it as a bundling option, easier to use than thingslike py2exe/cx_Freeze.

Nick Coghlansync is whether sure the thought for exposed, actually want a test that will generate a C file that imports everything "stable" and will break the buildbots if someone adds something new without explicitly adding it to the list of stable functions? The stable CPython ABI is actually tracked on http://upstream-tracker.org/versions/python_stable_api.html Ideally we'd be running those checks automatically as part of our own QA with http://ispras.linuxbase.org/index.php/ABI_compliance_checker,