The Story (Reloaded)

I won't tell you the language "threesome" story again. (if you don't know it, a bit of imagination might do the trick ) (No? OK... clue: me, Perl and C++) (You find it bad? That's alright, go on reading then :P.)

I released PXPerl more than a year ago, and it met a good success. Why? Because embedding Perl is very practical. Quickly developing applications is made very easy when Perl is available. Perl can achieve easily many things you would take a lot of time to program and debug in C++, and without guarantee of a significant speed gain.

I therefore decided to write a new, more powerful and comprehensive version. Also, more object-oriented, and thread-safe. For example, I wanted to be able to modify a scalar within a thread:

without crash (...)

that modification affects the interpreter for all threads.

Also, I wanted to access my C++ functions from within my Perl script. Hey, I wanted a real standard streams redirection too. And the wish-list doesn't stop here... I was full of new ideas :P

That revealed to be quite ambitious (I'm not a computer science student and I'm supposed to have not that time to spend on this...), but I did it. I learned much on Perl and Perl embedding while developing it.

While developing it, I decided to create my own Perl distribution. First, for a practical reason: not to bother users with "install SWIG in the following directory. Open XXX with Notepad. blah blah blah". This Perl distribution is loaded with a bunch of modules, and among them is GraphicsMagick, a powerful image manipulation library. It was compiled for maximum performance, with an average speed gain of 16% over ActivePerl. It was designed to ease the developer task: with syntax highlighted HTML documentation, Explorer integration, and SciTE bundled for visual Perl scripts editing.

So eventually, I decided to change the wrapper's name, to give name PXPerl to the Perl distribution, and PXPerlWrap to the wrapper/embedding solution.

PXPerlWrap comes packaged within PXPerl, and the all-in-one package is available on my site.

In a nutshell

Embed a world widely used scripting language without efforts: enjoy the power of Perl in minutes. Write high quality applications in seconds. Maintain them as easily. Enrich your applications of a scripting ability, extending to the infinity the possibilities for your end-users.

Features:

PXPerlWrap is a bi-directional wrapper. That is:

you can execute Perl code from your MFC application, and manipulate variables etc.

NEW! you can call your application, C/++ functions and classes from within your Perl scripts.

The first direction is achieved by PXPerlWrap. The second direction is achieved by SWIG, a free wrapper to several languages, on which you can find information here.

PXPerWrap is a namespace containing an intuitive set of classes, multithread-safe:

CPerlInterpreter: represents a Perl interpreter, persistent or not. Several different interpreters can be loaded.

parse scripts;

run scripts, several times;

get a variable object to manipulate it or simply retrieve its value(s);

eval a piece of Perl quickly.

CScript: a script object. Each script keeps record of its own properties towards each interpreter.

Easy installation within your existing project through the setup script.

You said persistent interpreter?

The idea of the persistent interpreter was found in perlembed. It consists of parsing once a script (see Behind the Persistent interpreter for this script) which will take care of parsing and running itself other scripts. Each scripts are assigned a different package name. This way, cleaning the package associated with a script will clean the variables used by the script, and hopefully free the memory associated with them.

The major benefit of the persistent interpreter is the ability to clean a script namespace, hence freeing memory for other scripts. I didn't perform any benchmark on it yet, but I think there is no real speed gain compared to a non-persistent interpreter: loading the interpreter is a bit longer (in fact, parsing the persistent script), but parsing a script is likely to be a little shorter. Running a script should take the same.

Therefore, prefer a persistent interpreter when you have a long running application, i.e., a long-running interpreter, so you can progressively clean scripts you don't need anymore.

Snippets

Here are pieces of code used in PXPerlWrap, which are interesting and may be useful to you.

How to redirect standard streams without process spawning

Developing this part of PXPerlWrap took me a lot of time. Why? Because I was looking for a bug in the wrong place. I got stuck for hours without figuring out why the standard streams got, either not at all, or partially redirected, in debug mode. In fact, and this may seem surely obvious to some people here, but not to me at this time, the problem was coming from the Perl DLL which was linked against a different CRT DLL (MSVCR71.DLL) than PXPerlWrap debug DLL (MSVCR71D.DLL). The output and error file descriptors are therefore not the same. And wanting to redirect them is vain. That's why I had to provide a Perl debug DLL.

Credits for most of the code presented below goes directly to Vladimir Schneider. He also provided me a great help to get this redirection working. Thank you!

Behind the Persistent interpreter

In order to maintain the persistent interpreter, we need a script (which I call perlsistent.pl) to parse and run "sub-scripts", and also clean them. Such a script is suggested in perlembed. I modified it a bit for PXPerlWrap. Here it is:

Setting up PXPerlWrap in your project

Setting up PXPerlWrap in your project is made quite simple by the supplied setup script. Here are the steps to set up PXPerlWrap in your projects (also available in PXPerl documentation):

Set your project binaries output directory to a single and unique one, for every configuration, if not already the case. Modify the output filenames as well.

Example:

Configuration

Output filename

Debug

bin/MyApp-d.exe

Release

bin/MyApp.exe

Debug Unicode

bin/MyApp-ud.exe

Release Unicode

bin/MyApp-u.exe

This way, you won't have to have several copies of PXPerlWrap.dll, Perl58.dll, and Perl default modules, i.e., one for each output directory.

Go to PXPerl installation directory.

Launch "Step 1 - Edit PXPerWrapSetup config file.bat".

You have to modify a couple of variable so as to get PXPerlWrap set up in your project; this is explained in the config file.

Launch "Step 2 - Launch PXPerWrapSetup.bat".

Several files will be created for you by the setup script in your project directory (i.e.: pxpw_*.bat, PXPerlWrap directory, DLL files under output directory, and modules directory).

You shouldn't see any error during this process. Otherwise, you probably specified a wrong path or option. Go to step 2 and read carefully. If the trouble persists, report it to me.

Add the files "PXPerlWrap.h" and "PXPerlWrap.cpp" to your project (located in the PXPerlWrap directory in your project directory); and only these files.

Correction: you may also add export.h, for access convenience, see step 8 for purpose of this file.

In your project settings, for each configuration, add a Pre-Build Event and Post-Build Event:

Configuration

Prebuild Event

Postbuild Event

Debug (Unicode or not)

PXPW_prebuild.bat

PXPW_postbuild_debug.bat

Release (Unicode or not)

PXPW_prebuild.bat

PXPW_postbuild.bat

For each configuration, add the additional include directory "$(PXPERL)\lib\CORE".

Now, wherever you want to use PXPerlWrap classes, just add this to your program header file:

#include<spanclass="code-string">"PXPerlWrap/PXPerlWrap.h"</span>

The prototypes of the functions you want to be exported with SWIG (i.e. available to your scripts) will have to appear in "export.h".

That's all

FAQ

Q: Grrrr, source code is not available!

A: I must admit I thought twice before deciding not making available the whole PXPerlWrap source code. And, well, although I encourage open source projects, it's a personal choice. Nevertheless, you can buy the whole source code of PXPerlWrap on my site.

However, I made some code pieces public because it may be useful to the community. See snippets.

Q: Is PXPerlWrap free or not?

A: Basically, yes. PXPerlWrap is free for using in freewares etc. Any commercial usage requires a commercial license, see details on my site. The PXPerl Perl distribution itself, apart from PXPerlWrap and other software bundled which come with their own, basically free, licenses, is available under the same license as Perl, that is the artistic license.

Q: Why is PXPerlWrap shipped with such a heavy package, PXPerl? Are the 16 MB necessary to PXPerlWrap, i.e., will I have to ship 16 MB in my application??

A: PXPerlWrap requires PXPerlWrap is shipped in PXPerl, the Perl distribution, for convenience. In my opinion, an all-in-one package is more practical. The other reason is that PXPerl and PXPerlWrap and tightly linked, the latter requires the first, through the setup script.

And, no, you won't have a 16 Mb overhead when using PXPerlWrap in your application. With a decent compression, and only standard modules embedded, you'll have roughly 2 MB.

Q: So, the end-user doesn't have to have PXPerl or any other Perl binary distribution installed on his machine?

A: That's it.

Supplying the PXPerlWrap*.dll (* = depends on the build you supply to end-user, either Unicode or MBCS), perl58.dll, PXPerl.opt, and the modules directory you specified during setup, is enough. Just keep this file tree and it will be OK.

Q: When I supply relative paths to PXPerlWrap, what happens?

A: PXPerlWrap will make these paths absolute, but not referring to the active directory, but to the PXPerlWrap DLL. I had to do that because, when debugging, Visual Studio sets an unobvious active directory which is the project directory. So be careful when loading script files.

Q: I built the Release [Unicode] version of my application, tested it, and it worked. I then compiled the Debug [Unicode] version, and the release executable was no longer working/frozen!

A: That's because your release executable require the release Perl DLL, while your debug version requires the Perl debug DLL. Both DLLs have the same name. Why? Because, this way, I don't have to supply all the modules compiled, especially for the debug DLL. And why is a Perl debug DLL needed? Because, otherwise your application debug executable and the Perl release DLL would be linked against a different CRT DLL, one debug, the other not, and redirection would not work.

Q: Wow, this DLL is too big!

A: Yeah, sometimes you have to choose between speed and size, I prefer speed. Use UPX packer to reduce significantly the DLL's size (particularly perl58.dll, 2.4 MB).

Q: I would like to have all these DLLs statically linked. Is it possible?

A: No. Perl can't be statically linked safely. No plan to make a static PXPerlWrap.

13 July 2003: v1.1: now using PerlIO::scalar and no real-files to store STD* outputs. Faster.

12 July 2003: Initial release (v1.0). Not released to the public

To come

A more comprehensive description/tutorial of the PXPerlWrap behavior towards UTF8 encoded strings in Unicode builds.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

About the Author

Comments and Discussions

- PXPerl's new name is PXPerlWrap, therefore the upcoming version is PXPerlWrap 2;

- PXPerl is now a Perl distribution, compatible with ActivePerl, and compiled with IC 8. It will include PXPerlWrap;

- I've nearly finished everything in PXPerlWrap (3000 lines of code) REDIRECTION WORKS PERFECTLY (erm, only in release mode actually, but I consider it working :P). Now I'm currently deeply testing and debugging every feature (lots :P).

In my actual project, I want to integrate a perlintetrpreter as a embedded scripting language in my c++ Application. ( like vb in office )

I need interaction between c++ and Perl, that is my Application receives data ( from the outside ) , witch then must be send to the perl-script, if it is running.
And on the other side, the perlscript must be able to send data to the application. ( Send a command to the outside, reaction )
( something like callbacks, sendMessage ?? )

Hardy_Smith wrote:First thank you for your work! And yes, Perl can be a 'love'!

;D thx

Well, I must say I haven't thought about this yet. This would require running the script in a separate thread, and, shame on me, I haven't tried yet, I can't predict if getting/setting variables dynamically during script execution will work. To be tested :p However the code should be multithreaded-safe.
Another problem is that retrieving script output (STDOUT, STDERR) during script execution isn't possible. But i should be in version 2 !!! So, what I suggest you is to wait a bit for version 2

What I can do is to set up a communication over sockets. That works good. And to avoid thread-save Problems I start my Perl-Scripts in a owe Process and the main app is running a ip-server. ( running as a thread should also work! )
I will play aroubd with that a little bit and keep you informed if I make same progress doint it with Perl more direct.

Trying to keep the differences between a stand alone version, a web server version and the embedded version of a perl progam at minimum, I ran into trouble with the @ARGV variable (and I guess there are lots of other variables that show similar problems - what about $0 ?).

I can use other names to feed data into an array before the perl program is executed, however I am not able to do this with an array named @ARGV.

I think this behaviour has fooled me before with @INC. I was able to fix this for the moment using a statement like:
use lib 'mypath/for/the/packages';

Anyone out there, who found s good solution for that class of problem? I have done some modifications to the perlsistent.pl and I was able to fake @ARGV using a real bizar construct. But I am absolutely unhappy with that 'solution'.