Qt for example uses the pimpl idiom for achieving ABI compatibility between
releases. The problem is additional heap allocations, additional indirection
and you pay for it, whether needed or not. (For example even derived classes in
the same library pay for it.)
I wondered whether this would still be necessary in D and I think not.
In D, as interface files are automatically generated, it could be possible to
have ones created with let's say a special " private_impl" property or
something. For these classes the object size would have to be stored as hidden
static member in the library. The new operator could then simply read the size
and allocate the needed space. Derived class methods can also use the size to
calculate the offset of the derived class data members.
So you would lose some optimizations, e.g. initializing of base members can't
be inlined and stuff. But this is not possible with pimpl either and you gain
the following:
- You only pay for it if you want it. You can also use the standard .di file
and lose the ABI compatiblity between versions if you so want and derived
classes in the same library do not need to pay any additional overhead either.
- It is completely transparent: If you later on decide you need ABI
compatibility between releases, it's just a matter of a compiler switch and
differend .di files.
- I think it will also be more efficient than pimpl in all regards.
I think this would be a real neat and very important feature, when it comes to
shared libraries. Is there any plan to implement something like that in the
future? Do I miss something?
Best regards,
Robert
--
"Feel free" - 10 GB Mailbox, 100 FreeSMS/Monat ...
Jetzt GMX TopMail testen: http://www.gmx.net/de/go/topmail

I think this would be a real neat and very important feature, when it
comes to shared libraries. Is there any plan to implement something
like that in the future? Do I miss something?

The pimpl pros and cons are the same for C++ and D.

Well, yes they are currently. But as D has pure reference semantics for
classes, it would be easier to handle the case when the size of an object is
not known at compile time. It basically comes down to having operator new read
the size from the library and having derived classes members do some offset
calculation. (Member initialization would have to be done by a function) Is
there something missing?
Is there a flaw in my proposal? Isn't it worth the effort? What are the
problems?
In Java pimpl for ABI compatibility seems not to be neccessary, according to:
http://java.sun.com/docs/books/jls/second_edition/html/binaryComp.doc.html
(It states that you can freely add and remove private fields)
but ok, Java is not a system programming language, they might cheat somehow.
Such a feature seems really cool to me, especially for library developers,
because they have to care less that they are implementing a dynamic library.
And you can easily avoid the costs if you don't need it, no rewrite necessary,
just a recompilation of the program. (Not even the library, if you don't care
about the hidden static size member and the initialization methods)
Maybe I am completely wrong, but if so I want to know
This is not really a feature request, as there are far more important things
to do at the moment. This is more a: "Is it possible?" "Would you like to have
it, if someone else implemented it?"
Best regards,
Robert
P.S.: Another benefit I forgot to mention: For such a not exposing .di file, I
think you can even remove all non public imports (if there are no inline
methods), as all you need to know is the size of the object.
I know that compilation time isn't that much of an issue as with C++, but
during development you could use such non exposing .di files, which would
change much less often. Thus recompilation of object files due to changes in
the di files could be decreased, shortening compile time for large projects.
In addition the compiler does not need to see, nor evaluate non public imports
in those di files.
What do you think?

I think this would be a real neat and very important feature, when it
comes to shared libraries. Is there any plan to implement something
like that in the future? Do I miss something?

The pimpl pros and cons are the same for C++ and D.

Well, yes they are currently. But as D has pure reference semantics for
classes, it would be easier to handle the case when the size of an object is
not known at compile time. It basically comes down to having operator new read
the size from the library and having derived classes members do some offset
calculation. (Member initialization would have to be done by a function) Is
there something missing?
Is there a flaw in my proposal? Isn't it worth the effort? What are the
problems?
In Java pimpl for ABI compatibility seems not to be neccessary, according to:
http://java.sun.com/docs/books/jls/second_edition/html/binaryComp.doc.html
(It states that you can freely add and remove private fields)
but ok, Java is not a system programming language, they might cheat somehow.

Java can recompile things on the fly.

Such a feature seems really cool to me, especially for library developers,
because they have to care less that they are implementing a dynamic library.
And you can easily avoid the costs if you don't need it, no rewrite necessary,
just a recompilation of the program. (Not even the library, if you don't care
about the hidden static size member and the initialization methods)
Maybe I am completely wrong, but if so I want to know
This is not really a feature request, as there are far more important things
to do at the moment. This is more a: "Is it possible?" "Would you like to have
it, if someone else implemented it?"

What you're asking for is dynamic typing. You can achieve something equivalent
using opDispatch and std.variant.

I think this would be a real neat and very important feature, when it
comes to shared libraries. Is there any plan to implement something
like that in the future? Do I miss something?

The pimpl pros and cons are the same for C++ and D.

Well, yes they are currently. But as D has pure reference semantics for
classes, it would be easier to handle the case when the size of an object is
not known at compile time. It basically comes down to having operator new read
the size from the library and having derived classes members do some offset
calculation. (Member initialization would have to be done by a function) Is
there something missing?
Is there a flaw in my proposal? Isn't it worth the effort? What are the
problems?
In Java pimpl for ABI compatibility seems not to be neccessary, according to:
http://java.sun.com/docs/books/jls/second_edition/html/binaryComp.doc.html
(It states that you can freely add and remove private fields)
but ok, Java is not a system programming language, they might cheat somehow.

Java can recompile things on the fly.

Such a feature seems really cool to me, especially for library developers,
because they have to care less that they are implementing a dynamic library.
And you can easily avoid the costs if you don't need it, no rewrite necessary,
just a recompilation of the program. (Not even the library, if you don't care
about the hidden static size member and the initialization methods)
Maybe I am completely wrong, but if so I want to know
This is not really a feature request, as there are far more important things
to do at the moment. This is more a: "Is it possible?" "Would you like to have
it, if someone else implemented it?"

What you're asking for is dynamic typing. You can achieve something equivalent
using opDispatch and std.variant.

What you're asking for is dynamic typing. You can achieve something
equivalent using opDispatch and std.variant.

I am not sure I understood you correctly. It is dynamic typing more or less,
just as polymorphic types like classes implement dynamic types, because you
can change the implementation at runtime, by assigning an object of another
derived class.
What I am asking for is hiding the implemenation of class completely, not only
to the programmer but also to the compiler.

I have been asking that for some time now, i am afraid you won't get much
of an audience.
You can get rid of both additional allocation and indirection but it is
not pretty. We could definitely use some help/sugar on this.
http://www.artima.com/cppsource/backyard3.html
On Thu, 19 Jan 2012 22:48:39 +0200, Roberto Caravani <JFanatiker gmx.at>
wrote:

Qt for example uses the pimpl idiom for achieving ABI compatibility
between releases. The problem is additional heap allocations, additional
indirection and you pay for it, whether needed or not. (For example even
derived classes in the same library pay for it.)
I wondered whether this would still be necessary in D and I think not.
In D, as interface files are automatically generated, it could be
possible to have ones created with let's say a special " private_impl"
property or something. For these classes the object size would have to
be stored as hidden static member in the library. The new operator could
then simply read the size and allocate the needed space. Derived class
methods can also use the size to calculate the offset of the derived
class data members.
So you would lose some optimizations, e.g. initializing of base members
can't be inlined and stuff. But this is not possible with pimpl either
and you gain the following:
- You only pay for it if you want it. You can also use the standard .di
file and lose the ABI compatiblity between versions if you so want and
derived classes in the same library do not need to pay any additional
overhead either.
- It is completely transparent: If you later on decide you need ABI
compatibility between releases, it's just a matter of a compiler switch
and differend .di files.
- I think it will also be more efficient than pimpl in all regards.
I think this would be a real neat and very important feature, when it
comes to shared libraries. Is there any plan to implement something like
that in the future? Do I miss something?
Best regards,
Robert

I have been asking that for some time now, i am afraid you won't get
much of an audience.
You can get rid of both additional allocation and indirection but it is
not pretty. We could definitely use some help/sugar on this.
http://www.artima.com/cppsource/backyard3.html

I have been asking that for some time now, i am afraid you won't get
much of an audience.
You can get rid of both additional allocation and indirection but it is
not pretty. We could definitely use some help/sugar on this.
http://www.artima.com/cppsource/backyard3.html

I have been asking that for some time now, i am afraid you won't get
much of an audience.
You can get rid of both additional allocation and indirection but it
is not pretty. We could definitely use some help/sugar on this.
http://www.artima.com/cppsource/backyard3.html

I have been asking that for some time now, i am afraid you won't get
much of an audience.
You can get rid of both additional allocation and indirection but it
is not pretty. We could definitely use some help/sugar on this.
http://www.artima.com/cppsource/backyard3.html

I have been asking that for some time now, i am afraid you won't get
much of an audience.
You can get rid of both additional allocation and indirection but it
is not pretty. We could definitely use some help/sugar on this.
http://www.artima.com/cppsource/backyard3.html

This will even work with plain new/this, as the allocation is done
in the constructor. The difficulty is that you can't build inheritable
pimpls.

Ah, good to know. One solution would be to just add some dummy space to
the class declaration (private void[256] dummy=void) so that the
implementation can use it for private members. (union{void[256] dummy;
struct{/*private_members*/}}) Of course, this is potentially wasteful.

Ah, good to know. One solution would be to just add some dummy space to
the class declaration (private void[256] dummy=void) so that the
implementation can use it for private members. (union{void[256] dummy;
struct{/*private_members*/}}) Of course, this is potentially wasteful.

This is what I did for the thread pimpl, but it requires you to manually
keep the size in sync with the implementation.

I have been asking that for some time now, i am afraid you won't
get
much of an audience.
You can get rid of both additional allocation and indirection but
it
is not pretty. We could definitely use some help/sugar on this.
http://www.artima.com/cppsource/backyard3.html

There is another issue Walter forgot to mention in the article.
I think there might be a way but looks like we also loose the
"destructor".
Which means we are all the way back to the
http://en.wikipedia.org/wiki/Opaque_pointer.
Walter, is there a way to get around destructor limitation?

This will even work with plain new/this, as the allocation is done
in the constructor. The difficulty is that you can't build inheritable
pimpls.

Hmm, I see some problems with this approach: It is not only impossible to
derive from this class without using the di file with the full implementation
(which might be acceptable)but you can't even use derived classes by an A
reference, because the compiler would not use the vtbl because it assumes
there are no derived classes. Which is not acceptable, in my opinion.

Ah, good to know. One solution would be to just add some dummy space to
the class declaration (private void[256] dummy=void) so that the
implementation can use it for private members. (union{void[256] dummy;
struct{/*private_members*/}}) Of course, this is potentially wasteful.

I also thought of such variants, but as you already noted, they are either
wasteful, or limited, or both.
The simplest and most efficient variant I could think of, was the one I
already explained, but I am illustrating it here with some code, to make it
more clear:
// base.d
import std.container;
class Base {
this() {
// ...
}
SList!int overrideMe() {
//....
}
private_impl:
int foo;
float bar;
}
//////
Generate two .di files, with --private-impl compiler switch:
dmd -H -c --private-impl base.d
The --private-impl switch would cause the following:
1. It creates two ".di" files, see below.
2. It creates a hidden static "size" field in the object file:
If you don't pass the --private-impl switch, just one standard .di file is
generated and the " private_impl:" is interpreted as plain "private:".
// base.d generated code:
import std.container;
class Base {
this() {
// ...
}
SList!int overrideMe() {
//....
}
private_impl:
int foo;
float bar;
static size_t __size_of_object=actual_base_object_size;
}
// base.d -- This is the public interface:
import std.container;
//Non public imports are still necessary, because the compiler needs to check
//that client code uses the same SList implementation.
// but imports only needed by method bodies, should be omitted, but how to
tell this the compiler?
class Base {
this(); // No inline, we are hiding the implementation!
SList!int overrideMe();
private_impl: // Omit private data, just a hint to the compiler that there
are private hidden fields (and that there exists a hidden size field)
}
// base_private.di -- the second .di file, just the normally generated one.
If I want to create a derived class and I want to be independent of the base
class implementation, I would simple import the public base.di file:
// derived.d
import base;
class D : Base {
SList!int overrideMe() {
// override it with something
// access my_field:
my_field=2;
// compiler would generate:
int* p=cast(int*)
((cast(void*)this)+Base.__size_of_object+offset_of_my_field);
*p=2;
// instead of:
int* p=cast(int*)
((cast(void*)this)+offset_of_my_field_including_statically_known_base_size);
*p=2;
// A class deriving from D, would also have to add Ds size and so on,
but as these offsets don't change during the life time of a program, the
calculated offsets could easily be cached.
}
private_impl:
int my_field;
}
If I don't care about implementation hiding, I would simple import the
base_private.di file, and I would have statically calculated offsets. Derived
classes in the same library would usually do that.
// main.d
import derived;
import std.container;
void main() {
D d=new D;
// I am using the public .di file, thus new has to look up the objects size
at runtime in the library and allocate the needed space, then it triggers the
now non inlined initialization sequence:
// - initialize all base members
// - initialize all derived members
// - call base class constructor
// - call derived class constructor
// -> can be optimized if derived class uses the non public interface.
// use d.
// d gets destroyed, simply call the destructors, and free the used space.
}
If you don't rename the private implementations, but simply use another module
lookup path, you can switch implementations by no code change at all, simply
comple main.d with the private module search path and you get the old
behaviour.
--> client code can choose, without a code change in the library, if you don't
mind the static size fields, not even a recompile of the library is necessary.
I hope it is now more clear what my basic approach is. I think the benefits
are overwhelming and it seems to be easy, maybe too easy. So I am pretty sure
it is not that simple, but what is it I am missing?
Best regards,
Robert

struct S {
static S* make(); // constructor
static void purge(S*); // destructor - you have to provide this as well
}
Limitation is that just like constructor, we also lose the destructor.
Which means we can't use "delete", and the tools designed for it.
As a result we don't have something new. It is exactly like C impl. hiding.
struct S;
S* make();
void purge(S*);

struct S {
static S* make(); // constructor
static void purge(S*); // destructor - you have to provide
this as well
}
Limitation is that just like constructor, we also lose the
destructor.
Which means we can't use "delete", and the tools designed for
it.
As a result we don't have something new. It is exactly like C
impl. hiding.
struct S;
S* make();
void purge(S*);

Well it is not bad, but how do you support derived classes with this scheme?
Derived classes would have to derive from the implementation, so it would have
to be public and compiled code using the derived class would break on changes
of the private fields. Am I misinterpreting something?
What's the purpose of the pimpl template?
What do you think about my proposal? Is it sane and feasible at all?
Best regards,
Robert

Well it is not bad, but how do you support derived classes with
this scheme? Derived classes would have to derive from the
implementation, so it would have to be public and compiled code
using the derived class would break on changes of the private
fields. Am I misinterpreting something?

Whole purpose of this is hiding implementation from user with
"zero" cost.
Derived classes has no places here. Using anything about derived
classes means "zero" cost is not your first concern. So just use
pure classes / interfaces, they are much cleaner and to the point.

What's the purpose of the pimpl template?

template<typename T>
struct pimpl : noncopyable
{
virtual ~pimpl() {}
};
"virtual ~pimpl() {}"
This is to get rid of leaking, you know we need to provide a
"virtual destructor" for an interface otherwise a base class has
no way of knowing it is a "base" class and when you delete a
pointer you would free only the memory base class allocated. But
i guess you are asking why it should be like this, another way is:
#define PIMPL(x) \
private: \
x(const x&); \
const x& operator=(const x&); \
protected: \
x() {} \
public: \
virtual ~x() {}
now we can just use:
struct lib
{
PIMPL(lib)
...
...
};
I don't understand why this didn't get much attention either, i
am now using it in my framework and it rocks!

What do you think about my proposal? Is it sane and feasible at
all?

I am having trouble understanding the need for supporting class
hierarchies.
Deriving from a private implementation feels quite wrong. A pimpl
IMO should be a black box.

Whole purpose of this is hiding implementation from user with
"zero" cost.
Derived classes has no places here. Using anything about derived
classes means "zero" cost is not your first concern. So just use
pure classes / interfaces, they are much cleaner and to the point.

What's the purpose of the pimpl template?

template<typename T>
struct pimpl : noncopyable
{
virtual ~pimpl() {}
};
"virtual ~pimpl() {}"
This is to get rid of leaking, you know we need to provide a
"virtual destructor" for an interface otherwise a base class has
no way of knowing it is a "base" class and when you delete a
pointer you would free only the memory base class allocated. But
i guess you are asking why it should be like this, another way is:

Yeah, I got that. What I don't get is why it needs to be a template?
Wouldn't a simple base class with a destructor suffice?

I don't understand why this didn't get much attention either, i
am now using it in my framework and it rocks!

What do you think about my proposal? Is it sane and feasible at
all?

I am having trouble understanding the need for supporting class
hierarchies.
Deriving from a private implementation feels quite wrong. A pimpl
IMO should be a black box.

Well my initial example was Qt. Take for example the QWidget class, you
usually derive from it in order to implement custom widgets and with the
PIMPL idiom it is ensured that your derived classes will not break upon
addition of private fields in QWidget. It is a black box, a derived
class does not have access to private fields.
The whole purpose of my proposal is, to really hide private fields so
that not just the API is stable, but also the ABI.
Best regards,
Robert

Yeah, I got that. What I don't get is why it needs to be a
template?
Wouldn't a simple base class with a destructor suffice?

Indeed it would suffice. Nothing really special, i didn't want
pimpl to be a "one true base class" that you could do things like:
vector<pimpl*> v;

Well my initial example was Qt. Take for example the QWidget
class, you
usually derive from it in order to implement custom widgets and
with the
PIMPL idiom it is ensured that your derived classes will not
break upon
addition of private fields in QWidget. It is a black box, a
derived
class does not have access to private fields.
The whole purpose of my proposal is, to really hide private
fields so
that not just the API is stable, but also the ABI.

I understand. My use case was that i needed an alternative to C
way of doing it.
I was thinking that C++/D got namespaces, modules, class/struct
methods. It should be possible to improve C pimpl.
So instead of:
libFun(handle, ...);
You would do:
handle.fun(...);
Now you have a cleaner code, cleaner global namespace with zero
cost.