Comments

I get the sense that many of you just don't want ANY answers that aren't "Yes, you're right, we should have implemented support for WinRT's object model and runtime enviroment in an ISO C++ solution and never seriously considered the benefits of a small, platform-specific (WinRT) language extension for simplying the expression of developer intent when programming shared, ref-counted COM objects in a new hetrogeneous programming model for Metro style application development on Windows 8".

No, Charles. We want either "Yes, you are right" or "No, you are wrong, here is why". What we get from Herb is "No, you are wrong, but I won't show you why, this is all terribly difficult, you just don't understand, you don't have a complete technical spec so I will just point out to various things which you haven't yet mentioned in your posts and this will supposedly show that you, well, don't have a complete technical spec and thus I win". This is pathetic.

C++/CX is real, not imaginary or speculative like the reverse of #import with wrapper classes solution.

Right, this is your entire argument. Since we seem to be at our final words, here is the final word from me:

What you did with C++/CX puts C++ developers into a position where they can not easily talk to WinRT from their native language. They have to use another language. Due to its viral nature and bad language integration, C++/CX is actually a worse choice for a C++ developer who wants to talk to WinRT than .NET languages or Javascript. The latter languages at least allow you to see the borders clearly.

Me and my team are going to stay away from C++/CX. This means staying away from WinRT for some time, too.

Do we agree that the route of "extending TLB #import" leads to a full ATL + MIDL like approach? ----- "No." ----- Then you're clearly mistaken. WinRT is COM (and then some), and in general you can't author COM types without all the additional information about the types you can specify in an IDL file -- whether those information-carrying extensions go into a separate IDL file (the MIDL approach) or into the main .cpp file (the C++/CX approach, though it could be done with different syntax, but it must necessarily be just as rich in order to to the job).

This is just word games. Yes, COM types are different from C++ types and there is more than one way to define the way C++ concepts translate to COM. Yes, we might have to convey some information to whoever is going to do the translation. No, you don't have to use MIDL. That's why I answered "No" above. If you think I should have answered "Yes", because using C++ is similar to using MIDL as long as both are used to define interfaces for COM, good.

Also, this doesn't avoid the need to generate wrappers, you will be generating wrappers anyway such as to change the std::wstring type to HSTRING and change all the function signatures to HRESULT returns and generate code to catch exceptions and translate them to HRESULTs and a dozen more things ...

Yes, it is necessary to generate wrappers. Our entire proposal is that you generate wrapper C++ code. That wrapper C++ code should wrap WinRT strings. What is your point here?

": public IInspectable" is not really C++ inheritance if you want it to have WinRT semantics.

We want interfaces to inherit from each other in the C++ way in the C++ code. We want them to do what's required for WinRT in wrapper code you generate. What is so difficult? Why do we have to go through this over and over?

There's not enough information to decide whether the intent is to define I1 as an interface or a class, since a class could be named I1 and could have all pure virtual methods.

Agree. This is something to talk about. Nothing too complex, but there are many possible solutions and we have to settle on one. Anything else?

As for the rest of I1, note that I1::G that returns a std::wstring will be slow if you provide only that much information.

See my reply to Jim Springfield which explains how you can avoid conversions between HSTRING and std::wstring.

We don't know whether each of X and Y is intended to be a class or an interface.

Yes, you have talked about this already. Again, this is something to talk about. There are many ways to go, we just have to settle on something.

WinRT requires X to have a default interface, but there's nothing in the code that says whether Y is to be the default interface for X. We could assume it must be, but then when you have "class X : public Y, public Z" how do you specify which interface should be the default interface?

Yes, this is something to talk about. Many ways to go again.

This is just scratching the surface. It is not by any means a complete list.

Of course. Nothing impossible or even really difficult though. A fair work, the kind you did for C++/CX, the kind we do day in and day out on our jobs. Nothing trivial, but nothing really difficult either. You just wake up, brush your teeth, drive to the office, get into the chair and do it.

I think it's clear that we aren't going to be successful designing this together in a comment thread, and if you refuse to accept that it isn't that simple, from the people who tried and understand the requirements and constraints deeply, then the only alternative is for you to invest the time to make the technical effort yourself -- not just sketch things that look plausible, but build the prototype (including correct and complete binding to WinRT features and high-quality VS product tooling integration) and demonstrate a proof of concept.

Oh, no. Please don't give me this "you just don't understand how difficult this really is, I have been doing this for years so I know much more about this than you do". C++/CX and WinRT aren't rocket science. Sorry, they just aren't. People have been mapping concepts between C++ and other technologies for years, it's a fair work, like I said, but nothing terribly difficult or grandiose like you pretend it is. Take a look at C# folks. They just took WinRT and mapped it into their language seamlessly with little fuss. Nobody has problems with what they did. Nobody. Why? Because WinRT is a better fit for .NET than C++? That's highly debatable. If you were among the C# guys you'd jump at that notion, you'd argue to death that WinRT is enormously closer to C++, since C++ is native and WinRT is native as well. No, the real reason the C# folks just took WinRT and mapped it seamlessly with little fuss is because they approached that as normal developers should, with little noise and no idiotic grand stands.

Herb, if you want to say that this all is really terribly difficult, please show it. Right now, you haven't shown anything particularly difficult. The total of your entire reply is this - how to specify whether a C++ class should map into a interface or a class or both in the WinRT sense, and how to specify the default interface for a class. That's it, everything else was just repetitions and irrelevancies. The question you bring up is not at all difficult. That said, it is very clear that you aren't really interested in discussing it. Your position is that since I don't have a complete technical specification for what is being proposed, this can't be done. This is pathetic.

The point is: how this mapping is implemented? How is the quality of the resulting code?

I agree. That was the original reason for the discussion.

There must be a way to tell the compiler that "Screen" is a WinRT class.

I agree. As I said earlier: "There are many different ways to direct the compiler to do this (pragmas, __declspec, predefined bases, separate command line switches, etc), we can talk about pros and cons of each."

If "ref" strictly before the declaration of a class was all C++/CX was adding, that might have been fine. But, no, we have hats, ref new, generics and everything else. I completely agree we have to talk about what exact code should be written in each case. __declspec is one way for the above, #pragma export(winrt) .. #pragma export would be another. That we have to talk specifically about each case does not mean it can't be done or is even terribly difficult to do. If you are skeptical about that and are willing to take Herb's word that this is terribly difficult, almost impossible, fine, but please realize that this turns it into a talk about faith-like concepts, not about technology. Yes, it is a pity I don't have a complete technical specification ready, I wish I would have one.

I think we can trust people in VC++ Team if they say they tried first without extending the C++ language, but it was not a good experience for the "end user" programmer, and they figured out C++/CX was a more productive tool for us.

Why does this have to be about faith? If there are technical arguments as to why not do wrappers, why not present them? Please forgive me for being blunt, but it very much looks like the team made a mistake in not going far enough with C++ and Herb is trying to justify that mistake by any means possible. So many posts from Herb with nothing resembling a counterpoint besides "it's so difficult, you don't even understand", this has to mean something.

However, it would be interesting if you and other guys could start an effort to try to offer a "pure C++ no-extension" approach to solve this problem of WinRT programming from C++, and let us know here. .. you have to build something concrete to convince me, ...

Fair enough.

As said, the approach suggested above requires extending an existing compiler. This isn't easy. Right now, the effort does not seem to be worth it, there is no hurry to use WinRT and if you absolutely have to use it, there are .NET languages and Javascript.

Assuming an almost 100% ISO C++ simple and elegant solution for creation and consumption of WinRT objects can be designed (will definitely love that), could it be implemented by a third-party entity (e.g. not Microsoft) or are there some technical limitations?

The sad thing and the only reason we are having this long discussion on this thread is that the ISO C++ solution as proposed above requires cooperation from the compiler and this is something that is much easier for Microsoft to do than for any third-party. The Microsoft compiler already parses C++ code and extracts WinRT concepts from that code, the only two things Microsoft has to do is make this work for regular C++ code instead of for their extended syntax, and to generate WinRT concepts as both metadata and C++ source code. We can achieve the same using, say, LLVM, but it won't be easy. If it was easy, or if no cooperation from the compiler was required, we wouldn't have had this thread, we'd just do it and use the result.

What would you say if you were proposing virtual functions for C, and someone asked, "Why do you recommend getting two notions of functions? Functions are functions, darn it, everyone knows that, virtual functions are just functions and I'm sure you can express them in ISO C without extensions, we have function pointers and everything you need."

My answer to that would be:

I am all for making better abstractions, but we have to do this in an organized way. Our language is complex and so if everyone just adds whatever he feels like adding, all benefits from these additions will be erased by the chaos that will arise. Additions have to be vetted so that they extend the language in natural, intuitive ways, so that they don't fight each other, so that there is not too many of them, etc. If you want to add virtual functions, go ahead and suggest this to the standards committee. If they take it on board, and I do think virtual functions are genuinely useful, so I don't see why they wouldn't, great. If they don't, so be it.

What you are doing with C++/CX is something else. You are putting the cart before the horse, like others said, with properties, events and numerous other things.

I think I have answered these. a) Yes b) n/a c) Because we didn't see this approach as the best solution for our users. We've tried to cover many of the issues that pushed us to C++/CX and I can understand if you disagree with some of them.

Yes, this is settled. We are yet to determine what exact issues of wrapper classes pushed you to C++/CX, but we are talking about this right now so hopefully it will become clear soon.

Regarding modules, we do NOT go through WinRT machinery when accessing a locally defined (within a binary) ref class. So, we can inline code and take advantage of other compiler optimizations.

That's good. This suggests to me that when you see a 'ref class', you already split it into a regular C++ class (whose methods you can then inline) and a WinRT class (whose methods are stubs that call methods of the regular class). Thus, you are doing much of what we are arguing for already, you only have to go a couple more steps and (1) generate the wrapper WinRT class in the source form, and (2) get rid of 'ref', altering the code that parses code exported for WinRT to support regular C++ definitions for several things you now do through C++/CX-only features.

I still agree with pretty much everything I said, including the need for further conformance to the new C++ standard now that it has stabilized and was published last month.

Sorry, no.

You equate C++/CX with C++/CLI point blank ignoring that while they look similarly, they wrap different underlying technologies, and what exactly they wrap matters. Suppose someone comes up with an idea of using hat pointers and ref new as an alternative way to work with regular C++ classes. Would you support the idea of such a language extension? Of course, not. You see, what exact implementation details hide behind the proposed syntax and rules matters. If the standards committee or the larger C++ community agrees on some aspect of C++/CLI, it does not at all follow that they would agree on the same aspect of C++/CX. Saying that you did it all right with C++/CLI and so it is all automatically right with C++/CX as well because it uses the same syntax and certain concepts are similar, too, is wrong. You simply can't say this, this isn't true.

Then again, did you really do everything right with C++/CLI? Take point 5, where you say a language extension should be something "not proprietary but rather the opposite: thing we already know is certain or likely to come" in the next version of the C++ standard and which you "hope all compiler vendors will provide too"? Is C++/CLI part of C++11? Is it certain to come in the next version of the C++ standard? Is C++/CLI provided or going to be provided by any major compiler vendor? You had more than 5 years, how much longer do we have to wait to see what you are talking about as a requirement for a language extension happen for C++/CLI? Is it going to happen at all?

As a side note, yes, C++/CLI is an ECMA standard, but that's not at all what we are after here and not what you have been talking about in the interview. The exact wording in the interview was: "things we already know are certain or likely to come in C++0x and which we hope all compiler vendors will provide too". We are talking about making extensions to C++, there is only one standard that really matters here - the ISO C++ standard for the language itself. It is nice if things outside of the standard are somewhat structured and standardized, but let's not be silly here, when we are talking about standard-compliant C++ code we mean only one standard - the ISO C++ one.

Same with points other than 5.

Since someone mentioned Bjarne Stroustrup, let's hear what he thinks of C++/CLI, too:

"I am happy that it makes every feature of the CLI easily accessible from C++ and happy that C++/CLI is a far better language than its predecessor "Managed C++". However, I am less happy that C++/CLI achieves its goals by essentially augmenting C++ with a separate language feature for each feature of CLI (interfaces, properties, generics, pointers, inheritance, enumerations, and much, much more). This will be a major source of confusion (whatever anyone does or says). The wealth of new language facilities in C++/CLI compared to ISO Standard C++ tempts programmers to write non-portable code that (often invisibly) become intimately tied to Microsoft Windows."

Heh, maybe C++/CLI will not become part of ISO C++ after all. Maybe it is not such a good and clear way to extend the language as Herb Sutter from Microsoft suggests. Maybe we indeed won't have any compiler vendors other than Microsoft supporting it anytime soon.

In sum, Herb, I am, frankly, amazed at your willingness to call black white, but... we aren't stupid. Black is black. White is white. What you did with C++/CX goes very much against the principles you outlined in the interview.

Thanks to both Jim and Herb for the replies. This is a reply to Herb, with another short reply and a short reply to Jim following.

@hsutter

Before I go further, let me sanity-check something: Do we agree that the route of "extending TLB #import" leads to a full ATL + MIDL like approach?

No. ATL + MIDL is a reasonable way to write code, but we are talking about something else. We are talking about not only "extending TLB #import" to cover WinRT modules, but also about doing the reverse of #import, that is, parsing regular C++ code and generating wrapper C++ code in order to expose regular code to WinRT.

The developer would write the above code, use class C in a regular way in the module, and direct the compiler to expose C for consumption by WinRT. There are many different ways to direct the compiler to do this (pragmas, __declspec, predefined bases, separate command line switches, etc), we can talk about pros and cons of each. The compiler would take the above code and generate wrapper C++ code representing C, I2 and I1 for WinRT. I1 would translate into __internal_I1 and would derive from IInspectable. I2 would translate to __internal_I2 and would derive fro IInspectable as well. __internal_I1::G would return a WinRT string. __internal_I2::H would return a refcounted pointer to __internal_I1. There would be a factory for creating a wrapper for C, which would return a refcounted pointer to it. And so on.

We can discuss how to do the translation for delegates, events and everything else. This is a lot to talk about, but from what we see, for every feature a regular C++ mapping of concepts can be done.

To create an event, you create a member of the type winrt_event<>, then return a reference to that member via an interface function. When you direct the compiler to expose your class for consumption by WinRT, it generates C++ wrapper code that maps winrt_event<> to equivalent constructs in WinRT. If you later import the same component in a different C++ project via #import, the compiler generates C++ wrapper code that maps WinRT constructs back to winrt_event<>. You subscribe to an event using the plus operator or the Subscribe function on winrt_event<>. The latter lets you save a cookie you can use pass to the Unsubscribe function on winrt_event<>.

If you subscribe to an event you expose in the same project, the call goes directly through C++, without passing through WinRT. This is a huge plus for many reasons (eg, optimization), but in case this ever becomes a minus, you can make the call go through WinRT by directing the compiler to export the class that exposes the event and then reimporting the result via #import.

I hope the above is sufficiently clear.

On to the post:

So I'll assume (please correct me if I'm wrong) that we agree that extending the #import model for the consumption side means that the authoring side would employ something like a MIDL language and compiler/toolchain -- so that programmers could specify the additional information that is needed for metadata -- plus a supporting structure of macros, base classes, and/or ATL-like smart pointers. Correct?

No. Smart pointers, base classes and possibly macros - yes. MIDL - no. I am not convinced that MIDL is worse than C++/CX either, but right now let's concentrate on using straight C++ without MIDL.

At the outset of the project, my team was asked by our management to take as primary design requirements that we find a programming model that should: R1. Enable full power and control, including both consumption and authoring. ... R2. Be as simple and elegant as possible for the developer.

Good goals. The approach proposed above achieves both - the wrapper classes allow R2 while being thin enough not to harm R1. It is possible to capture the code generated for wrapper classes and partially rewrite / reuse it to get more R1 if one needs this.

A. What is the minimal code syntax and build steps it would it take to express this in a MIDL-style approach, and across how many source/intermediate files? What's the best we could do in that kind of alternative approach?

B. How do you feel about these two goals? 1. Minimize the actual amount of non-portable non-ISO C++ code/tools developers have to write/use to use WinRT. 2. Make the code in the .cpp file itself appear to be using a C++ library, even though it actually is not (it is generated from Windows-specific language/extensions that are just located in a different non-.cpp file).

Ideally, the only non-portable non-ISO C++ code is #import and #pragmas. There is nothing wrong with having generated code. The requirements on the original code from which the code is generated do not have to be artificial. We are trying to add the ABI so that we can cross over to other languages and technologies, fine, let's do this by generating additional code and using that when we are crossing over. When we don't have to cross over, let's just use (instantiate, call, etc) the original code as is, to the maximum extent possible. Should I elaborate or is what I am saying here clear?

I strongly disagree that C++/CX minimizes the actual amount of non-portable non-ISO C++ code developers have to write to use WinRT. Yes, the above example is 6 lines of code in total and you turn it into C++/CX by adding a single 'ref' keyword, but this single 'ref' keyword makes all of the 6 lines non-ISO. 'Screen s;' looks deceptively like ISO, but if 'Screen' is a ref class, it is not ISO, because the ISO standard does not say what the compiler has to do to create an instance of a ref class. Same for the call to 's.DrawLine', the ISO standard does not say what the compiler has to do to call an instance of a ref class.

If we try and evolve your example just a tiny bit, the non-ISO nature of it will become much more apparent. Say, Screen has to be allocated dynamically, then 'Screen s' becomes 'Screen^ s = ref new Screen()'. Whoa, we added a hat and a second ref. That's 3 keywords for 6 lines now. If arguments to DrawLine have to be WinRT objects as well, there are more hats and refs. If we proceed in that direction we will very soon flood a good deal of our code with hats and refs. And we aren't even talking about non-C++ things like events yet, we are just talking about the trivial example you have chosen to start from.

Compare this mess with wrapper classes.

One can view C++ as a framework for using different technologies. We can use very different technologies like, say, ODBC and OLEDB, simultaneously, there are a lot of very diverse technologies like that which we can use, and we can do it all with next to no changes to the language (#import or an analog is all that is required, if that). What C++/CX does is take one such technology (WinRT) and shove it down into the language, breaking internal logic in many places along the way. What the proposed wrapper classes do is keep WinRT outside the language, making it cooperate via a layer of adapter code. It is clear to me that the approach with wrapper classes is a vastly better one.

You said many interesting things. In particular, you said that you "will be pushing for language extensions in Microsoft's C++ even before the product is fully compliant to today's standard". That said, you caveated this as follows:

"Don't get me wrong, I'm certainly not pushing for a lot of extensions, and certainly not anything where there isn't already a clear need and some community agreement. And when I propose those features within Microsoft, it will be after I've already talked with the world's top experts about how important they are and how they should be specified; I hate half-baked homebrew misfeatures and that's certainly not what I'm after here. I'm pushing for a handful of carefully chosen and carefully implemented extensions that the community is clamoring for, and which are not proprietary but rather the opposite: things we already know are certain or likely to come in C++0x and which we hope all compiler vendors will provide too. A standards committee can sometimes be inventive, but mostly it's supposed to standardize existing practice; I want to help Microsoft help the standard by providing just that "existing practice in the field" for these key upcoming features. Indeed, by trying to carefully wield Microsoft's product as a leader in this way I personally hope to help move the whole community forward on all compilers and platforms, as other products are encouraged to implement them too, which will benefit everybody no matter whose tools they're using."

I have a hard time reconciling the above with what happened with C++/CX. Did you significantly change your stance on extensions since the time you gave the interview? If not:

Isn't a third extension (on top of countless things marked as "Microsoft-specific" which we take as a given) in 10 years "a lot"?

Was there really "a clear need" for C++/CX given that a strict C++ alternative as suggested in this thread was possible as well (if you disagree it was possible, I'd appreciate knowing why, but I have already asked about that a number of times to no avail, so if you disagree but won't say why, just skip the question)?

Was there "some community agreement" around C++/CX?

Is C++/CX one of the things the community "has been clamoring for"?

Is C++/CX something that is "not proprietary but rather the opposite: thing we already know is certain or likely to come" in the next version of the C++ standard and which you "hope all compiler vendors will provide too"?

Are you encouraging other vendors to implement C++/CX "to the benefit of everybody no matter whose tools they're using"?

These are serious questions. What changed between 2002 and 2010 or whenever you started working on C++/CX? Why? Answers to these questions will help us understand what you are going to do with C++ in the future. I am not too optimistic, but, well, explain yourself. We'll listen.

I thought I would point out that every single optimization you describe you did for C++/CX would have been possible if you generated wrapper code:

There are two main places we are able to optimize. The first is for arguments to functions. In C++/CX, many parameters are hats (^). A ^ implies that we are doing automatic reference counting. For instance, if you have a ^ that is a member of struct and you assign to it, an AddRef will automatically be performed. In some ways, this is similar to a shared_ptr. However, when a ^ is passed as an argument to a function, we will not do an automatic AddRef/Release around it. To accomplish something similar with a shared_ptr, you would have the function take a "const shared_ptr<>&". However, this is fundamentally different in that it means at the ABI that a pointer to the variable is actually passed.

With wrapper classes, wrapper code would have taken "const shared_ptr<>&" or const reference to a different type of smart pointer, then it would have taken the actual value of the smart pointer, without doing an AddRef, make the underlying WinRT (COM) call, then returned the result without doing a Release. Done.

We will also do this for String^ parameters. String^ are different from other types in that there is no vtable for them and they may take one of two forms. There is what is called a "fast-pass" mechanism where a "normal" C/C++ string can be passed to an API that takes a String^ (HSTRING at ABI) without copying the string. We will automatically do this for string literals, wchar*, and all of the string classes we ship. Other string classes can easily support this as well. If the callee decides to hold onto the string it will be automatically copied if the string was fast-pass, but if it was not fast-pass then the reference count of the string can just be bumped. So, both the caller and callee can do the optimal thing.

A naive approach:

The wrapper code takes const std::wstring&. It extracts a const wchar_t* pointer from the wstring and passes that to WinRT without wrapping it into HSTRING. Returned HSTRINGs are converted to wstring.

A more involved approach, if the team thinks it is worth it to save on conversions from const wchar_t* (eg, literals) and HSTRING (return values from WinRT) to wstring:

The wrapper code implements methods with string arguments and or return values as more than one method with arguments and return values being either wstring or a library class that wraps all of wstring / HSTRING / const wchar_t*. There is no need to try all combinations, two combinations are all we need: either all types are wstring, or all types are the uber-wrapper class. The implementation code for the method that uses the uber-wrapper class uses the logic you have in C++/CX.

There are alternative approaches as well. Done.

The second opportunity for optimization occurs when calling methods of a class. Keep in mind that methods of a class may actually be on different interfaces at the ABI. If you have a WinRT class Foo, when you declare a Foo^, you are holding onto the default interface of Foo. If you call a method of Foo that is on the non-default interface, the compiler will generate a QueryInterface call to get the interface and then call that method. The optimization that is possible is when you make multiple calls to a non-default interface of a class (through a local variable that the compiler can prove can't change). Someone programming in COM today or using the low-level WinRT ABI would likely see this and cache the QI call themselves, but the compiler can do this automatically for ^.

The wrapper code for a WinRT class can, of course, cache all interfaces it ever calls. A very simple approach which might be enough for the vast majority of cases would be to simply QueryInterface for all supported interfaces in the constructor. If this ends up wasting too much time for classes implementing multiple interfaces (multiple calls to QueryInterface in constructor mean multiple interlocked calls, same for destructor), one can always cache queried interfaces on demand. There are lock-free techniques for this.

Alternatively, if any of the above is not acceptable, the wrapper code can always expose classes wrapping non-default interfaces. Anyone who finds out that the code spends too much time doing AddRef / Release on the arguments can request an instance of such a class and do the calls through that instance.

Now, let's see an optimization you would be able to do if you did everything in C++ instead of C++/CX:

Suppose you have a class you are exposing to WinRT. If you are also calling its methods in some of your other code in the same module, the compiler would be able to inline these calls.

Unlike saving a pair of calls to AddRef and Release here and there, having the ability to inline code actually matters. There are many cases where a developer has to split his code into several methods, as dictated by a design (witness property getters, for example), there is a lot of performance to be gained by putting these chunks of code together at runtime, and only the compiler can do this. The exact benefits depend on the design, of course, on average they are significant, and sometimes they are simply huge.

Are you doing the inlining of WinRT code in the same module right now? Across different compilation units? I don't think so. If you went with straight ISO C++ you would have gotten this for free.