Come learn about new Microsoft technologies that enable you to make your data available over the Web through a simple REST interface and using open formats such as plain XML, JSON or even RDF. We also discuss the underlying entity framework that makes it easy to model, publish, and program against your data over the Web.

Note: That's one of the members of LINQ-Related Sessions at MIX 07. I mentioned that Pablo Castro promised "to put a Web spin on the Entity Framework, which might include a side trip for LINQ to Entities." It turns out that I was right.

Project Astoria consists of a combination of patterns, libraries and an online service that explores the concept of data services for the web. Currently, Astoria data services use relational databases as the underlying store, but in general the nature of the store does not surface in the Astoria interfaces.

The goal of Astoria is to facilitate the creation of flexible data services that are naturally integrated with the web. As such, Astoria uses URIs to point to pieces of data and simple, well-known formats to represent that data, such as JSON and plain XML. This results in the data service being surfaced to the web as a REST-style resource collection that is addressable with URIs and that agents can interact with using the usual HTTP verbs such as GET, POST or DELETE.

In order for the system to understand and leverage semantics over the data that is surfacing, Astoria models the data exposed through the data service using a model called the Entity Data Model (EDM), an Entity-Relationship derivative. This organizes the data in the form of instances of "entity types", or "entities", and the associations between them. ...

Currently Astoria can represent data in plain XML, JSON (JavaScript Object Notation) and in a subset of RDF+XML.

The default representation format is XML. This is not a specific format; it is a fixed mapping of the entity structure to XML elements, with some added semantics to make use of the information provided by the EDM schema.

Q: What is the relationship between Astoria and Windows Communication Foundation (WCF)? A: All Astoria data services are essentially WCF services. You can see Astoria data services as a special form of WCF service that is aware of the nature of the data that is exposing.

Q: What is the relationship between Astoria and the Entity Data Model (EDM)? A: Astoria data services uses the EDM as the model for the data exposed through its services. Having a high-level data model allows Astoria to provided added semantics to the service, such as having a clear definition of what an “entity” is for a given service or how to navigate from one entity to another associated entity.

Q: What is the relationship between Astoria and the ADO.NET Entity Framework? A: Astoria exposes data modeled as EDM models. The ADO.NET Entity framework is the natural choice for an infrastructure that already supports the EDM and through mapping technology can present relational stores in EDM terms. The Astoria runtime that sits in the web servers and serves the requests to data services is built on top of the ADO.NET Entity Framework.

I plan to test the toolkit later this week and will add links to posts with my results here.

This is the second of the promised MIX07 announcements of projects that take advantage of the Entity Framework, which was recently cut from the Orcas release of Visual Studio. The first is here.

Update 5/31/2007: Granville Barnett has posted a new article on DotNetSlackers.com, "Interacting with Astoria Data Services," which describes Astoria Web services, explains their impact on rich Internet applications (RIA, "rich interactive applications" in Microsoftese), and provides examples of using Astoria with ASP.NET AJAX.

During Monday morning's MIX07 keynote, Andrew Conrad spilled the beans about the project he's been working on for the past few months: "Project Jasper." Also known as Dynamic ADO.NET, here's the official description of this new data-access technology from Andrew's blog post:

Microsoft Codename “Jasper” is a set of components aimed at fulfilling the need for a rapid and iterative development experience for data. With Jasper, you are able to just point at a database and immediately begin coding against its data using intuitive, domain-specific data classes. No configuration and no source code generation are required. Jasper works with existing application frameworks (including ASP.NET, WinForms, and WPF) and existing, real-world databases.

Jasper provides a programming interface to your data that is well-suited to rapid development. Jasper provides two main services:

Jasper leverages the power of dynamic languages and the concept of convention over configuration to provide a programming surface for data that enables rapid development of data-bound applications. While most other rapid data access frameworks are only capable of working against simple databases, Jasper can scale to almost any database, regardless of size or complexity. This is possible because Jasper takes advantage of the ADO.NET Entity Framework’s significant investments in mapping and conceptual data modeling. [Emphasis added.]

Jasper's dependence on the ADO.NET Entity Framework (EF) is significant, considering Saturday's announcement that Orcas has dropped the EF from the RTM, after an earlier postponement of the Entity Data Model (EDM) Designer.

According to Andrew:

For CTP release, we are specifically targeting Visual Basic 9 (Orcas version of VB.NET) and Iron Python 1.1. In other words, these are the languages we did a bulk of our testing for the CTP release. The reality is any CLR language that supports late binding can be used with the Jasper framework

I plan to give the MIX07 CTP a workout this week and will add links to reports of my results here.

Come learn about how new technologies from Microsoft bring together the concepts of dynamic languages and ADO.NET and allow you to quickly develop data driven Web pages using ASP.NET dynamic data controls, scalable to even the most complex databases.

The short answer is – We designed Jasper specifically for CLR languages with late-bound facilities such as VB and IronPython. C# doesn’t currently support late-bound calls and hence the answer would be no though some aspects of Jasper may still be applicable.

He then goes into an explanation of how C# might be used. The real question is "Why would anyone want to use C# in a dynamic, late-bound environment?" Andrew then moots the issue with:

[W]e are actively working on moving Jasper over to the dynamic language runtime - the freshly announced .Net addition to the runtime designed to extent the CLR for dynamic typed CLR languages (VBX, IronPython, IronRuby). Based on early analysis it is a strong possibility that we move away from the base class mechanism and support a more dynamic model where data class types (or even specific instances!) can be modified at runtime to allow the user to add their own business logic.

Project Jasper is geared toward iterative and agile development. You can start interacting with the data in your database without having to create mapping files or define classes. You can build user interfaces by naming controls according to your model without worrying about binding code. In this interview Shyam Pather, Carl Perry, Andrew Conrad and Jeff Derstadt, go deep whiteboarding and demoing Jasper.

Sunday, April 29, 2007

[W]e have decided to ship the ADO.NET Entity Framework and Tools during the first half of 2008 as an update to the Orcas release of the .NET Framework and Visual Studio.

Mike does his best to spin the bad news by starting the announcement with:

Microsoft is deepening its investment in the ADO.NET Entity Framework as a critical piece of Microsoft’s Data Platform vision.

Many Microsoft observers will read that sentence as:

Microsoft is deep-sixing its investment in the ADO.NET Entity Framework as a critical piece of Microsoft’s Data Platform vision.

It's too soon to make that judgment, but there's no question that Mike's announcement delivers yet another blow to those developers (like me) who followed Microsoft's primrose path into its previous object/relational mapping (O/RM) tool void.

We made a decision today to put an early end to our exploration of the Entity Framework for our current product work, and to fall back on NHibernate. ...

When the EDM designer started to drop out of the CTP's (post-August 2006), we had hoped that it would reappear around beta 1, and would be fairly solid around beta 2. We're scheduled to ship our current work product in coincidence with Orcas. We expect to start customer previews when a go-live license is offered. Without an editor for the metadata, we really can't justify the use of the EF to our customers.

The phone call [to the ADO.NET EF team] took place in two parts. The second part of the call got into some NDA-covered material that Kevin wasn't able to stick around for. Tim and Dan shared some info that gave me with a definitive and conclusive basis for making a sound call for our product. I'm very grateful to the EF team to have been enabled to make such a clear cut decision.

I won't remain committed to the Entity Framework if the EDM Designer doesn't arrive by Orcas RTM.

I see the EDM designer as one of the most compelling features of EF in its current incarnation. It's the piece that translates well to how we expect our customers will expect to work with the entities in our product.

If the EDM designer is not in place by RTM, then I don't expect that we can ship a product to our customers if part of the product's customization story is based on the presence of the EDM.

We would need to fall back on NHibernate, and we have had this contingency plan in mind since we began the foray into EF.

Scott implemented his contingency plan two days later. (He insists he's not a member of the "NHibernate Mafia".)

Yes, it turns out we do. Microsoft envisions an Entity Data Platform that enables customers to define a common Entity Data Model across data services and applications. The Entity Data Platform is a multi-release vision, with future versions of reporting tools, replication, data definition, security, etc. all being built around a common Entity Data Model.

Mike disclosed another part of the data access strategy in the first post:

Microsoft will be leveraging the Entity Data Model in future versions of Microsoft products such as SQL Server. This Data Platform vision enables customers to leverage their investment in a common conceptual entity model across product lines. [Emphasis added.]

The announcement speaks about Microsoft building applications on top of Entity Framework and also that the Entity Framework is moving towards SqlServer. Now, if you can add 1 and 1 together, you of course realize that if you have an application, say Sharepoint vNext, build on top of Entity Framework and you have one of your major cash cows SqlServer enabled with Entity Framework, you have a combination to earn more money: people who want the next version of your application also want your new database system.

The Entity Framework in ADO.NET vNext was designed to be database independent, and it might still be database independent when it eventually ships (I personally don't believe their H1 2008 mark). If you have one of your other major cash cows, say Sharepoint vNext, build on top of ADO.NET vNext, you then open up the road to host Sharepoint on say Oracle, DB2 or an open source database. This then would alienate one of your other cash cow products, SqlServer which will have the Entity Framework build-in.

LINQ to SQL supports rapid development of applications that query Microsoft SQL Server databases using objects that map directly to SQL Server schemas. LINQ to Entities supports more flexible mapping of objects to Microsoft SQL Server and other relational databases through extended ADO.NET Data Providers.

Despite the comments regarding support by LINQ to SQL for databases other than SQL Server by Dinesh Kulkarni, Scott, Guthrie, Matt Warren and Keith Farmer quoted in my Future LINQ to SQL Support for Multiple Databases? post of April 19, 2007, Mike has put that issue to bed. It's clear as part of Microsoft data access strategy that LINQ to SQL is LINQ to SQL Server (only).

Shades of WinFS, ObjectSpaces, Project Green, and the Microsoft Business Framework

Earlier, a major controversy about EF's lack of "Persistence Ignorance" and its need to support Plain Old CLR Objects (POCOs), test-driven development (TDD), and agile programming techniques erupted at Microsoft's MVP Summit conference in Redmond on March 12 - 15, 2007.

But the real problem is that .NET developers might equate the result of EF's integration into the next version of SQL Server (code-named "Katmai") with the demise of ObjectSpaces, EF's ill-fated predecessor OR/M tool, when it was integrated into WinFS. My June 26, 2006 Microsoft Bites the Bullet: WinFS Bites the Dust post delivers the grisly details of the deaths of WinFS, ObjectSpaces, Project Green, and the MBF.

ObjectSpaces is not participating in Whidbey beta 1, however, it remains part of the Whidbey/Yukon wave and will be made available as a downloadable add-on pack for the .NET Framework Whidbey shortly after Whidbey ships. This additional development and stabilization period will be focused on improving the overall ObjectSpaces programming experience and providing tighter integration with WinFS, the next generation file system in Longhorn. The schedule for the ObjectSpaces mapping tool is also being adjusted accordingly.

His post concludes with:

Bottom line - this will mean some change in the release schedule, but should not adversely effect anyone's plans to deploy the ObjectSpaces framework. In fact, I believe the tighter integration with WinFS will significantly justify the delay.

Does that language sound familiar? Compare the comments with those to Mike's initial post.

Even more ominous is Barbara Darrow's April 26, 2007 speculation in CRN that "Katmai, Orcas Link Could Delay Longhorn Release." If true, which I doubt, Katmai's EF dependency could delay it and Longhorn Server until probably late 2008.

I apologize for the “PR-spin” feel people have associated with this post. Even a short delay of the Entity Framework is a difficult message, especially to convey through the imprecise nature of written communication. How do we convey that we believe this to be the right decision, without people losing confidence in the technology, particularly in light of Microsoft’s less-than-perfect record in shipping ORM solutions, and in the wake of WinFS? I struggled with the wording. With the approach. With the message. Yes, marketing was involved in reviewing the content (of course). I regret now accepting some of their input, but the message is the same. The Entity Framework will not ship in Orcas. It will ship shortly there-after. The slip isn’t about Microsoft’s lack of commitment to the technology. It does give us more time to address some key feedback we’ve received to date from internal and external customers, including providing at least CTP-quality tools for defining flexible mapping between the store and the objects. There’s no way I can convince you that this isn’t another ObjectSpaces, or another WinFS. You’ll have to look at how we continue to deliver over the next few months. You’ll have to look at the quality of the bits. You’ll have to look at the other investments Microsoft is making around the Entity Data Platform, including announcements and demos this week at Mix.

This year I celebrate my 20th year at Microsoft. In that time I’ve seen Microsoft invests heavily in a number of technologies that, for one reason or another, never come to market. This isn’t one of them.

I'm inclined to believe Mike's comment. NetDocs also comes to mind as an example of a well-funded Microsoft technology that never came to market.

Roger Jennings, a developer and technology writer for OakLeaf Systems,
doesn't think the later ship date is going to be a big deal. "They decided that
they couldn't release the Entity Framework without the Entity Data Model
Designer, which was my contention to start with," he said. "And they've also
just announced a couple of incubation projects that depend on it that also might
have an influence on the delivery date." ...

"Almost everybody is considering [Entity Framework] to be ObjectSpaces
revisited, but I don't think that's the case," says Jennings.

Saturday, April 28, 2007

Jim Hugunin and John Lam [will be] talking about dynamic languages on .NET. (11:45am, Tuesday) This one has so much revolutionary stuff in it that they actually pulled it from the session descriptions on the Mix site. Long story short, search the web on musings about why John and Jim are at Microsoft, add to that the fact that the VB team has been working with them heavily, and you might be able to figure it out.

The original name of their currently anonymous presentation was Just Glue It! Dynamic Languages in "WPF/E":Jim Hugunin - Microsoft, John Lam - Microsoft.

Speakers: Jim Hugunin, John LamAbstract: The web was built using dynamic languages. Their plain-text format made it easy to mash up scripts to create the next great app. Similarly, dynamic languages will find a home in Silverlight applications where plain-text formats are common. Silverlight can be easily deployed, which means that a wider range of dynamic languages will be used in building browser-hosted applications. In this demo-centric talk, you will see this happen before your eyes as we rapidly create an application by combining code and markup from existing samples in Ruby, Python, JavaScript, and Visual Basic. This unprecedented level of integration is possible since all of these languages are implemented on top of the new Dynamic Language Runtime (DLR).

Catch a preview with a few screen captures from the video in this post.

"I joined Microsoft because I was so inspired by the possibilities of .Net," Hugunin said. "The chance to work with the Visual Basic team is wonderful. The chance to make changes into .Net to make it more amenable to dynamic languages is phenomenally exciting."

[W]ith Visual Basic, Hugunin said Microsoft is looking at leveraging some of the simplicity that Basic has. "That feels a lot like what people say they love today about dynamic languages," he said. "People say they love a language like Ruby because it has this simplicity and this dynamism."

And Visual Basic, in its formative years, "was state of the art in providing this simple, dynamic experience. So we see that there's this great potential for Basic to be another great dynamic language, where we keep all the great benefits that we put into Visual Basic .Net, but we have some of this dynamic feel that people love."

Hugunin demonstrated how, with simple changes in the code, Visual Basic could be made to behave like a dynamic language. "But that's just, at the moment, an experiment," he said with a smile.

So there's sure to be a VB hook in Jim and John session.

It appears that Andrew Conrad (one of the original ObjectSpaces devs and an early LINQ proponent) has been working on dynamic languages code for a MIX07 session or two. Andrew says:

More information to come in the new few weeks, but all I can say at this point is that the experience has been very dynamic. And if you can’t make it to Mix07, we should have some screen casts and white papers available some time in the next several weeks. [Emphasis Andrew's]

Plain Old Visual Basic (POVB) as a Dynamic Language?

The following MIX07 session underwent a dramatic metamorphosis recently:

ASP.NET dynamic data controls are part of a powerful, rich new framework that lets you create data driven ASP.NET applications extremely easily. ASP.NET dynamic data controls do this by automatically discovering the schema at runtime, deriving behavior from the database and finally creating an ASP.NET page. Anything that can be inferred from the schema works with almost no user effort. If needed, the page can be further customized either by using static languages such as Microsoft Visual C# or Visual Basic .NET, or dynamic languages such as IronPython, Visual Basic, Jscript, etc. In this talk, we show you how to build rich, database driven Web applications from scratch, such as TaskList along with other demos. [Emphasis added.]

Come learn about how new technologies from Microsoft bring together the concepts of dynamic languages and ADO.NET and allow you to quickly develop data driven web pages using ASP.NET dynamic data controls, scalable to even the most complex databases.

Notice that "dynamic ... Visual Basic" in the preceding description is plain VB, not VB .NET, which I'm betting is a reincarnation of VBScript for the CLR. Polita is the developer of Blinq, so there might be a LINQ hook in the session.

One of the topics of MTS07 was "When is the veil of silence going to be lifted?" It's been almost eight months since Jim added a post to his Thinking Dynamic blog. The last post announced the release of IronPython 1.0 on CodePlex and included a brief history of the three-year project. Microsoft released IronPython 1.1 with little fanfare on April 17, 2007.

Over the last several months, because of my current project I have had to program in a bunch of different languages. (The reason for this will become crystal clear early next week). Basically, I have had to code in C#, VB, and IronPython while at the same time I have been trying learning Haskell and occasionally Ruby - and some Javascript just for fun on the weekends.

What's the chance of an announcement that IronPython (and perhaps even RubyCLR) will support LINQ?

IronPython 1.0 will not support LINQ. We have started to look at it, but will seriously look at it after the summer. It is a meaty issue given that many of the concepts in LINQ are based on static typing.

I think the interesting bit for us, now, is that all this runs on the [.NET Framework] 2.0 runtime. From what I can tell, all it'd take would be some compiler modifications to recognize extension methods (marked by attribute) and to create an easy way to create expression trees, and IronPython would be set for it.

Yes, I'm excited about LINQ.. I've seen many gems come out of .NET (and a few blunders). This, I think, is just plain beautiful.

The example uses C#'s typeof() instead of VB's GetType() function in lines 2 and 3, is missing the Function keyword on line 4, and uses the unsupported ? symbol for Nullable(Of T) on line 4. There are obvious errors on 50% of the lines of the example. Further, the following required Imports statements are not to found anywhere in the document:

Modern applications operate on data in several different forms: Relational tables, XML documents, and in-memory objects. Each of these domains can have profound differences in semantics, data types, and capabilities, and much of the complexity in today’s applications is the result of these mismatches. In this talk, we will explain how the Orcas release of Visual Studio aims to unify the programming models through LINQ capabilities in C#, a strongly typed data access framework, and an innovative Application Programming Interface (API) for manipulating and querying XML.

Database-centric applications have traditionally had to rely on two distinct programming languages: one for the database and one for the application. In the second part of this talk we will introduce LINQ to SQL, a component of the LINQ project designed to help integrate relational data and queries with C# and Visual Basic. LINQ to SQL enables developers to express queries and updates in terms of their local programming language without sacrificing the server-side execution model of today’s high-performance SQL-based approaches. Using these advances, database queries that previously were stored as opaque strings now benefit from static type checking, CLR metadata, design-time type inference, and of course IntelliSense. LINQ to SQL also supports a rich update capability that lets you save changes to an object graph back to the database using optimistic concurrency or transactions.

LINQ to SQL receives about the last 45 minutes of the presentation, most of which uses C#.

Saturday, April 21, 2007

Conventional single-layer DVD-R/RW and DVD+R/RW discs hold 4.7 GB. So unless you have a double-layer DVD burner and media, you won't be able to run the 5.32-GB Visual Studio Orcas Beta1 Setup program from a physical DVD.

Setup fails before completion of .NET Fx 3.5 when run from a networked drive. Believe Setup's Start Page where it says:

Some components require that network connections be temporarily suspended
during setup.

If you try installing from a network drive, which includes third-party apps that mount ISO images to virtual CD or DVD drives, you might experience what appears to be a non-fatal timeout message while Setup is copying its initial set of files. Even if not, you'll undoubtedly see this message during the installation of .NET Fx 3.5:

This means that your double-layer DVD burner must be local to the machine on which you want to install Beta 1. Virtual machines connect to the host OS's physical drives as networked drives, so you're out of luck if you want to install Orcas Beta 1 on Vista as a guest OS.

The Solution

Use Daemon Tools' free Virtual Daemon Manager (VDM) to mount the en_visual_studio_orcas_beta_1_professional_dvd_23591.iso image file to a virtual DVD drive. The current VDM version (4.09) works fine under Windows Vista as a host or guest OS, despite what you might have heard on the Web.

Note: When you install VDM on virtualized Vista, save and run the setup file. You must reboot to finish VDM's installation. Under Virtual Server 2005 R2 RC1 with Vista as the guest OS, you must explicitly run daemon4091-x86.exe a second time to complete the installation after rebooting.

Update: 4/24/2007: Commenters have suggested using WinRAR or IsoBuster to extract the image to executable files. WinRAR isn't free (it has a 40-day trial license) and IsoBuster requires the PRO version ($29.95 personal, $49.95 professional) to open an image of this size. Some commenters have suggested other virtual DVD drive apps, but VDM is free and has proven itself under Vista.

Thursday, April 19, 2007

In my recent tests of the Orcas March 2007 CTP's implementation of LINQ to SQL and the Object/Relational Designer, I tried substituting connection strings to SQL Server Compact Edition (SSCE) and Oracle 10g Express Edition with managed providers, and other databases with OLE DB drivers. Most resulted in "unsupported product" messages and all threw exceptions when attempting to open a connection with the DataContext object. One DataContext constructor accepts an IDbConnection object, which implies at least some degree of database independence.

SqlMetal.exe generates an XML external mapping file (NwindLINQ.xml for this example) that you can use in lieu of decorating entity classes with LINQ to SQL-specific attributes. This feature enables generating POCO (plain old CLR objects), which many developers prefer. The "LINQ to SQL: .NET Language-Integrated Query for Relational Data" white paper's "External Mapping" section indicates that the top-level <database> element of NwindLINQ.xml should have a ProviderType="System.Data.Linq.SqlClient.Sql2005Provider" attribute. It's missing from the mapping file I created with SqlMetal.exe, but it did point me to the System.Data.Linq.SqlClient namespace.

Inspecting SqlClient in Object Browser shows it contains about 167 classes having interesting names such as Funcletizer, which contains Friend Shared Function Funcletize(ByVal expression As System.Linq.Expressions.Expression) As System.Linq.Expressions.Expression. I remembered Funcletizer.Funcletize from Mike Taulty's blog and found it near the end of his Deconstructing LINQ to SQL (Part 2) post:

You have to love the naming here :-) Note that the provider looks to have a SQL 2000 and a SQL 2005 mode on it which makes sense but I hadn't thought about it before even though I had thought that functions such as Take might use something 2000/2005 specific.

From there on in, it gets "really hard, really fast" or at least it did to me. From BuildQuery onwards there are a lot of classes (SqlSelect, SqlBinder, SqlRetyper, SqlMultiplexer, SqlFormatter and a whole lot more) that look to be involved in taking what we have as a tree and turning it into T-SQL.

It's obvious that it won't be a walk in the park to write the corresponding "really hard" extension code to implement another managed data provider and SQL flavor, such as Oracle PL/SQL or IBM DB2 SQL.

Last month, the ADO.NET team hosted a number of ADO.NET Data Provider writers on campus to present information on how to enhance an existing provider to support the Entity Framework features in the upcoming Orcas release. To help provider writers who were unable to attend, we’re publishing the material we presented to our team blog.
In this blog post, you’ll find information about how to extend an existing provider. The blog also has a [Resources].zip attachment that contains the slide decks presented at the event, a help file that covers the CommandTree class, and the source code for a sample ADO.NET Data Provider.

The Resources.zip file contains OrcasSampleProvider.exe, but the source code snippets in "Extending an Existing Provider" specification don't appear to me to comprise the complete source code for an extended provider.

According to the April 2007 "Looking to LINQ" story in Redmond Developer News:

Microsoft is working with other database vendors to build interoperability into their repositories as well. Oracle Corp. officials would only say they're monitoring LINQ's progress, but IBM Corp. says it plans to support LINQ in both its DB2 database and the Informix IDS platform.

Curt Cotner, an IBM Fellow and CTO for database servers, says LINQ could very well be critical in eliminating the fragmentation that exists today among programmers and database developers. Still, Cotner expects LINQ to have its share of both ardent supporters and critical detractors. "It's going to be popular for a significant segment of customers, but there will be another segment that will see it as not down the path they're trying to follow," he says. ...

But Cotner doesn't see [the Java Persistence API,] JPA and LINQ as competing specifications per se. "It addresses a different segment of the population," he says. And frankly, he admits from a developer standpoint, JPA can't touch LINQ in terms of its ability to build native queries into code from the language and development environment.

VistaDB Software, Inc. announced the intent to support LINQ in a future version of VistaDB. VistaDB 3 is an embedded SQL database for .NET, the .NET Compact Framework, and Mono that supports datatypes compatible with SQL Server 2005 (except xml) and uses T-SQL syntax. Jason Short wrote in an April 7, 2007 "LINQ and VistaDB" blog post:

I feel quite strongly that LINQ support in the future will be a vital feature of VistaDB. Since we are already a fully managed database for Dot Net, the ability to make fully type safe queries will definitely set us apart from other database systems. And the fact that we will not have to drop to SQL code to perform that queries will mean a faster running database as well (and that is what you really wanted to hear, right?). ...

LINQ support is on the list for research right now. When will be release it? I don’t know. I would really like to hear from you about your plans to support LINQ. I think it is quite obvious that LINQ is going to change the way a lot of applications are written to query their database.

We don't yet have a LINQ to SQL (DLINQ), but we do have ADO.NET V3 Entities working over SQLce. If you install the Sync Services CTP, you'll see we install a Entity dll for SQLce as well. This CTP isn't directly compatible with the Orcas Feb CTP, so don't expect to do anything with it just yet. It's possible the Feb CTP that was just released would enable Entities over SQLce, but I have to check to see what made it into that build.

LINQ to SQL actually stands for LINQ to 'databases that use SQL' or in other words LINQ for the relational data model. LINQ to Entities means LINQ for the Entity-Data-Model which is a kind of a relational++ model.

"Thanks for your interest in the LINQ project and feedback. Support for other providers indeed is a very important area that we are looking into. Because of schedule reasons, we were not able to fit it in V1.

"As for OLEDB/ODBC, the problem is that these standards really don't fully address the semantic differences between SQL dialects and database type systems. So I suspect that we will have to emphasize managed ADO.NET providers (e.g. Oracle provider, DB2 provider etc.) over ODBC/OLEDB. Unlike in case of ADO.NET v2 APIs where the user decides the SQL semantics by creating appropriate string, LINQ to SQL (aka DLinq) has to understand the semantics of the generated string. Hence, we need separate providers that understand the corresponding SQL flavor and type system.

"That said, there may be enough value in handling a common subset. However, such a 'generic' provider will likely have limitations."

"Unfortunately we haven't had enough time to do a provider SDK or even a public provider API in Orcas. This is definitely a high priority post-Orcas release item for us. Thanks again for your feedback."

We are working closely with many DB vendors, including IBM, to help them update
their ADO.NET providers to support the Entity Framework. Unfortunately, I
do not have any information as to when they may release a new provider.

Update: 5/7/2007: Jay Hickerson of the Visual Basic Team responding to a LINQ Project General forum question about the purpose of the O/R Provider property of LINQ to SQL's DataContext object:

The O/R Provider was originally placed there so that we could generate code
that would work correctly with databases other than MS SQL Server at runtime.
However, the runtime support for this did not materialize in time to get it into
the designer for Orcas. The property has been removed for Beta 2 and there
will not be support for different providers from the designer. We are
considering adding some level of support back in a future release, possibly a
service pack.

I doubt that the ADO.NET team would devote the resources necessary to port LINQ to SQL to competing RDBMSs, especially those for which Microsoft and the database vendor have released managed .NET providers, such as Oracle, but also encompassing products such as IBM DB2 9 and VistaDB 3.0 for which managed data providers are available from the vendor RDBS only.

Presumably, the Entity Framework (EF), which was designed from the get-go as the foundation of database-agnostic O/RM, and LINQ to Entities is the solution for Oracle, DB2, Informix IDS, and PostgreSQL. The ADO.NET team is supporting developers with extending providers for the EF's CommandTree class, which isn't likely to be the same as that for LINQ to SQL. However, the EF and LINQ to Entities seems to me to be overkill for embedded and other lighter-weight RDBMSs such as SSCE, VistaDB 3, MySQL, and Firebird (formerly Interbase). These databases are more likely to be used in projects with simpler object graphs that match the underlying persistence database's schema.

I'm not sanguine about the prospects for a future "provider SDK or even a public provider API" for LINQ to SQL from Microsoft. A future Microsoft LINQ to SQL provider for SSCE might arrive, but SSCE already has a LINQ to Entities provider that, hopefully, will be operable in Beta 1.

I think it's a good bet that an IBM DB2 LINQ to SQL EntityClient provider will arrive around the same date as the Orcas RTM in 2008H1 (?) that Microsoft finally releases EF and its components. I'm not holding my breath for the announcement of an EntityClient provider from Oracle.

Tuesday, April 17, 2007

Mike Taulty, a member of Microsoft UK's Developer & Platform Group who frequently blogs on LINQ-topics, has created a series of short (about 15-minute) video segments devoted to LINQ to SQL as implemented by the Orcas March 2007 CTP. Mike plans to post a new segment each business day until he exhausts his collection.

Following is a complete list of all segments (only two of which were available when I first posted this):

This session demonstrates building a rich interactive application (RIA) using Silverlight and managed code. We'll cover how to use Visual Studio to create applications, how to create UI using XAML markup and code, how to build a custom control, how to retrieve data from a Web service, how to manipulate data with XML and LINQ, and how to use the available programming languages in Silverlight.

Note: Microsoft appears to have redefined RIA from rich Internet application (the common usage) to rich interactive application.

Update 4/28/2007 from Channel9 Buzzcast #13: ﻿In this episode, Aaron Dunnington, Program Manager on the Data Programmability XML Team talks about harnessing the power of the semantic web using Microsoft XML technology stack.

Modern applications operate on data in several different forms: Relational tables, XML documents, and in-memory objects. Each of these domains have profound differences in semantics, data types, and capabilities, and much of the complexity in today's applications is the result of these mismatches. Anders Hejlsberg, Microsoft Technical Fellow and Chief Architect for Microsoft Visual C# and LINQ, will explain how LINQ (Language Integrated Query) unifies these programming models and dramatically improves the experience of creating data intensive Web applications. Anders promises to have few slides and lots of live code demos!

Note: Anders appears to have expanded his title of Chief Architect for Microsoft Visual C# to include LINQ. This might explain the dearth of LINQ sample code in Visual Basic.

Update 4/28/2007: This session should shed some light on the forthcoming LINQDataSource for ASP.NET. Beth Massi says, "Be sure not to miss Anders typing the 'Dim' keyword!"

Bonus Session: Pablo Castro promises to put a Web spin on the Entity Framework, which might include a side trip for LINQ to Entities. Will Pablo disclose more details about CloudDB or Blue/Cloud, a.k.a. Pacifica? Don't miss:

Come learn about new Microsoft technologies that enable you to make your data available over the Web through a simple REST interface and using open formats such as plain XML, JSON or even RDF. We also discuss the underlying entity framework that makes it easy to model, publish, and program against your data over the Web.

Office Pool: How many years ago was Pablo's mug shot for DevTeach taken?

Update 4/29/2007: Pablo's presentation should be even more interesting considering that Microsoft announced last night that Orcas has dropped "the underlying entity framework."

Andrew Conrad, one of the members of the original ObjectSpaces team, says he's been working on the following project for the last year:

Update 4/28/2007 New description: ASP.NET dynamic data controls are part of a powerful, rich new framework that lets you create data driven ASP.NET applications extremely easily. ASP.NET dynamic data controls do this by automatically discovering the schema at runtime, deriving behavior from the database and finally creating an ASP.NET page. Anything that can be inferred from the schema works with almost no user effort. If needed, the page can be further customized either by using static languages such as Microsoft Visual C# or Visual Basic .NET, or dynamic languages such as IronPython, Visual Basic, Jscript, etc. In this talk, we show you how to build rich, database driven Web applications from scratch, such as TaskList along with other demos. [Emphasis added.]

Come learn about how new technologies from Microsoft bring together the concepts of dynamic languages and ADO.NET and allow you to quickly develop data driven web pages using ASP.NET dynamic data controls, scalable to even the most complex databases.

Notice that "dynamic ... Visual Basic" in the preceding description is plain VB, not VB .NET. Polita is the developer of Blinq, so there might be a LINQ hook in the session.

More information to come in the new few weeks, but all I can say at this
point is that the experience has been very dynamic. And if you can’t make it to
Mix07, we should have some screen casts and white papers available some
time in the next several weeks. [Emphasis Andrew's,]

Friday, April 13, 2007

One of the reasons for adopting the new template for the OakLeaf Systems blog was gaining more horizontal space for code samples. Here's what Microsoft says about Consolas and its downloadable installer:

Consolas is intended for use in programming environments and other circumstances where a monospaced font is specified. All characters have the same width, like old typewriters, making it a good choice for personal and business correspondence. Optimizing the font specifically for ClearType allowed a design with proportions closer to normal text than traditional monospaced fonts like Courier. This allows for more comfortable reading of extended text on-screen.

The package will change the default text face in Visual Studio to the Consolas family.

This package is only intended for licensed users of Microsoft Visual Studio 2005.

Another plus is that Consolas has a narrower pitch than Courier New/Courier, similar to the difference between the traditional Elite and Courier typewriter fonts, but Consolas is more readable than Courier New in a smaller height and width. Here's a bit-map example of code from

I've modified the CSS for the code samples in the blog to use Consolas as the default. Windows Vista, Windows 2003 Server R2, and most Longhorn builds install the Consolas font family. Strangely, Orcas doesn't install it.

If you don't have it installed, you'll see line wrapping in some examples because of the greater width of Courier New characters.

Thursday, April 12, 2007

Entity change tracking is required to detect and resolve conflicts that can occur when more than one user updates an instance of a particular entity, such as a SalesOrder or MedicalRecord object, concurrently. Thus the common term for this problem, which occurs only with multi-user systems, is concurrency conflict.

There are two methods of handling concurrency conflicts: pessimistic locking and optimistic locking. If you're familiar with optimistic locking and handling concurrency conflicts, jump to the "Resolving Concurrency Conflicts with LINQ to SQL and the Entity Framework" topic.

Pessimistic Locking to Prevent Concurrency Issues

If the application's data store, usually a relational database, uses pessimistic locking, the following actions occur in response to an entity update request:

User 1 requests a copy of the entity from the data store.

The data store locks the record(s) associated with the entity to prevent other users from editing them. (For a SalesOrder, there is typically a record for a SalesOrderHeader and one for each SalesOrderLineItem.)

User 1 edits the entity.

User 1 submits the changes to the data store directly (two-tier) or through a business services layer (n-tier).

The business services layer, the data store, or both determine that the changes don't violate either business policies or relational integrity.

The data store completes the transaction, updates the record(s) and releases its locks on the records.

User 2 repeat steps 1 through 6.

Pessimistic locking prevents concurrency conflicts at the expense of data accessibility. If User 1 goes to lunch or home after completing step 1 but before performing step 4, other users can't edit (and applications might not even be able to read) the locked data until User 1 returns to work and completes step 4 or an application-imposed time limit expires, which cancels the editing operation.

Optimistic Locking and Handling Concurrency Issues

Optimistic locking maximizes data accessibility but introduces uncertainty into data values when multiple users edit the same records simultaneously. The degree of uncertainty increases when multiple users edit independent copies of the data and submit the changes to the data store. This problem arises in any application that makes use of object/relational mapping techniques or ADO.NET DataSets. Data uncertainty is further exacerbated as the copy grows older; that is, the original data becomes stale.

If the application's data store uses optimistic locking, the following actions can occur in response to User 1 and User 2 obtaining a copy of the same entity concurrently:

User 1 requests and obtains a copy of the entity. The data store doesn't lock the associated records.

User 2 requests and obtains a copy of the entity before User 1 submits her edits.

User 1 submits her edits which conform to business rules and don't violate relational integrity.

A short time later, User 2 submits his edits which also conform to business rules and don't violate relational integrity.

If the client, business layer (if present), and data store support concurrency management, a concurrency exception occurs. The exception is needed because User 2 might have made no changes or different changes if he was aware of the changes made by User 1.

The conflict must be resolved in step 6 because User 2 has captured original member (property) value(s) or a timestamp value in step 2. The data store detects concurrency conflict(s) by comparing original values with corresponding member values in the corresponding table cells or a timestamp cell. If any pair of values differ, the data store rolls back the pending transaction and reports that no records were affected by the submission. In this case, the data store, business layer, or client code throws an exception. Repeated attempts by User 2 to update the data with his values will fail until User 2 either abandons the editing attempt or synchronizes his original or timestamp value(s) with those of the data store.

Note: My "Optimize Update-Concurrency Tests" article from the October 2003 issue of Visual Studio Magazine compares the performance of concurrency management by testing member values of typed DataSet tables or SQL Server 2000 timestamp values. The tests results showed timestamp-based test with relatively large objects, such as a SalesOrderHeader, outperform member value tests by a factor of about 2.1.

Resolving Concurrency Conflicts with LINQ to SQL and the Entity Framework

Concurrency conflicts can be resolved by blindly allowing the last update to prevail (called the "last in wins" approach) or by presenting the user with resolution options and (hopefully) sufficient information on which to make an informed judgment.

Orcas March 2007 CTP's LINQ to SQL implementation offers developers the capability to easily create a conflict resolution message box, such as this:

The conflict occurred because the original ContactName value (Maria Anders) didn't match User 1's change to the database value (Maria Anders-Brown). User 2's option descriptions are taken directly from the Orcas March 2007 CTP's online help file. The database value is retrieved with the exception raised by the database (SQL Server 200x only for LINQ to SQL). Current database values are needed to make an informed conflict resolution decision. (Timestamp values aren't meaningful; they only indicate that underlying data has changed while an edit was in progress.)

Conflict resolution by user intervention is much more practical in a conventional, 2-tier (client/server) environment than with n-tier or service-oriented architecture (SOA) architecture.

The Entity Framework's ObjectStateManager detects concurrency conflicts but doesn't offer a built-in handler to resolve them at the member level. The ObjectStateManager provides all current and original values (but not database values) for comparison:

The Entity Framework developer must add procedural code to resolve the conflicts; retrieving database values requires a round-trip to the server with a DataReader to retrieve the current row values.

The Current Need for Do-It-Yourself Change Tracking

The key to detecting concurrency conflicts is to maintain original member values or the timestamp value. The DataContext object handles change tracking for LINQ to SQL; the ObjectStateManager does the same for the Entity Framework's Object Services layer. Neither object tracking approach is well suited to n-tier, SOA, or ASP.NET Web applications that update data because the object copy being edited in the client tier is detached from the change tracker in the middle tier or service layer. Currently, the developer must write change tracking code to track changes to the detached object and pass the change information along with the edited object to the middle tier. I use the term detached (from the change-tracker) so as not to confuse the problem with disconnected (from the database.)

Why does LINQ to SQL not implement a change tracking mechanism on an entity level?

Without the entity tracking mechanism, it would seem like the framework is primarily targeted at developers working in a truly connected state (i.e. always use a DataContext to query for entities and remain connected (to a DataContext) throughout the lifetime of the entities).

I consider it a really big obstacle to fully accept LINQ to SQL as a mature ORM if I have to implement the change tracking mechanism every time I design an architecture with more than 2 tiers (e.g. a standard ASP.NET application that creates a DataContext for each request, makes changes and disposes resources).

I used ASP.NET as an example to show the stupidity of this EDM/Linq to Sql design flaw, but any setup where an entity object gets disconnected from the session/context which fetched it, will result in this [requirement for] 'DIY changetracking' code. And for what? Why isn't this solved properly? Beats me... you'd think that after all these years, Microsoft would come up with a framework for data access / entity management which would really help the developer with taking care of the plumbing crap and let the developer focus on what really matters: the application code.

Although Frans' post is directed primary to the Entity Framework, his complaint about loss of original values in n-tier applications and the requirement for developers to roll their own value tracking code echo's Anders' complaint.

Do-it-yourself change tracking becomes more practical when you compare timestamp values, as Frans observes in this comment. Timestamp and primary key values don't change (unless your table has a composite primary key), so you can use these values as original values in most cases. If you choose the timestamp approach, you must make a roundtrip to the database to compare current values with database values to determine which member values have pending changes. However, database values don't necessarily correspond to original values at the time editing began.

The LINQ to SQL Team's Proposed Change Tracking Solution

I found it strange that Anders' 4/9/2007 post hadn't elicited a response from members of the LINQ to SQL team by 4/12/2007. It turned out that the thread had moved to an earlier and somewhat related update existing disconnected objects (dlinq) thread that started on 2/1/2007. Microsoft's Matt Warren delivered this initial response on 4/11/2007:

The reconnecting API's you have available to you at this time do not support
the scenario you guys are discussing. There is more work coming that makes
3-tier scenario's easier. The re-attach and playback API's are included
because they are the bare minimum required to make any reconnecting work at all,
however cumbersome it may seem at this time. There is
an upcoming feature that automatically databinds LINQ to SQL objects
directly to webforms that use these API's to communicate changes back into the
DataContext. There is also a technology in the works that automatically
serializes LINQ to SQL objects to other tiers, change tracks and data binds the
objects for you on that tier and serializes the objects back, solving the
cyclic reference serialization problem and the indescribable schema problem for
objects with change sets; which seems to match what many of you are
describing. [Emphasis added.]

Matt's subsequent message clarifies the technology in the works:

The mechanism that does the change tracking on the client is similar to a
mini-connectionless DataContext. It is a type that packages up all the
objects, lists of objects and graphs that you want to send to the client.
It serializes itself and everything you've given it automatically. (It
implements IXmlSerializable.) On the client, the same class also manages
change-tracking for all the objects that were serialized with it. When
serialized again (on the trip back) it serializes both the objects and their
change information that it logged. Back on the middle tier, you just make
one call to re-attach the whole package to a new DataContext instance and then
call SubmitChanges.

The remaining question is how the Entity Framework folks plan to deal with this issue.

Another concurrency-related issue is the necessity to store and test the values of all related objects in the graph when undetected modifications to dependent objects might have serious medical or economic consequences. I'll reserve that issue for a future round of the Entity Change Tracking in LINQ to SQL and the Entity Framework quartet.

Updated 4/12/2007: Fixed errors due to uploading a draft instead of the final version of the initial post and added responses from Matt Warren in the LINQ Project General forum.

The dual Web role application has been running in Microsoft's South Central US (San Antonio) data center since September 2009. I believe it is the oldest continuously running Windows Azure application.

About Me

I'm a Windows Azure Insider, a retired Windows Azure MVP, the principal developer for OakLeaf Systems and the author of 30+ books on Microsoft software. The books have more than 1.25 million English copies in print and have been translated into 20+ languages.

Full disclosure: I make part of my livelihood by writing about Microsoft products in books and for magazines. I regularly receive free evaluation software from Microsoft and press credentials for Microsoft Tech•Ed and PDC. I'm also a member of the Microsoft Partner Network.