Recent posts about LINQ and related topics - Page 1 by ThinqLinq

Actions

My Books:

While reviewing the recent updates to RxJs and looking at some of the tutorial information, I started to get perplexed at the difference between select or map, selectMany or flatMap, and where or filter. I’ve typically used the LINQ/SQL methods of select, selectMany and where, but while reading the learnrx tutorial I noticed he was using map, flatMap, and filter and became confused as to how these are related. A quick peek at the source (which is easy with JavaScript libraries) reveals how. Here’s the signature of select and map for example:

So we can see that select and map both extend the Observable prototype and alias the same function. The same is true of selectMany/flatMap and where/filter. As a result, which should you use? It doesn’t matter. Use the syntax that you’re more comfortable with. As a LINQ guy who started with Access in my professional career, I thinq I’ll probably stick to the SQL like methods, but don’t be surprised to see map/filter sneak in from time to time.

For several years, I’ve been demonstrating using Reactive Extensions to handle taking user input and providing real time translations into multiple languages. This code sample has been so popular that even Soma posted an example using the Bing translator in his introducing Rx post. In most cases, they only issue a single request or issue multiple individual requests for multiple languages. To add a bit of fun to my demo, I decided to spice it up a bit and turn the list of destination languages into another observable that I can join with as well. I’ve had this sample available in my Rx downloads for some time including one for RxJs, but haven’t updated them to newer versions (curently 2.2.2) until now. I figured it was about time that I shared my version.

To begin, let’s create a UI. We’ll keep it simple to allow us to focus on the code that comes later:

Next comes the fun part. Before you can do anything though, you do need to sign up to get a free Bing App ID. I’m not going to share mine . Once you have that, your ready to code. But what are we going to do in our code? This example combines three observable sequences: User key presses, destination languages, and service request/responses. In order to work with the observables and to make it easy to work with our DOM, we need to start by importing two JavaScript libraries: jQuery and Rx (we only need the rx.lite.js version for this example).

Of our three observables, the second one is the easiest, so let’s get it out of the way first. To turn an array of languages into an observable sequence, we use the fromArray generator on a simple JavaScript array.

With that out of the way, Let’s take a look at how we are going to issue the service requests to Bing and turn them into an observable stream. We’ll use the jQuery ajax method to issue our request to the Bing translator using jsonp and pass in the required app id, destination language (to), source language (I speak American English, so ‘from’ is ‘en-us’) and the text we want to translate (text). We can then use JavaScript promise to access this invoker and use RxJs’s fromPromise to turn it into an observable stream that returns the string result.

Now that we have these two observables, we need to wire this up with the user’s key presses. Instead of going on and on about each step, here’s the code in it’s glory detail annotated with what each line is doing.

We start by getting the keyup events for the input textbox and turning that into an observable stream (using fromEvent). This is an expensive service operation, so we throttle the request for 1/2 of a second before continuing to let the user finish typing. There’s no reason to send each keypress through the translator. The do method allows us to cause side effects, in this case to clear the last translated results if any exist. We then transform (using select or map) the result to get the user’s supplied text and pass that as the resulting stream. Similar to throttle, we also want to make sure that the user hasn’t just entered the same value twice which would result in the same translation results. We optimize our requests by not reissuing the request using distinctUntilChanged.

The fun stuff comes next. Here we start to merge our observables using selectMany (or flatMap) to match up the input text with the observable list of destination languages. Once we have that, we selectMany again, this time passing in the source string and destination language into the translate method that returns the observable string result and based on that result, we project out the result along with the enclosing destination language.

To conclude, we subscribe to the end observable stream and either append the result to the output list in the case of onNext, or append the error message for onError. Since this is set up over observables, the user can just start typing another string and the process will react to the request all over again. To put this all together, here is the complete code. Remember to get your AppId before you use it because the one I’m showing here won’t work.

Now that I’ve been at Slalom Consulting for 3 years, it’s about time for a hardware refresh. In preparation for that, and inspired by the recent list by Rocky Lhotka of top Win8/RT apps, I thought I would share the list of software/tools that I have found indispensible over the last several years creating software for customers. This is by no means as extensive as the impressive list of tools that Scott Hanselman puts out each year. A while back, I published a list of LINQ tools if you’re looking for some of those.

This list is primarily intended as a list to help me when I build my next laptop and offered here in case you find it helpful as well. It is by no means an endorsement of any specific product and I’m not paid to publish any of these. Feel free to add to this list in the comments as I may well have forgotten something along the way.

System apps and tools

I use this to seamlessly backup all of the computers on my home network. It’s saved my rear multiple times and makes trying out new builds of Windows and other applications a painless task. I’m still using the original version and it’s been grinding along smoothly for years now.

Do you ever wander what’s taking up so much space on your drives? WinDirStat is a nifty way to visualize your space usage and can break it down by folder/file type/etc. It’s an easy way to find those screen capture videos, vhd images, and audio files that can get scattered around the drive over time.

Over 2500 metro-style icon templates that are free (in exchange for giving them your email address that they will use to spam up) and customizable either as XAML or png images. I’ve used these a number of times when needing an icon for a customer and changed the colors to match their brand.

GUI to manage your boot image. Helps out in a pinch when the boot sector gets corrupted or you want to configure dual boot to a VHD image for testing new OS images. It has a free non-commercial version as well as a minimal cost commercial version.

Free virtual machine tool. I’ve found this to be lighter weight and easier to configure than HyperV or VmWare. I’m sure there are features that it doesn’t include, but so far I haven’t had the need for the bigger tools except for running the Windows Phone emulator which does require HyperV.

Low level debugging tool for when you want to see in minute detail what your drives are doing. The information that it provides can be overwhelming, but I’ve found it useful for tracking down pesky file not found issues where a system is looking for a file in a different path than where it actually exists.

This has been my go-to twitter client for years. Unfortunately due to Twitter’s access token limit, they recently announced that they will no longer support new users or upgrades. I may have to switch back to Tweetdeck or Tweetium once the new computer comes but wanted to leave this on my list in tribute to the app that was.

Development Tools

The best development IDE out there. I spend most of my time working in the .Net world, but have had to work with a couple others including Eclipse and am much happier and productive with VS than the other tools.

Any time I want to test out some code snippets but don’t want to crank up a new project and build a UI, I reach out for LINQ Pad. We liked it so much that we published all of the LINQ in Action samples for you to use in LinqPad.

Although not as full featured as the no longer free Reflector, I reach out to the open-source .NET assembly browser and decompiler, ILSpy whenever I want to know how a .Net DLL works under the covers and I don’t have access to the original source code.

This simple tool lets you check the project dependencies of compiled modules which is helpful when trying to understand how existing systems are constructed so that we can try to extend them with additional functionality.

Anytime I need to sniff web requests or test services, Fiddler is my tool of choice. Not only are you able to see the requests and responses, but you can easily create new ones by dragging a previous request to the compose tab and modifying the URI/header/footers to meet your needs and issue the request directly without needing to mess with a browser.

Often I need to compare files or directories to merge them between different versions. There are a number of diffing tools, but this one lets you quickly diff and merge them.

Visual Studio Extensions

I typically do not install many extensions including the popular CodeRush, Resharper, or JustCode because I’m frequently giving presentations and attendees often are more focused on the plug in tools that I’m using than the task at hand. As a result, I typically use a cleaner install of Visual Studio with just a handful of useful tools.

If you’re using Visual Studio and haven’t heard of Nuget, you’ve probably been living under a bridge for the last couple of years. NuGet is the way to manage external packages for your applications. Increasingly, parts of .Net including MVC, Entity Framework, Web API, and others are being distributed through NuGet rather than as part of large .Net version upgrades. It’s a must have extension for Visual Studio at this point.

Visual Studio plug in that allows you to run JavaScript QUnit, Jasmine, and Mocha unit tests the same way you run your MSTest/XUnit tests for .Net code—directly within Visual Studio. Also allows you to debug the JavaScript unit tests quickly and easily.

As I spend more time reviewing code with clients and on public forums, I’m constantly seeing cases where people have issues with Outer joins in LINQ and the various flavors of LINQ. In this post, I’m going to look at a couple options from a syntax perspective that you can use to make working with outer joins easier with LINQ. Naturally, we have a much deeper discussion of outer joins in our book that you’re welcome to dig into as well.

Typically, if you do a join in LINQ, it will perform an inner join where it only returns the records that contain results in both sides of the evaluation (in both tables). If a child table doesn’t have a record for a parent row, that result will be excluded. Consider the case in Northwind where an Order record exists, but there are no order detail lines. The following LINQ query won’t return that record as part of its result set:

from o in Orders
join od in OrderDetails on o.OrderID equals od.OrderID
select new {o, od}

This translates into the following SQL statement (Note to DBA’s that are concerned by the * below: EF does list the individual columns in the actual SQL. I’m reducing them to * for the purposes of this post to focus on the join clauses):

If you want to include the orders regardless of whether it has any detail lines, you would need to turn this inner join into an outer join using the DefaultIfEmpty extension method. LINQ only supports left outer joins. If you want a right outer join, you need to flip the logic of your query to turn it into a left outer join. In order to use the DefaultIfEmpty, you typically need to push the joined set into a temporary value first and then select from it using the DefaultIfEmpty method:

from o in Orders
join innerOD in OrderDetails on o.OrderID equals innerOD.OrderID into Inners
from od in Inners.DefaultIfEmpty()
select new {o, od}

The problem that I have with this is that the syntax seems overly verbose to accomplish this change. As is often the case, Microsoft often gives multiple ways to accomplish the same goal. One method that I’ve started to find helpful is to revert to more of an ANSI 82 style syntax where the joins were accomplished in a where clause instead of a join clause. By mixing and matching the LINQ query comprehensions and lambda syntax, we can restate the above query as follows:

If we check the SQL, we can see that this generates the same SQL as the more verbose version above. Often when I’m doing this in application code, I’ll put the where and DefaultIfEmpty on the same line to let me focus on what I’m fetching from, not how I’m joining them unless I need to focus on that.

There is an issue with using this syntax when joining nullable values or strings. Since the where statement doesn’t know about the cardinality relationship (1-Many, 0-1 – Many, etc), Entity Framework adds an additional check where the nullable value is not null to allow for join cases where both sides have a null value. Changing our query to use Northwind’s Customers and Orders which joins on the string CustomerID values, we can write the following LINQ query which is nearly identical to the one we created before for Orders and OrderDetails:

Note that in this case, we have an additional clause checking for Extent2.CustomerID IS NOT NULL. This may seem innocuous, but I have found in at least one case for the query execution to be significantly slower due to the use of an index scan rather than an index seek caused by this clause. Here you have to be careful when crafting your queries and monitor performance even more carefully to avoid performance bottlenecks unnecessarily.

While this version works better, I still prefer to use associations rather than joins to think about the problem from more of an object graph perspective rather than set based operations. As long as you have a natural association between the entities I much prefer using the associations to navigate through than to have to worry about building out the joins manually each time. We can restate our join even more simply as follows using associations:

from o in Orders
from od in o.Order_Details.DefaultIfEmpty()
select new {o, od}

Note that if you omit the DefaultIfEmpty clause, you would get an Inner join rather than left outer join.

If you have other ways of creating outer joins in LINQ that you prefer, let me know what you thinq.

Coming from LINQ to SQL, I’ve long been a fan of its logging simplicity where you just set the context.log to a stream (console.out) to see the stream of sql statements being issued to the database. Prior to EF6, this has been a rather frustrating omission that has finally been rectified with EF 6. Now, you can log SQL statements just as simply, but with a slightly different implementation. In EF, you set the DbContext’s Database.Log property to an Action delegate that takes a string as input parameter. Thus to log requests to the Diagnostics Trace implementation, you simply set the following:

context.Database.Log = Sub(val) Diagnostics.Trace.WriteLine(val)

This is fine as long as you are willing to accept the default logging implementation. If you want to customize the log output, things get a bit trickier. In my example, I only want to log the calling method and elapsed time that the query took to execute. I’m not as concerned in my current need for the SQL string, but that is easily included as I’ll point out below.

To start, I need to set the calling method from my code that I’ll be able to access in the logging implementation. Since the calling code is actually in ASP.Net Web API calls returning IQueryables, and the database is not being executed until long after my application code has completed. I need to explicitly identify the application calling method rather than the last method from the call stack which would otherwise be one of the Web API internal methods. To handle this and centralize my set-up logic for the context, I’ll create a factory method which configures the context and use that instead of relying on the context’s constructor. Thanks to partial classes, I can extend the generated entity class and include this new property and factory constructor. To make matters even easier, I’ll take advantage of the CallerMemberName attribute to automatically pull the member name from the method that is calling the factory.

Now that we’ve set up the logger, we need to customize the output that is generated. To do this, we need to add a new class that derives from DatabaseLogFormatter (in the System.Data.Entity.Infrastructure.Interception namespace). If you don’t have this namespace, you may need to upgrade to EF6 on order to access the logging functionality. Since the base class doesn’t have a default parameterless constructor, we need to make a new one and simply delegate to the base implementation. With that out of the way, we can supply our own logging. The base implementation gives us hooks to the following interception points.

Method

Description

LogCommand

Writes the SQL Statement to the action implementation prior to executing the statement

LogResult

Writes when the SQL statement has completed

LogParameter

Writes the parameter(s) used in the query

Executing/Executed

Called before and after the database request is made

NonQueryExecuting/NonQueryExecuted

Called for queries that don’t return results (insert/update/delete and non-result return stored procedures)

ReaderExecuting/ReaderExecuted

Called for queries that return tabular data results (select)

ScalarExecuting/ScalarExecuted

Called for queries that return single value results (user defined functions)

Write

Base write implementation that each of the loggers use to format the output.

As stated before in our example, we want to write a log entry when the command completes including the calling method name and execution time. To do this we need to perform the following adjustments to the base implementation: 1) suppress logging the SQL statements in the LogCommand method by simply creating a noop and not delegating to the base implementation, and 2) replace the default result log information with our custom output in the LogResult method. To get the elapsed time, we directly leverage the Stopwatch property of the base class. Here then is our new custom formatter.

One step remains in order for this new logging implementation to take effect. We need to register the custom logger in our solution. To do this manually in code, we create a new class and derive from DbConfiguration. Then in the constructor, call the base class’s SetDatabaseLogFormatter to set the formatter to our new CustomDbLogFormatter. For more information on configuring the logging via your config file rather than in code, see AJ Vicker’s post today on EF 6.1 turning on SQL logging without recompiling your app.

Naturally, the value that we are outputting is just an example, you’re free to make your logging as complex as you want given these basic building blocks. For more details, make sure to read the EF team’s posts on the logging implementation.