Obtaining a memory dump

In this first example, we’ll use a running ASP.NET MVC 5 application hosted by IIS, but the steps here can be used on a normal .NET framework Windows application. Let’s start by taking a full memory dump of a running application.

Download ProcDump and copy it to the server that runs the application you want to debug. Obtain the process ID from the application you want to profile by using Task Manager, and then pass it as an argument to procdump.

procdump -ma <pid>

You should now have a dump named similar to w3wp_171229_151050.dmp in the working directory.

Note:

If you’re running several applications under a single app pool in IIS, it may be easier to debug by changing the app to run under its own application pool, which allows the ASP.NET app to run under a dedicated process.

Inspecting the ASP.NET application state (.NET Framework)

Now that we have a memory dump, it’s time to look at the suspended state of the application. Copy the dump file to your workstation, and then open it via File > Open Crash Dump in WinDBG. Your screen should look like this:

Load the SOS debugging extension, which will allow us to inspect the managed threads:

!loadby sos clr

Then, list the stack trace of every thread:

!eestack

Note:

If get an exception when running this command and you are using IIS Express, try the command again. There appears to be a bug that throws an exception only for the first command run from WinDbg, which should not affect the rest of your debugging session.

You should see a lot of threads in the output. To narrow the results down, search for the namespace of your project in the output text.

We can see that there is an external web request being made in Thread 34. Let’s look at what external URL is being requested. Switch to the thread, and then run clrstack -p to get some more detailed information about each method call.

~34 s
!clrstack -p

Note:

You may see many arguments that contain the value <no data>. This can be caused by compiler optimizations; inspecting the state of these parameters is beyond the scope of this article.

The controller is present in this call stack, so let’s inspect the object instance by clicking on the this instance address, which is a shortcut for the !DumpObj command.

This instance contains a field named _request, which contained a field named requestUri, which has the original URI for this request:

That’s it! The commands vary slightly for dumping different field types.

.NET Core application on Linux

Required:

LLDB 3.9

Locally-built copy of the SOS plugin in the CoreCLR repo - instructions

In this next scenario, we’ll look at inspecting a core dump from a .NET Core app running on an Ubuntu x64 instance. The instance will have a core dump taken while a request is processing, which we will then inspect.

Take a core dump of the process using the createdump utility. These commands assume you have the coreclr repo checked out to ~/git/coreclr, and that you’re running an application built with .NET Core 2.0.

sudo ~/git/coreclr/bin/Product/Linux.x64.Debug/createdump -u (pid)

Load the dump in LLDB. This command also loads the SOS debugging extension.

After a few moments, a CLI will become available. Run eestack to dump the state of all CLR threads. If you get an empty output or a segmentation fault, verify that you are running the correct version of lldb and are loading libsosplugin from the bin directory, and that you have created the core dump with createdump.

eestack

There is an instance of HomeController in the stack of Thread 17. Switch to it to reveal more information about the current request. This time, we’ll inspect the state of an internal .NET Core request frame, since information about the current request isn’t as accessible as it was in ASP.NET MVC 5.

thread select 17
sos DumpStackObjects

Look for the address of Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Frame'1[[Microsoft.AspNetCore.Hosting.Internal.HostingApplication+Context, Microsoft.AspNetCore.Hosting]] in the output, and then dump the object. The name of this class might differ slightly based on the version of the framework you’re running.

Identify the QueryString field address:

Dumping that field reveals the query part of the URL the browser requested!

If debugging applications like this sounds interesting to you, join us! Thanks to Kyle Sletten, Justin Brooks, and Bradley Grainger for reviewing early drafts of this post.

Great to be here!
<script>window.location='https://example.com'</script>

It’s necessary to encode user-generated content so that browsers don’t mistake it for markup, and execute an untrusted script. This is easy to do for plain text, but what if a page needs to render user-generated HTML? Here’s an example of HTML that contains inline Javascript, which browsers could execute:

This content must be sanitized before rendering. Libraries such as HTMLAgilityPack or DOMPurify provide a way to parse the HTML and strip out elements or attributes known to execute scripts.

Sanitization is important, but what if an attacker has discovered a way around the filter? This is where Content Security Policy comes in.

If the Content-Security-Policy header is present when retrieving the page, and contains a script-src definition, scripts will be blocked unless they match one of the sources specified in the policy. A policy might look something like:

script-src 'self'; object-src 'none'; base-uri 'none';

This policy disallows:

External scripts not hosted on the same domain as the current page.

Inline script elements, such as <script>

Evaluated Javascript, such as <img src="" onerror="alert(0)" />

Base elements, which could break scripts loaded from a relative path

Object elements, which can host interactive content, such as Flash

Whitelisting inline scripts

Sometimes it is necessary to run inline scripts on your page. In these cases, the nonce attribute on script elements can be used to whitelist scripts that you control.

<script nonce="00deadbeef">doSomething()</script>

A matching nonce must be present in the CSP for the script to run. For compatibility with older browsers, unsafe-inline allows scripts to run if the nonce tag is unsupported.

It is critical that this nonce is derived from a cryptographic random number generator so that an attacker can’t guess a future nonce. In .NET, RNGCryptoServiceProvider.GetBytes can be used to fill a 16 byte array:

Whitelisting external scripts

strict-dynamic can be used to allow scripts hosted on a third-party domain to be loaded by scripts that you control. However, at the time of writing, this isn’t supported by all major browsers, so a host whitelist should be used as well as a fallback until it has broad support.

Might return window.location='http://google.com';test({}) in the JSONP response, which would cause arbitrary code to be executed!

There are other policies that you can define to strengthen your site’s security, such as restricting where stylesheets are loaded from. This post only focuses on mitigating cross-site scripting attacks.

Because the method might mutate this, a copy has to be made to ensure the readonly value isn’t modified

Property accessors are instance methods

Every time a property on an in parameter is accessed in CalculateDistance, the compiler
has to defensively create a temporary copy of the parameter.
We’ve now gone from avoiding one copy per argument (at the call site) to three copies per argument
(inside the method body)!

Solution

If we change public struct Point3D to public readonly struct Point3D (the implementation doesn’t
have to change because all fields are already readonly), then the compiler knows it can elide
the temporary copy inside the body of CalculateDistance. This makes the method faster than
passing the structs by value.

Note that we could have achieved the same effect in C# 7.1 by passing the struct by ref. However,
this allows the caller to mutate its fields (if it’s mutable) or reassign the entire variable to a new
value. Using in expresses the intent that the caller will not modify the variable at all (and the
compiler enforces that).

Demonstration

I’ve created a test harness that benchmarks the various
combinations of in, ref, struct and readonly struct. (Note
that I increased the struct size to 56 bytes to make the differences more obvious; smaller structs
may not be impacted as much.) The full benchmark results are in that repo;
the summary is:

Method

Mean

PointByValue

25.09 ns

PointByRef

21.77 ns

PointByIn

34.59 ns

ReadOnlyPointByValue

25.29 ns

ReadOnlyPointByRef

21.78 ns

ReadOnlyPointByIn

21.79 ns

Summary

If you’re using in to express design intent (instead of ref), be aware that there may be a slight performance penalty when passing large structs.

If you’re using in to avoid copies and improve performance, only use it with readonly struct.

]]>
Mon, 30 Oct 2017 09:00:00 +0000
We advise the following best practices for using MySQL Server with .NET code.

General Recommendations

Increase max_allowed_packet

The max_allowed_packet server variable controls the maximum size of a SQL statement that can
be sent to the server, or the largest individual row that can be returned. Before MySQL 8.0, this
defaulted to 4MiB, which is too small for many applications.

In MySQL 8.0, the new default is 64MiB; if you’re running an earlier server version you should set
max_allowed_packet=64M (or higher).

Use MySqlConnector

MySqlConnector is a replacement ADO.NET connector library
(with many contributions from Faithlife developers). It provides true async I/O, has better performance, and
fixes many bugs in
MySQL Connector/NET (aka MySql.Data).

Many stored string values are IDs or tokens, which don’t need case-insensitive comparison

If you have human-readable textual data where you’re certain that you want the database to perform
non-ordinal equality comparison and ordering, then consider using utf8mb4_0900_ai_ci (or decide
if another collation is more appropriate for your use case).

Note: The default character set has changed from latin1 to utf8mb4in MySQL 8.0.

Store bool as TINYINT(1)

In MySQL Server, BOOL is an alias for TINYINT(1).
The MySQL ADO.NET connector understands this convention and will marshal TINYINT(1) back
to managed code as the C# bool type (System.Boolean).

Use the BOOL alias when defining columns in your SQL statements. Do not use BIT(1) (which gets
mapped as a ulong) to represent a Boolean value.

Avoid TINYINT(1)

As a corollary to the above, avoid explicitly using TINYINT(1). If you need a one-byte integer,
use TINYINT (or TINYINT UNSIGNED). The (1) suffix simply indicates the “display width”
(which is typically ignored by .NET programs), not the number of bytes used for storage.

Store Guid as CHAR(36)

To store a Guid, use CHAR(36) [NOT NULL] COLLATE ascii_general_ci.

The MySQL UUID() function returns a GUID in this format. The MySQL ADO.NET connector understands this
convention and will marshal CHAR(36) back to managed code as the .NET Guid type.

The collation is ascii_general_ci because:

Using a one-byte-per-character character set allows MySQL to use fixed-length storage

Index prefix length limits are measured in bytes, not characters

Case-insensitive collation allows GUIDs to be queried using uppercase or lowercase hex digits

If you’re storing millions of GUIDs, you may wish to consider using BINARY(16) and
performing custom data transfer. This will save 20 bytes per value, which may significantly
reduce your data storage needs. Note that for a BINARY(16) column, MySqlDataReader.GetValue
will return a byte[], but MySqlDataReader.GetGuid will reinterpret the value as a Guid.

Connection String Options

CharacterSet=utf8mb4

Uses Unicode when transferring queries to and results from the server, avoiding potential data
loss from encoding errors.

Note: this option is utf8mb4 by default in MySqlConnector

ConnectionReset=True

With connection pooling on, but ConnectionReset=False, temporary tables and session variables from
the last connection to the database will be retained. This is almost never what you want, and can lead
to bugs.

Note: This option is True by default in MySqlConnector

SslMode=None

If you’re OK with the risk of traffic between your web server and MySQL server being intercepted and read, then
setting SslMode=None will use just a plaintext TCP connection. This increases connection speed (less handshaking
to set up the connection) and reduces overhead (of TLS encryption) when transmitting queries & results. The
overhead ranges from 10% (reading thousands of small rows) to 500% (reading a few large BLOBs).

Note also that SslMode=Preferred or SslMode=Required doesn’t really increase security because a MITM
attack can just replace the server’s certificate; you need to specify SslMode=VerifyCA or SslMode=VerifyFull
to ensure a secure connection.

UseAffectedRows=True

Setting UseAffectedRows=True matches the default expectations of most ADO.NET programmers; for
example, DbCommand.ExecuteNonQuery is documented in MSDN
as returning “the number of rows affected”.

Note: This option is True by default in MySqlConnector

]]>
Wed, 02 Aug 2017 16:30:00 +0000
C# 7 introduces local functions. By default,
the compiler generates very efficient code for local functions, even if the argument to the method is captured.

However, there is an edge case in C# 7. If your local function would be implemented with a compiler-generated class (two examples I’ve
found are iterator methods and async methods), then that class will always be allocated when the outer function is invoked, whether it’s used
or not.

If the “early out” condition is taken >90% of the time, and if this function is called frequently, you may wish to avoid the
overhead of the unnecessary allocations of the compiler-generated class. Fortunately, there is a workaround: instead
of implicitly capturing the outer function’s parameter, explicitly alias it:

Now the “happy path” has zero allocations and the compiler-generated class is only instantiated if the local
function is actually called.

I wouldn’t recommend doing this by default, but if profiling indicates that extra allocations are a problem,
you may want to perform this refactoring to avoid them. I strongly recommend checking the compiled IL
to ensure that you’ve solved it. If your outer function has many parameters or locals and (due to a typo)
you inadvertently capture just one of them, then the optimization won’t kick in.

And note that in the typical “argument validation” use case for local functions, there’s no point in doing this
because the outer function should call the local function 100% of the time. (The only reason it wouldn’t
is if a boneheaded exception is thrown because the function was called incorrectly.)
This refactoring is only useful if the local function ends up being called extremely infrequently.

In that specific scenario, the MySQL client library was reading bytes from a TCP socket. Most of the time, the
bytes have already arrived and are in an OS buffer. Thus, we can immediately (and synchronously) return a
ValueTask<int> containing the number of bytes copied into the caller’s buffer. Infrequently, we have to
asynchronously wait on network I/O. Only then do we want the overhead of allocating the compiler-generated
async state machine in order to return a (wrapped) Task<int> to the caller.

]]>
Thu, 30 Mar 2017 12:00:00 +0000
The HttpClient class is used in modern .NET applications to make HTTP requests. It was introduced in .NET 4.5 as a replacement for HttpWebRequest and WebClient.

This post collects some usage guidelines for HttpClient that may not be obvious.

Reuse HttpClient instances

All HttpClient methods for making HTTP requests are thread-safe, so as long as you don’t change any of its properties (BaseAddress, DefaultRequestHeaders, Timeout, etc.) after you start making requests, reusing the same instance on any thread will be fine.

The simplest way to reuse HttpClient instances is to create a static readonly field in the class that needs to make HTTP requests. Of course, that still results in one instance per class, so consider allowing the HttpClient instance to be specified via constructor or method parameter.

HttpClient is disposable, but there’s no harm in never disposing it if it is used for the life of the application.

Dispose HttpRequestMessage and HttpResponseMessage

The SendAsync method is the most flexible way to send an HTTP request with HttpClient. It accepts an HttpRequestMessage and returns an HttpResponseMessage. Note that both classes are disposable; be sure to dispose of instances of both classes when you are finished with them.

Example: FacilityCSharp has tests that run a number of HTTP requests in a row that POST a JSON body and process a JSON result. After a few dozen requests, HttpClient would asynchronously deadlock. I never got to the bottom of it, but I did find a solution: dispose the HTTP requests and responses.

Handle timeouts properly

When an HTTP request times out, an OperationCanceledException exception is thrown, not a TimeoutException.

The default timeout is 100 seconds. If you are using a single HttpClient instance (see above), you’ll want to make sure that HttpClient.Timeout is set as high as any request might need.

Aside:FacilityCSharp does not address this problem, but if you’re a Facility user and experience it, please file an issue and let us know!

]]>
Mon, 03 Jun 2013 15:30:00 +0000
In our last video, we looked at using the async and await keywords in a console application. This week’s video uses async and await in a WPF application.

First we create a simple event handler using async and await. Then we simulate what happens behind-the-scenes with await by implementing the same behavior using continuation tasks. Just like await, we capture the SynchronizationContext and use the Post method to run the continuation on the UI thread.

Next we use DBpedia’s SPARQL endpoint to asynchronously execute a query against its structured data from Wikipedia. We then see what happens when an exception is thrown in an awaited task.

]]>
Tue, 14 May 2013 10:06:00 +0000
By default, ASP.NET uses shadow copying, which “enables assemblies that are used in an application domain to be updated without unloading the application domain.” Basically, it copies your assemblies to a temporary folder and runs your web app from there to avoid locking the assemblies, which would prevent you from updating those assemblies.

This works great for managed DLLs, but not very well for native DLLs. ASP.NET doesn’t copy the native DLLs, so the web app fails with an error that doesn’t even make it obvious that a native DLL is missing or which one it is.

The simplest solution I’ve found is to turn off shadow copying, which causes the DLLs to be loaded directly from the bin directory. This is the strategy now being used in production by Biblia.com. Just add a <hostingEnvironment> element to your web.config:

This also works for local development, but you may find that you need to restart the app pool in order to rebuild the app. Biblia has a pre-build step that runs appcmd stop apppool and a post-build step that runs appcmd start apppool with the corresponding /apppool.name argument.

Alternatively you could consider removing <hostingEnvironment> from your local web.config and putting your bin folder in the PATH system environment variable, but that will be problematic if you have multiple web apps that depend on different builds of the native DLLs.

]]>
Fri, 10 May 2013 15:45:00 +0000
As multi-core processors are quickly becoming ubiquitous, it becomes increasingly important to use parallel and asynchronous programming techniques to create responsive, high-performance applications. The latest .NET releases have responded to this need by introducing the Task Parallel Library (TPL) in .NET 4, and the async/await keywords in C# 5.

We have created a set of fast-paced, code-driven videos on asynchronous programming in C# using TPL and async/await, with a focus on the Task-based Asynchronous Pattern (TAP). If you want a concise introduction to Tasks and async/await, these videos are for you! The videos are under 5 minutes each, and are intended to give a quick overview of each subject. The accompanying blog posts have links for further study.

]]>
Mon, 19 Nov 2012 13:40:00 +0000
As mentioned in my last post (Sharing Code Across Projects),
developers work on the head of the master branch “by convention”. This
is fine for day-to-day work, but we’d like something a little more rigorous
for our continuous integration builds.

For this, we use “build repositories”. A build repository contains a
submodule for each repository required to build the project. (In the App1
example, the App1 build repo would have App1, Framework and Utility
submodules.) The CI server simply gets the most recent commit on the master
branch of the build repo, recursively updates all the submodules, then builds
the code.

The problem now: how is the build repository updated? We solve this using a
tool we developed named Leeroy. (So
named because we use Jenkins as a CI server, and
Leeroy starts the Jenkins builds. We weren’t the first ones to think of
this.)

Leeroy uses the GitHub API on our
GitHub Enterprise instance to watch for
changes to the submodules in a build repo. When it detects one, it creates
a new commit (again, through the GitHub API) that updates that submodule in
the build repo. After committing, it requests the “force build” URL on the
Jenkins server to start a build. Jenkins’ standard git plugin updates the
code to the current commit in each submodule and builds it.

The benefit is that we now have a permanent record of the code included in
each build (by finding the commit in the build repo for that build, then
following the submodules). For significant public releases, we also tag the
build repo and each of the submodules (for convenience).

Posts in the “Building Code at Logos” series:

]]>
Sat, 17 Nov 2012 09:00:00 +0000
We often have common code that we’d like to share across different projects.
(For example, our Utility library is useful in both the desktop software
and in an ASP.NET website.)

One way of sharing code is to place it in its own repository, and add it as
a submodule to all repos
that need it. But submodules are a bit of a pain to work with on a daily
basis (for example, git checkout doesn’t automatically update submodules
when switching branches; you have to remember to do this every time, or
create an alias).

Submodules also make it difficult to “compose” libraries. For example, App1
and App2 might both use Utility, but they might also both use Framework,
a desktop application framework that’s not general-purpose enough to live in
Utility, but is in its own repo. If Framework itself uses Utility as a
submodule, then the App1 and App2 repos might contain both /ext/Utility
and /ext/Framework/ext/Utility. This is a maintenance nightmare.

Our choice at Logos is to clone all necessary repositories as siblings of
each other. In the App1 example above, we might have C:\Code\App1,
C:\Code\Framework and C:\Code\Utility as independent repos. Dependencies
are expressed as relative paths that reference files outside the current
repo, e.g., ..\..\..\Utility\src\Utility.csproj. We’ve written a shell script
that clones all necessary repos (for a new developer) or updates all
subfolders of C:\Code (to get the latest code).

By convention, developers are working on the master branch on each repo
(or possibly a feature branch in one or more repos for a complex feature).
It’s theoretically possible for someone to push a breaking change to
Utility and forget to push the corresponding change to App1 (a problem
that submodules do prevent), but this happens very infrequently.

Posts in the “Building Code at Logos” series:

]]>
Fri, 16 Nov 2012 09:50:00 +0000
Some of our repositories reference third-party code. In many cases,
this can be managed using NuGet, but sometimes
we need to make private modifications and build from source.

We accomplish this by creating a repository for the third-party code. In some cases, this repository is added as a submodule under ext in the repositories that need it; in other cases, the binaries created from the code are committed to another repository’s lib folder. The decision depends on how complicated it is to build the code versus how useful it is for developers to have the source (and not just precompiled binaries).

In the third-party repository, the upstream branch contains the unmodified upstream code, while the master branch contains the Logos-specific modifications.

When cloning the repository locally, the origin remote refers to our repository containing the third-party code. If the original third-party code is available via git, then we add an upstream remote that references the original maintainer’s code.

Example: Creating a ThirdParty repo from source on GitHub

# clone a local copy of the remote third-party repository
gitclonehttps://github.com/user/ExampleProject.gitcdExampleProject# rename the "origin" remote (created by clone) to "upstream"
gitremoterenameoriginupstream# add Logos' repo as the "origin" remote
gitremoteaddorigingit@git:ThirdParty/ExampleProject.git# use this code as the "upstream" branch
gitcheckout-bupstreamgitpushoriginupstream# work on Logos-specific modifications
gitcheckoutmaster**makemodificationsgitcommit-am"Some important changes."# push the changes to our repo
gitpushoriginmaster

Example: Creating a ThirdParty repo from source in Subversion

# create the git repo
mkdirExampleProjectcdExampleProjectgitinit# add Logos' repo as the "origin" remote
gitremoteaddorigingit@git:ThirdParty/ExampleProject.git# seed it with the upstream code
svnexport--forcehttp://source.example.org/repos/example/tags/1.0.# add all the code
gitadd-Agitcommit-m"Add Example 1.0"# use this code as the "upstream" branch
gitcheckout-bupstreamgitpushoriginupstream# work on Logos-specific modifications
gitcheckoutmaster**makemodificationsgitcommit-am"Some important changes."# push the changes to our repo
gitpushoriginmaster

Once the repository is created, we will want to update it with new versions of the third-party code when they are released (then merge in our changes).

The new code gets committed to the upstream branch, then that gets merged into master. If necessary, conflicts are resolved, or our changes are edited/removed to reflect changes in the upstream code.

Example: Updating a ThirdParty repository from source on GitHub

# switch to the "upstream" branch, which contains the latest external code
gitcheckoutupstream# get the latest code from the "master" branch in the "upstream" repo
gitpullupstreammaster# switch to our local "master" branch, which contains Logos changes
gitcheckoutmaster# merge in the latest upstream code
gitmergeupstream**fixanyconflicts,andcommitifnecessary# push the latest merged code to our repo
gitpushoriginmaster

Example: Updating a ThirdParty repository from source in Subversion

# switch to the "upstream" branch, which contains the latest external code
gitcheckoutupstream**deleteallfilesintheworkingcopy,exceptthe'.git'directory# get the latest version of the third-party code
svnexport--forcehttp://source.example.org/repos/example/tags/1.1.# add all files in the working copy, then commit them
gitadd-Agitcommit-m"Update to Example 1.1."# switch to our local "master" branch, which contains Logos changes
gitcheckoutmaster# merge in the latest upstream code
gitmergeupstream**fixanyconflicts,andcommitifnecessary# push the latest merged code to our repo
gitpushoriginmaster

At the root of the repository, we have the Visual Studio solution file (used
by developers) and the build script (used by the build server). This might be
a NAnt build file, a
psake build script, a shell script, or
something similar.

The src and tests folders contain the bulk of the code we write; this is
the code that gets shipped to users or deployed to a web server, and tests
that get run by the build server.

ext, lib, and packages contain third-party code (or sometimes Logos code
that is consumed as a precompiled binary, rather than as source). Folders
under ext are git submodules that reference third-party repos. lib
contains pre-compiled DLLs and static libraries. packages is reserved for
use by NuGet. (This folder is added to .gitignore
if this repo uses NuGet Package Restore.)

build is never committed, but is reserved to contain build output. tools
contains code that is required to build the project, but doesn’t get shipped.
This would include things like NUnit (or another test
framework), StyleCop plugins, NAnt
extensions, mocking frameworks, etc.