Posted in WordPress on April 9th, 2014 by Richard – Comments Off on WordPress Updating Under Windows & IIS

The Common Error

…would be something like:

Warning: copy(some local path/wp-admin/includes/update-core.php) [function.copy]:
failed to open stream: Permission denied in
the same local path\wp-admin\includes\class-wp-filesystem-direct.php on line 131

(a) file ownership: all of your WordPress files must be owned by the user under which your web server executes.

[…]

(b) file permissions: all of your WordPress files must be either owner writable by, or group writable by, the user under which your Apache server executes.

Of course that "under which your Apache server executes" is the give-away. Under a
different web server things are likely to be different.

Permissions on IIS 7 & 7.5

Typically this means running WordPress under PHP using fast-cgi with the Application Pool
(AppPool) running under “Application Pool Identity”.

But because of the way the AppPool identities are implemented, just giving that identity
write and delete permissions on the WordPress folder (and all of its contents,
recursively) is not enough. (To assign permissions to an AppPool identity on Windows
Server 2008 R2 (and later) use the name IIS APPPOOL\name (which is a local
account), on Server 2008 you need to use the command line).

You also need to give the local account IUSR the same permissions.

And suddenly updates start working. Of course to completely test this I need
a new update to the WordPress code... but 3.9 will hopefully be along shortly.

One final note: this blog runs on Apache, but I run a local test instance on Windows
for testing. And, eventually, my own theme (because I've yet to find anything that
really suits me :-) ).

Updated: WordPress 3.8.3

Helpfully a new WordPress has jjust been released, thus able to test
this with more than a theme or plugin update. And it just worked:
updated from 3.8.2 to 3.8.3 without problem.

Configuration Inheritance: There Are Multiple Configuration Files

Introduction

Note, the contents of the post covers non-ASP.NET applications. ASP.NET uses its own
approach to configuration inheritance, and this article already contains
more than enough information.

Also note, I call this configuration inheritance because it
expresses the idea of one thing overriding another. Given the limited conceptual
documentation about .NET’s configuration system I don’t think there is
an official term. Until there is I will stick with “inheritance”.

When one uses the .NET configuration APIs to load settings into an application
there is a rich structure supporting not just the kind of data that can be held,
but also the scope of that data. Different applications can share data at the
machine level, different AppDomains in the same process can have different
configuration, and users can have their own settings either in the roaming part of
their profile or purely local, or both.

Where this data is stored, and what influences where is the subject of this
posting, but on the way there is a little coverage of how to get it and
manage it but mostly that will be for the future.

The Four Levels

For non-ASP.NET there are four files, with each able—if the
section allows it—to override the previous level. Some times of
collection can also be modified by subsequence levels: clearing
the collection, adding or removing members. The four levels are

Machine

Application

User Roaming

User Local

The Machine Configuration Level

This is the global file, normally stored in
%SystemRoot%\Microsoft.NET\Framework64\‹Version›\Config\machine.config
for 64bit .NET or
%SystemRoot%\Microsoft.NET\Framework\‹Version›\Config\machine.config
for 32bit. Where ‹Version› is the full version, e.g. v4.0.30319
for .NET 4.

It is instructive to read through this file—if you have enabled local active content in
Internet Explorer the ability to collapse elements makes this easier—because you will
see all the standard sections and section groups defined with the classes that implement them.
For instance <appSettings> is implemented by
System.Configuration.AppSettingsSection in assembly System.Configuration. It
is also in the <connectionStrings> section at the machine level that
connection “LocalSqlServer” is defined to be SQL Express using data file
…\App_Data\aspnetdb.mdf which is then used as the store for
ASP.NET profiles in their default configuration. (ASP.NET also has a global configuration
“web.config” in the same folder with yet more global settings.)

As this file is read, by the configuration system, first any of the other levels can
override it, unless the section
is defined with allowDefinition or allowExeDefinition attribute
is set to “MachineOnly” to ensure it can only be used at the machine level.

The Application Level

This is the most familiar level, creating a file called “app.config” in
a Visual Studio project will create a file
“‹assembly-name›.config”
in the project output directory.

User Roaming and Local Levels

If you set things up correctly, per user files can also be used. The roaming
file will be part of the user’s roaming profile, shared across machines if
roaming is enabled. The local file will stay local. But you can override
the filenames (see below) and as the
content of a user’s roaming profile is based
on a set of configurable folders, these names are strictly speaking indicative rather
than absolute.

User levels are most useful when the ability to save configuration content
is enabled. Rather than storing user settings and options in the registry
use these configuration files.

The only problem with these files is the default naming, it is designed to
ensure different versions of the same thing running side by side have
separate configuration. If the location, assembly version or signing key
changes then the filenames will be changed and any previous content (e.g.
from before an upgrade) will be lost. This is bad news. The good news is
that it is easy to specify your own filenames—and I would suggest
doing that in most cases.

Using The User Levels

If you just use the static properties of ConfigurationManager (or
one of the other configuration entry points) then machine and application
configuration files are included, but user files are not.

To access either user-roaming, or user-roaming and user-local levels (you cannot
use the local level without the roaming level—the roaming level could be left empty)
one just uses one of the
static ConfigurationManager methods that take a
ConfigurationUserLevel parameter. The ConfigurationUserLevel
enumeration allows selection of no user levels: ConfigurationUserLevel.None,
just roaming: ConfigurationUserLevel.PerUserRaoming, or
all levels: ConfigurationUserLevel.PerUserRaomingAndLocal.

If the configuration is opened with the
ConfigurationManager.OpenExeConfiguration
method then the default filenames, created by the configuration system, are
used. Alternately using ConfigurationManager.OpenMappedExeConfiguration
allows the filenames to be specified (more on this below, after covering the default
filenames).

What Are The Default Filenames?

Disclaimer: none of this, as far as I can determine, is formally documented. Treat
this information as subject to change in a future .NET version.

Default Filenames For The Default AppDomain of a Unsigned Executable

This is the simplest, and likely starting, case. The application configuration
is the starting assembly’s path with “.config” appended.
The user files will be:

companyName: Is the value of the AssemblyCompanyAttribute if
set. If not then the value of the AssemblyProductAttribute.
If neither is set then this path fragment is not included (one less
directory in the path).

AssemblyBase: is the first part of the assembly’s filename (including the
first period and two further characters).

hash: is an alphanumeric string created by hashing the full path of the assembly
in some implementation determined way.

version: is the value of the AssemblyVersionAttrinbute.

With Code Signing

If the assembly is signed then there is a small change. Rather than
‹AssemblyBase›_Url_‹hash›
we have
‹AssemblyBase›_StrongName_‹hash›
with the hash being based on the signing key (the rest of the path
is the same). This means by signing the assembly
the location of the user configuration files becomes independent of the location of
the assembly. If you want xcopy deployment then signing would
seem to be a good idea to allow the user to relocate the application while
preserving their configuration.

With Other AppDomains

If the configuration is loaded from a different AppDomain the machine and
application configuration files remain the same. However the user levels file names change.
In the above path AssemblyBase is replaced by the AppDomain name.
Another case of trying to ensure uniqueness.

One of the options on creating an AppDomain is to specify a different configuration
file:

In this case replacing the AssemblyName.config with
AppDomain.config. This results in the assembly level configuration
file being the filename specified, but no other levels are changed. Specifically
the user level filenames remain the same whatever application level configuration
is specified.

Note, if you don’t give an AppDomain a name on creation, .NET will:
there are no unnamed AppDomains.

Defining You Own Filenames

As noted above the default naming is not ideal (if you want to be
able to revise the application) except in being confident of its uniqueness.
You can however define all the filenames, except at machine level, yourself.

To specify your own filenames you need to specify all the levels you are going
to use, except the machine level. This includes the application level (to use the
default just use AppDomain.CurrentDomain.SetupInformation.ConfigurationFile
(this seems to be always populated whether not specified on AppDomain creation or
in the default AppDomain) or Assembly.GetEntryAssembly().Location + ".config".

The real benefit is being able to specify you own path, to be independent of
assembly/AppDomain name, assembly version, signing key and assembly location. Just
construct the path you need, and remember with a new version you need to
ensure that structural changes are either backwards compatible, or you include
conversion code. Something like:

This can be called with each of Environment.SpecialFolder.ApplicationData
and Environment.SpecialFolder.LocalApplicationData to get the current
users roaming and local profile root folders (respectively). (The GetAttribute<T>()
method is a helper extension method over Assembly I have which gets
the single instance of attribute T from the assembly.) Normally this would
be passed the entry assembly, but for an extension could be passed the extension‘s
assembly to keep configuration separate (and thus avoid conflicts) from the containing
application.

Once you have the filenames you want to use, populate an instance of
ExeConfigurationFileMap and pass to
ConfigurationManager.OpenMappedExeConfiguration to get
a Configuration instance, which is used in exactly the
same way as an instance obtained any other way.

Discovering the Names Being Used At Runtime

There are two approaches: defined and laborious, or using reflection to
peer inside the Configuration instance. The safe method
is based on Configuration.FilePath being the path
of the lowest level of configuration in use. So open with different
ConfigurationUserLevel values and get each path in turn.

The reflection method relies on Configuration having
private member _configRecord of private type
System.Configuration.MgmtConfigurationRecord. This
is a linked list of MgmtConfigurationRecord instances, one
per level. Each has a _configName field which gives the
level as a string (“MACHINE”, “EXE”, …) giving
the level, and a ConfigurationFilePath property with the path.
Given an instance of Configuration and a few helpers a
populated ExeConfigurationFileMap can be returned.

Reference

In the posting:
“Client Settings FAQ”
Raghavendra Prabhu covers some details of the user-roaming and user-local configuration levels.
Including that the hash following the _Url_ or _StrongName_
is based on the evidence—i.e. of the URL or signing key.

Configuration Sections

Introduction

There are two remaining approaches to providing structure that I have not yet
covered: collections of elements and section groups. Element collections will
be a topic for several future posts as there are a number of different approaches
to creating element collections, and two of them make little sense without
first covering configuration inheritance
(see part 5).

Section groups are much simpler, and mostly a matter of declaring them.

Creating Section Groups

When custom sections are declared using the <configSections>
element of a configuration file, one can declare both sections (with the
<sections> element, using attributes to define the associated
custom type and its name) and section groups.

To define a section group use the—unsurprisingly—<sectionsGroup>
element. This needs a name and a type. Often the type will be
System.Configuration.ConfigurationSectionGroup which adds no behaviour
(but one can customise this, more below) and just allows programmatic access to
its child sections and section groups (there is no specified limit to
the depth of structure you can define—but don’t go overboard with
deep structures which will become increasingly hard to understand and maintain).

Which defines a userSettings element to act as a container for (presumably)
multiple sections describing many aspects of user behaviour.

Note, replace the strong name for System.Configuration with
“System.Configuration.ConfigurationSectionGroup, System.Configuration,
Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”
for .NET versions 2, 3 and 3.5.

Reading Section Groups and Contained Sections

In previous parts of this series the application code used
Configuration.GetSection(‹name›) to get
to its custom section (and the instance of Configuration was obtained
with ConfigurationManager.OpenExeConfiguration).
This works for configuration sections that are immediate children of the root
<configuration> node.

To get a section defined with a group simple get the group, and open the
section from within that group. For instance, based on the above sample
configuration:

GetSectionGroup returns a ConfigurationSectionGroup instance
which has GetSection and GetSectionGroup methods as
Configuration has.

Configuration Section and Group Collection Properties

You might notice that the Configuration and
ConfigurationSectionGroup also have properties
Sections and SectionGroups which
can be indexed by name. These give another approach to getting
a section or section group:

Custom Section Group Types

It is also possible to create your own custom configuration section
group by deriving from ConfigurationSectionGroup, and
a number of .NET Framework types do this. E.g.
System.Xml.Serialization.Configuration.SerializationSectionGroup
defined in the System.Xml assembly, which adds properties
to type safely access contained sections (i.e. the properties
contain the access into the Sections collection
and cast the results to the correct type).

Validating Values

Introduction

Parts 1 and
2 added structure and
data, but this is often not enough, one needs valid data—the
domain of the data for your configuration being narrower than the
domain of the underlying type of the parsed values. Hence the need to extend the custom
types representing the structure with more types and attributes to limit the
ranges of those values.

Value Validation

Checking single values is the easiest part, albeit needing two custom types
to be defined for each distinct kind of validation, but at least each case
can be parameterised (i.e. if you need to check a date range, you only
need two types, not four).

First: The Validator

The first type is derived from System.Configuration.ConfigurationValidatorBase.
This will actually do the validation in two steps:

Overriding CanValidate to confirm that the type can be validated.
Return true to indicate you can, otherwise validation will fail.
This override is usually used to confirm the type of the value matches the
expectations of the next method.

Override Validate to perform the validation. To indicate failure
throw an exception (which will be wrapped by the configuration runtime, into
a ConfigurationErrorsException.

NB. the validation needs to handle the default (before the configuration
content is parsed) value. E.g. a DateTime value will be initially
validated with a value of DateTime.MinValue before being called
again with the value read from the configuration file. (For types which can
be used with code attributes—see §17.1.3 of the C# specification,
the default value is an optional parameter for the ConfigurastionProperty
attribute. For other types, the code attribute based declaration of configuration
values can be replaced with a more programmatic one, which I should cover later
in this series.)

For instance to check that a DateTime value has a minimum year the
following validation code will work (note the check to allow MinValue):

Second: The Attribute

The second type is derived from ConfigurationValidatorAttribute
(which itself derives from System.Attribute). This is used to
(1) annotate the configuration property in the ConfigurationElement (or
ConfigurationSection) type, and (2) be an object factory for the first,
validation, type. Additional parameters can be passed from this attribute to
the validator.

The key override is ValidatorInstance which needs to return an
initialised instance of the validator class.

Note the use of an explicit lower limit for StartDate but not for
EndDate. (DateTime is not a type with literals
available for attribute parameters, hence just using the year here.)

What Happens on Validation Failure?

Validation is performed when the configuration section is read, i.e. when
Configuration.GetSection(‹name›) is
called. Importantly this means that if configuration sections are not used,
validation will not be performed, and no errors will be reported, however many
invalid, missing or extra values there are. Also note that extra attributes
and elements will be reported as errors, this behaviour can be modified by
overriding ConfigurationElement’s
OnDeserializeUnrecognizedAttribute or
OnDeserializeUnrecognizedElement.

When an error, validation or otherwise (including malformed XML) occurs a
ConfigurationErrorsException is thrown. If this is based
on another exception being thrown internally (e.g. on malformed XML, a
XmlException) then that original exception may be the
ConfigurationErrorsException’s InnerException
(sometimes this seems to be the base, other times not—I don’t see
much consistency).

Any reporting of this to the user or logging is up to the application, this is
not trivial as while the text of the message does include key information
(like where the error was in the configuration file) it is not exactly in
a user friendly format. But then configuration files are not targeted at
(typical) end user direct editing.

Checking Values Together

There is no direct support for further validation after
each individual value has been loaded and (given suitable attributes)
validated. But ConfigurationElement does have the
PostDeserialize method, which is called at the right time.

But the error message is not ideal, starting with the text:
“An error occurred creating the configuration section handler for
‹section-name›”. But it does work.

Allowing Richer Structures by Adding Custom Elements

Introduction

Part 1 kept things really
simple. But for non-trivial custom configuration just having XML attributes to
work with starts to get very hard to edit in the file. Rather one wants to
employ the richer hierarchical structures that XML allows: child elements.
This turns out to be very easy.

Creating A Custom Element

To keep things simple, this example will just change part 1’s structure:

<custom date="2010-05-13" name="element name" />

into this:

<custom>
<data date="2010-05-21" name="custom data name"/>
</custom>

All this takes is deriving a new class from System.Configuration.ConfigurationElement
with code attributes for the two configuration attributes, and then using this new
type as the type of the code property in the ConfigurationSection derived
class. See the source code below: DataConfigurationElement
is the new custom element, and AppConfigSection is the updated custom section,
now with just a single code property.

One limitation of this use of attributes is that text content is not supported
(<element>text content</element>), custom elements still
just use attributes, or child custom elements (there does not seem to be any
depth limit). Unless one completely overrides the de-serialisation code
that makes custom elements easy (by overriding the
ConfigurationElement.DeserializeElement
method), but then all the work—including parsing the attributes—would
have to be manually coded using the passed XmlReader instance.

Previous Part of This Series on .NET Custom Configuration

A Simple Starting Point: Single Element with Attributes

Introduction

The <appSettings> element of a .NET
.config with the ConfigurationSettings.AppSettings
static collection to read them make adding a few simple string configuration
settings to an application easy.
But, limited to simple name-value pairs, this is (to put it nicely) limited.
And then one compares it to the richness of structure (for example) the CLR
and ASP.NET support.
However all that capability to create rich hierarchical structures, including
multiple types of collections with all values validated is also open to
the developer.

It is just not obvious where to start (at least the documentation for .NET 4,
e.g. here,
is a big improvement over .NET 2's).

Hence this series of posts which aim to be an aide-mémoire of various
scenarios for different types of .config structures.

Keeping It Simple: A Single Element

How to add this

<custom date="2010-05-13" name="some text" />

to the application (or ASP.NET) configuration file?

Note: both XML and C# code have attributes, so I’ll write
configuration attribute for XML attributes on the config file, and
code attribute for those in the source code.

Setting Up The Project

Any kind of .NET project can be used, I’m using
C# in a console project here (so I can just focus on configuration).
This will also work against a .NET 2, 3, 3.5 or 4
target. The sample here is a console application called
ConfigurationDemoPart1, so that will also
be the namespace.

Start by adding a reference to the System.Configuration
assembly.

Then add a new public class, for this sample called AppConfigSection.
Derive from System.Configuration.ConfigurationSection. This base
is needed for any custom type that is referenced by the
<configSections> to define the configuration file section
being created.

Defining Attributes on the Configuration Element

To create each configuration attribute to appear on the custom element add a
property to the section class.

Each property needs a System.Configuration.ConfigurationPropertyAttribute
code attribute with the name of the configuration attribute, and must use
the default dictionary inherited from ConfigurationSection as a backing store: do
not use automatic properties or a local field. The key name used in the
dictionary must match the name in the ConfigurationProperty.

Accessing the Custom Configuration

Without this the whole effort would be pointless, why write configuration
file content if you are not going to use it?

Create an instance of System.Configuration.Configuration by using
the OpenExeConfiguration method of
System.Configuration.ConfigurationManager. Pass
System.Configuration.ConfigurationUserLevel.None to just
access the application configuration file (or web.config).
Per-user configuration files, and directly loading a specific file (i.e.
not the default name) is also possible, but that’s something for
later.

Get to the custom configuration with GetSection(name)
and casting to the custom configuration type. The properties will have the
values from the configuration file.

You can also set the value of the properties. While this will change the
value subsequent reads of the properties returns it will not change the
configuration file. It is possible to save changes, but that’s a topic
for a later post.

Adding To The Configuration File

As well as adding the new custom element, the configuration system
needs to be told about the new element. Do this with the
<configSections> element, giving the new section
a name (which you also need to pass, in code, to GetSection)
and a type:

Possible Errors

A null reference on accessing a property in code to get a configuration
attribute’s value probably means one of the code attribute on
the custom configuration property, the index into the values dictionary
or the configuration attribute is spelt differently. (This took ages to
work out the first time it happened to me, now it is always the first
thing to check.)

Posted in .NET Futures, Rx on March 18th, 2010 by Richard – Comments Off on Creating a Reusable “ObservableSource” for the Reactive Extensions (part 6.1)

Preparing For Implementing Concurrency

Introduction

The next step in this series on creating my own
IObservable<T> implementation is concurrency.
(See below for links to previous parts,
Part 5
has a full code listing.)

This is the hard part. Too easy to think it is OK when it isn’t, and
too hard to validate that is really thread safe.

The only option is careful analysis, and then double (triple, …) check
everything.

After all even the best set of unit tests will fail to find problems if
it takes a particular combination of OS version, hardware and other
processes running to show the problem. With race conditions, deadlocks
and all the other concurrency fun such is not just a possibility
but likely.

But first, the new 1.0.2350.0 (2010-03-15) build of Rx makes
no change to my code, all tests pass without any changes. And this
is now running on my main system (Rx has go-live licence, so why not)
with its multiple codes (previously was working on a VM with a single
virtual CPU).

What Needs Protection?

At the heart of my implementation there is some shared data. This needs
protecting where multiple threads could operate concurrently. E.g. while
events are pushed to subscribers using a copy of the list of subscribers
(this allows re-entrant subscription and un-subscription) taking a copy of
the values of a Dictionary<K,V> certainly is not
safe.

Some operations are inherently safe (defined by the platforms object model,
see Joe Duffy’s blog
for more details). In this case reading or writing an aligned, volatile
int or reference is safe (such reads and writes are inherently
atomic). And only a single field needs to be checked, written to, or both
compared & exchanged (via Interlocked.CompareExchange):

Checking if the endOfEvents field (reference to a
Notification<T>) is null.

The boolean field pushingEvents used only in PushEvents
(the method that loops through a snapshot of the current subscribers) to avoid
re-entrant or multi-threaded pushing of events.

When it is read, and true in a simple check there is no problem
(just return from PushEvents. But if it is false it then
needs to be set. So two operations are needed: a read and a write.
Interlocked.CompareExchange will take care of this without
a lock. (Before returning from pushing events it is set back to false,
but a single write is also OK.)

As there is no overload of Interlocked.CompareExchange for
bool the field will need to be an int with
values 0 and 1.

To avoid races in Next, Completed
and Error between checking for endOfEvents
and setting it one can use Interlocked.CompareExchange.

There are also two pieces of functionality that require multiple operations
on fields (or objects that are not thread safe). In each case a lock is needed.

The collection of subscribers and subscriber id counter. With operations
to add or remove a subscriber, and to snapshot a list of all subscribers.
This will require a lock, there is no way (I can see) around this.

To operate on the queue of pending events can be handled by changing to
an instance of ConcurrentQueue<T> with modifications
in PushEvents to use TryDequeue rather than a
combination of checking the count and then Dequeue.

What Doesn’t Need Protection

This implementation is not about being fair. So if two threads
both try and use an instance together, with one calling
Next and the other Completed only
the Windows thread scheduler gets any say in which wins (if
Completed is called first, no subscriber will see
the value passed to Next).

Ready to Implement?

Even as I wrote the analysis above (which was already the second pass through
what would be needed) I noted that I could avoid a lock for Next,
Completed, and Error and use CompareExchange.
So I’ve probably still missed something: time for another pass over the code.

And another question is how to test…? Throwing a load of cocurrent tasks
from the Parallel Extensions (see here)
will be a start, but cannot really cover mixing in Completed and
Error because I cannot determine which thread will win and which
will not, and thus no way to verify the “right one” won in a unit
test assert.

Posted in .NET Futures, Rx on March 9th, 2010 by Richard – Comments Off on Creating a Reusable “ObservableSource” for the Reactive Extensions (part 5a)

Updates for Next Reactive Extensions (Rx) Code Drop

The new code drop came through, 2½ months after the last drop, on 5th March. The
release notes
are suitably long. With a fair amount of change, much of it under the covers to edge cases.

But a fair number of things have moved assemblies and changed name.

Impact on My Implementation

Given all the change, there was relatively little impact. The only thing that broke was
my use of Notification<T>: the Current
property is now called Value. And it now has an Exception
property, which saves a cast when handling an OnError notification.

My (unpublished) test harness used GroupDisposable to clean up
subscriptions, this was renamed CompositeDisposable and moved to
the CoreEx assembly.

Both of these renames give slightly better and more consistent names, and the addition
of the Exception property is definitely better. So overall the new drop is,
for me, an improvement so far: once the code compiled all tests just passed.

The only function that changed was SendNotification so following DRY helped.
The new implementation:

So Where’s The Concurrency Support…?

Coming! :-)
I've been rather head down in learning WPF for something else, and trying to avoid
too many distractions while getting my head around the basics of WPF. Concurrency
support is perhaps the biggest challenge here and I therefore want to be able to focus
on it and get it right. This will mean taking a block of focused time.

Posted in .NET Futures, Rx on February 24th, 2010 by Richard – Comments Off on Creating a Reusable “ObservableSource” for the Reactive Extensions (part 5)

Adding Completed and Error Methods

Introduction

So far in this series (links below) the focus has been on implementing
Next and Subscribe in a way that is re-entrant.
But that leaves two methods uncovered. Both Completed and
Error end the sequence, but in different ways.
Error passes an exception indicating the source has failed,
whereas Completed just indicates it has ended.

Based on Subject<T> when a sequence of events is ended:

Existing subscribers received a call to their OnError
or OnCompleted (as applicable).

Further calls to any of Subject<T>'s
IObserver<T> methods is a no-op.

Any new subscribers immediately get a call to their OnError
or OnCompleted (as applicable) and nothing more.

Previously

Notification<T> As A Way to Store An Observable Sequence

The reactive Framework (Rx) distributable comes with three assemblies:

System.CoreEx

System.Reactive

System.Interactive

System.CoreEx contains types that would have been in System or System.Core
had .NET been designed with Rx in it from the start. System.Interactive
extends LINQ to Objects by back-porting many of IObservable<T>'s
extension methods (or ’operators”) to IEnumerable<T>.
System.Reactive implements LINQ like operators for IObservable<T>.

My observable implementation almost doesn’t need any of these
three assemblies (IObservable<T> and IObserver<T>
being defined in the mscorlib assembly).

However one rather useful group of types defined in the System.CoreEx
assembly in namespace System.Collections.Generic are Notification<T>
and its subtypes. These types allow the “value” of an observable event
to be stored. Using one of the three subtypes (one each for OnNext,
OnCompleted and OnError) defined as members of
Notification<T> (hence created as, for example,
new Notification<int>.OnNext(1)). Therefore a collection
of events covering all three IObserver<T> methods can be
easily created without some custom helper types.

These types are provided for the Materialize and
Dematerialise enumerable and observable operators (the former being
one the back-ported operators in System.Interactive).

Overview of The Implementation of Completed and OnError

Both follow the same pattern as the existing Next implementation:

If ended, return

Add the new event to a queue of pending events.

If not already pushing an event, process the queue.

The first of these steps is new, and handles the requirement that either
Completed or OnError ends the sequence.

To make handling new subscriptions after the end of the sequence
the endOfSequnec field is a
Notification<T> reference. If null the sequence has not
ended, otherwise this is the event to immediately push. Also a helper
from System.Reactive is used as the return value:
System.Disposables.Disposable.Empty which is an instance of
a type which implements IDisposable as a no-op and these is
no need to track this after the end subscriber (it will never be called
again).

The New Implementation

The single biggest change to support storing notifications (rather than
values of type T>) is in PushEvents where
different kinds of notifications need to call different
IObserver<T> methods, so a switch is needed. This is
in its own method (SendNotification) because it can then
be reused to send the ending event to post-end subscribers.

The implementations of Next, Completed and
Error end up being very similar and simple.

Overall, compared to the code from
part 4, the
implementation is a somewhat longer, but certainly better factored with
each method doing one thing. The most complex (PushEvents)
does have to loop over both pending events and subscribers, but that
is all it does.

Further Steps

The toughest part to get right is still left: concurrency. This will be
difficult as much to test as to implement, tests using workers are never
simple and just being confident that they are doing things concurrently
is hard (after all a simple Interlocked.Increment might
be slow enough to effectively eliminate any concurrency due to forcing
CPU cache synchronisation even with multiple idle cores).

Posted in .NET Futures, Rx on February 22nd, 2010 by Richard – Comments Off on Ractive Framework’s Observer.Create Doesn’t Create a Pure Wrapper

What Happens When One’s Test Doesn’t Fail When It Should, or, Why
Observer.Create<T> Isn’t Good For Testing Observables

Introduction

In building my own implementation of IObservable<T>
across a series of posts
(Part 1,
Part 2,
Part 3 and
Part 4)
I’ve been making heavy use of Observer.Create<T> to create
helper observer instances to ensure that the right methods of the
subscribed observers are called, the right number of times (and
even—although somewhat harder to track in all but the
simplest cases—the right order).

The Discovery

In working towards Part 5 which will cover implementing the two,
so far missing, methods of IObserver<T>:
OnCompleted and OnError I was rather surprise
when this test passed:

When I knew that Observable4<int>.Completed had no
logic to prevent IObserver<T>.OnComplete being called
multiple times. Under the debugger it was quite clear that the
third lambda was not being called (briefly I considered a debugger bug,
but decided to check more fully first).

I looked at the implementation of Observer.Create<T> in
Reflector. That factory method
creates an instance of an internal type, which derives from another internal
type AbstractObserver<T>. So far largely as expected.

But in looking at the implementation of the IObserver<T>
methods I saw:

I.e. it keeps track of a call to OnCompleted and blocks
further events. All three IObserver<T> methods check
for IsStopped, and it is also set in OnError.

The Implication

If I want an observer that I can use to test an observable for compliance
to the IObserver<T> semantic contract, I cannot use
Observer.Create<T>. It will hide breaking the
“OnCompleted or OnError end the
observable’s event sequence”.

The Solution

In the end the fix is rather easy: make my own
Observer.Create<T> which doesn’t contain any such
logic. This is rather simple: