Monday, February 6, 2012

Building the Platform Information
This is the final part of this series and we will discuss how to create the platform information using the illustrative samples from the previous article.
The platform information simply provides additional metadata to the build system, making is possible to display extra information about the API. You will see this extra information as platform icons displayed by the API, showing which API is supported by a given platform.
Note that this feature is not currently customizable, that is it applies only to MSDN defined customization. With the current MSDN release, there are two main framework groups or version frameworks; the .NET Framework and the Silverlight Framework, under these various platforms are defined as outlined below:

.NET Compact Framework: Not available in newer versions of the MSDN.Versions: 3.0, 2.0, 1.0.

XNA Framework: Icons are displayed by newer MSDN, but versions are displayed only by the older MSDN.Versions: 3.0, 2.0, 1.0

Portable Class Library: No platform and version identifiers are defined by the Sandcastle.Version: Portable Class Library.

Silverlight Framework Group

Silverlight (main platform): No icon supported.Versions: 5.0, 4.0. 3.0 (earlier versions are not available in MSDN).

Silverlight for Windows Phone: Sandcastle does not current define version identifier for the Windows Phone OS 7.1, but could be easily added.Versions: Windows Phone OS 7.1, Windows OS 7.0

XNA Framework: The icon supported is the same was that under the .NET Framework by the Sandcastle. However, the MSDN currently displays different icon for this.Versions: XBox 360, Windows Phone OS 7.0

Clearly, the newer platforms (Portable Class Library, .NET Framework Client Profile etc) and versions (.NET Framework 4.5, Silverlight 5.0 etc) are not supported by the current release of the Sandcastle. Versions identifiers (IDs) are easy to extrapolate (even though some inconsistency exists in the naming), but platform identifiers (IDs) are not.

Also, the current MSDN elevated Silverlight to its own framework group, but is not well supported by the current Sandcastle, since at the time of release, no Silverlight documentation existed in the MSDN.

These issues can be fixed if required, and we will be looking into them as part of the Sandcastle Assist release issues.

Platform Builder

The platform metadata is defined in a file specifying the platform (marked by the platform ID) under which each API in a documentation is supported.

Sandcastle is open source project developed internally by Microsoft, but the sources for some tools such as the VersionBuilder.exe are not released. The platform file format and the tool for building it are also not released.

The information we are presenting here are obtained from the study of the platform build component, transformation styles and the resource contents. We take the responsibility for any error contained in this article.

Platform File

From the study, this is the format of the platform file we came out with, it is an XML file:

The following points are not obvious from the format as presented above:

Multiple Platform IDsYou can have multiple platform IDs for a specific version. The multiple platform IDs are separated by comma (,).

Multiple Versions IDsSimilar to the multiple platform IDs, you can have multiple version IDs for a particular platform. The multiple version IDs are separated by comma (,).

Except to keep it compact, using multiple platform IDs or version IDs does not make any special difference in the file format, since these are split up by the build component that consumes this file. We have not found any use of the multiple version IDs. However, for the multiple platform IDs we have, and the provided code for building the platform file handles this automatically.

The following is a partial platform file generated from the attached sample:

Platform File Builder
In this section, we will present information on the tool used to generate the platform information file starting with the input files.

PlatformBuilder: Reflection Data Input Files
In the second part of this series we discussed the VersionBuilder tool, which uses as input the raw reflection data files and combines this to create a raw single reflection data file.
Unlike the VersionBuilder, our PlatformBuilder uses the processed reflection data files as input files. That is, after the document model is applied to the reflection data. It might be possible to use the raw reflection data, but for simplicity and our desire to minimize the memory requirements by using XmlWriter, we found the processed format more easier to consume.
Now, revisiting the version building process, the platform building process can be added in a modified diagram as shown below (the extension is shown in blue, the version builder part is removed for simplicity):

PlatformBuilder: Configuration Input File
This is really not a requirement of the code presented with this article. The code is presented in the source form as private static methods so that it can be modified and used in various forms.
For this article, we used this procedure, which we think will fit in most applications; reuse the VersionBuilder configuration file. We exposed one public static method which uses this and has this declaration:

configFile: The VersionBuilder configuration file generated in the version information processing.

platformFile: The output platform file.

isSilverlight: Specifies whether we are dealing with the Silverlight Framework Group or the .NET Framework Group. This parameter is currently not used.

This method will parse the configuration file, retrieve the raw reflection data files, change the extensions to .xml (from the .org), load the processed reflection data files and then build the platform file.

NOTE:

This method will not convert the raw reflection data files to processed reflection data files, it assumed this was done and processed files are in the same directory as the configuration file.

Using this procedure will limit you to creation of a single platform information file, even though this is not a requirement of the build component.

In the third sample project to demonstrate the platform information, we modified the PreBuilder.bat to add the support for transforming the raw reflection data files to the processed reflection data files.

Processing the Documentations
After creating the platform file(s), the only other requirement for platform information is to configure the build component.
The build component used here is: PlatformsComponent.
The configuration of this component is similar to the following:

<component type="Microsoft.Ddue.Tools.PlatformsComponent" assembly="%DXROOT%\ProductionTools\BuildComponents.dll">
<!-- The order of filter files in this config determines the order of platforms in the output. -->
<filter files=".\PlatformFile.xml" />
</component>

Note that this build component is not added to the default Sandcastle configuration file and you must add it yourself.
This is a Pre-Transform component and the position selected is based on sample configuration files found in the Sandcastle source repository. We have not tested other pre-transform positions in the configuration file, you are invited to experiment.

We are ready for the processing, so lets consider some test configurations. Since platform icons are not supported by user/custom platform IDs, we will reconfigure the previous version information file to use the standard platform and version IDs. You do not have to worry about the VersionSharedContent.xml file since its content is useless and will not have any effect on the build process.

.NET Framework Group
For the .NET Framework Group, we will use the following version build configuration (VersionBuilder.config) to demonstrate the platform information:

NOTE: As you can observe when you compare the second images from each output, the framework members are not displayed or are filtered out in the Silverlight group. This same issue affects the user/custom platform.
The only reason we could attribute to this is; Sandcastle currently treats the Silverlight (and its related frameworks) as sub-platforms on the .NET Framework - this being the main platform. We can modify the transformation styles to resolve this issue if required.

Sample Test Applications
The test applications in this article are available for download. Use this to test the platform information.

The sample and codes are released under the Ms-PL license, which is used by the Sandcastle. You can use it anyway you like.NOTE: This is a zip file shared from the Google Docs. If the download button is not displayed when clicked, you can download the zip file from the File menu.

Conclusions
This completes the information required to build a better platform and version information using the Sandcastle tools.
In the Sandcastle Assist project, we hope to make the Sandcastle tools easier to use, and as with the Sandcastle conceptual topics supports, which we documented and exposed to all, we have documented and hopefully exposed the platform/version information support to all.

Friday, February 3, 2012

Building the Version Information
We will discuss how to build the version information using very simple illustrative code samples. We will create a standalone application (independent of other libraries) so that the knowledge can easily be used in other applications.
Lets review the build process in the Sandcastle and then examine the changes required for the version information

General API Processing
In this section, we will discuss the procedure used by the Sandcastle system to build a documentation. By providing the outline of the processing, we will not only make it easy for all to get an overview of the documentation process, but also make it easy to mark the points in the processing stages that must be changed for version and platform information.
This is the processing procedure for a typical Sandcastle API documentation:

1. Initialization
This is nothing special, clean up previous build outputs and/or create working/output directories.
The main inputs; the assemblies to be built and the corresponding XML comments files, may also be copied to the appropriate directories during this stage.
The most important processing here is to copy the Sandcastle resources (styles, scripts, icons or images) for the chosen model to the working or output directory. There is a batch file provided if you wish to do this with the command line.
Run this in the output/working directory:

...\Presentation\vs2005\copyOutput.bat

2. Reflection Builder
This involves the use of the MrefBuilder.exe tool provided by the Sandcastle. This is also the first most important stage for the version information. Let consider the input sources:

XML Comment Input FilesWith the exception of custom application-defined processing and filtering, the XML comments files generated by the C# and VB.NET compilers are not further processed.

Assembly Input FilesBy .NET reflections, the input assembly files are converted into an XML format, which represents the various API defined in the target assembly files.The MrefBuilder.exe tool is used and the output file is usually named Reflection.org.

The basic command line to this tool is:

MrefBuilder TestLibrary.dll /out:Reflection.org

config:MrefBuilder.config [options]

NOTE: The configuration file for this tool, MrefBuilder.config, determines the contents of the output. This is where API filters are defined. It also provides support for add-ins, which are used to modify and/or add extra information to the reflection output file, such as information for extension methods. This file is, however, optional. If not provided, a default file available in the Sandcastle installed directory is used.

3. Documentation Model Builder

Sandcastle provides support for building various documentation model, such as the VS2005, Hana and Prototype models. At this stage we transform the Reflection.org file to a format defined by the documentation model of choice using the XslTransform.exe tool provided by the Sandcastle. This tool uses XSL files in the ProductionTransforms directory for the processing. The output of this processing is usually named Reflection.xml.

NOTE: For multiple assemblies, you may apply the MergeDuplicates.xsl transform through the XslTransform.exe tool before this step to merge duplicates. It is not used in this case for simplicity.

Now, for the VS2005 model

Use: ApplyVSDocModel.xsl to create the model.

Use: AddFriendlyFilenames.xsl or similar naming XSL to create the file names of the various API defined by the assemblies.

The above two are normally combined on a single command line as (omitting the directory information for simplicity):

XslTransform /xsl:"ApplyVSDocModel.xsl" Reflection.org

/xsl:"AddFriendlyFilenames.xsl" /out:Reflection.xml [options]

Below is a diagram representing the steps required to build the reflection data from various assemblies for a single version of the documentation.

4. Manifest Builder

The manifest file is the list of topics or APIs that will be documented. The manifest generator is also an XSL file and therefore processed by the XslTransform.exe tool. The input is the reflection file, Reflection.xml, from the previous step and output is usually named Manifest.xml.

The command line for generating the manifest file is given below:

XslTransform /xsl:"ReflectionToManifest.xsl"

Reflection.xml /out:Manifest.xml

5. Creating Table of Contents (TOC)

Like many other operations, the table of contents of the documentation is created for each model using the appropriate XSL file. The XslTransform.exe tool is used with the reflection file, Reflection.xml, as the input. The output is usually named Toc.xml file.

The command line is as follows (omitting the directories):

XslTransform /xsl:"CreateVSToc.xsl" Reflection.xml /out:Toc.xml

6. Processing the Documentations

The main processing of the documentation is done with the BuildAssembler.exe tool. The input of this tool is the manifest file created in the previous step and a configuration file. The output is usually the HTML files of the documentation.

The command line to this tool is given below:

BuildAssembler /config:"Sandcastle.config" Manifest.xml

We will not present detail description of the operation of this tool, but highlight some of the main operations and where the version information is applied.
This tool starts with a skeleton or template XML file, which defines the structure of the document model. The tool loops through each of the topics listed in the manifest file and using the information defined in the configuration file to copy in the necessary parts of the skeleton or template.

The actual processing is broken down into simpler steps, each processed by units called the Build Component. This is similar to the assembling line in a manufacturing system, giving the name of the tool Build Assembler.
The interesting thing to note about the build components is that it provides an extensible build system, so you can provide custom build component to either replace a default Sandcastle component or provide new features in the assembling process.

These are the processing steps in the build assembler:

6-1: Create skeleton documentIt is similar to car assembly line; starting with a frame, fasten all other components to it and finally paint it.The build component used here is named: CopyFromFileComponent.The configuration of this component is shown below:

6-3: Generate syntax
The syntax for the various supported or selected programming languages showing the declaration of the API is generated here.
The main build component used here is named: SyntaxComponent.
The Syntax Component uses language specific syntax generators to handle the syntax generation. Each generator derives from SyntaxGeneratorTemplate class, which in turn derives from SyntaxGenerator class.
The configuration of this component, which is normally wrapped in conditional component is given below:

6-4: Copying in comments
This involves indexing the XML comments of your API and comments from the .NET Framework. This also requires a large amount of memory. The CopyFromIndexComponent is configured differently as shown below:

Specialized copying operations, such as the handling of inherited documents, are handled by different components called the Copy Component, which derived from the CopyComponent class.
Like the reflection data copying, this is also broken down into several parts.

6-5: Transformation
After the API document is completely composed in memory, it is transformed to XHTML format using transformation styles defined by the selected model.
Most user or custom components in the assembling process is applied either before or after the transformation, and are classified as Pre-Transform components and Post-Transform components.
The build component used here is named: TransformComponent.
The configuration of this component is given below:

The transformation to the XHTML is really not completed yet. Some XML tags are inserted by the transformation process and must be resolved.

6-6: Resolving Shared Content
The shared content is the string resources used by the transformation and other steps for customization and localization of the document.
The string resources are defined in XML file in a simply key-value format as shown below:

NOTE: These resources include the version and platform information of the document, making this a step that must be modified to support the version information.
The build component used here is named: SharedContentComponent.
The configuration of this component is given below:

6-7: Resolving reference links
The reference links inserted by the transformation process is resolved at this stage of the processing, using the your reflection data and the .NET framework reflection data, which is used to resolve the MSDN links to the framework documentations.
The build component used here is named: ResolveReferenceLinksComponent.
The configuration of this component is given below:

6-8: Saving the document
Finally, the XHTML document is saved to a file and ready for compilation to any format.
The build component used here is named: SaveComponent.
The configuration of this component is shown below:

At this stage, the HTML files are compiled into HtmlHelp 1, 2, 3 or WebHelp. In some cases, output specific tools are provided to generate the help compiler projects and in some case process the XHTML files.
For the Microsoft HTMLHelp 1 outputs, which we are creating for the sample projects, a tool named ChmBuilder.exe is provided to handle such operations. This requires a configuration file and the command line is shown below:

A sample illustrative project is provided to demonstrate how to build a simple Sandcastle API document. It uses three batch files to break down the build process:

ReflectionBuilder.bat: This is used to create the reflection files, and it is the only batch that will change when we consider the support of version information. We will use the default configuration file for the MrefBuilder.exe tool for simplicity.

PreBuilder.bat: This applies the document model.

PostBuilder.bat: This processes the documentation and compiles the help file.

A screenshot of the Project Explorer is shown below. In this, we provided custom general and feedback contents and configuration files for various tools.

As shown, the GeneralBuilder project is the main application for the general Sandcastle API testing. There are four test projects; TestProject1, TestProject2, TestProject3 and TestProject4, all of which are class libraries containing either one or two classes. However, only one project is used for this test sample for simplicity.
We will now continue to discuss what must be changed to add version information to the documentation.

NOTE: All Sandcastle projects are built using the standard or default installation and styles. Custom styles such as from the Sandcastle Styles project are not used.

Version Information API Processing
We have so far presented the API documentation processing by Sandcastle. Now, we will discuss what will change if you have to support version and platform information.
For the version information, we will change only two steps in the previous procedure to make it work

Reflection Builder

Processing the Documents

1. Reflection Builder

For the version information, instead of a set of assemblies defining a version, you have a number of sets each defining a version. Each set defines version platform and will produce reflection data. We must combine the reflection data from each set to produce a final reflection data.

Using the Sandcastle article on this, the sequence of processing version information are

Run the MrefBuilder.exe tool on each version's assembly set. Note that the assembly set for each version of a project includes the complete set of assemblies, not just the assemblies that are new or changed. This means, each assembly set, which is equivalent to version platform, can consists of various versions of the platform.For instance, the .NET Framework platform has various releases; 1.0 1.1, 2.0, 3.0, 3.5, 4.0 and 4.5. All these releases define a set.

Run the MergeDuplicates.xsl (through the XslTransform.exe) on each of the version-specific reflection files.

Run the VersionBuilder.exe with the version-specific reflection files as input and the combined reflection file as output.

Proceed with the rest of the Sandcastle build using the combined reflection file.

The reflection and documentation model builder diagram can be modified as shown below:

2. Version Builder

We will now discuss the version builder tool, VersionBuilder.exe, which is used to create the combined reflection data.

The command line to this tool is given below:

VersionBuilder /config:VersionBuilder.config /out:reflection.org

NOTE: A third parameter of this tool is /rip+|-, is a switch; /rip+ to turn on (or true) and /rip- to turn off (or false). The default is on (or true). The option specifies whether to rip old APIs that are not supported by the latest versions. We will come back to this option again, since it is easier if you see it in action.

In order to understand the version builder configuration file, we will return to the illustrative sample and define the documentation structure we will to build with version information.
We will define three platforms with the following information:

Platform 1

Platform 2

Platform 3

Platform ID

PlatformId1

PlatformId2

PlatformId3

Platform Label

First Platform

Second Platform

Third Platform

Versions (Projects)

TestProject1

TestProject2

TestProject3
TestProject4

We will define the versions with the following information:

TestProject1

TestProject2

TestProject3

TestProject4

Version ID

VersionId1

VersionId2

VersionId3

VersionId4

Version Label

1.0

2.0

3.0

3.0 SP

From these we expect the version information section of the output to be as shown below:

Now, the VersionBuilder configuration file specifies the input reflection files that MrefBuilder generated for each version. It also specifies a name for the project and names for each version. The names correspond to the IDs of shared content items for the project and version names that appear in the Version Information section of an API's documentation.
Here is the content of the VersionBuilder configuration file required to define our documentation structure:

NOTE: The TestProject4 reflection data is placed before the TestProject3 reflection data. This is related to the /rip+|- option, and I will illustrate this later.

3. Processing the Documentations
We need to update the BuildAssembler configuration file to include the version information. We will update the following two build component configurations

Copying in Comments

Resolving Shared Contents

Copying in Comments

In this case, we just want to highlight important points to avoid confusion.

The original Sandcastle tutorial sample provided by Microsoft on version information used only one comment file, let's explain why. Both the tutorial and the sample presented the basic use of version information, whereby the documentation project has a set of assemblies for a release and next release contains updated versions of the set of assemblies with/without additional assemblies.

The new release contains updated information and anything not included in the new release from the old is considered an old API, which is ripped off by the version builder tool (the /rip+|- option).

In practice, you are dealing with different platforms with many unrelated APIs, such as .NET Compact Framework and XNA Framework. Therefore, the included comments must cover all the supported API in the documentations.

For the illustrative sample, here is the configuration of the CopyFromIndexComponent component.

Resolving Shared Contents
We defined the configuration file for the version builder tool using only the platform and version IDs. We will define the platform and version labels in a shared content file and include it in build assembler configuration.
For the provided sample, this file is named VersionsSharedContent.xml, and the configuration of the shared build component is shown below:

The sample project, VersionInfoBuilder, is completed and is shown below:

Ripping Old API: /rip+|- Option
As promised, we will now illustrate this option. The version builder does not check your assembly versions, and to make sure you see it; all the test projects versions are set to 1.0.0.0.
The version builder, however, uses the order in which the versions are defined in a platform.
Let us illustrate this with the Platform 3, which consist of two versions; Version 3 and Version 4. At the code level, the only difference between these two versions is that Version 4 includes an additional property named Value in the class TestOther. With the configuration shown above and default rip off option (on or true), the output is correctly shown below:

However, if you reverse the order of Version 3 and 4, placing the Version 3 before the Version 4 and with the rip off (on or true), the version builder will consider the Value property as old and it will be ripped off as shown below:

Fixing Grouped or Container Pages
In our sample, we defined our own platform and version IDs. Sandcastle, however, assumes only the platform and version IDs defined for the MSDN will be used. The result is that when you compiled the version information help file, the container or grouped pages such as the Members, Constructors, Methods and Properties pages in your document will be empty as shown below:

One quick way to fix this issue is to apply the fix provided by a Sandcastle user by name SanderSaares on Sandcastle Styles forum.

The sample is released under the Ms-PL license, which is used by the Sandcastle. You can use it anyway you like.NOTE: This is a zip file shared from Google Docs. If the download button is not displayed when clicked, you can download the zip file from the File menu.

Conclusions
Whether you build your documentations directly using the Sandcastle command line tools or you use third-party GUI tools, you need a better understanding of the Sandcastle version information processing to structure your documentation project for better results.
In this article, we have examined the Sandcastle build process and extended that knowledge to building documentations with version information.
The last part of this series will examine platform information support. Thank you.

Saturday, January 21, 2012

Introduction
Version information is one of the most advanced features supported by the Microsoft Developer Network (MSDN) help system.
The feature was introduced in September 2007 release of the Sandcastle with blogs detailing both the feature and how is it supported by the Sandcastle tools. The articles introducing the version information are listed below for reference:

The information provided in those articles and tools were not sufficient to implement the full version information features in the MSDN Help System. In the Sandcastle Assist project, we implemented most of the missing parts but one main feature was still missing, and that is the target of this blog.

What is Version Information?
The MSDN help system for managed codes provides you with the ability to view the various API released in each version of the .NET and Silverlight frameworks.
We will illustrate the various parts of the feature so that we could discuss the complete implementations or how to realize them.
For better understanding of this feature, I will introduce three terms associated with it:

Versions
Versions are easy to recognize because these are directly marked by the release version numbers.
For the .NET framework; we have version labels; 1.0, 1.1 to 4.5 (in beta).
In the implementations, versions are defined by two parameters

Version IDThis uniquely identify each release version, and for the MSDN there are a number of predefined versions identifiers provided in the Sandcastle content files.

Version LabelThis is the text representation of the version, and in most cases are numbers such as the 1.0, 1.1 etc. However, for the version information, this can be any text.

Examples of predefined version ids and labels provided by the Sandcastle in the reference_content.xml file are:

ID

Label

netfw40

4.0

netfw35_1

3.5 SP1

silverlight_mobile_v1

Windows Phone OS 7.0

NOTE: In that file, you will find these labelled as framework version ids.

The version parts are marked in the image shown below:

Fig 1: Version Information

Platforms
The platforms are confusing when working the Version Information because in Sandcastle there is another term, platform note, which relates mainly to the Windows platforms like Win95, WinXP, WinVista, Win7 etc.
For version information, platform is essentially versions collections (or categories). Similar to the version, platforms are defined by two parameters:

Platform IDThis uniquely identify the platform (or versions collection), and there are predefined identifiers for the MSDN.

Platform LabelThis is the descriptive name of the platform.

Examples of predefined platform ids and labels provided by the Sandcastle in the reference_content.xml file are:

ID

Label

netfw

.NET Framework

netcfw

.NET Compact Framework

silverlight

Silverlight

NOTE: In that XML file, you will find this labelled as framework ids.

The platform parts are marked in the image below (again, this is extended to include the platform note to avoid confusion):

Fig 2: Platform Information

The platforms are more visually expressed by icons that are directly attached to the API in the documentations as shown below:

Fig 3: Platform Icons

There are few point to take note of in the above image

The main platform (.NET Framework in this case) is not represented by an icon.

Most other platforms (XNA Framework and Portable Class Library in this case) have icons specifying which APIs are supported.

Like the main platform, not all need an icon. Platform icon is not a requirement as in the case of .NET Framework Client Profile shown earlier.

It is possible to provide platform icons, even if the version information are not provided, as shown by the X-like
XNA Framework icon. The version information and platform information can be used independent of each other.

Finally, the platform is used for filtering, which allows you to display the API for the platform you wish to see. This is available on the older online version of the MSDN help and is shown below:

Fig 4: Platform Filters

Frameworks

This may also be a little confusing when dealing with version information. By the version information, there are only two frameworks: .NET Framework and Silverlight Framework.
Whereas the Sandcastle allows you to mix version information easily from each version framework, to comply with the MSDN, you will have to create your documentation to work with the defined frameworks.
The confusion may be due to the fact that Silverlight documentation was later added to the MSDN and Sandcastle.
The image in Fig 3 is for a MemoryStream class defined in the .NET Framework. The same image for the Silverlight Framework, also taken from the MSDN, is shown below:

Fig 5: MemoryStream under Silverlight Version Framework

As you might have observed by now, I prefer the term Version Framework to avoid confusion.
Note that the .NET Version Framework is labelled by what may be called the version framework identifiers in the MSDN (see the image below). However, this is defined by the main platform in your documentation.

Fig 6: Framework Filter

In short, the Help 1 and Help 2 provide the platform filters in a framework, the Web Helps and Help 3 provide main platform or framework filters.

Conclusion
The version information is an advanced Sandcastle feature and working with it can be confusing. This article documents my understanding of the feature, and how I approach it. There is no complete formal documentation of this feature and I claim the responsibility for any error in this article.
Next, we will consider the implementation; first an updated version information and then the platform information.