Into ALM with TFShttps://intovsts.net
blogging about Application Lifecycle Management with Team Foundation ServerThu, 08 Dec 2016 19:36:50 +0000enhourly1http://wordpress.com/https://s2.wp.com/i/buttonw-com.pngInto ALM with TFShttps://intovsts.net
Migrate inline images to VSTShttps://intovsts.net/2016/09/14/migrate-inline-images-to-vsts/
https://intovsts.net/2016/09/14/migrate-inline-images-to-vsts/#commentsWed, 14 Sep 2016 09:00:41 +0000http://intovsts.net/?p=1227]]>Lately I have been planning a number of migrations to move small/big companies from TFS on-premises to Visual Studio Team Services (VSTS).

There are a number of options to migrate data from TFS to VSTS, but option 3 [high-fidelity database migration] is unfortunately not yet available. So, most of the time I still use custom/third-party tooling to perform the migration which is not always straightforward and may be very time-consuming.

One serious issue that popped up in a migration towards VSTS (using the TFS Integration Tools), was that the inline images in the Description field (or other html fields) of migrated work items were not properly migrated. The inline images are actually still referring to the old TFS on-premises environment because the html value of the Description field contains an <img> element with a source set to http://<tfs-on-prem>:8080/tfs/<tpc>/workitemtracking/v1.0/attachfilehandler.ashx?filenameguid=<guid>&filename=<filename>.png.

As a result, all inline images are only stored in the old TFS environment and they have not been uploaded to VSTS. The html value of the Description field has been migrated as-is. Initially you might not notice this after the migration of the selected work items because as long as the old TFS environment is still available, the inline image will be displayed. But what if the old TFS environment has been archived/destroyed?

To correct this and to upload the original images to VSTS, I have written some code (TFS API) to loop over the VSTS work items to detect image links to the old TFS environment. Using the source link of the original image, I can download the image to my local disk and upload it as an attachment to the VSTS work item. Finally, I’m replacing the original image source link with the new VSTS image link. Good to know is that once in-line images are detected inside the html field, those images are stored on the server and the temporary image file attachments may be deleted as well.

]]>https://intovsts.net/2016/09/14/migrate-inline-images-to-vsts/feed/1pietergheysensRetain VSTS Build Indefinitely – Fetch Build ID from RM artifact variablehttps://intovsts.net/2016/06/26/retain-vsts-build-indefinitely/
https://intovsts.net/2016/06/26/retain-vsts-build-indefinitely/#respondSun, 26 Jun 2016 16:03:06 +0000http://intovsts.wordpress.com/?p=1221]]>While showing a Visual Studio Release Management demo in a Practical DevOps training, I stressed how important it was that the build artifact, which was used during a release, was not destroyed by the built-in retention policy. By default, the output of a build run is only stored for 10 days. So, in case you really want to keep the build and build artifacts, you must take care of this yourself.

For that purpose I created a powershell script which calls the VSTS REST API to accomplish this.

The powershell script is called from a release management task at the beginning of the release process.

At the moment I worked on this activity (probably still in private preview of VSRM – September 2015) it was not possible yet to fetch the exact Build ID through the build artifact which is linked in the release definition. That’s why I was dropping a simple text file with the Build ID in the build process which was also stored in the build artifact. That file was then used in the release management process to parse the Build ID.

Apparently, this workaround is not necessary anymore and you can now immediately fetch the Build Id from the build artifact via a pre-defined release management artifact variable RELEASE_ARTIFACTS_[source-alias]_[variable-name]. Read more about the available RM artifact variables.

]]>https://intovsts.net/2016/06/26/retain-vsts-build-indefinitely/feed/0pietergheysensKeepBuildArtifactBuildIdMigrating TFVC (TFS on-premises) to Git (VSTS)https://intovsts.net/2016/06/09/migrating-tfvc-tfs-on-premises-to-git-vsts/
https://intovsts.net/2016/06/09/migrating-tfvc-tfs-on-premises-to-git-vsts/#commentsThu, 09 Jun 2016 12:21:25 +0000http://intovsts.net/?p=1155]]>In the last couple of months I do get more requests to move TFVC version control history to a git repository in Visual Studio Team Services (VSTS). The migration from TFVC to TFVC is at the moment possible via the TFS Integration Tools and is not that straightforward to accomplish. Migrating to a git repository is much simpler and is certainly the way to go if you were already planning to adopt git in the future. The migration can be done via Git-TF which is a set of cross-platform command-line tools that facilitate sharing of changes between Team Foundation Server, Visual Studio Team Services and Git.

Add the extracted git-tf path to the system environment variable “path”

Create a new “git” Team Project in VSTS

Migration Steps:

Open a command-line prompt and navigate to a directory where you want to host the local git repository

Call git-tf clone to push all TFS changeset info from TFVC to a new local git repo. The first argument is the Team Project Collection url. You pass the TF version control path to the exact branch in the second argument and you end the command with the “deep” flag to ensure that the full history of the branch is moved into separate commits in the git repo. Pass your credentials to connect to TFS and execute the command.

Once you have a local git repository it’s easy to push it towards an empty central VSTS git repository. First use the git remote add command to link your local git repo to the remote “origin” and afterwards you can push all changes via git push.

Navigate to the Code Hub in your VSTS Team Project and you should see all code history inside the git repo. What’s a big plus is that the original changeset date/time stamps are now part of the git commit info.

]]>https://intovsts.net/2016/06/09/migrating-tfvc-tfs-on-premises-to-git-vsts/feed/2pietergheysensgit-tf-clone1git-tf-clone2Work Item Query via TFS API and the dayPrecision parameterhttps://intovsts.net/2016/03/07/work-item-query-via-tfs-api-and-the-dayprecision-parameter/
https://intovsts.net/2016/03/07/work-item-query-via-tfs-api-and-the-dayprecision-parameter/#respondMon, 07 Mar 2016 09:30:51 +0000http://intovsts.net/?p=1133]]>By default TFS doesn’t pay attention to the time part in work item queries when comparing datetime values. If you want to launch a query and you need to take into account the exact timestamp, you must switch off the dayPrecision parameter in the Query constructor.

]]>https://intovsts.net/2016/03/07/work-item-query-via-tfs-api-and-the-dayprecision-parameter/feed/0pietergheysensdayPrecision(upcoming) ALM/DevOps Community Talkshttps://intovsts.net/2016/02/08/upcoming-almdevops-community-talks/
https://intovsts.net/2016/02/08/upcoming-almdevops-community-talks/#respondMon, 08 Feb 2016 09:30:09 +0000http://intovsts.net/?p=1109]]>Q1 of 2016 has already been quite busy with the organisation of VISUG 10 Years and the upcoming conference Techorama 2016. The user group and all community activities are simply not a simple side job anymore.

For the celebration of VISUG 10 years I delivered a session DevOps with Visual Studio Release Management.

]]>https://intovsts.net/2016/02/08/upcoming-almdevops-community-talks/feed/0pietergheysensDestroying deleted branches in TFShttps://intovsts.net/2016/01/13/destroying-deleted-branches-in-tfs/
https://intovsts.net/2016/01/13/destroying-deleted-branches-in-tfs/#commentsWed, 13 Jan 2016 11:24:40 +0000http://intovsts.wordpress.com/?p=1059]]>Everyone using a version control system will have ever used a delete command to delete files/folders which are not required anymore.

Team Foundation Verion Control (TFVC) treats a delete [tf delete] as a pendingchange in your workspace (type = delete). The final checkin command removes the item from the version control server, but does not delete the item permanently. All historical changes to the file remain available for lookup. In fact it’s also possible to undelete [tf undelete] an item and bring it back into play.

In only a few scenarios you might want to go one step further and really destroy the deleted files from TFS …

Recently a customer was complaining about the exponential growth of their TFS 2013 databases (exceeding 400GB for a specific TPC database). Running some SQL scripts on the tbl_content table of the Team Project Collection database revealed that a lot of version control and file container data was added monthly to the TFS database. Ranging from 2.5GB a few months ago up to 8GB in the last months.

After some investigation I noticed that the development team was creating a lot of feature branches (150+) for a specific application to isolate their feature development. This shouldn’t have a big impact on the exponential growth of the TFS databases because a branch is like a shallow copy of each of the files from the source branch. Both branches will originally refer to the same copy of each file in TFS. Extra changes will trigger a deltafication process to optimize storage. More info of this complex procedure is explained in this old TFS 2010 blog post which should still be valid for the latest version of TFS. But the main problem was that the feature branches also contained a lot of references (NuGet packages / binary files) which were added to version control as well and many dependency changes were initiated in the feature branches. As described in the old blog post, TFS will also try to compute deltas for binary files, but it’s way more difficult to predict database growth based on the sizes of these deltas. The results from deltafication vary greatly depending on what type of file it is, and the nature of the changes.

After further discussions, we decided to permanently destroy old deleted feature branches in version control and see the impact of the database size. I wrote a powershell script to iterate over a version control root folder to search for deleted branches and to destroy it only if the latest check-in did not occur in the last 100 days. In the script I used the startcleanup switch to immediately trigger the clean up process in TFS instead of waiting for the daily clean up background process.

I don’t remember the exact results on the database size, but it certainly decreased the total size of the TFS database for the involved Team Project Collection. With the upcoming planned upgrade to TFS 2015, we also used the Test Attachment Cleaner (part of the TFS Power Tools) to remove old diagnostic test data. So, in the end we were able to reduce the total size of the TPC database with a fair amount of GB in order to also speed up the duration of the future upgrade to TFS 2015.

Another initiative was started to not use TFS anymore to keep (physical) track of the dependencies. TFS is indeed a version control repository to store sources, but there are better tools out there to store software packages/artefacts.

]]>https://intovsts.net/2016/01/13/destroying-deleted-branches-in-tfs/feed/1pietergheysensTFS Build 2015 … and versioning!https://intovsts.net/2015/08/24/tfs-build-2015-and-versioning/
https://intovsts.net/2015/08/24/tfs-build-2015-and-versioning/#commentsMon, 24 Aug 2015 13:06:21 +0000http://intovsts.net/?p=1039]]>Lately I got some time to play a bit more with the new build system which was released with TFS 2015 and which is also available for Visual Studio Online. The new build system was in the beginning announced as build vNext, but now with the release of TFS 2015, it’s safe to call it Team Foundation Build 2015 (TFBuild 2015) while the “old” build system can be referred to as the xaml (workflow) build system. Colin Dembovsky has a great post why you should switch to the new build sytem.

In the last years, I had to implement a lot of customizations into the xaml build system and I became very productive with the workflow activities. Along the way I developed a number of generic activities which I could reuse for other assignments and I really knew my way around in the build workflow. In many cases, the TFS Build Extensions were used to not reinvent the wheel. So, in the first phase I was a bit sceptic about the rise of yet another build system, but I clearly saw some interesting advantages which are explained in the post by Colin. One disadvantage of the xaml build system is the steep learning curve to master the customization process and also the deployment mechanism to refresh the TFS build controller(s). But like I experienced, once you got there, you were able to integrate very powerful customizations into the build process. Anyway, the “old” build system won’t disappear and you can still rely on this functionality for quite some time in the future, but I recommend to have a good look at the new build system and use it for your new/future build definitions.

In this post I want to share how I integrated a common activity in the build process: Versioning. With the available build steps it has become extremely simple to hook your own scripts into the build process. In your scripts you will have access to some predefined build variables.

In the screenshot above you will see that I launch a powershell script (PreBuild.ps1) before building the solution and I pass one argument productVersion to the script. The powershell script will do the magic in the background to replace all versioning values for AssemblyVersion, AssemblyFileVersion and AssemblyInformationalVersion in the Assembly Info files, based on this product version. The product version will be passed as a whole to the AssemblyVersion and the AssemblyInformationalVersion attributes. The AssemblyFileVersion will be replaced with a full version number which will consist of the major and minor version number of the product version, a Julian based date and an incremental build number.

Assembly File Version = 1.0.15236.3

1 => taken from “Major” product version

0 => taken from “Minor” product version

15236 => generated by build process: “15” = year 2015, “236” = day of year 2015

3 => third build, run on day 236 in year 2015

Looking at the assembly details of a custom built Fabrikam assembly now reveals correct meta data:

I also modified the build number format to have some more version information displayed in the build run.

I added a gist at GitHub to share the powershell script. Note that the script has been used for experimentation and may not be ready to be used for production. it certainly lacks some proper validation and error-handling. Use at your own risk.

Also have a look at some similar inspiring blog posts about versioning TFS Builds which helped me to develop the powershell script that works for my scenario.

Without a methodical (assembly) version numbering strategy, the ability to determine what changes were included in which version is lost. In my opinion, you always need to know exactly which source files went into which build and which version of the sofware is currently deployed in a specific environment. A random version numbering system creates confusion and will soon or later cause deployment risks. It will become a nightmare to fetch the exact source files to reproduce a bug from production.

All versioning of .NET assemblies that use the common language runtime is done at the assembly level. The specific version of an assembly and the versions of dependent assemblies are recorded in the assembly’s manifest. The default version policy for the runtime is that applications run only with the versions they were built and tested with, unless overridden by explicit version policy in configuration files.

Each .NET project has an AssemblyInfo file which contains an AssemblyVersion attribute and an AssemblyFileVersion attribute.

AssemblyVersion: this is the version number used by the .NET framework during build and at runtime to locate, link and load the assemblies. When you add a reference to any assembly in your project, it is this version number which gets embedded. At runtime, the CLR looks for assembly with this version number to load. But remember this version is used along with name, public key token and culture information only if the assemblies are strong-named signed. If assemblies are not strong-named signed, only file names are used for loading.

AssemblyFileVersion: This is the version number given to a file as in file system. It is displayed by Windows Explorer. It’s never used by .NET framework or runtime for referencing.

But what about this difference between AssemblyVersion and AssemblyFileVersion? Many times, I see that the same version is applied to both attributes … but why are these two (different) attributes provided by the .NET Framework? The AssemblyVersion should be the public version of an entire software application, while the AssemblyFileVersion is more the version of a specific component which may only be a small part of the entire application. The AssemblyFileVersion is the best place to put extra build version information which can be important for patching individual components of a software application.

Please follow the Semantic Versioning recommendations to dictate how the AssemblyVersion should be assigned and incremented. For the AssemblyFileVersion, I tend to include specific build information. Often, you will need to build (and test) a number of time a specific SemVer version of your software.

For example: release 1 of a software application could have the AssemblyVersion set to 1.0.0 (all components), while the AssemblyFileVersion of the individual components could be set to 1.0.15234.2 which refers to a unique build number of the build system and is linked to a particular date and a revision: “15” = year 2015; “234” = day number in 2015; “2” = second build processed that day. This also allows to later patch individual components in production with a similar AssemblyVersion (1.0.0), but a different AssemblyFileVersion (1.0.15235.1).

So, let’s try to apply this to a test project in Visual Studio and see the assembly details after building the project …

Now you should be confused! Why does the Product Version display the AssemblyFileVersion and where’s the AssemblyVersion? The problem here is that a new Visual Studio project doesn’t include a third version attribute AssemblyInformationalVersion which is intended to represent the public version of your entire software application. Note that the CLR doesn’t care about this third (optional) version attribute. In short, the same Semantic Versioning rules of AssemblyVersion should be applied to AssemblyInformationalVersion.

Aha! This looks much better right? Now it’s also easy to extract this metadata from your deployed assemblies and this information can be nicely listed in the about box of your software. The only issue with this approach is that the AssemblyFileVersion doesn’t include the “patch” number (Semantic Versioning) of the AssemblyVersion, but this can be ignored with the fact that the AssemblyFileVersion will be unique and can be linked to a unique build run in the build system. This way of working is my personal interpretation of how versioning can be properly applied in complex software applications and doesn’t reflect official guidelines from Microsoft. My goal here is to make software developers aware of the potential risks of not having a clear versioning strategy.

Now, forget about manually setting version information in the AssemblyInfo files and never ever release software from a local Visual Studio build. In a streamlined build process, generating unique build and version numbers are centrally coordinated. For effective troubleshooting and traceability, it’s imperative that the generated assemblies are stamped with a unique identifier that can be easily traced back to a system build number.

In a next post I will talk about how you can achieve this global .NET versioning strategy with the new build system in TFS 2015.

]]>https://intovsts.net/2015/08/24/global-net-versioning-strategy-assemblyinformationalversion/feed/5pietergheysensVersioning-1Versioning-2Deep copy of Test Cases to a TFS Team Project in another Team Project Collectionhttps://intovsts.net/2015/07/22/deep-copy-of-test-cases-to-a-tfs-team-project-in-another-team-project-collection/
https://intovsts.net/2015/07/22/deep-copy-of-test-cases-to-a-tfs-team-project-in-another-team-project-collection/#commentsWed, 22 Jul 2015 19:03:24 +0000http://intovsts.net/?p=1024]]>A few days ago I got stuck in a project to migrate a set of test cases from an “old” Team Project to another (new) Team Project in another Team Project Collection (TFS 2013 Update 4 environment). Via the TFS API I used the work item Copy method to perform a deep copy [including links/attachments] with the WorkItemCopyFlags set to CopyFiles.

This always worked when copying test cases to a Team Project in the same Team Project Collection (TPC) as the original Team Project, but copying the test cases to a Team Project in another TPC always caused an error:

TF237136: File attachment was not found on attachment server.

I expected this to work, but after I verified that the MTMCopyTool also generated the same error when copying test cases across Team Projects, I decided to search for a workaround … downloading and uploading the attachments as after-processing.

Not perfect, but good enough to keep going and moving all attachments to the other Team Project in the other Team Project Collection.

]]>https://intovsts.net/2015/07/22/deep-copy-of-test-cases-to-a-tfs-team-project-in-another-team-project-collection/feed/1pietergheysensDownloadUploadAttachmentsReporting Services issues after migration to TFS 2015 RC2 (from TFS 2010)https://intovsts.net/2015/07/10/reporting-services-issues-after-migration-to-tfs-2015-rc2-from-tfs-2010/
https://intovsts.net/2015/07/10/reporting-services-issues-after-migration-to-tfs-2015-rc2-from-tfs-2010/#commentsFri, 10 Jul 2015 17:31:34 +0000http://intovsts.net/?p=1018]]>Yesterday, I blogged about the migration activities to get to TFS 2015 RC2, but I was still stuck at an error during new Team Project creation.

Looking into the logs it looked like there was some kind of issue during the upload of the first Reporting Services report “Bug Status” (part of the latest TFS 2015 Agile process template).

First I thought there was something wrong with permissions on the Reporting Services site because I also applied a certificate to offer accessibility via https, but that wasn’t it. All seemed to be setup correctly. A sign to look further into the details of the Bug Status .rdl report definition and why it couldn’t be correctly uploaded to the Reporting Services site. After downloading the process template from the Team Project Collection and opening the report with notepad, I knew I was getting closer to the root cause.

As shown in the picture above, the (default TFS 2015) Data Source mentioned in the report definition did not exist in the Report Server database after the migration from TFS 2010 SP1. In the TFS 2010 timeframe, all report defintions were linked to a “2010” Data Source.

Instead of renaming the existing Data Sources (that would break the existing reports, created from a TFS 2010 Team Project), I duplicated the entries to provide extra Data Sources with the expected names for usage with TFS 2015.