Author Archive: Andrew S Higginbotham

OverviewOver the years, I’ve found a common reason for blogging is not only to share information with others, but also to help yourself when enough days have passed that you’ve forgotten your own advice. In my role as a Domain/Practice Lead in our Support organization, there are certain posts of mine that I frequently refer people to as well as find myself using in the field.

With that in mind, here’s a list of some of my most commonly referenced posts, along with reasons why they’ve proven useful:

Note: Like having a resource for Exchange troubleshooting tips? I’d also recommend the Exchange Server Troubleshooting Companion that Paul Cunningham and I wrote. You’ll likely find many of these within it as well.

Info: Active Directory and DNS issues are one of the most common Exchange support issues. When Exchange is having service startup issues or random failures, it’s useful to be able to utilize Event Viewer to determine if Exchange is properly able to access the Global Catalog servers in the environment.

Info: The most useful piece of information from this post (aside from explaining the differences between SMTP Relay and Submit) is the below command I frequently use to check for Receive Connectors that have been configured as an Open Relay:

Info: With Exchange 2013, it became extremely important to ensure you were running on the appropriate (and supported) version of .NET Framework. A quick method to determine this (given to me by my good friend and fellow Exchange MCM Mark Henderson) is to use the below command to pull the currently installed .NET version, then compare it to the versions listed in the post:

Info: Probably my most commonly referenced topic when it comes to Exchange networking; IPv6. Microsoft’s statement is fairly simple when it comes to IPv6 (this goes for every product line), they perform zero testing or validation on Windows with IPv6 disabled. Simply put, good luck with disabling it. The point of my post is that should you choose to disable it, do it via the registry and NOT just unchecking it on the NIC.

Info: Transport Agents are a common cause of mail flow issues with Exchange, at least when they’re misbehaving. At the very least, know how to utilize the “Get-TransportAgent” command and what each Transport Agent does, especially the third-party agents.

Info: This is a fairly common issue with Exchange 2013 (though technically the GUI should now prevent this issue from happening) where two different Exchange Transport services could end up listening on the same port number and causing issues

Info: Working for a hardware vendor, I spent a lot of my time helping customers with their storage solutions for Exchange. I commonly get pulled into Exchange Calculator or Jetstress escalations and this post has become a very useful reference for explaining the importance Controller Caching, even when using an Exchange JBOD architecture.

Info: I could retire if I had a dollar for every hour I’ve spent helping customers overcome corruption issues with Exchange, usually the result of running an ESEUTIL /p and not vacating the database afterwards. This is a great reference I like to send customers and frontline phone agents which describes how to recover from such corruption.

Info: In past conferences like IT Dev Connections, I’ve said that CPU overcomittment and the resulting contention is THE most common Exchange Virtualization support issue I encounter. This is a great article to send to someone who is struggling to understand how CPU overcomittment works and still somehow thinks that virtualization is just magic, where you can give a VM as many resources as you want and it will just work 🙂

Info: These articles are great references when attempting to explain or understand Exchange Transaction logging. This topic is important to understand when working with Exchange Backups, DAG log shipping, and HA recovery.

Info: A common issue when attempting to delete an Exchange Mailbox Database (typically the first one which was created by the system). The important commands to remember from this post are the following:

Info: Going back to my previous statement about improper DNS settings being one of the most common causes of Exchange issues, this post discusses the impact NIC DNS settings can have on an Exchange Server.

Intro: I honestly didn’t expect this to be a popular post, but oddly enough, the topic of Dynamic Distribution Lists is a very common one on the forums. While the issue I experienced wasn’t extremely common on its own, the explanations within the post about how DDL’s work has become a common point of reference.

Info: Having spent a lot of time working with small businesses and SBS, I wrote this post hoping to shine a light on the benefits of the Essentials Office 365 integration tools and how they’re a great alternative to using Directory Synchronization for small businesses. I tend to send this link to customers and colleagues once a month as I’ve found most people don’t even know what Essentials is.

As I opened my Friday session on Dive deep into Microsoft Exchange Server High Availability at Microsoft Ignite, I made mention of it being a session focused on on-premises Exchange. This was met with much applause from the attendees. This actually didn’t surprise me much, as the feedback from customers throughout the week had been their strong desire to see more on-premises sessions (especially for Exchange). During the panel session I participated in on Tuesday, I actually attempted to explain this trend from my perspective.

Before I drag on, I’d actually like to call out the fact that there were several Breakout sessions (at least on Exchange) that had a primary focus on-premises:

I call these out not only for your viewing pleasure, but also to dissuade the idea that the Exchange Product Marketing team has forgotten on-premises Exchange.

So back to my rambling. Throughout the week, myself and other speakers kept getting feedback that attendees wanted more on-premises content. Whether it was via verbal polls during sessions, audience feedback, or in the session evaluations, the message was clear. As someone who still spends the majority of his time working with on-premises customers, I could certainly understand their feedback.

One interesting thing my colleagues and I noticed however was that the majority of attendee questions involved Hybrid or Office 365 in some way. I took this to mean most attendees were at least interested in “dipping their toes” in the cloud waters so to speak. While I understand the cloud isn’t for everyone, I find it hard to envision a world of the future where most customers don’t have at least some kind of cloud footprint.

Not only is change a sensitive topic for some, nostalgia can trigger a very strong emotional response. The explanation I gave in the panel was that it likely feels as though the Exchange on-premises content has dropped off because we just rode a huge wave of Exchange innovation over the past 5-6 years.

Let’s look at just some of the changes the product has seen since Exchange 2010 was released:

Database Availability Groups (removal of Single Copy Cluster as a deployment option)

Death of traditional Public Folders and birth of Modern Public Folders

Death of ISA/TMG

Multiple new mail clients (Outlook App/OWA App/etc.)

Feature rich integration with Lync/Skype for Business/SharePoint

Multitude of new migration options to Office 365 (IMAP/Cutover/Staged/Hybrid/etc.)

Death of Small Business Server and birth of Windows Server Essentials Office 365 Integration Tools

Introduction of MAPI over HTTP

Introduction of Modern Authentication

An extensive list, consisting of many game-changing features which drastically changed the way companies use email and how Exchange Professionals make a living. Before these changes, we were still deploying Exchange on expensive SANs, the only mailboxes in the “cloud” were via hosting companies, nobody cared who Edward Snowden was, and hosted email archiving was a pretty good business to be in. And to think, all this happened within ~6 years. It seemed every post on the Exchange Team Blog had some new drastic change that was altering the way we use email, usually for the better. At TechEd/MEC, you had to decide which Exchange session to skip because so many were available.

This influx of new functionality not only changed the landscape of on-premises deployments, it enabled Office 365 to become what it is today. There’s no way Microsoft could offer reasonably priced high-performing 50 GB mailboxes in Office 365 if they still relied upon traditional backups and SAN storage using 15k disks.

In my opinion, we’ve simply transitioned from on-premises innovations enabling Exchange Online to exist, to Office 365 gracing on-prem with its breakthroughs. There simply aren’t enough new things to talk about in the on-premises Exchange world. Aside from all the new Hybrid goodness (much of which was brought to us by Tim Heeney at Microsoft Ignite), most new on-premises functionality have centered around High Availability (Replay Lag Manager/Read From Passive/Automatic Database Balancing/etc.).

So maybe this post has been a long winded way of saying that Microsoft isn’t ignoring On-Premises, but rather that there’s less to talk about there. I don’t view it as them abandoning their customers, but focusing on where the educational needs are greatest due to rapid growth. Also, it’s hard to blame a publicly-traded company (who have a responsibility to their shareholders to make money) for focusing on what has been to date, one of the most profitable things they’ve ever done.

For the third year in a row, I’ll be speaking in Vegas once again at IT Dev Connections, a week-long conference covering Enterprise Collaboration, Cloud/Data Center, Development, Mobility, Enterprise Management & much more. This year I’ll be presenting:

Discuss migration options for Small Businesses as well as discuss new features Microsoft is rolling out which small businesses may find useful. Cover the best tools for managing a small Office 365 tenant that still has an on-premises presence (via Directory Synchronization or file services). This session will describe options & provide guidance on the following: Cutover vs Staged vs IMAP vs Hybrid migration options for small businesses; Discuss pros and cons of each option; When directory synchronization makes sense and when it doesn’t; Pros and cons of Azure AD Connect and Exchange management; Windows Server Essentials pseudo-Password Synchronization option; Discuss options the Windows Server Essentials role brings to customers and service providers; Discuss features in Office 365 many customers can take advantage of which they may not be aware of.

Taking a spotlight approach to the topics of Performance, Disaster Recovery, and Migration troubleshooting to provide the audience with useful tools/techniques in the field. This session will cover the following topics: Monitoring Exchange in key performance areas to maintain a properly performing Exchange installation; Tools and methodologies for troubleshooting Exchange Performance issues; Common Disaster Recovery scenarios you should prepare for; Methodologies and procedures for navigating Disaster Recovery scenarios; Common hurdles when performing an Exchange migration; Tools and methodologies for troubleshooting and navigating Exchange Migration hurdles.

October in Vegas can be warm at times, but still comfortable. However, the Aria Resort & Casino is always inviting and full of fun. Book your spot now, I hope to see you there & please feel free to chat me up at the conference!

Paul Cunningham and I have spent a lot of time troubleshooting Microsoft Exchange. Not only in our careers as IT Professionals but also in various communities and forums. We’ve often encountered other IT Professionals asking what they can do to grow their Exchange skillset, or learn how to become a more effective Exchange Server Troubleshooter. Maybe it’s someone beginning their careers and wishing to become an Exchange expert, or someone who is established in the IT profession but has recently been charged with maintaining an Exchange environment. Or maybe someone wishing to give their helpdesk staff a bit of a boost in proficiency of handling inbound Exchange cases.

When asked, I’ve often tried to point them to published books, blogs, videos, etc. However, much of what’s available is either scattered in various different places or is written from a product feature or marketing perspective. This past year we began work on a book that focused on troubleshooting Exchange as well as how the product works “under the hood” so to speak. With a focus on function and a layout which allows you to read at your own pace (or to use it as a troubleshooting reference with many relevant links to tools/blog/etc.), I’m very proud of the product we’ve produced.

To create this book, Paul and I spent a lot of time comprising much of our own “tips and tricks”, “do’s and don’ts”, and best practices for Exchange troubleshooting that we’ve gathered over the years. Along with it, we wrote in-depth explanations for how the product works as well as our beliefs of how an IT Professional should go about troubleshooting Microsoft Technologies.

Who is the book for?

New IT Professional expected to manage or support Exchange Server

Seasoned IT Professional who may be tasked with supporting Exchange Server, though they may not have extensive experience with the product

Helpdesk staff required to support Exchange Server or various clients used to connect to Exchange Server or Office 365

IT Professional looking for practical, real-world scenario, training material for Exchange Server

While most of my troubleshooting blogging gets recorded on Exchangemaster.Wordpress.com, my other blogging and various activities are a bit spread out. It was a busy 2015 for me so I thought I’d make a digest here of all my Exchange-related activities for the year.

In less than two months’ time I’ll be flying up to Vegas once again to speak at IT/Dev Connections, a week-long conference covering Enterprise Collaboration, Cloud/Data Center, Development, Enterprise Management & much more. Following-up from my sessions last year, this year I’ll be presenting:

In last year’s session I covered many useful tools & techniques for troubleshooting Exchange Server. Due to popular demand, I’ll be delivering a similar session with a slightly different spin. I’ll be covering commonly seen break-fix scenarios based on real-life customer support calls. We’ll go from identifying the issue, gathering data, resolving the issue, & preventing similar issues from happening again. I’ll showcase useful troubleshooting techniques as well as some lesser known tools to keep in your tool belt.

Much time has been spent debating which storage solution Exchange should be deployed on (often dependent upon cost, simplicity, politics, existing infrastructure, etc.) but this session will instead focus on how to best deploy on the various storage solution options once that decision has been made. I’ll also go into proper storage testing & validation procedures for when it’s time to actually go live with it all.

So while Vegas will likely be pretty hot the week of Sept 14th-17th, The Aria Resort & Casino is quite comfortable. Book your spot now, I hope to see you there & please feel free to chat me up at the conference!

I managed to get the session down to around 60min but I still kept all the content in the deck. There’s a ton of information in the slide notes as well as many hidden slides, so be sure to download the deck afterwards if you’re an attendee. If you’re not an attendee, then how’s a last minute trip to Vegas sound?

However, I decided to use this blog to expand on a topic I just couldn’t do justice within a 60min time frame. Hopefully this can give people a look at the type of content being presented at Exchange Connections as well as a starting point if they’d like to grow their troubleshooting skills. So in this post I’d like to cover common break-fix issues seen with Client Access Servers; even though technically some of these components live in the Mailbox Server role now.

Overview

The first step to troubleshooting any technology, I feel, is to understand the functionality of its core components during normal operation. Often time people are given a set of tools to be used in troubleshooting but never truly understand how to interpret the data they’re looking at. Similar to how using NetMon will be of little use to someone who doesn’t have a solid understanding of TCP/IP, looking at Exchange Client Access or IIS data will not prove useful if you do not understand how each of the components interact with each other. Let’s begin by looking at IIS.

What we see above is the IIS Manager on my Server 2012 R2 Exchange 2013 CU6 multi-role server. We find the various web sites as well as the Application Pools that correspond to each application like ActiveSync, PowerShell, or OWA. Because this server is multi-role (has both CAS & Mailbox Roles installed) you will see two separate Exchange web sites:

Default Web Site = Client Access Server Role

Exchange Back End=Mailbox Server Role

The two main services associated with IIS are the IIS Admin Service (inetinfo.exe) & the World Wide Web Publishing Service (w3wp.exe). To oversimplify it, inetinfo.exe corresponds to IIS configuration information whereas w3wp.exe corresponds to each of the various Application Pools. After changing IIS configuration information (like Auth Settings, etc.), the IIS Admin Service will typically be what you’ll want to restart. Whereas, if a particular application still isn’t updating after you’ve made a change (like OWA or ActiveSync) then you may need to Recycle that Application Pool & at worst, restart the WWW Publishing Service.

However, in many cases it’s recommended to simply stop/start the website or recycle the application pool rather than restarting the services or using iisreset (Reference-AReference-BReference-C). This is because it’s possible IIS has not saved the necessary changes in time & those changes could be lost by a forcible service restart. Starting/Stopping the websites, recycling the application pools, or using the “/noforce” switch for iisreset is preferred. However, sometimes killing a service is all you can do in a troubleshooting scenario.

Web Sites & Application Pools in Exchange 2013

When troubleshooting IIS, I commonly find myself looking at the Web Site Bindings. These are what “bind” an IP Address, Port Number, Host Name, & (potentially) a Certificate to the web site. Let’s look at the bindings using both PowerShell as well as the GUI.

Using the above series of commands (reference) I was able to import the IIS PowerShell Module & query the bindings of my two web Sites in IIS. I’ve found that using PowerShell is a very handy way to query this data fairly quickly. It’s also useful for when you need to send a customer a set of commands they can run & send the data back to you. Here’s a few of my preferred information gathering commands:

The above series of commands has me navigating to the “Default Web Site” & viewing the various Applications & Virtual Directories underneath it. Notice how the commands work similar to navigating a folder structure. If I need to go back a level I can simply use “cd ..”. Alternatively, if I wanted to export this to a text file I could repeat the last command but with a Format-List at the end “dir | fl > C:\IISOutput.txt”. This can be useful when comparing a known working server to a problematic one. Of course there’s also any number of ways this data can be scripted/manipulated/etc. to fit your needs.

Note: The Default Web Site has bindings of 80 & 443 for HTTP & HTTPS while Exchange Back End has 81 & 444. When you make a connection to Exchange using HTTPS you’re connecting to the Default Web Site & it’s proxying it back to the Exchange Back End web site. Do not change the bindings on the Exchange Back End website.

Now if I go back to the root I can see a list of all the Application Pools in IIS.

Alternatively you could just use the Exchange Management Shell for some of these commands but you might find the IIS Module gives you a bit more flexibility.

Now to look at these settings in the GUI may seem easier but it does require a bit more mouse clicks to get all the same data:

How it can break

Bindings & Firewalls

So we know how things are laid out but now let’s look at what I most commonly see broken from customers. I’ve always said one of the best ways to learn something is to break it in a lab 🙂

Excluding Certificates (which I’ll discuss later), the most common IIS-related issue I see is related to IIS Bindings. I’ll commonly encounter customers trying to install a 3rd party application or a Microsoft application that is not explicitly supported on an Exchange 2013 Server (SharePoint, RDS, Lync, etc.) & in the process their bindings will get messed up. Allow me to demonstrate.

Say I’m logged into OWA on my multi-role 2013 server.

Now within IIS I right-click Exchange Back End>Edit Bindings & change the HTTPS binding from 444 to 445

If I now refresh my browser I’ll be greeted with a blank page.

This is because, by design, the Default Web Site has the traditional web server bindings for port 80 & 443, while the Exchange Back End website uses ports 81 & 444 for HTTP/HTTPS connectivity. When the Client Access Server role is communicating with the Mailbox Server role for IIS –related functions, it proxies these connections via HTTPS using port 444. So the expected flow for UserA logging into OWA on ServerA (single server environment for this 1st example) would be:

As you can see, in this scenario while the client connects to OWA using 443, CAS proxies that connection to the relevant Mailbox Server over 444 (over the network). If you really want to see this in action then you can use a tool like NETSTAT to view connections between your servers:

In the below example I see a local connection to 443 & an associated Process ID (PID). I can use Task Manager to see that PID correlates to an instance of Internet Explorer (iexplore.exe), which I have open & connected to OWA (https://127.0.0.1/owa).

The below command was run from the same server but for port 444; this output is quite a bit busier. There’s the connection to the local server for the OWA session that I’m logged into (the mailbox I’m logged in with is on a database that’s mounted locally). However, you’ll also find there’s a connection to 10.180.62.191, which is one of my other Exchange servers in the environment. This is for another instance of OWA I have open for a mailbox that’s currently mounted on that server. In that case the PID corresponds to an instance of w3wp.exe (World Wide Web Publishing Service). The other PIDs correspond to background processes like Microsoft.Exchange.ServiceHost.exe (MSEXchange Service Host Service), MSExchangeHMWorker.exe (MSExchange Health Manager Service), & MSExchangeMailboxAssistants.exe (MSExchange Mailbox Assistants Service). These are all background processes that are constantly running behind the curtains to keep Exchange up & running (synthetic transactions, maintenance tasks, etc.).

So it’s fairly common to see customers accidentally change the bindings or delete them. Unfortunately, their attempts to repair the web sites typically result in them using the incorrect port numbers (like putting 443 on the Exchange Back End site). Alternatively, customers (or their network security admins) may block port 444 traffic between servers & suddenly find their servers in a state of sad uselessness.

Recreating the various Virtual Directories has been a useful troubleshooting step in the past but I’ll be honest when I say that it’s usually done as a last ditch step whenever every other avenue of troubleshooting hasn’t helped. In fact, if recreating the vDir doesn’t resolve the issue I’m usually looking at a /RecoverServer install as the next step. But it has been useful when OWA/ECP/ActiveSync/EWS/OAB/PowerShell/AutoDiscover don’t work as expected & you’d like to reset the relevant Virtual Directory to defaults.

Note: Recreating the Virtual Directories will reset any settings or customizations you have done to it so I recommend running a “Get-OWAVirtualDirectory | FL” or similar command beforehand to grab the existing settings. In fact, if you use EAC to reset the VDirs then you’ll be prompted to save the configuration to a network path.

There are two ways to perform this action, EAC (GUI) or EMS (Shell). Let’s look at the EAC method first:

You can go to EAC>Servers>Virtual Directories, select the Virtual Directory you wish to reset & then click the Reset button.

Here we see the prompt you’ll receive to backup the current Virtual Directory settings before resetting it.

After clicking “Reset” the Virtual Directory will be removed & then recreated. Afterwards you’ll need to restart IIS.

Now how would we do this with shell? It’s fairly simple:

Now this works when we have an issue with the Default Web Site but I’ve actually run across a case where I had to recreate the OWA Virtual Directory on the Exchange Back End site as well. To do this I would run the below commands:

PowerShell

Now what if you’re having issues with the PowerShell vDir? You likely can’t connect to that server to manage it via EMS or EAC so you’re going to have to load the local PowerShell snap-in using the below commands:

Since we’re on the topic of PowerShell, on occasion I’ve found myself having to verify all the proper Modules are added for the PowerShell vDIR.

The best advice I can give you is to compare the loaded modules here to a known working server (or lab machine). On several occasions I’ve found the kerbauth module to be missing & I’ve had to re-add it. I saw it on several occasions in Exchange 2010 but not yet in 2013; but regardless, the proper modules will be needed in order for things to work properly on any version of Exchange.

Note: Also make sure that any & all file directory paths have the proper permissions set on them. Again, it’s helpful to have a known working server to use as a comparison. Also, be sure that all proper Anti-Virus Exclusions have been configured (extremely common scenario). (Reference)

Certificates & Naming

By far, Certificates are the most common CAS/IIS-related support issue I see; which is odd considering the core concepts are not that difficult. Much like understanding core TCP/IP functionality, I feel core PKI & SSL knowledge should be something every IT professional should learn early on in their careers.

You don’t have to be an expert but you should understand the 3 golden rules of trust: Do I trust the issuer of this certificate? Is the certificate expired? Is the name I’m using to connect to this service listed on the certificate?

Knowing these things will help us to understand which names we need to put onto our Exchange certificate when requesting it. You can technically get away with only having 1 name on your certificate in a simplistic environment with limited requirements (which also seem to be the environments where less experienced customers are unsure of their options). For instance:

In this example, everything would work except for non-domain joined Outlook clients & ActiveSync automatic profile creation. This is because you won’t have AutoDiscover.Contoso.com on your certificate so the process will not be seamless. You’ll either be greeted with certificate warnings or the connection just won’t work. Now technically you can get non-domain joined Outlook clients to work if you create an SRV record for AutoDiscover but there’s no workaround for ActiveSync. Your users will have to manually enter in the server name when creating ActiveSync devices. Also, depending on how your device handles certificates, you may or may not be able to connect.

Example-B (never seen it in the wild but it would technically work fine)

Of course the downside of this configuration is your users would have to use https://autodiscover.contoso.com/owa to access OWA & I haven’t found a customer yet who was willing to do that. However, all services would work, including Outlook/ActiveSync profile autoconfiguration.

I brought these examples up not to tell you how to deploy Exchange (by all means, get a multi-name or wildcard cert) but instead to explain that in the end, all that matters is that the names you configure in Exchange are resolvable to CAS & listed on the cert. You could literally make your Outlook Anywhere namespace “randomseriesofcharacters.contoso.com” & as long as it was on your cert & as long as the name resolved to CAS then it would work.

If you remember nothing else about certificates, just remember Do I trust the issuer of this certificate? Is the certificate expired? Is the name I’m using to connect to this service listed on the certificate?

Certificates-Miscellaneous

Certificates are bound to both the Default Web Site as well as the Exchange Back End site in IIS. If you right-click on Default Web Site>Edit Bindings>Select HTTPS & click Edit you can see the current certificate bound to the site. When you run “Enable-ExchangeCertificate –Thumbprint <Thumbprint> -Services IIS”, this is what it configures. The image below shows my certificate generated by my internal Certificate Authority:

I often see the incorrect certificate listed here or I may see certificates missing. Many customers mistakenly think that the Exchange tools are the only way to Import/Export certificates, but the Certificates MMC Snap-In is a very handy troubleshooting tool.

Below is the Local Computer account’s Personal Certificates store; where manually installed certificates are likely to be stored. In short, when you run “Import-ExchangeCertificate” the certificate ends up here. So similarly you can use this console to Import/Export certificates as well.

Note: Your Personal store will likely look different than mine as my lab server is also a DC/CA.

Certificate issues have historically revolved around generating the request, but the Certificate Request GUI’s found in Exchange 2010 & 2013 have made those customer calls much easier.

However, a problem I still see with customers is that don’t understand that when you generate the certificate request on the Exchange server, you need to leave that request intact until you receive the new certificate from your issuing Certificate Authority. If you don’t then your certificate will be missing the private key & be effectively useless. I see this frequently when customers are requesting a certificate multiple times or if they try to use a different server to import the cert on than the one they issued the request from. Once a request has been generated, you’ll see the pending request in the EAC Certificates console; along with an option to Complete the request when you’ve received the certificate from your CA (this process generates the Private Key).

Additional Logging

Lastly, I can’t leave out the plethora of logging that’s now present in the install directory (typically C:\Program Files\Microsoft\Exchange Server\V15\Logging) of every Exchange Server. In fact, the logging is so vigorous that you’ll often find it taking up quite a bit of your disk space. Luckily there are methods to truncate unneeded logs. These logs have come in handy when I’ve had to troubleshoot odd issues in the past related to CAS proxy behavior. I’d suggest taking time to look through these logs using notepad or even better, Log Parser Studio. It’s a tool frequently used by Microsoft Support & great for when you have to parse through many log files trying to find a needle in a haystack.

Conclusion

As this post has already grown quite long, I suppose we’ll end it there. I’m sure I could find something to continue rambling on about but hopefully I’ve done this topic enough justice. If you make it to Dev Connections then I’d be happy to chat with you sometime during the conference about any other oddities that surround the world of troubleshooting Exchange.

This article will see us use Server 2012 R2 Essentials to integrate with Office 365 for Exchange. This will be one of the options I’ll discuss during my session titled “Exploring options for moving a small Exchange or Small Business Server environment to Office 365 or remaining on-premises” at Exchange Connections 2014 in Las Vegas Sept 15th-19th. It’s a long title for a session but it’s fitting considering the many options you’ll have for you or your customer’s business.

First we should know what Server 2012 R2 Essentials is. You could say it’s the spiritual successor to Small Business Server 2011, which was essentially (pun intended) discontinued. For a detailed walkthrough of Windows Server 2012 R2 Essentials, watch this great multi-part video series on it from Microsoft. It’ll answer all your licensing, deployment, & configuration questions.

History & Now

The last version of Small Business Server was SBS 2011. This was a Server 2008 R2 Domain Controller which ran Exchange 2010, along with WSUS, File Services, SharePoint Foundation, & SQL. The solution had several other cool features like Remote Web Workplace which was a web portal (running on top of Remote Desktop Services) that allowed any remote user to connect through a web interface & remote into a client machine on the company LAN. It also included a singular management interface named the SBS Console where Admins could manage Users, Groups, Mailboxes, Backups, Alerts, & Updates. This solution was aimed at small businesses (as the name implied) with less than 75 users.

Fast forward to today & we have Server 2012 R2 Essentials which has new features but also is a bit lighter. Exchange, SharePoint, & SQL are no longer included with the product; however you can still utilize these applications either on-premises (on another server) or in the cloud. The Windows Server Essentials Dashboard allows you to perform many of the same user management functions as the SBS Console, as well as manage mailboxes if you’ve chosen to run Exchange on another server on-premises or have connected your server to Office 365 (more on that later). New with 2012 R2 is the ability to install Essentials as a Server Role on Server 2012 R2 Standard or Datacenter; this is called the Windows Server Essentials Experience Role. See the aforementioned video series to learn about all the additional features of 2012 R2 Essentials.

Our Goal

In this article I’ll be Installing the Windows Server Essentials Experience Role, configuring the environment as if I were a small business, & connecting my environment to an Office 365 Tenant. I chose to use the server role (instead of a full Essentials OS install) in this guide for my own convenience. The experience should be the same, it’s really just the licensing that differs. Once we’ve performed these steps we’ll explore management of users; most notably, managing Office 365 mailboxes from the Essentials Dashboard. We’ll also see how passwords are synched to O365 (yes that’s right, Essentials had PasswordSync before the full version of DirSync did!).

Proceed to the Select Server Roles page & place a checkmark next to Windows Server Essentials Experience. Take note of the additional Features that will be installed. Select Add Features & click Next a few times & then click Install. Reboot afterwards.

Exploring & Configuring Essentials

Upon logging back into the server post-reboot, open Server Manager & click the flag in the top-right corner of the screen indicating post-deployment tasks requiring action. Click Configure Windows Server Essentials

Take note of the initial page & click Next.

As part of the Role configuration, your server will be made into a Domain Controller of a new Active Directory Domain (if you do not already have a domain). In this case I’ll put “ASH-Exchange Inc” for my Company Name (a fake Company Name I chose in hopes of not being sued 😉 ). I’ll also put “Ashdrewness” as the Internal Domain Name (I own ashdrewness.com; replace that domain in this guide with a domain you own). Click Next.

When asked for an Administrator account I put “Andrew-Admin” & entered my preferred password twice. Click Configure. This next phase may take up to 30min, feel free to grab your favorite beverage. The server will eventually reboot.

After the server has rebooted you can log back in with the account you were using to perform the install/configuration (Administrator in my case) or the account you just created. After opening Server Manager you’ll be greeted with a dialog box saying the configuration is complete. Click Close.

My server is now a Domain Controller for the Ashdrewness.local domain. From the Desktop, click the icon for the “Dashboard.” You’ll be greeted by the Essentials Dashboard.

Now at this point there are many things I could setup on the server from within this wizard such as Backup, Anywhere Access, & adding computers to the domain; but for now I’ll just go straight to the “Services” section of the Dashboard.

With “Services” selected, click “Integrate with Office 365.” Then click “Integrate with Office 365” again. This wizard will help us setup integration with an Office 365 tenant. You can either have a tenant already created or you can use the wizard to create one (or create a trial tenant). We’ll use the wizard to create our tenant now.

Click Next & then click “Try” under “Microsoft Office 365 for midsize business and enterprises”. On the page it takes you to, input the requested information. On the next page I’ll create a username of “Andrew” & put “ashdrewnesstrial” as my company name. After creating a password click Next. You may be asked to provide a phone number for text message verification to prove that you’re not a bot. Afterwards click “Create my account.”

Afterwards you’ll be given some information to write down, similar to :

Click on “You’re ready to go” to be taken to the main dashboard of your Office 365 tenant. (Note: as time goes on, some of these O365 menu options may change so please use your best judgment to get through them. Also, If your browser crashes or you need to get back to the O365 Admin portal then just login to https://portal.office.com using the account you just created; andrew@ashdrewnesstrial.onmicrosoft.com in my case.)

From the top of the Dashboard click Admin>Office 365. On the next page click “Domains”. Click “Add Domain”.

I own “ashdrewness.com” so at this point I’ll add it to this Office 365 tenant (Note: a domain can only exist in one Office 365 tenant at a time). Use this wizard to add your domain. In my case the domain is hosted by GoDaddy so I’m prompted to have the Office 365 wizard confirm ownership for me by creating a randomly generated TXT record. I will be prompted for my GoDaddy credentials & the required records will be created for me. Click Finish when complete.

At this point of the wizard I’ll be asked to add users & assign licenses. Click “Start step 2” & say “I don’t want to add users right now” & then Next.

Click “Start step 3” & then click Next to say I wish to use this domain for Exchange & Lync Online.

The next page will inform you that a strong password policy will be configured. Check the box & click Next (by doing this, users may be prompted to change their passwords).

After completing the wizard & restarting the Dashboard, click on “Services” again to see that your Windows Azure Active Directory & Office 365 integrations are now Enabled.

Matching your Active Directory Users with an Office 365 Mailbox

Click on “Users” within the Essentials Dashboard & then select John Smith. In the right-hand pane click “Assign a Microsoft online account”

On the first page of the wizard, leave the default selection of “Create a new Microsoft Online Services account & assign to this user account”. Click Next

On the next page, at this time just select Exchange & Lync (while I’m sure all your users will be anxious to start using Yammer, they can be patient). Click Next.

The next page should provide verification that the account has been created & will also notify you that John Smith will be prompted to change his credentials the next time he signs in. Click Close.

At this point if I look at the user account of John Smith in Active Directory I can see that his User Principal name has been changed from JohnSmith@ashdrewness.local to JohnSmith@ashdrewness.com. Also, if I open the Office 365 Admin Center I’ll see that John Smith has been added to the tenant with a Status of “in cloud” which is different than it would show up if it were synched from my local Active Directory using DirSync. This is because Essentials does not use DirSync but its own custom framework for synching users to the cloud & keeping Passwords in sync (in fact, it could do Password Synchronization before DirSync could). Speaking of passwords…..

Now if you were to try & login to the Office 365 portal right now as JohnSmith@Ashdrewness.com you would get a login error. What you must first do is either try to login to your local Active Directory environment, have it prompt you to change your credentials, change them to a password that meets your new complexity policy OR an admin could use the Essentials Dashboard or Active Directory Users & Computers to reset the password.

Afterwards, connect to https://portal.office.com & login using JohnSmith@Ashdrewness.com with your new credentials. You’ll be presented with the below page. Note: In my testing, it is a matter of seconds between changing a password on-premises & having it take effect in the Office 365 tenant. This is much faster than DirSync; however using the Essentials Office 365 features (and its own custom web-based sync service) is only supported in a single Domain Controller environment, aka small environments (UPDATE: It is now supported in multi-DC environments per this article).

We’ll now act as if we already have an Office 365 mailbox created & now we wish to map it to an on-prem AD user account. For instance, this would be the case if you had migrated from another version of Exchange to Office 365 using a Cutover Migration or similar.

Login to the Office 365 admin portal with Andrew@ashdrewnesstrial.onmicrosoft.com & navigate to “Users and groups”. Click the plus symbol to create a new user called “Mike Smith” with a user name of “MikeSmith”.

Now back in the Essentials Dashboard, select Mike Smith & then click “Assign a Microsoft online account.”

This time select “Assign an existing Microsoft Online Services account to this user’s account”. Notice how a drop-down will appear which will automatically populate with your tenant accounts which have yet to be mapped to an on-prem user account. Select MikeSmith@ashdrewness.com & click Next.

Managing your Office 365 properties from the Essentials Dashboard

From the Users pane of the Essentials Dashboard, double-click John Smith’s user account. On the Properties of John’s account, click the “Microsoft online” tab. Notice it’ll take a second to connect.

Notice you can do things like block access to the service, add additional email addresses, & assign/un-assign services to the account. Close out of the Properties of John’s account.

Within the Users pane of the Essentials Dashboard, click the Distribution Groups tab & then click “Add a distribution group”.

This company has Exchange 2013 installed & are using Modern Public Folders. They have a business process where customers email a mail-enabled public folder & have setup a rule (using the Folder Assistant) to fire once the email has been received. This rule sends an email to the customer essentially saying “Thank you for your email, we will get back to you shortly”.

Employees then use the message Categorization feature to assign a Category to the email so it can then be tracked accordingly. Unfortunately, once the message is marked with any category, the rule fires again & sends another email saying “Thank you for your email, we will get back to you shortly” to the customer. Say for instance the categorization is changed 3 times, then the customer would receive 3 additional emails (quite SPAMMY). Issue happens when using Outlook (tried both 2010 & 2013) as well as OWA.

This environment recently moved from Legacy Public Folders (03/07/10) where this issue did not occur with the same configuration. I was also able to reproduce this issue in my lab; I’ve detailed the process below:

Issue Reproduction

I duplicated the configuration in two different environments, one with Exchange 07/10/13 & Public Folders on Exchange 2007 (Legacy Public Folders) while the other environment was purely Exchange 2013 SP1 using Modern Public Folders. In either scenario the configuration was the same:

Create a Public folder named “TestPF”, Mail-Enable it, & grant an account Owner permissions for it.

Open Outlook as the Owner account & navigate to the TestPF public folder.

Select “Reply With” & then click “Template”. Within the Outlook window that opens, compose the template you wish to send out & then click “Save & Close” (see image below).

Click “Ok” twice.

Now compose an example customer email to be sent to the public folder’s address

You should then receive an email from the public folder’s SMTP address containing the text you placed into the template earlier.

Now navigate to the TestPF public folder. On the newly received customer email, right-click, hover over Categorize, & select a Category (see image below).

10. Shortly after assigning a Category you (the customer in this case) should receive an identical template email which the Folder Assistant fired off again.

If you perform these same steps on a Legacy Public Folder, the issue should not occur (no duplicate emails after changing the Category).

Workaround

I initially tried sending this to some internal Microsoft people via the Ranger & MVP DLs but never received a response. It seems like one of those odd corner cases that would be hard to get traction on. I suggested opening a MS Support case to attempt to get it classified as a bug. Unfortunately not everyone has a Premier Support contract & given there’s no guarantee of MS agreeing something is a bug (thus refunding their money) the company was hesitant to create the case & invest the time. This decision was made after they determined that a workaround for this issue was to create a subfolder called “Received” (which did not have any Rules configured) & move the emails there before applying any message Categorization.

Note: An important concept to understand is that every bug that needs to be fixed is a business decision. It takes extensive development as well as testing/validation effort to make it happen. So knowing that, it is almost always necessary to write out not only the problem description but also a business case for why a certain behavior is negatively impacting your business. It may seem a bit tedious but this is the type of ammunition needed by Support to get traction on an issue with any development team at Microsoft.

Conclusion

This is certainly an odd issue but I wanted to put it out there in the event someone used a similar process & were either currently having issues or were planning the move to 2013 Modern Public Folders.

First off, I still plan on blogging the troubleshooting related issues I come across at exchangemaster.wordpress.com while this blog will be for other Exchange/Office 365-related topics (guides/technical deep-dives/etc). My good friend, fellow MCM/MCSM, & co-worker Jedidiah Hammond started that blog several years ago with the intent of helping the Exchange community’s troubleshooting efforts & I wouldn’t want to deviate from that goal with some of my own occasional projects.

Exploring options for moving a small Exchange or Small Business Server environment to Office 365 or remaining on-premises

With the retirement of Small Business Server (SBS) & customers no longer able to
purchase SBS 2011, small businesses will be exploring their options going forward.
This session will describe options & provide high level guidance for:
-Supported migration options for moving Exchange to O365 for SBS 03/07/10 (PST
vs Cutover vs Staged vs Hybrid vs IMAP vs 3rd party) with pros/cons of each.
-Which services to take to cloud vs leaving on-prem (Exchange/SharePoint/AD/File
Storage)
-What benefits does Server 2012 Essentials bring & how to scale up from a single server
deployment; Including virtualizing Essentials/Exchange/SharePoint as
separate VMs on same host. Risks/benefits of each.
-The dangers of trying to recreate your own “2013-SBS” by placing
Exchange/SharePoint/RD Gateway/AD/WSUS on same box.
-How to handle authentication (DirSync with Password Sync vs Separate
Credentials vs ADFS)
Session will be high-level & aimed at both small business owners as well as
consultants who work with many small businesses. Session will also include
lessons learned via support with customers already exploring the various options.

Advanced troubleshooting procedures & tools for Exchange 2013

Using lessons learned from support, as well as insights from Microsoft, this session
will focus on the following:
-Proper troubleshooting methodology
-Common support issues since the release of 2013 & what can be learned from
them
-(bulk of session)Under-the-hood look at various core & supporting components of
Exchange functionality (Client Access/Transport/Mailbox/High Availability/Unified
Messaging/Active Directory/Operating System resources/Hybrid) & the logs/tools
required for troubleshooting them.
-Resources to follow to be updated on emerging issues (blogs/social media/etc)
-Time permitting, practical examples of beginning to end troubleshooting of
Exchange components.