Monday, March 25, 2013

One of my customers experienced an issue where Outlook clients randomly lost their HTTPS connection to the Exchange server. All Outlook clients at my customer connect to the Exchange server using http/RPC rather then TCP (MAPI) both internally and externally. Randomly once a day the Outlook HTTP connection would break and fall back to TCP (internally) or break completely for external users.

Running iisreset would fix the problem but the problem would always re-emerge.

To ensure that Outlook clients retain their connection with the Exchange server using HTTP and not TCP both "On fast networks, conect using HTTP first" and "On slow networks, connect using HTTP first" must be selected.

Activesync and webmail continued to work ok and were not effected by this issue.

This issue was caused by the RPC web application using te Default Application Pool (DefaultAppPool) which is configured to recycle worker processes every 1740 minutes (29 hours). During the recycling process, IIS allows active worker threads an additional 90 seconds to finish servicing requests before IIS terminates the active threads.

Because RPC over HTTP uses long-running connections, the connections may not finish within an additional 90 secosd that were given to the worker threads. In this scenario, the connections are terminated. Therefore Outlook loses connectivity with IIS. When this action occurs, Outlook immediately tries to reconnect. If many Outlook clients are disconnected at the same time, the large number of concurrent reconnections may overwhelm the server.

To resolve this problem create a new Application Pool dedicated to the RPC over HTTP web application with a larger HTTP sys que limit. Please refer to the following TechNet article with instructions on how to perform this procedure:

On my Office 2010 installation, ScanPST can be found in the following path:

C:\Program Files\Microsoft Office\Office14

Below is a screenshot of ScanPST.

Stellar Phoenix labs performed testing with ScanPST and from their testing they discovered that the free PST repair tool is capable of repairing PST files with only minor structural errors. PST files with severe correction or PST files where the indexing table is completely removed, ScanPST will not repair the file.

Stellar Phoenix claim that their tool Outlook PST Repair v4.5 can repair a corrupt PST file and bring it back to a consistent state regardless how severe. I questioned this with Stellar Phoenix as 100% of corruption is a big claim however the company was confident to back it. All content within the corrupt the PST file which is in its valid state can be recovered. Data which has been lost due to corruption is gone, no tool will be able to recover this.

Outlook PST Repair v4.5 has been designed to look like Microsoft Outlook to provide users and administrators with a familiar user experience. When a corrupt PST loaded, all content which is still readable inside the corrupt PST file will be displayed. Companies have the flexibility of recovering individual emails, attachments, sub folders or entire PST files.

Below is a screenshot of the the Outlook PST Repair tool:

To begin using the tool simply the Open an Outlook File to Repair.

Select the location of the PST file which is corrupt. In my case I have a corrupt PST file called test.pst.

Hit the Start button and Outlook PST Repair will go through and scan for all recoverable content.

The tool displays all data which is now recoverable. The user is able to browse mail items, calendars, contacts, tasks, notes everything which can be displayed in Outlook using the Outlook PST Repair tool.

The user is able to do the following things once a corrupt PST file has been loaded in Outlook PST Repair v4.5:

Export all content which is readable within the corrupt PST into a new PST file.

Export select content from a corrupt PST file by selecting what content they wish to export.

Extract attachments from emails

Export individual emails to MSG or EML format

Outlook PST Repair v4.5 comes in a demo version and a full version. What is the difference between the demo version and the full version?

The demo version allows you to see all items which can be discovered, read email and look at calendar items however it does not allow you to extract any information out of the corrupt PST file including attachments, individual items or folders.

The full version allows you to browse a corrupt PST file and export content from a corrupt PST file to a new PST.

There are two licencing versions for purchasing the full version of Outlook PST Repair 4.5. Both licences come are lifetime and come with 24/5 technical support free with the purchase.

Single User Licence ($129 USD). Users receive a key which they use to activate the Outlook PST Repair tool. Once activated the key will only ever work on the Windows instance in which Outlook PST Repair tool was activated. In the event the user purchases a new computer or re-installs Windows, the user must contact support to transfer the licence.

Technician Licence (299 USD). The technician licence can be used unlimited times on different workstations. However a USB key must be connected to the machine to activate the licence and perform the recovery. Only one recovery can be performed at a time. One technician licence must be purchased per office. Stellar Phoenix ship the USB key to the customer upon purchase.

Note: All pricing is subject to change, to get the latest pricing please visit www.stellarinfo.com

Summary

The Outlook PST Repair 4.5 tool is a fantastic tool for fixing corrupt PST files. If Scanpst.exe fails to recover a corrupt PST file or you need to perform granular recovery from a corrupt PST file I encourage you to give Stellar Phoenix Outlook PST Repair a shot.

For more information or to obtain a copy of Stellar Phoenix Outlook PST Repair please visit the following website:

Tuesday, March 19, 2013

Today we required the ability to port forward two public IP addresses both listening on TCP25 to the same internal IP address listening on TCP25. By default a Cisco router will not let you do this. However there is an extenable option which you can put on the end of your command to allow you to do this.

For example to allow TCP25 from both 3.3.3.3 and 3.3.3.4 to 10.1.1.40 on TCP25 we would do the following:

Today one of my customers was listed on the SpamHaus XBL list. The Spamhaus Exploits Block List (XBL) is a realtime database of IP addresses of hijacked PCs infected by illegal 3rd party exploits, including open proxies (HTTP, socks, AnalogX, wingate, etc), worms/viruses with built-in spam engines, and other types of trojan-horse exploits.

My customer had all client workstations access the Internet from the same public IP address as what the Exchange 2010 server relayed email from. Workstations did not connect to the Internet through a proxy, just sandard network address translation (NAT).

My customer did block TCP25 Outbound (SMTP Traffic) from all hosts on the network but the internal IP address of there Exchange 2010 server. Despite this my customer was still added to the XBL.SpamHaus.org blocklist and as a result had difficulties sending and receiving email from many companies especially because SpamHaus is one of the more popular blocklists.

This was because a few workstations on their network was infected with the Pushdo trojan which was performing denial of service (DOS) attacks against target web servers.

Below is the reason why we were RBLed extracted from the SpamHaus.org website:

To get around this problem we changed the outgoing IP address of email, ensured a PTR record exists for the new IP address, updated the Sender Policy Framework (SPF) TXT record on the DNS zone. Finally we updated the port forward on the router and MX records to ensure all mail relay went through a dedicated email.

So what did we learn from this?

If possible always use a dedicated public IP address for relaying mail (if possible)

Use a proxy server for your users to surf the net and block HTTP/HTTPS and other ports if possible outbound to the Internet.

Regarding the Pushdo botnet, we got around to cleaning that up too to ensure my customers network was not used to DOS innocent web servers on the net.

The above event error 12292 it provided us the Provider ID: {5fdb6ef5-6ead-4610-995b-401c88626115}. Looking in the registery under HKLM\System\CurrentControlSet\services\VSS\Providers\{5fdb6ef5-6ead-4610-995b-401c88626115} it shows this provider as the Backup Exec VSS Provider.

For some reason WBAdmin is trying to use the Backup Exec VSS Provider instead of the Microsoft VSS Provider.

I added the registry DWORD UseMicrosoftProvider to HKLM\SYSTEM\CurrentControlSet\Control\BackupRestore with a value of "1" which is meant to force the backup to use the Microsoft provider.

This key had no effect, the backup still attempted to use the Symantec VSS Provider. Next I used the following Symantec article 130940 to completely remove the Symantec backup exec agent from the server including removing registry keys.

After removing the Symantec backup exec agent I ran a test backup and the backup failed again with the same error. Running a "vssadmin list providers" revealed that the Symantec VSS Provider was still in place despite following Symantec article 130940 which was meant to completely remove backup exec from a windows server.

Again we see same GUID of the Symantec provider which was presented in the event error and the registry, {5fdb6ef5-6ead-4610-995b-401c88626115}.

I then followed Symantec article 77585 to completely remove the Backup Exec VSS Provider by deleting the {5fdb6ef5-6ead-4610-995b-401c88626115} key from the following location in the registry:

VSS Event ID 8193 says that the VSS provider was denied access when opening a registry key under the security context of SYSTEM

SYSTEM\CurrentControlSet\Services\VSS\Diag,...).

Damn it cut off! We could use Sysinternals ProcMon to get the full path however lets just force FULL access for tye System account from the DIAG key downwards.

After making this change I then tested another wbadmin. Made no difference. :-(

I searched the entire registry for the GUID of the Backup Exec VSS Provider to ensure nothing was missed. My search found nothing. Whilst I have isolated the problem to the VSS Provider provided by Symantec, a change made by the Symantec Backup Exec agent remains and as a result wbadmin will not function.

If there is someone out there who has fixed this issue can you please comment below with your resolution to ensure others with this issue have a fix as this is not documented anywhere on the Internet.

Tuesday, March 12, 2013

I was about to do a large number of Active Directory changes on a domain controller and needed to grab a system state backup before proceeding. I took a system state backup using the wbadmin utility on a 2008 R2 SP1 domain controller by using the following command from a command prompt:

wbadmin start systemstatebackup -backupTarget:c:

When running the command I received the following error message:

The backup storage location is invalid. You cannot use a volume that is included in the backup as a storage location.

The resolution for this problem was adding a new Key and DWORD value to the system registry. Create the following:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\wbengine.
Create a key
called "SystemStateBackup"
Set the value of this entry as follows:
Name:
AllowSSBToAnyVolume
Data type: DWORD
Value data: 1

After creating the registry key the system state backup now completed successfully.

Tuesday, March 5, 2013

I am currently in the process of deploying the open source Nagios monitoring platform for a customer to provide them the ability to monitor their Windows machines, network infrastructure and virtual environment.

There are a couple of Nagios agents for Windows out there such as NSClient++, NC_Net and WINRPE which all do a great job of extracting Event Logs, Disk Utilization, Process Status, Service Status, Schedule Tasks, Windows Update Status, Anti Virus protection and much more.

In my deployment I chose to implement the NSClient++ on my Windows Server infrastructure. The copy of NSClient++ I'm using is NSCP-0.4.1.90-x64.msi which I downloaded from the downloads page: