Lately, I’ve been doing a lot of work provisioning and decommissioning LUNs on some Fibre Channel arrays. As you all know, this process can be somewhat tedious when provisioning a new LUN on vSphere or removing said LUN.

In the past, I would always use this PowerCLI command

PowerShell

1

Get-VMhost-Cluster"Cluster1"|Get-VMHostStorage-Refresh

This would first get all the ESXi hosts in the cluster “Cluster1” and then launch the refresh cmdlet on each host. The problem with this is that it can take a really long time for the command to complete because the refresh is executed host by host. Before starting the next refresh, the command waits for the previous one to finish. While this is fine for smaller environments, the time really does add up when you have a decent sized cluster.

Using PowerShell jobs

In order to reduce the time this process takes, I decided this would be a great use case for PowerShell jobs. A job is essentially another PowerShell instance that runs in the background. This means that your main script or PowerShell window doesn’t wait for it to finish before going to the next step. Using jobs will allow us to start the rescan on each host as a job and then start the next job. Meaning the rescans will happen in parallel.

This seems to be a known issue for Lenovo servers. It turns out they cram a whole lot of drivers in there, more than other vendors. The only known workaround is to remove VIBs from the host that are not needed. This is also described in https://kb.vmware.com/s/article/2144200.

To find out what VIBs are installed, log onto the affected host using SSH and run this command:

PowerShell

1

esxcli software vib list

Once you have identified a driver that you want to remove, put the host in maintenance mode and this command:

PowerShell

1

esxcli software vib remove-nqlogic-nx2-provider

In my case I wasn’t using the qlogic nx2 driver so I removed it. Now you should be able to update the host using VUM,

While doing my morning check of our monitoring system I came across a strange issue…Two of our servers had been unavailable during the night, this had also happened the night before. Time to dig a bit deeper!

Both servers are domain controllers running Server 2012 R2, no updates were installed in the last few weeks and nothing had been changed either. Pulling up the event log of both servers, I found the exact same event logged on both machines at the time they were unavailable to our monitoring system.

After I quick search, I found out that aeinv.dll is part of the Microsoft Customer Experience Improvement Program (CEIP). If you’re signed up to the CEIP, which you are by default when you install Server 2012 R2, three scheduled tasks will run each night to upload your anonymized date to Microsoft.

AitAgent

Microsoft compatibility Appraiser

ProgramDataUpdater

We had the unavailability after the first and third task ran.

Luckily you can easily opt out of the CEIP by opening Server Manager, going to Local Server and clicking on the link behind Customer Experience Improvement Program.

There you will see that the radio button, Yes, I want to participate in the CEIP is selected. Click on No, I don’t want to participate and click OK.

You’re now opted out and the scheduled tasks will no longer run.

UPDATE 11/07: Turns out that just opting out of the CEIP does not actually prevent the scheduled tasks from running. In order to fully disable them, you need to disable them through the task scheduler or, even better, disable them through PowerShell

Last week I was moving some VMs for a client using Veeam Quick Migration. During the migration, I happened to stumble upon some strange behavior.

The source VMs make up an Oracle RAC cluster, using shared VMDKs. One of the requirements for setting up shared VMDKs, using the multi-writer option, is that they are thick eager zeroed disks. When the VMs got to the other side, they wouldn’t boot. I figured something went wrong during the migration so I tried it again. Once the second run completed, the VMs still wouldn’t boot so I started digging around some more.

The error message I was getting was rather vague; “Incompatible device backing specified for device ‘0’.”. After verifying the config of both nodes I eventually decided to look at the disk type on the destination side. That’s when I noticed the disk type was thick provisioned lazy zeroed. Ahah, that’s why they didn’t want to boot! After manually inflating the disks, they were up and running again. I’m starting to suspect that this is a bug.

Running some tests

I started building some more test VMs just to prove that this was, in fact, a bug. One of the options you can set during the Quick Migration wizard, is the disk type. You can explicitly select each of the types, or you can have Veeam use the same format as on the source side. Explicitly selecting thick provisioned eager zeroed or using the same as source also produced a VM with lazy zeroed disks. Time to submit a ticket!

As usual, Veeam support was very helpful and investigated the issue. A couple days later they came back to me and confirmed this was indeed a bug that will be fixed in an upcoming version.

Workaround

This bug is a minor inconvenience since there is an easy workaround. You can login to an ESXi server using SSH and convert the VMDK using the command

A customer asked me if it was possible to have a room mailbox automatically accept meeting requests from external parties. They would also like to publish the calendar of that specific room publicly.

Accept meetings from external parties

Let’s start with the first question. By default, resource mailboxes only accept requests from internal senders. As you might guess, you can’t change this behavior through the GUI, Powershell to the rescue!

Since I didn’t know the cmdlet that would let me change this behavior, the first thing I did was look for all “Calendar cmdlets”. After connecting to the Office 365 PowerShell, I ran this command

PowerShell

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

get-command*calendar*

CommandType Name Version Source

----------------------------

FunctionGet-CalendarDiagnosticAnalysis1.0tmp_xse1ew1u.eif

FunctionGet-CalendarDiagnosticLog1.0tmp_xse1ew1u.eif

FunctionGet-CalendarDiagnosticObjects1.0tmp_xse1ew1u.eif

FunctionGet-CalendarNotification1.0tmp_xse1ew1u.eif

FunctionGet-CalendarProcessing1.0tmp_xse1ew1u.eif

FunctionGet-MailboxCalendarConfiguration1.0tmp_xse1ew1u.eif

FunctionGet-MailboxCalendarFolder1.0tmp_xse1ew1u.eif

FunctionSet-CalendarNotification1.0tmp_xse1ew1u.eif

FunctionSet-CalendarProcessing1.0tmp_xse1ew1u.eif

FunctionSet-MailboxCalendarConfiguration1.0tmp_xse1ew1u.eif

FunctionSet-MailboxCalendarFolder1.0tmp_xse1ew1u.eif

Seems like there are a few cmdlets concerning calendars, good info for the second question! The Get-CalendarProcessing cmdlet looks promising, let’s try it out!

PowerShell

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

get-mailbox"test room"|Get-CalendarProcessing|fl

RunspaceId:9f1b6e5d-09a6-40d9-9b83-8006a50d4284

AutomateProcessing:AutoUpdate

AllowConflicts:False

BookingWindowInDays:180

MaximumDurationInMinutes:1440

AllowRecurringMeetings:True

EnforceSchedulingHorizon:True

ScheduleOnlyDuringWorkHours:False

ConflictPercentageAllowed:0

MaximumConflictInstances:0

ForwardRequestsToDelegates:True

DeleteAttachments:True

DeleteComments:True

RemovePrivateProperty:True

DeleteSubject:True

AddOrganizerToSubject:True

DeleteNonCalendarItems:True

TentativePendingApproval:True

EnableResponseDetails:True

OrganizerInfo:True

ResourceDelegates:{}

RequestOutOfPolicy:{}

AllRequestOutOfPolicy:False

BookInPolicy:{}

AllBookInPolicy:True

RequestInPolicy:{}

AllRequestInPolicy:False

AddAdditionalResponse:False

AdditionalResponse:

RemoveOldMeetingMessages:True

AddNewRequestsTentatively:True

ProcessExternalMeetingMessages:False

RemoveForwardedMeetingNotifications:False

MailboxOwnerId:test room

Identity:test room

IsValid:True

ObjectState:Changed

As you can see on the highlighted line, this is exactly the property we were looking for. Let’s change it so we get the desired behavior. In the get-command output, I saw a cmdlet Set-CalendarProcessing, this seems like the right one.

This change will only affect new meeting requests, requests that have already been refused won’t be automatically accepted.

Publish calendar publicly

In the cmdlets we got earlier, there wasn’t really one that stood out as a “possible match” so let’s look at the attributes of the calendar itself. In essence, the calendar is just a folder inside of a mailbox object. Let’s query that folder directly.

PowerShell

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

Get-MailboxCalendarFoldertestroom@domain.com:\calendar|fl

RunspaceId:

Identity:test room:\calendar

PublishEnabled:False

PublishDateRangeFrom:ThreeMonths

PublishDateRangeTo:ThreeMonths

DetailLevel:AvailabilityOnly

SearchableUrlEnabled:False

PublishedCalendarUrl:

PublishedICalUrl:

CalendarSharingOwnerSmtpAddress:

CalendarSharingPermissionLevel:Null

SharingLevelOfDetails:None

SharingPermissionFlags:None

SharingOwnerRemoteFolderId:AAA=

IsValid:True

ObjectState:Changed

That’s everything we need and more! As you can see, we can set the PublishEnabled attribute to true but we can do so much more. You can choose the detail level and even set how far back and forth the published calendar needs to go.

Let’s publish the calendar and run the Get-MailboxCalendarFolder cmdlet again to get the URL.

When I got into the office this morning, I noticed that on particular copy job hadn’t done its` job over the weekend. This particular job copies the daily restore points to a separate scale-out repository and enforces the GFS scheme that’s been set.

The job report displayed

Not that much to go on if you ask me. First, I checked to see if all extents in the repository still had enough room, this was the case. While I was doing that, I verified that all my proxies were still up and running. Before heading to my good friend Google, I decided to remove the copy job restore points from the configuration.

After this, I did a rescan of the repository and retried the job. It ran without a hitch, a nice and easy fix 🙂 I hope this won’t become a common thing, time will tell.

It’s been fairly quiet on the blog front lately, with this post I’m trying to pick it back up again 🙂 I decided to put the blog in a fresh new theme, I really like how it turned out! It looks a lot cleaner now.

2016 in review

When I started this blog last year, I made a goals page that lists everything I wanted to get done that year. A quick review:

Earn the VCP6-DCV certificate –> FAIL. I only managed to get part 1/2 done, I haven’t gotten to part 2 yet, but it is one of the top priorities this year.

Become CCNA –> FAIL. I will be taking this one off the list. When I have some more time, I might pick it up. But for the moment, I feel my knowledge is good enough to get by.

Learn PowerShell –> PASS. Last year I started performing routine tasks using PowerShell. Eventually, I was able to automate some of the things that I had to do frequently. Over the past year, I’ve gotten a pretty good feel for the language and I’m constantly discovering new things! I will be continuing with this until I feel that I have mastered it.

Publish code to GitHub –> PASS. I published 2 scripts that I created. I also made my first ever pull request and added some tests to the awesome Vester project (If you don’t know it, check it out here!). When I get some more free time, I will be looking for some more projects to contribute to.

Attend VMUG(s) –> PASS. I managed to go to both the Belgian as the Dutch VMUGs and I’m hoping to do it this year as well.

2016 was also a big year on a personal level. In august, I changed jobs and started working for Realdolmen as a system engineer. This was one of the best decisions I have made recently, working here gives me the chance to interact with some of the smartest people I know. I get to work with complex and interesting environments and I’m learning new things every day!

But the most important thing I did, was asking my girlfriend to marry me. She said yes and we’re getting married this coming May, I’m very much looking forward to it!

Looking forward to 2017

Obviously the biggest thing for me this year is my wedding. Shortly after that, 29 colleagues and I will be climbing the legendary Mont Ventoux by bike. I’m riding a lot again and found the joy of cycling again.

On a professional level, I will also be setting a few goals for the coming year.

Earn the VCP6-DCV certificate –> This is the first thing I want to get done education wise this year. It’s possible I won’t have time to do this until the summer, though.

Continue learning PowerShell –> There’s a ton I don’t know yet, and a lot that I can do better. I’ve started to put most of my code in functions and I will be looking into building some modules where I can.

Upgrade my MCSA to 2016 –> With the release of Server 2016, it’s time to upgrade my MCSA. I don’t want to let it expire, which would mean I would have to take the first 3 exams again.

Keep the blog more active this year –> Changing jobs in the summer, starting cycling again and preparing for our wedding has eaten up most of my free time since august. I’m hoping to find some more time to keep this blog going with some new content!

Wildcard –> I’m keeping this one open for something else to do the coming year. I’m not entirely sure what it is yet, it will all depend on the amount of free time I have and how the other goals have come this year.

A short list this year, but with a lot going on in my personal life, this feels reasonable.

When running Office 365 in a hybrid deployment, it is possible to have a mailbox both on premises as on Office 365. This can happen when you assign the user an Exchange Online license before the mailbox has been migrated to Office 365.

If the user’s outlook is still configured to use the on-premises mailbox, this can create some funky issues. For example, sent items will be in the on-premises mailbox but new items will arrive in Office 365.

Verifying the mailbox on Office 365

The first thing you have to do is verify if the user has a mailbox on 365, Powershell lets us do this. This command returns all

PowerShell

1

2

3

Connect-MsolService

Get-MsolUser|where-object{$_.userprincipalname-like"*user*"}

Unsyncing user from Office 365

Once you’ve identified the user, that has a mailbox in Office 365, you have to remove the object from Office 365. You can do this by moving the user to an OU that is not synced with AADConnect. After that, run another AD Sync or wait until the next time the scheduled task runs. This sync will remove the user from Azure AD and flag the mailbox as “SoftDeleted”. That puts the mailbox in the recycle bin, there it will stay for 30 days before it will be automatically deleted.

Removing the mailbox from the recycle bin

In order to delete the user from the cloud recycle bin, you can use the following Powershell commands

PowerShell

1

2

3

Get-MsolUser-ReturnDeletedUsers-|FLUserPrincipalName,ObjectID

Remove-MsolUser-ObjectId<GUID>-RemoveFromRecycleBin-Force

Because of the distributed nature of Office 365, it can take up to 15 minutes for the changes to replicate. Now we can hard delete the mailbox

PowerShell

1

Remove-Mailbox-Identityuser@domain.com-PermanentlyDelete

Finishing up

As a final step, move the user back to an OU that is synced. The next time the scheduled task runs, the user will be recreated in Azure AD and will appear in the Office 365 admin center.

A couple of weeks ago we had an issue which resulted in some users getting deleted from Office 365. Because of a sync tool we use, we had to recreate the user accounts instead of restoring them. As a result of this (different SID), the new users also got a new Onedrive for Business library and could no longer access their existing library.

Listing all disabled user profiles

In the Office 365 admin center, go to the SharePoint admin center and click User profiles and Manage User Profiles.

While using various searches, I noticed that every user had i:0#.f|membership in front of their UPN. So I decided to use that as the search term, turns out this lists all users. After changing the view to Profiles missing from import, I got a list of all the disabled users.

Changing Onedrive for Business library permissions

Now that we can see the profiles, we can change the permissions. Click the three dots behind the profile and select Manage Site collection owners.

There you can set the new user as a Site collection administrator. Once you click OK, the users can access the old library using the web version of Onedrive for Business.

There is one caveat to all this, the old profiles will be deleted after 30 days, as part of the Office 365 automated cleanup.

This will be another short one, but I figured someone else will have run into this.

While I was doing a rework for a Veeam implementation, I noticed on several jobs that there were exclusions set inside the jobs. I wanted a list of all jobs with their respective exclusions, time for Powershell!

The script starts by getting a list of all Veeam jobs. Next, it will go through all jobs and look for objects that have the type “Exclude” set. What follows is a bit of code to match the job name to the different exclusions and dump everything into a CSV. I struggled a bit with getting the contents of the array listed properly in the CSV, I kept getting the array listed as “System Object[]“. Turns out I just needed to put the $VMExclusions variable between quotes.

The CSV will look something like this, the job name is listed on the left and the excluded objects on the right.

One last note, this script needs to be run from the Veeam server itself.

As always, you can find the most recent version of the script on Github. The initial version can be found below.