Archive

So this was one of those posts that I always wanted to write related to Symmetrix V-Max and SRDF enhancements that were incorporated with the 5874 microcode.

Yesterday morning had a chat with a friend and ended up talking about SRDF and then later in the day had another interesting conference call on SRDF with a potential customer. So I really thought, today was the day I should go ahead and finish this post.

Back in April 2009 when the V-Max systems were initially launched, Storagezilla had a post on V-Max and SRDF features, he covers quite a bit of ground related to the Groups and the SRDF/EDP (Extended Distance Protection).

Here are the highlights of SRDF for V-Max Systems

SRDF Groups:

250 SRDF Groups with Symmetrix V-Max (5874) Systems. In the prior generation Symmetrix DMX-4 (5773), it had support for 128 groups. Logically even with 2PB of storage, very seldom do customers hit that mark of 250 groups.

64 SRDF groups per FC / GigE channel. In the previous generation Symmetrix DMX-4 (5773), there was support for 32 groups per channel.

SRDF Consistency support with 2 mirrors:

Each leg is placed in a separate consistency group so it can be changed separately without affecting the other.

Active SRDF Sessions and addition/removal of devices:

Now customers can add or remove devices from a group without invaliding the entire group, upon the device becoming fully synced it should be added to the consistency group (with previous generation Symmetrix DMX-4, one device add or remove would cause the entire group to invalidate requiring the customers to run full establish again).

SRDF Invalid Tracks:

The “long tail” – last few tracks search has been vastly improved. The search procedure and methods for the “long tail’ has been completely redesigned. It is a known fact with SRDF, that the last invalid tracks take a lot of time to sync as its going through the cache search.

The SRDF establish operations speed is at least improved by 10X; see the numbers below in the performance data.

Timefinder/Clone & SRDF restores:

Customers can now restore Clones to R2 and R2’s to R1’s simultaneously, initially with the DMX-4’s this was a 3-step process.

SRDF /EDP (Extended Distance Protection):

3-way SRDF for long distance with secondary site as a pass through site using Cascaded SRDF.

For Primary to Secondary sites customers can use SRDF/S, for Secondary to Tertiary sites customer can use SRDF/A

Diskless R21 pass-through device, where the data does not get stored on the drives or consume disk. R21 is really in cache so the host is not able to access it. Needs more cache based on the amount of data transferred.

So these lights are USB powered through USB cables, a very simple idea though a genius one. Enterprise class arrays and the use of USB ports at the backend of these systems keep these Blue flashy lights on.

Here are the pictures of those USB connectors from the front. Believe it or not, they are redundant as well.

(Look at the USB cable connectors that go on the front door, 2 of them right above the blue light)

As Storage Anarchist says in his Blog post“The first thing you’ll probably notice about the new Symmetrix V-Max is the packaging – and specifically the glossy-black panel with the blazing blue LED light bar that underscores the name Symmetrix on every door. The design team had a lot of fun blending the modern gloss-black look of today’s popular personal communications devices with the image of stability and security that customers expect from Symmetrix.”

Yea that is right,,,,,,,,,, this post was about the Blue lights on the Symmetrix V-Max Systems

Oh…the big question, will it call home through the EMC ESRS Gateway if one of these Blue lights fail?

There are two types of sparing strategies available on EMC Symmetrix Series of machines.
Dynamic Hot Sparing: Starting the Symmetrix 4.0, EMC had introduced dynamic hot spares in its Enginuity code to support customers against failing disk drives and reducing the probability of a data loss. Available there onwards on each version of Symmetrix, customers have been able to use this Hot Sparing technology. Today the Dynamic sparing is available on Symmetrix 4.0, Symmetrix 4.8, Symmetrix 5.0, Symmetrix 5.5, DMX, DMX2, DMX3, and DMX4 systems.

Permanent Spares: Was introduced starting the Symmetrix DMX3 products, now available on DMX4’s and V-Max systems. I believe, Enginuity code 5772 started supporting Permanent Spares to guard customers against failing disk drives to further help reduce any performance, redundancy and processing degradation on the Symmetrix systems with features that were not available with the Dynamic Hot Sparing.
Highlights of Permanent Sparing
Due to some design, performance, redundancy limitations and Symmetrix mirror positions, dynamic hot spares were becoming a bottleneck related to customer internal job processing, example: a failed 1TB SATA drive sync to dynamic spare might take more than 8 to 48 hours. While a similar process to remove the dynamic spare and equalize the replaced drive might take the same. During this time the machine is more or less in a lock down (Operational but not configurable).

Due to these limitations, a concept of Permanent spares was introduced on EMC Symmetrix systems, which would help fulfill some gaps the Dynamic hot spares technology has. Following are the criteria for Dynamic Hot Spares.

Supported through microcode (Enginuity) version starting Symmetrix Family 4.0, support extended through all later releases of Enginuity until DMX-4 (5773).

Dynamic Hot Spares configured and enabled in the backend by an EMC CE.

No BIN file change is performed as the Dynamic Hot Spare gets invoked or removed upon a disk drive failure.

No BIN file change is allowed until the Dynamic Hot Spare is removed from the active used devices pool and inserted back into the Spares pool.

An EMC CE will need to attend site to replace the failed drive and put the dynamic hot spare back in the pool of devices available for sparing.

Enginuity does not check for performance and redundancy when the dynamic hot spare is invoked.

In the previous generation of Symmetrix systems, an exact match (speed, size, block size) was required with Dynamic hot spares. Starting I believe the 5772 (DMX3 onwards) version of microcode that requirement is not necessary. Now larger or smaller multiple dynamic spares can be spread across protecting multiple devices not ready, the one to one relationship (failed drive to dynamic spare) is not true any more.

Related to performance on DMX3 systems and above, if correct dynamic spares are not configured, customers can see issues around redundancy and performance. Example, A 10K drive can be invoked automatically against a failed drive that is 15K causing performance issues. Also a drive on the same loop as other raid group devices can be invoked as a hot spare, potentially causing issues if the entire loop was to go down.

Dynamic spares will not take all the characteristics of failed drives. Example, mirror positions.

While the Permanent Spare or Dynamic Hot Spare is not invoked and is sitting in the machine waiting for a failure, these devices are not accessible from the front end (customer). The folks back at the PSE labs, will still be able to interact with these devices and invoke it for you incase of a failure or a proactive failure or for any reasons the automatic invoke fails.

If a Permanent Spare fails to invoke, a Dynamic Hot Spare is invoked, if a Dynamic Hot Spare fails to invoke, the customer data stays unprotected.

Dynamic Hot Spare is supported with RAID-1, RAID-10, RAID-XP, RAID-5 and various configurations within each Raid type. Dynamic hot sparing does not work with RAID-6 devices.

As far as I know for the V-Max systems, Dynamic hot sparing is not supported.

Some important benefits of Dynamic Hot Sparing

Dynamic Hot Sparing kicks in when Permanent Sparing fails to invoke

Provides additional protection against data loss

No BIN file change is performed with Dynamic Hot Sparing

As a requirement to all the new systems that are configured now, sparing is required. Hope this provides a vision into configuring your next EMC Symmetrix on the floor.

Virtualization

Categories

Disclaimer

The opinions expressed here are my - StorageNerve opinions. This blog and the content published here is not read or approved in advance by my employer (Accenture) or clients and does not reflect their views and opinions.