I have a wide scope of interests in IT, which includes hyper-v private cloud, remote desktop services, server clustering, PKI, network security, routing & switching, enterprise network management, MPLS VPN on enterprise network etc. Started this blog for my quick reference and to share technical knowledge with our team members.

Friday, January 4, 2013

Cisco Multicast Expansion Table (MET) is Full

Today, some users aren't able to join multicast groups to watch their favorite video. After pinpointing to the upstream Cisco 7600 router, I noticed the following error message in the log:

%MCAST-SP-4-MET_THRESHOLD_EXCEEDED: Multicast Expansion table has exceeded 98% of its capacity and is reaching its maximum%MMLS-SP-6-MET_LIMIT_EXCEEDED: Failed to allocate MET entry, exceeded system limit of (32744) entries. Number of times MET limit is exceeded in the last 1 min : 1%CONST_MFIB_LC-SP-6-MET_MCAST_ALLOC_FAILURE: Failed to allocate MET entries for IP multicast entry (S:*.*.*.*, G:*.*.*.*)
The router is configured PIM in sparse mode and an RP manually set.

Look like the Multicast Expansion Table (MET) is fully utilized. In Cisco multicast architecture, MET is populated with Output Interface Lists (OILs e.g. VLAN, routed interfaces) that requires packets to be replicated over. For example: (S,G) with three OIFs, multicast replication creates three copies of every packet received from source (S) destined to group (G). When MET is full, new users won't be able to join the igmp group, as their (S,G) and connected interface won't be added in the table.

There doesn't seem to be a way to clear MET entries except to reload the router. Cisco has a pdf guide that is useful for better understanding and troubleshooting of multicast in Catalyst 6500. The guide did mention about monitoring of MET utilization but short on the resolution.

After-note: Packet replication can be performed either egress (default) or ingress. Strangely, after changing the direction (not rebooting), the clients are able to join the IGMP group. The direction can be changed using:
(config)# mls ip multicast replication-mode ingress or the newer command (IOS 15.x)
(config)# ip multicast hardware-switching replication-mode ingressUpdate:
According to Cisco TAC, there was a known bug (CSCtg53299) whereby MET entries are not deleted properly, which in turn, led to the filling up of MET table. A solution is to upgrade the firmware to version 15.1(01)S1 and above.

3 comments:

Multicast PIM is designed to replicate the same packet stream on interfaces subscribing to the same group. You may have 1000 users on one VLAN or just 10 users on 10 VLAN interfaces. Hence, the number of users is really immaterial.