These are less of notes and more of links and what I did when I encountered this issue. Just for my future self.

At work we had a host which was giving HA errors. The message was along the lines that vCenter could not contact HA. So I tried reconfiguring it for HA (right click the host and select “Reconfigure for vSphere HA”) upon which I got a new error: Cannot install the vCenter Server agent service. Cannot upload agent.

To investigate further I tried logging on to the server. I couldn’t enable SSH and ESXi Shell from the Configuration tab – it gave me an error. So I iLO’d into the server DCUI and enabled SSH and ESXi Shell. SSH still refused to let me in, and when I’d press Alt+F1 on the console to get the login prompt it was filled with messages like these: /bin/sh cant fork. Initially I thought it might be to do with HP AMS memory leak (see this and this) but it wasn’t.

I pressed Alt+F12 to see the on-screen logs. It was filled with messages like these:

Blimey!

There was nothing more I could do here basically. Couldn’t login to the server at all, heck I couldn’t even Shutdown/ Restart it gracefully via F12 in DCUI (nothing would happen). So I cold booted it and that got it working.

It’s been about 2 hours since I did that and the server seems stable so maybe it was a one off-thing. I looked at more logs though and here’s what I found.

/var/log/syslog.log

2015-08-14T08:37:39Zsfcb-CIMXML-Processor[22385291]:TicketCache---Can't open '/var/run/sfcb/52cbb0d0-da3a-9ad5-322d-361a1caafbcc', Error: 'No space left on device'

2015-08-14T08:37:40Zsfcb-CIMXML-Processor[22385292]:TicketCache---Can't open '/var/run/sfcb/52cbb0d0-da3a-9ad5-322d-361a1caafbcc', Error: 'No space left on device'

2015-08-14T08:37:40Zsfcb-CIMXML-Processor[22385293]:TicketCache---Can't open '/var/run/sfcb/52cbb0d0-da3a-9ad5-322d-361a1caafbcc', Error: 'No space left on device'

2015-08-14T08:37:40Zsfcb-CIMXML-Processor[22385294]:TicketCache---Can't open '/var/run/sfcb/52cbb0d0-da3a-9ad5-322d-361a1caafbcc', Error: 'No space left on device'

2015-08-14T08:37:41Zsfcb-CIMXML-Processor[22385295]:TicketCache---Can't open '/var/run/sfcb/52cbb0d0-da3a-9ad5-322d-361a1caafbcc', Error: 'No space left on device'

2015-08-14T08:37:41Zsfcb-CIMXML-Processor[22385296]:TicketCache---Can't open '/var/run/sfcb/52cbb0d0-da3a-9ad5-322d-361a1caafbcc', Error: 'No space left on device'

2015-08-14T08:37:41Zsfcb-CIMXML-Processor[22385297]:TicketCache---Can't open '/var/run/sfcb/52cbb0d0-da3a-9ad5-322d-361a1caafbcc', Error: 'No space left on device'

2015-08-14T08:37:41Zsfcb-CIMXML-Processor[22385298]:TicketCache---Can't open '/var/run/sfcb/52cbb0d0-da3a-9ad5-322d-361a1caafbcc', Error: 'No space left on device'

2015-08-14T08:37:41Zsfcb-CIMXML-Processor[22385299]:TicketCache---Can't open '/var/run/sfcb/52cbb0d0-da3a-9ad5-322d-361a1caafbcc', Error: 'No space left on device'

2015-08-14T08:37:42Zsfcb-CIMXML-Processor[22352532]:TicketCache---Can't open '/var/run/sfcb/52cbb0d0-da3a-9ad5-322d-361a1caafbcc', Error: 'No space left on device'

/var/log/vmkwarning.log

(Contains: A summary of Warning and Alert log messages excerpted from the VMkernel logs)

1

2

3

4

5

6

7

2015-08-13T19:56:19.608Zcpu2:22382164)WARNING:VisorFSObj:1940:Cannot create file/var/run/sfcb/527fb83b-7c0b-4fe2-0152-d81fb0bac853 forprocess sfcb-CIMXML-Pro because the inode table of its ramdisk(root)isfull.

2015-08-13T20:00:14.737Zcpu4:34191opID=ee934b0f)WARNING:VisorFSObj:1940:Cannot create file/var/run/vmware/tickets/vmtck-52f258cf-a87b-e1 forprocess hostd-worker because the inode table of its ramdisk(root)isfull.

2015-08-13T20:04:46.110Zcpu30:34194opID=ee934b0f)WARNING:VisorFSObj:1940:Cannot create file/var/run/vmware/tickets/vmtck-52c87856-17ee-61forprocess hostd-worker because the inode table of its ramdisk(root)isfull.

2015-08-13T20:09:17.481Zcpu3:36506opID=ee934b0f)WARNING:VisorFSObj:1940:Cannot create file/var/run/vmware/tickets/vmtck-529ddabc-6196-ddforprocess hostd-worker because the inode table of its ramdisk(root)isfull.

2015-08-13T20:13:48.849Zcpu11:7960868opID=ee934b0f)WARNING:VisorFSObj:1940:Cannot create file/var/run/vmware/tickets/vmtck-5278454b-65e6-1dforprocess hostd-worker because the inode table of its ramdisk(root)isfull.

2015-08-13T20:15:53.301Zcpu6:21329945)WARNING:VisorFSObj:1940:Cannot create file/var/run/vmware/tickets/vmtck-7f09012d-0b29-44forprocess cimslp because the inode table of its ramdisk(root)isfull.

2015-08-13T20:16:48.853Zcpu12:35008opID=ee934b0f)WARNING:VisorFSObj:1940:Cannot create file/var/run/vmware/tickets/vmtck-5257e7ba-7c96-d0 forprocess hostd-worker because the inode table of its ramdisk(root)isfull.

/var/log/vob.log

(Contains: VMkernel Observation events)

1

2

3

4

5

6

7

2015-08-17T00:15:19.220Z:[VisorfsCorrelator]17133398447519us:[vob.visorfs.ramdisk.inodetable.full]Cannot create file/var/run/vmware/tickets/vmtck-52b5db61-d61e-8dforprocess hostd-worker because the inode table of its ramdisk(root)isfull.

2015-08-17T00:15:19.220Z:[VisorfsCorrelator]17133319127883us:[esx.problem.visorfs.ramdisk.inodetable.full]The filetable of the ramdisk'root'isfull.Asaresult,the file/var/run/vmware/tickets/vmtck-52b5db61-d61e-8dcould notbe created by the application'hostd-worker'.

2015-08-17T00:21:20.587Z:[VisorfsCorrelator]17133759815799us:[vob.visorfs.ramdisk.inodetable.full]Cannot create file/var/run/vmware/tickets/vmtck-52ac40ae-4240-e3 forprocess hostd-worker because the inode table of its ramdisk(root)isfull.

2015-08-17T00:21:20.587Z:[VisorfsCorrelator]17133680494786us:[esx.problem.visorfs.ramdisk.inodetable.full]The filetable of the ramdisk'root'isfull.Asaresult,the file/var/run/vmware/tickets/vmtck-52ac40ae-4240-e3 could notbe created by the application'hostd-worker'.

2015-08-17T00:25:51.966Z:[VisorfsCorrelator]17134031195582us:[vob.visorfs.ramdisk.inodetable.full]Cannot create file/var/run/vmware/tickets/vmtck-520e1b5c-35f2-21forprocess hostd-worker because the inode table of its ramdisk(root)isfull.

2015-08-17T00:25:51.966Z:[VisorfsCorrelator]17133951873623us:[esx.problem.visorfs.ramdisk.inodetable.full]The filetable of the ramdisk'root'isfull.Asaresult,the file/var/run/vmware/tickets/vmtck-520e1b5c-35f2-21could notbe created by the application'hostd-worker'.

2015-08-17T00:30:23.342Z:[VisorfsCorrelator]17134302572394us:[vob.visorfs.ramdisk.inodetable.full]Cannot create file/var/run/vmware/tickets/vmtck-52b93f74-9429-59forprocess hostd-worker because the inode table of its ramdisk(root)isfull.

/var/log/vmkernel.log

2015-08-17T01:22:09.441Zcpu30:22401956)WARNING:VisorFSObj:1940:Cannot create file/var/run/sfcb/5277bce4-a843-d718-aacc-a7bf06d5768a forprocess sfcb-CIMXML-Pro because the inode table of its ramdisk(root)isfull.

2015-08-17T01:22:09.740Zcpu27:22401957)WARNING:VisorFSObj:1940:Cannot create file/var/run/sfcb/5277bce4-a843-d718-aacc-a7bf06d5768a forprocess sfcb-CIMXML-Pro because the inode table of its ramdisk(root)isfull.

2015-08-17T01:22:10.060Zcpu13:22401959)WARNING:VisorFSObj:1940:Cannot create file/var/run/sfcb/5277bce4-a843-d718-aacc-a7bf06d5768a forprocess sfcb-CIMXML-Pro because the inode table of its ramdisk(root)isfull.

2015-08-17T01:22:10.367Zcpu9:22401960)WARNING:VisorFSObj:1940:Cannot create file/var/run/sfcb/5277bce4-a843-d718-aacc-a7bf06d5768a forprocess sfcb-CIMXML-Pro because the inode table of its ramdisk(root)isfull.

2015-08-17T01:22:10.629Zcpu26:22401962)WARNING:VisorFSObj:1940:Cannot create file/var/run/sfcb/5277bce4-a843-d718-aacc-a7bf06d5768a forprocess sfcb-CIMXML-Pro because the inode table of its ramdisk(root)isfull.

2015-08-17T01:22:10.869Zcpu22:22401963)WARNING:VisorFSObj:1940:Cannot create file/var/run/sfcb/5277bce4-a843-d718-aacc-a7bf06d5768a forprocess sfcb-CIMXML-Pro because the inode table of its ramdisk(root)isfull.

2015-08-17T01:22:11.113Zcpu25:22401966)WARNING:VisorFSObj:1940:Cannot create file/var/run/sfcb/5277bce4-a843-d718-aacc-a7bf06d5768a forprocess sfcb-CIMXML-Pro because the inode table of its ramdisk(root)isfull.

2015-08-17T01:22:11.359Zcpu17:22401968)WARNING:VisorFSObj:1940:Cannot create file/var/run/sfcb/5277bce4-a843-d718-aacc-a7bf06d5768a forprocess sfcb-CIMXML-Pro because the inode table of its ramdisk(root)isfull.

2015-08-17T01:22:11.668Zcpu12:22401970)WARNING:VisorFSObj:1940:Cannot create file/var/run/sfcb/5277bce4-a843-d718-aacc-a7bf06d5768a forprocess sfcb-CIMXML-Pro because the inode table of its ramdisk(root)isfull.

/var/log/hostd.log

(Contains: Host management service logs, including virtual machine and host Task and Events, communication with the vSphere Client and vCenter Server vpxa agent, and SDK connections.)

1

2

3

4

5

6

2015-08-15T09:11:54.083Z[2B781B70info'Vimsvc.ha-eventmgr']Event9898:The filetable of the ramdisk'root'isfull.Asaresult,the file/var/run/vmware/tickets/vmtck-52533095-569d-5ccould notbe created by the application'hostd-worker'.

From these logs one thing was clear. The ESXi RAMdisk hosting the root filesystem had run out of inodes. Possibly caused by the SFCB service. Because of this the root filesystem had run out of space and everything was failing. Great!

In Linux I am used to the df command to check filesystem usage. But in ESXi df only seems to be give info on the mounted filesystems whereas vdf gives the local filesystems (like RAMdisks and Tardisks (whatever that is)).

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

~# vdf -h

Tardisk Space Used

sb.v00148M148M

s.v00295M295M

misc_cni.v0024K21K

net_bnx2.v00304K301K

net_bnx2.v011M1M

net_cnic.v00140K137K

...

imgdb.tgz400K400K

state.tgz28K27K

-----

Ramdisk Size Used Available Use%Mounted on

root32M1M30M5%--

etc28M260K27M0%--

tmp192M532K191M0%--

hostdstats1053M8M1044M0%--

snmptraps1M0B1M0%--

Above output is after a reboot and all seems fine. To check the inode usage use the stat command.

1

2

3

4

5

6

~# stat -f /

File:"/"

ID:100000000Namelen:127Type:visorfs

Block size:4096

Blocks:Total:492406Free:331548Available:331548

Inodes:Total:524288Free:519997

Or use exscli. It gives you the free space as well as the inode count!

1

2

3

4

5

6

7

8

~# esxcli system visorfs ramdisk list

Ramdisk Name System Include inCoredumps Reserved Maximum Used Peak Used Free Reserved Free Maximum Inodes Allocated Inodes Used Inodes Mount Point

Note to self: Make a habit of using the esxcli command as that seems to be the VMware preferred way of doing things. Plus it’s one command with various namespaces you can use for networking and other info.

In my case things look to be fine now.

KB 2037798 talks about this problem. Apparently it is fixed via a patch released in 2013, and as far as I can tell we are properly patched so we shouldn’t have been hit by this issue. If it happens again though the same KB article talks about creating a separate RAMdisk for SFCB so even if it eats up all the inodes your root file system isn’t affected. This involves creating a new RAMdisk at boot time by modifying rc.local (nice!). The esxcli command can be used to create a new ramdisk and mount it at the mount point required by SFCB: