Zimbra Collaboration Suite Appliance died on me

hello folks,

I'm having a serious issue here.

We run the appliance version of Zimbra 6.07. Everything was running fine until I noticed today that some mails were bounced. I tried to open the GUI management console and got nothing. It waited forever for the services status to display but nothing happened. So finally, I decided to reboot the appliance.

After the reboot, I suddenly was welcomed with 'localhost.localdomain'. This is NOT good. After a bit of trouble shooting I noticed that the localconfig.xml could not be found according to the logs. The file was there, but it was empty???? I copied a backup copy into place but I'm still stuck here. Nothing is working.

I tried the troubleshooting tips from alot of posts around here (DNS, MX, hosts file, resolv.conf) All naming related things seem to be ok, zmsshkeygen pukes me an error message which I think could be related.

java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl .java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSoc ketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.j ava:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.j ava:366)
at java.net.Socket.connect(Socket.java:525)
at java.net.Socket.connect(Socket.java:475)
at java.net.Socket.<init>(Socket.java:372)
at java.net.Socket.<init>(Socket.java:215)
at com.mysql.jdbc.StandardSocketFactory.connect(Stand ardSocketFactory.java:256)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:271)
at com.mysql.jdbc.Connection.createNewIO(Connection.j ava:2921)
at com.mysql.jdbc.Connection.<init>(Connection.java:1 555)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonReg isteringDriver.java:285)
at java.sql.DriverManager.getConnection(DriverManager .java:582)
at java.sql.DriverManager.getConnection(DriverManager .java:154)
at org.apache.commons.dbcp.DriverManagerConnectionFac tory.createConnection(DriverManagerConnectionFacto ry.java:68)
at com.zimbra.cs.db.DbPool$ZimbraConnectionFactory.cr eateConnection(DbPool.java:239)
at org.apache.commons.dbcp.PoolableConnectionFactory. makeObject(PoolableConnectionFactory.java:294)
at org.apache.commons.pool.impl.GenericObjectPool.bor rowObject(GenericObjectPool.java:1148)
at org.apache.commons.dbcp.PoolingDataSource.getConne ction(PoolingDataSource.java:96)
at com.zimbra.cs.db.DbPool.getConnection(DbPool.java: 309)
at com.zimbra.cs.mailbox.Mailbox$MailboxChange.getCon nection(Mailbox.java:310)
at com.zimbra.cs.mailbox.Mailbox.getOperationConnecti on(Mailbox.java:1130)
at com.zimbra.cs.db.DbMailItem.saveMetadata(DbMailIte m.java:670)
at com.zimbra.cs.mailbox.MailItem.saveMetadata(MailIt em.java:2856)
at com.zimbra.cs.mailbox.MailItem.saveMetadata(MailIt em.java:2849)
at com.zimbra.cs.mailbox.Folder.checkpointRECENT(Fold er.java:565)
at com.zimbra.cs.mailbox.Mailbox.recordImapSession(Ma ilbox.java:2436)
at com.zimbra.cs.imap.ImapFolder.snapshotRECENT(ImapF older.java:154)
at com.zimbra.cs.imap.ImapFolder.unregister(ImapFolde r.java:143)
at com.zimbra.cs.imap.ImapHandler.unsetSelectedFolder (ImapHandler.java:825)
at com.zimbra.cs.imap.TcpImapHandler.dropConnection(T cpImapHandler.java:155)
at com.zimbra.cs.imap.ImapHandler.dropConnection(Imap Handler.java:3719)
at com.zimbra.cs.tcpserver.ProtocolHandler.hardShutdo wn(ProtocolHandler.java:246)
at com.zimbra.cs.tcpserver.ProtocolHandler.gracefulSh utdown(ProtocolHandler.java:237)
at com.zimbra.cs.tcpserver.TcpServer.shutdownActiveHa ndlers(TcpServer.java:116)
at com.zimbra.cs.tcpserver.TcpServer.shutdown(TcpServ er.java:147)
at com.zimbra.cs.tcpserver.TcpServer.shutdown(TcpServ er.java:131)
at com.zimbra.cs.imap.ImapServer.shutdownImapServers( ImapServer.java:111)
at com.zimbra.cs.util.Zimbra.shutdown(Zimbra.java:299 )
at com.zimbra.soap.SoapServlet.destroy(SoapServlet.ja va:138)
at org.mortbay.jetty.servlet.ServletHolder.destroyIns tance(ServletHolder.java:318)
at org.mortbay.jetty.servlet.ServletHolder.doStop(Ser vletHolder.java:289)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.servlet.ServletHandler.doStop(Se rvletHandler.java:174)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.jetty.servlet.SessionHandler.doStop(Se ssionHandler.java:125)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.jetty.handler.ContextHandler.doStop(Co ntextHandler.java:591)
at org.mortbay.jetty.webapp.WebAppContext.doStop(WebA ppContext.java:505)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerCollection.doStop (HandlerCollection.java:169)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerCollection.doStop (HandlerCollection.java:169)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.jetty.handler.DebugHandler.doStop(Debu gHandler.java:135)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.handler.HandlerWrapper.doStop(Ha ndlerWrapper.java:142)
at org.mortbay.jetty.Server.doStop(Server.java:283)
at org.mortbay.component.AbstractLifeCycle.stop(Abstr actLifeCycle.java:76)
at org.mortbay.jetty.Server$ShutdownHookThread.run(Se rver.java:561)

small addition: when I try to start the server daemons, the following output is displayed:
Host zimbra.infra.local
Unable to determine enabled services from ldap.
Unable to determine enabled services. Cache is out of date or doesn't exist
[fail]

small addition: when I try to start the server daemons, the following output is displayed:
Host zimbra.infra.local
Unable to determine enabled services from ldap.
Unable to determine enabled services. Cache is out of date or doesn't exist
[fail]

That error is usually caused by a DNS error or an incorrect /etc/hosts file, go to the Split DNS article and in the 'Verify...' section you'll see details on how to check the DNS records and the format of the hosts file.

It would be odd as Zimbra has worked fine for us for more than 3 months. As far as I can tell, all is well in the named department. The server isn't published with a public name to the internet but behind a firewall and a spam gateway.

How about the output of the dig commands just for completeness? You should also only have one DNS server in your resolve.conf unless both of your DNS server have identical records for the Zimbra server.

so, basically, what you're saying is that Zimbra needs to resolve all hosted domains via (local) dns to it's own (internal) IP. Odd then that the appliance version does not have a Bind instance installed.

I need to move to a different location now, I will be back in approx 2 hours. I will then install Bind on the local machine and configure the hosted domains. Do I need to put an MX record in for the local domain "infra.local" as well?

so, basically, what you're saying is that Zimbra needs to resolve all hosted domains via (local) dns to it's own (internal) IP.

You only need the DNS A & MX records for the primary domain on the Zimbra server.

Originally Posted by alex vmguru.nl

Odd then that the appliance version does not have a Bind instance installed.

That's for flexibility, you don't specifically need the DNS server on the appliance or on the 'normal' Zimbra server. If you already have a DNS server in your LAN then feel free to use that, the Split DNS article is just one example of how to set-up that facility.

Originally Posted by alex vmguru.nl

I need to move to a different location now, I will be back in approx 2 hours. I will then install Bind on the local machine and configure the hosted domains. Do I need to put an MX record in for the local domain "infra.local" as well?

If that's the primary domain then, yes you'll need the records for that (as I mentioned above).