I followed Rob's advice and enabled the http debug logging. I'm no expert here
but I didn't really see anything that indicates an error. I do see a series of
connections to the server and GSS/kerberos log messages (like "client delegates
us their credentials" and "GSS-API token of length 22 bytes will be sent
back..."). The log just stops and apache returns an authentication error.

While reading the logs, I did notice a reference to s4u2proxy. When I looked
around for that, I ran across a blog article from idra at samba.org where he
described how they modified mod_auth_kerb for FreeIPA.
On suspecting something went wrong with the installation (like some update that
replaced mod_auth_kerb or something related), I did a "yum reinstall
freeipa-server". After it completed, everything started working as it had in
the past.
So while I don't know for sure, I believe some other package that was updated
(perhaps apache) overwrote something in FreeIPA (perhaps mod_auth_kerb). In any
case, I'm whole again.
Brian
On Oct 17, 2012, at 7:49 AM, Rob Crittenden wrote:
> Brian Vetter wrote:
>> I had a happy, working 2.2 FreeIPA installation humming along last
>> week. I had to do some maintenance so I shut everything down. When I
>> brought everything up, I can no longer log into the web admin tool. I
>> get a "Kerberos ticket is no longer valid" error.
>>
>> Using the troubleshooting pages on the wiki as a guide, I can kinit
>> successfully and see the tickets using klist. I can use the ldapsearch
>> tool using GSSAPI to authenticate as well and can return results from
>> the ldap server. 'ldapsearch -Y GSSAPI -b "dc=dcc,dc=mobi" uid=admin'
>> returns a valid ldap recors for my admin user. I ran this command kinit
>> from multiple kerberos principals/users and each worked.
>>
>> I verified my config settings again with firefox and they are still set
>> correctly (auth.delegation-uris, auth.trusted-users both set to my
>> domain .dcc.mobi). The cert was accepted (no warnings about the cert not
>> being trusted because I had already set it to trusted). I turned on the
>> NSPR logging as described in the documents, and didn't see an error,
>> although I can't tell if this is correct:
>>
>> 492291904[7f4d1d31f6a0]: using REQ_DELEGATE
>> 492291904[7f4d1d31f6a0]: service = geonosis.dcc.mobi
>> 492291904[7f4d1d31f6a0]: using negotiate-gss
>> 492291904[7f4d1d31f6a0]: entering nsAuthGSSAPI::nsAuthGSSAPI()
>> 492291904[7f4d1d31f6a0]: Attempting to load gss functions
>> 492291904[7f4d1d31f6a0]: entering nsAuthGSSAPI::Init()
>> 492291904[7f4d1d31f6a0]: nsHttpNegotiateAuth::GenerateCredentials()
>> [challenge=Negotiate]
>> 492291904[7f4d1d31f6a0]: entering nsAuthGSSAPI::GetNextToken()
>> 492291904[7f4d1d31f6a0]: leaving nsAuthGSSAPI::GetNextToken [rv=0]
>> 492291904[7f4d1d31f6a0]: Sending a token of length 1394
>>
>>
>> There is nothing in /var/log/httpd/error_log. /var/log/httpd/access_log
>> does have a few entries:
>>
>> 10.1.1.10 - - [16/Oct/2012:18:05:26 -0500] "POST /ca/ocsp HTTP/1.1"
>> 200 2298 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:16.0)
>> Gecko/20100101 Firefox/16.0"
>> 10.1.1.10 - - [16/Oct/2012:18:05:26 -0500] "POST /ipa/session/json
>> HTTP/1.1" 401 -
>> 10.1.1.10 - - [16/Oct/2012:18:05:26 -0500] "GET
>> /ipa/session/login_kerberos HTTP/1.1" 401 1861
>> 10.1.1.10 - ad...@dcc.mobi
>> [16/Oct/2012:18:05:26 -0500] "GET /ipa/session/login_kerberos
>> HTTP/1.1" 200 -
>> 10.1.1.10 - - [16/Oct/2012:18:05:26 -0500] "POST /ipa/session/json
>> HTTP/1.1" 401 -
>>
>>
>> The 401's aren't surprising here since somehow, something is not
>> properly authenticating.
>>
>> I also looked in /var/log/krb5kdc.log and see the following line when
>> authenticating:
>>
>> Oct 16 18:12:17 geonosis.dcc.mobi krb5kdc[1193](info): TGS_REQ (1
>> etypes {18}) 10.1.1.10: ISSUE: authtime 1350424404, etypes {rep=18
>> tkt=18 ses=18}, ad...@dcc.mobi for
>> krbtgt/dcc.m...@dcc.mobi
>>
>>
>> I don't believe this describes an error, but I'm not an expert reading
>> that log type.
>>
>> From what I can tell, the problems seem to be between the apache server
>> and the browser. Both worked fine together last week. Is there something
>> I can turn on in Apache (perhaps in the ipa.conf or auth_kerb.conf
>> files) that can help debug this? Or better yet, anyone else seen this
>> and have an answer? Is there some key/ticket/etc associated with the
>> http server that might be "wrong" now (somehow whacked during the reboot)?
>
> If you set LogLevel debug in /etc/httpd/conf.d/nss.conf and restart the httpd
> service you'll get a lot of debugging output from ipa and mod_auth_kerb, one
> of which may provide some pointers.
>
> rob
>