Moving this to a kernel bug for now. cifs-utils doesn't do much beyond send the password down to the kernel. That's where the real work happens.
sec=ntlm is just a workaround. It's absolutely not secure enough for general usage these days since the hashes are crackable in seconds with rainbow tables. The real issue is why mounting samba with sec=ntlmssp is failing for you.
It seems to work fine for me, but my client and server are x86_64 I'll note that you're testing this on ppc64. That makes me suspect that there's an endianness bug somewhere.
Does it work if you attempt to mount this server using sec=ntlmssp from an x86_64 client? How about if you do the reverse and mount an x86_64 server from a ppc64 client?

------- Comment From sanpatr1@in.ibm.com 2013-06-07 11:51 EDT-------
(In reply to comment #9)
> Moving this to a kernel bug for now. cifs-utils doesn't do much beyond send
> the password down to the kernel. That's where the real work happens.
>
> sec=ntlm is just a workaround. It's absolutely not secure enough for general
> usage these days since the hashes are crackable in seconds with rainbow
> tables. The real issue is why mounting samba with sec=ntlmssp is failing for
> you.
>
> It seems to work fine for me, but my client and server are x86_64 I'll note
> that you're testing this on ppc64. That makes me suspect that there's an
> endianness bug somewhere.
>
> Does it work if you attempt to mount this server using sec=ntlmssp from an
> x86_64 client? How about if you do the reverse and mount an x86_64 server
> from a ppc64 client?
As request I tried similar scenario and here are the results
I have one pcc64 and one x86_64 Fedora19 systems. I configured samba server in both the system. Try to mount within and across the systems. Here are the results
Samba Server mount client default sec=ntlm
pcc64 ppc64 failure success
x86_64 x86_64 success not required
ppc64 x86_64 success not required
x86_64 ppc64 failure success

Many thanks. Based on that, it does seem likely that there's an endianness problem in the cifs.ko NTLMSSP auth code. The trick now is to track down where it is.
What might be good is to turn up samba debugging code, and see whether it gives us any hint as to which field isn't getting proper endianness conversion.

------- Comment From sanpatr1@in.ibm.com 2013-06-07 13:02 EDT-------
(In reply to comment #11)
> Many thanks. Based on that, it does seem likely that there's an endianness
> problem in the cifs.ko NTLMSSP auth code. The trick now is to track down
> where it is.
>
> What might be good is to turn up samba debugging code, and see whether it
> gives us any hint as to which field isn't getting proper endianness
> conversion.
>
> One other question...
>
> Do any of the failing configurations work if you use sec=ntlmv2?
No it is also not working. Only ntlm is working in case failing case
>That sec= option and sec=ntlmssp share some code, so that may help us pinpoint where
> to look.

I have a setup where I can recreate this bug.
It does not have to be a Samba server running on x86 box and
a cifs client running on a ppc64 box, a Windows server on x86 and
cifs client on ppc64 box is a good enough setup, which is what I have.

Yeah, I imagine that'll reproduce it. It might be easier to get info out of a samba server though if you (for instance) run it with -d10. With that, it may tell you something about why it's rejecting the auth request...

Focussing on nlmv2 code since it is failing and that is what
ntlmssp uses as well.
Comparing the wireshark traces of session setup from a Power box (be)
and an x86 (le) box, I see that Unicode password lengths are different
even though cifs client from both of the boxes is attempting to
authenticate with the same server as the same user with the same password.
Looking into it.

Unicode password lengths were different because I had older code
in function build_avpair_blob() on my test machine running
2.6.32-279.el6.e1000.ppc64.
Even after changing function build_avpair_blob() to make it same as mainline
code, ntlmv2 authentication on this Power Big Endian machine still fails.
Wondering whether hmac-md5 crypto code has any big endian bugs, it is possible but not sure, so will spend some time on that.

Nice! Thanks again for chasing this down. I'll leave it to you to post the patch since you did most of the legwork.
Also, while you're in there...
The code that handles the domainName or serverName indicates that it should be converting those strings to uppercase, but it doesn't look like it does that. Do we need to fix those as well?

*********** MASS BUG UPDATE **************
We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 19 kernel bugs.
Fedora 19 has now been rebased to 3.11.1-200.fc19. Please test this kernel update and let us know if you issue has been resolved or if it is still present with the newer kernel.
If you experience different issues, please open a new bug report for those.