Thanks for the suggestions and with these suggestions it went ahead, I am getting below errors(Underlined) at server side while accessing the secured ejb. Now user name and password are getting passed to server, however password is not matching. Same database was working for jboss6.

Thanks for the detailed output - at this point this suggests the realm / domain configuration is correctly configured to be working together, we may need to double check that the correct password is being passed to the login module but from the information I have if the correct username is arriving there I would not suspect the wrong password to be arriving there.

Within your security domain configuration you have configured the module to use both a hashAlgorithm and hashEncoding - what is the actual value stored in the password field of that row of the table?

I would suggest that as a next step to at least verify end to end of this call that you remove both the hashAlgorithm and hashEncoding values from the domain configuration and update that row of the table so that the password field contains the plain text value 'admin' - if the call can be verified to that point it will mean that attention can then be focussed on the final settings for the DatabaseLoginModule.

Looking at your stack trace the error is reported when the connection is being established for the InitialContext, have a look towards the end of the following post with examples to set the same xnio Options for the JNDI connection: -

Are either of you familiar with debugging JBoss AS source? If so the easiest thing would be to set a breakpoint and check the values being passed into the validatePassword of UsernamePasswordLoginModule and verify that they are as expected (To be getting the error we see one of them must be wrong)

If not I will see if we can use byteman to output the values for us to vefify.

One other point revealed by this thread (and a couple of others) is that it is quite difficult to get enough information from the logs to accurately see why a comparison is failing - all logging that outputs passwords has been removed so that passwords are not accidentally stored in the logs but I think there are still a couple of scenarios where there is no other option than logging the values so that they can be manually compared to make sure they are as expected.

I am going to have a look at what options we have for a development environment at least to essentially evaluate the end to end authentication process so that the point of failure can be pinpointed more quickly.