Details

Description

We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

Activity

Currently, UGI has getShortUserName which truncates the user name at the first '@' or '/'. I propose we replace that with a getLocalName that applies the configured mapping to create the local user name.

The administrator creates a file (user.mapping) with one rule per a line, the rules are attempted in the order listed in the file, and only the first rules that applies is used. '*' is a wildcard that matches 0 or more characters other than '/' and '@'. The value that matched the nth '*' is available to the rules as \n.

The translation fails with an exception if the resulting name contains either '/' or '@'.

The default rules would be:

*/*@* -> \1
*@* -> \1

which just keeps the prefix of each principal.

There will be a command line tool that you can invoke to translate a list of long names into their local equivalents.

Owen O'Malley
added a comment - 30/Jan/10 17:11 - edited Currently, UGI has getShortUserName which truncates the user name at the first '@' or '/'. I propose we replace that with a getLocalName that applies the configured mapping to create the local user name.
The administrator creates a file (user.mapping) with one rule per a line, the rules are attempted in the order listed in the file, and only the first rules that applies is used. '*' is a wildcard that matches 0 or more characters other than '/' and '@'. The value that matched the nth '*' is available to the rules as \n.
The translation fails with an exception if the resulting name contains either '/' or '@'.
The default rules would be:
*/*@* -> \1
*@* -> \1
which just keeps the prefix of each principal.
There will be a command line tool that you can invoke to translate a list of long names into their local equivalents.

Owen O'Malley
added a comment - 30/Jan/10 23:43 The annoying thing is that Kerberos has a function (and configuration) that does it, but doesn't export the API or CLI to do it. sigh
If someone has a cross-platform solution short of reimplmenting it ourselves, I'd love to hear it.

Ok, after some investigation I wasn't happy.
1. The Java Kerberos library doesn't export their auth_to_local rule translation.
2. The Java Kerberos library has bugs (ie. simplifications) that mean they skip over the auth_to_local rules in their parsing of the Kerberos config file.

So here is some code where you can cut and paste the rules from your krb5.conf's auth_to_local rules into core-site.xml. The downside is that the best documentation for those rules are in an_to_ln.c. sigh

So the default rule is just "DEFAULT" which takes all principals in your default domain to their first component. "omalley@APACHE.ORG" and "omalley/admin@APACHE.ORG" to "omalley", if your default domain is APACHE.ORG.

The translations rules have 3 sections:
<base><filter><substitution>

The base consists of a number that represents the number of components in the principal name excluding the realm and the pattern for building the name from the sections of the principal name. The base uses $0 to mean the realm, $1 to mean the first component and $2 to mean the second component.

The filter is a regex in parens that must the generated string for the rule to apply.

"(.*%admin)" will take any string that ends in "%admin"
"(.*@ACME.COM)" will take any string that ends in "@ACME.COM"

Finally, the substitution is a sed rule to translate a regex into a fixed string.

"s/@ACME\.COM//" removes the first instance of "@ACME.COM".
"s/@[A-Z]*\.COM//" removes the first instance of "@" followed by a name followed by ".COM".
"s/X/Y/g" replaces all of the "X" in the name with "Y"

So, if your default realm was APACHE.ORG, but you also wanted to take all principals from ACME.COM that had a single component "joe@ACME.COM", you'd do:

RULE:[1:$1@$0](.@ACME.ORG)s/@.//
DEFAULT

To also translate the names with a second component, you'd make the rules:

Owen O'Malley
added a comment - 07/Mar/10 19:59 Ok, after some investigation I wasn't happy.
1. The Java Kerberos library doesn't export their auth_to_local rule translation.
2. The Java Kerberos library has bugs (ie. simplifications) that mean they skip over the auth_to_local rules in their parsing of the Kerberos config file.
So here is some code where you can cut and paste the rules from your krb5.conf's auth_to_local rules into core-site.xml. The downside is that the best documentation for those rules are in an_to_ln.c. sigh
So the default rule is just "DEFAULT" which takes all principals in your default domain to their first component. "omalley@APACHE.ORG" and "omalley/admin@APACHE.ORG" to "omalley", if your default domain is APACHE.ORG.
The translations rules have 3 sections:
<base><filter><substitution>
The base consists of a number that represents the number of components in the principal name excluding the realm and the pattern for building the name from the sections of the principal name. The base uses $0 to mean the realm, $1 to mean the first component and $2 to mean the second component.
[1:$1@$0] translates "omalley@APACHE.ORG" to "omalley@APACHE.ORG"
[2:$1] translates "omalley/admin@APACHE.ORG" to "omalley"
[2:$1%$2] translates "omalley/admin@APACHE.ORG" to "omalley%admin"
The filter is a regex in parens that must the generated string for the rule to apply.
"(.*%admin)" will take any string that ends in "%admin"
"(.*@ACME.COM)" will take any string that ends in "@ACME.COM"
Finally, the substitution is a sed rule to translate a regex into a fixed string.
"s/@ACME\.COM//" removes the first instance of "@ACME.COM".
"s/@ [A-Z] *\.COM//" removes the first instance of "@" followed by a name followed by ".COM".
"s/X/Y/g" replaces all of the "X" in the name with "Y"
So, if your default realm was APACHE.ORG, but you also wanted to take all principals from ACME.COM that had a single component "joe@ACME.COM", you'd do:
RULE: [1:$1@$0] (. @ACME.ORG)s/@. //
DEFAULT
To also translate the names with a second component, you'd make the rules:
RULE: [1:$1@$0] (. @ACME.ORG)s/@. //
RULE: [2:$1@$0] (. @ACME.ORG)s/@. //
DEFAULT
If you want to treat all principals from APACHE.ORG with /admin as "admin", your rules would look like:
RULE [2:$1%$2@$0] (. %admin@APACHE.ORG)s/. /admin/
DEFAULT

Several tests and code at other places too had to be changed because createRemoteUser and similar APIs have been changed to throw IOException. This patch would be simiplified if those APIs throw a RuntimeException instead of IOException. Do we really need to throw IOException?

Jitendra Nath Pandey
added a comment - 17/Mar/10 02:11 Patch for hadoop 20 uploaded.
Several tests and code at other places too had to be changed because createRemoteUser and similar APIs have been changed to throw IOException. This patch would be simiplified if those APIs throw a RuntimeException instead of IOException. Do we really need to throw IOException?

Latest patch introduces src/test/krb5.conf which is needed by a couple of tests only. The use of this configuration file for some tests is enabled by the property java.security.krb5.conf. Kerberos has a bug in the implementation of the logic around this property (see http://bugs.sun.com/view_bug.do?bug_id=6857795)

This badly affects any tests running from under ant environment (i.e. Herriot tests (HADOOP-6332)) and on another hand isn't sufficient for Eclipse environment.

Konstantin Boudnik
added a comment - 27/Apr/10 01:46 Latest patch introduces src/test/krb5.conf which is needed by a couple of tests only. The use of this configuration file for some tests is enabled by the property java.security.krb5.conf. Kerberos has a bug in the implementation of the logic around this property (see http://bugs.sun.com/view_bug.do?bug_id=6857795 )
This badly affects any tests running from under ant environment (i.e. Herriot tests ( HADOOP-6332 )) and on another hand isn't sufficient for Eclipse environment.

Another issue is that the setting affects all tests. This is especially bad for tests which are running on an actual cluster but from the source workspace i.e. Herriot tests. This settings forces default realm to be set to APACHE.ORG which is non-sensical in environments with different realm names.

A better way is to set this property directly in the functional tests requiring this config file. Other tests shouldn't be affected.

This is dirty hack to workaround the problem, although we shouldn't be modifying the whole build just because of a couple of tests requiring a custom config file.

Konstantin Boudnik
added a comment - 27/Apr/10 01:49 Another issue is that the setting affects all tests. This is especially bad for tests which are running on an actual cluster but from the source workspace i.e. Herriot tests. This settings forces default realm to be set to APACHE.ORG which is non-sensical in environments with different realm names.
A better way is to set this property directly in the functional tests requiring this config file. Other tests shouldn't be affected.
This is dirty hack to workaround the problem, although we shouldn't be modifying the whole build just because of a couple of tests requiring a custom config file.

fixed release warning for krb5.conf
javadoc and javac warnings are for "warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release[exec][javac] import sun.security.krb5.Config; "

Hadoop QA
added a comment - 05/Jun/10 02:07 -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12446384/HADOOP-6526-1.patch
against trunk revision 951624.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 11 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/565/console
This message is automatically generated.

both javadoc and javac warnings are related to this:
Constructing Javadoc information...[exec][javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:29: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release[exec][javadoc] import sun.security.krb5.Config;[exec][javadoc] ^[exec][javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:30: warning: sun.security.krb5.KrbException is Sun proprietary API and may be removed in a future release[exec][javadoc] import sun.security.krb5.KrbException;[exec][javadoc] ^[exec][javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:77: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release[exec][javadoc] private static Config kerbConf;