If I set the Sharepoint repository connector to use the authority 'None (Global Authority)', then allow_token_document will contain SIDs that are not prefixed with any authority name, for example:

S-1-5-21-1234567890-1234567890-1234567890-1234

It is therefore not possible to get any search results, because the authority service tokens will not match the stored tokens (because they are prefixed with authority names).

If I set the Sharepoint repository connector to use one of the AD authorities 'InternalAD', then allow_token_document will contain SIDs that are prefixed with 'InternalAD', for example:

TOKEN:InternalAD:S-1-5-21-1234567890-1234567890-1234567890-1234

However, the prefix is always 'InternalAD', even if the user/group actually belongs to the external.com domain. Therefore it is not possible for users in the external.com domain to get any search results, because the authority service tokens will not match the stored tokens.

In essence, there seems to be a mismatch between the tokens that the authority service outputs, and those that repository connectors output.

Perhaps one solution would be to use the authority 'None (Global Authority)', and modify the authority service to take an extra query parameter that prevents it from prefixing SIDs with the authority name.

Activity

If you are using SharePoint, the active directory domains must be related in some way. SharePoint, as far as I know, does not support a model where two unrelated domain controllers authorize documents. Can you provide details about how the domains are related?

The obvious way to look at the problem is extending the Active Directory Authority to be able to deal with related domains, but until we understand the use case, that's going to be hard.

Karl Wright
added a comment - 11/Apr/12 11:29 If you are using SharePoint, the active directory domains must be related in some way. SharePoint, as far as I know, does not support a model where two unrelated domain controllers authorize documents. Can you provide details about how the domains are related?
The obvious way to look at the problem is extending the Active Directory Authority to be able to deal with related domains, but until we understand the use case, that's going to be hard.

I assure you it works fine in SharePoint 2007, and has been doing so for years now

internal.com is a forest root for 3 regional domains, e.g. ap.internal.com, while external.com is it's own forest. Each of the internal regional domains is configured to trust external.com. That is really as far as they are 'related'.

Colin Anderson
added a comment - 11/Apr/12 11:48 I assure you it works fine in SharePoint 2007, and has been doing so for years now
internal.com is a forest root for 3 regional domains, e.g. ap.internal.com , while external.com is it's own forest. Each of the internal regional domains is configured to trust external.com. That is really as far as they are 'related'.
external.com
internal.com
ap.internal.com > trusts external.com
eu.internal.com > trusts external.com
am.internal.com > trusts external.com

So, how is SharePoint configured to authenticate users who log into it? What domain controller does it use for authentication (i.e. what domain is the SharePoint server(s) joined to)?

I am not an active directory expert, but I've never found any indication that you can join a Windows machine to more than one domain. I presume, therefore, that the joined domain controller delegates authentication to the external.com domain controller. If that is that case, everything might already work for you if you simply specify the internal.com DC in your active directory authority connection.

Karl Wright
added a comment - 11/Apr/12 12:11 So, how is SharePoint configured to authenticate users who log into it? What domain controller does it use for authentication (i.e. what domain is the SharePoint server(s) joined to)?
I am not an active directory expert, but I've never found any indication that you can join a Windows machine to more than one domain. I presume, therefore, that the joined domain controller delegates authentication to the external.com domain controller. If that is that case, everything might already work for you if you simply specify the internal.com DC in your active directory authority connection.

Colin Anderson
added a comment - 11/Apr/12 12:47 The SharePoint servers are joined to the internal.com domain.
When logging in, users authenticate with their usernames in the form 'DOMAIN\USERNAME', which I guess tells SharePoint which domain to use for authentication.
I already specify the internal.com DC in the 'InternalAD' authority connector, but as above, this doesn't work. The authority service returns something like this:
AUTHORIZED:ExternalAD
TOKEN:ExternalAD:S-1-5-21-1234569890-1234569890-1234569890-1234
USERNOTFOUND:InternalAD
TOKEN:InternalAD:DEAD_AUTHORITY
while the document tokens are still all prefixed by 'InternalAD'

The prefixes are not the issue. In ManifoldCF, there should be one authority that is capable of dealing with all the interacting domains, just like SharePoint understands all the domains by talking with just one domain controller. An authority's job is to look up the SIDs for a given username, and since your SharePoint instance can do this by talking with only one DC, ManifoldCF's active directory authority connector ought to be able to do the same thing.

The issue, then, is that the active directory authority connector doesn't know how to deal with external, trusted domains. The authority must locate the DC(s) for these domains within LDAP (I believe) and route requests to the appropriate DC. Alternatively, if automatic discovery of the external DC is too hard, we can modify the AD authority connector to simply allow the user to describe multiple domain controllers.

Karl Wright
added a comment - 11/Apr/12 13:05 - edited Hi Colin,
The prefixes are not the issue. In ManifoldCF, there should be one authority that is capable of dealing with all the interacting domains, just like SharePoint understands all the domains by talking with just one domain controller. An authority's job is to look up the SIDs for a given username, and since your SharePoint instance can do this by talking with only one DC, ManifoldCF's active directory authority connector ought to be able to do the same thing.
The issue, then, is that the active directory authority connector doesn't know how to deal with external, trusted domains. The authority must locate the DC(s) for these domains within LDAP (I believe) and route requests to the appropriate DC. Alternatively, if automatic discovery of the external DC is too hard, we can modify the AD authority connector to simply allow the user to describe multiple domain controllers.
Does this sound reasonable to you?

Karl Wright
added a comment - 11/Apr/12 13:29 Oh, and one other thing: the ManifoldCF authorities ONLY support incoming user names of the form: name@domain. They do not support domain\name. The AD authority connector also has this restriction.

I'm not sure that SharePoint is just talking to one DC Actually, I'm fairly sure it talks to both the internal.com and external.com DCs. Whether the internal.com domain 'tells' it to talk to extranet.com, I don't know.

But regardless it still seems like a good idea to be able to specify multiple domains within a single AD authority connector. We should be able to set the different parameters for each DC (username, password, auth type, AD attribute).

Colin Anderson
added a comment - 11/Apr/12 13:36 - edited Hi Karl,
I'm not sure that SharePoint is just talking to one DC Actually, I'm fairly sure it talks to both the internal.com and external.com DCs. Whether the internal.com domain 'tells' it to talk to extranet.com , I don't know.
But regardless it still seems like a good idea to be able to specify multiple domains within a single AD authority connector. We should be able to set the different parameters for each DC (username, password, auth type, AD attribute).
About the format of usernames, I have no issue with that

If SharePoint talks to multiple DC's, it is nevertheless mediated by the main DC, otherwise it would have no idea what it should trust. Otherwise, if you logged in as "colin@baddomain.com" and set up your own baddomain.com DC, it would authenticate you even though it shouldn't.

But that's neither here nor there. I agree that just supporting multiple DC's in the AD authority is probably the way to go. The only other way would be to locate trusted domains in the main domain's LDAP; I've asked a colleague whether that info exists and is likely to be useful or not. Stay tuned.

Karl Wright
added a comment - 11/Apr/12 13:47 Hi Colin,
If SharePoint talks to multiple DC's, it is nevertheless mediated by the main DC, otherwise it would have no idea what it should trust. Otherwise, if you logged in as "colin@baddomain.com" and set up your own baddomain.com DC, it would authenticate you even though it shouldn't.
But that's neither here nor there. I agree that just supporting multiple DC's in the AD authority is probably the way to go. The only other way would be to locate trusted domains in the main domain's LDAP; I've asked a colleague whether that info exists and is likely to be useful or not. Stay tuned.

Even if it could automatically discover trusted domains, I think it best to have the option to manually specify the other domains, so we can exclude some if desired, and so we can supply username/password etc for those domains we do want.

Colin Anderson
added a comment - 11/Apr/12 14:20 Hi Karl,
Even if it could automatically discover trusted domains, I think it best to have the option to manually specify the other domains, so we can exclude some if desired, and so we can supply username/password etc for those domains we do want.
I will stay tuned

According to my colleague, it does not appear that we can get much help out of LDAP. So I think the right approach is as you describe - entering multiple domains one at a time, with complete information for each. For backwards compatibility, and to handle the case where there is no domain suffix supplied for a user, we'll keep the concept of a primary or default domain.

Karl Wright
added a comment - 11/Apr/12 15:37 According to my colleague, it does not appear that we can get much help out of LDAP. So I think the right approach is as you describe - entering multiple domains one at a time, with complete information for each. For backwards compatibility, and to handle the case where there is no domain suffix supplied for a user, we'll keep the concept of a primary or default domain.

The doc is not done yet, but the way it's supposed to work is that you create a sequence of rules. Each rule has a "suffix"; if that matches the end of the domain attached to the username (everything case insensitive), then the corresponding domain controller will be the one that is used to resolve that user's SIDs.

Karl Wright
added a comment - 13/Apr/12 10:14 The branch https://svn.apache.org/repos/asf/incubator/lcf/branches/CONNECTORS-460 contains a revised active directory authority connector (plus one other fix that's needed to make it work). Colin, if you can check out and build this branch, I'd love to hear if it works for you.
The doc is not done yet, but the way it's supposed to work is that you create a sequence of rules. Each rule has a "suffix"; if that matches the end of the domain attached to the username (everything case insensitive), then the corresponding domain controller will be the one that is used to resolve that user's SIDs.
Please let me know what you find.

(1) When you set up a connection with the old Active Directory connector using the same (identical) parameters, do you get a successful connection?
(2) Please look carefully at the connection on the view page. Did all your settings seem to get saved correctly? (other than the passwords, which you can't see obviously).
(3) Do you see any exceptions in manifoldcf.log? They may be helpful in figuring out what is going wrong.

Meanwhile I'll eyeball the code and see if I can find something obvious...

Karl Wright
added a comment - 13/Apr/12 13:39 No idea what is wrong offhand. But we can debug.
Some questions:
(1) When you set up a connection with the old Active Directory connector using the same (identical) parameters, do you get a successful connection?
(2) Please look carefully at the connection on the view page. Did all your settings seem to get saved correctly? (other than the passwords, which you can't see obviously).
(3) Do you see any exceptions in manifoldcf.log? They may be helpful in figuring out what is going wrong.
Meanwhile I'll eyeball the code and see if I can find something obvious...

I hope this works for you now; if not, I'm going to be unavailable until Sunday afternoon, at which point I can look at this again (or, hopefully, just update the documentation and commit it to trunk!)

Karl Wright
added a comment - 13/Apr/12 13:47 I hope this works for you now; if not, I'm going to be unavailable until Sunday afternoon, at which point I can look at this again (or, hopefully, just update the documentation and commit it to trunk!)

Colin Anderson
added a comment - 13/Apr/12 15:10 Hi Karl,
I can create the authority with multiple domains now, so that side seems OK.
When crawling, I get allow_token_document values all prefixed with the name of new, single authority.
But the ManifoldCF authority service doesn't work - if I call:
http://localhost:8345/mcf-authority-service/UserACLs?username=123456@ap.enterdir.com
I get:
UNREACHABLEAUTHORITY:Active+Directory
TOKEN:AD:DEAD_AUTHORITY
And in the log I see:
WARN 2012-04-13 15:06:07,253 (Auth check thread 0) - Authority connection error: null
java.lang.NullPointerException
at org.apache.manifoldcf.authorities.authorities.activedirectory.ActiveDirectoryAuthority$AuthorizationResponseDescription.getCriticalSectionName(ActiveDirectoryAuthority.java:1024)
at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCreateSection(CacheManager.java:343)
at org.apache.manifoldcf.authorities.authorities.activedirectory.ActiveDirectoryAuthority.getAuthorizationResponse(ActiveDirectoryAuthority.java:260)
at org.apache.manifoldcf.authorities.system.AuthCheckThread.run(AuthCheckThread.java:92)
WARN 2012-04-13 15:06:07,253 (13242994@qtp-32105264-0) - Authority 'Active Directory' is unreachable for user '123456@ap.enterdir.com'
I get the same if I try with a user in the external.com domain.

Karl Wright
added a comment - 13/Apr/12 16:06 service doesn't handle multi-domain environments
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
Looks simple to fix next time I have internet service.
Karl
Sent from my Windows Phone
From: Colin Anderson (Commented) (JIRA)
Sent: 4/13/2012 10:13 AM
To: connectors-dev@incubator.apache.org
Subject: [jira] [Commented] ( CONNECTORS-460 ) ManifoldCF authority
service doesn't handle multi-domain environments
[ https://issues.apache.org/jira/browse/CONNECTORS-460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253396#comment-13253396
]
Colin Anderson commented on CONNECTORS-460 :
-------------------------------------------
Hi Karl,
I can create the authority with multiple domains now, so that side seems OK.
When crawling, I get allow_token_document values all prefixed with
the name of new, single authority.
But the ManifoldCF authority service doesn't work - if I call:
http://localhost:8345/mcf-authority-service/UserACLs?username=123456@ap.enterdir.com
I get:
UNREACHABLEAUTHORITY:Active+Directory
TOKEN:AD:DEAD_AUTHORITY
And in the log I see:
WARN 2012-04-13 15:06:07,253 (Auth check thread 0) - Authority
connection error: null
java.lang.NullPointerException
at org.apache.manifoldcf.authorities.authorities.activedirectory.ActiveDirectoryAuthority$AuthorizationResponseDescription.getCriticalSectionName(ActiveDirectoryAuthority.java:1024)
at org.apache.manifoldcf.core.cachemanager.CacheManager.enterCreateSection(CacheManager.java:343)
at org.apache.manifoldcf.authorities.authorities.activedirectory.ActiveDirectoryAuthority.getAuthorizationResponse(ActiveDirectoryAuthority.java:260)
at org.apache.manifoldcf.authorities.system.AuthCheckThread.run(AuthCheckThread.java:92)
WARN 2012-04-13 15:06:07,253 (13242994@qtp-32105264-0) - Authority
'Active Directory' is unreachable for user '123456@ap.enterdir.com'
I get the same if I try with a user in the external.com domain.
–
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA
administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Karl Wright
added a comment - 14/Apr/12 00:23 I fixed the problem with the cache key computation; it was a simple typo. If you synch up and try again, I feel pretty good about it working completely this time.

Colin Anderson
added a comment - 16/Apr/12 10:13 Hi Karl,
It works!
I also had to remove the '@' from the start of the domain suffix (so it is ap.internal.com rather than @ap.internal.com )
Many thanks for your swift work on this
I expect this functionality will be useful to other large organisation too, so it's good to have it in ManifoldCF