Introduction

This interceptor is used to handle referrals. According to RFC 2251/4511, a referral is returned inside a LdapResult, if the result code is set to referral, or as one or more SearchResultReference

Info on Referrals and JNDI

• RFC 2251
• RFC 3296
• JNDI Tutorial on Referral Behavoir

ApacheDS Implementation Notes

Referral handling must be accounted for in two respects. At the protocol level in the MINA LDAP protocol provider and at the JNDI level in the core ApacheDS JNDI Provider. Both must behave according to their respective specifications when dealing with refferals.

Structure

Behavior: ManageDsaIT Control & Context.REFFERAL

The ASN.1 subsystem must be updated to be able to handle the ManageDsaIT control and the server must publish that it supports this control in the RootDSE. The control as with all others should be added to the requestControl property of the LdapContext used to conduct JNDI operations within the MINA LDAP protocol provider. Meaning the protocol provider passes controls down into the JNDI provider so they can be used by the core to make critical decisions about request handing behavoir. The ManageDsaIT control does not need this however like other controls because of the use of the Context.REFFERAL property in the environment. This is discussed below.

The Context.REFERRAL property in the JNDI environment effects the way referrals are handled by the JNDI provider. According to JNDI specifications:

A JNDI application uses the Context.REFERRAL(in the API reference documentation)
("java.naming.referral") environment property to indicate to the service providers how
to handle referrals. The following table shows the values defined for this property. If this
property has not been set, then the default is to ignore referrals.

Based on the entry point, (via protocol or embedded JNDI) two mechanisms exist for controling the underlying Referral handing mechanism.  One uses the ManageDsaIT contrrol and the other uses the Context.REFERRAL property.  The presence of the ManageDsaIT control is the same as setting the environment property to ignore or not even setting the property in the environment since by absense the property is defaulted to ignore.

Referral Handling Scenarios

ou: People
ref: ldap://hostb/OU=People,DC=example,DC=com
ref: ldap://hostc/OU=People,O=MNN,C=WW
objectClass: referral
objectClass: extensibleObject

ou: Roles
ref: ldap://hostd/ou=Roles,dc=apache,dc=org
objectClass: referral
objectClass: extensibleObject

Case #2: Target is a referral

When the target is a referral, the refs are returned back to the client with a resultCode of REFFERAL. (example from RFC) If for example the client issues a modify for the target of "OU=People,O=MNN,C=WW", the server will return the following when the ManageDsaIT control is NOT present:

ModifyResponse (referral) {
    ldap://hostb/OU=People,DC=example,DC=com
    ldap://hostc/OU=People,O=MNN,C=WW
}

In this situation, without the ManageDsaIT control, the ApacheDS LDAP frontend (MINA provider) will set the value of the Context.REFFERAL property to "throw" before issuing JNDI calls to the core. The JNDI operation on the ApacheDS JNDI DirContext will throw a ReferralException which shall contain everything needed for the LDAP frontend to respond properly. This also allows, embedding applications to see the same results they would encounter from the SUN JNDI LDAP Provider operating against a remote LDAP server.

Case #3: Target's parent is a referral

According to the RFC 3296 it appears as though the remaining name past the referral is appended to the DN of the ref attributes, if the values are LDAP URLs. Also if they are LDAP URLs the scope, filter and attribute terms are removed. The result is returned back. To illustrate this let's consider the example from the RFC where an add operation is performed with the target DN of "CN=Manager,OU=Roles,O=MNN,C=WW".

The dynamics of the add operation must be considered first WRT the ApacheDS JNDI provider. This operation can proceed in two ways. First via the lookup of the parent context, "OU=Roles,O=MNN,C=WW", followed by a createSubcontext() operation on it using the RDN of the target entry. Other way to perform the add operation is by looking up an ancestor context above the parent, "O=MNN,C=WW" for example, followed by a createSubcontext() operation using a name fragment like "CN=Manager,OU=Roles". The last situation is not performed by the ApacheDS LDAP frontend but it can be performed by an embedding application against the JNDI interfaces.

When the Context.REFERRAL environment property is not set (an implicit ignore) or is explicity set to the "ignore" String, the createSubcontext() operation, regardless of what parent or ancestor it is issued upon will create the target entry under the referral parent. Remember when referrals are ignored all referrals are processed as regular entries. The dynamics get interesting when the Context.REFFERAL property is set to "throw". Incidentally we will ignore the "follow" value for the property for the time being. In the first case where the createSubcontext() operation is performed on the parent, which is the the referral entry "OU=Roles,O=MNN,C=WW", the attempt to create a non-existing child will succeed unless logic is put into place. The logic must allow the Context implementation to detect the fact that it is a referral, and that the createSubcontext() operation being performed with the Context.REFERRAL property set to "throw" must be prevented. BTW If an attempt is made to lookup the parent context with the Context.REFERRAL property set to "throw" then a ReferralException will occur. So to get the parent we would have had to look it up with referral's ignored.

In the latter case, the createSubcontext() operation is being performed upon a (non-referral) ancestor with a name fragment for the target, "CN=Manager,OU=Role". The context used to perform the operation does not care what mode the Context.REFERRAL property is set to since it is not a referral itself. It will issue the add request against the nexus with the computed target DN. The only way to prevent incorrect creation of this entry, is to check through the target's lineage for an ancestor that is a referral. If no ancestor up to the root suffix context is a referral then the operation may proceed. We cannot just check if the parent is a referral because the parent may be a regular entry hidden under another referral: meaning the target's parent may have been created while ignoring referrals. So we must exhaust the entire lineage or short the process when we find a parent or ancestor that is a referral. This must happen within the JNDI provider when the Context.REFERRAL mode is set to "ignore".

Case #4: Target's ancestor (not parent) is a referral

This is very similar to the latter half of case #3 above. When Context.REFERRAL="throw", The ApacheDS JNDI provider must test to see if any ancestors of the target entry are referrals. The biggest difference here is in the processing of ref attribute value DN fields (if they are LDAP URLs). Here the remaining name after the referral ancestor is tacked onto the DN components of the ref value. So if we were performing a createSubcontext() to add "CN=OneDown,CN=Manager,OU=Role,O=MNN,C=WW", the ancestor "OU=Role,O=MNN,C=WW" is a referral. Now the parent "CN=Manager" may or may not exist. Whether the parent exists or not we have to check for the presence of a referral ancestor before allowing the add operation to proceed. Again the best place for this is within the JNDI provider. In this example the returned AddResponse would be:

AddResponse (referral) {
        ldap://hostd/CN=OneDown,CN=Manager,ou=Roles,dc=apache,dc=org
    }

Protocol Handlers

Protocol handlers should delegate these checks for referrals to the JNDI provider. Meaning they should get whatever root context they user for a client's session and issue operations against that context using relative DN fragments (not same as single name component RDNs). They should not attempt to first lookup the parent to perform operations. This will lead to them having to replicate some of the logic in the JNDI provider for handling referrals. These handlers will have to interpolate ReferralExceptions on parent lookups to the target. Meaning if they blow chunks looking for the parent they have to massage the ref values returned by the ReferralException to represent the DN of the target on the remote server. Even with the add op you can issue a relative JNDI createSubcontext() on the rootDSE JNDI Context using the DN supplied by the request: even here there is no need to explicitly attempt to lookup the parent context.

Finding base of search operations

Here we discuss referral handling for finding the search base which is very similar to finding the targets of other operations. Unlike the other operations if we encounter a referral while finding the search base we must add a search scope specifier to the ref if it's value is an LDAP URL. Also critical extensions MUST NOT be trimmed nor modified.

Another difference to calculating ref values is in factoring in alias dereferrencing. This is where things seem to get a little tricky but not really. Whether or not the name for the discovered referral is an aliased name or the primary processing of the URL DN in ref values is the same. Only the remaining name, the remaining part of the search base DN after the referral's DN, is needed and appended to the the DN of the URL in the ref value. Either it's easier in ApacheDS because of the architecture or we're way oversimplifying this. Now let's review the cases for referral handling while finding the search base.

1. base is a normal entry (default)
2. base is a referral
3. base's parent is a referral
4. base's parent is or is not present, but an ancestor is a referral

Case #1 is the default case and will be skipped for consideration/elaboration.

Case #2: Same as for finding target entries with URL handling differences

An example is best for this case. We'll take the one in the RFC. If the client issues a subtree search in which the base object is "OU=Roles,O=MNN,C=WW", the server will return:

SearchResultDone (referral) {
        ldap://hostd/ou=Roles,dc=apache,dc=org??sub
    }

Notice the extra subtree scope parameter tacked onto the URL.

Case #3: Same as for finding target entries with URL handling differences

Again an example is best for this case. We'll take the one in the RFC. If the client issues a base scoped search in which the base object is "CN=Manaager,OU=Roles,O=MNN,C=WW", the server will return:

SearchResultDone (referral) {
        ldap://hostd/CN=Manager,ou=Roles,dc=apache,dc=org??base
    }

Notice the extra subtree scope parameter tacked onto the URL.

I won't elaborate on Case #4 since it's pretty much the same concept.

Generating search references

This is bad! Muy mal!

Yep we're not talking about a small change anymore. However referral indices above the level of partitions however does sound interesting. It gives us the ability to comply with yet another requirement made not by RFC 3296 but of JNDI. JNDI requires us to return all non-referral entries first in a search before returning referrals. Using this mechanism a search result enumeration filter can remove referral results if at all selected by the search filter. Or another approach would be to add a negated ( ! ( objectClass=referral) ) branch. However this is costly to have. It might just be better to return what is selected and use an enumeration filter to remove the rest. The indices then can be used to return all referrals after regular entries are returned.

We can use a similar indexing mechanism as is used with alias handling within the DefaultPartition implementation. We would have a oneReferral and a subReferral index. Let's describe them one by one. The oneReferral index would map the normalized DNs of regular and referral entries to a set of child referral RDNs. So the oneReferral index maps parents to child referrals. If we wanted to find all referrals immediately subordinate to a DN then it would be very easy to lookup this information very rapidly. The subReferral index is similar, except it maps the normalized DN of ancestors to their ancestor relative names. For any DN search base DN, we can ask this index for a list of all referral's in scope. I don't know if its better to store the full DN or to store relative names which can be added to the parent or ancestor DN to conserve space and prevent unnecessary DN component parsing.

This technique can btw be applied to aliases to pull the alias dereferencing mechanism out of partitions. Today aliases are bound by their partition because they are implemented within the default paritition. Going futher on this tangent, other things like DN normalization can also be pulled out of backends so they do not have to track both normalized and user provided names. This information can be provided.

These "master" indices can however cause other serious problems on second thought. First their data cannot be offlined with a partition. Meaning when a partition is taken offline either temporarily or forever the indices remain. To solve this B+Tree keys can be partition name or ID prefixed. Extended and management operations can be used to rebuild, and prune indices for perminently offlined partitions. Again the complexity grows. However this is not much of an issue for managing referrals since they will not be in as much abundance as regular entries. So it's still a reasonable approach. Some garbage may be collected but it will not be that much and it will not be looked up out of the index inadvertantly: meaning if we lookup referrals using the search base then we only see what is immediately subordinate to it or an ancestor of it. Offlined partition referrals can't really creep up into the picture.

Eureka!

if ( env.get( Context.REFERRAL ) != null || ! env.get( Context.REFERRAL ).equals( "ignore" ) )
{
   filter = ( | (objectClass=referral) (filter) )
}

Ok this referral index outside of partitions is a bit too much. This solution is better. The external index idea creates a subtle management issue along with cleanup overheads. Where do you put these indices and how do you scrub them down when partitions are dismounted? How do you make sure a dismounted partition is not mounted elsewhere by another server instance and altered to make indices inconsistent? You can't without rebuilding/rechecking index entries for that partition every time you mount or unmount it.

I'm going to go with this filter alteration solution for now.

Back on track with search continuations

So for each referral within scope, we have to return a SearchResultReference using the URI compoents of the ref attribute. Here's what we have to do to transform that URI:

• If the URI component is not a LDAP URL, it should be
  returned as is.
• If the LDAP URL's DN part is absent or empty, the DN
  part must be modified to contain the DN of the referral object.
• If the URI component is a LDAP URL, the URI SHOULD be modified to
  add an explicit scope specifier.

Subtree Example (From RFC 3296)

Subtree search of "O=MNN,C=WW" with filter (objectClass=*) might return:

SearchEntry for O=MNN,C=WW
SearchResultReference {
    ldap://hostb/OU=People,DC=example,DC=com??sub
    ldap://hostc/OU=People,O=MNN,C=WW??sub
}
SearchEntry for CN=Manager,O=MNN,C=WW
SearchResultReference {
    ldap://hostd/OU=Roles,dc=apache,dc=org??sub
}
SearchResultDone (success)

One Level Example (From RFC 3296)

Same search but scope is one level on the same base:

SearchResultReference {
    ldap://hostb/OU=People,DC=example,DC=com??sub
    ldap://hostc/OU=People,O=MNN,C=WW??sub
}
SearchEntry for CN=Manager,O=MNN,C=WW
SearchResultReference {
    ldap://hostd/OU=Roles,dc=apache,dc=org??sub
}
SearchResultDone (success)

Processing Considerations for Other Operations

Operations

We won't have to implement every operations in the interceptor : some of them are not necessary, like operations which do not modify the entries. For instance, bind() operation is not implemented.

Here is the list of operations defined in the interface, and the list of operations we implement in ReferralService (the missing methods are already implemented in the intermediate abstract class) :

Add

AddContextPartition

Compare

Delete

Init 

Modify 

Modify DN

If the newSuperior is a referral object or is subordinate to a
    referral object, the server SHOULD return affectsMultipleDSAs.  If
    the newRDN already exists but is a referral object, the server SHOULD
    return affectsMultipleDSAs instead of entryAlreadyExists.

Move

RemoveContextPartition

Search 

Conclusions

We will need to alter the ApacheDS JNDI provider, and the LDAP server frontend (MINA LDAP protocol provider) to handle referrals correctly. Here are the changes required for each subsystem.

Changes to JNDI Provider

• Add logic to check for parent and ancestor referrals when referral handling is not ignored
• Throw the appropriate ReferralExceptions with ref modification with referral handling set to throw
• Implement follow handling to chase referrals
• Add special handling for search to properly modify referral LDAP URLs
• Add code to alter the search operation when referral handling is not ignored
• Create and add search result enumeration filter to collect referrals and save them for returning last after the underlying enumeration has been exhausted of regular entries. This way we can return named continuation referrences last as JNDI LDAP providers are supposed to do.

Changes to MINA LDAP Frontend (Protocol Provider)

• Make handlers set the Context.REFERRAL property approapriate (ignore or throw)
• Make handlers correctly deal with ReferralExceptions for non-search operations
• Handle search continuations properly