Save the datasources, save the world.

by sil on April 5th, 2011

The Problem

While going about a routine hey, what’s that do at work, we discovered that, by default, Weblogic is configured to allow remote access to all JNDI resources. These resources include JDBC datasources. Datasources are defined to connect to a database as a specific user (this can be a proxy user, but not necessarily). What this means, is some intelligent person could gain access to the datasources configured on a Weblogic server and have all the powers allowed by the user defined in that datasource. This takes various levels of knowledge to be able to gain the correct host and port, and to be able to pass the right SQL to perform malicious operations. However difficult it may be, it is still a possibility.

With that in mind, Weblogic does provide a way to lock this down via security constraints. In fact, you can lock down many resources via these security policies which appear to be pretty open by default. However, enabling constraints can cause problems. Stack traces like the following can appear:

This presents us with the heart of the problem, how do we go about limiting access to these datasources and still allow the applications we maintain to function.

The Concept

The concept to protect these resources is simple, only allow access to the datasources to trusted entities (whether those be people or applications). The interesting thing about the error message above is Weblogic does not appear to know what application is running, by way of the application “null” response. It seems like they intended to know if the application was local or not, but does not appropriately track that. If that worked, one would think it would be possible to limit these JDBC datasources to only applications that the server knew about. At this time, this does not appear to be the case. Either the configuration options are not publicly known, or the implementation of the above behavior is not present.

So just what can we do? Weblogic provides a set of, somewhat restricted, constraints that can be applied. The constraints are added as policy conditions to a security policy defined for the chosen resource. Constraints include:

Role

Group

User

Access occurs before the specified day of the month

Context element’s value is greater than a numeric constant

Access occurs after

Access occurs on the specified day of the month

Access occurs between the specified hours

Access occurs on specified days of the week

Allow access to everyone

Deny access to every

Access occurs after the specified day of the month

Context element defined

Context element’s value equals a string constant

Context element’s value is less than a numeric constant

Access occurs before

Server is in development mode

Context element’s value equals a numeric constant

Some of these are obviously more useful than others. These restrictions could be applied at the JNDI level, or the JDBC level.

JNDI has the following operations (abridged definitions):

modify – ability to modify the JNDI tree in any way

lookup – ability to look up an object in the JNDI tree

list – ability to list the contents of a JNDI tree

Security policies can be applied to each operation individually, to all, or to a combination.

JDBC has the following datasource actions (kinda like operations (abridged definitions)):

admin – perform admin operations

reserve – reserve a connection in the datasource

reset – resets the datasource connection by shutting down and re-establishing all physical database connections

shrink – shrink the number of connections in the datasource

Security policies can be applied to each operation individually, to all, or a combination on a per datasource basis. It appears global settings (root level policies) can be applied to all JDBC resources, however, it appears it will add it to all actions at this level. i.e. you cannot have a global shrink setting different from a global reset setting, all or nothing!

This leads us to a crossroads; protect the datasources via JNDI access, JDBC access, or both! There are pros and cons to both, mostly dealing with how much control the application has over certain actions. Restricting either should be sufficient. If JNDI is protected, the application can never lookup the datasource, and therefore never create connections. If JDBC is protected, the application can never call reserve and therefore never create a connection either.

The JNDI Way

The JNDI Way focuses on protecting the datasource at the JNDI level. It may be possible that simply providing the right credentials during a JNDI lookup is all that will be needed, and once the datasource has been found operations can proceed as normal. Likewise, other environments that wire their own JNDI lookups through handwritten Java may be able to provide the correct credentials, and all will be good when it comes to JNDI operations. Certainly sounds promising. There are many ways that a JNDI lookup could take place, especially when dealing with libraries and frameworks. The Spring Framework, for instance, uses a bean JNDI lookup as like the following (abridged):

This bean gets used elsewhere as part of a property to Spring’s org.springframework.orm.hibernate3.LocalSessionFactoryBean, which is important later (JDBC side). It may be possible to extend, or create our own JNDI lookup bean that can be used by the LocalSessionFactoryBean that Spring uses. We hope to find a way to extend and efficiently authenticate this lookup.

The JDBC Way

The JDBC Way can be really simple if you are lucky, or incredibly complex if you are not.

In the infinity that is the Web, there appear to be a number of options that can help once access to the datasource has been restricted. However, as an application becomes more complex it often uses newer technologies (faster, stronger). These are sometimes black boxes and impossible to know what is truly happening inside (Weblogic). Sometimes these are glass boxes (sounds fitting) where you can see inside but changes must be delicate lest ye smash it to bits (Spring, Hibernate). Sometimes these are a box of legos where you have absolute control.

The focus of The JDBC Way will be the handful of possible entry points for authenticating the JDBC connections. Two big ones we plan to look at are: to do it at the statement level, and at the acquire connection level. It will really depend on how nice of a box we have to work with. If we are lucky the boxes will have nice entry points to let us extend, and manipulate. Our worst case is if there is a black box and we have no way of knowing what is happening inside. We hope to find the cleanest, simplest, and most efficient way to authenticate to the datasource.

The Ultimate Options of Ultimate Destiny

So what can we actually do?

The XMLs

XML configuration can be our friend. It is often high level, out of the way, not technically a code change. The Java Servlet spec has a run-as element that can be applied to servlet tags that appear in theweb.xml of a WAR. This tag appears to have the purpose to run functions within the scope of the servlet as a provided role (this is a role, not a user). It looks like the following example (abridged):

It appears this requires a security-role companion tag in the web.xml (it apparently wants to know you weren’t fooling and that you really provided a role it should know about):

<security-role><role-name>MYAWESOMEROLE</role-name></security-role>

This is where it gets container specific. Now that you have told the servlet to run as a role, and defined your no foolery security role, you have to tell the container what that role means. In WARs, Weblogic also contains a weblogic.xml that you will put this missing piece. You have two options at this point. You can define a security-role-assignment tag or you can define a run-as-role-assignment tag. The former appears to be a bit more flexible as it does exactly what latter does plus one more thing. The run-as-role-assignment tag maps a run-as role to a specific username:

This sounds great, like you could change this user at run-time right? Well, not so much. While that is how defined externally functions when dealing with the web.xmlsecurity-constraint tags, it is not the case for servlet run-as tags. The server simply errors because it cannot determine what the role means at startup time. So it becomes a roll a die choice.

That’s it. All looks pretty simple and straightforward to setup. So how well does it work? Turns out pretty dismally. While this can force the some part of the controller / action methods to be run as this particular role, it does not have the desired effect on the JDBC statement execution as a whole. It does also have the side effect of populating the user principal with the defined user and applicable roles that would be found had the user actually logged in themselves. Which also means, that a user cannot login themselves. They will be overwritten. So it looks like it does force the principal to be that particular user at some point, but really at all the wrong points. It forces the user, to well, be a another user and it’s not the point at which JDBC communication is happening, and that is what we really wanted. It also would not be at the time JNDI lookup is happening, at least in the case of Spring, as that is done completely outside the scope of a servlet call. The JNDI lookup happens during context initialization, which is at deploy time.

It might be of some use in the situation where you have a box of legos and can decide to put JNDI lookups or JDBC connections in the servlet methods themselves. I’d caution against it because if it is doing what I think it is doing, this setup will make a round trip authentication every time the servlet is accessed, even if no database action is needed to be performed. Let alone the fact that it potentially wipes out any previous authentication in the process. Best possible use is applications where user authentication is not expected, but protection of datasource is desired. However, as stated, even this may not appropriately associate the subject to the database operations.

The Weblogic Express

Java and Weblogic offer run-as annotations and methods to perform security like operations. The run-as annotations appear to be associated to The XMLs method as described above, it is just that you annotate the servlets instead of placing the information in XML. I found (and lost) an article that talked a little bit about Java’s doAs and doAsPriviledged vs Weblogics runAs method. They had mentioned that doAs did not associated the Subject with the actions performed in Weblogic, but that did not make much sense as I thought that was the whole point. I have not explored it at this time, but it is probably comparable to the following Weblogic runAs implementation. And since we are dealing with a Weblogic problem, why not use Weblogic methods. What’s the worst that can happen?!

Weblogic’s runAs is found in weblogic.security.Security. This method takes an already authenticated subject and an action class and then runs the class as though the provided subject initiated it. In order to accomplish you will need a little bit up set up. Imagine some doInitialize method where you want to get some stuff from the database. As stated, the runAs method needs a subject. In order do to that in Weblogic when you have no request and response object available, you have to create a login context, and login. The following example illustrates (slight namespace qualification for benefit):

Notice that you create a LoginContext and pass it two arguments. The first is the string “Sample”. This is a reference to a JAAS LoginModule. JAAS clients rely on a configuration file to maps names of JAAS login modules to their implementation as well as additional parameters. I recommend keeping it all one line. It appeared to dislike putting the additional parameters on separate lines.

The configuration file needs to be defined in a jaas.config file that gets passed to the server at startup. Why you cannot simply reference a name in your Weblogic security realm, I do not know. The easiest way is to define the argument and pass it as a startup variable.

-Djava.security.auth.login.config={PATH}jaas.config

The second argument is a CallBackHandler. This will be used by the LoginModule to retreive the username, password, and the URL of the Weblogic instance that will authenticate the subject. TheCallBackHandler can be very generic. Only one need exist. The following example illustrates our CallBackHandler (slight namespace qualification for benefit):

After the LoginContext is established, a login() is performed to authenticate the credentials and populate the subject, and then the subject is retrieved for use by Weblogic’s runAs method. This method’s second argument is a PrivledgedAction (or class that extend it) object which will perform (or initiate) the work we want done. The following example illustrates our PrivilegedAction extension (slight namespace qualification for benefit):

This action class can have a constructor to take whatever input parameters are required, and can return whatever object is necessary. The difficult part is using it in the right spot. Obviously you could write an action class for every database query you want to perform and wrap it inside the action. In a black box this may be your only option. It would save a lot of repetitive code to take this a step back and wrap your connection to the datasource creation in this type of action. Might be possible for this in glass boxes, though you may find yourself having to rewrite up a very long chain of classes to get to the appropriate level you need which can be a major downside when you go to upgrade versions. This is certainly an option for those with a box of legos, you wrote everything after all.

It is pretty flexible, and the best part is it does what we want it to. We perform whatever action to the database as the desired user and do not interfere with the current subject. The downside is we are making lots of round trip authentication connections whenever we want to do something to (or get data from) the database. We also have to store the password to the user somewhere. Sure we could get a little fancy and write an encryption utility to mask the password to anyone unless they have that encryption utility, the key to decrypt, and the encrypted password. Certainly makes us feel a little better. We also have to know the URL to connect to the Weblogic instance. And we are just plain writing a whole lot of repetitive code.

The Extendables

Mild mannered Java classes upon exposure to cosmic ray radiation turned into super heroes that saved us from the diabolical … I mean what? This section is all about Java classes extensions. The Weblogic Express gave us something that can work, but is not very efficient. So what are our alternatives? We already mentioned moving the action code further up the chain. So lets take a look at how we could go about doing that in our glass box with Spring and Hibernate.

Spring has a session factory bean implementation called LocalSessionFactoryBean (remember from earlier? I told you it was important!) for Hibernate (as well as other ORMs I believe) that is the central point for Hibernate management. It creates the session factory that will in turn will be used by the HibernateTransactionManger to manage all database transactions. Those look like prime candidates for extension. From the stack traces I received, I could see that the LocalSessionFactoryBean.buildSessionFactory method was being invoked in at least one case. Using what we learned above with the Weblogic Express, we can do the following:

Now this will act just like Spring’s implementation, except that it will build the session factory as though it were being performed by the provided subject. So what does this buy us? Apparently not all that much. This class is very near the surface for the general use. However, during deployment, is does appear to be along the right track as it stops an exception during context initialization (immediately following: Initializing connection provider: org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider). So it is somewhat useful.

LocalDataSourceConnectionProvider points us to something though, it is the Spring class that calls getConnection(). This sounds like a great place to wrap in an action. And it probably would could I only find out how where it gets set to use that class. It looks pretty integrated into the LocalSessionFactoryBean. There is where we will have to start breaking glass.

Lucky for us, Spring seems to have a pretty nice commitment to extendability. You just have to know where to look. Spring provides an interesting and useful processing method for us inLocalSessionFactoryBean.postProcessConfiguration. It is called near the tail end of the LocalSessionFactoryBean.buildSessionFactory method and is even said it is meant for subclasses that want to perform custom post-processing of the Configuration object after default initialization is performed. Sounds perfect! So what do we have to do?

First, we need hack our little MyAwesomeLocalSessionFacotryBean class up a bit. It no longer needs to wrap anything in an action, but it does need to override the postProcessConfiguration method. All this method needs to change is the connection provider class to a custom class we will make, instead of the LocalDataSourceConnectionProvider that is set by default. It does this by setting our custom class to the Configuration property Environment.CONNECTION_PROVIDER. We call the super method just in case Spring puts something in there later (currently it does not look like it does).

Second, we need to create ourselves a fancy class that extends LocalDataSourceConnectionProvider and overrides the getConnection we are so found of. It’s my precious. I mean our precious. Our! Since we are now experts at wrapping methods in actions, we slap one of those in there to reserve a connection to the datasource as though our desired user connected. Piece of cake.

Lastly, there is one important step that is easily overlooked. We need to tell Spring to use our awesome class in the applicationContext.xml. The only change here is to change the provided class of the session factory bean from Spring’s LocalSessionFactoryBean to our MyAwesomeLocalSessionFactoryBean class.

Wave the magic wand around a little bit, mumble up an incantation, and presto! Connections are created as the desired user. This appears to give us what we want, right? What do you mean right? We went through all that for just a right? Are you telling me this might not be what we want? Really?! Unfortunately, yes. This may not be what you want! While this certainly, somewhat elegantly (relatively speaking), gives us our desired effect. It also creates an overhead. Every single attempted database connection is going to first attempt to authenticate to whatever authentication mechanism you have in place. If your data access layer performs numerous little transactions, this overhead could be noticeable. Unfortunately, I cannot make that call for you. You will have to see if it negatively impacts performance in your application. This also has almost all the same cons as The Weblogic Express. We are still making numerous round trips to the authentication mechanism, storing that same sensitive info. In fact, all that is improved is time, effort, and code required. Don’t get me wrong, The Extendables is way easier to maintain. Way. But is it good enough?

The Filter of Antioch

A path to efficiency lay in the glorious filter. Yes, the holy hand grenade ain’t got nothin’ on the filter, except in Worms. The filter presents us with an opportunity. Filters intercept requests, and have access to the request and response objects the Weblogic needs for a certain servlet authentication call it can make. That’s right. I am suggesting using a filter to pre-process a request, check to see if an entity (user or our filter) has already authenticated against the server, and subsequently login as a designated user configured internally.

Since we all like Spring Security, lets take a look at an example there. The following is a custom filter that will be placed in Spring Security’s filter chain. The important part of the filter is Weblogic’sServletAuthentication.login method. This takes the user credentials, the request and response object, authenticates the user, and populates the subject. This means until the user themselves logs in, all actions the user takes will still be performed as an authenticated user.

//You may want this to be more sophisticated. The point is you don't want to login//unnecessarilyif(request.getRemoteUser()==null||("".equals(request.getRemoteUser())){try{int loginResultint = ServletAuthentication.login("username","password", request,
response);

We use the wordy XML version of Spring Security’s configuration, so we have a FilterChainProxy defined in the springSecurity_applicationContext.xml. However, this can just as easily be done using Spring Security namespaces version, and interject the filter where desired. The following is our Spring Security configuration tid bits. We placed it near the beginning because it seemed appropriate for our needs. But I am not necessarily sure it needs to be there. Our requirements are kinda goofy.

//You may want this to be more sophisticated. The point is you don't want to login//unnecessarilyif(request.getRemoteUser()==null||("".equals(request.getRemoteUser())){try{int loginResultint = ServletAuthentication.login("username","password", request,
response);

Now that we have the class defined, the web.xml needs the filter and filter-mapping defined. We are going map this filter to all URLs from the base context so that it can trigger on any resource access.

So there are the two filters. Look pretty similar don’t they? The main difference to note is the Spring filter we extend Spring’s GenericFilterBean and the generic filter we implement Java’s Filter class. On the pro side, this saves us a significant amount of authentication calls to whatever mechanism handles that operation, potentially alleviating us of our performance consideration. It does place a filter in the chain of request, but this filter only needs to check for an existing Subject, and login otherwise. It should be nice and efficient. We no longer need the Weblogic instance connection URL. On the con side we still have to store the user password, and this filter makes our datasource a little vulnerable. It now has to be configured to allow anyone who is authenticated to be able to access that datasource. You may be okay with that. Ideally there would be some sort of audit trail that could track down that a user did something nefarious. However, they could still do something, well, nefarious.

It is also not necessarily a complete solution. Your application may have initialization methods that try to load objects from the database. The filter will not be involved at that point. Your application may have server side processing. The filter will not be involved at that point either. This means you either have to re-design some part of the application, or end up having to use The Weblogic Express in some instances. So enter The Extra Round Trip Login Strikes Back and Return of the Connection URL.

The Secret Move

Finish him! URDLDR A+B B B+C. Fatality! That’s what we’re hoping to find anyway: a swift, decisive finishing move to defeat this problem. Remember The JNDI Way from oh so long ago? The solutions thus far have generally been focused on The JDBC Way, mostly because that way is the most accessible. But there is that shining light off in the distance. Could it be what we are looking for, or is it a mirage?

The JNDI Way discussed the potential of locking down just the JNDI operations. It should be sufficient to lock down JNDI operations to force restricted access to JDBC datasources. Let’s take a look at our glass box (Spring) again. Spring provides a way to do this as a generic bean via their JndiObjectFactoryBean class. The class performs the actual lookup in the afterPropertiesSet method. Extendables, let’s transform and roll out!

Now instead of using the jndi-lookup tag from The JNDI Way we instead have our own little bean in the applicationContext.xml (or however wherever it is defined). It is probably possible to extend the JEE schema that Spring has for the jndi-lookup tag to preserve that one line XML entry. However, I did not consider it worth the time to look into for the reward of saving two lines of XML but possibly creating more lines of Java.

Now if you have a box of legos,The Secret Move should be a piece of cake since you are all master of The Weblogic Express at this point. Wherever you do your JNDI lookup, you will have to at wrap it in an action. As long as you have a similar mindset, that you do not continually perform a lookup operation when access to the datasource is needed, you should tremendously save on round trip authentication calls. In fact, this should not even be noticeable. You’re rambling. I’m getting the feeling there is a pending but. You are right. Spring is not the only shiny glass box out there you have to worry about. I cannot, obviously, address them all since I am not currently using them all. However, there is one nagging box just waiting to be cracked, and that is Quartz.

Since Quartz is a well established open source project, I am sure it expects requirements from developers to be able to extend functionality, like our case. And luckily for us, they have been kind enough to give us an entry point. Out of the box, in their quarty.properties file, you can provide the initial context factory, url, username, and password as optional properties. However, once you provide one, you must provide all four or it gets cranky.

Well that sure was simple. Custom made properties just for us. It’s like they knew we were on a rant. The glaring downside is the password is sitting there in plain text. If this was a Java class, we could write our own custom encryption utility and decrypt the password on demand. But as a property file our hands are tied a little bit as no matter what, at some point it will appear in plain text within this property file. That may be an acceptable risk. Someone would need to get access to the source to be nefarious. But let’s see if Quartz has any more tricks for us, shall we?

Quartz allows you define your very own custom connection provider class by implementing Quartz’s ConnectionProvider class. This class is responsible implementing the getConnection method that returns a connection to the datasource, and for implementing the shutdown method to perform any cleanup. I put the JNDI lookup in a static block as a way to only do that lookup once. Is that necessary? Probably not, but I’m trying to minimize here! And with Quartz, it can ask for a connection every couple seconds. So let’s be friendly to ourselves.

@Overridepublicvoid shutdown()throwsSQLException{//I don't know what to do here!}}

Now that we have our class, we need to tell Quartz to use it. All we need is two lines in the quartz.properties file to define our datasource and connection provider itself. Quartz does allow parameters to be passed via property definitions as well should you need them, and then you can access them bean style with getters and setters.

Blamo! Our charged up special move ignited right in the face of our problem. It’s clean, efficient, simple, and short. We have the control we want, and are not hammering the authentication mechanism with login attempts. We still have to store user credentials and connection info, but we don’t need to open the datasource to all users. We can still work some encryption magic as needed on the password and not worry about it sitting in plaintext anywhere. While I was not able to demonstrate how to do this in all JNDI lookup situations, hopefully there is enough ground work to give you an idea. So what’s left?

The Final Chapter

Let’s recap.

The XMLs showed us the limitations of the passwordless configuration attempt.

The Weblogic Express gave us a way to wrap methods in privileged actions to execute them as a specific user, but was too repetitive and time consuming to do at the level of a statement.

The Extendables showed us we can use what we learned from The Weblogic Express to work at a higher level than statements to write simpler and easier to maintain actions, but still forced an authentication round trip for each connection.

The Filter of Antioch brought to light there is a better way than authenticating for every datasource connection, but had limitations on server initiated processing.

The Secret Move changed gears and gave us a clean, simple solution like The Extendables but gave use the limited authentication attempts of The Filter of Antioch.

Each option has a place, depending on the complexity of your application and the box you have to work with, whether it is a black box, a glass box, or a box of legos. While we had a black box (Weblogic) to constrain us, we were able to get around it by working with the other boxes that were involved. I personally like The Secret Move the best, but if the performance hit is okay, The Extendables is also a nice solution. Both need to have a designated user defined in the security policy for the JNDI or JDBC resource respectively. Both need to know the username and password for this user. And both need to know the connection URL for the Weblogic instance.

The examples were intended to be simple and generic. I have some things you may want to think on. Make required info properties so they can be configurable without a newly compiled EAR. Also, use some encryption mechanism on the password so it is not plain text. Sure, if they get a hold of the EAR it will have all they need (the decryption key and the functions to do it), but there may be good libraries out there for this (I really not have looked closely). Weblogic might even let you use theirs and not have to decrypt it within the application. You may be able to just pass the encrypted string to Weblogic and it will do the rest. I mentioned Java has similar methods to Weblogic’s specific runAs method in The Weblogic Express. It may be possible to make a Java Express to avoid vendor lock-in should other application servers behave similarly.

All is well. Our enemy is broken, charred, frozen, kersploded; you name it, we did it. It was that epic. Feels good to beat the final boss, doesn’t it?