Advanced Caching with Redis Cache

About application caching

Application caching is critical to running any large-scale web application. Understanding how application caching works and what some of the options you have to help your Blackboard Learn instance run efficiently, helps you make decisions about investing in hardware and software.

Ehcache

Ehcache is an open source cache library that Blackboard Learn uses by default. Its configurations for the core caches are defined in your Blackboard Learn installation inside the config/cache-settings.properties file.

Redis Cache

The Redis Cache Building Block enables Blackboard Learn's cache framework to use Redis as an additional type of memory store outside the JVM heap. Redis is an open source, high-performance, networked advanced key-value store. Other companies that use Redis include Twitter, Craigslist, Github, Stackoverflow, and Instagram.

To get started with the Redis Cache Building Block, you must have Blackboard Learn 9.1 SP 11 or later and a Redis server.

Performance, scalability, and availability highlights

Increases request throughput and response times

No risk of performance loss

Cache is shared across all application servers and the hottest subset of data is kept local

Cache is available under JVM heap pressure

Enables the system to scale with a manageable JVM heap size

Bigger cache and faster overflow storage

Fast and efficient object graph serialization

Asynchronous puts and updates via write-behind strategy

Increases off-heap caching opportunity

Reduces on-heap cluster invalidation overhead

Caches that survive application restarts and crashes

Rich monitoring features for preventive maintenance and troubleshooting

Redis Cache in action

Discover how to use Redis Cache to boost your efficiency when diagnosing and troubleshooting your Blackboard Learn servers.

Install and activate Redis Cache

You can use Software Updates to install the Redis Cache building block directly from the Administrator Panel. Alternatively, you can visit Behind the Blackboard Downloads and find the Redis Cache building block to download and install manually.

Software Updates

Software Updates are located on the System Admin panel. When a new building block is available or has been updated, you are notified by a change in the icons. Select the Feature Building Blocks Available icon and locate the Redis Cache building block. Select Install. You will need to make the building block available after it has been installed.

Behind the Blackboard

Download the Redis Cache building block from Behind the Blackboard Downloads home, located under Featured Building Blocks for Learn. Save the file on a local drive. Unzip the package to access the .war file.

On the Administrator Panel, go to Building Blocks > Installed Tools.

Select Upload Building Blocks.

Browse for the Redis Cache building block .war file.

Select Submit.

You will get a success message when the building block is installed. Set the building block to Active and select Approve.

Enable Redis Cache

After you install a Redis server, go to the Redis Cache building block settings page on the Administrator Panel.

Select the list of caches to enable Redis.

Provide Redis connection settings in the Host, Port, and Password fields. In newer versions of Blackboard Learn, these settings are in the bb-config.properties file.

Select Submit.

Make sure that the Redis server port and the RMI ports are open among Redis server and the Blackboard Learn application servers.

Restart all Blackboard Learn application servers. A rolling restart is recommended for production environments.

Navigate to the Redis connection status page on the Administrator Panel.

Verify that all application servers are connected to Redis.

Install Redis server

These steps assume that you are installing Redis on Red Hat Enterprise Linux 6.

Prepare a Linux server. A dedicated machine or VM instance is highly recommended. Don't run Redis on Windows or Solaris. Blackboard Learn on Windows and Solaris can use Redis on Linux.

Set the Linux kernel overcommit memory setting to 1 by adding vm.overcommit_memory=1 to /etc/sysctl.conf. Reboot the server or run the command sysctl vm.overcommit_memory=1 for the change to take effect. More information is available in the Redis FAQ.

Redis is a single-threaded server. It favors fast CPUs with large caches instead of many cores. Intel CPUs are currently known to be the best fit. Redis documentation states that it is not uncommon to get only half the performance on an AMD Opteron CPU compared to similar Nehalem EP, Westmere EP, or Sandy bridge Intel CPUs.

Redis throughput is limited by the network well before the CPU. Use fast and/or multiple NICs.

(Optional) Redis runs slower on a VM, so they recommend running Redis on a physical box.

Redis Cache Server Monitoring

Version 1.2 and above: Redis Cache Server Monitoring is accessible through the Redis Cache link in Tools and Utilities on the Administrator Panel.

Version 1.1 and below: Redis Cache Server Monitoring is accessible through the Admin Console. Look for Redis Cache navigation link in the main menu. If you do not have Admin Console installed, you can access Redis Cache Server Monitoring by manually pointing your browser to the URL path, /webapps/bb-redis-cache-BBLEARN/execute/redisStats.

Security

Harden the Redis server

Two components secure the Redis server:

Adding proper security configurations to the Redis application

Adding secure configurations to the server running Redis

Secure Redis settings make sure that the Redis application is as secure internally as it can be. Server configurations secure the server and the Redis application without involving the Redis application itself. The following sections describe two types of configurations.

Secure Redis settings

Password

The Redis server is very efficient at serial string lookups, which is exactly what the Redis password lookup request is. Additionally, the Redis password is required only by the Blackboard application. Users do not need to remember it, so the password used should be much stronger than a traditional password. Normally you should follow the owasp password length and complexity guidelines for setting password length, but because end users do not need to remember this password, the convention of choosing passwords based on users' ability to remember the password can be ignored.

Guideline

Reason

Use at least 16 characters, instead of 8

A brute force attack would take n to the 8th power times as long as an 8 character password, which is the normal minimum.

Mix lowercase and uppercase characters, numbers, and symbols

As with normal passwords, mixing case, numbers, and symbols reduces the likelihood of a successful dictionary attack and adds complexity for a brute force attack because all types of characters need to be checked.

Try not to use words or phrases in your password

Because this password does not have to be remembered, you can remove dictionary attacks entirely by not using any words or phrases.

The password setting itself must be added to the Redis configuration file, using the requirepass directive, such as:

Block Redis commands

Because Blackboard is using Redis just as a cache, a number of commands are not being used. Some of these commands could be used by a malicious user to cause Denial of Service via cache misses, denial of service by running the server out of space, as well as information exposure by allowing malicious users to obtain the entire set of cache data. Redis provides an interface for altering or dropping these commands entirely, which Blackboard recommends for a few specific commands:

Key

Removal recommended because

APPEND

Allows modification of keys to force cache misses

BGSAVE

Command that saves the dataset to disk, which could allow information exposure if a user obtains the file

RENAME

Renames a key, which would force cache misses

SAVE

Same as BGSAVE

SPOP

Removes a random member from the set

SREM

Removes one or more members from the set

To block these commands, edit the Redis configuration file to add the following:

By renaming the command to the empty string, you block anything from accessing it. Additionally, any new Redis commands that are created should also be blocked in this way because the building block will not be using them.

Secure server configuration

Securing the server running Redis means taking steps to ensure that only users who need access to the Redis server can access it, as well as preventing unauthorized access to the Redis application through file and network permissions.

Run Redis without root context

The user root, on Linux machines, has full access to any file on the system, regardless of permissions or access control. This is why it is dangerous to use the root user to run applications because well-crafted attacks could allow a malicious user to control the account running that application. Because Redis does not need that level of access to the system, a user account with decreased permissions, just enough to run Redis, should be used to install and run the Redis application.

Blackboard recommends not using this account for anything except running the Redis server. It should not have any permissions to the operating system or other applications or daemons running on this server.

Secure the Redis configuration file

The password used by the Redis server, which was described in an earlier section, is stored in cleartext within the configuration file. This is outside of Blackboard's control because this is how Redis functions at this time. So, you need to limit the users who have access to the file. It should be limited only to the user that runs the Redis application.

This can be accomplished using the following command, assuming that the redis configuration file is redis.conf:

chown 600 redis.conf

Block unnecessary traffic to Redis server

The Redis server, assuming the only application running on it is Redis, should not be accessible to the open internet. It should be able to be reached only in the following circumstances:

An administrator needs to do work on the Redis server

Blackboard communications over the Redis port

This will prevent unauthorized users from attempting to bring down the Redis server, run unauthorized commands in Redis, and so on. Assuming that the default Redis port, 6379 is in use, add the following types of rules to the firewall for this server:

Rule

Explanation

Block all traffic to and from the server

This should be the default behavior, except in the case of the exceptions below.

Create an exception for loopback requests

The server should be able to access itself, if for no other reason than monitoring incoming requests to Redis using the redis-cli commandmonitor.

Create an exception for traffic from and to the Blackboard servers only over Redis port

The Blackboard server should be able to communicate with Redis over port 6379, and Blackboard should be able to get responses from the Redis server.

Create an exception allowing for remote administration from a specific machine

Administrators may need access to the server for the purposes of upgrading Redis, checking logs, monitoring performance, and so on. Unless administrators have direct access to the machine, they will need to be able to access the server remotely. SSH traffic should be limited to a single machine, or a small set of machines, so normal users can't communicate to the server over port 22.

Administrators will also need to create exceptions for environment specific items, such as required shared resources. These will be specific to each institution.

Implement traffic monitoring

In case of attack against the Redis server from an outside source, some form of intrusion detection should be in place. A few items that can be monitored to flag traffic as possibly malicious, or drop the traffic entirely:

If anything other than a Blackboard application server or system administration server tries to access the Redis server, it should be flagged.

If a bad password is entered, it should be flagged.

If too many requests are made to Redis server from a single location, flag that traffic. Monitoring typical usage patterns, as well as speculating on edge cases (say, during finals week), can help determine how many requests per minute are expected.

Implementing these intrusion detection rules will help determine whether anyone is trying to penetrate the Redis server or application.

Building Block permissions

Java Type

Name

Actions

Why is this permission necessary?

persist

*

*

The Settings page needs to persist and load configuration data into SYSTEM_REGISTRY table

Persists and loads Redis server and cache statistics for the specified duration to the Stats database for trend analysis

java.io.FilePermission

1. BB_HOME/-

2. BB_CONTENT/-

3. "${java.home}/-

1. read, write, delete

2. read, write, delete

3. read

Reads properties files under Blackboard Learn home directory

Saves a hash value unique to a Blackboard Learn instance, shared by all Blackboard Learn application servers within a cluster. It is used in Redis cache key to avoid collision from other Redis cache usage by another Learn instance or other applications.

Ehcache diskoverflow feature

java.lang.reflect.
ReflectPermission

suppressAccessChecks

*

Used by Spring framework and an object graph serialization framework, Kryo

java.lang.RuntimePermission

*

*

Used by various Blackboard Learn components including log services and service managers