Introduction

This environment is based on Sutton and Barto's (1998) Mountain Car Task as described in Reinforcement Learning: An Introduction. This implementation was ported to RL-Glue from Sutton's Mountain Car Software (2000).
Consider a task where an agent must drive an underpowered car up a steep mountain road. Since gravity is stronger than the car’s engine, even at full throttle the car cannot simply accelerate up the steep slope. The car’s movement is described by two continuous output variables, position and velocity, and one discrete input representing the acceleration of the car.

This task is interesting because the car’s position on the hill and its velocity are real-valued. Therefore, a learning algorithm must use a function approximator to learn a good policy. Mountain car is also interesting because a successful control policy must drive the car backwards, up the other side of the valley, to gain enough momentum to drive forwards up the hill. This means the learning algorithm must move away from the goal, incurring additional negative reward, to discover the solution. Finally, actions do not have immediately measurable effects on the state of the system. Thus, learning algorithms must assign credit to actions taken several time steps in the past.

The Mountain Car task was originally presented by Andrew Moore in his PhD disertation (1990) and has been widely studied since. Singh and Sutton (1996) later used Mountain Car in their work on eligibility traces and formalized the state update equations for the position and velocity of the car based on Moore's original problem specification. Over the years there have been several variations on Singh and Sutton's version of the problem: different reward functions, starting states and termination conditions.

Technical Details

Observation Space

2 dimensional, continuous valued

car position (-1.2 to 0.6)
car velocity (-0.07 to 0.07)

Action Space

1 dimensional, discrete valued

reverse, neutral and forward

Rewards

negative reward per step

Termination

if car position is >= 0.6

Additional technical details

deterministic dynamics

optional random starting position and velocity

Parameters

Mountain Car currently has three parameters that can be set with RL-Viz:

Random Seed

Random Start States

Transition Noise

These parameters are not relevant to the conventional Mountain Car problem; they exist for experiments for generalized mountain car. This page will be updated at some near-future date to explain the implications of setting those parameters. For now, do not use special values for those settings when doing traditional Mountain Car experiments.

Using This Download

This download can be used to augment your existing local RL-Library (if you have one), or as the basis to start a new one.

This Is Your First Project

#Create a directory for your rl-library. Call it whatever you like.
mkdir rl-library

That's all you have to do special for the first time you download a rl-library component. Continue on now
to the next section.

Adding To An Existing RL-Library Download

#First, download the file. Depending on your platform, you might have to do this manually with a web browser.
#If you are on Linux, you can use wget which will download MountainCar-Java-R1308.tar.gz for you
wget http://rl-library.googlecode.com/files/MountainCar-Java-R1308.tar.gz
#Copy the download to your local rl-library folder (whatever it is called)
cp MountainCar-Java-R1308.tar.gz rl-library/
cd rl-library
#This will add any project-specific things necessary to system and products folders
#It will also create a folder for this particular project
tar -zxf MountainCar-Java-R1308.tar.gz
#Clean up
rm MountainCar-Java-R1308.tar.gz

After this step is completed, you will have several new files:

rl-library/

products/

MountainCar.jar

MountainCar-Java-R1308/

src/

build.xml

LICENSE.TXT

README.TXT

system/

common/

libs/

ant/

Compiling This Project

You must have Apache Ant installed to build this project using these instructions.

You don't have to compile this project, because the JAR file has been compiled
and placed into the products directory already. However, if you want to
make changes and recompile, type:

Running This Project

You can run this project by typing:

java -jar products/MountainCar.jar
#or from within the project's directory
cd MountainCar-Java-R1308
ant run

You can also use it in conjunction with RL-Viz by putting the JAR file
products/MountainCar.jar in the appropriate directory, as long as the
RL-Viz library jar file is in the appropriate relative location from
where you put MountainCar.jar. The location is:
../system/common/libs/rl-viz/RLVizLib.jar