2
Reinforcement Learning (Ch , Ch. 20) Learner passive active Sequential decision problems Approaches: 1.Learn values of states (or state histories) & try to maximize utility of their outcomes. Need a model of the environment: what ops & what states they lead to 2.Learn values of state-action pairs Does not require a model of the environment (except legal moves) Cannot look ahead

4
Reinforcement Learning … M = 0.8 in direction you want to go 0.2 in perpendicular 0.1 left 0.1 right Policy: mapping from states to actions An optimal policy for the stochastic environment: utilities of states: Environment Observable (accessible): percept identifies the state Partially observable Markov property: Transition probabilities depend on state only, not on the path to the state. Markov decision problem (MDP). Partially observable MDP (POMDP): percepts does not have enough info to identify transition probabilities.

5
Partial observability in previous figure (2,1) vs. (2,3) U(A)  0.8*U(A) in (2,1) + 0.2*U(A) in (2,3) Have to factor in the value of new info obtained by moving in the world

8
Does not require there to exist a “last step” unlike dynamic programming

9
The utility values for selected states at each iteration step in the application of VALUE-ITERATION to the 4x3 world in our example Thrm: As t  , value iteration converges to exact U even if updates are done asynchronously & i is picked randomly at every step.

12
Value Determination Algorithm The VALUE-DETERMINATION algorithm can be implemented in one of two ways. The first is a simplification of the VALUE-ITERATION algorithm, replacing the equation (17.4) with and using the current utility estimates from policy iteration as the initial values. (Here Policy(i) is the action suggested by the policy in state i) While this can work well in some environments, it will often take a very long time to converge in the early stages of policy iteration. This is because the policy will be more or less random, so that many steps can be required to reach terminal states

13
Value Determination Algorithm The second approach is to solve for the utilities directly. Given a fixed policy P, the utilities of states obey a set of equations of the form: For example, suppose P is the policy shown in Figure 17.2(a). Then using the transition model M, we can construct the following set of equations: U(1,1) = 0.8u(1,2) + 0.1u(1,1) + 0.1u(2,1) U(1,2) = 0.8u(1,3) + 0.2u(1,2) and so on. This gives a set of 11 linear equations in 11 unknowns, which can be solved by linear algebra methods such as Gaussian elimination. For small state spaces, value determination using exact solution methods is often the most efficient approach. Policy iteration converges to optimal policy, and policy improves monotonically for all states. Asynchronous version converges to optimal policy if all states are visited infinitely often.

17
Passive learning … start (a) (b) (c) (a) A simple stochastic environment. (b) Each state transitions to a neighboring state with equal probability among all neighboring states. State (4,2) is terminal with reward –1, and state (4,3) is terminal with reward +1. (c) The exact utility values.

18
LMS – updating [Widrow & Hoff 1960] function LMS-UPDATE(U,e,percepts,M,N) returns an update U if TERMINAL?[e] then reward-to-go  0 for each e i in percepts (starting at end) do reward-to-go  reward-to-go + REWARD[e i ] U[STATE[e i ]]  RUNNING-AVERAGE (U[STATE[e i ]], reward-to-go, N[STATE[e i ]]) end Average reward-to-go that state has gotten simple average batch mode

19
Converges slowly to LMS estimate or training set

20
But utilities of states are not independent! NEW U = ? OLD U = -0.8 P=0.9 P= An example where LMS does poorly. A new state is reached for the first time, and then follows the path marked by the dashed lines, reaching a terminal state with reward +1.

21
Adaptive DP (ADP) Idea: use the constraints (state transition probabilities) between states to speed learning. Solve = value determination. No maximization over actions because agent is passive unlike in value iteration. using DP  Large state space e.g. Backgammon: equations in variables

22
Temporal Difference (TD) Learning Idea: Do ADP backups on a per move basis, not for the whole state space. Thrm: Average value of U(i) converges to the correct value. Thrm: If  is appropriately decreased as a function of times a state is visited (  =  [N[i]]), then U(i) itself converges to the correct value

23
Algorithm TD( ) (not in Russell & Norvig book) Idea: update from the whole epoch, not just on state transition. Special cases: =1: LMS =0: TD Intermediate choice of (between 0 and 1) is best. Interplay with  …

24

25
Convergence of TD( ) Thrm: Converges w.p. 1 under certain boundaries conditions. Decrease  i (t) s.t. In practice, often a fixed  is used for all i and t.

26
Passive learning in an unknown environment unknown ADP does not work directly LMS & TD( ) will operate unchanged … Changes to ADP Construct an environment model (of ) based on observations (state transitions) & run DP Quick in # epochs, slow update per example As the environment model approaches the correct model, the utility estimates will converge to the correct utilities.

27
Passive learning in an unknown environment ADP: full backup TD: one experience back up As TD makes a single adjustment (to U) per observed transitions, ADP makes as many (to U) as it needs to restore consistency between U and M. Change to M is local, but effects may need to be propagated throughout U.

28

29
Passive learning in an unknown environment TD can be viewed as a crude approximation of ADP Adjustments in ADP can be viewed as pseudo experience in TD A model for generating pseudo-experience can be used in TD directly: DYNA [Sutton] Cost of thinking vs. cost of acting Approximating iterations directly by restricting the backup after each observed transition. Prioritized sweeping heuristic prefers to make adjustments to states whose likely successors have just undergone large adjustments in U(j) - Learns roughly as fast as full ADP (#epochs) - Several orders of magnitude less computation  allows doing problems that are not solvable via ADP - M is incorrect early on  minimum decreasing adjustment size before recompute U(i)

30
Active learning in an unknown environment Agent considers what actions to take. Algorithms for learning in the setting (action choice discussed later) ADP: TD( ): Unchanged! Learn instead of as before Model-based (learn M) Model-free (e.g. Q-learning) Which is better? open Tradeoff

31
Q-learning Q (a,i) Direct approach (ADP) would require learning a model. Q-learning does not: Do this update after each state transition:

32
Exploration Tradeoff between exploitation (control) and exploration (identification) Extremes: greedy vs. random acting (n-armed bandit models) Q-learning converges to optimal Q-values if * Every state is visited infinitely often (due to exploration), * The action selection becomes greedy as time approaches infinity, and * The learning rate  is decreased fast enough but not too fast (as we discussed in TD learning)