*Reinforcement learning is like many topics with names ending in-ing, such as machine learning, planning, and mountaineering, in that it is simultaneously a problem, a class of solution methods that work well on the class of problems, and the field that studies these problems and their solution methods. Reinforcement learning problems involve learning what to do—how to map situations to actions—so as to maximize a numerical reward signal.
~Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction*
I am exploring RL in the context of robot learning.
Builds on the Markov decision process (MDP)
basically says next action depends on the current state only.
At each timestep $t$, an agent observes the current state $s_t$ and chooses an action $a_t$. The rule the agent uses to choose actions is called its policy, denoted by $\pi$. A policy can be deterministic, where it directly selects an action, or stochastic, where it defines a probability distribution over possible actions. In here we consider stochastic actions.
$$ \pi(a \mid s) = P(a_t = a \mid s_t = s) $$
After the agent takes an action, the environment provides a reward $r_t$ and transitions to a new state $s_{t+1}$. Repeating this process produces a trajectory.
$$ \tau = (s_0,a_0,r_0,s_1,a_1,r_1,\ldots) $$
The goal is not necessarily to maximize the immediate reward $r_t$. An action may give a small reward now but lead to much larger rewards later. We therefore define the return of a trajectory as the cumulative discounted reward.
what is $\gamma^t$?
this is the discount factor. we discount the reward because the reward now worth more than reward in the future.
$$ R(\tau) = \sum_{t=0}^{\infty} \gamma^t r_t
$$
rewards can be discrete or sparse (non smooth, +1 or -1), which means not differentiable. To find the optimal policy that produce high reward actions, we need to have a function differentiable over policy parameters $(\theta)$ . That is why we are considering expectation.
By definition, the expected value (expectation $\mathbb{E}$) means the sum of probability multiplied by the rewards.
Because the policy and/or environment can be stochastic, following the same policy does not necessarily produce the same trajectory every time. Instead of maximizing the return of one particular trajectory, we therefore maximize the expected return over trajectories generated by the policy. So we can define the policy objective $J(\pi)$:
$$ J(\pi) = \mathbb{E}{\tau \sim \pi}\left[\sum{t=0}^{\infty} \gamma^t r_t\right] $$