# Reinforcement Learning

Ce TP concerne la découverte de l'apprentissage par renforcement appliqué au domaine du gaming. Les observations seront données par un émulateur de jeux atari. L'objectif de ce tp est la mise en oeuvre de différents algorithmes types **policy gradient**.

## 1. Setup

Import de quelques modules

```python
# Python ≥3.5 is required
import sys
assert sys.version_info >= (3, 5)

# Scikit-Learn ≥0.20 is required
import sklearn
assert sklearn.__version__ >= "0.20"

# Installation de openai gym et des agents
try:
    # %tensorflow_version only exists in Colab.
    %tensorflow_version 2.x
    !apt update && apt install -y libpq-dev libsdl2-dev swig xorg-dev xvfb
    !pip install -q -U tf-agents-nightly pyvirtualdisplay gym[atari]
    IS_COLAB = True
    try:
        import pyvirtualdisplay
        display = pyvirtualdisplay.Display(visible=0, size=(1400, 900)).start()
    except ImportError:
        pass
except Exception:
    IS_COLAB = False

# TensorFlow ≥2.0 is required
import tensorflow as tf
from tensorflow import keras
assert tf.__version__ >= "2.0"

if not tf.config.list_physical_devices('GPU'):
    print("No GPU was detected. CNNs can be very slow without a GPU.")
    if IS_COLAB:
        print("Go to Runtime > Change runtime and select a GPU hardware accelerator.")

# Common imports
import numpy as np
import os

# to make this notebook's output stable across runs
seed = 42
np.random.seed(seed)
tf.random.set_seed(seed)

# To plot pretty figures
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
mpl.rc('axes', labelsize=14)
mpl.rc('xtick', labelsize=12)
mpl.rc('ytick', labelsize=12)

# To get smooth animations
import matplotlib.animation as animation
mpl.rc('animation', html='jshtml')

```

# 2. Introduction to OpenAI gym

Dans ce notebook, l'environnement et les observations/données seront fournis par le framework [OpenAI gym](https://gym.openai.com/). Il fournit de nombreux environnements avec lesquels peut interagir et apprendre votre *agents*.
L'import se fait classiquement par:

```python
import gym
```

La liste des environnements disponibles est donnée par:

```python
gym.envs.registry.all()
```

    dict_values([EnvSpec(Copy-v0), EnvSpec(RepeatCopy-v0), EnvSpec(ReversedAddition-v0), EnvSpec(ReversedAddition3-v0), EnvSpec(DuplicatedInput-v0), EnvSpec(Reverse-v0), EnvSpec(CartPole-v0), EnvSpec(CartPole-v1), EnvSpec(MountainCar-v0), EnvSpec(MountainCarContinuous-v0), EnvSpec(Pendulum-v0), EnvSpec(Acrobot-v1), EnvSpec(LunarLander-v2), EnvSpec(LunarLanderContinuous-v2), EnvSpec(BipedalWalker-v3), EnvSpec(BipedalWalkerHardcore-v3), EnvSpec(CarRacing-v0), EnvSpec(Blackjack-v0), EnvSpec(KellyCoinflip-v0), EnvSpec(KellyCoinflipGeneralized-v0), EnvSpec(FrozenLake-v0), EnvSpec(FrozenLake8x8-v0), EnvSpec(CliffWalking-v0), EnvSpec(NChain-v0), EnvSpec(Roulette-v0), EnvSpec(Taxi-v3), EnvSpec(GuessingGame-v0), EnvSpec(HotterColder-v0), EnvSpec(Reacher-v2), EnvSpec(Pusher-v2), EnvSpec(Thrower-v2), EnvSpec(Striker-v2), EnvSpec(InvertedPendulum-v2), EnvSpec(InvertedDoublePendulum-v2), EnvSpec(HalfCheetah-v2), EnvSpec(HalfCheetah-v3), EnvSpec(Hopper-v2), EnvSpec(Hopper-v3), EnvSpec(Swimmer-v2), EnvSpec(Swimmer-v3), EnvSpec(Walker2d-v2), EnvSpec(Walker2d-v3), EnvSpec(Ant-v2), EnvSpec(Ant-v3), EnvSpec(Humanoid-v2), EnvSpec(Humanoid-v3), EnvSpec(HumanoidStandup-v2), EnvSpec(FetchSlide-v1), EnvSpec(FetchPickAndPlace-v1), EnvSpec(FetchReach-v1), EnvSpec(FetchPush-v1), EnvSpec(HandReach-v0), EnvSpec(HandManipulateBlockRotateZ-v0), EnvSpec(HandManipulateBlockRotateZTouchSensors-v0), EnvSpec(HandManipulateBlockRotateZTouchSensors-v1), EnvSpec(HandManipulateBlockRotateParallel-v0), EnvSpec(HandManipulateBlockRotateParallelTouchSensors-v0), EnvSpec(HandManipulateBlockRotateParallelTouchSensors-v1), EnvSpec(HandManipulateBlockRotateXYZ-v0), EnvSpec(HandManipulateBlockRotateXYZTouchSensors-v0), EnvSpec(HandManipulateBlockRotateXYZTouchSensors-v1), EnvSpec(HandManipulateBlockFull-v0), EnvSpec(HandManipulateBlock-v0), EnvSpec(HandManipulateBlockTouchSensors-v0), EnvSpec(HandManipulateBlockTouchSensors-v1), EnvSpec(HandManipulateEggRotate-v0), EnvSpec(HandManipulateEggRotateTouchSensors-v0), EnvSpec(HandManipulateEggRotateTouchSensors-v1), EnvSpec(HandManipulateEggFull-v0), EnvSpec(HandManipulateEgg-v0), EnvSpec(HandManipulateEggTouchSensors-v0), EnvSpec(HandManipulateEggTouchSensors-v1), EnvSpec(HandManipulatePenRotate-v0), EnvSpec(HandManipulatePenRotateTouchSensors-v0), EnvSpec(HandManipulatePenRotateTouchSensors-v1), EnvSpec(HandManipulatePenFull-v0), EnvSpec(HandManipulatePen-v0), EnvSpec(HandManipulatePenTouchSensors-v0), EnvSpec(HandManipulatePenTouchSensors-v1), EnvSpec(FetchSlideDense-v1), EnvSpec(FetchPickAndPlaceDense-v1), EnvSpec(FetchReachDense-v1), EnvSpec(FetchPushDense-v1), EnvSpec(HandReachDense-v0), EnvSpec(HandManipulateBlockRotateZDense-v0), EnvSpec(HandManipulateBlockRotateZTouchSensorsDense-v0), EnvSpec(HandManipulateBlockRotateZTouchSensorsDense-v1), EnvSpec(HandManipulateBlockRotateParallelDense-v0), EnvSpec(HandManipulateBlockRotateParallelTouchSensorsDense-v0), EnvSpec(HandManipulateBlockRotateParallelTouchSensorsDense-v1), EnvSpec(HandManipulateBlockRotateXYZDense-v0), EnvSpec(HandManipulateBlockRotateXYZTouchSensorsDense-v0), EnvSpec(HandManipulateBlockRotateXYZTouchSensorsDense-v1), EnvSpec(HandManipulateBlockFullDense-v0), EnvSpec(HandManipulateBlockDense-v0), EnvSpec(HandManipulateBlockTouchSensorsDense-v0), EnvSpec(HandManipulateBlockTouchSensorsDense-v1), EnvSpec(HandManipulateEggRotateDense-v0), EnvSpec(HandManipulateEggRotateTouchSensorsDense-v0), EnvSpec(HandManipulateEggRotateTouchSensorsDense-v1), EnvSpec(HandManipulateEggFullDense-v0), EnvSpec(HandManipulateEggDense-v0), EnvSpec(HandManipulateEggTouchSensorsDense-v0), EnvSpec(HandManipulateEggTouchSensorsDense-v1), EnvSpec(HandManipulatePenRotateDense-v0), EnvSpec(HandManipulatePenRotateTouchSensorsDense-v0), EnvSpec(HandManipulatePenRotateTouchSensorsDense-v1), EnvSpec(HandManipulatePenFullDense-v0), EnvSpec(HandManipulatePenDense-v0), EnvSpec(HandManipulatePenTouchSensorsDense-v0), EnvSpec(HandManipulatePenTouchSensorsDense-v1), EnvSpec(Adventure-v0), EnvSpec(Adventure-v4), EnvSpec(AdventureDeterministic-v0), EnvSpec(AdventureDeterministic-v4), EnvSpec(AdventureNoFrameskip-v0), EnvSpec(AdventureNoFrameskip-v4), EnvSpec(Adventure-ram-v0), EnvSpec(Adventure-ram-v4), EnvSpec(Adventure-ramDeterministic-v0), EnvSpec(Adventure-ramDeterministic-v4), EnvSpec(Adventure-ramNoFrameskip-v0), EnvSpec(Adventure-ramNoFrameskip-v4), EnvSpec(AirRaid-v0), EnvSpec(AirRaid-v4), EnvSpec(AirRaidDeterministic-v0), EnvSpec(AirRaidDeterministic-v4), EnvSpec(AirRaidNoFrameskip-v0), EnvSpec(AirRaidNoFrameskip-v4), EnvSpec(AirRaid-ram-v0), EnvSpec(AirRaid-ram-v4), EnvSpec(AirRaid-ramDeterministic-v0), EnvSpec(AirRaid-ramDeterministic-v4), EnvSpec(AirRaid-ramNoFrameskip-v0), EnvSpec(AirRaid-ramNoFrameskip-v4), EnvSpec(Alien-v0), EnvSpec(Alien-v4), EnvSpec(AlienDeterministic-v0), EnvSpec(AlienDeterministic-v4), EnvSpec(AlienNoFrameskip-v0), EnvSpec(AlienNoFrameskip-v4), EnvSpec(Alien-ram-v0), EnvSpec(Alien-ram-v4), EnvSpec(Alien-ramDeterministic-v0), EnvSpec(Alien-ramDeterministic-v4), EnvSpec(Alien-ramNoFrameskip-v0), EnvSpec(Alien-ramNoFrameskip-v4), EnvSpec(Amidar-v0), EnvSpec(Amidar-v4), EnvSpec(AmidarDeterministic-v0), EnvSpec(AmidarDeterministic-v4), EnvSpec(AmidarNoFrameskip-v0), EnvSpec(AmidarNoFrameskip-v4), EnvSpec(Amidar-ram-v0), EnvSpec(Amidar-ram-v4), EnvSpec(Amidar-ramDeterministic-v0), EnvSpec(Amidar-ramDeterministic-v4), EnvSpec(Amidar-ramNoFrameskip-v0), EnvSpec(Amidar-ramNoFrameskip-v4), EnvSpec(Assault-v0), EnvSpec(Assault-v4), EnvSpec(AssaultDeterministic-v0), EnvSpec(AssaultDeterministic-v4), EnvSpec(AssaultNoFrameskip-v0), EnvSpec(AssaultNoFrameskip-v4), EnvSpec(Assault-ram-v0), EnvSpec(Assault-ram-v4), EnvSpec(Assault-ramDeterministic-v0), EnvSpec(Assault-ramDeterministic-v4), EnvSpec(Assault-ramNoFrameskip-v0), EnvSpec(Assault-ramNoFrameskip-v4), EnvSpec(Asterix-v0), EnvSpec(Asterix-v4), EnvSpec(AsterixDeterministic-v0), EnvSpec(AsterixDeterministic-v4), EnvSpec(AsterixNoFrameskip-v0), EnvSpec(AsterixNoFrameskip-v4), EnvSpec(Asterix-ram-v0), EnvSpec(Asterix-ram-v4), EnvSpec(Asterix-ramDeterministic-v0), EnvSpec(Asterix-ramDeterministic-v4), EnvSpec(Asterix-ramNoFrameskip-v0), EnvSpec(Asterix-ramNoFrameskip-v4), EnvSpec(Asteroids-v0), EnvSpec(Asteroids-v4), EnvSpec(AsteroidsDeterministic-v0), EnvSpec(AsteroidsDeterministic-v4), EnvSpec(AsteroidsNoFrameskip-v0), EnvSpec(AsteroidsNoFrameskip-v4), EnvSpec(Asteroids-ram-v0), EnvSpec(Asteroids-ram-v4), EnvSpec(Asteroids-ramDeterministic-v0), EnvSpec(Asteroids-ramDeterministic-v4), EnvSpec(Asteroids-ramNoFrameskip-v0), EnvSpec(Asteroids-ramNoFrameskip-v4), EnvSpec(Atlantis-v0), EnvSpec(Atlantis-v4), EnvSpec(AtlantisDeterministic-v0), EnvSpec(AtlantisDeterministic-v4), EnvSpec(AtlantisNoFrameskip-v0), EnvSpec(AtlantisNoFrameskip-v4), EnvSpec(Atlantis-ram-v0), EnvSpec(Atlantis-ram-v4), EnvSpec(Atlantis-ramDeterministic-v0), EnvSpec(Atlantis-ramDeterministic-v4), EnvSpec(Atlantis-ramNoFrameskip-v0), EnvSpec(Atlantis-ramNoFrameskip-v4), EnvSpec(BankHeist-v0), EnvSpec(BankHeist-v4), EnvSpec(BankHeistDeterministic-v0), EnvSpec(BankHeistDeterministic-v4), EnvSpec(BankHeistNoFrameskip-v0), EnvSpec(BankHeistNoFrameskip-v4), EnvSpec(BankHeist-ram-v0), EnvSpec(BankHeist-ram-v4), EnvSpec(BankHeist-ramDeterministic-v0), EnvSpec(BankHeist-ramDeterministic-v4), EnvSpec(BankHeist-ramNoFrameskip-v0), EnvSpec(BankHeist-ramNoFrameskip-v4), EnvSpec(BattleZone-v0), EnvSpec(BattleZone-v4), EnvSpec(BattleZoneDeterministic-v0), EnvSpec(BattleZoneDeterministic-v4), EnvSpec(BattleZoneNoFrameskip-v0), EnvSpec(BattleZoneNoFrameskip-v4), EnvSpec(BattleZone-ram-v0), EnvSpec(BattleZone-ram-v4), EnvSpec(BattleZone-ramDeterministic-v0), EnvSpec(BattleZone-ramDeterministic-v4), EnvSpec(BattleZone-ramNoFrameskip-v0), EnvSpec(BattleZone-ramNoFrameskip-v4), EnvSpec(BeamRider-v0), EnvSpec(BeamRider-v4), EnvSpec(BeamRiderDeterministic-v0), EnvSpec(BeamRiderDeterministic-v4), EnvSpec(BeamRiderNoFrameskip-v0), EnvSpec(BeamRiderNoFrameskip-v4), EnvSpec(BeamRider-ram-v0), EnvSpec(BeamRider-ram-v4), EnvSpec(BeamRider-ramDeterministic-v0), EnvSpec(BeamRider-ramDeterministic-v4), EnvSpec(BeamRider-ramNoFrameskip-v0), EnvSpec(BeamRider-ramNoFrameskip-v4), EnvSpec(Berzerk-v0), EnvSpec(Berzerk-v4), EnvSpec(BerzerkDeterministic-v0), EnvSpec(BerzerkDeterministic-v4), EnvSpec(BerzerkNoFrameskip-v0), EnvSpec(BerzerkNoFrameskip-v4), EnvSpec(Berzerk-ram-v0), EnvSpec(Berzerk-ram-v4), EnvSpec(Berzerk-ramDeterministic-v0), EnvSpec(Berzerk-ramDeterministic-v4), EnvSpec(Berzerk-ramNoFrameskip-v0), EnvSpec(Berzerk-ramNoFrameskip-v4), EnvSpec(Bowling-v0), EnvSpec(Bowling-v4), EnvSpec(BowlingDeterministic-v0), EnvSpec(BowlingDeterministic-v4), EnvSpec(BowlingNoFrameskip-v0), EnvSpec(BowlingNoFrameskip-v4), EnvSpec(Bowling-ram-v0), EnvSpec(Bowling-ram-v4), EnvSpec(Bowling-ramDeterministic-v0), EnvSpec(Bowling-ramDeterministic-v4), EnvSpec(Bowling-ramNoFrameskip-v0), EnvSpec(Bowling-ramNoFrameskip-v4), EnvSpec(Boxing-v0), EnvSpec(Boxing-v4), EnvSpec(BoxingDeterministic-v0), EnvSpec(BoxingDeterministic-v4), EnvSpec(BoxingNoFrameskip-v0), EnvSpec(BoxingNoFrameskip-v4), EnvSpec(Boxing-ram-v0), EnvSpec(Boxing-ram-v4), EnvSpec(Boxing-ramDeterministic-v0), EnvSpec(Boxing-ramDeterministic-v4), EnvSpec(Boxing-ramNoFrameskip-v0), EnvSpec(Boxing-ramNoFrameskip-v4), EnvSpec(Breakout-v0), EnvSpec(Breakout-v4), EnvSpec(BreakoutDeterministic-v0), EnvSpec(BreakoutDeterministic-v4), EnvSpec(BreakoutNoFrameskip-v0), EnvSpec(BreakoutNoFrameskip-v4), EnvSpec(Breakout-ram-v0), EnvSpec(Breakout-ram-v4), EnvSpec(Breakout-ramDeterministic-v0), EnvSpec(Breakout-ramDeterministic-v4), EnvSpec(Breakout-ramNoFrameskip-v0), EnvSpec(Breakout-ramNoFrameskip-v4), EnvSpec(Carnival-v0), EnvSpec(Carnival-v4), EnvSpec(CarnivalDeterministic-v0), EnvSpec(CarnivalDeterministic-v4), EnvSpec(CarnivalNoFrameskip-v0), EnvSpec(CarnivalNoFrameskip-v4), EnvSpec(Carnival-ram-v0), EnvSpec(Carnival-ram-v4), EnvSpec(Carnival-ramDeterministic-v0), EnvSpec(Carnival-ramDeterministic-v4), EnvSpec(Carnival-ramNoFrameskip-v0), EnvSpec(Carnival-ramNoFrameskip-v4), EnvSpec(Centipede-v0), EnvSpec(Centipede-v4), EnvSpec(CentipedeDeterministic-v0), EnvSpec(CentipedeDeterministic-v4), EnvSpec(CentipedeNoFrameskip-v0), EnvSpec(CentipedeNoFrameskip-v4), EnvSpec(Centipede-ram-v0), EnvSpec(Centipede-ram-v4), EnvSpec(Centipede-ramDeterministic-v0), EnvSpec(Centipede-ramDeterministic-v4), EnvSpec(Centipede-ramNoFrameskip-v0), EnvSpec(Centipede-ramNoFrameskip-v4), EnvSpec(ChopperCommand-v0), EnvSpec(ChopperCommand-v4), EnvSpec(ChopperCommandDeterministic-v0), EnvSpec(ChopperCommandDeterministic-v4), EnvSpec(ChopperCommandNoFrameskip-v0), EnvSpec(ChopperCommandNoFrameskip-v4), EnvSpec(ChopperCommand-ram-v0), EnvSpec(ChopperCommand-ram-v4), EnvSpec(ChopperCommand-ramDeterministic-v0), EnvSpec(ChopperCommand-ramDeterministic-v4), EnvSpec(ChopperCommand-ramNoFrameskip-v0), EnvSpec(ChopperCommand-ramNoFrameskip-v4), EnvSpec(CrazyClimber-v0), EnvSpec(CrazyClimber-v4), EnvSpec(CrazyClimberDeterministic-v0), EnvSpec(CrazyClimberDeterministic-v4), EnvSpec(CrazyClimberNoFrameskip-v0), EnvSpec(CrazyClimberNoFrameskip-v4), EnvSpec(CrazyClimber-ram-v0), EnvSpec(CrazyClimber-ram-v4), EnvSpec(CrazyClimber-ramDeterministic-v0), EnvSpec(CrazyClimber-ramDeterministic-v4), EnvSpec(CrazyClimber-ramNoFrameskip-v0), EnvSpec(CrazyClimber-ramNoFrameskip-v4), EnvSpec(Defender-v0), EnvSpec(Defender-v4), EnvSpec(DefenderDeterministic-v0), EnvSpec(DefenderDeterministic-v4), EnvSpec(DefenderNoFrameskip-v0), EnvSpec(DefenderNoFrameskip-v4), EnvSpec(Defender-ram-v0), EnvSpec(Defender-ram-v4), EnvSpec(Defender-ramDeterministic-v0), EnvSpec(Defender-ramDeterministic-v4), EnvSpec(Defender-ramNoFrameskip-v0), EnvSpec(Defender-ramNoFrameskip-v4), EnvSpec(DemonAttack-v0), EnvSpec(DemonAttack-v4), EnvSpec(DemonAttackDeterministic-v0), EnvSpec(DemonAttackDeterministic-v4), EnvSpec(DemonAttackNoFrameskip-v0), EnvSpec(DemonAttackNoFrameskip-v4), EnvSpec(DemonAttack-ram-v0), EnvSpec(DemonAttack-ram-v4), EnvSpec(DemonAttack-ramDeterministic-v0), EnvSpec(DemonAttack-ramDeterministic-v4), EnvSpec(DemonAttack-ramNoFrameskip-v0), EnvSpec(DemonAttack-ramNoFrameskip-v4), EnvSpec(DoubleDunk-v0), EnvSpec(DoubleDunk-v4), EnvSpec(DoubleDunkDeterministic-v0), EnvSpec(DoubleDunkDeterministic-v4), EnvSpec(DoubleDunkNoFrameskip-v0), EnvSpec(DoubleDunkNoFrameskip-v4), EnvSpec(DoubleDunk-ram-v0), EnvSpec(DoubleDunk-ram-v4), EnvSpec(DoubleDunk-ramDeterministic-v0), EnvSpec(DoubleDunk-ramDeterministic-v4), EnvSpec(DoubleDunk-ramNoFrameskip-v0), EnvSpec(DoubleDunk-ramNoFrameskip-v4), EnvSpec(ElevatorAction-v0), EnvSpec(ElevatorAction-v4), EnvSpec(ElevatorActionDeterministic-v0), EnvSpec(ElevatorActionDeterministic-v4), EnvSpec(ElevatorActionNoFrameskip-v0), EnvSpec(ElevatorActionNoFrameskip-v4), EnvSpec(ElevatorAction-ram-v0), EnvSpec(ElevatorAction-ram-v4), EnvSpec(ElevatorAction-ramDeterministic-v0), EnvSpec(ElevatorAction-ramDeterministic-v4), EnvSpec(ElevatorAction-ramNoFrameskip-v0), EnvSpec(ElevatorAction-ramNoFrameskip-v4), EnvSpec(Enduro-v0), EnvSpec(Enduro-v4), EnvSpec(EnduroDeterministic-v0), EnvSpec(EnduroDeterministic-v4), EnvSpec(EnduroNoFrameskip-v0), EnvSpec(EnduroNoFrameskip-v4), EnvSpec(Enduro-ram-v0), EnvSpec(Enduro-ram-v4), EnvSpec(Enduro-ramDeterministic-v0), EnvSpec(Enduro-ramDeterministic-v4), EnvSpec(Enduro-ramNoFrameskip-v0), EnvSpec(Enduro-ramNoFrameskip-v4), EnvSpec(FishingDerby-v0), EnvSpec(FishingDerby-v4), EnvSpec(FishingDerbyDeterministic-v0), EnvSpec(FishingDerbyDeterministic-v4), EnvSpec(FishingDerbyNoFrameskip-v0), EnvSpec(FishingDerbyNoFrameskip-v4), EnvSpec(FishingDerby-ram-v0), EnvSpec(FishingDerby-ram-v4), EnvSpec(FishingDerby-ramDeterministic-v0), EnvSpec(FishingDerby-ramDeterministic-v4), EnvSpec(FishingDerby-ramNoFrameskip-v0), EnvSpec(FishingDerby-ramNoFrameskip-v4), EnvSpec(Freeway-v0), EnvSpec(Freeway-v4), EnvSpec(FreewayDeterministic-v0), EnvSpec(FreewayDeterministic-v4), EnvSpec(FreewayNoFrameskip-v0), EnvSpec(FreewayNoFrameskip-v4), EnvSpec(Freeway-ram-v0), EnvSpec(Freeway-ram-v4), EnvSpec(Freeway-ramDeterministic-v0), EnvSpec(Freeway-ramDeterministic-v4), EnvSpec(Freeway-ramNoFrameskip-v0), EnvSpec(Freeway-ramNoFrameskip-v4), EnvSpec(Frostbite-v0), EnvSpec(Frostbite-v4), EnvSpec(FrostbiteDeterministic-v0), EnvSpec(FrostbiteDeterministic-v4), EnvSpec(FrostbiteNoFrameskip-v0), EnvSpec(FrostbiteNoFrameskip-v4), EnvSpec(Frostbite-ram-v0), EnvSpec(Frostbite-ram-v4), EnvSpec(Frostbite-ramDeterministic-v0), EnvSpec(Frostbite-ramDeterministic-v4), EnvSpec(Frostbite-ramNoFrameskip-v0), EnvSpec(Frostbite-ramNoFrameskip-v4), EnvSpec(Gopher-v0), EnvSpec(Gopher-v4), EnvSpec(GopherDeterministic-v0), EnvSpec(GopherDeterministic-v4), EnvSpec(GopherNoFrameskip-v0), EnvSpec(GopherNoFrameskip-v4), EnvSpec(Gopher-ram-v0), EnvSpec(Gopher-ram-v4), EnvSpec(Gopher-ramDeterministic-v0), EnvSpec(Gopher-ramDeterministic-v4), EnvSpec(Gopher-ramNoFrameskip-v0), EnvSpec(Gopher-ramNoFrameskip-v4), EnvSpec(Gravitar-v0), EnvSpec(Gravitar-v4), EnvSpec(GravitarDeterministic-v0), EnvSpec(GravitarDeterministic-v4), EnvSpec(GravitarNoFrameskip-v0), EnvSpec(GravitarNoFrameskip-v4), EnvSpec(Gravitar-ram-v0), EnvSpec(Gravitar-ram-v4), EnvSpec(Gravitar-ramDeterministic-v0), EnvSpec(Gravitar-ramDeterministic-v4), EnvSpec(Gravitar-ramNoFrameskip-v0), EnvSpec(Gravitar-ramNoFrameskip-v4), EnvSpec(Hero-v0), EnvSpec(Hero-v4), EnvSpec(HeroDeterministic-v0), EnvSpec(HeroDeterministic-v4), EnvSpec(HeroNoFrameskip-v0), EnvSpec(HeroNoFrameskip-v4), EnvSpec(Hero-ram-v0), EnvSpec(Hero-ram-v4), EnvSpec(Hero-ramDeterministic-v0), EnvSpec(Hero-ramDeterministic-v4), EnvSpec(Hero-ramNoFrameskip-v0), EnvSpec(Hero-ramNoFrameskip-v4), EnvSpec(IceHockey-v0), EnvSpec(IceHockey-v4), EnvSpec(IceHockeyDeterministic-v0), EnvSpec(IceHockeyDeterministic-v4), EnvSpec(IceHockeyNoFrameskip-v0), EnvSpec(IceHockeyNoFrameskip-v4), EnvSpec(IceHockey-ram-v0), EnvSpec(IceHockey-ram-v4), EnvSpec(IceHockey-ramDeterministic-v0), EnvSpec(IceHockey-ramDeterministic-v4), EnvSpec(IceHockey-ramNoFrameskip-v0), EnvSpec(IceHockey-ramNoFrameskip-v4), EnvSpec(Jamesbond-v0), EnvSpec(Jamesbond-v4), EnvSpec(JamesbondDeterministic-v0), EnvSpec(JamesbondDeterministic-v4), EnvSpec(JamesbondNoFrameskip-v0), EnvSpec(JamesbondNoFrameskip-v4), EnvSpec(Jamesbond-ram-v0), EnvSpec(Jamesbond-ram-v4), EnvSpec(Jamesbond-ramDeterministic-v0), EnvSpec(Jamesbond-ramDeterministic-v4), EnvSpec(Jamesbond-ramNoFrameskip-v0), EnvSpec(Jamesbond-ramNoFrameskip-v4), EnvSpec(JourneyEscape-v0), EnvSpec(JourneyEscape-v4), EnvSpec(JourneyEscapeDeterministic-v0), EnvSpec(JourneyEscapeDeterministic-v4), EnvSpec(JourneyEscapeNoFrameskip-v0), EnvSpec(JourneyEscapeNoFrameskip-v4), EnvSpec(JourneyEscape-ram-v0), EnvSpec(JourneyEscape-ram-v4), EnvSpec(JourneyEscape-ramDeterministic-v0), EnvSpec(JourneyEscape-ramDeterministic-v4), EnvSpec(JourneyEscape-ramNoFrameskip-v0), EnvSpec(JourneyEscape-ramNoFrameskip-v4), EnvSpec(Kangaroo-v0), EnvSpec(Kangaroo-v4), EnvSpec(KangarooDeterministic-v0), EnvSpec(KangarooDeterministic-v4), EnvSpec(KangarooNoFrameskip-v0), EnvSpec(KangarooNoFrameskip-v4), EnvSpec(Kangaroo-ram-v0), EnvSpec(Kangaroo-ram-v4), EnvSpec(Kangaroo-ramDeterministic-v0), EnvSpec(Kangaroo-ramDeterministic-v4), EnvSpec(Kangaroo-ramNoFrameskip-v0), EnvSpec(Kangaroo-ramNoFrameskip-v4), EnvSpec(Krull-v0), EnvSpec(Krull-v4), EnvSpec(KrullDeterministic-v0), EnvSpec(KrullDeterministic-v4), EnvSpec(KrullNoFrameskip-v0), EnvSpec(KrullNoFrameskip-v4), EnvSpec(Krull-ram-v0), EnvSpec(Krull-ram-v4), EnvSpec(Krull-ramDeterministic-v0), EnvSpec(Krull-ramDeterministic-v4), EnvSpec(Krull-ramNoFrameskip-v0), EnvSpec(Krull-ramNoFrameskip-v4), EnvSpec(KungFuMaster-v0), EnvSpec(KungFuMaster-v4), EnvSpec(KungFuMasterDeterministic-v0), EnvSpec(KungFuMasterDeterministic-v4), EnvSpec(KungFuMasterNoFrameskip-v0), EnvSpec(KungFuMasterNoFrameskip-v4), EnvSpec(KungFuMaster-ram-v0), EnvSpec(KungFuMaster-ram-v4), EnvSpec(KungFuMaster-ramDeterministic-v0), EnvSpec(KungFuMaster-ramDeterministic-v4), EnvSpec(KungFuMaster-ramNoFrameskip-v0), EnvSpec(KungFuMaster-ramNoFrameskip-v4), EnvSpec(MontezumaRevenge-v0), EnvSpec(MontezumaRevenge-v4), EnvSpec(MontezumaRevengeDeterministic-v0), EnvSpec(MontezumaRevengeDeterministic-v4), EnvSpec(MontezumaRevengeNoFrameskip-v0), EnvSpec(MontezumaRevengeNoFrameskip-v4), EnvSpec(MontezumaRevenge-ram-v0), EnvSpec(MontezumaRevenge-ram-v4), EnvSpec(MontezumaRevenge-ramDeterministic-v0), EnvSpec(MontezumaRevenge-ramDeterministic-v4), EnvSpec(MontezumaRevenge-ramNoFrameskip-v0), EnvSpec(MontezumaRevenge-ramNoFrameskip-v4), EnvSpec(MsPacman-v0), EnvSpec(MsPacman-v4), EnvSpec(MsPacmanDeterministic-v0), EnvSpec(MsPacmanDeterministic-v4), EnvSpec(MsPacmanNoFrameskip-v0), EnvSpec(MsPacmanNoFrameskip-v4), EnvSpec(MsPacman-ram-v0), EnvSpec(MsPacman-ram-v4), EnvSpec(MsPacman-ramDeterministic-v0), EnvSpec(MsPacman-ramDeterministic-v4), EnvSpec(MsPacman-ramNoFrameskip-v0), EnvSpec(MsPacman-ramNoFrameskip-v4), EnvSpec(NameThisGame-v0), EnvSpec(NameThisGame-v4), EnvSpec(NameThisGameDeterministic-v0), EnvSpec(NameThisGameDeterministic-v4), EnvSpec(NameThisGameNoFrameskip-v0), EnvSpec(NameThisGameNoFrameskip-v4), EnvSpec(NameThisGame-ram-v0), EnvSpec(NameThisGame-ram-v4), EnvSpec(NameThisGame-ramDeterministic-v0), EnvSpec(NameThisGame-ramDeterministic-v4), EnvSpec(NameThisGame-ramNoFrameskip-v0), EnvSpec(NameThisGame-ramNoFrameskip-v4), EnvSpec(Phoenix-v0), EnvSpec(Phoenix-v4), EnvSpec(PhoenixDeterministic-v0), EnvSpec(PhoenixDeterministic-v4), EnvSpec(PhoenixNoFrameskip-v0), EnvSpec(PhoenixNoFrameskip-v4), EnvSpec(Phoenix-ram-v0), EnvSpec(Phoenix-ram-v4), EnvSpec(Phoenix-ramDeterministic-v0), EnvSpec(Phoenix-ramDeterministic-v4), EnvSpec(Phoenix-ramNoFrameskip-v0), EnvSpec(Phoenix-ramNoFrameskip-v4), EnvSpec(Pitfall-v0), EnvSpec(Pitfall-v4), EnvSpec(PitfallDeterministic-v0), EnvSpec(PitfallDeterministic-v4), EnvSpec(PitfallNoFrameskip-v0), EnvSpec(PitfallNoFrameskip-v4), EnvSpec(Pitfall-ram-v0), EnvSpec(Pitfall-ram-v4), EnvSpec(Pitfall-ramDeterministic-v0), EnvSpec(Pitfall-ramDeterministic-v4), EnvSpec(Pitfall-ramNoFrameskip-v0), EnvSpec(Pitfall-ramNoFrameskip-v4), EnvSpec(Pong-v0), EnvSpec(Pong-v4), EnvSpec(PongDeterministic-v0), EnvSpec(PongDeterministic-v4), EnvSpec(PongNoFrameskip-v0), EnvSpec(PongNoFrameskip-v4), EnvSpec(Pong-ram-v0), EnvSpec(Pong-ram-v4), EnvSpec(Pong-ramDeterministic-v0), EnvSpec(Pong-ramDeterministic-v4), EnvSpec(Pong-ramNoFrameskip-v0), EnvSpec(Pong-ramNoFrameskip-v4), EnvSpec(Pooyan-v0), EnvSpec(Pooyan-v4), EnvSpec(PooyanDeterministic-v0), EnvSpec(PooyanDeterministic-v4), EnvSpec(PooyanNoFrameskip-v0), EnvSpec(PooyanNoFrameskip-v4), EnvSpec(Pooyan-ram-v0), EnvSpec(Pooyan-ram-v4), EnvSpec(Pooyan-ramDeterministic-v0), EnvSpec(Pooyan-ramDeterministic-v4), EnvSpec(Pooyan-ramNoFrameskip-v0), EnvSpec(Pooyan-ramNoFrameskip-v4), EnvSpec(PrivateEye-v0), EnvSpec(PrivateEye-v4), EnvSpec(PrivateEyeDeterministic-v0), EnvSpec(PrivateEyeDeterministic-v4), EnvSpec(PrivateEyeNoFrameskip-v0), EnvSpec(PrivateEyeNoFrameskip-v4), EnvSpec(PrivateEye-ram-v0), EnvSpec(PrivateEye-ram-v4), EnvSpec(PrivateEye-ramDeterministic-v0), EnvSpec(PrivateEye-ramDeterministic-v4), EnvSpec(PrivateEye-ramNoFrameskip-v0), EnvSpec(PrivateEye-ramNoFrameskip-v4), EnvSpec(Qbert-v0), EnvSpec(Qbert-v4), EnvSpec(QbertDeterministic-v0), EnvSpec(QbertDeterministic-v4), EnvSpec(QbertNoFrameskip-v0), EnvSpec(QbertNoFrameskip-v4), EnvSpec(Qbert-ram-v0), EnvSpec(Qbert-ram-v4), EnvSpec(Qbert-ramDeterministic-v0), EnvSpec(Qbert-ramDeterministic-v4), EnvSpec(Qbert-ramNoFrameskip-v0), EnvSpec(Qbert-ramNoFrameskip-v4), EnvSpec(Riverraid-v0), EnvSpec(Riverraid-v4), EnvSpec(RiverraidDeterministic-v0), EnvSpec(RiverraidDeterministic-v4), EnvSpec(RiverraidNoFrameskip-v0), EnvSpec(RiverraidNoFrameskip-v4), EnvSpec(Riverraid-ram-v0), EnvSpec(Riverraid-ram-v4), EnvSpec(Riverraid-ramDeterministic-v0), EnvSpec(Riverraid-ramDeterministic-v4), EnvSpec(Riverraid-ramNoFrameskip-v0), EnvSpec(Riverraid-ramNoFrameskip-v4), EnvSpec(RoadRunner-v0), EnvSpec(RoadRunner-v4), EnvSpec(RoadRunnerDeterministic-v0), EnvSpec(RoadRunnerDeterministic-v4), EnvSpec(RoadRunnerNoFrameskip-v0), EnvSpec(RoadRunnerNoFrameskip-v4), EnvSpec(RoadRunner-ram-v0), EnvSpec(RoadRunner-ram-v4), EnvSpec(RoadRunner-ramDeterministic-v0), EnvSpec(RoadRunner-ramDeterministic-v4), EnvSpec(RoadRunner-ramNoFrameskip-v0), EnvSpec(RoadRunner-ramNoFrameskip-v4), EnvSpec(Robotank-v0), EnvSpec(Robotank-v4), EnvSpec(RobotankDeterministic-v0), EnvSpec(RobotankDeterministic-v4), EnvSpec(RobotankNoFrameskip-v0), EnvSpec(RobotankNoFrameskip-v4), EnvSpec(Robotank-ram-v0), EnvSpec(Robotank-ram-v4), EnvSpec(Robotank-ramDeterministic-v0), EnvSpec(Robotank-ramDeterministic-v4), EnvSpec(Robotank-ramNoFrameskip-v0), EnvSpec(Robotank-ramNoFrameskip-v4), EnvSpec(Seaquest-v0), EnvSpec(Seaquest-v4), EnvSpec(SeaquestDeterministic-v0), EnvSpec(SeaquestDeterministic-v4), EnvSpec(SeaquestNoFrameskip-v0), EnvSpec(SeaquestNoFrameskip-v4), EnvSpec(Seaquest-ram-v0), EnvSpec(Seaquest-ram-v4), EnvSpec(Seaquest-ramDeterministic-v0), EnvSpec(Seaquest-ramDeterministic-v4), EnvSpec(Seaquest-ramNoFrameskip-v0), EnvSpec(Seaquest-ramNoFrameskip-v4), EnvSpec(Skiing-v0), EnvSpec(Skiing-v4), EnvSpec(SkiingDeterministic-v0), EnvSpec(SkiingDeterministic-v4), EnvSpec(SkiingNoFrameskip-v0), EnvSpec(SkiingNoFrameskip-v4), EnvSpec(Skiing-ram-v0), EnvSpec(Skiing-ram-v4), EnvSpec(Skiing-ramDeterministic-v0), EnvSpec(Skiing-ramDeterministic-v4), EnvSpec(Skiing-ramNoFrameskip-v0), EnvSpec(Skiing-ramNoFrameskip-v4), EnvSpec(Solaris-v0), EnvSpec(Solaris-v4), EnvSpec(SolarisDeterministic-v0), EnvSpec(SolarisDeterministic-v4), EnvSpec(SolarisNoFrameskip-v0), EnvSpec(SolarisNoFrameskip-v4), EnvSpec(Solaris-ram-v0), EnvSpec(Solaris-ram-v4), EnvSpec(Solaris-ramDeterministic-v0), EnvSpec(Solaris-ramDeterministic-v4), EnvSpec(Solaris-ramNoFrameskip-v0), EnvSpec(Solaris-ramNoFrameskip-v4), EnvSpec(SpaceInvaders-v0), EnvSpec(SpaceInvaders-v4), EnvSpec(SpaceInvadersDeterministic-v0), EnvSpec(SpaceInvadersDeterministic-v4), EnvSpec(SpaceInvadersNoFrameskip-v0), EnvSpec(SpaceInvadersNoFrameskip-v4), EnvSpec(SpaceInvaders-ram-v0), EnvSpec(SpaceInvaders-ram-v4), EnvSpec(SpaceInvaders-ramDeterministic-v0), EnvSpec(SpaceInvaders-ramDeterministic-v4), EnvSpec(SpaceInvaders-ramNoFrameskip-v0), EnvSpec(SpaceInvaders-ramNoFrameskip-v4), EnvSpec(StarGunner-v0), EnvSpec(StarGunner-v4), EnvSpec(StarGunnerDeterministic-v0), EnvSpec(StarGunnerDeterministic-v4), EnvSpec(StarGunnerNoFrameskip-v0), EnvSpec(StarGunnerNoFrameskip-v4), EnvSpec(StarGunner-ram-v0), EnvSpec(StarGunner-ram-v4), EnvSpec(StarGunner-ramDeterministic-v0), EnvSpec(StarGunner-ramDeterministic-v4), EnvSpec(StarGunner-ramNoFrameskip-v0), EnvSpec(StarGunner-ramNoFrameskip-v4), EnvSpec(Tennis-v0), EnvSpec(Tennis-v4), EnvSpec(TennisDeterministic-v0), EnvSpec(TennisDeterministic-v4), EnvSpec(TennisNoFrameskip-v0), EnvSpec(TennisNoFrameskip-v4), EnvSpec(Tennis-ram-v0), EnvSpec(Tennis-ram-v4), EnvSpec(Tennis-ramDeterministic-v0), EnvSpec(Tennis-ramDeterministic-v4), EnvSpec(Tennis-ramNoFrameskip-v0), EnvSpec(Tennis-ramNoFrameskip-v4), EnvSpec(TimePilot-v0), EnvSpec(TimePilot-v4), EnvSpec(TimePilotDeterministic-v0), EnvSpec(TimePilotDeterministic-v4), EnvSpec(TimePilotNoFrameskip-v0), EnvSpec(TimePilotNoFrameskip-v4), EnvSpec(TimePilot-ram-v0), EnvSpec(TimePilot-ram-v4), EnvSpec(TimePilot-ramDeterministic-v0), EnvSpec(TimePilot-ramDeterministic-v4), EnvSpec(TimePilot-ramNoFrameskip-v0), EnvSpec(TimePilot-ramNoFrameskip-v4), EnvSpec(Tutankham-v0), EnvSpec(Tutankham-v4), EnvSpec(TutankhamDeterministic-v0), EnvSpec(TutankhamDeterministic-v4), EnvSpec(TutankhamNoFrameskip-v0), EnvSpec(TutankhamNoFrameskip-v4), EnvSpec(Tutankham-ram-v0), EnvSpec(Tutankham-ram-v4), EnvSpec(Tutankham-ramDeterministic-v0), EnvSpec(Tutankham-ramDeterministic-v4), EnvSpec(Tutankham-ramNoFrameskip-v0), EnvSpec(Tutankham-ramNoFrameskip-v4), EnvSpec(UpNDown-v0), EnvSpec(UpNDown-v4), EnvSpec(UpNDownDeterministic-v0), EnvSpec(UpNDownDeterministic-v4), EnvSpec(UpNDownNoFrameskip-v0), EnvSpec(UpNDownNoFrameskip-v4), EnvSpec(UpNDown-ram-v0), EnvSpec(UpNDown-ram-v4), EnvSpec(UpNDown-ramDeterministic-v0), EnvSpec(UpNDown-ramDeterministic-v4), EnvSpec(UpNDown-ramNoFrameskip-v0), EnvSpec(UpNDown-ramNoFrameskip-v4), EnvSpec(Venture-v0), EnvSpec(Venture-v4), EnvSpec(VentureDeterministic-v0), EnvSpec(VentureDeterministic-v4), EnvSpec(VentureNoFrameskip-v0), EnvSpec(VentureNoFrameskip-v4), EnvSpec(Venture-ram-v0), EnvSpec(Venture-ram-v4), EnvSpec(Venture-ramDeterministic-v0), EnvSpec(Venture-ramDeterministic-v4), EnvSpec(Venture-ramNoFrameskip-v0), EnvSpec(Venture-ramNoFrameskip-v4), EnvSpec(VideoPinball-v0), EnvSpec(VideoPinball-v4), EnvSpec(VideoPinballDeterministic-v0), EnvSpec(VideoPinballDeterministic-v4), EnvSpec(VideoPinballNoFrameskip-v0), EnvSpec(VideoPinballNoFrameskip-v4), EnvSpec(VideoPinball-ram-v0), EnvSpec(VideoPinball-ram-v4), EnvSpec(VideoPinball-ramDeterministic-v0), EnvSpec(VideoPinball-ramDeterministic-v4), EnvSpec(VideoPinball-ramNoFrameskip-v0), EnvSpec(VideoPinball-ramNoFrameskip-v4), EnvSpec(WizardOfWor-v0), EnvSpec(WizardOfWor-v4), EnvSpec(WizardOfWorDeterministic-v0), EnvSpec(WizardOfWorDeterministic-v4), EnvSpec(WizardOfWorNoFrameskip-v0), EnvSpec(WizardOfWorNoFrameskip-v4), EnvSpec(WizardOfWor-ram-v0), EnvSpec(WizardOfWor-ram-v4), EnvSpec(WizardOfWor-ramDeterministic-v0), EnvSpec(WizardOfWor-ramDeterministic-v4), EnvSpec(WizardOfWor-ramNoFrameskip-v0), EnvSpec(WizardOfWor-ramNoFrameskip-v4), EnvSpec(YarsRevenge-v0), EnvSpec(YarsRevenge-v4), EnvSpec(YarsRevengeDeterministic-v0), EnvSpec(YarsRevengeDeterministic-v4), EnvSpec(YarsRevengeNoFrameskip-v0), EnvSpec(YarsRevengeNoFrameskip-v4), EnvSpec(YarsRevenge-ram-v0), EnvSpec(YarsRevenge-ram-v4), EnvSpec(YarsRevenge-ramDeterministic-v0), EnvSpec(YarsRevenge-ramDeterministic-v4), EnvSpec(YarsRevenge-ramNoFrameskip-v0), EnvSpec(YarsRevenge-ramNoFrameskip-v4), EnvSpec(Zaxxon-v0), EnvSpec(Zaxxon-v4), EnvSpec(ZaxxonDeterministic-v0), EnvSpec(ZaxxonDeterministic-v4), EnvSpec(ZaxxonNoFrameskip-v0), EnvSpec(ZaxxonNoFrameskip-v4), EnvSpec(Zaxxon-ram-v0), EnvSpec(Zaxxon-ram-v4), EnvSpec(Zaxxon-ramDeterministic-v0), EnvSpec(Zaxxon-ramDeterministic-v4), EnvSpec(Zaxxon-ramNoFrameskip-v0), EnvSpec(Zaxxon-ramNoFrameskip-v4), EnvSpec(CubeCrash-v0), EnvSpec(CubeCrashSparse-v0), EnvSpec(CubeCrashScreenBecomesBlack-v0), EnvSpec(MemorizeDigits-v0)])

Le [Cart-Pole](https://gym.openai.com/envs/CartPole-v1/) est un environnement très simple dans lequel un chariot peut bouger soit vers la gauche, soit vers la droite. Un bâton/piquet est placé sur ce dernier. L'agent doit alors bouger le chariot à gauche ou à droite pour que le bâton reste droit.

L'environnement est défini par la commande `make()`:

```python
env = gym.make('CartPole-v1')
```

Il faut ensuite initialiser l'environnement en appelant la méthode `reset()` qui retourne la première observation.

```python
env.seed(seed)
obs = env.reset()
obs
```

    array([-0.01258566, -0.00156614,  0.04207708, -0.00180545])

Les observations dépendent de l'environnement considéré. Dans ce cas, les observations sont regroupées dans un array Numpy 1D soit 1 vecteur composé de 4 floats. Ce vecteur regroupe:
- la position horizontale du chariot (<0 gauche; 0 = verticale; >0 droite)
- la vitesse du chariot
- l'angle du bâton (<0 gauche; 0 = verticale; >0 droite)
- la vitesse angulaire

Un environnement peut-être visualisé par appel à la méthode `render()`. Pour cet environnement deux modes de rendering exitents:
- si on veut récupérer les observations (sous la forme d'image), `img = env.render(mode="rgb_array")`
- ou si on veut afficher l'environnement  `env.render()`

```python
env.render()
```

    True

Pour récupérer une observation (une image) de l'environnement sous la forme d'un array NumPy:

```python
img = env.render(mode="rgb_array")
img.shape
```

    (400, 600, 3)

La fonction `plot_environment()` permet de récupérer et d'afficher l'observation courante de l'environnement.

```python
def plot_environment(env, figsize=(5,4)):
    plt.figure(figsize=figsize)
    img = env.render(mode="rgb_array")
    plt.imshow(img)
    plt.axis("off")
    return img
```

```python
plot_environment(env)
plt.show()
```

Pour interagir avec l'environnement, l'agent doit sélectionner une action à partir de l'espace des actions possibles. Pour chaque environnement, la commande `action_space` permet de connaitre cet espace.

```python
env.action_space
```

    Discrete(2)

Pour le cartpole,deux actions sont possible: gauche ou droite.

Dans l'état actuel, le bâton est penché vers la droite (`obs[2] > 0`); passons l'action de déplacer le chariot vers la droite.

```python
# action
action = 1  # right

# interaction with the environment
obs, reward, done, info = env.step(action)

# observation
obs
```

    array([-0.01261699,  0.19292789,  0.04204097, -0.28092127])

Remarquez que:
- le chariot se déplace maintenant vers la droite (`obs[1] > 0`);
- le bâton est toujours incliné vers la droite (`obs[2] > 0`) ;
- mais sa vitesse angulaire est maintenant négative (`obs[3] < 0`), il est ainsi vraissemblable qu'il soit incliné vers la gauche après la prochaine action.

```python
plot_environment(env);
```

L'interaction avec l'environnement:
- génère l'observation suivante obs
- renseigne l'agent sur la récompense générée par l'action prise: reward
- retourne `done=True` si le jeu est fini (pour cet environnement, le jeu est fini si l'angle du baton est de plus de 12° ou si le chariot s'est déplacé de plus 2.4 unités par rapport au centre.)
- info est un dictionnaire spécifique à chaque environnement qui peut contenir des informations supplémentaires pour le debugging ou l'apprentissage. Pour cet environnement, info est vide.

```python
reward
```

    1.0

```python
done
```

    False

```python
info
```

    {}

Comme nous l'avons vu en cours, un **épisode** correspond à la séquence d'interactions entre le moment où l'environnement est lancé `reset()` et le moment ou `done=True`. Pour relancer un épisode, il faut refaire appel à la méthode `reset()`.

```python
if done:
    obs = env.reset()
```

**Charger consécutivement les environnements suivants:**
- **SpaceInvaders-v0**
- **CarRacing-v0**

**Déterminer pour chaque environnement:**
- **l'espace d'actions**
- **l'espace d'observation**

**et créer une image/copie de chaque environnement**

```python
####################
# YOUR CODE HERE: SpaceInvaders
env1 = gym.make('SpaceInvaders-v0')

img = plot_environment(env1)
plt.show()

print('action space: {}'.format(env1.action_space))
print('observation space: {}'.format(img.shape))
env1.close()
####################

```

```python
####################
# YOUR CODE HERE: CarRacing-v0

# pip install gym[Box2D]
env2 = gym.make('CarRacing-v0')
obs = env2.reset() # attention apparemment tant que le reset n'est pas fait aucune donnée
#img = env2.render(mode="rgb_array") #["human", "rgb_array", "state_pixels"]
img = plot_environment(env2)
#plt.show()
print('action space: {}'.format(env2.action_space))
print('observation space: {}'.format(img.shape))
env2.close()
####################

```

# 3. Définition d'une politique simple (policy)

Now how can we make the poll remain upright? We will need to define a _policy_ for that. This is the strategy that the agent will use to select an action at each step. It can use all the past actions and observations to decide what to do.

Revenons au **CartPole** et essayons de le maintenir droit. Nous avons besoin d'une policy (politique) pour cela. Il s'agit d'une stratégie que l'agent utilisera pour sélectionner une action à chaque étape. Cette stratégie pourra utiliser potentiellement utiliser toutes les actions et observations passées pour faire cela.

Pour le moment, nous allons définir une politique très simple consistant à déplacer le chariot vers la gauche (resp. la droite) quand il penche vers la gauche (resp. la droite). Ici, la policy $\pi_{\theta}(\mathbf{a}_t|\mathbf{o}_t)$ est déterministe pourra donc s'écrire $\mathbf{a}_t = \pi(\mathbf{o}_t)$.

**Question:**
Pour l'implémentation vous créerez:
- une fonction `simple_policy(obs)` retournant l'action à prendre en fonction de l'observation
- un script qui permet de
 - générer `episode_nb` (500) épisodes (ou échantillons, trajectoires, roll_out)
 - chacun de ces épisodes comprend un nombre max d'interaction avec l'environnement fixé à `interaction_max_nb = 200`
 - pour chaque interaction, décider de l'action à prendre (appel à `simple_policy()`), jouer cette action en interagissant avec l'environnement et récupérer la reward et calculer la reward totale
 - calculer les performances de l'algorithme (les statistiques: moyenne, std, min, max)

**Notes**: deux boucles imbriquées sont à coder, vous pouvez suivre le squellette suivant

```python
# fixer la graine aléatoire
env.seed(seed)

# policy definition
def simple_policy(obs):

    ####################
    # YOUR CODE HERE

    ####################
    return action

# parameters
episode_nb=500
interaction_max_nb=200
episode_rewards_history=[]

####################
# YOUR CODE HERE

# loop over the episodes
for ...

    # starting a new episode

    # ...with interaction_max_nb interactions
    for step in range(interaction_max_nb):

        # starting a new interaction

        # select action

        # interaction

        # cumulate reward

        # continue or not
        if done:
            break

    # performance statistics
    episode_rewards_history.append(episode_rewards)

####################

```

```python
# Display performance statistics
np.mean(episode_rewards_history), np.std(episode_rewards_history), np.min(episode_rewards_history), np.max(episode_rewards_history)

```

**Questions**: Quelles sont vos conclusions sur les performances obtenues?

**Your answer here**

Vous pouvez visualiser l'évolution du cartpole au cours d'un episode avec la boucle suivante.
**Question**:
Complétez les lignes demandées.

```python
# on fixe la graine aléatoire
env.seed(seed)

# initialisation
frames = []
obs = env.reset()

# boucle d'interactions
for step in range(interaction_max_nb):

    # recuperation de l'environnement courant
    img = env.render(mode="rgb_array")
    frames.append(img)

    # select action
    ####################
    # YOUR CODE HERE

    ####################

    # interaction
    ####################
    # YOUR CODE HERE

    ####################

    # continue or not
    if done:
        print('break at {}'.format(step))
        print('obs: {}'.format(obs))
        print('reward: {}'.format(reward))
        break

```

Vous pouvez définir un lecteur d'animation.

```python
def update_scene(num, frames, patch):
    patch.set_data(frames[num])
    return patch,

def plot_animation(frames, repeat=False, interval=40):
    fig = plt.figure()
    patch = plt.imshow(frames[0])
    plt.axis('off')
    anim = animation.FuncAnimation(
        fig, update_scene, fargs=(frames, patch),
        frames=len(frames), repeat=repeat, interval=interval)
    plt.close()
    return anim
```

```python
plot_animation(frames)
```

# 4. Neural Network Policies

Nous allons la remplacer la policy avec une stratégie trop simple en remplacant la fonction par un réseau de neurones permettant plus de liberté.

**Question**: Commencer par créer un model séquentiel de réseau de neurones à deux couches dense (la couche cachée sera de 4 neurones) avec keras:
- Ce réseau prendra une observation en entrée
- Ce réseau prédira l'action à prendre pour chaque observation. Pour choisir l'action, le réseau estimera la probabilité pour chaque action.
- Quel est le nombre de neurones de la couche de sortie? Quelle sera la fonction d'activation de la dernière couche? Pourquoi?

**Notes sur observations et états**:
Dans cet environnement particulier, les actions passées et les observations peuvent être ignorées car chaque observation contient l'état complet de l'environnement. Si par exemple, l'observation ne contenait pas la vitesse du chariot, on aurait besoin de quelques observations successives de la position du chariot pour l'estimer et répondre au problème. Si la qualité des observations était très mauvaises par exemple dans le cas de données bruitées, il faudrait probablement mettre dans les observations, quelques observations successives.

**Réponse**:
- nb de neurones 1 et fonction d'activation sera sigmoid

```python
keras.backend.clear_session()
tf.random.set_seed(seed)
np.random.seed(seed)

####################
# YOUR CODE HERE
model=...
####################

```

**Question:**
Complétez la fonction suivante vise le rendu d'un épisode et sort les frames pour afficher l'animation.

**Notes sur le compromis entre exploration et action**
Vous devez définir sous le commentaire `# exploration or action strategy` une stratégie entre choisir une action la plus probable et exploration d'une autre action. Cette approche laisse l'agent trouver le bon équilibre entre explorer de nouvelles actions et exploiter les actions qui sont connus pour bien fonctionner. Imaginer que vous alliez dans un restaurant pour la première fois et que tous les plats vous semblent de la même manière appétissants. Vous en choisissez un au hasard. S'il vous satisfait, vous allez augmenter la probabilité de le choisir à votre prochain passage dans ce restaurant. Si vous fixez cette probabilité à 1, vous n'essaierez jamais les autres plats...

Ici une stratégie simple sera:
- réaliser un tirage aléatoire uniforme d'un réell compris entre 0 et 1
- vérifier si ce tirage est supérieur à la probabilité de déplacer le chariot vers la gauche
- l'`action` sera `False` avec la probabilité `left_proba` ou `True` avec la probabilité `1 - left_proba`
- il suffit alors de "caster" ce booléen en nombre.

```python
def render_policy_net(model, interaction_max_nb=200, seed=42):

    # on fixe la graine aléatoire
    env = gym.make("CartPole-v1")
    env.seed(seed)
    np.random.seed(seed)

    # initialisations
    frames = []
    obs = env.reset()

    # boucle d'interactions
    for step in range(interaction_max_nb):

        ####################
        # YOUR CODE HERE

        # recuperation de l'environnement courant

        # select action

        # exploration or action strategy

        # interaction

        ####################

        # continue or not
        if done:
            break

    # output
    env.close()
    return frames

```

On peut alors voir le déroulement d'un épisode.

```python
frames = render_policy_net(model)
plot_animation(frames)
```

A ce stade, les poids du réseau de neurones sont aléatoires. Aucun apprentissage n'a eu lieu.
L'algorithme REINFORCE de type policy gradient nous permettra de faire cet apprentissage.

# 5. Policy Gradients

Pour entraîner ce réseau de neurones, nous devrons définir les probabilités cibles `y`. Si une action est bonne, nous devons augmenter sa probabilité, et inversement si elle est mauvaise, nous devons la réduire. Mais comment savoir si une action est bonne ou mauvaise ? Le problème est que la plupart des actions ont des effets différés, donc lorsque vous gagnez ou perdez des points dans un épisode, il n'est pas clair quelles actions ont contribué à ce résultat : était-ce seulement la dernière action ? Ou les dix dernières ? Ou une seule action 50 étapes plus tôt ? C'est ce qu'on appelle le _problème d'attribution de crédit_.

L'algorithme _Policy Gradients_ s'attaque à ce problème en jouant d'abord plusieurs épisodes, puis en rendant les actions des bons épisodes légèrement plus probables, tandis que les actions des mauvais épisodes sont rendues légèrement moins probables. On joue d'abord, puis on revient en arrière et on réfléchit à ce qu'on a fait.

Pour mémoire, la fonction de coût pour les algorithmes PG est :
	\begin{align*}
		J(\theta) &= E_{\tau \sim p_\theta(\tau)} \left[\sum\limits_{t}^T r(\mathbf{s}_{t},\mathbf{a}_{t}) \right] \approx \frac{1}{N} \sum\limits_{i=1}^N  \sum\limits_{t} r(\mathbf{s}_{i,t},\mathbf{a}_{i,t})
%		\nabla_{\theta}J(\theta) &= E_{\tau \sim p_\theta(\tau)} \left[ \left(\sum\limits_{t}^T r(\mathbf{s}_{t},\mathbf{a}_{t}) \right) \cdot \left(  \sum\limits_{t=1}^{T} \nabla_{\theta} \log \pi_\theta(\mathbf{a}_t|\mathbf{s}_t) \right) \right]
	\end{align*}

Elle peut être estimée à partir de $N$ trajectoires
	\begin{align*}
		J(\theta) &\approx \frac{1}{N} \sum\limits_{i=1}^N \left[ \left(\sum\limits_{t=1}^T \nabla_{\theta} \log \pi_\theta(\mathbf{a}_{i,t}|\mathbf{s}_{i,t})\right) \left( \sum\limits_{t=1}^T r(\mathbf{s}_{i,t},\mathbf{a}_{i,t}) \right) \right]\\
	\end{align*}

L'algorithme d'apprentissage REINFORCE est alors:
- Sample $ \{ \tau^i\}_{i=1,\dots,N} $ from $\pi_{\theta}(\mathbf{a}_t|\mathbf{s}_t)$
- Evaluate the gradient:
\begin{align}
		\nabla_{\theta}J(\theta) &\approx \frac{1}{N} \sum\limits_{i=1}^N \left[ \left(\sum\limits_{t=1}^T \nabla_{\theta} \log \pi_\theta(\mathbf{a}_{i,t}|\mathbf{s}_{i,t})\right) \left( \sum\limits_{t=1}^T r(\mathbf{s}_{i,t},\mathbf{a}_{i,t}) \right) \right]
		\end{align}
- Update the parameters $ \theta \leftarrow \theta + \eta \nabla_{\theta} J(\theta) $

Comme mentionné en cours, nous allons bénéficier de l'algorithme de calculs automatiques du gradient offert par tensorflow. L'algorithme consiste alors à réaliser:
- générer une interaction en calculant la loss et ses gradients et gardant la reward
- générer un épisode composé d'un ensemble d'interactions
- itérer pour appliquer les gradients à la policy par `gradient ascent`

**Question**:
La loss peut ici être estimée par la fonction `tf.keras.losses.binary_crossentropy` pourquoi?

**Your Answer here:**

Commençons par créer une fonction `one_interaction` pour réaliser une seul interaction:
 - en utilisant le modèle de réseau de neurones
 - en calculant la loss et ses gradients (nous allons simplement enregistrer ces gradients pour l'instant, et les modifier plus tard en fonction de la qualité ou de la mauvaise de l'action)

Notes:
- pour calculer un gradient avec Tensorflow (sans Keras), il faut calculer la `loss` dans le contexte `tf.GradientTape()`. Les gradients sont ainsi calculés par la méthode `gradient` (voir ci-dessous).
- `y_target` sera 1 si l'action est d'aller à gauche (=0) et 0 dans le cas contraire.
- `tf.reduce_mean(loss_fn(...)` permettra d'estimer la loss

```python
def one_interaction(env, obs, model, loss_fn):

    # compute gradients
    with tf.GradientTape() as tape:

        ####################
        # YOUR CODE HERE
        # forward pass of the model outputs the probability of going left
        #(attention le model attend un batch d'une seule observation)

        # exploration or action strategy

        # compute losses

        ####################

    # gradient de la loss en fonction des paramètres entrainables du model
    grads = tape.gradient(loss, model.trainable_variables)

    # interaction

    # sorties
    return obs, reward, done, grads
```

```python
# test de la fonction
loss_fn = tf.keras.losses.binary_crossentropy
obs, reward, done, grads = one_interaction(env, obs, model, loss_fn)
obs
```

**Question**: Maintenant, créez une autre fonction qui s'appuiera sur la fonction `one_interaction()` pour jouer plusieurs épisodes, en retournant toutes les récompenses et les gradients, pour chaque épisode et chaque étape.

```python
def generate_multiple_episodes(env, episode_nb, interaction_max_nb, model, loss_fn):

    ####################
    # YOUR CODE HERE

    # initialisation

    # episode loop
    for episode ...

        # initialisation

        # interaction loop
        for interaction ...:

            # interaction

            # update current rewards and grads

            # continue episode or not
            if done:
                break

        # update rewards and grads histories

    ####################

    # outputs
    return rewards_hist, grads_hist
```

```python
# test de la fonction
loss_fn = tf.keras.losses.binary_crossentropy
episode_nb = 3
rewards_hist, grads_hist = generate_multiple_episodes(env, episode_nb, interaction_max_nb, model, loss_fn)
print('rewards:', rewards_hist[0])

print('grads:', grads_hist[0])
```

L'algorithme PG utilise le modèle pour jouer l'épisode plusieurs fois (par exemple 10 fois), puis il revient en arrière et examine toutes les récompenses pour calculer:
- la "reward-to-go" avec un "discount" facteur:
\begin{align*}
\nabla_{\theta}J(\theta) &\approx \frac{1}{N} \sum\limits_{i=1}^N \left[ \sum\limits_{t=1}^{T} \left( \nabla_{\theta} \log \pi_\theta(\mathbf{a}_{i,t}|\mathbf{s}_{i,t}) \cdot  \sum\limits_{t'=t}^T \gamma^{t'-t} r(\mathbf{s}_{i,t'},\mathbf{a}_{i,t'}) \right) \right]
\end{align*}
- la "reward-to-go" avec une baseline moyenne.
 \begin{align*}
\nabla_{\theta}J(\theta) &\approx \frac{1}{N} \sum\limits_{i=1}^N \left[ \sum\limits_{t=1}^{T} \left( \nabla_{\theta} \log \pi_\theta(\mathbf{a}_{i,t}|\mathbf{s}_{i,t}) \cdot  \left(\sum\limits_{t'=t}^T  \gamma^{t'-t} r(\mathbf{s}_{i,t'},\mathbf{a}_{i,t'}) - b\right) \right)\right]\\
b &= \frac{1}{N}\sum\limits_{i=1}^{N} R(\tau)
\end{align*}

- la "reward-to-go" avec une baseline moyenne et normalisation par un ecart type.
 \begin{align*}
\nabla_{\theta}J(\theta) &\approx \frac{1}{N} \sum\limits_{i=1}^N \left[ \sum\limits_{t=1}^{T} \left( \nabla_{\theta} \log \pi_\theta(\mathbf{a}_{i,t}|\mathbf{s}_{i,t}) \cdot  \frac{\left(\sum\limits_{t'=t}^T \gamma^{t'-t}  r(\mathbf{s}_{i,t'},\mathbf{a}_{i,t'})\right) - b }{s}\right) \right]\\
b &= \frac{1}{N}\sum\limits_{i=1}^{N} R(\tau) \;\;\;\;\; s = std
\end{align*}

Ces différentes fonctions "reward-to-go" servent à réduire la variance de l'estimateur du gradient connu pour être très bruité. Leur expression sont respectivement

**Question**: Créez quatre fonctions permettant à partir d'une liste de rewards de sortir les trois types de reward:
- la première `compute_discounted_rewards` calculera les récompenses réduites (entrées: liste de rewards et discount_rate)
- la seconde `episode_discounted_rewards` calculera les récompenses réduites moins la moyenne sur de nombreux épisodes (entrées: liste de listes de rewards et discount_rate).
- la troisième `episode_discounted_rewards_mean` calculera les récompenses réduites moins la moyenne sur de nombreux épisodes.(entrées: liste de listes de rewards et discount_rate).
- la quatière `episode_discounted_rewards_mean_std` normalisera les récompenses réduites sur de nombreux épisodes.(entrées: liste de listes de rewards et discount_rate).

```python
def compute_discounted_rewards(rewards, discount_rate):
    ####################
    # YOUR CODE HERE

    ####################

    return discounted

def episode_discounted_rewards(all_rewards, discount_rate):
    ####################
    # YOUR CODE HERE

    ####################
    return all_discounted_rewards

def episode_discounted_rewards_mean(all_rewards, discount_rate):
    ####################
    # YOUR CODE HERE

    ####################
    return all_discounted_rewards

def episode_discounted_rewards_mean_std(all_rewards, discount_rate):
    ####################
    # YOUR CODE HERE

    ####################
    return all_discounted_rewards

```

```python
#Test de la fonction (on doit obtenir array([-22, -40, -50]))
compute_discounted_rewards([10, 0, -50], discount_rate=0.8)
```

```python
#Test de la fonction (on doit obtenir [array([-22., -40., -50.]), array([26., 20.])])
episode_discounted_rewards([[10, 0, -50], [10, 20]], discount_rate=0.8)
```

```python
#Test de la fonction (on doit obtenir [array([ -8.8, -26.8, -36.8]), array([39.2, 33.2])])
episode_discounted_rewards_mean([[10, 0, -50], [10, 20]], discount_rate=0.8)
```

```python
#Test de la fonction (on doit obtenir [array([-0.28435071, -0.86597718, -1.18910299]),array([1.26665318, 1.0727777 ])]
episode_discounted_rewards_mean_std([[10, 0, -50], [10, 20]], discount_rate=0.8)
```

Finalement, on a tous les blocs pour l'algorithme final que vous compléterez.

```python
# parametres
n_iterations = 150
episode_nb_per_update = 10
interaction_max_nb = 200
discount_rate = 0.95
```

```python
# optimiseur
optimizer = keras.optimizers.Adam(lr=0.01)

# loss fonction
loss_fn = keras.losses.binary_crossentropy
```

```python
# model
keras.backend.clear_session()
np.random.seed(seed)
tf.random.set_seed(seed)

####################
# YOUR CODE HERE
model = ...
####################

```

```python
# initialisation
env = gym.make("CartPole-v1")
env.seed(seed);

# gradient descent loop
for iteration in range(n_iterations):

    ####################
    # YOUR CODE HERE

    # generate samples for on policy algorithm through multiple episodes

    # performances
    total_rewards = sum(map(sum, all_rewards))
    print("\rIteration: {}, mean rewards: {:.1f}".format(
        iteration, total_rewards / episode_nb_per_update), end="")

    # compute discounted reward rewards

    # compute gradients weighted by discounted reward-to-go
    all_mean_grads = []
    for var_index in range(len(model.trainable_variables)):
        mean_grads = tf.reduce_mean(
            [final_reward * all_grads[episode_index][step][var_index]
             for episode_index, final_rewards in enumerate(all_final_rewards)
                 for step, final_reward in enumerate(final_rewards)], axis=0)
        all_mean_grads.append(mean_grads)

    # apply gradients to policy
    optimizer.apply_gradients(zip(all_mean_grads, model.trainable_variables))

env.close()
```

```python
frames = render_policy_net(model)
plot_animation(frames)
```

**Question**: Testez les trois différentes fonctions *reward-to-go* et comparez leurs performances.

## 6. (Optionnel) Testez l'algorithme en l'adaptant pour un autre environnement de votre choix

```python

```


# Let's start!
OPEN the jupyter notebook **Tutorial1-Part1** downloaded from the **indico timetable: https://indico.cern.ch/event/1088622/timetable/#20220111** to work locally or from the following link: **https://github.com/fusterma/JUAS2022** to work online.

# Tutorials summary

The goal of these workshops is to do numerical exercices using MAD-X to visualize transverse dynamics concepts from a different point of view.

**Friday 14th of January**
- **Tutorial 1 - Part 1**: Introduction to the tools, small numerical exercises (all together).
- **Tutorial 1 - Part 2**: My first circular accelerator: FODO cell – optics and first matching (groups of 3/4 students).
- **Tutorial 1 - Part 3**: Adding dipoles to the FODO cell – MAD-X matching block (groups of 3/4 students).

$\color{red}{\text{WEEKEND: homework exercice}}$

**Monday 17th of January**
- **Tutorial 2 - Part 1**: Natural chromaticity – MAD-X tracking module (groups of 3/4 students).
- **Tutorial 2 - Part 2**: Chromaticity correction – impact of non-linearities (groups of 3/4 students).
- **Tutorial 2 - Part 3**: Design of a transfer line  - optics and matching (groups of 3/4 students).

$\color{red}{\text{Homework + Tutorial 1 and Tutorial 2 jupyter-notebooks (to be delivered as late on Wednesday 19th to nuria.fuster@ific.uv.es)}}$. This will be considered as a BONUS to pass the accelerator design workshop oral exam.

$\color{red}{\text{VERY IMPORTANT}}$: Save your jupyter-notebooks and download them to your computer after finishing the tutorials!! Otherwise your work will be lost!

- The tutorials solutions will be uploaded on the indico timetable on Wednesday 19th.

$\color{blue}{\text{Notes}}$:

- For most of the tutorials we will split in groups of 3/4 students. These groups will be kept also for the accelerator design workshops on the third and fourth weeks of the JUAS course.

- The timing of the tutorials (except Tutorial 1 - Part 1) will be:
    - 5 minutes for the introduction to the problem.
    - 25 minutes to work within your team and with your tutor on the problem.
    - 15 minutes for going through the solutions and discussion.

- Tutors (Axel, Guido, Tessa, Davide and Nuria) will go around to help you and answer questions!

- You can work as a team in the groups or by yourself but when you need help please ask your team mates or the tutor and use the screen share option.

- The problems are long with many questions, we don't expect you to solve all of them. Some BONUS questions are there for discussion or homework.

- We encourage you to use the Slack MAD-X channel during the workshops for the discussion part, during the weekend and all JUAS to post questions.

# Tutorial 1: Part 1

Objectives:

- [Get familiar with the jupyter-notebooks.](#introjupyter)

- [Get familiar with the basic python commands that we will use during the tutorials.](#intropython)

- [How do we compute the optics of a lattice?](#firstexercice)

- [Get familiar with python commands to send the information to the MAD-X code and review the main MAD-X blocks for optics calculations.](#intromadx)

# Jupyter notebook

- **OPEN** a jupyter notebook go to **FILE -> OPEN**.

- **EDIT/INSERT/DELATE** a cell.

- **RUN** (press bottom on the top command line or press CAPS+ENTER).

- If working online **SAVE and DOWNLOAD**: after we finish one tutorial you need to SAVE and DOWNLOAD the jupyter-notebook into your PC. Otherwise your progres will be lost!

- **SAVE TO BROWSER STORAGE** using the "cloud" icon (for those working on BINDER).

# Basic python commands

The python universe has a huge number of libraries that extend the capabilities of python.
Nearly all of these are open source. For this workshop we will use the following:

```python
############################
# Import special libraries #
############################
#For plotting
from matplotlib import pyplot as plt
# For numerical calulations (np.max(), np.min(), np.mean()...)
import numpy as np
# For symbolic computation (solving algebra problems)
import sympy as sp
# For structuring the data, visualization of tables and data manipultion
import pandas as pd
# Library that allows us to use the MAD-X models
from cpymad.madx import Madx
# Plot display
%matplotlib notebook
```

If you want to learn more about **python**:

https://www.youtube.com/watch?v=kqtD5dpn9C8

https://www.kaggle.com/learn/python

More about the **cpymad** library: http://hibtc.github.io/cpymad/getting-started

# Scalars, arrays and matrices in Python

```python
# Scalar
a=20
b=30
```

```python
# Arrays and matrices
sp.Matrix([1,2,3,4]) # 1D array
```

$\displaystyle \left[\begin{matrix}1\\2\\3\\4\end{matrix}\right]$

```python
sp.Matrix([[1,2],[3,4]]) # 2x2 matrix
```

$\displaystyle \left[\begin{matrix}1 & 2\\3 & 4\end{matrix}\right]$

```python
A=sp.Matrix([[1,2],[3,4]])
B=sp.Matrix([[1,2],[3,4]])
A+B
```

$\displaystyle \left[\begin{matrix}2 & 4\\6 & 8\end{matrix}\right]$

```python
A*B
```

$\displaystyle \left[\begin{matrix}7 & 10\\15 & 22\end{matrix}\right]$

# Plots in Python

```python
# Plot
%matplotlib notebook
x=[0,1,2,3,4,5,6,7,8,9,10]
y=[0,1,2,3,4,5,6,7,8,9,10]
plt.plot(x,y,'.-b',label='test')
plt.legend()
plt.grid()
plt.xlabel('s [m]')
plt.ylabel('x[m]')
```

    <IPython.core.display.Javascript object>

    Text(0, 0.5, 'x[m]')

# How do we compute the optics of a lattice?

- Here we want to motivate the use of optics codes such as MAD-X and at the same time illustrate the basic numerical approach behind some of the methods.

- The TWISS method is based on matrix multiplications where first and second order transport matrices are used to get the optics of the machine.

# FODO cell
- Compute the linear optics functions of a FODO cell, which is the simplier combination of quadrupoles required to focuse the beam in both, vertical and horizontal planes.

# Thin lens approximation (f >> $l_q$)

- To do some first estimations analytically one uses the thin lens approximation.

```python
# Symbolic computation

# Symbols defintion
K = sp.Symbol("K", positive = True)
Lq = sp.Symbol("Lq", positive = True)
Ld = sp.Symbol("Ld", positive = True)
```

```python
#Matrices definition: A=sp.Matrix([[1,2],[3,4]])

#Mfoc=??

#Mdefoc=??

#Mdrift=??
```

```python
#################################################
# Transport matrix of a focusing quadrupole     #
#################################################
Mfoc
```

```python
################################################
# Transport matrix of defocusing quadrupole    #
################################################
Mdefoc
```

```python
##############################################
# Transport matrix of a drift                #
##############################################
Mdrift
```

```python
#Matrix multiplication and you can use the sp.simplify(M) command to simplify the elements of the matrix.

```

```python
# Matrix elements computation
# f=200 m, lq=1 m, ld= 30 m
M_thin = M.subs(K, 1/(200*1)).subs(Lq, 1).subs(Ld,30) # units K in m-2, Lq and Ld in m
M_thin
```

```python
#And for 3 FODO cells?
```

# What can we do with the  transfer matrix?
This matrix describes the optical properties of the lattice and defines the beam parameters.

 - We can propagate the phase space coordinates of a particle with a given set of initial coordinates.

```python
# Try with a paralel particle going through the center of the first quadrupole or with a certain amplitude

```

```python
# Try with paralel particle going through the center of the first quadrupole or with a certain amplitude
x=sp.Matrix([[1],[0]])
x
```

```python
x2=M*x
x2
```

- We can compute the periodic solution TWISS functions.

```python
# Transfer matrix
R11, R12, R21, R22 = sp.symbols('R11,R12,R21,R22')

Mt=sp.Matrix([[R11,R12],[R21,R22]])

Mt
```

$\displaystyle \left[\begin{matrix}R_{11} & R_{12}\\R_{21} & R_{22}\end{matrix}\right]$

```python
# In case of periodic conditions in the accelerator there is another way to describe the particles trjectories.
# Periodic solution one-turn-transfer matrix in terms of twiss functions:

a, b, g, m = sp.symbols(r'\alpha,\beta, \gamma,\mu')

M=sp.Matrix([[sp.cos(m)+a*sp.sin(m),b*sp.sin(m)],[-g*sp.sin(m),sp.cos(m)-a*sp.sin(m)]])

M
```

$\displaystyle \left[\begin{matrix}\alpha \sin{\left(\mu \right)} + \cos{\left(\mu \right)} & \beta \sin{\left(\mu \right)}\\- \gamma \sin{\left(\mu \right)} & - \alpha \sin{\left(\mu \right)} + \cos{\left(\mu \right)}\end{matrix}\right]$

```python
#For the phase advance we use the comparison of the trace of the matrix
sp.Eq(sp.cos(m),(R11+R22)/2)
```

$\displaystyle \cos{\left(\mu \right)} = \frac{R_{11}}{2} + \frac{R_{22}}{2}$

```python
#For the beta function we use the R12 matrix element
sp.Eq(b,R12/sp.sin(m))
```

$\displaystyle \beta = \frac{R_{12}}{\sin{\left(\mu \right)}}$

```python
#For the alfa function we use the trace of the matrix also
sp.Eq(a,(R11-R22)/(2*sp.sin(m)))
```

$\displaystyle \alpha = \frac{R_{11} - R_{22}}{2 \sin{\left(\mu \right)}}$

```python
#For the gamma we use the R21 element
sp.Eq(g,-(R21/sp.sin(m)))
```

$\displaystyle \gamma = - \frac{R_{21}}{\sin{\left(\mu \right)}}$

Once you have computed the periodic TWISS functions you can propagate them to any point in the machine using the transfer matrix of the TWISS functions from Transverse dynamics course.

```python
sp.Matrix([[R11**2, -2*R12*R11, R12**2],[-R11*R21,R12*R21+R22*R11, -R12*R22],[R21**2,-2*R22*R21, R22**2]])
```

$\displaystyle \left[\begin{matrix}R_{11}^{2} & - 2 R_{11} R_{12} & R_{12}^{2}\\- R_{11} R_{21} & R_{11} R_{22} + R_{12} R_{21} & - R_{12} R_{22}\\R_{21}^{2} & - 2 R_{21} R_{22} & R_{22}^{2}\end{matrix}\right]$

**Using the periodic one-turn-matrix one can define some interesting relations between the TWISS parameters and the magnetic properties of the lattice.**

# Figure 1: Relation between $\Delta \mu$, K, $L_{cell}$, $L_q$

```python
# NOTE: here we consider that both quadrupoles hve the same strength
# Relation between the phase advance of the cell and K, Lcell, Lq (from periodic solution an stbility condition)
a, b, g, m, d, Lq, Lc, K, pi = sp.symbols(r'\alpha,\beta, \gamma,\mu, \Delta, L_{q}, L_{cell} K \pi')
sp.Eq(d*m/pi,2*sp.asin(K*Lq*Lc/4))
```

```python
#Parametric plots
%matplotlib notebook
plt.rcParams['savefig.dpi'] = 90
plt.rcParams['figure.dpi'] = 90

x=np.arange(0,6,0.01)
y=2*np.arcsin(x/4)/np.pi
fig, ax1 = plt.subplots()
ax1.plot(x,y,'-')
ax1.set_ylabel("$\Delta \mu / \pi [rad]$", fontsize=16)
ax1.set_xlabel("$K*L_{quad}*L_{cell}$ [-]", fontsize=16)
ax1.grid()
ax1.tick_params(axis='both', labelsize=16)
plt.tight_layout()
```

    <ipython-input-2-2b874282a2b0>:7: RuntimeWarning: invalid value encountered in arcsin
      y=2*np.arcsin(x/4)/np.pi

    <IPython.core.display.Javascript object>

# Exercice:
- What is the quadrupole strenght to match a FODO cell phase advance of 45$^\circ$ if the $L_{quad}$=5 m and $L_{cell}$=100 m?

- And for a FODO cell phase advance of 90$^\circ$?

- What is the maximum phase advance in a FODO cell?

```python

```

```python

```

```python

```

# Figure 2: Relation between $\beta_{max}$ and $\beta_{min}$ with K, $L_{cell}$, $L_q$

```python
# Relation between the beta of the cell and K, Lcell, Lq
a, bmin, bmax, g, m, d, Lq, Lc, K, pi = sp.symbols(r'\alpha,\beta_{min}, \beta_{max}, \gamma,\mu, \Delta, L_{q}, L_{cell} K \pi')
sp.Eq(bmin/Lc,(1-(K*Lq*Lc/4))/(sp.sin(2*sp.asin(K*Lq*Lc/4))))
```

$\displaystyle \frac{\beta_{min}}{L_{cell}} = \frac{- \frac{K L_{cell} L_{q}}{4} + 1}{\sin{\left(2 \operatorname{asin}{\left(\frac{K L_{cell} L_{q}}{4} \right)} \right)}}$

```python
sp.Eq(bmax/Lc,(1+(K*Lq*Lc/4))/(sp.sin(2*sp.asin(K*Lq*Lc/4))))
```

$\displaystyle \frac{\beta_{max}}{L_{cell}} = \frac{\frac{K L_{cell} L_{q}}{4} + 1}{\sin{\left(2 \operatorname{asin}{\left(\frac{K L_{cell} L_{q}}{4} \right)} \right)}}$

```python
%matplotlib notebook
plt.rcParams['savefig.dpi'] = 90
plt.rcParams['figure.dpi'] = 90

x=np.arange(0.4,3.90,0.01)
betamax=(1+(x/4))/(np.sin(2*np.arcsin(x/4)))
betamin=(1-(x/4))/(np.sin(2*np.arcsin(x/4)))
fig, ax1 = plt.subplots()
ax1.plot(x,betamax,'-',label=r"$\beta_{max}/L_{cell}$")
ax1.plot(x,betamin,'-',label=r"$\beta_{min}/L_{cell}$")
ax1.set_ylabel("[-]", fontsize=16)
ax1.set_xlabel("$K*L_{quad}*L_{cell}$ [-]", fontsize=16)
plt.grid()
plt.legend()
plt.tick_params(axis='both', labelsize=16)
plt.tight_layout()
```

    <IPython.core.display.Javascript object>

# Exercice:
- What is K and $L_{cell}$ to match a FODO cell with a phase advance of 90$^\circ$ and a $\beta_{max}$ of 200 m? The  $L_{quad}$=2 m.

HINT: You may need to combine the data from both plots.

```python

```

```python

```

```python

```

At the end, the exact solution of the particle motion has to be calculated in full detail but using some approximations we can make the first steps easier and estimate the order of magnitud of some magnetic properties of our lattice.

# Thick lens computation

```python
K = sp.Symbol("K")
Lq = sp.Symbol("Lq")
Ld= sp.Symbol("Ld")

Mfoc=sp.Matrix([[sp.cos(sp.sqrt(K)*Lq),1/(sp.sqrt(K))*sp.sin(sp.sqrt(K)*Lq)],[-(sp.sqrt(K))*sp.sin(sp.sqrt(K)*Lq),sp.cos(sp.sqrt(K)*Lq)]])

Mdefoc=sp.Matrix([[sp.cosh(sp.sqrt(K)*Lq),1/(sp.sqrt(K))*sp.sinh(sp.sqrt(K)*Lq)],[(sp.sqrt(K))*sp.sinh(sp.sqrt(K)*Lq),sp.cosh(sp.sqrt(K)*Lq)]])

Mdrift=sp.Matrix([[1,Ld],[0,1]])
```

```python
#################################################
# Transport matrix of a focusing quadrupople    #
#################################################
Mfoc
```

$\displaystyle \left[\begin{matrix}\cos{\left(\sqrt{K} Lq \right)} & \frac{\sin{\left(\sqrt{K} Lq \right)}}{\sqrt{K}}\\- \sqrt{K} \sin{\left(\sqrt{K} Lq \right)} & \cos{\left(\sqrt{K} Lq \right)}\end{matrix}\right]$

```python
################################################
# Transport matrix of a defocusing quadrupople #
################################################
Mdefoc
```

$\displaystyle \left[\begin{matrix}\cosh{\left(\sqrt{K} Lq \right)} & \frac{\sinh{\left(\sqrt{K} Lq \right)}}{\sqrt{K}}\\\sqrt{K} \sinh{\left(\sqrt{K} Lq \right)} & \cosh{\left(\sqrt{K} Lq \right)}\end{matrix}\right]$

```python
##############################################
# Transport matrix of a dipole               #
##############################################
Mdrift
```

$\displaystyle \left[\begin{matrix}1 & Ld\\0 & 1\end{matrix}\right]$

```python
M=Mdrift*Mdefoc*Mdrift*Mfoc
M=sp.simplify(M)
M
```

$\displaystyle \left[\begin{matrix}\left(\sqrt{K} Ld \sinh{\left(\sqrt{K} Lq \right)} + \cosh{\left(\sqrt{K} Lq \right)}\right) \cos{\left(\sqrt{K} Lq \right)} - \left(2 \sqrt{K} Ld \cosh{\left(\sqrt{K} Lq \right)} + K Ld^{2} \sinh{\left(\sqrt{K} Lq \right)} + \sinh{\left(\sqrt{K} Lq \right)}\right) \sin{\left(\sqrt{K} Lq \right)} & \frac{\left(\sqrt{K} Ld \sinh{\left(\sqrt{K} Lq \right)} + \cosh{\left(\sqrt{K} Lq \right)}\right) \sin{\left(\sqrt{K} Lq \right)} + \left(2 \sqrt{K} Ld \cosh{\left(\sqrt{K} Lq \right)} + K Ld^{2} \sinh{\left(\sqrt{K} Lq \right)} + \sinh{\left(\sqrt{K} Lq \right)}\right) \cos{\left(\sqrt{K} Lq \right)}}{\sqrt{K}}\\- \sqrt{K} \left(\left(\sqrt{K} Ld \sinh{\left(\sqrt{K} Lq \right)} + \cosh{\left(\sqrt{K} Lq \right)}\right) \sin{\left(\sqrt{K} Lq \right)} - \cos{\left(\sqrt{K} Lq \right)} \sinh{\left(\sqrt{K} Lq \right)}\right) & \left(\sqrt{K} Ld \sinh{\left(\sqrt{K} Lq \right)} + \cosh{\left(\sqrt{K} Lq \right)}\right) \cos{\left(\sqrt{K} Lq \right)} + \sin{\left(\sqrt{K} Lq \right)} \sinh{\left(\sqrt{K} Lq \right)}\end{matrix}\right]$

```python
M_thick = M.subs(K, 1/(200*1)).subs(Lq, 1).subs(Ld,30) # units K in m-2, Lq and Ld in m
M_thick
```

$\displaystyle \left[\begin{matrix}0.821745966061752 & 66.6422445425746\\-0.000766666456349212 & 1.15474570697446\end{matrix}\right]$

```python
M_thin
```

$\displaystyle \left[\begin{matrix}0.8275 & 64.5\\-0.00075 & 1.15\end{matrix}\right]$

```python
#And for 3 FODO cells?
```

In real world applications, lattices (including FODO) are not designed by hand but
dedicated software is used to do the design and simulation as for example MAD-X.

The TWISS in MADX it is based in matrix multiplications similar to what has been shown here.

# Wha is next?

- Now we are going to do optics calculations using MAD-X TWISS command (handle thousands of elements).

- We will use the MATCHING MAD-X tool to compute the required magnetic properties for a desired TWISS functions.

- We will use MAD-X to visulize the impact of some properties of the lattice on the TWISS and on the single particle DYNAMICS.

# An introduction to MAD-X using the python interface

In this first part we are going to get familiar with MAD-X syntax.

For more information please refer to the [MAD-X online manual](http://cern.ch/madx/releases/last-rel/madxuguide.pdf).

**Basic steps:**

   - Load the cpymad library.
   - Instantiate the MADX class (we create an object of the class).
   - Access the methods in the class.
       - From the methods available we will be mainly using the method "input" to send to MAD-X the commands.

```python
#Load the cpymad library
from cpymad.madx import Madx
```

```python
#Launching MAD-X
myMad = Madx(stdout=True)
```

```python
#String that will be interpreted by MAD-X
myString='''
stop;
'''
```

```python
#Using the "input" method to send the commandas to the MAD-X class
myMad.input(myString);
```

With the 'stop;' instruction we exit from MAD-X, so, as done in the following cell we need to re-instantiate our MAD-X object with the
**myMad = Madx()** instruction.

---
It is a good practice to make header, please use '!' to comment the single line.

```python
# Define and print a value
myMad = Madx(stdout=True)
myString='''

!***************************************
! It is a good practice to make a header
!***************************************

a= 20;
value a;

'''
myMad.input(myString);
```

---
Use the **help** keyword (very rudimental help)

```python
# To get information about the MAD-X methods (twiss, beam,match... ) use the commnd "help"
myString='''
help, twiss;
'''
myMad.input(myString);
```

```python
myString='''
help, drift;
'''
myMad.input(myString);
```

# Let's define the main ingredients to get some results from MAD-X!

-Definition of machine parameters

-Magnets definition

-Sequence definition

-Beam definition

-Activate the sequence

-Actions

```python
myString='''
! *********************************************************************
! Definition of some parameters
! *********************************************************************

l_cell=60;
quadrupoleLenght=1;
myK:=0.005;// m^-2

! *********************************************************************
! Definition of magnets
! *********************************************************************
QF: quadrupole, L=quadrupoleLenght, K1:=myK;
QD: quadrupole, L=quadrupoleLenght, K1:=-myK;

! *********************************************************************
! Definition of sequence
! *********************************************************************
myCell:sequence, refer=entry, L=L_CELL;
quadrupole1: QF, at=0;
marker1: marker, at=15;
quadrupole2: QD, at=30;
endsequence;

! *********************************************************************
! Definition of beam
! *********************************************************************
beam, particle=proton, energy=1;

! *********************************************************************
! Use of the sequence
! *********************************************************************
use, sequence=myCell;

! *********************************************************************
! TWISS
! *********************************************************************

select, flag=TWISS, column=keyword, name, s, betx, bety,alfx, alfy, x, y, dx, dy;
twiss, file=Test.madx;
!plot, haxis=s, vaxis=betx,bety,dx,colour=100,file=Test;

'''
myMad.input(myString);
```

The OUTPUT generated by MADX can be found by accessing the jupyter-notebook files-view.
- For accessing: CLIC on the jupyter-logo on the top of the page using the right buttom of your mouse and use the option "Open Link in a New Tab".

**Output:**

- SUMM table

- TWISS table

- TWISS .txt file

- TWISS .ps plot

# Accessing the data

- Open the files generated by MAD-X.

- Use python and output the required data on the jupyter-notebook.

    - Using MAD-X commands and the **input()** method.

    - Using **cpymad** methods:
        - **myMad.table.twiss.dframe()**
        - **myMad.table.summ.dframe()**

```python
#######################
#Using MAD-X commands #
#######################
myString='''
value, table(SUMM,Q1);
value, table(SUMM,betxmax);
'''
myMad.input(myString);
```

And for the vertical plane?

```python
#######################
#Using MAD-X commands #
#######################
myString='''
value, table(SUMM,Q2);
value, table(SUMM,betymax);
'''
myMad.input(myString);
```

```python
myString='''
value, table(TWISS,MYCELL$END,betx);
value, table(TWISS,MYCELL$END,bety);
'''
myMad.input(myString);
```

# Using python pandas library

Pandas dataframe are very convenient, have a look in https://pandas.pydata.org/Pandas_Cheat_Sheet.pdf.

```python
# Using another method from cpymad "table.twiss.dframe()"
myDF=myMad.table.twiss.dframe()
```

```python
myDF
```

      name
      keyword
      s
      betx
      alfx
      mux
      bety
      alfy
      muy
      x
      ...
      sig54
      sig55
      sig56
      sig61
      sig62
      sig63
      sig64
      sig65
      sig66
      n1

      #s
      mycell$start:1
      marker
      0.0
      463.623288
      -1.156109
      0.000000
      369.779162
      0.929316
      0.000000
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      quadrupole1
      quadrupole1:1
      quadrupole
      5.0
      463.623288
      1.156109
      0.001709
      369.779162
      -0.929316
      0.002161
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      drift_0[0]
      drift_0:0
      drift
      25.0
      419.394867
      1.055312
      0.008930
      408.967742
      -1.030113
      0.010350
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      marker1
      marker1:1
      marker
      25.0
      419.394867
      1.055312
      0.008930
      408.967742
      -1.030113
      0.010350
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      drift_1[0]
      drift_1:0
      drift
      50.0
      369.779162
      0.929316
      0.019041
      463.623288
      -1.156109
      0.019493
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      quadrupole2
      quadrupole2:1
      quadrupole
      55.0
      369.779162
      -0.929316
      0.021202
      463.623288
      1.156109
      0.021202
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      drift_2[0]
      drift_2:0
      drift
      100.0
      463.623288
      -1.156109
      0.038533
      369.779162
      0.929316
      0.038533
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      #e
      mycell$end:1
      marker
      100.0
      463.623288
      -1.156109
      0.038533
      369.779162
      0.929316
      0.038533
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

8 rows × 256 columns

```python
myDF[['name','s','betx','bety','alfx','alfy']]
```

      name
      s
      betx
      bety
      alfx
      alfy

      #s
      mycell$start:1
      0.0
      463.623288
      369.779162
      -1.156109
      0.929316

      quadrupole1
      quadrupole1:1
      5.0
      463.623288
      369.779162
      1.156109
      -0.929316

      drift_0[0]
      drift_0:0
      25.0
      419.394867
      408.967742
      1.055312
      -1.030113

      marker1
      marker1:1
      25.0
      419.394867
      408.967742
      1.055312
      -1.030113

      drift_1[0]
      drift_1:0
      50.0
      369.779162
      463.623288
      0.929316
      -1.156109

      quadrupole2
      quadrupole2:1
      55.0
      369.779162
      463.623288
      -0.929316
      1.156109

      drift_2[0]
      drift_2:0
      100.0
      463.623288
      369.779162
      -1.156109
      0.929316

      #e
      mycell$end:1
      100.0
      463.623288
      369.779162
      -1.156109
      0.929316

```python
myDF["s"]
```

    #s               0.0
    quadrupole1      5.0
    drift_0[0]      25.0
    marker1         25.0
    drift_1[0]      50.0
    quadrupole2     55.0
    drift_2[0]     100.0
    #e             100.0
    Name: s, dtype: float64

```python
myDF["betx"]
```

    #s             463.623288
    quadrupole1    463.623288
    drift_0[0]     419.394867
    marker1        419.394867
    drift_1[0]     369.779162
    quadrupole2    369.779162
    drift_2[0]     463.623288
    #e             463.623288
    Name: betx, dtype: float64

```python
myDF2=myMad.table.summ.dframe()
```

```python
myDF2
```

      length
      orbit5
      alfa
      gammatr
      q1
      dq1
      betxmax
      dxmax
      dxrms
      xcomax
      ...
      ycorms
      deltap
      synch_1
      synch_2
      synch_3
      synch_4
      synch_5
      synch_6
      synch_8
      nflips

      #e
      100.0
      -0.0
      0.0
      0.0
      0.038533
      -0.043847
      463.623288
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

1 rows × 27 columns

```python
myDF2["q1"]
```

    #e    0.038533
    Name: q1, dtype: float64

# Basic plot

```python
#Plot
plt.plot(myDF['s'],myDF['betx'],'.-b',label='$\\beta_x$')
plt.plot(myDF['s'],myDF['bety'],'.-r',label='$\\beta_y$')
#Labels of the plot
plt.xlabel('s [m]')
plt.ylabel('[m]')
#Legend and grid
plt.legend(loc='best')
plt.grid()
```

# For reference...

---
This is an example to get familiar with the use of the physical constants and the formatting of the output. Have a look on the difference.

```python
myString='''
a=pi;
value a;
set, format="22.20e";
value a;
'''
myMad.input(myString);
```

#
This is an example to get familiar with if and deferred expression. Please note the after the block delimited with {...} the ; can be omitted. Pay attention to circular call!

```python
myString='''
if (1==1){
option, echo=false, info=true;
a=pi;
b:=a;
c=a;
value a;
value b;
value c;
a=CLIGHT*cos(a);
value a;
value b;
value c;}
! BEWARE of circular call!
!a:=a+1;
! When evaluating you will get a fatal error
! value a;
option, echo=true, info=true;
'''
myMad.input(myString);
```

# ---
This is an example to get familiar with **while** and **macros** loops.

```python
myString='''
a(myvariable1,myvariable2): macro = {
value, myvariable1;
value, myvariable1*myvariable2;
}

N=1;
while (N<10){
exec, a(N,N);
N=N+1;
}
'''
myMad.input(myString);
```

---
### List of functions
In MAD-X the following functions are available

- SQRT(x) square root,
- LOG(x) natural logarithm,
- LOG10(x) logarithm base 10,
- EXP(x) exponential,
- SIN(x) trigonometric sine,
- COS(x) trigonometric cosine,
- TAN(x) trigonometric tangent,
- ASIN(x) arc sine,
- ACOS(x) arc cosine,
- ATAN(x) arc tangent,
- SINH(x) hyperbolic sine,
- COSH(x) hyperbolic cosine,
- TANH(x) hyperbolic tangent,
- SINC(x) cardinal sine function,
- ABS(x) absolute value,
- ERF(x) Gauss error,
- ERFC(x) complementary error,
- FLOOR(x) floor, largest previous integer,
- CEIL(x) ceiling, smallest next integer,
- ROUND(x) round, closest integer,
- FRAC(x) fractional part of number,
- RANF() random number, uniformly distributed in [0,1],
- GAUSS() random number, gaussian distribution with unit standard deviation,
- TGAUSS(x) random number, gaussian distribution with unit standard deviation, truncated at x standard deviations;

---
### List of physical constant

| MAD-X name  | symbol  |  value |unit|
|:-:|:-:|:-:|:-:|
|PI| π |4 * atan(1)| 1|
|TWOPI|2π| 2 * PI| 1|
|DEGRAD| 180/π |180 / PI| deg/rad|
|RADDEG| π/180 |PI / 180 |rad/deg|
|E| e |exp(1) |1|
|EMASS| me |0.510998928e−3| GeV|
|PMASS| mp |0.938272046| GeV|
|NMASS| u |0.931494061| GeV|
|MUMASS| mµ| 0.1056583715 |GeV|
|CLIGHT| c| 299792458| m/s|
|QELECT| e| 1.602176565e−19| A.s|
|HBAR| ¯h| 6.58211928e−25| MeV.s|
|ERAD| re| 2.8179403267e−15| m|
|PRAD| re(me/mp)| ERAD*EMASS/PMASS| m|


[Table of Contents](http://nbviewer.ipython.org/github/rlabbe/Kalman-and-Bayesian-Filters-in-Python/blob/master/table_of_contents.ipynb)
\appendix
# Installation, Python, NumPy, and FilterPy

```python
#format the book
%matplotlib inline
from __future__ import division, print_function
from book_format import load_style
load_style()
```

<style>
@import url('http://fonts.googleapis.com/css?family=Source+Code+Pro');
@import url('http://fonts.googleapis.com/css?family=Lora');

//@import url('http://fonts.googleapis.com/css?family=Open+Sans');
//@import url('http://fonts.googleapis.com/css?family=Vollkorn');
//@import url('http://fonts.googleapis.com/css?family=Karla');
//@import url('http://fonts.googleapis.com/css?family=Poppins');
//@import url('http://fonts.googleapis.com/css?family=Arimo');
//@import url('http://fonts.googleapis.com/css?family=Roboto');
//@import url('http://fonts.googleapis.com/css?family=Lato');
//@import url('http://fonts.googleapis.com/css?family=Domine');
//@import url('http://fonts.googleapis.com/css?family=Chivo');
//@import url('http://fonts.googleapis.com/css?family=Cardo');
//@import url('http://fonts.googleapis.com/css?family=Arvo');
//@import url('http://fonts.googleapis.com/css?family=Crimson+Text');
//@import url('http://fonts.googleapis.com/css?family=Ubuntu');
//@import url('http://fonts.googleapis.com/css?family=Fontin');
//@import url('http://fonts.googleapis.com/css?family=Raleway');
//@import url('http://fonts.googleapis.com/css?family=Merriweather');

.CodeMirror pre {
    font-family: 'Source Code Pro', Consolas, monocco, monospace;
}
    div.cell{
        //width: 950px;
        margin-left: 0% !important;
        margin-right: auto;
    }
    div.text_cell_render{
        font-family: 'Lora';
        //font-family: 'Open Sans';
        //font-family: 'Karla',verdana,arial,sans-serif;
        //font-family: 'Roboto',verdana,arial,sans-serif;
        //font-family: 'Lato',verdana,arial,sans-serif;
        //font-family: 'Domine',verdana,arial,sans-serif;
        //font-family: 'Chivo',verdana,arial,sans-serif;
        //font-family: 'Cardo',verdana,arial,sans-serif;
        //font-family: 'Arvo',verdana,arial,sans-serif;
        //font-family: 'Poppins',verdana,arial,sans-serif;
        //font-family: 'Ubuntu',verdana,arial,sans-serif;
        //font-family: 'Fontin',verdana,arial,sans-serif;
        //font-family: 'Raleway',verdana,arial,sans-serif;
        //font-family: 'Merriweather',verdana,arial,sans-serif;
        //font-family: 'Crimson Text', verdana,arial,sans-serif;
        //font-family: verdana,arial,sans-serif;
        //font-family: arial,sans-serif;
        line-height: 125%;
        font-size: 130%;
        text-align: justify;
        text-justify:inter-word;
    }
    div.text_cell code {
        background: transparent;
        color: #000000;
        font-weight: 400;
        font-size: 12pt;
        //font-style: bold;
        font-family:  'Source Code Pro', Consolas, monocco, monospace;
   }
    h1 {
        font-family: 'Open sans',verdana,arial,sans-serif;
	}

    div.input_area {
        background: #F6F6F9;
        border: 1px solid #586e75;
    }

    .text_cell_render h1 {
        font-weight: 200;
        font-size: 30pt;
        line-height: 100%;
        color:#c76c0c;
        margin-bottom: 0.5em;
        margin-top: 1em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h2 {
        font-family: 'Open sans',verdana,arial,sans-serif;
        text-align: left;
    }
    .text_cell_render h2 {
        font-weight: 200;
        font-size: 16pt;
        font-style: italic;
        line-height: 100%;
        color:#c76c0c;
        margin-bottom: 0.5em;
        margin-top: 1.5em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h3 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }
    .text_cell_render h3 {
        font-weight: 200;
        font-size: 14pt;
        line-height: 100%;
        color:#d77c0c;
        margin-bottom: 0.5em;
        margin-top: 2em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h4 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }
    .text_cell_render h4 {
        font-weight: 100;
        font-size: 14pt;
        color:#d77c0c;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
        white-space: nowrap;
    }
    h5 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }

    .text_cell_render h5 {
        font-weight: 200;
        font-style: normal;
        color: #1d3b84;
        font-size: 16pt;
        margin-bottom: 0em;
        margin-top: 0.5em;
        display: block;
        white-space: nowrap;
    }
    div.output_subarea.output_text.output_pyout {
        overflow-x: auto;
        overflow-y: scroll;
        max-height: 50000px;
    }
    div.output_subarea.output_stream.output_stdout.output_text {
        overflow-x: auto;
        overflow-y: scroll;
        max-height: 50000px;
    }
    div.output_wrapper{
        margin-top:0.2em;
        margin-bottom:0.2em;
}

    code{
        font-size: 6pt;

    }
    .rendered_html code{
    background-color: transparent;
    }
    ul{
        margin: 2em;
    }
    ul li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
    }
    ul li li{
        padding-left: 0.2em;
        margin-bottom: 0.2em;
        margin-top: 0.2em;
    }
    ol{
        margin: 2em;
    }
    ol li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
    }
    ul li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.2em;
    }
    a:link{
       color:#447adb;
    }
    a:visited{
       color: #1d3b84;
    }
    a:hover{
       color: #1d3b84;
    }
    a:focus{
       color:#447adb;
    }
    a:active{
       font-weight: bold;
       color:#447adb;
    }
    .rendered_html :link {
       text-decoration: underline;
    }
    .rendered_html :hover {
       text-decoration: none;
    }
    .rendered_html :visited {
      text-decoration: none;
    }
    .rendered_html :focus {
      text-decoration: none;
    }
    .rendered_html :active {
      text-decoration: none;
    }
    .warning{
        color: rgb( 240, 20, 20 )
    }
    hr {
      color: #f3f3f3;
      background-color: #f3f3f3;
      height: 1px;
    }
    blockquote{
      display:block;
      background: #fcfcfc;
      border-left: 5px solid #c76c0c;
      font-family: 'Open sans',verdana,arial,sans-serif;
      width:680px;
      padding: 10px 10px 10px 10px;
      text-align:justify;
      text-justify:inter-word;
      }
      blockquote p {
        margin-bottom: 0;
        line-height: 125%;
        font-size: 100%;
      }
</style>

This book is written in Jupyter Notebook, a browser based interactive Python environment that mixes Python, text, and math. I choose it because of the interactive features - I found Kalman filtering nearly impossible to learn until I started working in an interactive environment. It is difficult to form an intuition about many of the parameters until you can change them and immediately see the output. An interactive environment also allows you to play 'what if' scenarios. "What if I set $\mathbf{Q}$ to zero?" It is trivial to find out with Jupyter Notebook.

Another reason I choose it is because most textbooks leaves many things opaque. For example, there might be a beautiful plot next to some pseudocode. That plot was produced by software, but software that is not available to the reader. I want everything that went into producing this book to be available to you. How do you plot a covariance ellipse? You won't know if you read most books. With Jupyter Notebook all you have to do is look at the source code.

Even if you choose to read the book online you will want Python and the SciPy stack installed so that you can write your own Kalman filters. There are many different ways to install these libraries, and I cannot cover them all, but I will cover a few typical scenarios.

## Installing the SciPy Stack

This book requires IPython, Jupyter, NumPy, SciPy, SymPy, and Matplotlib. The SciPy stack of NumPy, SciPy, and Matplotlib depends on third party Fortran and C code, and is not trivial to install from source code. The SciPy website strongly urges using a pre-built installation, and I concur with this advice.

I use the Anaconda distribution from Continuum Analytics. This is an excellent distribution that combines all of the packages listed above, plus many others. Installation is very straightforward, and it can be done alongside other Python installations you might already have on your machine. It is free to use. You may download it from here: http://continuum.io/downloads I strongly recommend using the latest Python 3 version that they provide.

There are other choices for installing the SciPy stack. You can find instructions here: http://scipy.org/install.html

Many Linux distributions come with these packages preinstalled. However, they are often somewhat dated and they will need to be updated as the book depends on recent versions of all. Updating a specific Linux installation is beyond the scope of this book. An advantage of the Anaconda distribution is that it does not modify your local Python installation, so you can install it and not break your linux distribution.

## Installing FilterPy

FilterPy is a Python library that implements all of the filters used in this book, and quite a few others. Installation is easy using `pip`. Issue the following from the command prompt:

     pip install filterpy

FilterPy is written by me, and the latest development version is always available at https://github.com/rlabbe/filterpy.

## Downloading and Running the Book

The book is stored in a github repository. From the command line type the following:

    git clone https://github.com/rlabbe/Kalman-and-Bayesian-Filters-in-Python.git

If you do not have git installed, browse to https://github.com/rlabbe/Kalman-and-Bayesian-Filters-in-Python where you can download the book via your browser.

Now, from the command prompt change to the directory that was just created, and then run Jupyter notebook:

    cd Kalman-and-Bayesian-Filters-in-Python
    juptyer notebook

A browser window should launch showing you all of the chapters in the book. Browse to the first chapter by clicking on it, then open the notebook in that subdirectory by clicking on the link.

More information about running the notebook can be found here:

http://jupyter-notebook-beginner-guide.readthedocs.org/en/latest/execute.html

## Using Juptyer Notebook

A complete tutorial on Jupyter Notebook is beyond the scope of this book. Many are available online. In short, Python code is placed in cells. These are prefaced with text like `In [1]:`, and the code itself is in a boxed area. If you press CTRL-ENTER while focus is inside the box the code will run and the results will be displayed below the box. Like this:

```python
print(3+7.2)
```

    10.2

If you have this open in Jupyter Notebook now, go ahead and modify that code by changing the expression inside the print statement and pressing CTRL+ENTER. The output should be changed to reflect what you typed in the code cell.

## SymPy

SymPy is a Python package for performing symbolic mathematics. The full scope of its abilities are beyond this book, but it can perform algebra, integrate and differentiate equations, find solutions to differential equations, and much more. For example, we use use it to compute the Jacobian of matrices and expected value integral computations.

First, a simple example. We will import SymPy, initialize its pretty print functionality (which will print equations using LaTeX). We will then declare a symbol for SymPy to use.

```python
import sympy
sympy.init_printing(use_latex='mathjax')

phi, x = sympy.symbols('\phi, x')
phi
```

$$\phi$$

Notice how it prints the symbol `phi` using LaTeX. Now let's do some math. What is the derivative of $\sqrt{\phi}$?

```python
sympy.diff('sqrt(phi)')
```

$$\frac{1}{2 \sqrt{\phi}}$$

We can factor equations

```python
sympy.factor(phi**3 -phi**2 + phi - 1)
```

$$\left(\phi - 1\right) \left(\phi^{2} + 1\right)$$

and we can expand them.

```python
((phi+1)*(phi-4)).expand()
```

$$\phi^{2} - 3 \phi - 4$$

You can evauate an equation for specific values of its variables:

```python
w =x**2 -3*x +4
print(w.subs(x, 4))
print(w.subs(x, 12))
```

    8
    112

You can also use strings for equations that use symbols that you have not defined:

```python
x = sympy.expand('(t+1)*2')
x
```

$$2 t + 2$$

Now let's use SymPy to compute the Jacobian of a matrix. Given the function

$$h=\sqrt{(x^2 + z^2)}$$

find the Jacobian with respect to x, y, and z.

```python
x, y, z = sympy.symbols('x y z')

H = sympy.Matrix([sympy.sqrt(x**2 + z**2)])

state = sympy.Matrix([x, y, z])
H.jacobian(state)
```

$$\left[\begin{matrix}\frac{x}{\sqrt{x^{2} + z^{2}}} & 0 & \frac{z}{\sqrt{x^{2} + z^{2}}}\end{matrix}\right]$$

Now let's compute the discrete process noise matrix $\mathbf Q$ given the continuous process noise matrix
$$\mathbf Q = \Phi_s \begin{bmatrix}0&0&0\\0&0&0\\0&0&1\end{bmatrix}$$

The integral is

$$\mathbf Q = \int_0^{\Delta t} \mathbf F(t)\mathbf Q\mathbf F^T(t)\, dt$$

where
$$\mathbf F(\Delta t) = \begin{bmatrix}1 & \Delta t & {\Delta t}^2/2 \\ 0 & 1 & \Delta t\\ 0& 0& 1\end{bmatrix}$$

```python
dt = sympy.symbols('\Delta{t}')
F_k = sympy.Matrix([[1, dt, dt**2/2],
                    [0,  1,      dt],
                    [0,  0,       1]])
Q = sympy.Matrix([[0,0,0],
                  [0,0,0],
                  [0,0,1]])

sympy.integrate(F_k*Q*F_k.T,(dt, 0, dt))
```

$$\left[\begin{matrix}\frac{\Delta{t}^{5}}{20} & \frac{\Delta{t}^{4}}{8} & \frac{\Delta{t}^{3}}{6}\\\frac{\Delta{t}^{4}}{8} & \frac{\Delta{t}^{3}}{3} & \frac{\Delta{t}^{2}}{2}\\\frac{\Delta{t}^{3}}{6} & \frac{\Delta{t}^{2}}{2} & \Delta{t}\end{matrix}\right]$$