IBM Quantum Challenge Africa 2021

Introduction and the Crop-Yield Problem

Quantum Computing has the potential to revolutionize computing, as it can solve problems that are not possible to solve on a classical computer. This extra ability that quantum computers have is called quantum advantage. To achieve this goal, the world needs passionate and competent end-users: those who know how to apply the technology to their field.

In this challenge you will be exposed, at a high-level, to quantum computing through Qiskit. As current or future users of quantum computers, you need to know what problems are appropriate for quantum computation, how to structure the problem model/inputs so that they are compatible with your chosen algorithm, and how to execute a given algorithm and quantum solution to solve the problem.

This is the first notebook for the IBM Quantum Challenge Africa. Before starting here, ensure you have completed the Week 0 content in preparation for the following exercises.

## Initialization

To ensure the demonstrations and exercises have the required Python modules and libraries, run the following cell before continuing.

```python
# Import auxiliary libraries
import numpy as np

# Import Qiskit
from qiskit import IBMQ, Aer
from qiskit.algorithms import QAOA, VQE, NumPyMinimumEigensolver
from qiskit.utils import QuantumInstance, algorithm_globals
from qiskit.providers.aer.noise.noise_model import NoiseModel

from qiskit_optimization import QuadraticProgram
from qiskit_optimization.algorithms import MinimumEigenOptimizer
from qiskit_optimization.converters import QuadraticProgramToQubo

import qiskit.test.mock as Fake
```

## Table of Contents

The notebook is structured as follows:

1. Initialization
2. Table of Contents
3. Qiskit and its parts
4. Setting up the Qiskit Environment
5. Quadratic Problems
6. Crop-Yield Problem as a Quadratic Problem
7. Solving the Crop-Yield Problem using Quantum Computing
8. Simulating a Real Quantum Computer for the Crop-Yield Problem

## Qiskit and its parts

Qiskit is divided into multiple modules for different purposes, with Terra at the core of the Qiskit ecosystem. It helps to be familiar with what each module can do and whether you will need to utilize the software contained within them for your specific problem. There are four modules that deal with the application of quantum computing; with tools and algorithms built specifically for their fields.

Have a quick look at the [Qiskit documentation](https://qiskit.org/overview) and this IBM Research [blog post](https://www.ibm.com/blogs/research/2021/04/qiskit-application-modules/) to see what these modules are called. Once you have done so, you can complete the first exercise of the challenge: replace the ficticious module names, in the python cell below, with the correct module names. Though it would be fun to have a Qiskit Gaming module, it has not yet been developed. If you are interested in contributing to the open-source Qiskit community, have a look at the [contribution guide](https://qiskit.org/documentation/contributing_to_qiskit.html).

### Exercise 1a: Qiskit Applications Modules

```python
# Definitely real Qiskit module names
qiskit_module_names = [
    "Qiskit Gaming",
    "Qiskit Cooking",
    "Qiskit VR",
    "Qiskit Word Processing",
]
```

```python
# Correct names
qiskit_module_names_correct = [
    "Qiskit finance",
    "qiskit nature",
    "qiskit optimization",
    "qiskit machine learning",
]
```

Run the following python cell to check if you have the correct module names.

```python
from qc_grader import grade_ex1a

grade_ex1a(qiskit_module_names)
```

```python
from qc_grader import grade_ex1a

grade_ex1a(qiskit_module_names_correct)
```

    /opt/conda/lib/python3.8/site-packages/pyscf/lib/misc.py:47: H5pyDeprecationWarning: Using default_file_mode other than 'r' is deprecated. Pass the mode to h5py.File() instead.
      h5py.get_config().default_file_mode = 'a'

    Submitting your answer for ex1/partA. Please wait...
    Congratulations 🎉! Your answer is correct and has been submitted.

### Qiskit Applications Modules

In this notebook, we will use the `Qiskit Optimization` applications module for a specific problem to illustrate how

1. classical and mathematical problem definitions are represented in Qiskit,
2. some quantum algorithms are defined,
3. to execute a quantum algorithm for a given problem definition,
4. and algorithms and Qiskit problems are executed on real and simulated quantum computers.

We will also be using Qiskit Terra and Aer as they provide the foundation of Qiskit and high-performance quantum simulators respectively.

## Quadratic Problems

Some computational problems can be formulated into quadratic equations such that the minimum of the quadratic equations is the optimal solution, if any exist. These problems are encountered in finance, agriculture, operations and production management, and economics.

Quadratic programming is also used to identify an optimal financial portfolio with minimum risk and optimizing the layout of production components in a factory to minimize the travel distance of resources. This notebook focuses on agriculture as it is a relevant application of quantum computing to problems facing the African continent. However, all of these applications share two common characteristics: the system can be modelled as a quadratic equation and the system variables may be constrained, with their values limited to within a given range.

---

Quadratic problems take on the following structure. Given a vector of $n$ variables $x\in\mathbb{R}^n$, the quadratic function to minimize is as follows.

$$
\begin{align}
\text{minimize}\quad & f\left(x\right)=\frac{1}{2}x^\top{}\mathbf{Q}x + c^\top{}x &\\
\text{subject to}\quad & \mathbf{A}x\leq{}b&\\
& x^\top{}\mathbf{Q}_ix + c_{i}^\top{}x\leq{}r_i,\quad&\forall{}i\in[1,k_q]\\
& l_i\leq{}x_i\leq{}u_i,\quad&\forall{}i\in[1,k_l]\\
\end{align}
$$

$\mathbf{Q}$, $\mathbf{Q}_i$, and $\mathbf{A}$ are $n\times{}n$ symmetric matrices. $c$ and $c_i$ are $n\times{}1$ column vectors. $\mathbf{Q}_i$, $\mathbf{A}$, $c_i$, $l_i$, and $u_i$ define constraints on the variables in $x$. The quadratic equation at the core of the quadratic problem is found by multiplying out the matrices in the minimization function. Though '$\leq{}$' is used in the constraint equations above, any identity relationship may be used for any number of constraints: i.e. "$<$", "$=$", "$>$", "$\geq$", or "$\leq$".

A valid solution to the quadratic must satisfy all conditions for the problem. Examples of some constraints are given below. The first two are linear constraints whereas the third example is a quadratic constraint.

$$ x_1 + x_4 \leq{} 10$$

$$ x_2 - 3x_6 = 10$$

$$x_1x_2 - 4x_3x_4 + x_5 \leq{} 15 $$

Qiskit has Python code that allows you to implement a quadratic problem as a `QuadraticProgram` instance. Though our definition above used matrices to define the coefficients, `QuadraticProgram` allows you to define the objective (function overwhich to minimize) directly. To illustrate how to use `QuadraticProgram`, we will use the following quadratic problem definition, with three integer variables.

$$\begin{align}
\text{minimize}\quad{} & f(x)=(x_1)^2 + (x_2)^2 - x_1x_2 - 6x_3 \\
\text{subject to}\quad{} & x_1 + x_2 = 2             \\
                         & x_2x_3 \geq{} 1           \\
                         & -2 \leq{} x_2 \leq{} 2    \\
                         & -2 \leq{} x_3 \leq{} 4    \\
\end{align}$$

The figure below shows the constraints on $x_1$ and $x_3$, with some simplifcation. The shaded area denotes valid values for $x_1$ and $x_3$, within which $f(x)$ must be minimized.

In the following code, the above quadratic problem is defined as a `QuadraticProgram` instance. Have a look at the [Qiskit documentation for `QuadraticProgram`](https://qiskit.org/documentation/stubs/qiskit.optimization.QuadraticProgram.html), as it can be very useful in helping your understanding of its interface.

The quadratic to minimize, called an objective, is implemented using dictionaries. This allows you, the developer, to explicitly define coefficients for specific variables and terms. The keys in the dictionaries are the variable names identifying a term in $f(x)$. For example, `("x_1","x_2")` is for $x_1x_2$. The values for each item are the coefficients for said terms. Terms that are subtracted in $f(x)$ must have a negative coefficient.

```python
quadprog = QuadraticProgram(name="example 1")
quadprog.integer_var(name="x_1", lowerbound=0, upperbound=4)
quadprog.integer_var(name="x_2", lowerbound=-2, upperbound=2)
quadprog.integer_var(name="x_3", lowerbound=-2, upperbound=4)
quadprog.minimize(
    linear={"x_3": -6},+
    quadratic={("x_1", "x_1"): 1, ("x_2", "x_2"): 1, ("x_1", "x_2"): -1},
)
quadprog.linear_constraint(linear={"x_1": 1, "x_2": 1}, sense="=", rhs=2)
quadprog.quadratic_constraint(quadratic={("x_2", "x_3"): 1}, sense=">=", rhs=1)
```

    <qiskit_optimization.problems.quadratic_constraint.QuadraticConstraint at 0x7f5f0c236e20>

A `QuadraticProgram` can have three types of variables: binary, integer, and continuous. The Qiskit implementation of the algorithms we are going to use currently only support binary and integer variables. There are other algorithms that allow for the simulation of continuous variables, but they are not covered in this notebook. If you want to know more about them though, have a look at this Qiskit tutorial on [algorithms to solve mixed-variable quadratic problems](https://qiskit.org/documentation/tutorials/optimization/5_admm_optimizer.html).

We can visualize the `QuadraticProgram` as an LP string, a portable text-based format used representating the model as a **L**inear **P**rogramming problem.

```python
print(quadprog.export_as_lp_string())
```

    \ This file has been generated by DOcplex
    \ ENCODING=ISO-8859-1
    \Problem name: example 1

    Minimize
     obj: - 6 x_3 + [ 2 x_1^2 - 2 x_1*x_2 + 2 x_2^2 ]/2
    Subject To
     c0: x_1 + x_2 = 2
     q0: [ x_2*x_3 ] >= 1

    Bounds
           x_1 <= 4
     -2 <= x_2 <= 2
     -2 <= x_3 <= 4

    Generals
     x_1 x_2 x_3
    End

Any optimization problem that can be represented as a single second-order equation, in that the greatest exponent of any term is 2, can be transformed into a quadratic problem or program of the form given above. The above example is arbitrary and does not necessarily represent a given real-world problem. The main problem of this notebook focuses on optimizing the yield of a farm, though only the problem definition need be changed to apply this technique to other quadratic problem applications.

## Crop-Yield Problem as a Quadratic Problem

To show how to solve your quadratic program using a quantum computer, we will use two algorithms to solve the crop-yield problem. It is a common need to optimize the crops and management of a farm to reduce risk while increasing profits. One of the big challenges facing Africa and the whole world is how to produce enough food for everyone. The problem here focuses not on profits but on the tonnage of crops harvested. Imagine you have a farm with three hectares of land suitable for farming. You need to choose which crops to plant from a selection of four. Furthermore, you also need to determine how many hectares of each you should plant. The four crops you can plant are wheat, soybeans, maize, and a push-pull crop. The fourth cannot be sold once harvested but it can help increase the yield of the other crops.

            Our beautiful three hectare farm

            Wheat

            Soybeans

            Maize

            Push-Pull

There are three types of farming methods we can use: monocropping, intercropping, and push-pull farming. These are shown below. Monocropping is where only one crop is farmed. This is can make the farm susceptible to disease and pests as the entire yield would be affected. In some instances, growing two different plants nearby each other will increase the yield of both, though sometimes it can decrease the yield. Intercropping is the process where different plants are chosen to _increase_ the yield. Push-Pull crops are pairs of plants that repel pests and attract pests respectively. Integrating them into a larger farm increases the yield of harvested food but with the cost of not necessarily being able to use the harvest of Push-Pull crops as part of the total yield. This is because the Push-Pull crop may not be usable or even edible.

            Monocropping

            Intercropping

            Push-Pull farming

---
Only in certain cases can quadratic programming problems be solved easily using classical problems. In their general sense, they are NP-Hard; a class of problems that is difficult to solve using classical computational methods. In fact, the best classical method to solve these problems involves heuristics, a technique that finds an approximate solution. Quantum Computers have been shown to provide significant speed-up and better scaling for some heuristic problems. The crop-yield problem is a combinatorial problem, in that the solution is a specific combination of input parameters. Though the problem shown here is small enough to solve classically, larger problems become intractable on a classical computer owing to the number of combinations of which to optimize.

Solving the above problem using quantum computing involves three components:

1. Defining the problem
2. Defining the algorithm
3. Executing the algorithm on a backend

Many problems in Qiskit follow this structure as the algorithm you use can typically be swapped for another without significantly redefining your problem. Execution on different backends is the easiest, as long as the device has sufficient resource. The first component is given below, with the second and third in sections 1.5 and 1.6.

### Define the Crop-Yield problem

The following problem is defined for you but the `QuadraticProgram` is not implemented. Your task at the end of this section is to implement the `QuadraticProgram` for the given crop-yield model.

Your farm has three hectares available, $3~ha$, with each crop taking up $0~ha$ or $1~ha$. We define the yield of the farm as a quadratic function where the influence of each crop on eachother is represented by the quadratic coefficients. The variables in this quadratic are the number of hectares of the crop to be planted and the objective function to maximize is the yield of usable crops in tons. Here is the mathematical model for the problem. In this scenario, all crops increase the yield of other crops. However, the problem to solve is which crops to use to achieve the maximum yield.

The farm yield, in tons, is modelled as a quadratic equation, given below, with constraints on the hectares used by each crop and the total hectares available. Each crop is shown using a different symbol, as shown above, representing the number of hectares to be planted of said plant. Note that we can only plant up to 1 hectare of each crop and that our farm is constrained to 3 hectares.

----
#### Non-graphical notation
Here is a non-graphical representation of the above model, if you are struggling to interpret the above graphic.

$$
\begin{align}
    \text{maximize} \quad & 2(\operatorname{Wheat}) + \operatorname{Soybeans} + 4(\operatorname{Maize}) \\
    & + 2.4(\operatorname{Wheat}\times\operatorname{Soybeans}) \\ & + 4(\operatorname{Wheat}\times\operatorname{Maize})\\
    &+ 4(\operatorname{Wheat}\times\operatorname{PushPull}) \\ & + 2(\operatorname{Soybeans}\times\operatorname{Maize}) \\
                          & + (\operatorname{Soybeans}\times\operatorname{PushPull}) \\ & + 5(\operatorname{Maize}\times\operatorname{PushPull})
\end{align}
$$

$$
\begin{align}
\text{subject to} \quad & \operatorname{Wheat} + \operatorname{Soybeans} + \operatorname{Maize} + \operatorname{PushPull} \leq{} 3\\
& 0\leq{}\operatorname{Wheat}\leq{}1\\
& 0\leq{}\operatorname{Soybeans}\leq{}1\\
& 0\leq{}\operatorname{Maize}\leq{}1\\
& 0\leq{}\operatorname{PushPull}\leq{}1
\end{align}
$$

### Exercise 1b: Create Quadratic Program from crop-yield variables

Your first exercise is to create a `QuadraticProgram` that represents the above model. Write your implementation in the `cropyield_quadratic_program` function below. Remember to use the example as a guide, and to look at the [QuadraticProgram documentation](https://qiskit.org/documentation/tutorials/optimization/1_quadratic_program.html?highlight=quadraticprogram) and [Qiskit reference](https://qiskit.org/documentation/stubs/qiskit.optimization.QuadraticProgram.html?highlight=quadraticprogram#qiskit.optimization.QuadraticProgram).

**Note:** Ensure your variables are named `Wheat`, `Soybeans`, `Maize,` and `PushPull`. This is necessary for the grader to work.

```python
def cropyield_quadratic_program():
    cropyield = QuadraticProgram(name="Crop Yield")
    ##############################
    # Put your implementation here
    crops = ["Wheat", "Soybeans", "Maize", "PushPull"]

    linear_ = {"Wheat":2, "Soybeans":1, "Maize":4, "PushPull":0}

    quadratic_ = {("Wheat", "Soybeans"): 2.4,
        ("Wheat", "Maize"): 4,
        ("Wheat", "PushPull"): 4,
        ("Soybeans", "Maize"): 2,
        ("Soybeans", "PushPull"): 1,
        ("Maize", "PushPull"): 5}
    lin_const = {"Wheat":1, "Soybeans":1, "Maize":1, "PushPull":1}

    for crop in crops:
        cropyield.integer_var(name=crop, lowerbound=0, upperbound=1)
    cropyield.maximize(
    linear=linear_,
    quadratic=quadratic_,
)
    cropyield.linear_constraint(linear=lin_const, sense="<=", rhs=3)

    #
    ##############################
    return cropyield
```

```python
def cropyield_quadratic_program_solution():
    ### Define crop-yield quadratic coefficients
    # Constraint variables on hectares
    max_hectares_per_crop = 1
    hectares_available = 3

    # Crops to be planted
    crops = ["Wheat", "Soybeans", "Maize", "PushPull"]

    # Monoculture crop yields, in tons/hectare
    yield_monoculture_per_hectare = {
        "Wheat": 2,
        "Soybeans": 1,
        "Maize": 4,
        "PushPull": 0,
    }

    # Intercrop crop yields, in tons/hectare
    yield_intercrop_per_hectare = {
        ("Wheat", "Soybeans"): 2.4,
        ("Wheat", "Maize"): 4,
        ("Wheat", "PushPull"): 4,
        ("Soybeans", "Maize"): 2,
        ("Soybeans", "PushPull"): 1,
        ("Maize", "PushPull"): 5,
    }
    # Create a QuadraticProgram
    cropyield = QuadraticProgram(
        name="Crop Yield",
    )

    # Add crop-yield variables
    for crop in crops:
        # Note that we add these as integer variables but max_hectares_per_crop is 1
        cropyield.integer_var(lowerbound=0, upperbound=max_hectares_per_crop, name=crop)

    # Add crop-yield quadratic using monoculture and intercrop yield variables
    cropyield.maximize(
        linear=yield_monoculture_per_hectare,
        quadratic=yield_intercrop_per_hectare,
    )

    # Add constraint for the total farm area
    hectares_available_linear_dict = dict([(crop, 1) for crop in crops])
    cropyield.linear_constraint(
        linear=hectares_available_linear_dict,
        sense="<=",
        rhs=hectares_available,
        name="Hectares Availabe",
    )
    return cropyield
```

Once you feel your implementation is correct, you can grade your solution in the following cell.

```python
# Execute this cell to grade your solution
from qc_grader import grade_ex1b

cropyield = cropyield_quadratic_program()
grade_ex1b(cropyield)
```

    Submitting your answer for ex1/partB. Please wait...
    Congratulations 🎉! Your answer is correct and has been submitted.

```python
# This shows that the 'solution' is marked as correct
from qc_grader import grade_ex1b

cropyield = cropyield_quadratic_program_solution()
grade_ex1b(cropyield)
```

### Converting QuadraticPrograms

If we want to estimate how many qubits this quadratic program requires, we can convert it to an Ising Model and print the `num_qubits` parameter. An [ising model](https://qiskit.org/documentation/apidoc/qiskit.optimization.applications.ising.html?highlight=ising) is a special system model type that is well suited for quantum computing. Though we will not be using an ising model explicitly, the algorithms and Qiskit classes we are using do this conversion internally.

```python
# Estimate the number of qubits required
ising_operations, _ = (
    QuadraticProgramToQubo()
    .convert(
        cropyield,
    )
    .to_ising()
)
print(f"Number of qubits required is {ising_operations.num_qubits}")
```

    Number of qubits required is 6

Even though quadratic programs are widely used in Qiskit, the algorithms we are going to use require binary variables. Qiskit provides an automated method for converting our integer variables into binary variables. The binary-only form is called a _Quadratic Unconstrained Binary Optimization_ problem, or `QUBO`. The conversion is done using `QuadraticProgramToQUBO` from the Qiskit optimization module. Every integer variable, and their associated constraints, are transformed into binary variables.

Run the following code to see how the QUBO version of the cropyield problem looks. Notice how the quadratic becomes longer and more variables are added. This is to account for the bits in each variable, including the constraints. When we run our quantum algorithm to solve this QuadraticProgram, it is converted to a QUBO instance within the Qiskit algorith implementation, implicitly.

```python
QuadraticProgramToQubo().convert(cropyield)
```

    \ This file has been generated by DOcplex
    \ ENCODING=ISO-8859-1
    \Problem name: Crop Yield

    Minimize
     obj: - 160.400000000000 Wheat@0 - 159.400000000000 Soybeans@0
          - 162.400000000000 Maize@0 - 158.400000000000 PushPull@0
          - 158.400000000000 c0@int_slack@0 - 316.800000000000 c0@int_slack@1 + [
          52.800000000000 Wheat@0^2 + 100.800000000000 Wheat@0*Soybeans@0
          + 97.600000000000 Wheat@0*Maize@0 + 97.600000000000 Wheat@0*PushPull@0
          + 105.600000000000 Wheat@0*c0@int_slack@0
          + 211.200000000000 Wheat@0*c0@int_slack@1 + 52.800000000000 Soybeans@0^2
          + 101.600000000000 Soybeans@0*Maize@0
          + 103.600000000000 Soybeans@0*PushPull@0
          + 105.600000000000 Soybeans@0*c0@int_slack@0
          + 211.200000000000 Soybeans@0*c0@int_slack@1 + 52.800000000000 Maize@0^2
          + 95.600000000000 Maize@0*PushPull@0
          + 105.600000000000 Maize@0*c0@int_slack@0
          + 211.200000000000 Maize@0*c0@int_slack@1 + 52.800000000000 PushPull@0^2
          + 105.600000000000 PushPull@0*c0@int_slack@0
          + 211.200000000000 PushPull@0*c0@int_slack@1
          + 52.800000000000 c0@int_slack@0^2
          + 211.200000000000 c0@int_slack@0*c0@int_slack@1
          + 211.200000000000 c0@int_slack@1^2 ]/2 + 237.600000000000
    Subject To

    Bounds
     0 <= Wheat@0 <= 1
     0 <= Soybeans@0 <= 1
     0 <= Maize@0 <= 1
     0 <= PushPull@0 <= 1
     0 <= c0@int_slack@0 <= 1
     0 <= c0@int_slack@1 <= 1

    Binaries
     Wheat@0 Soybeans@0 Maize@0 PushPull@0 c0@int_slack@0 c0@int_slack@1
    End

## Solving the Crop-Yield Problem using a Quantum Computer

There are three ways to _run_ a quantum algorithm using Qiskit:
1. on a simulator locally on your own machine
2. on a simulator hosted in the cloud by IBM
3. on an actual quantum computer accessible through IBM Quantum.

All of these are called backends. In all cases, the _backend_ can easily be swapped for another as long as the simulator or device has appropriate resources (number of qubits etc.). In the code below, we show how to access different backends. We demonstrate this using the local Aer QASM simulator from Qiskit. The Aer QASM simulator models the physical properties of a real quantum computer so you, researchers, and developers can test their quantum computing code and algorithms before running on real devices.

```python
# We will use the Aer provided QASM simulator
backend = Aer.get_backend("qasm_simulator")

# Given we are using a simulator, we will fix the algorithm seed to ensure our results are reproducible
algorithm_globals.random_seed = 271828
```

We would like to compare our quantum solution to that obtained classically. Secondly, we also want to try different algorithms. The following three subsections show how these different methods for solving the Crop-Yield problem are implemented in Qiskit. The two algorithms used are the [_Quantum Approximate Optimization Algorithm_](https://qiskit.org/documentation/stubs/qiskit.algorithms.QAOA.html?highlight=qaoa#qiskit.algorithms.QAOA) `QAOA` and the [_Variational Quantum Eigensolver_](https://qiskit.org/documentation/stubs/qiskit.algorithms.VQE.html?highlight=vqe#qiskit.algorithms.VQE) `VQE`.

Both of these algorithms are hybrid, in that they use a classical _optimizer_ to alter parameters that affect the quantum computation. The VQE algorithm is used to find the lowest eigenvalue of a matrix, which can describe a system to optimize. The QAOA also finds the lowest eigenvalue, but achieves this is a different way to VQE. Both are very popular algorithms, with varying applications and strengths.

### Classical Solution

The classical solution to the crop-yield problem can easily be found using Numpy and Qiskit. The QUBO problem can be solved by finding the minimum eigenvalue of its underlying matrix representation. Fortunately, we don't have to know what this matrix looks like. We only need to pass it to a `MinimumEigensolver` and `MinimumEigenOptimizer`.

The optimizer translates the provided problem into a parameterised representation which is then passed to the solver. By optimizing the paramters, the solver will eventually give the minimum eigenvalue for the parameterized representation and thus the solution to the original problem. Here we use a classical solver from NumPy, the `NumPyMinimumEigensolver`.

```python
def get_classical_solution_for(quadprog: QuadraticProgram):
    # Create solver
    solver = NumPyMinimumEigensolver()

    # Create optimizer for solver
    optimizer = MinimumEigenOptimizer(solver)

    # Return result from optimizer
    return optimizer.solve(quadprog)
```

If we execute the classical method for our crop-yield problem, we get a valid solution that maximises the yield.

```python
# Get classical result
classical_result = get_classical_solution_for(cropyield)

# Format and print result
print("Solution found using the classical method:\n")
print(f"Maximum crop-yield is {classical_result.fval} tons")
print(f"Crops used are: ")

_crops = [v.name for v in cropyield.variables]
for cropIndex, cropHectares in enumerate(classical_result.x):
    print(f"\t{cropHectares} ha of {_crops[cropIndex]}")
```

    Solution found using the classical method:

    Maximum crop-yield is 19.0 tons
    Crops used are:
    	1.0 ha of Wheat
    	0.0 ha of Soybeans
    	1.0 ha of Maize
    	1.0 ha of PushPull

### QAOA Solution

To solve our problem using QAOA, we need only replace the classical_solver with a `QAOA` class instance. Now that we are running a quantum algorithm, we need to tell the solver where to execute the quantum component. We use a `QuantumInstance` to store the backend information. The QAOA is an iterative algorithm, and thus is run multiple times with different internal parameters. The parameters are tuned classically during the optimization step of the computation. To determine how many iterations there are, we define a callback function that runs for each iteration and stores the number of evaluations thus far. At the end of our algorithm execution, we return the result and the number of iterations.

```python
def get_QAOA_solution_for(
    quadprog: QuadraticProgram,
    quantumInstance: QuantumInstance,
):
    _eval_count = 0

    def callback(eval_count, parameters, mean, std):
        nonlocal _eval_count
        _eval_count = eval_count

    # Create solver
    solver = QAOA(
        quantum_instance=quantumInstance,
        callback=callback,
    )

    # Create optimizer for solver
    optimizer = MinimumEigenOptimizer(solver)

    # Get result from optimizer
    result = optimizer.solve(quadprog)
    return result, _eval_count
```

If we execute the QAOA method for our crop-yield problem, we get the same result as the classical method, showing that 1) the quantum solution is correct and 2) that you now know how to use a quantum algorithm! 🌟

```python
# Create a QuantumInstance
simulator_instance = QuantumInstance(
    backend=backend,
    seed_simulator=algorithm_globals.random_seed,
    seed_transpiler=algorithm_globals.random_seed,
)

# Get QAOA result
qaoa_result, qaoa_eval_count = get_QAOA_solution_for(cropyield, simulator_instance)

# Format and print result
print("Solution found using the QAOA method:\n")
print(f"Maximum crop-yield is {qaoa_result.fval} tons")
print(f"Crops used are: ")
for cropHectares, cropName in zip(qaoa_result.x, qaoa_result.variable_names):
    print(f"\t{cropHectares} ha of {cropName}")

print(f"\nThe solution was found within {qaoa_eval_count} evaluations of QAOA.")
```

    Solution found using the QAOA method:

    Maximum crop-yield is 19.0 tons
    Crops used are:
    	1.0 ha of Wheat
    	0.0 ha of Soybeans
    	1.0 ha of Maize
    	1.0 ha of PushPull

    The solution was found within 3 evaluations of QAOA.

### VQE Solution

The `VQE` algorithm works in a very similar way to the `QAOA`. Not only in a mathematical modelling and algorithmic perspective, but also programmatically. There is a quantum solver and a classical optimizer. The `VQE` instance is also iterative, and so we can measure how many iterations are needed to find a solution to the Crop-Yield problem.

```python
def get_VQE_solution_for(
    quadprog: QuadraticProgram,
    quantumInstance: QuantumInstance,
):
    _eval_count = 0

    def callback(eval_count, parameters, mean, std):
        nonlocal _eval_count
        _eval_count = eval_count

    # Create solver and optimizer
    solver = VQE(quantum_instance=quantumInstance, callback=callback)

    # Create optimizer for solver
    optimizer = MinimumEigenOptimizer(solver)

    # Get result from optimizer
    result = optimizer.solve(quadprog)
    return result, _eval_count
```

And we should get the exact same answer as before.

```python
# Create a QuantumInstance
simulator_instance = QuantumInstance(
    backend=backend,
    seed_simulator=algorithm_globals.random_seed,
    seed_transpiler=algorithm_globals.random_seed,
)

# Get VQE result
vqe_result, vqe_eval_count = get_VQE_solution_for(cropyield, simulator_instance)

# Format and print result
print("Solution found using the VQE method:\n")
print(f"Maximum crop-yield is {vqe_result.fval} tons")
print(f"Crops used are: ")
for cropHectares, cropName in zip(vqe_result.x, vqe_result.variable_names):
    print(f"\t{cropHectares} ha of {cropName}")

print(f"\nThe solution was found within {vqe_eval_count} evaluations of VQE")
```

    Solution found using the VQE method:

    Maximum crop-yield is 19.0 tons
    Crops used are:
    	1.0 ha of Wheat
    	0.0 ha of Soybeans
    	1.0 ha of Maize
    	1.0 ha of PushPull

    The solution was found within 25 evaluations of VQE

### Exercise 1c: Classical and Quantum Computational Results

From the above computations you received six results, the maximum crop-yield and the number of evaluations for three different methods. The maximum yield values should be the same.

Run the code cell below to see if the maximum yields you computed are correct.

```python
from qc_grader import grade_ex1c

max_yield_qaoa = qaoa_result.fval
max_yield_vqe = vqe_result.fval

grade_ex1c(tonnage_qaoa=max_yield_qaoa, tonnage_vqe=max_yield_vqe)
```

    Submitting your answer for ex1/partC. Please wait...
    Congratulations 🎉! Your answer is correct and has been submitted.

_You could always verify your result with the classical method, though this is only possible here given the size of the problem. Larger problems become more difficult to verify._

## Simulating a Real Quantum Computer for the Crop-Yield Problem

Sometimes one would want to _simulate_ a real quantum computer to see how the actual hardware may impact the performance of the algorithm. All quantum computers have an underlying architecture, different noise characeristics, and error rates. These three aspects impact how well the algorithm can perform on a given deivce. To test the impact a given quantum computer has on the QAOA instance, we can utilize a _fake_ instance of the device in Qiskit to tell our simulator what parameters to use. In this example we will be simulating `ibmq_johannesburg`, a device named after the city of Johannesburg in South Africa.

```python
fake_device = Fake.FakeJohannesburg()
```

We can inspect what this device _looks_ like using the Qiskit Jupyter tools, shown below. You do not need to know about this structure to execute quantum programs on a device, but it is useful to visualize the parameters.

```python
import qiskit.tools.jupyter

fake_device
```

    VBox(children=(HTML(value="<h1 style='color:#ffffff;background-color:#000000;padding-top: 1%;padding-bottom: 1…

    <FakeJohannesburg('fake_johannesburg')>

The three aforementioned components of a quantum computer are represented as a noise model, coupling map, and the basis gate set. The noise model is a representation of how the noise and errors in the computer behave. The coupling map and basis gate set are core to the architecture of the device. The coupling map represents how the physical qubits can interact whereas the basis gate set is analogous to the set of fundamental computatonal instructions we can use. You can see the coupling map in the above widget as the lines connecting each qubit in the architecture diagram.

To simulate `ibmq_johannesburg`, we must pass these three components to our Aer simulator.

```python
# Create the noise model, which contains the basis gate set
noise_model = NoiseModel.from_backend(fake_device)

# Get the coupling map
coupling_map = fake_device.configuration().coupling_map
```

Next we create a new `QuantumInstance` with these parameters

```python
fake_instance = QuantumInstance(
    backend=backend,
    basis_gates=noise_model.basis_gates,
    coupling_map=coupling_map,
    noise_model=noise_model,
    seed_simulator=algorithm_globals.random_seed,
    seed_transpiler=algorithm_globals.random_seed,
)
```

We can then execute the `QAOA` from before on this new _fake_ quantum device.

```python
# Get QAOA result
qaoa_result, qaoa_eval_count = get_QAOA_solution_for(cropyield, fake_instance)

# Format and print result
print("Solution found using the QAOA method:\n")
print(f"Maximum crop-yield is {qaoa_result.fval} tons")
print(f"Crops used are: ")
for cropHectares, cropName in zip(qaoa_result.x, qaoa_result.variable_names):
    print(f"\t{cropHectares} ha of {cropName}")

print(f"\nThe solution was found within {qaoa_eval_count} evaluations of QAOA.")
```

    Solution found using the QAOA method:

    Maximum crop-yield is 19.0 tons
    Crops used are:
    	1.0 ha of Wheat
    	0.0 ha of Soybeans
    	1.0 ha of Maize
    	1.0 ha of PushPull

    The solution was found within 3 evaluations of QAOA.

### Scaling of the Quantum Solution vs Classical

When we created our quadratic program for the crop-yield problem, we saw that the Ising Model required 6 qubits. We had constrained our problem such that we could only plant up to 1 hectare per crop. However, we could change the model so that we can plot 3 hectares per crop, upto our maximum available farm area of 3 hectares.

How many qubits would this ising model require?

---

Furthermore, what if we had more land to farm? We know that this problem is NP-Hard and thus classical solutions are mostly found using heuristics. This is the core reason why quantum computers are promising to solve these kinds of problems. But what are the resource requirements for the quantum solution, with a larger farm and crops that can be planted in more hectares?

To illustrate this, we've provided a function that returns the number of qubits required by the underlying Ising Model for the Crop-Yield Problem. We then see the estimated number of qubits needed for different problem parameters. Feel free to modify the variables being used to see how the qubit resource requirements change.

```python
ls
```

    crop_empty.svg               lab1.ipynb
    crop_maize.svg               lab1_solutions.ipynb
    crop_pushpull.svg            maize.svg
    crop_soybeans.svg            pushpull.svg
    crop_wheat.svg               quadratic_example.svg
    ex1_utils.py                 qubo_problem_graphical.svg
    farm_intercrop_pushpull.svg  qubo_problem_graphical_variables.svg
    farm_intercrop.svg           soybeans.svg
    farm_mono.svg                wheat.svg
    index.md

```python
import ex1_utils
from ex1_utils import estimate_number_of_qubits_required_for
```

    Traceback (most recent call last):
      File "<ipython-input-34-041cd1a2605a>", line 1, in <module>
        import ex1_utils
    ModuleNotFoundError: No module named 'ex1_utils'

    Use %tb to get the full traceback.

<button class='button iqx-button'>Search for solution online</button>

```python
import numpy as np
from qiskit_optimization import QuadraticProgram
from qiskit_optimization.converters import QuadraticProgramToQubo

def estimate_number_of_qubits_required_for(
    max_hectares_per_crop=1,
    hectares_available=3,
):
    return 4 * np.ceil(np.log2(max_hectares_per_crop + 1)) + np.ceil(
        np.log2(hectares_available + 1)
    )
```

```python
#from ex1_utils import estimate_number_of_qubits_required_for

# Our new problem parameters
hectares_available = 10
max_hectares_per_crop = 10

# Retrieving the number of qubits required
number_of_qubits_required = estimate_number_of_qubits_required_for(
    max_hectares_per_crop=max_hectares_per_crop, hectares_available=hectares_available
)

print(
    f"Optimizing a {hectares_available} ha farm with each crop taking up to {max_hectares_per_crop} ha each,",
    f"the computation is estimated to require {number_of_qubits_required} qubits.",
)
```

    Optimizing a 10 ha farm with each crop taking up to 10 ha each, the computation is estimated to require 20.0 qubits.

The number of qubits required is related to the constraints in the quadratic program and how the integer variables are converted to binary variables. In fact, the scaling of the number of qubits, as a function of the hectares available, is logarithmic in nature; owing to this conversion.

## Running on real quantum hardware

To use the IBM Quantum platform is easy. First you need to load the account you enabled in the week 0 content. If you didn't complete this, follow this [quick guide](https://quantum-computing.ibm.com/lab/docs/iql/manage/account/ibmq) on connecting your IBM Quantum account with Qiskit in python and Jupyter.

```python
IBMQ.load_account()
```

    <AccountProvider for IBMQ(hub='ibm-q', group='open', project='main')>

IBM Quantum backends are accessed through a provider, which manages the devices to which you have access. Here we're going to use the open access provider that everyone is assigned to when they register an IBM Quantum account.

```python
provider = IBMQ.get_provider(hub="ibm-q", group="open", project="main")
```

To retrieve a backend from the provider, one needs only request it by name. For example, we can request `ibmq_santiago` as follows.

```python
backend_santiago = provider.get_backend("ibmq_santiago")
```

We can also list all available backends as follows. Notice that this also includes the IBM Quantum hosted high-performance simulators.

```python
for _backend in provider.backends():
    print(_backend.name())
```

    ibmq_qasm_simulator
    ibmq_armonk
    ibmq_santiago
    ibmq_bogota
    ibmq_lima
    ibmq_belem
    ibmq_quito
    simulator_statevector
    simulator_mps
    simulator_extended_stabilizer
    simulator_stabilizer
    ibmq_manila

Qiskit provides visual tools to view backend information in a jupyter notebook. To accomplish this, one needs to import the `jupyter` submodule and call the appropriate _magic comment_. With `qiskit_backend_overview` you can view all devices accessible by the current IBMQ account. Notice how it does not include simulators. Furthermore, you should see that all devices available through the _open group_ have at most 5 qubits. This is a problem for solving the crop-yield problem we created earlier, as we showed it requires 6 qubits.

To demonstrate how a real quantum device is used, a smaller `QuadraticProgram` is provided, requiring a maximum of 4 qubits.

```python
%qiskit_backend_overview
```

    VBox(children=(HTML(value="<h2 style ='color:#ffffff; background-color:#000000;padding-top: 1%; padding-bottom…

If you want access to larger and more sophisticated quantum computers through IBM, see if your university or company is part of the [IBM Quantum Network](https://www.ibm.com/quantum-computing/network/members/). Researchers are institutions that are part of the [African Research Universities Alliance (AURA)](https://arua.org.za/) can also apply for access through the University of the Witwatersrand, in South Africa; which is a member of the IBM Quantum Network. If you are a researcher, you can also apply for access through the [IBM Quantum Researchers Program](https://www.ibm.com/quantum-computing/researchers-program/). If you're a student at a highschool or university, you can ask your teachers or lecturers to apply for access through the [IBM Quantum Educators Program](https://www.ibm.com/quantum-computing/educators-program/).

Given that we have imported `qiskit.tools.jupyter`, Jupyter will now display a helpful widget when an IBMQ backend is displayed. We do not have to use a _magic comment_ here as the jupyter submodule defines how some variables are displayed in a jupyter notebook, without requiring a _magic comment_.

```python
backend_santiago
```

    VBox(children=(HTML(value="<h1 style='color:#ffffff;background-color:#000000;padding-top: 1%;padding-bottom: 1…

    <IBMQBackend('ibmq_santiago') from IBMQ(hub='ibm-q', group='open', project='main')>

The IBMQ provider also allows us to select a device by how busy it is. `least_busy` will select the device, from a given list. which has the shortest queue: i.e. which quantum computer is predicted to be available the soonest. Below we filter the backends to only those that can execute our small crop-yield problem: those with at least 4 qubits that are not simulators.

```python
# Create a list of backends with >= 4 qubits that aren't simulators
backends_with_at_least_4_qubits = [
    _backend
    for _backend in provider.backends(simulator=False, operational=True)
    if _backend.configuration().num_qubits >= 4
]
```

```python
# Find the least busy backend from the list of selected backends
from qiskit.providers.ibmq import least_busy

backend_least_busy = least_busy(backends_with_at_least_4_qubits)
backend_least_busy
```

    VBox(children=(HTML(value="<h1 style='color:#ffffff;background-color:#000000;padding-top: 1%;padding-bottom: 1…

    <IBMQBackend('ibmq_belem') from IBMQ(hub='ibm-q', group='open', project='main')>

We can create a new `QuantumInstance` object to contain our _least busy_ backend, similar to how we created one to manage our simulator. For real devices, there is an extra parameter we can set: `shots`. The output of a quantum computing algorithm is probabilistic. Therefore, we must execute the quantum computation multiple times, sampling the outputs to estimate their probabilities. The number of `shots` is the number of executions of the quantum computation. Here we set out `QuantumInstance` to use the least busy backend with 2048 shots.

```python
quantum_instance_least_busy = QuantumInstance(backend_least_busy, shots=2048)
```

The VQE algorithm and QAOA are iterative, meaning that they incorporate a classical-quantum loop which repeats certain computations, _hopefully_ converging to a valid solution. In each iteration, or evaluation, the quantum backend will execute the quantum operations 2048 times. Each shot is quite fast, so we do not have to worry about a significant increase in processing time by using more shots.

----
We now define our small crop-yield problem, which only requires 4 qubits. In this example, only Wheat and Maize are used. The model is altered to illustrate the impact of growing too much of a single crop, with the yield decreasing as the number of hectares of a single crop is increased. However, utilizing both wheat and maize increases yield, showing the benefits of intercropping.

**NB:** The maximum number of hectares available is 4, but given that the model would never exceed this limit, the linear constraint defining the maximum number of hectares is not included. This reduces the number of qubits required from 6 to 4.

```python
# Create a small crop-yield example quadratic program
cropyield_small = QuadraticProgram(name="Small Crop-Yield")

# Add two variables, indicating whether we grow 0, 1, or 2 hectares for two different crops
cropyield_small.integer_var(lowerbound=0, upperbound=2, name="Wheat")
cropyield_small.integer_var(lowerbound=0, upperbound=2, name="Maize")

# Add the objective function defining the yield in tonnes
cropyield_small.maximize(
    linear={"Wheat": 3, "Maize": 3},
    quadratic={("Maize", "Wheat"): 1, ("Maize", "Maize"): -2, ("Wheat", "Wheat"): -2},
)

# This linear constraint is not used as the model never reaches this. This is because the
# sum of the upperbounds on both variables is 4 already. If this constraint is applied, the
# model would require 6 qubits instead of 4.
# cropyield_small.linear_constraint(linear={"Wheat": 1, "Maize": 1}, sense="<=", rhs=4)

print(cropyield_small)
```

    \ This file has been generated by DOcplex
    \ ENCODING=ISO-8859-1
    \Problem name: Small Crop-Yield

    Maximize
     obj: 3 Wheat + 3 Maize + [ - 4 Wheat^2 + 2 Wheat*Maize - 4 Maize^2 ]/2
    Subject To

    Bounds
           Wheat <= 2
           Maize <= 2

    Generals
     Wheat Maize
    End

Here we verify that our small crop-yield problem requires only 4 qubits.

```python
# Estimate the number of qubits required
ising_operations_small, _ = (
    QuadraticProgramToQubo()
    .convert(
        cropyield_small,
    )
    .to_ising()
)
print(f"Number of qubits required is {ising_operations_small.num_qubits}")
```

    Number of qubits required is 4

Now that we know the problem can be run on our chosen device, we can execute the VQE algorithm to solve the problem.

```python
result_least_busy = get_VQE_solution_for(cropyield_small,quantum_instance_least_busy)
```

    /opt/conda/lib/python3.8/site-packages/qiskit/utils/run_circuits.py:695: UserWarning: max_credits is not a recognized runtime option and may be ignored by the backend.
      return backend.run(circuits, **run_kwargs)

```python
result_least_busy
```

    (optimal function value: 3.0
     optimal value: [1. 1.]
     status: SUCCESS,
     17)

```python
job_real =backend_least_busy.jobs()[0]

print(f"VQE job created at {job_real.creation_date()} and has a job id of {job_real.job_id()}")
```

    VQE job created at 2021-09-09 15:32:28.398000+00:00 and has a job id of 613a290cc32c4f2f26210acf

```python

from qc_grader import grade_ex1d

job_id = '613a290cc32c4f2f26210acf'

grade_ex1d(job_id)
```

    Submitting your answer for ex1/partD. Please wait...
    Congratulations 🎉! Your answer is correct and has been submitted.

## References

[1] A. A. Nel, ‘Crop rotation in the summer rainfall area of South Africa’, South African Journal of Plant and Soil, vol. 22, no. 4, pp. 274–278, Jan. 2005, doi: 10.1080/02571862.2005.10634721.

[2] H. Ritchie and M. Roser, ‘Crop yields’, Our World in Data, 2013, [Online]. Available: https://ourworldindata.org/crop-yields.

[3] G. Brion, ‘Controlling Pests with Plants: The power of intercropping’, UVM Food Feed, Jan. 09, 2014. https://learn.uvm.edu/foodsystemsblog/2014/01/09/controlling-pests-with-plants-the-power-of-intercropping/ (accessed Feb. 15, 2021).

[4] N. O. Ogot, J. O. Pittchar, C. A. O. Midega, and Z. R. Khan, ‘Attributes of push-pull technology in enhancing food and nutrition security’, African Journal of Agriculture and Food Security, vol. 6, pp. 229–242, Mar. 2018.

```python
import qiskit.tools.jupyter

%qiskit_version_table
```


# Simulation in air

# Purpose
Check the equation of motion

# Methodology
Conduct some simulations to see that the object is behaving in air.

# Setup

```python
# %load imports.py
## Local packages:

%matplotlib inline
%load_ext autoreload
%autoreload 2
%config Completer.use_jedi = False  ## (To fix autocomplete)

## External packages:
import pandas as pd
pd.options.display.max_rows = 999
pd.options.display.max_columns = 999
pd.set_option("display.max_columns", None)

import numpy as np
import os
import matplotlib.pyplot as plt
#if os.name == 'nt':
#    plt.style.use('presentation.mplstyle')  # Windows

import plotly.express as px
import plotly.graph_objects as go

import seaborn as sns
import sympy as sp
from sympy.physics.mechanics import (dynamicsymbols, ReferenceFrame,
                                      Particle, Point)
from sympy.physics.vector.printing import vpprint, vlatex
from IPython.display import display, Math, Latex
from src.substitute_dynamic_symbols import run, lambdify

import pyro

import sklearn
import pykalman
from statsmodels.sandbox.regression.predstd import wls_prediction_std
import statsmodels.api as sm

from scipy.integrate import solve_ivp

## Local packages:
from src.data import mdl

from src.symbols import *
from src.parameters import *
import src.symbols as symbols
from src import prime_system
from src.models import regression
from src.visualization.plot import track_plot
from src.equation import Equation

## Load models:
# (Uncomment these for faster loading):
#import src.models.vmm_abkowitz  as vmm_abkowitz
import src.models.vmm_air as vmm_air
```

```python
Math(vlatex(vmm_air.X_eq))
```

```python
Math(vlatex(vmm_air.Y_eq))
```

```python
Math(vlatex(vmm_air.N_eq))
```

```python
ship_parameters = pd.Series({

        'L' : 5,
        'B' : 1,
        'x_G' : 0,  # motions are expressed at CG
        'm' : 2,
        'I_z': 2,
    })

parameters = {}

```

```python
t_ = np.linspace(0,10,1000)
df = pd.DataFrame(index=t_)
f_ext_x_ = 2
a_x_ = f_ext_x_/ship_parameters.m

df['u'] = a_x_*t_
df['v'] = 0
df['r'] = 0
df['x0'] = a_x_*t_**2/2
df['y0'] = 0
df['psi'] = 0
df['beta'] = -np.arctan2(df['v'],df['u'])
df['f_ext_x']=f_ext_x_
df['f_ext_y']=0
df['m_ext_z']=0

result_air = vmm_air.simulator.simulate(df_=df,
                                        parameters=parameters,
                                        ship_parameters=ship_parameters,
                                        control_keys=['f_ext_x','f_ext_y','m_ext_z'])
result_air.plot_compare()
```

```python

```

```python
t_ = np.linspace(0,10,1000)
df = pd.DataFrame(index=t_)
V_=6
r_ = 2*np.pi/t_[-1]

f_ext = ship_parameters.m*r_*V_

df['beta'] = np.deg2rad(30)
df['u'] = V_*np.cos(df['beta'])
df['v'] = -V_*np.sin(df['beta'])
df['r'] = r_

R_=V_*t_[-1]/(2*np.pi)

phi_ = r_*t_
df['psi'] = phi_ + df['beta']
df['x0'] = R_*np.sin(phi_)
df['y0'] = R_-R_*np.cos(phi_)

df['f_ext_x']=f_ext*np.sin(df['beta'])
df['f_ext_y']=f_ext*np.cos(df['beta'])
df['m_ext_z']=0

result_air = vmm_air.simulator.simulate(df_=df,
                                        parameters=parameters,
                                        ship_parameters=ship_parameters,
                                        control_keys=['f_ext_x','f_ext_y','m_ext_z'])
result_air.plot_compare()
```

```python

```


# Tutorial 1: Active inference from scratch

[](https://colab.research.google.com/github/infer-actively/pymdp/blob/master/docs/notebooks/active_inference_from_scratch.ipynb)

*Author: Conor Heins*

This tutorial guides you through the construction of an active inference agent "from scratch" in a simple grid-world environment. This tutorial is designed to walk you through the various mathematical operations required for running active inference in discrete state spaces, as well as introduce you to the way discrete probability distributions are represented in `pymdp` (through the use of numpy vectors and matrices). We also show you how to compute the expected free energy in terms of risk and ambiguity, and how to use it to plan sequences of actions.

By the end of this tutorial, you should be able to write simple active inference models from scratch using the main components of any POMDP generative model: the `A`, `B`, `C`, and `D` arrays, as well as be able to write the active inference loop for an agent engaged in a perception-action exchange with its environment.

*Note*: When running this notebook in Google Colab, you may have to run `!pip install inferactively-pymdp` at the top of the notebook, before you can `import pymdp`. That cell is left commented out below, in case you are running this notebook from Google Colab.

```python
# ! pip install inferactively-pymdp
```

### Imports

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
```

### Define some auxiliary functions

Here are some plotting functions that will come in handy throughout the tutorial.

```python
def plot_likelihood(matrix, xlabels = list(range(9)), ylabels = list(range(9)), title_str = "Likelihood distribution (A)"):
    """
    Plots a 2-D likelihood matrix as a heatmap
    """

    if not np.isclose(matrix.sum(axis=0), 1.0).all():
      raise ValueError("Distribution not column-normalized! Please normalize (ensure matrix.sum(axis=0) == 1.0 for all columns)")

    fig = plt.figure(figsize = (6,6))
    ax = sns.heatmap(matrix, xticklabels = xlabels, yticklabels = ylabels, cmap = 'gray', cbar = False, vmin = 0.0, vmax = 1.0)
    plt.title(title_str)
    plt.show()

def plot_grid(grid_locations, num_x = 3, num_y = 3 ):
    """
    Plots the spatial coordinates of GridWorld as a heatmap, with each (X, Y) coordinate
    labeled with its linear index (its `state id`)
    """

    grid_heatmap = np.zeros((num_x, num_y))
    for linear_idx, location in enumerate(grid_locations):
      y, x = location
      grid_heatmap[y, x] = linear_idx
    sns.set(font_scale=1.5)
    sns.heatmap(grid_heatmap, annot=True, cbar = False, fmt='.0f', cmap='crest')

def plot_point_on_grid(state_vector, grid_locations):
    """
    Plots the current location of the agent on the grid world
    """
    state_index = np.where(state_vector)[0][0]
    y, x = grid_locations[state_index]
    grid_heatmap = np.zeros((3,3))
    grid_heatmap[y,x] = 1.0
    sns.heatmap(grid_heatmap, cbar = False, fmt='.0f')

def plot_beliefs(belief_dist, title_str=""):
    """
    Plot a categorical distribution or belief distribution, stored in the 1-D numpy vector `belief_dist`
    """

    if not np.isclose(belief_dist.sum(), 1.0):
      raise ValueError("Distribution not normalized! Please normalize")

    plt.grid(zorder=0)
    plt.bar(range(belief_dist.shape[0]), belief_dist, color='r', zorder=3)
    plt.xticks(range(belief_dist.shape[0]))
    plt.title(title_str)
    plt.show()

```

## **The Basics: categorical distributions**

Let's start by creating a simple categorical distribution $P(x)$. We will represent this distribution numerically with a 1-D `numpy` array (i.e. a `numpy.ndarray` where `ndim == 1`). The discrete entries of this vector can be thought of as the probabilities of seeing each of the $K$ alternative outcomes (the support of the distribution), if we were to sample once from the distribution. We can write the distribution as a vector of probabilities, follows:

$$
P(x) = \begin{bmatrix} P(x = 0) \\\ P(x = 1)\\\ \vdots \\\ P(x = K) \end{bmatrix}
$$

We can use `numpy` and the `norm_dist` function from `utils` library of `pymdp` to quickly create a random categorical distribution.

```python
from pymdp import utils
```

```python
my_categorical = np.random.rand(3)
# my_categorical = np.array([0.5, 0.3, 0.8]) # could also just write in your own numbers
my_categorical = utils.norm_dist(my_categorical) # normalizes the distribution so it integrates to 1.0

print(my_categorical.reshape(-1,1)) # we reshape it to display it like a column vector
print(f'Integral of the distribution: {round(my_categorical.sum(), 2)}')
```

    [[0.11879564]
     [0.37417953]
     [0.50702484]]
    Integral of the distribution: 1.0

 We can sample a discrete outcome from this distribution using the `sample()` function from the `utils` module

```python
sampled_outcome = utils.sample(my_categorical)
print(f'Sampled outcome: {sampled_outcome}')
```

    Sampled outcome: 1

Now plot the beliefs using our `plot_beliefs()` function that we defined in the beginning.

```python
plot_beliefs(my_categorical, title_str = "A random (unconditional) Categorical distribution")
```

Now let's move onto _conditional_ categorical distributions or likelihoods, i.e. how we represent the distribution of one discrete random variable X, conditioned on the settings of another discrete random variable Y.

We would write these mathematically as:

$$ \begin{align}
P(X | Y)
\end{align}
$$

And can represent them using 2-D `numpy.ndarrays` i.e. matrices

```python
# initialize it with random numbers
p_x_given_y = np.random.rand(3, 4)
print(p_x_given_y.round(3))
```

    [[0.051 0.178 0.847 0.826]
     [0.03  0.971 0.328 0.378]
     [0.071 0.506 0.524 0.899]]

```python
# normalize it
p_x_given_y = utils.norm_dist(p_x_given_y)
print(p_x_given_y.round(3))
```

    [[0.336 0.108 0.499 0.393]
     [0.196 0.587 0.193 0.18 ]
     [0.468 0.306 0.308 0.427]]

Now print the first column (i.e. $P(X | Y = 0)$) and show that it's a proper categorical distribution

```python
print(p_x_given_y[:,0].reshape(-1,1))
print(f'Integral of P(X|Y=0): {p_x_given_y[:,0].sum()}')
```

    [[0.33580292]
     [0.19582781]
     [0.46836927]]
    Integral of P(X|Y=0): 1.0

So column `i` of the matrix `p_x_given_y` represents the conditional probability of $X$, given the `i`-th level of the random variable $Y$, i.e. $P(X | Y = i)$

Next: taking expectations of random variables using matrix-vector products

```python
""" Create a P(Y) and P(X|Y) using the same numbers from the slides """

p_y = np.array([0.75, 0.25]) # this is already normalized - you don't need to `utils.norm_dist()` it!

# the columns here are already normalized - you don't need to `utils.norm_dist()` it!
p_x_given_y = np.array([[0.6, 0.5],
                        [0.15, 0.41],
                        [0.25, 0.09]])

print(p_y.round(3).reshape(-1,1))
print(p_x_given_y.round(3))
```

    [[0.75]
     [0.25]]
    [[0.6  0.5 ]
     [0.15 0.41]
     [0.25 0.09]]

Calculate expected value of $X$, given our current _belief_ about $Y$, i.e. $P(Y)$ using a simple matrix-vector product, of the form $\mathbf{A}\mathbf{x}$.

```python
""" Calculate the expectation using numpy's dot product functionality """

# first version of the dot product (using the method of a numpy array)
E_x_wrt_y = p_x_given_y.dot(p_y)

# second version of the dot product (using the function np.dot with two arguments)
# E_x_wrt_y = np.dot(p_x_given_y, p_y)
```

Now print it and verify the result is a proper categorical distribution (integrates to 1.0)

```python
print(E_x_wrt_y)
print(f'Integral: {E_x_wrt_y.sum().round(3)}')
```

    [0.575 0.215 0.21 ]
    Integral: 1.0

## **A simple environment: Grid-world**
Now that we understand categorical distributions and how to take conditional expectations of random variables, with categorical conditional and prior distributions, let's move onto a worked example of Active Inference, so-called 'Grid-World'.

Before we write down the generative model $P(o,s)$ of an active inference agent who will navigate Grid-World, let's start by establishing what exactly this environment is and how we're going to represent it mathematically.

```python
import itertools

""" Create  the grid locations in the form of a list of (Y, X) tuples -- HINT: use itertools """
grid_locations = list(itertools.product(range(3), repeat = 2))
print(grid_locations)
```

    [(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2), (2, 0), (2, 1), (2, 2)]

```python
plot_grid(grid_locations)
```

## **Building the generative model**: $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$, and $\mathbf{D}$

### 1. The **A** matrix or $P(o\mid s)$.

---

The generative model's "prior beliefs" about how hidden states relate to observations

```python
""" Create variables for the storing the dimensionalities of the hidden states and the observations """

n_states = len(grid_locations)
n_observations = len(grid_locations)

print(f'Dimensionality of hidden states: {n_states}')
print(f'Dimensionality of observations: {n_observations}')
```

    Dimensionality of hidden states: 9
    Dimensionality of observations: 9

```python
""" Create the A matrix  """

A = np.zeros( (n_states, n_observations) )
```

```python
""" Create an umambiguous or 'noise-less' mapping between hidden states and observations """

np.fill_diagonal(A, 1.0)

# alternative:
# A = np.eye(n_observations, n_states)
```

Likelihood matrices show relationships between indices of the state space, which are indexed linearly. Remember the (Y, X) coordinates of the grid world don't necessarily neatly map onto to the locations in the matrix here, which has 'unwrapped' the Grid World's (Y, X) locations into columns/rows.

```python
plot_likelihood(A, title_str = "A matrix or $P(o|s)$")
```

How could we introduce uncertainty into this A matrix? I.e. how do we introduce ambiguity or uncertainty into the agent's model of the world?

```python
""" Remind yourself of the mapping between linear indices (0 through 8) and grid locations (Y, X) """
plot_grid(grid_locations)
```

```python
A_noisy = A.copy()

# this line says: the probability of seeing yourself in location 0, given you're in location 0, is 1/3, AKA P(o == 0 | s == 0) = 0.3333....
A_noisy[0,0] = 1 / 3.0 # corresponds to location (0,0)

# this line says: the probability of seeing yourself in location 1, given you're in location 0, is 1/3, AKA P(o == 1 | s == 0) = 0.3333....
A_noisy[1,0] = 1 / 3.0 # corresponds to one step to the right from (0, 1)

# this line says: the probability of seeing yourself in location 3, given you're in location 0, is 1/3, AKA P(o == 3 | s == 0) = 0.3333....
A_noisy[3,0] = 1 / 3.0 # corresponds to one step down from (1, 0)
```

```python
plot_likelihood(A_noisy, title_str = 'modified A matrix where location (0,0) is "blurry"')
```

```python
""" Let's make ake one grid location "ambiguous" in the sense that it could be easily confused with neighbouring locations """
my_A_noisy = A_noisy.copy()

# locations 3 and 7 are the nearest neighbours to location 6
my_A_noisy[3,6] = 1.0 / 3.0
my_A_noisy[6,6] = 1.0 / 3.0
my_A_noisy[7,6] = 1.0 / 3.0

# Alternatively: you could have the probability spread among locations 3, 4, 6, and 7. This is basically saying, that whole lower-left corner of grid-world is blurry, if you're in location 6
# Remember to make sure the A matrix is column normalized. So if you do it this way, with the probabilities spread among 4 perceived locations, then you'll have to make sure the probabilities sum to 1.0
# my_A_noisy[3,6] = 1.0 / 4.0
# my_A_noisy[4,6] = 1.0 / 4.0
# my_A_noisy[6,6] = 1.0 / 4.0
# my_A_noisy[7,6] = 1.0 / 4.0
```

Now plot the new A matrix, that now has two "blurry" locations

```python
plot_likelihood(my_A_noisy, title_str = "Noisy A matrix now with TWO ambiguous locations")
```

### 2. The **B** matrix or $P(s_{t}\mid s_{t-1}, u_{t-1})$.

---

The generative model's "prior beliefs" about (controllable) transitions between hidden states over time. Namely, how do hidden states at time $t$ result from hidden states at some previous time $t-1$. These transition dynamics are further conditioned on some past action $u_t$.

```python
actions = ["UP", "DOWN", "LEFT", "RIGHT", "STAY"]

def create_B_matrix():
  B = np.zeros( (len(grid_locations), len(grid_locations), len(actions)) )

  for action_id, action_label in enumerate(actions):

    for curr_state, grid_location in enumerate(grid_locations):

      y, x = grid_location

      if action_label == "UP":
        next_y = y - 1 if y > 0 else y
        next_x = x
      elif action_label == "DOWN":
        next_y = y + 1 if y < 2 else y
        next_x = x
      elif action_label == "LEFT":
        next_x = x - 1 if x > 0 else x
        next_y = y
      elif action_label == "RIGHT":
        next_x = x + 1 if x < 2 else x
        next_y = y
      elif action_label == "STAY":
        next_x = x
        next_y = y
      new_location = (next_y, next_x)
      next_state = grid_locations.index(new_location)
      B[next_state, curr_state, action_id] = 1.0
  return B

B = create_B_matrix()
```

Let's now explore what it looks to "take" an action, using matrix-vector product of an action-conditioned "slice" of the $\mathbf{B}$ array and a previous state vector

```python
""" Define a starting location"""
starting_location = (1,0)

"""get the linear index of the state"""
state_index = grid_locations.index(starting_location)

"""  and create a state vector out of it """
starting_state = utils.onehot(state_index, n_states)

plot_point_on_grid(starting_state, grid_locations)
```

```python
plot_beliefs(starting_state, "Categorical distribution over the starting state")
```

Now let's imagine we're moving **"RIGHT"** - write the conditional expectation, that will create the state vector corresponding to the new state after taking a step to the right.

```python
""" Redefine the action here, just for reference """
actions = ["UP", "DOWN", "LEFT", "RIGHT", "STAY"]

""" Generate the next state vector, given the starting state and the B matrix"""
right_action_idx = actions.index("RIGHT")
next_state = B[:,:, right_action_idx].dot(starting_state) # input the indices to the B matrix

""" Plot the next state, after taking the action """
plot_point_on_grid(next_state, grid_locations)
```

Now let's imagine we're moving **"DOWN"** from where we just landed - write the conditional expectation, that will create the state vector corresponding to the new state after taking a step down.

```python
""" Generate the next state vector, given the previous state and the B matrix"""
prev_state = next_state.copy()
down_action_index = actions.index("DOWN")
next_state = B[:,:,down_action_index].dot(prev_state)

"""  Plot the new state vector, after making the movement """
plot_point_on_grid(next_state, grid_locations)
```

### 3. The prior over observations: the $\mathbf{C}$ vector or $\tilde{P}(o)$

---

The (biased) generative model's prior preference for particular observations, encoded in terms of probabilities.

```python
""" Create an empty vector to store the preferences over observations """
C = np.zeros(n_observations)

""" Choose an observation index to be the 'desired' rewarding index, and fill out the C vector accordingly """
desired_location = (2,2) # choose a desired location
desired_location_index = grid_locations.index(desired_location) # get the linear index of the grid location, in terms of 0 through 8

C[desired_location_index] = 1.0 # set the preference for that location to be 100%, i.e. 1.0

"""  Let's look at the prior preference distribution """
plot_beliefs(C, title_str = "Preferences over observations")
```

### 3. The prior over hidden states: the $\mathbf{D}$ vector or $P(s)$

---

The generative model's prior belief over hidden states at the first timestep.

```python
""" Create a D vector, basically a belief that the agent has about its own starting location """

# create a one-hot / certain belief about initial state
D = utils.onehot(0, n_states)

# demonstrate hwo belief about initial state can also be uncertain / spread among different possible initial states
# alternative, where you have a degenerate/noisy prior belief
# D = utils.norm_dist(np.ones(n_states))

""" Let's look at the prior over hidden states """
plot_beliefs(D, title_str = "Prior beliefs over states")
```

## **Hidden state inference**

Hidden state inference proceeds by finding the setting of the optimal variational posterior $q(s_t)$ that minimizes the variational free energy. For the simple POMDP generative model and posterior we're considering here, this update reduces to a remarkably simple update rule, that is essentially proportional to Bayes` rule:

$$
\begin{align}
q(s_t) = \sigma\left(\ln \mathbf{A}[o,:] + \ln\mathbf{B}[:,:,u] \cdot q(s_{t-1})\right)
\end{align}
$$

Import the `softmax` function and a numerically-stable version of `np.log` from `pymdp.maths`

```python
from pymdp.maths import softmax
from pymdp.maths import spm_log_single as log_stable
```

Below we implement the `infer_states()` function

```python
""" Create an infer states function that implements the math we just discussed"""

def infer_states(observation_index, A, prior):

  """ Implement inference here -- NOTE: prior is already passed in, so you don't need to do anything with the B matrix. """
  """ This function has already been given P(s_t). The conditional expectation that creates "today's prior", using "yesterday's posterior", will happen *before calling* this function"""

  log_likelihood = log_stable(A[observation_index,:])

  log_prior = log_stable(prior)

  qs = softmax(log_likelihood + log_prior)

  return qs
```

Let's do a simulated single timestep of hidden state inference

```python
qs_past = utils.onehot(4, n_states) # agent believes they were at location 4 -- i.e. (1,1) one timestep ago

last_action = "UP" # the agent knew it moved "UP" one timestep ago
action_id = actions.index(last_action) # get the action index for moving "UP"
```

Get "today's prior" using the past posterior and the past action, i.e. calculate:

$$ \begin{align}
P(s_t) = \mathbf{E}_{q(s_{t-1})}\left[P(s_t | s_{t-1}, u_{t-1})\right]
\end{align}
$$

and choose an observation that is consistent with the new location

```python
prior = B[:,:,action_id].dot(qs_past)

observation_index = 1
```

Now run the `infer_states()` function, using the observation, the $\mathbf{A}$ matrix, and the prior we just calculated above

```python
qs_new = infer_states(observation_index, A, prior)
plot_beliefs(qs_new, title_str = "Beliefs about hidden states")
```

Let's see what happens if observation and the prior "disagree" with eachother?

```python
""" Get an observation that 'conflicts' with the prior """
observation_index = 2 # this is like the agent is seeing itself in location (0, 2)
qs_new = infer_states(observation_index, A, prior)
plot_beliefs(qs_new)
```

What happens if likelihood and prior are partially ambiguous?

```python
""" Create an ambiguous A matrix """
A_partially_ambiguous = softmax(A)
print(A_partially_ambiguous.round(3))
```

    [[0.254 0.093 0.093 0.093 0.093 0.093 0.093 0.093 0.093]
     [0.093 0.254 0.093 0.093 0.093 0.093 0.093 0.093 0.093]
     [0.093 0.093 0.254 0.093 0.093 0.093 0.093 0.093 0.093]
     [0.093 0.093 0.093 0.254 0.093 0.093 0.093 0.093 0.093]
     [0.093 0.093 0.093 0.093 0.254 0.093 0.093 0.093 0.093]
     [0.093 0.093 0.093 0.093 0.093 0.254 0.093 0.093 0.093]
     [0.093 0.093 0.093 0.093 0.093 0.093 0.254 0.093 0.093]
     [0.093 0.093 0.093 0.093 0.093 0.093 0.093 0.254 0.093]
     [0.093 0.093 0.093 0.093 0.093 0.093 0.093 0.093 0.254]]

```python
""" ... and a noisy prior """
noisy_prior = softmax(prior)
plot_beliefs(noisy_prior)
```

```python
""" Do inference with the new, partially-ambiguous A matrix and the noised-up prior """
qs_new = infer_states(observation_index, A_partially_ambiguous, noisy_prior)
plot_beliefs(qs_new)
```

## **Action Selection and the Expected Free Energy**

Now we'll write functions to compute the expected states $Q(s_{t+1}|u_t)$, expected observations $Q(o_{t+1}|u_t)$, the entropy of $P(o|s)$: $\mathbf{H}\left[\mathbf{A}\right]$, and the KL divergence between the expected observations and the prior preferences $\mathbf{C}$.

```python
""" define component functions for computing expected free energy """

def get_expected_states(B, qs_current, action):
  """ Compute the expected states one step into the future, given a particular action """
  qs_u = B[:,:,action].dot(qs_current)

  return qs_u

def get_expected_observations(A, qs_u):
  """ Compute the expected observations one step into the future, given a particular action """

  qo_u = A.dot(qs_u)

  return qo_u

def entropy(A):
  """ Compute the entropy of a set of conditional distributions, i.e. one entropy value per column """

  H_A = - (A * log_stable(A)).sum(axis=0)

  return H_A

def kl_divergence(qo_u, C):
  """ Compute the Kullback-Leibler divergence between two 1-D categorical distributions"""

  return (log_stable(qo_u) - log_stable(C)).dot(qo_u)
```

Now let's imagine we're in some starting state, like (1,1)
N.B. This is the generative process we're talking about -- i.e. the true state of the world

```python
""" Get state index, create state vector for (1,1) """

state_idx = grid_locations.index((1,1))
state_vector = utils.onehot(state_idx, n_states)
plot_point_on_grid(state_vector, grid_locations)
```

And let's furthermore assume we start with the (accurate) belief about our location, that we are in location (1,1). So let's just make our current $Q(s_t)$ equal to the true state vector. You could think of this as if we just did one 'step' of inference using our current observation along with precise `A`/`B` matrices.

```python
""" Make qs_current identical to the true starting state """
qs_current = state_vector.copy()
plot_beliefs(qs_current, title_str ="Where do we believe we are?")
```

And we prefer to be in state (1,2). So we express that as high probability over the corresponding entry of the $\mathbf{C}$ vector -- that happens to be index 5.

```python
plot_grid(grid_locations)
```

```python
""" Create a preference to be in (1,2) """

desired_idx = grid_locations.index((1,2))

C = utils.onehot(desired_idx, n_observations)

plot_beliefs(C, title_str = "Preferences")
```

Now to keep things simple, let's evaluate the expected free energies $\mathbf{G}$ of just two actions that we could take from our current position -- either moving LEFT or moving RIGHT. Let's remind ourselves of what action indices those are

```python
left_idx = actions.index("LEFT")
right_idx = actions.index("RIGHT")

print(f'Action index of moving left: {left_idx}')
print(f'Action index of moving right: {right_idx}')
```

    Action index of moving left: 2
    Action index of moving right: 3

And now let's use the functions we've just defined, in combination with our `A`, `B`, `C` arrays and our current posterior `qs_current`, to compute the expected free energies for the two actions

```python
""" Compute the expected free energies for moving left vs. moving right """
G = np.zeros(2) # store the expected free energies for each action in here

"""
Compute G for MOVE LEFT here
"""

qs_u_left = get_expected_states(B, qs_current, left_idx)
# alternative
# qs_u_left = B[:,:,left_idx].dot(qs_current)

H_A = entropy(A)
qo_u_left = get_expected_observations(A, qs_u_left)
# alternative
# qo_u_left = A.dot(qs_u_left)

predicted_uncertainty_left = H_A.dot(qs_u_left)
predicted_divergence_left = kl_divergence(qo_u_left, C)
G[0] = predicted_uncertainty_left + predicted_divergence_left

"""
Compute G for MOVE RIGHT here
"""

qs_u_right = get_expected_states(B, qs_current, right_idx)
# alternative
# qs_u_right = B[:,:,right_idx].dot(qs_current)

H_A = entropy(A)
qo_u_right = get_expected_observations(A, qs_u_right)
# alternative
# qo_u_right = A.dot(qs_u_right)

predicted_uncertainty_right = H_A.dot(qs_u_right)
predicted_divergence_right = kl_divergence(qo_u_right, C)
G[1] = predicted_uncertainty_right + predicted_divergence_right

""" Now let's print the expected free energies for the two actions, that we just calculated """
print(f'Expected free energy of moving left: {G[0]}\n')
print(f'Expected free energy of moving right: {G[1]}\n')
```

    Expected free energy of moving left: 36.841361487904734

    Expected free energy of moving right: 0.0

Now let's use formula for the posterior over actions, i.e.

$$ \begin{align}
Q(u_t) = \sigma(-\mathbf{G})
\end{align} $$

to compute the probabilities of each action

```python
Q_u = softmax(-G)

""" and print the probability of each action """
print(f'Probability of moving left: {Q_u[0]}')
print(f'Probability of moving right: {Q_u[1]}')
```

    Probability of moving left: 9.999999999999965e-17
    Probability of moving right: 1.0

For usefulness later on, let's wrap the expected free energy calculations into a function

```python
def calculate_G(A, B, C, qs_current, actions):

  G = np.zeros(len(actions)) # vector of expected free energies, one per action

  H_A = entropy(A) # entropy of the observation model, P(o|s)

  for action_i in range(len(actions)):

    qs_u = get_expected_states(B, qs_current, action_i) # expected states, under the action we're currently looping over
    qo_u = get_expected_observations(A, qs_u)           # expected observations, under the action we're currently looping over

    pred_uncertainty = H_A.dot(qs_u) # predicted uncertainty, i.e. expected entropy of the A matrix
    pred_div = kl_divergence(qo_u, C) # predicted divergence

    G[action_i] = pred_uncertainty + pred_div # sum them together to get expected free energy

  return G
```

## **Complete Recipe for Active Inference**

1.   Sample an observation $o_t$ from the current state of the environment
2.   Perform inference over hidden states i.e., optimize $q(s)$ through free-energy minimization
3. Calculate expected free energy of actions $\mathbf{G}$
4. Sample action from the posterior over actions $Q(u_t) \sim \sigma(-\mathbf{G})$.
5. Use the sampled action $a_t$ to perturb the generative process and go back to step 1.

---

Let's start by creating a class that will represent the Grid World environment (i.e., _generative process_) that our active inference agent will navigate within. Note that we don't need to specify the generative process in terms of `A` and `B` matrices - the generative process will be as arbitrary and complex as the environment is. The `A` and `B` matrices are just the agent's _representation_ of the world and the task, which in this case happen to capture the Markovian, noiseless dynamics of the world perfectly.

```python
class GridWorldEnv():

    def __init__(self,starting_state = (0,0)):

        self.init_state = starting_state
        self.current_state = self.init_state
        print(f'Starting state is {starting_state}')

    def step(self,action_label):

        (Y, X) = self.current_state

        if action_label == "UP":

          Y_new = Y - 1 if Y > 0 else Y
          X_new = X

        elif action_label == "DOWN":

          Y_new = Y + 1 if Y < 2 else Y
          X_new = X

        elif action_label == "LEFT":
          Y_new = Y
          X_new = X - 1 if X > 0 else X

        elif action_label == "RIGHT":
          Y_new = Y
          X_new = X +1 if X < 2 else X

        elif action_label == "STAY":
          Y_new, X_new = Y, X

        self.current_state = (Y_new, X_new) # store the new grid location

        obs = self.current_state # agent always directly observes the grid location they're in

        return obs

    def reset(self):
        self.current_state = self.init_state
        print(f'Re-initialized location to {self.init_state}')
        obs = self.current_state
        print(f'..and sampled observation {obs}')

        return obs

env = GridWorldEnv()
```

    Starting state is (0, 0)

Now equipped with an environment and a generative model (our `A`, `B`, `C`, and `D`), we can code up the entire active inference loop

---

To have everything in one place, let's re-create the whole generative model

```python
""" Fill out the components of the generative model """

A = np.eye(n_observations, n_states)

B = create_B_matrix()

C = utils.onehot(grid_locations.index( (2, 2) ), n_observations) # make the agent prefer location (2,2) (lower right corner of grid world)

D = utils.onehot(grid_locations.index( (1,2) ), n_states) # start the agent with the prior belief that it starts in location (1,2)

actions = ["UP", "DOWN", "LEFT", "RIGHT", "STAY"]
```

Now let's initialize the environment with starting state `(1,2)`, so that the agent has accurate beliefs about where it's starting.

```python
env = GridWorldEnv(starting_state = (1,2))
```

    Starting state is (1, 2)

Let's write a function that runs the whole active inference loop, and then run it for `T = 5` timesteps.

```python
""" Write a function that, when called, runs the entire active inference loop for a desired number of timesteps"""

def run_active_inference_loop(A, B, C, D, actions, env, T = 5):

  """ Initialize the prior that will be passed in during inference to be the same as `D` """
  prior = D.copy() # initial prior should be the D vector

  """ Initialize the observation that will be passed in during inference - hint use env.reset()"""
  obs = env.reset() # initialize the `obs` variable to be the first observation you sample from the environment, before `step`-ing it.

  for t in range(T):

    print(f'Time {t}: Agent observes itself in location: {obs}')

    # convert the observation into the agent's observational state space (in terms of 0 through 8)
    obs_idx = grid_locations.index(obs)

    # perform inference over hidden states
    qs_current = infer_states(obs_idx, A, prior)

    plot_beliefs(qs_current, title_str = f"Beliefs about location at time {t}")

    # calculate expected free energy of actions
    G = calculate_G(A, B, C, qs_current, actions)

    # compute action posterior
    Q_u = softmax(-G)

    # sample action from probability distribution over actions
    chosen_action = utils.sample(Q_u)

    # compute prior for next timestep of inference
    prior = B[:,:,chosen_action].dot(qs_current)

    # update generative process
    action_label = actions[chosen_action]

    obs = env.step(action_label)

  return qs_current

""" Run the function we just wrote, for T = 5 timesteps """
qs = run_active_inference_loop(A, B, C, D, actions, env, T = 5)

```

## **Planning**

But there's a problem. Imagine we started the agent at the top left corner (coordinate `(0,0)`) rather than one step away from the target location

```python
D = utils.onehot(grid_locations.index((0,0)), n_states) # let's have the agent believe it starts in location (0,0)

env = GridWorldEnv(starting_state = (0,0))
qs = run_active_inference_loop(A, B, C, D, actions, env, T = 5)
```

Because the expected free energy of each action is only evaluated for one timestep in the future, the agent has no way of knowing which action to initially take, to get closer to its final "goal" of `(2,2)`. This is because the expected divergence term for all actions $\operatorname{D}_{KL}(Q(o|u) \parallel \mathbf{C})$ is identical for all considered actions (which are just 1-step moves away from starting from `(0,0)` ). This speaks to the importance of planning, or _multiple timestep policies_.

Now let's do active inference with multi-step policies

We can rely on a useful function from `pymdp`'s `control` module called `construct_policies()` to automatically generate a list of all the policies we want to entertain, for a given number of control states (actions) and a desired temporal horizon.

```python
from pymdp.control import construct_policies
```

```python
policy_len = 4
n_actions = len(actions)

# we have to wrap `n_states` and `n_actions` in a list for reasons that will become clear in Part II
all_policies = construct_policies([n_states], [n_actions], policy_len = policy_len)

print(f'Total number of policies for {n_actions} possible actions and a planning horizon of {policy_len}: {len(all_policies)}')
```

    Total number of policies for 5 possible actions and a planning horizon of 4: 625

Let's re-write our expected free energy function, but now we loop over policies (sequences of actions), rather than actions (1-step policies)

```python
def calculate_G_policies(A, B, C, qs_current, policies):

  G = np.zeros(len(policies)) # initialize the vector of expected free energies, one per policy
  H_A = entropy(A)            # can calculate the entropy of the A matrix beforehand, since it'll be the same for all policies

  for policy_id, policy in enumerate(policies): # loop over policies - policy_id will be the linear index of the policy (0, 1, 2, ...) and `policy` will be a column vector where `policy[t,0]` indexes the action entailed by that policy at time `t`

    t_horizon = policy.shape[0] # temporal depth of the policy

    G_pi = 0.0 # initialize expected free energy for this policy

    for t in range(t_horizon): # loop over temporal depth of the policy

      action = policy[t,0] # action entailed by this particular policy, at time `t`

      # get the past predictive posterior - which is either your current posterior at the current time (not the policy time) or the predictive posterior entailed by this policy, one timstep ago (in policy time)
      if t == 0:
        qs_prev = qs_current
      else:
        qs_prev = qs_pi_t

      qs_pi_t = get_expected_states(B, qs_prev, action) # expected states, under the action entailed by the policy at this particular time
      qo_pi_t = get_expected_observations(A, qs_pi_t)   # expected observations, under the action entailed by the policy at this particular time

      kld = kl_divergence(qo_pi_t, C) # Kullback-Leibler divergence between expected observations and the prior preferences C

      G_pi_t = H_A.dot(qs_pi_t) + kld # predicted uncertainty + predicted divergence, for this policy & timepoint

      G_pi += G_pi_t # accumulate the expected free energy for each timepoint into the overall EFE for the policy

    G[policy_id] += G_pi

  return G

```

Let's write a function for computing the action posterior, given the posterior probability of each policy:

```python
def compute_prob_actions(actions, policies, Q_pi):
  P_u = np.zeros(len(actions)) # initialize the vector of probabilities of each action

  for policy_id, policy in enumerate(policies):
    P_u[int(policy[0,0])] += Q_pi[policy_id] # get the marginal probability for the given action, entailed by this policy at the first timestep

  P_u = utils.norm_dist(P_u) # normalize the action probabilities

  return P_u
```

Now we can write a new active inference function, that uses temporally-deep planning

```python
def active_inference_with_planning(A, B, C, D, n_actions, env, policy_len = 2, T = 5):

  """ Initialize prior, first observation, and policies """

  prior = D # initial prior should be the D vector

  obs = env.reset() # get the initial observation

  policies = construct_policies([n_states], [n_actions], policy_len = policy_len)

  for t in range(T):

    print(f'Time {t}: Agent observes itself in location: {obs}')

    # convert the observation into the agent's observational state space (in terms of 0 through 8)
    obs_idx = grid_locations.index(obs)

    # perform inference over hidden states
    qs_current = infer_states(obs_idx, A, prior)
    plot_beliefs(qs_current, title_str = f"Beliefs about location at time {t}")

    # calculate expected free energy of actions
    G = calculate_G_policies(A, B, C, qs_current, policies)

    # to get action posterior, we marginalize P(u|pi) with the probabilities of each policy Q(pi), given by \sigma(-G)
    Q_pi = softmax(-G)

    # compute the probability of each action
    P_u = compute_prob_actions(actions, policies, Q_pi)

    # sample action from probability distribution over actions
    chosen_action = utils.sample(P_u)

    # compute prior for next timestep of inference
    prior = B[:,:,chosen_action].dot(qs_current)

    # step the generative process and get new observation
    action_label = actions[chosen_action]
    obs = env.step(action_label)

  return qs_current

```

Now run it to see if the agent is able to navigate to location `(2,2)`

```python
D = utils.onehot(grid_locations.index((0,0)), n_states) # let's have the agent believe it starts in location (0,0) (upper left corner)
env = GridWorldEnv(starting_state = (0,0))
qs_final = active_inference_with_planning(A, B, C, D, n_actions, env, policy_len = 3, T = 10)
```