Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Yves Dubief, 2016. NSF for support via NSF-CBET award #1258697.

```python
"""
importing the necessary libraries, do not modify
"""
%matplotlib inline
from IPython.display import display,Image, Latex

from IPython.display import clear_output

import matplotlib.pyplot as plt
import numpy as np
import math
import scipy.constants as sc

import sympy as sym

```

# Jupyter notebooks to solve heat transfer problems

## Module schemdraw
In order to use jupyter notebooks for your assignments you first need to install a module to draw resistance circuits.

### Windows

Search and launch anaconda prompt

### Mac OS

In `Applications`>`Utilities` open Terminal

### Command

In the prompt or terminal execute:

`python -m pip install schemdraw`

## Inserting sketches/photos

The very step for solving any engineering problem is to draw a sketch. The notebook is an awesome tool but it does not allow for drawing yet. Using your favorite app to sketch or simply taking a picture or scan of your hand-drawn sketch, use the following command to insert your image in a `markdown` cell:

``

or

``

If you only enter one attribute (width or height), the browser should preserve the width:height ratio.

```python

import schemdraw as schem
import schemdraw.elements as e
```

# Thermodynamics Module

The thermodynamic properties necessary for the treatment of heat transfer problem include:
* Density $\rho \text{ (kg/m$^3$)}$
* Specific heat at constant pressure $C_p \text{ (J/(kg.K)}$
* Dynamic viscosity $\mu \text{ (kg/(m.s))}$
* Thermal conductivity $k \text{ (W/(m.K))}$
* Kinematic viscosity $\nu \text{ (m$^2$/s)}$
* Thermal diffusivity $\alpha \text{ (m$^2$/s)}$
* Prandtl number $Pr$, ratio of kinematic viscosity to thermal diffusivity, dimensionless
* Thermal expansion coefficient $\beta \text{ (1/K)}$

The module from Libraries import thermodynamics as thermo  gives these thermodynamics properties for water, air, argon and krypton under $1\text{ atm}$ over a given range of temperature $T$ (dependent upon the fluid). Air properties are tabled between $-150\text{$^\circ$C}$ and $400\text{$^\circ$C}$, water between $274\text{K}$ and $373\text{K}$, Argon between $100\text{K}$ and $700\text{K}$ and Krypton between $150\text{K}$ and $750\text{K}$

    When using the library thermodynamics, the input temperature must be in Kelvin

Use the library's functions C2K, K2C, K2F  and F2K  to convert temperatures to Kelvin, Celsius or Fahrenheit. Thermodynamic properties are linearly interpolated from the two nearest temperature states.

## How does it work?

First load up the module

```python
from Libraries import thermodynamics as thermo
```

Second define a variable as your fluid. For example:

```python
water_pipe = thermo.Fluid('water',thermo.F2K(70.))
```

The class is called Fluid, the capital letter F is necessary. If you don't remember which fluids are available, execute the following:

```python
? thermo.Fluid
```

Now you can print any of the quantities defined above:

```python
print('Density %4.3f kg/m^3' %water_pipe.rho)
print('Specific heat %4.3f J/Kg.K' %water_pipe.Cp)
print('Dynamic viscosity %1.3e kg/m.s' %water_pipe.mu)
print('Thermal conductivity %4.4f W/m.K' %water_pipe.k)
print('Kinematic viscosity %1.3e m^2/s' %water_pipe.nu)
print('Thermal diffusivity %1.3e m^2/s' %water_pipe.alpha)
print('Prandtl number %4.3f ' %water_pipe.Pr)
print('Thermal expansion coefficient %1.3e 1/K' %water_pipe.beta)
```

    Density 997.973 kg/m^3
    Specific heat 4183.343 J/Kg.K
    Dynamic viscosity 9.750e-04 kg/m.s
    Thermal conductivity 0.6004 W/m.K
    Kinematic viscosity 9.770e-07 m^2/s
    Thermal diffusivity 1.438e-07 m^2/s
    Prandtl number 6.793
    Thermal expansion coefficient 2.181e-04 1/K

## Thermal Resistance Module

The class thermal resistance is imported with from Libraries import HT_thermal_resistance as res.

Thermal resistances are derived from an analogy between heat transfer and electric circuit. Each mode of heat transfer can be represented by a thermal resistance, linking the heat flux ($q''\text{ W/m$^2$}$), heat rate per unit length ($q'\text{ W/m}$) or heat rate ($q\text{ W}$) to the temperature difference between two points ($T_a-T_b$, with $T_a>T_b$:

$$
q''=\frac{1}{R''_\text{th}}\left(T_a-T_b\right),\; q'=\frac{1}{R'_\text{th}}\left(T_a-T_b\right),\; q=\frac{1}{R_\text{th}}\left(T_a-T_b\right)
$$

The heat transfer rate, also called heat loss or heat gain, is analog to the current, and the temperature to the potential difference.

The first step is to define an array of thermal resistance, and give each resistance its own label.

Here we use the class Resistance in the module HT_thermal_resistance. The first call requires the label (in Latex format with \$ $\) and the dimensional unit of the associated flux

```python
from Libraries import HT_thermal_resistance as res

Rth = []
Rth.append(res.Resistance('$R_{conv,in}$','W'))
Rth.append(res.Resistance('$R_{cond,pipe}$','W'))
Rth.append(res.Resistance('$R_{conv,out}$','W'))
Rth.append(res.Resistance('$R_{rad,out}$','W'))

print('number of resistances %2i' %len(Rth))
```

    number of resistances  4

Typically the next step is to draw the thermal circuit using the module `schemdraw` already imported in the first cell:

import schemdraw as schem

and

import schemdraw.elements as e

```python
d = schem.Drawing()
L1=d.add( e.LINE, d = 'right')
d.labelI(L1,'q')
d.add( e.DOT, label='$T_{\infty.in}$')
R0 = d.add( e.RES, d='right', label=Rth[0].name )
d.add( e.DOT, label='$T_a$')
R1 = d.add( e.RES, d='right', label=Rth[1].name )
d.add( e.DOT, rgtlabel='$T_b$')
L0 = d.add( e.LINE, d = 'up', l = 1.5)
R2 = d.add( e.RES, d='right', label=Rth[2].name )
L1 = d.add( e.LINE, d = 'down', l = 1.5)
L2 = d.add( e.LINE, d = 'right', l = 1.5)
d.add(e.DOT, label = '$T_\infty.out$')
L3 = d.add( e.LINE, d = 'down', xy = R1.end, l = 1.5)
R3 = d.add( e.RES, d = 'right', label = Rth[3].name)
L4 = d.add( e.LINE, d = 'up', l = 1.5)
d.draw()
```

For 1D conduction in a solid or a fluid at rest, the expression of the thermal resistance depends upon the geometry of the system:
* **Plane wall** or planar conduction is declared by the function `R[i].cond_plane(k,L)` for calculation of heat flux ($W/m^2$) or `R[i].cond_plane(k,L,A)` for calculation or heat rate ($W$). `k` is the thermal conductivity, `L` the wall thickness and `A` is the cross sectional area of the wall (perpendicular to the flux).

$$
R_\text{th,cond}=\frac{L}{kA}
$$

* **Cylindrical shell**: `R[i].cond_cylinder(k,ra,rb)` or `R[i].cond_cylinder(k,ra,rb,L=Lpipe ,)` or `R[i].cond_cylinder(k,ra,rb,angle)` or `R[i].cond_cylinder(k,ra,rb,L=Lpipe ,angle = angle_pipe)` . The first computes $R'$ for a unit length of the pipe, the second $R$ for the whole pipe, the third $R'$ for a cylindrical shell with a specific angular dimenion (i.e. smaller than $2\pi$) and the fourth is the same as the third but for the whole length of the shell. `ra` and `rb` are the internal and external radii of the shell.

$$
R_\text{th,cond}=\cfrac{\ln\left(\cfrac{r_b}{r_a}\right)}{2\pi kL}
$$

* **Spherical shell**: `R[i].cond_sphere(k,ra,rb)` or `R[i].cond_sphere(k,ra,rb,scale)`. Here scale is a value <= 1., 1. by default for a full spherical shell, less than one for a fraction shell. For instance, a cornea is about 1/3 of spherical shell, therefore scale=1./3.

$$
R_\text{th,cond}=\cfrac{\cfrac{1}{r_a}-\cfrac{1}{r_b}}{4\pi k}
$$

* **Contact resistance**: `R[i].contact(Rc)` or `R[i].contact(Rc,A)`. `A` is optional and `A=1.0` by default when calculating heat flux. `Rc`, the contact resistance is obtained from tables based on the materials involved and their respective roughness.

* **Convection**: `R[i].convection(h)` or `R[i].convection(h)`. `A` is optional and `A=1.0` by default when calculating heat flux. `h` is the convection heat transfer coefficient ($W/(m^2.K)$ obtained from correlations specific to the geometry and flow parameters of the problem.

$$
R_\text{th,conv}=\frac{1}{hA}
$$

* **Radiation**: `R[i].radiation(eps,T_s,T_sur)` and `R[i].radiation(eps,T_s,T_sur,A)`. `A` is optional and `A=1.0` by default when calculating heat flux. `eps` is the permissivity of the material, `T_s` the surface temperature and `T_sur` is the temperature of the distant surface emitting radiation. **Both temperatures must be in Kelvin**. `R[i].h` outputs the radiation heat transfer coefficient $h_r$:

$$
R_\text{th,rad}=\frac{1}{h_rA}\, \text{ with } h_r = \varepsilon\sigma(T_s+T_\text{sur})(T_s^2+T_\text{sur}^2)
$$
 To calculate $h_r$, temperatures must be in Kelvin

Like a thermal circuit, $n$ thermal resistances in serial are equivalent to one total resistance. If the $n$ resistances are stored in an array $R[i]$, the total resistance of resistances in series starting at $m$ and finishing at $n$ is
serial_sum(R,m,n)

$$
R_\text{th,tot}=\sum_{i=m}^{n}R[i]
$$

Similarly, thermal resistances in parallel between $n$ and $m$ reduce to
parallel_sum(R,m,n)

$$
R_\text{th,tot}=\left(\sum_{i=m}^{n}\cfrac{1}{R[i]}\right)^{-1}
$$

```python
"""Example: heated water at T_infty_in flowing in a pipe with convection heat transfer h_i. The pipe of length L is made of copper, length L,
inner radius r_a, outer radius r_b. The pipe is immersed in air at T_infty_out and the heat transfer from the outer
surface of the pipe to the air is through convection and radiation. Find the heat loss per unit length
"""
L_pipe = 1. #m
r_a = 0.05/2.  #m
r_b = 0.055/2. #m
h_i = 60. #W/m^2.K
k_pipe = 80. #W/m.K
h_o = 2. #W/m^2.K
eps = 0.8
T_infty_in = 320. #C
T_infty_out = 5. #C
T_sur = T_infty_out #radiation reference temperature
A_i = 2.*np.pi*r_a*L_pipe
A_o = 2.*np.pi*r_b*L_pipe
print(A_i,A_o)

""" By construction, the radiation resistance needs the surface temperature at r=r_b. """
T_b = 310 #C a guess at this poin

Rth = []
Rth.append(res.Resistance("$R'_{conv,in}$","W/m"))
Rth.append(res.Resistance("$R'_{cond,pipe}$","W/m"))
Rth.append(res.Resistance("$R'_{conv,out}$","W/m"))
Rth.append(res.Resistance("$R'_{rad,out}$","W/m"))

d = schem.Drawing()
#L1=d.add( e.LINE, d = 'right')
#d.labelI(L1,'q')
d.add( e.DOT, label='$T_{\infty,in}$')
R0 = d.add( e.RES, d='right', label=Rth[0].name )
d.add( e.DOT, label='$T_a$')
R1 = d.add( e.RES, d='right', label=Rth[1].name )
d.add( e.DOT, rgtlabel='$T_b$')
L0 = d.add( e.LINE, d = 'up', l = 1.5)
R2 = d.add( e.RES, d='right', label=Rth[2].name )
L1 = d.add( e.LINE, d = 'down', l = 1.5)
L2 = d.add( e.LINE, d = 'right', l = 1.5)
d.add(e.DOT, label = '$T_\infty,out$')
L3 = d.add( e.LINE, d = 'down', xy = R1.end, l = 1.5)
R3 = d.add( e.RES, d = 'right', label = Rth[3].name)
L4 = d.add( e.LINE, d = 'up', l = 1.5)
d.draw()

Rth[0].convection(h_i,A_i)
Rth[1].cond_cylinder(k=k_pipe,ra = r_a, rb = r_b, L = L_pipe)
Rth[2].convection(h_o,A_o)
error = np.inf
threshold = 0.001
iteration = 0
while (error > threshold) and (iteration < 10):

    Rth[3].radiation(eps,thermo.C2K(T_b),thermo.C2K(T_sur),A_o)
    R_tot = res.serial_sum(Rth,0,1) + res.parallel_sum(Rth,2,3)
    q = R_tot*(T_infty_in - T_infty_out)
    T = np.zeros(4)
    T_b_old = T_b
    T[0] = T_infty_in
    T[1] = T[0] - q*Rth[0].R
    T[2] = T[1] - q*Rth[1].R
    T[3] = T_infty_out
    T_b = T[2]
    print(q)
    error = np.abs(T_b - T_b_old)
    iteration += 1

print("T_b = %.2f C"%T_b)
print("Number of iterations: %2i" %iteration)
print("Heat rate: %.0f W" %q)

```

    0.15707963267948966 0.17278759594743862
    133.05054277632627
    134.516288064436
    134.57191534774122
    134.5740272051045
    T_b = 305.70 C
    Number of iterations:  4
    Heat rate: 135 W

```python

```


# Lab 3 Exercises for COMP 432 Machine Learning

In this lab you'll cluster and fit mixture models to data using the popular _scikit-learn_ package. Lab3 requires a good understanding of Numpy and Matplotlib. Please complete Lab1 before attempting Lab3.

**Run the code cell below** to import the required packages.

```python
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import sklearn
import sklearn.cluster         # For KMeans class
import sklearn.mixture         # For GaussianMixture class
import sklearn.preprocessing   # For scale function
import mpl_toolkits.mplot3d    # For enabling projection='3d' feature in Matplotlib
```

# Introduction
This "introduction" group of code cells has no exercises. The goal is to provide you with examples. Run the code cells and learn from them.

Load the image *ladybug.png* as a Numpy array. The image should already have been unzipped into the same directory as this notebook.

```python
image = plt.imread('ladybug.png')
print(image.shape)
```

    (100, 100, 3)

Notice the shape of the array. Entry _image[y,x]_ in the array is a length-3 vector representing the RGB value of pixel at location $(x, y)$ in the image.

Plot the image using Matplotlib's _imshow_ function.

```python
plt.imshow(image);
```

Reshape the image into a 2-dimensional matrix suitable for clustering the pixels by their RGB colour value. The data should be in array format

$$
\begin{bmatrix}
r_1 & g_1 & b_1\\
r_2 & g_2 & b_2\\
\vdots & \vdots & \vdots\\
r_N & g_N & b_N
\end{bmatrix}
$$

where $N=10000$ and each $\begin{bmatrix} r_i & g_i & b_i \end{bmatrix}$ is the colour of pixel with index $i$.

```python
RGB = image.reshape(-1, 3)
print(RGB.shape)
print(RGB)
```

    (10000, 3)
    [[0.30588236 0.5568628  0.2       ]
     [0.27058825 0.53333336 0.16470589]
     [0.27058825 0.53333336 0.16470589]
     ...
     [0.4509804  0.6745098  0.34901962]
     [0.4392157  0.6784314  0.34509805]
     [0.46666667 0.6901961  0.37254903]]

The top-left pixel (index $i=0$) has colour $\begin{bmatrix} 0.31 & 0.56 & 0.20 \end{bmatrix}$, which is green-ish. Makes sense!

Plot the pixel RGB values in 3-dimensional colour space.

```python
def plot_colour_space(RGB, title=None, hold=False, **kwargs):
    """
    Plots Nx3 matrix RGB in 3 dimensions.

    The keyword arguments are passed to Matplotlib's scatter() function.
    If hold=True, the points will be added to the previous plot. Otherwise a new plot is generated.
    """
    if hold:
        ax = plt.gca()
    else:
        fig = plt.figure(figsize=(8,8))
        ax = fig.add_subplot(111, projection='3d')
    kwargs.setdefault('alpha', 1.0)
    ax.scatter(*RGB.T, **kwargs)
    ax.set_xlabel('R', color='r', fontweight='bold', fontsize=15)
    ax.set_ylabel('G', color='g', fontweight='bold', fontsize=15)
    ax.set_zlabel('B', color='b', fontweight='bold', fontsize=15)
    if title is not None:
        ax.set_title(title)

plot_colour_space(RGB, c=RGB, s=0.5, marker='s', title="Pixels plotted in colour space")
```

Run the *K*-means clustering algorithm with *K*=5, using each RGB pixel colour vector as a data point.

```python
def run_kmeans(data, k):
    """
    Runs K-means on an NxD array using k clusters.

    Returns a KxD matrix of centroids and a length-N vector of labels (cluster assignments).
    """
    kmeans = sklearn.cluster.KMeans(n_clusters=k, random_state=0).fit(data)
    return kmeans.cluster_centers_, kmeans.labels_

centroids, labels = run_kmeans(RGB, k=5)

print(centroids)
print(labels, len(labels))
```

    [[0.918301   0.11951366 0.04882497]
     [0.21993323 0.46929815 0.09348458]
     [0.8889072  0.67274106 0.681143  ]
     [0.07332361 0.23711264 0.04663859]
     [0.39092913 0.5740279  0.2899987 ]]
    [4 1 1 ... 4 4 4] 10000

The centroids array has shape (5,3) because there are *K*=5 centroids, a 3-dimensional vector. The labels are given as integer indices, as in the "alternate formulation for *K*-means" from Lecture 2.

Plot the centroids in RGB colour space.

```python
plot_colour_space(centroids, facecolors=centroids, edgecolor='black', s=1000, title="Centroids only")
```

Plot the pixels in RGB space but with colour matching that of the centroid the pixel was assigned to. Include the centroids themselves in the same plot.

```python
# Create a new (10000,3) ndarray where values in row i are copied from centroids[labels[i],:]
RGB_recoloured = centroids[labels]

# Plot the re-coloured pixels, along with the corresponding centroids
plot_colour_space(RGB, c=RGB_recoloured, marker='s', s=0.5, title="Centroids and recoloured pixels")
plot_colour_space(centroids, facecolors=centroids, edgecolor='black', s=1000, hold=True)
```

```python
plt.imshow(RGB_recoloured.reshape(image.shape));
```

Compare the above to Figure 9.3 (p.429) of the Bishop textbook.

# 1. Spatial clustering with K-means

Exercises 1.1–1.4 ask you to apply scikit-learn's **[sklearn.cluster.KMeans](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html)** object to the pixels from the introduction, but on slightly different data.

It requires that you have already **run the code cells from the introduction**.

```python
assert 'image' in globals(), "Did you run the code cells from introduction?"
assert 'RGB' in globals(), "Did you run the code cells from introduction?"
```

### Exercise 1.1 — Add pixel location features

The introduction showed you how to cluster pixels by colour, like Figure 9.3 (p.429) of the Bishop book. The data that we clustered was an array of 3-dimensional features $\begin{bmatrix} r_i & g_i & b_i \end{bmatrix}$, stored in the array object referenced by variable *RGB*.

In exercises 1.2–1.4, you'll be asked to cluster the pixels by _colour_ (rgb) and _location_ (xy). To do this, you must first add pixel location features to the data.

**Write a few lines of code** to create a new array called *RGBXY* where each row is a 5-dimensional feature vector $\begin{bmatrix} r_i & g_i & b_i & x_i & y_i\end{bmatrix}$, with $(x_i, y_i)$ being the location in the image where pixel $i$ came from in the original image. Use the *image* variable to get the height and width of the original ladybug image. Then use the [**np.arange**](https://numpy.org/doc/stable/reference/generated/numpy.arange.html) and [**np.meshgrid**](https://numpy.org/doc/stable/reference/generated/numpy.meshgrid.html) function to generate two arrays, each containing the individual $x_i$ and $y_i$ values respectively. Finally, use a matrix stacking function like [**np.column_stack**](https://numpy.org/doc/stable/reference/generated/numpy.column_stack.html) to create the new array object *RGBXY* from the original *RGB* data and your new pixel location arrays.

If you have done everything correctly, your data with extra "pixel location features" should look like this:
```python
>>> RGBXY
[[ 0.30588236  0.55686277  0.2         0.          0.        ]
 [ 0.27058825  0.53333336  0.16470589  1.          0.        ]
 [ 0.27058825  0.53333336  0.16470589  2.          0.        ]
 ...
 [ 0.4509804   0.67450982  0.34901962 97.         99.        ]
 [ 0.43921569  0.67843139  0.34509805 98.         99.        ]
 [ 0.46666667  0.6901961   0.37254903 99.         99.        ]]
```

```python
# Your code here. Aim for 2-6 lines.
y,x,d = image.shape
mesh = np.array(np.meshgrid(np.arange(y), np.arange(y))).T.reshape(-1, 2).copy()
mesh[:, [1, 0]] = mesh[:, [0, 1]]
RGBXY = np.column_stack((image.reshape(x*y, 3), mesh)).astype(RGB.dtype)
```

**Check your answer** by running the code cell below.

```python
assert 'RGBXY' in globals(), "You didn't create a variable called RGBXY"
assert isinstance(RGBXY, np.ndarray), "Expected RGBXY to be ndarray"
assert RGBXY.dtype == RGB.dtype, "RGBXY has wrong dtype"
assert RGBXY.shape == (100*100,5), "RGBXY has wrong shape"
assert np.array_equal(RGBXY[99:101],
                      np.array([[0.38039216, 0.57647060, 0.15686275, 99., 0.],
                                [0.25490198, 0.52156866, 0.14509805,  0., 1.]], dtype=np.float32)), "RGBXY wrong data"
print("Correct!")
```

### Exercise 1.2 — Cluster pixels spatially

**Write a few lines of code** to do the following:
1. Cluster the $\begin{bmatrix} r_i & g_i & b_i & x_i & y_i\end{bmatrix}$ values that are currently stored in the _RGBXY_ variable. Use *K*=6 clusters. You can use *sklearn.cluster.KMeans* directly or you can use the _run_kmeans_ function from the introduction.
2. Plot the re-coloured image, where each pixel's colour has been replaced by its corresponding centroid colour, as we did in the introduction.

Your plot should look like this:

_Hint:_ Your _RGBXY_ array has shape (10000,5), but the _imshow_ function expects an array of shape (*height*, *width*, 3). So you will need to slice out colour data and reshape it into a form that _imshow_ expects.

```python
clusters, lab = run_kmeans(RGBXY, 6)
colorData = clusters[lab][:, :3].reshape(x, y, 3)
plt.imshow(colorData)
```

### Exercise 1.3 — More clusters!

**Repeat Exercise 1.2** but this time use *K*=50 clusters. It may take a few seconds. If you do not understand the result that you see, ask for insight from a TA or from someone else.

```python
clusters50, lab2 = run_kmeans(RGBXY, 50)
colorData2 = clusters50[lab2][:, :3].reshape(x, y, 3)
plt.imshow(colorData2)
```

### Exercise 1.4 — Feature scaling

**Repeat Exercise 1.3** but this time scale the $x$ and $y$ features by a factor of 0.02 before clustering. Your answer should use Numpy's "broadcasting" mechanism to achieve the scaling in a single step.

Be sure to store the transformed values in a new array called _RGBXY_scaled_ rather than modifying the original _RGBXY_ array. Otherwise you may end up having to re-run the code cell that generated the _RGBXY_ variable.

```python
RGBXY_scaled = RGBXY * [1, 1, 1, 0.02, 0.02]
# RGBXY_scaled[:, [3,4]] = RGBXY_scaled[:, [3,4]] * 0.02
scaledCluseter, lab3 = run_kmeans(RGBXY_scaled, 50)
colorData3 = scaledCluseter[lab3][:, :3].reshape(x, y, 3)
plt.imshow(colorData3)
```

The purpose of this exercise is to see that the clustering method is very sensitive to the scale of different features. If you were to scale the _XY_ components all the way to zero, you would be back to clustering _only by colour_, as we did in the introduction.

Before continuing, try scaling $(x,y)$ by different values, such as $(0.05, 0.02)$ and see how that effects the pixel clusters.

# 2. Selecting *K* in *K*-means

Exercises 2.1–2.2 ask you to generate synthetic data and to apply the "elbow" heuristic for selecting optimal *K* in *K*-means.

**Run the code cell below** to define a function for sampling data from a mixture of Gaussians.

```python
def sample_gmm(means, covs=None, weights=None, N=1):
    """
    Samples a K-component D-dimensional Gaussian mixture.

    The means are KxD.
    The covariances are KxDxD. The default covariance is the DxD identity matrix.
    The weights are length K and must sum to 1. The default is uniform weights.

    Returns (X, c) where X is an NxD array of samples and c is a length-N vector
    of component indices, i.e. X[i] was sampled from mixture component c[i].
    """
    K, D = means.shape

    # Valudate inputs and set default values if needed
    if covs is None:
        covs = np.tile(np.eye(D), (K,1,1))  # Stack of K D-dimensional identity matricies
    if weights is None:
        weights = np.full(K, 1/K)
    assert covs.shape == (K,D,D)

    # Sample a vector of component choices in proportion to weights, one for each sample
    c = np.random.choice(K, N, p=weights)  # choice() checks that weights has shape (K,1) and sums to 1

    # Fill an array of N samples, one component at a time
    X = np.empty((N,D))
    for k in range(K):
        X[k==c] = np.random.multivariate_normal(means[k], covs[k], np.count_nonzero(k==c))
    return X, c
```

### Exercise 2.1 — Generate synthetic GMM data

**Write a few lines of code** to generate 1000 samples from a 9-component Gaussian mixture, where the means are arranged in a 3x3 grid as shown below.

Use a single call to the *sample_gmm* function to generate your data. That means you must build a _means_ matrix with shape (9,2) containing the 9 means. You should know how to do this procedurally, either with for-loops or with a call to the _np.meshgrid_ and *np.column_stack* functions. (Do not write out all 9 of the mean vectors by hand.)

Create a variable called _X_ that refers to your samples, and create a variable called _c_ that refers to the mixture component indices (this value is directly returned by *sample_gmm*, so just store the result).

The `np.random.seed(0)` ensures you always generate the same data each time you run the code cell.

```python
np.random.seed(0)
seedx, seedy = np.meshgrid(np.linspace(-5, 5, 3), np.linspace(-5, 5, 3))
X, c = sample_gmm(np.column_stack((seedx.flatten(), seedy.flatten())), N=1000)
```

    (9, 4) (1000, 2)

**Plot your data** by running the code cell below.

```python
assert 'X' in globals(), "You must create a variable called 'X' that refers to your samples!"
assert 'c' in globals(), "You must create a variable called 'c' that refers to your component indices!"
assert X.shape == (1000,2), "X should have shape (1000, 2)!"
assert c.shape == (1000,), "c should have shape (1000,)!"
plt.scatter(*X.T, c=c, s=5)     # Plot each point (X[i,0], X[i,1]) using colour index c[i] and point size 5
plt.title("Synthetic GMM data")
plt.xticks([-5, 0, 5])
plt.yticks([-5, 0, 5])
plt.gca().set_aspect('equal')   # It's important to see this data with equal scales on each axis
```

### Exercise 2.2 — Choosing *K* with *K*-means

When applying *K*-means, it may be difficult to choose the best *K* for a particular data set. Recall the "elbow curve" method for selecting *K* as reviewed in Lecture 2. You are asked to apply this method to select a "good" *K* for the data set generated in Exercise 2.1.

**Write a few lines of code** to generate a plot of *K* ($x$-axis) versus the corresponding *K*-means objective value ($y$-axis) for each choice of $K \in \{2, \ldots 15\}$. Use the **[sklearn.cluster.KMeans](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html)** object in your answer. Once you have called the _fit_ method on a _KMeans_ object you can recover the final objective value (at convergence) from the object's *intertia_* attribute (as in `kmeans_object.inertia_`).

If you have done this correctly, you should see a 'kink' at *K*=9, which makes sense given that the data was indeed generated from 9 clusters.

```python
# Your code here. Aim for 2-4 lines of code plus a few lines for plotting.
for i in range(2,16):
    plt.scatter(i, sklearn.cluster.KMeans(i).fit(X).inertia_)

plt.plot()

```

# 3. Fitting a Gaussian Mixture Model (GMM)

Exercises 3.1–3.4 ask you fit 2D data with scikit-learn's **[sklearn.mixture.GaussianMixture](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html)** object.

**Run the code cell below** to define a function that will help to visualize the placement and shape of GMM components.

```python
def plot_gmm(gmm):
    """
    Plots the placement of components in a Gaussian Mixture.

    The gmm object should be of type sklean.mixture.GaussianMixture
    """
    ax = plt.gca()
    for weight, mean, cov in zip(gmm.weights_, gmm.means_, gmm.covariances_):
        v, w = np.linalg.eigh(cov)
        v = 2*np.sqrt(2*v)
        u = w[0] / np.linalg.norm(w[0])
        angle = 180 * (1 + np.arctan(u[1]/u[0]) / np.pi)  # Ellipse() function needs degrees
        ell = mpl.patches.Ellipse(mean, v[0], v[1], angle, edgecolor='k',
                                  facecolor='none', linestyle='--', linewidth=2, alpha=0.8)
        ax.add_artist(ell)
```

### Exercise 3.1 — Fit a GMM to non-overlapping, isotropic data

**Write fitting code** to fit a **[sklearn.mixture.GaussianMixture](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html)** object to the synthetic data from Exercise 2.1. Use 9 mixture components in your answer. You should use the argument *random_state=0* to ensure reproducibility. Create a variable called _gmm_ that refers to your new *GaussianMixture* object.

```python
gmm = sklearn.mixture.GaussianMixture(9, random_state=0)
print(gmm)
```

    GaussianMixture(n_components=9, random_state=0)

**Write plotting code** using the *plot_gmm* function provided. Once you have the ellipses plotting correctly, use the **[predict](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html#sklearn.mixture.GaussianMixture.predict)** method of the *GaussianMixture* object and use them to colour the points in the scatter plot. Your plot should look similar to:

If you find that the GMM components do not fit perfectly to the data, try adding *n_init=5* to the arguments of your _GaussianMixture_ object. This will run the EM algorithm multiple times from different randomized initializations, and may give a higher chance of getting the 'right' clustering.

```python
# Modify the code below. Aim for 2 additional lines.
plot_gmm(gmm.fit(X))
plt.scatter(*X.T, s=5, c=gmm.predict(X))
plt.title('GMM fitted to anisotropic data')
plt.gca().set_aspect('equal')
```

### Exercise 3.2 — Generate overlapping, anisotropic synthetic GMM data

**Write some code** to generate 500 samples from 2-dimensional 3-component Gaussian mixture having the following parameters:

$$
\begin{align}
\pi_1 = 0.5, \quad &\boldsymbol{\mu}_1 = \begin{bmatrix} -2.0 & 0.0 \end{bmatrix}, &\boldsymbol{\Sigma}_1 = \begin{bmatrix}\phantom{-}2.0 & \phantom{-}1.0 \\ \phantom{-}1.0 & \phantom{-}2.0\end{bmatrix}\\
\pi_2 = 0.3, \quad &\boldsymbol{\mu}_2 = \begin{bmatrix} \phantom{-}2.0 & 0.0 \end{bmatrix}, &\boldsymbol{\Sigma}_2 = \begin{bmatrix}1.0 & -0.9 \\ -0.9 & 1.0\end{bmatrix}\\
\pi_3 = 0.2, \quad &\boldsymbol{\mu}_3 = \begin{bmatrix} \phantom{-}0.0 & 0.0 \end{bmatrix}, &\boldsymbol{\Sigma}_3 = \begin{bmatrix}10.0 & \phantom{-}0.0 \\ \phantom{-}0.0 & 10.0\end{bmatrix}\\
\end{align}
$$

Use the *sample_gmm* function from part 2. Create a variable called _X_ that refers to your samples, and create a variable called _c_ that refers to the mixture component indices (this value is directly returned by *sample_gmm*, so just store the result).

```python
np.random.seed(0)
means = np.array([
    [-2.0, 0.0],
    [2.0, 0.0],
    [0.0, 0.0]
])

covariance = np.array([
    [[2.0, 1.0], [1.0, 2.0]],
    [[1.0, -0.9], [-0.9, 1.0]],
    [[10.0, 0.0], [0.0, 10.0]]
])

weights = np.array([0.5, 0.3, 0.2])
X, c = sample_gmm(means=means, covs=covariance, weights=weights, N=500)
```

**Plot your data** by running the code cell below. The plot should look like this:

```python
assert 'X' in globals(), "You must create a variable called 'X' that refers to your samples!"
assert 'c' in globals(), "You must create a variable called 'c' that refers to your component indices!"
assert X.shape == (500,2), "X should have shape (500, 2)!"
assert c.shape == (500,), "c should have shape (500,)!"
plt.scatter(*X.T, c=c, s=5)     # Plot each point (X[i,0], X[i,1]) using colour index c[i] and point size 5
plt.title("Synthetic GMM data")
plt.xticks([-5, 0, 5])
plt.yticks([-5, 0, 5])
plt.gca().set_aspect('equal')   # It's important to see this data with equal scales on each axis
```

### Exercise 3.3 — Fit a GMM to the anisotropic, overlapping data

**Write fitting code** to fit a **[sklearn.mixture.GaussianMixture](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html)** object to the synthetic data you generated in Exercise 3.2. Use 3 mixture components in your answer. You should use the argument *random_state=0* to ensure reproducibility. Create a variable called _gmm_ that refers to your new *GaussianMixture* object.

```python
gmm = sklearn.mixture.GaussianMixture(3, random_state=0)
```

**Write plotting code** just as you did for Exercise 3.1 using the **[predict](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html#sklearn.mixture.GaussianMixture.predict)** method. Your plot should look like this:

```python
plot_gmm(gmm.fit(X))
plt.scatter(*X.T, s=5, c=c)
plt.title('GMM fitted to anisotropic data')
plt.gca().set_aspect('equal')
```

**Write plotting code** to visualize the probability $p(z_{ik} \mid \mathbf{x}_i)$ that component $k$ generated data point $\mathbf{x}_i$. Use the **[predict_proba](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html#sklearn.mixture.GaussianMixture.predict_proba)** method of the _GaussianMixture_ object to get the probabilities. Because there are $K=3$ components, you can visualize these probabilities by using $\begin{bmatrix} z_{i1} & z_{i2} & z_{i3} \end{bmatrix}$ as the values colour $\begin{bmatrix} r_i & g_i & b_i \end{bmatrix}$ for data point $i$. Your plot should look like this:

```python
plot_gmm(gmm.fit(X))
plt.scatter(*X.T, s=20, c=gmm.predict_proba(X))
plt.title('GMM fitted to anisotropic data')
plt.gca().set_aspect('equal')
```

### Exercise 3.4 — Sample from spatially clustered pixels using a GMM

**Write fitting code** to fit a **[sklearn.mixture.GaussianMixture](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html)** object to the _RGBXY_ data (the ladybug) that you generated in Exercise 3.2. Use **15** mixture components in your answer. You should use the argument *random_state=0* to ensure reproducibility. Create a variable called _gmm_ that refers to your new *GaussianMixture* object.

```python
gmm = sklearn.mixture.GaussianMixture(15, random_state=0).fit(RGBXY)
```

**Generate 5000 samples** from your fitted mixture model by using the **[sample](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html#sklearn.mixture.GaussianMixture.sample)** method of the _GaussianMixture_ object. The resulting sample matrix should have shape (5000,5) where each row is a vector $\begin{bmatrix}r & g & b & x & y\end{bmatrix}$. Create a variable _X_ to refer to your new samples. Note that the _sample_ function also returns a component membership vector, but you can ignore it.

```python
X, c = gmm.sample(5000)
print(X)
```

    [[ 3.46851326e-01 -9.02360738e-02 -2.43623256e-02  7.95827847e+01
       5.22453003e+01]
     [ 1.38434547e+00  1.96541881e-01  4.96918224e-02  7.19965214e+01
       4.97476077e+01]
     [ 4.79247157e-01  1.89137918e-01  1.10851799e-01  7.20397440e+01
       7.23885958e+01]
     ...
     [ 2.85009137e-01  2.93851123e-01  4.14639804e-01  3.44345202e+01
       4.05519970e+01]
     [ 8.55497336e-01  8.40602537e-01  9.04088935e-01  4.14135879e+01
       5.18982229e+01]
     [ 5.11923476e-01  5.04867657e-01  5.92077136e-01  2.38742913e+01
       5.89054347e+01]]

**Plot the samples** as if they were pixels at location $(x,y)$ with colour $(r,g,b)$. Use a single call to Matplotlib's _scatter_ function. Your ladybug should look something like this:

*Hint:* The _scatter_ function will fail if you use any colour values outside range $[0.0,1.0]$, yet there's no guarantee that a Gaussian will produce values within this range. Use the **[np.clip](https://numpy.org/devdocs/reference/generated/numpy.clip.html)** function to clamp the $(r,g,b)$ values to this range.

```python
plt.scatter(*X[:, 3:5].T, c=np.clip(X[:, 0:3], 0, 1), s=10)
# Keep these lines at the end.
plt.gca().invert_yaxis()         # This line makes sure the ladybug appears upright!
plt.gca().set_aspect('equal')    # This line makes sure the ladybug isn't squished!
```


## Atomic Units

Atomic Units are standardly used in Computatoinal Chemistry code packages as use of the meteric or imperial units will result in very tiny numerical values that can result in large numerical values when manipulated computationaly. Atomic Units are larger for the purposes of atomic values and constants providing improved computatoinal accuracy and performance.

Below Follows a list of Atomic Units and a description of each unit. a.u. stands for atomic units.

$$
\begin{align}
\hbar&=\frac{h}{2\pi} = 1 &&\mbox{Planck Constant represents the Angular Momentum and is set to 1 a.u.} \\
m_e &= 1 &&\mbox{Mass of an electron is set to 1 a.u.} \\
e &= 1 &&\mbox{Electron charge is set to 1 a.u.} \\
4\pi\epsilon_0 &= 1 &&\mbox{Permittivity of Free space is 1 a.u.} \\
a_0 &= \frac{\hbar^2}{me^2} = 1 &&\mbox{Length in Bohr Radius} \\
E_h &= 1 &&\mbox{Energy in Hartrees} \\
\end{align}
$$

All information in this notebook originates from Introduction to Computational Physical Chemistry,  by Joshua Schrier.


# Ferreira et.al. 2018
Following is its notation and description

|      Parameter       | description                                                  |
| :------------------: | :----------------------------------------------------------- |
|         $K$          | Total number of available price vectors                      |
|         $N$          | Total number of products                                    |
|         $M$          | Total number of kinds of resource                            |
|         $T$          | Total number of periods                                      |
|        $[x]$         | We define $[x]$ as a set, $[x]=\{1,2,\cdots, x\}$            |
|         $i$          | Index of products                                            |
|         $j$          | Index of resources                                           |
|         $t$          | Index of period                                              |
|     $I_j,I_j(t)$     | $I_j$ is the initial inventory for each resource $j\in [M]$, $I_j(t)$ is the inventory at the end of period $t$. $I_j(0)=I_j$ |
|       $a_{ij}$       | When we produced one unit item $i$, it would consume $a_{ij}$ unit $j$ |
|        $c_j$         | we define $c_j=\frac{I_j}{t}$                                |
|        $p_k$         | We define $\{p_1,p_2,\cdots,p_K\}$ as the admissible price vectors, each $p_k$ is a $N\times 1$ vector, specifying the price of each product, $p_k=(p_{1k},\cdots,p_{Nk})$, where $p_{ik}$ is the price of product $i$, for $i\in [N]$. We define $p_{\infty}$ as a "shut-off" price, such that the demand for any product under this price is zero. |
|        $P(t)$        | We denote by $P(t)=(P_1(t),\cdots,P_N(t))$ the prices chosen by the retailer in this period, and require that $P(t)\in \{p_1,p_2,\cdots,p_K,p_{\infty}\}$ |
|        $D(t)$        | We denote by $D(t) = (D_1(t),\cdots,D_N(t))$ the demand of each product at period $t$. We assume that given $P(t)=p_k$, the demand $D(t)$ is sampled from a probability distribution on $\mathbb{R}^{N}_+$ with joint cumulative distribution function (CDF) $F (x_1,\cdots,x_N, pk, \theta )$. $D(t)$ is independent of the history $\mathcal{H}_{t-1}$, given $P(t)$ |
|       $\theta$       | $\theta$ is the parameter of demand distribution, takes values in the parameter space $\Theta\subset\mathbb{R}^l$. The nature would sample $\theta$ from a prior distribution at the beginning form the process. The distribution is assumed to be subexponential; |
|  $\mathcal{H}_{t }$  | $\mathcal{H}_{t}=(P(1),D(1),\cdots,P(t),D(t))$               |
|       $\xi(t)$       | At the beginning of each period $t\in[T]$, the retailer observes some context $\xi(t)$, $\xi(t)$ belongs to some discrete set $\mathcal{X}$. We assume $\xi(t)$ is sampled i.i.d from a known distribution. |
 | $d_{ik}(\xi|\theta)$ | The mean demand of product $i\in [N]$ under price vector $p_k$, $\forall k\in[K]$, given context $\xi$ and parameter $\theta$ |

## 1 st algorithm

We implement the 4th algorithm of Ferreira et.al. 2018

The most difficult task in the implementation of this algorithm is the update rule of posterior. In fact, for most of the prior distribution, it is nearly impossible to derive the explict expression of posterior distribution. Here, we would like to simplify this process.

We generate K * N matrix from beta(1, 1), in other words, uniform distribution in $[0, 1]$. Each entry corresponds to a product in each pricing vector. We let $d_{k,i}$ denote the demand of $i^{th}$ product given $k^{th}$ pricing vector. We assume
$$
\begin{align}
Pr(d_{k,i} = 1) & = \Theta_{k,i}\\
Pr(d_{k,i} = 0) & = 1 - \Theta_{k,i}\\
\end{align}
$$

In this case, it would be quite easy to update the posterior distribution. Assume we adopt $k^{th}$ price vector $p_k$ in period $t_1, t_2, \cdots, t_\tau$,

For each $k\in [K], i\in [N]$, the poseterior density function of $\Theta_{k,i}$ is
$$
\begin{align}
f(\theta_{k,i} | (p^{(t_1)}_{k},d^{(t_1)}_k),\cdots, (p^{(t_\tau)}_{k},d^{(t_\tau)}_k))
& = \frac{Pr(d^{(t_1)}_k,\cdots, ,d^{(t_\tau)}_k \ | \ \theta_{k,i}) * 1}{\int_{0}^{1}Pr(d^{(t_1)}_k,\cdots, ,d^{(t_\tau)}_k \ | \ \theta)*1d\theta}\\
&=\frac{\prod_{j=1}^\tau Pr(d^{(t_j)}_k| \ \theta_{k,i}) * 1}{\int_{0}^{1}\prod_{j=1}^\tau Pr(d^{(t_j)}_k| \ \theta)*1d\theta}\\
&=\frac{\theta_{k,i}^{\sum_{j=1}^\tau d^{(t_j)}_k} (1-\theta_{k,i})^{\tau-\sum_{j=1}^\tau d^{(t_j)}_k}}{\int_{0}^{1}\theta^{\sum_{j=1}^\tau d^{(t_j)}_k} (1-\theta)^{\tau-\sum_{j=1}^\tau d^{(t_j)}_k}d\theta}\\
&=beta(\sum_{j=1}^\tau d^{(t_j)}_k, \tau-\sum_{j=1}^\tau d^{(t_j)}_k)
\end{align}
$$

$d^{(i)}_j$ is the demand of $j^{th}$ product in $i^{th}$ period

Well, there are also tough problems. That is, there would be counter-intuitive random number. For example, **higher price lead to higher demand**. But anyway, I tried my best to find a suitable prior distribution, but I failed to do so. So I decided to just use this easier one.

```python
%reset -f
import numpy as np
import pandas as pd # we would restore our result in csv file .\\Result_Ferreira et.al. 2018 Algorithm 1\\
import datetime

import pyscipopt
from pyscipopt import quicksum

random_seed = 888
np.random.seed(random_seed)
```

```python
# Generate parameters
# the maximum value of np.random.randint(low, high) would be high - 1
K = np.random.randint(low = 2, high = 10) # Total number of available price vectors,
N = np.random.randint(low = 2, high = 10) # Total number of products
M = np.random.randint(low = 2, high = 10) # Total number of kinds of resource
T = 1000 # Total number of periods

# each row represent an admissible pricing strategy
P_list = np.float64(np.random.randint(low = 1, high = 10, size = (K, N)))

# initialize inventory
# c is the limitation of resources in each round
c = np.random.randint(low = 10, high = 15, size = M)
I_0 = c * T

# initialize a_ij
# A is the cost of each product
A = np.float64(np.random.randint(low = 10, high = 15, size = (N,M)))

# initialize real parameter theta
theta = np.random.beta(a = 200, b = 100, size = (K,N))
```

```python
class FerreiraAgent:
    # this agent would adopt the 1st algorithm
    def __init__(self, K, N, M, T, P_list, A, I_0):
        self.K = K
        self.N = N
        self.M = M
        self.T = T
        self.P_list = P_list
        self.A = A
        self.I_0 = I_0

        # initialize history
        self.H_P = np.zeros(shape = T) # the index of pricing vector we used in each period
        self.H_D = np.zeros(shape = (T, N)) # the demand of products in each period
        self.H_I = np.zeros(shape = (T + 1, M)) # avaliable remained inventory in each period
        self.H_I[0, :] = np.float64(I_0)
        self.H_reward = np.zeros(T) # the reward in each period
        self.H_bestX = np.zeros(shape = (T, K + 1)) # the best solution in each optimization

        # each realization of price vector, index of period,
        # corresponds to a estimate of theta
        self.H_alpha = np.zeros(shape = (T + 1,K,N))
        self.H_beta = np.zeros(shape = (T + 1,K,N))
        self.H_alpha[0, :, :] = 1 * np.ones(shape = (K,N))
        self.H_beta[0, :, :] = 1 * np.ones(shape = (K,N))

        # initialize the constraint value in each round
        # M kinds of resources correspond to M constraints, and one more constraint is x1 + ... + xN <=1
        self.H_constraint_value = np.zeros(shape = (T, M + 1))

        # estimated theta in each round
        self.H_theta = np.zeros(shape = (T,K,N))

        # initialize the index of period
        self.t = 1

        # initialize the average consumption of resource
        self.c = I_0 / T

    def Action(self):
        # vectorize beta sample function to accelerate
        mybeta = np.vectorize(np.random.beta)

        if not all(self.H_I[self.t - 1] >= 0):
            # resource are not enough
            price_offered_index = self.K + 1
            self.H_P[self.t - 1] = price_offered_index # record the index of offered price
            return price_offered_index

        # first step, sample from posterior distribution
        # H_alpha[t-1, :, :], H_beta[t-1, :, :] is the history data from 0 to t
        # H_theta[t-1, :, :] is the sample theta we used in round t
        self.H_theta[t-1, :, :] = mybeta(self.H_alpha[t-1, :, :], self.H_beta[t-1, :, :])

        # first step, calculate the mean demand given sample theta
        demand_mean = self.H_theta[t-1, :, :]

        # second step, optimize a linear function
        model = pyscipopt.Model("Optimization in Rount {:d}".format(t))
        # generate decision variable
        x = {}
        for xindex in range(1, K+1):
            x[xindex] = model.addVar(vtype="C", lb = 0, ub = 1, name="x{:d}".format(xindex))

        # second step, generate object function
        obj_coefficient = np.sum(demand_mean * self.P_list, axis = 1)# obj_coefficient[k] = $\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1}$
        model.setObjective(quicksum(x[xindex]*obj_coefficient[xindex-1] for xindex in range(1, K+1)), "maximize")
        # objective = $\sum_{k=1}^K(\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1})x_{k}$

        # second step, add constraint x_1+...+x_k<=1
        constraint_index = {}
        constraint_index[0] = model.addCons(quicksum(x[xindex] for xindex in range(1, K+1)) <= 1)

        # second step, for each resources, we require \sum_{k=1}^K\sum_{i=1}^N d_{i,k}a_{i,j}x_l<=c_j
        for jj in range(1, M+1):
            con_coefficient = self.A[:, jj - 1].dot(np.transpose(demand_mean))# con_coefficient[k] = $\sum_{i=1}^N a_{i,j}d_{i,k+1}$
            constraint_index[jj] = model.addCons(quicksum(x[xindex] * con_coefficient[xindex - 1] for xindex in range(1, K+1)) <= self.c[jj - 1])

        # second step, optimize the problem
        model.optimize()
        bestx = np.zeros(K+1) # p_{K+1} would force the demand be zero
        for xindex in range(1,K+1):
            bestx[xindex - 1] = model.getVal(x[xindex])
        bestx[K] = 1 - np.sum(bestx[0:K])
        eliminate_error = lambda x: 0 if np.abs(x) < 1e-10 else x #there would be numerical error in the best solution
        bestx = np.array([eliminate_error(x) for x in bestx])
        bestx = bestx / np.sum(bestx)

        # third step, offer price
        price_offered_index = np.random.choice(np.arange(1, K + 2), p = bestx)

        # fourth step, update estimate of parameter
        self.H_P[self.t - 1] = price_offered_index # record the index of offered price

        # fourth step, record the constraint value in optimization
        self.H_constraint_value[self.t - 1, 0] = np.sum(bestx[0:K])
        for jj in range(1, M+1):
            con_coefficient = np.array(list(model.getValsLinear(constraint_index[jj]).values()))
            self.H_constraint_value[self.t - 1, jj] = np.sum(bestx[0:K] * con_coefficient)

        # fourth step, record the optimal solution in this round
        self.H_bestX[self.t - 1, :] = bestx

        return price_offered_index

    def Update(self, demand):
        # record the realized demand
        self.H_D[self.t - 1, :] = demand

        # record the reward
        price_offered_index = np.int64(self.H_P[self.t - 1])# record the index of offered price
        if price_offered_index < K + 1:
            self.H_reward[self.t - 1] = self.P_list[price_offered_index - 1, :].dot(self.H_D[self.t - 1, :])
        else: # the demand must be zero
            self.H_reward[self.t - 1] = 0

        # update the remaining inventory
        self.H_I[self.t] = self.H_I[self.t - 1] - np.transpose(self.A).dot(self.H_D[self.t - 1, :])

        # update the estimation of alpha and beta
        if price_offered_index < K + 1:
            # if demand = 1, then alpha plus 1; if demand = 0, then alpha remain unchanged
            self.H_alpha[self.t, :, :] = self.H_alpha[self.t - 1, :, :]
            self.H_alpha[self.t, price_offered_index - 1, :] = self.H_alpha[self.t, price_offered_index - 1, :] + self.H_D[self.t - 1, :]

            # if demand = 1, then beta remained unchanged; if demand = 0, then beta plus 1
            self.H_beta[self.t, :, :] = self.H_beta[self.t - 1, :, :]
            self.H_beta[self.t, price_offered_index - 1, :] = self.H_beta[self.t - 1, price_offered_index - 1, :] + np.ones(N) - self.H_D[self.t - 1, :]
        else: # the demand must be zero, then all the estimate remain unchanged
            self.H_alpha[t, :, :] = self.H_alpha[t - 1, :, :]
            self.H_beta[t, :, :] = self.H_beta[t - 1, :, :]

        # update the index of period
        self.t = self.t + 1
```

```python
class UniformAgent:
    # this agent would uniformly pick pricing vector
    def __init__(self, K, N, M, T, P_list, A, I_0):
        self.K = K
        self.N = N
        self.M = M
        self.T = T
        self.P_list = P_list
        self.A = A
        self.I_0 = I_0

        # initialize history
        self.H_P = np.zeros(shape = T) # the index of pricing vector we used in each period
        self.H_D = np.zeros(shape = (T, N)) # the demand of products in each period
        self.H_I = np.zeros(shape = (T + 1, M)) # avaliable remained inventory in each period
        self.H_I[0, :] = np.float64(I_0)
        self.H_reward = np.zeros(T) # the reward in each period

        # initialize the index of period
        self.t = 1

    def Action(self):
        # vectorize beta sample function to accelerate
        mybeta = np.vectorize(np.random.beta)

        if not all(self.H_I[self.t - 1] >= 0):
            # resource are not enough
            price_offered_index = self.K + 1
            self.H_P[self.t - 1] = price_offered_index # record the index of offered price
            return price_offered_index

        price_offered_index = np.random.randint(low = 1, high = self.K + 2)
        # the maximum value of np.random.randint(low, high) would be high - 1
        self.H_P[self.t - 1] = price_offered_index
        return price_offered_index

    def Update(self, demand):
        # record the realized demand
        self.H_D[self.t - 1, :] = demand

        # record the reward
        price_offered_index = np.int64(self.H_P[self.t - 1])# record the index of offered price
        if price_offered_index < K + 1:
            self.H_reward[self.t - 1] = self.P_list[price_offered_index - 1, :].dot(self.H_D[self.t - 1, :])
        else: # the demand must be zero
            self.H_reward[self.t - 1] = 0

        # update the remaining inventory
        self.H_I[self.t] = self.H_I[self.t - 1] - np.transpose(self.A).dot(self.H_D[self.t - 1, :])

        # update the index of period
        self.t = self.t + 1
```

```python
class Environment:
    # we use an object as environment to cooperate with agents
    def __init__(self, theta, K, N):
        self.theta = theta
        self.K = K
        self.N = N

    def Response(self, price_offered_index):
        # vectorize binomial sample function to accelerate
        mybinomial = np.vectorize(np.random.binomial)
        if price_offered_index < self.K + 1:
            demand = mybinomial(np.ones(self.N), theta[price_offered_index - 1, :]) # record the realization of demand
        else: # the demand must be zero
            demand = np.zeros(self.N)
        return demand
```

```python
# Carry on experiments
random_seed = 12345
np.random.seed(random_seed)

exp_num = 5

rewardferreira = np.zeros(shape = (exp_num, T))
rewarduniform = np.zeros(shape = (exp_num, T))

for exp_index in range(exp_num):
    ferreiraagent = FerreiraAgent(K, N, M, T, P_list, A, I_0)
    env = Environment(theta, K, N)
    for t in range(1, T+1):
        price_offered_index = ferreiraagent.Action()
        demand = env.Response(price_offered_index)
        ferreiraagent.Update(demand)
    rewardferreira[exp_index, :] = ferreiraagent.H_reward

    uniformagent = UniformAgent(K, N, M, T, P_list, A, I_0)
    env = Environment(theta, K, N)
    for t in range(1, T+1):
        price_offered_index = uniformagent.Action()
        demand = env.Response(price_offered_index)
        uniformagent.Update(demand)
    rewarduniform[exp_index, :] = uniformagent.H_reward

rewardferreira = np.mean(rewardferreira, axis = 0)
rewarduniform = np.mean(rewarduniform, axis = 0)
```

```python
# calculate the benchmark
model = pyscipopt.Model("Optimization in Rount {:d}".format(t))

x = {}
for xindex in range(1, K+1):
    x[xindex] = model.addVar(vtype="C", lb = 0, ub = 1, name="x{:d}".format(xindex))

# Generate object function
demand_mean = theta
obj_coefficient = np.sum(demand_mean *  P_list, axis = 1)# obj_coefficient[k] = $\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1}$
model.setObjective(quicksum(x[xindex]*obj_coefficient[xindex-1] for xindex in range(1, K+1)), "maximize")
# objective = $\sum_{k=1}^K(\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1})x_{k}$

# add constraint x_1+...+x_k<=1
constraint_index = {}
constraint_index[0] = model.addCons(quicksum(x[xindex] for xindex in range(1, K+1)) <= 1)

# add, constraint for each resources, we require \sum_{k=1}^K\sum_{i=1}^N d_{i,k}a_{i,j}x_l<=c_j
for jj in range(1, M+1):
    con_coefficient = A[:, jj - 1].dot(np.transpose(demand_mean))# con_coefficient[k] = $\sum_{i=1}^N a_{i,j}d_{i,k+1}$
    constraint_index[jj] = model.addCons(quicksum(x[xindex] * con_coefficient[xindex - 1] for xindex in range(1, K+1)) <= c[jj - 1])

# optimize the problem
model.optimize()

# get the best solution
bestx = np.zeros(K+1) # p_{K+1} would force the demand be zero
for xindex in range(1,K+1):
    bestx[xindex - 1] = model.getVal(x[xindex])
bestx[K] = 1 - np.sum(bestx[0:K])
eliminate_error = lambda x: 0 if np.abs(x) < 1e-10 else x #there would be numerical error in the best solution
bestx = np.array([eliminate_error(x) for x in bestx])
bestx = bestx / np.sum(bestx)

Opt = model.getObjVal()
```

```python
import matplotlib.pyplot as plt

bestreward = Opt * np.ones(T)
plt.figure()
plt.plot(range(1, T + 1), np.cumsum(rewardferreira))
plt.plot(range(1, T + 1), np.cumsum(rewarduniform))
plt.plot(range(1, T + 1), np.cumsum(bestreward))
plt.xlabel("index of round")
plt.ylabel("total reward")
plt.legend(["Ferreira Algorithm", "Uniform", "Benchmark"])
```

```python
# bestreward = Opt * np.ones(T)
# plt.figure()
# plt.plot(range(1, T + 1), np.cumsum(ferreiraagent.H_reward))
# plt.plot(range(1, T + 1), np.cumsum(uniformagent.H_reward))
# plt.plot(range(1, T + 1), np.cumsum(bestreward))
# plt.xlabel("index of round")
# plt.ylabel("total reward")
# plt.legend(["Ferreira Algorithm", "Uniform", "Benchmark"])
```

```python
plt.figure()
plt.plot(range(10, T), np.cumsum(rewardferreira)[10:T] / np.cumsum(bestreward)[10:T])
plt.plot(range(10, T), np.cumsum(rewarduniform)[10:T] / np.cumsum(rewardferreira)[10:T])
plt.legend(["Ferreira / Best", "Uniform / Ferreira"])
plt.plot(range(10, T), np.ones(T)[10:T])
plt.plot(range(10, T), 0.9 * np.ones(T)[10:T])
plt.show()
```

```python
np.sum(rewardferreira)
```

    4704.6

```python
Opt * T
```

    4758.011663603385

t, M, N, K
constraint_index
model.getValsLinear(constraint_index [1]),
model.getValsLinear(constraint_index [2])
x
model.getVal(x[1]), model.getVal(x[2]), model.getVal(x[3])
bestx
0.530018723612461 * 10.376003106593089, 0.530018723612461 * 17.74088269167503,
c
model.getValsLinear(constraint_index [1])
model.getValsLinear(constraint_index [2])
model.getConss()
model.getNConss()
model.getValsLinear(model.getConss()[0])
model.getValsLinear(model.getConss()[1])

```python
# generate the csv data file and restore it into csv file
from datetime import datetime
moment = str(datetime.now()).replace(':', '-').replace(' ', '_')[:-7]
# the value of moment:  '2021-10-21_11-39-24', we used this varibale to name our files

# we firstly input the parameter into txt file
prameter_filename = moment + "_K-{:d}_N-{:d}_M-{:d}_T-{:d}_randomseed-{:d}_ParameterList.txt".format(K, N, M, T, random_seed)
with open(".\\Result_Ferreira et.al. 2018 Algorithm 1\\" + prameter_filename, 'w') as f:
    f.write("K = {:d}\n".format(K))
    f.write("N = {:d}\n".format(N))
    f.write("M = {:d}\n".format(M))
    f.write("randomseed = {:d}\n".format(random_seed))

    f.write("-----Admissible Prive Vector------\n")
    for kindex in range(K):
        f.write("Price vector {:d} : {:s}\n".format(kindex, str(P_list[kindex, :])))

    f.write("-----Initial Inventory------\n")
    f.write("Initial Inventory: {:s}\n".format(str(I_0)))

    f.write("-----Resource Consumption------\n")
    for nindex in range(N):
        f.write("Cost of resource of product {:d}: {:s}\n".format(nindex + 1, str(A[nindex, :])))

    f.write("-----Resouce Constraint in each round------\n")
    for nindex in range(N):
        f.write("c : {:s}\n".format(str(c)))

    f.write("-----Real theta------\n")
    for kindex in range(K):
        f.write("Given price vector {:d}, the real theta is {:s}\n".format(kindex + 1, str(theta[kindex, :])))

# then we input the experiment into csc file, we use period as index
data = pd.DataFrame({"Pricing_index": H_P, "Reward": H_reward})
data['Period_index'] = range(1, T+1)
# add the demand of each product
data = pd.concat([data, pd.DataFrame(H_D, columns = ["product_{:d}_demand".format(nindex) for nindex in range(1,N+1)])], axis = 1)
# add the remain invetory of each resource
data = pd.concat([data, pd.DataFrame(H_I[0:T, :], columns = ["resouce_{:d}".format(mindex) for mindex in range(1,M+1)])], axis = 1)
# add the constraint value
data = pd.concat([data, pd.DataFrame(H_constraint_value[0:T, :], columns = ["constraint_{:d}".format(conindex) for conindex in range(0,M+1)])], axis = 1)
# add the best solution in each round
data = pd.concat([data,
                  pd.DataFrame(H_bestX,\
                              columns = ["use_price_index_{:d}".format(kindex) for kindex in range(1,K+1)] + ["use_infinite_price"])],
                  axis = 1)
# add the estimation of alpha
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_alpha[0:T, :, :], newshape =(T, K * N)),
                             columns = ["alpha_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
# add the estimation of alpha
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_beta[0:T, :, :], newshape =(T, K * N)),
                             columns = ["beta_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
# add the estimation of theta
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_theta[0:T, :, :], newshape =(T, K * N)),
                             columns = ["theta_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
exp_filename = moment + "_K-{:d}_N-{:d}_M-{:d}_T-{:d}_randomseed-{:d}_Experiment.csv".format(K, N, M, T, random_seed)
data.to_csv(".\\Result_Ferreira et.al. 2018 Algorithm 1\\" + exp_filename, index = False)
```

```python
# clear the memory
H_reward = np.zeros(T) # the reward in each period
H_bestX = np.zeros(shape = (T, K + 1)) # the best solution in each optimization
H_P = np.zeros(shape = T) # the index of pricing vector we used in each period
H_D = np.zeros(shape = (T, N)) # the demand of products in each period
H_I = np.zeros(shape = (T + 1, M)) # avaliable inventory in each period
H_I[0, :] = np.float64(I_0)
```

```python
# if we replace step 1 with true theta
for t in range(1, T+1):
    # first step, sample from posterior distribution
    # H_alpha[t-1, :, :], H_beta[t-1, :, :] is the history data from 0 to t
    # H_theta[t-1, :, :] is the sample theta we used in round t
#     H_theta[t-1, :, :] = mybeta(H_alpha[t-1, :, :], H_beta[t-1, :, :])

    # first step, calculate the mean demand given sample theta
    demand_mean = theta

    # second step, optimize a linear function
    model = pyscipopt.Model("Optimization in Rount {:d}".format(t))
    # generate decision variable
    x = {}
    for xindex in range(1, K+1):
        x[xindex] = model.addVar(vtype="C", lb = 0, ub = 1, name="x{:d}".format(xindex))

    # second step, generate object function
    obj_coefficient = np.sum(demand_mean *  P_list, axis = 1)# obj_coefficient[k] = $\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1}$
    model.setObjective(quicksum(x[xindex]*obj_coefficient[xindex-1] for xindex in range(1, K+1)), "maximize")
    # objective = $\sum_{k=1}^K(\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1})x_{k}$

#     # second step, add constraint x_1+...+x_k<=1
#     model.addCons(quicksum(x[xindex] for xindex in range(1, K+1)) <= 1)

#     # second step, for each resources, we require \sum_{k=1}^K\sum_{i=1}^N d_{i,k}a_{i,j}x_l<=c_j
#     for jj in range(1, M+1):
#         # size(A) = [N, M], size(demand_mean) = [K, N]
#         con_coefficient = A[:, jj - 1].dot(np.transpose(demand_mean))# con_coefficient[k] = $\sum_{i=1}^N a_{i,j}d_{i,k+1}$
#         model.addCons(quicksum(x[xindex] * con_coefficient[xindex - 1] for xindex in range(1, K+1)) <= c[jj - 1])

#     # second step, optimize the problem
#     model.optimize()
#     bestx = np.zeros(K+1) # p_{K+1} would force the demand be zero
#     for xindex in range(1,K+1):
#         bestx[xindex - 1] = model.getVal(x[xindex])
#     bestx[K] = 1 - np.sum(bestx[0:K])
#     eliminate_error = lambda x: 0 if np.abs(x) < 1e-10 else x #there would be numerical error in the best solution
#     bestx = np.array([eliminate_error(x) for x in bestx])
#     bestx = bestx / np.sum(bestx)

#     # third step, offer price
#     price_offered_index = np.random.choice(np.arange(1, K + 2), p = bestx)

    # second step, add constraint x_1+...+x_k<=1
    constraint_index = {}
    constraint_index[0] = model.addCons(quicksum(x[xindex] for xindex in range(1, K+1)) <= 1)

    # second step, for each resources, we require \sum_{k=1}^K\sum_{i=1}^N d_{i,k}a_{i,j}x_l<=c_j
    for jj in range(1, M+1):
        # size(A) = [N, M], size(demand_mean) = [K, N]
        con_coefficient = A[:, jj - 1].dot(np.transpose(demand_mean))# con_coefficient[k] = $\sum_{i=1}^N a_{i,j}d_{i,k+1}$
        constraint_index[jj] = model.addCons(quicksum(x[xindex] * con_coefficient[xindex - 1] for xindex in range(1, K+1)) <= c[jj - 1])

    # second step, optimize the problem
    model.optimize()
    bestx = np.zeros(K+1) # p_{K+1} would force the demand be zero
    for xindex in range(1,K+1):
        bestx[xindex - 1] = model.getVal(x[xindex])
    bestx[K] = 1 - np.sum(bestx[0:K])
    eliminate_error = lambda x: 0 if np.abs(x) < 1e-10 else x #there would be numerical error in the best solution
    bestx = np.array([eliminate_error(x) for x in bestx])
    bestx = bestx / np.sum(bestx)

    # third step, offer price
    price_offered_index = np.random.choice(np.arange(1, K + 2), p = bestx)

    # fourth step, update estimate of parameter
    H_P[t - 1] = price_offered_index # record the index of offered price

    # fourth step, record the constraint value in optimization
    H_constraint_value[t - 1, 0] = np.sum(bestx[0:K])
    for jj in range(1, M+1):
        con_coefficient = np.array(list(model.getValsLinear(constraint_index[jj]).values()))
        H_constraint_value[t - 1, jj] = np.sum(bestx[0:K] * con_coefficient)

    # fourth step, update estimate of parameter
    H_P[t - 1] = price_offered_index # record the index of offered price

    # fourth step, record the optimal solution in this round
    H_bestX[t - 1, :] = bestx

    # fourth step, record the realization of demand
    if price_offered_index < K + 1:
        H_D[t - 1, :] = mybinomial(np.ones(N), theta[price_offered_index - 1, :]) # record the realization of demand
    else: # the demand must be zero
        H_D[t - 1, :] = np.zeros(N)

    # fourth step, record the reward in this period
    if price_offered_index < K + 1:
        H_reward[t - 1] = P_list[price_offered_index - 1, :].dot(H_D[t - 1, :])  # record the realization of demand
    else: # the demand must be zero
        H_reward[t - 1] = 0

    # fourth step, record the remain inventory, size(A) = [N, M]
    H_I[t, :] = H_I[t - 1, :] - np.transpose(A).dot(H_D[t - 1, :])
    if not all(H_I[t, :] >= 0):
        break

print("Total reward of known distribution: {:f}".format(np.sum(H_reward)))
```

    Total reward of known distribution: 4254.000000

```python
# then input the experiment into csc file, we use period as index
data = pd.DataFrame({"Pricing_index": H_P, "Reward": H_reward})
data['Period_index'] = range(1, T+1)
# add the demand of each product
data = pd.concat([data, pd.DataFrame(H_D, columns = ["product_{:d}_demand".format(nindex) for nindex in range(1,N+1)])], axis = 1)
# add the remain invetory of each resource
data = pd.concat([data, pd.DataFrame(H_I[0:T, :], columns = ["resouce_{:d}".format(mindex) for mindex in range(1,M+1)])], axis = 1)
# add the constraint value
data = pd.concat([data, pd.DataFrame(H_constraint_value[0:T, :], columns = ["constraint_{:d}".format(conindex) for conindex in range(0,M+1)])], axis = 1)
# add the best solution in each round
data = pd.concat([data,
                  pd.DataFrame(H_bestX,\
                              columns = ["use_price_index_{:d}".format(kindex) for kindex in range(1,K+1)] + ["use_infinite_price"])],
                  axis = 1)
# add the estimation of alpha
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_alpha[0:T, :, :], newshape =(T, K * N)),
                             columns = ["alpha_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
# add the estimation of alpha
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_beta[0:T, :, :], newshape =(T, K * N)),
                             columns = ["beta_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
# add the estimation of theta
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_theta[0:T, :, :], newshape =(T, K * N)),
                             columns = ["theta_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
exp_filename = moment + "_K-{:d}_N-{:d}_M-{:d}_T-{:d}_randomseed-{:d}_RealThetaExperiment.csv".format(K, N, M, T, random_seed)
data.to_csv(".\\Result_Ferreira et.al. 2018 Algorithm 1\\" + exp_filename, index = False)
```

## 2 nd algorithm

We implement the 4th algorithm of Ferreira et.al. 2018

$c_j(t) = \frac{I_j(t-1)}{T-t+1}$

```python
%reset -f
import numpy as np
import pandas as pd

import pyscipopt
from pyscipopt import quicksum

random_seed = 12345
np.random.seed(random_seed)
```

```python
# Generate parameters
K = np.random.randint(low = 1, high = 5) # Total number of available price vectors
N = np.random.randint(low = 1, high = 5) # Total number of products
M = np.random.randint(low = 1, high = 5) # Total number of kinds of resource
T = 1000 # Total number of periods

# generate the demand

# each row represent an admissible pricing strategy
P_list = np.float64(np.random.randint(low = 1, high = 10, size = (K, N)))

# initialize inventory
I_0 = np.float64(np.random.randint(low = 9000, high = 11000, size = M))
c = np.zeros(shape = (T, M)) # we would update c in each period

# initialize a_ij
A = np.float64(np.random.randint(low = 5, high = 15, size = (N,M)))

# initialize real parameter theta
theta = np.random.beta(a = 1, b = 1, size = (K,N))

# initialize history
H_P = np.zeros(shape = T) # the index of pricing vector we used in each period
H_D = np.zeros(shape = (T, N)) # the demand of products in each period

H_I = np.zeros(shape = (T + 1, M)) # avaliable inventory in each period
H_I[0, :] = np.float64(I_0)

H_bestX = np.zeros(shape = (T, K + 1)) # the best solution in each optimization

H_reward = np.zeros(T) # the reward in each period

# each realization of price vector, index of period,
# corresponds to a estimate of theta
H_alpha = np.zeros(shape = (T + 1,K,N))
H_beta = np.zeros(shape = (T + 1,K,N))
H_alpha[0, :, :] = 1*np.ones(shape = (K,N))
H_beta[0, :, :] = 1*np.ones(shape = (K,N))

# initialize the constraint value in each round
# M kinds of resources correspond to M constraints, and one more constraint is x1 + ... + xN <=1
H_constraint_value = np.zeros(shape = (T, M + 1))

# vectorize beta sample function to accelerate
mybeta = np.vectorize(np.random.beta)
H_theta = np.zeros(shape = (T,K,N))

# vectorize binomial sample function to accelerate
mybinomial = np.vectorize(np.random.binomial)
```

```python
# implementation of algorithm 2
for t in range(1, T+1):
    # first step, sample from posterior distribution
    # H_alpha[t-1, :, :], H_beta[t-1, :, :] is the history data from 0 to t
    # H_theta[t-1, :, :] is the sample theta we used in round t
    H_theta[t-1, :, :] = mybeta(H_alpha[t-1, :, :], H_beta[t-1, :, :])

    # first step, calculate the mean demand given sample theta
    demand_mean = H_theta[t-1, :, :]

    # second step, optimize a linear function
    model = pyscipopt.Model("Optimization in Rount {:d}".format(t))
    # generate decision variable
    x = {}
    for xindex in range(1, K+1):
        x[xindex] = model.addVar(vtype="C", lb = 0, ub = 1, name="x{:d}".format(xindex))

    # second step, generate object function
    obj_coefficient = np.sum(demand_mean *  P_list, axis = 1)# obj_coefficient[k] = $\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1}$
    model.setObjective(quicksum(x[xindex]*obj_coefficient[xindex-1] for xindex in range(1, K+1)), "maximize")
    # objective = $\sum_{k=1}^K(\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1})x_{k}$

#     # second step, add constraint x_1+...+x_k<=1
#     model.addCons(quicksum(x[xindex] for xindex in range(1, K+1)) <= 1)

#     # second step, for each resources, we require \sum_{k=1}^K\sum_{i=1}^N d_{i,k}a_{i,j}x_l<=c_j
#     c[t - 1, :] = H_I[t - 1, :] / (T - t + 1)
#     for jj in range(1, M+1):
#         # size(A) = [N, M], size(demand_mean) = [K, N]
#         con_coefficient = A[:, jj - 1].dot(np.transpose(demand_mean))# con_coefficient[k] = $\sum_{i=1}^N a_{i,j}d_{i,k+1}$
#         model.addCons(quicksum(x[xindex] * con_coefficient[xindex - 1] for xindex in range(1, K+1)) <= c[t - 1, jj - 1])

#     # second step, optimize the problem
#     model.optimize()
#     bestx = np.zeros(K+1) # p_{K+1} would force the demand be zero
#     for xindex in range(1,K+1):
#         bestx[xindex - 1] = model.getVal(x[xindex])
#     bestx[K] = 1 - np.sum(bestx[0:K])
#     eliminate_error = lambda x: 0 if np.abs(x) < 1e-10 else x #there would be numerical error in the best solution
#     bestx = np.array([eliminate_error(x) for x in bestx])
#     bestx = bestx / np.sum(bestx)

#     # third step, offer price
#     price_offered_index = np.random.choice(np.arange(1, K + 2), p = bestx)

    # second step, add constraint x_1+...+x_k<=1
    constraint_index = {}
    constraint_index[0] = model.addCons(quicksum(x[xindex] for xindex in range(1, K+1)) <= 1)

    # second step, for each resources, we require \sum_{k=1}^K\sum_{i=1}^N d_{i,k}a_{i,j}x_l<=c_j
    c[t - 1, :] = H_I[t - 1, :] / (T - t + 1)
    for jj in range(1, M+1):
        # size(A) = [N, M], size(demand_mean) = [K, N]
        con_coefficient = A[:, jj - 1].dot(np.transpose(demand_mean))# con_coefficient[k] = $\sum_{i=1}^N a_{i,j}d_{i,k+1}$
        constraint_index[jj] = model.addCons(quicksum(x[xindex] * con_coefficient[xindex - 1] for xindex in range(1, K+1)) <= c[t - 1, jj - 1])

    # second step, optimize the problem
    model.optimize()
    bestx = np.zeros(K+1) # p_{K+1} would force the demand be zero
    for xindex in range(1,K+1):
        bestx[xindex - 1] = model.getVal(x[xindex])
    bestx[K] = 1 - np.sum(bestx[0:K])
    eliminate_error = lambda x: 0 if np.abs(x) < 1e-10 else x #there would be numerical error in the best solution
    bestx = np.array([eliminate_error(x) for x in bestx])
    bestx = bestx / np.sum(bestx)

    # third step, offer price
    price_offered_index = np.random.choice(np.arange(1, K + 2), p = bestx)

    # fourth step, update estimate of parameter
    H_P[t - 1] = price_offered_index # record the index of offered price

    # fourth step, record the constraint value in optimization
    H_constraint_value[t - 1, 0] = np.sum(bestx[0:K])
    for jj in range(1, M+1):
        con_coefficient = np.array(list(model.getValsLinear(constraint_index[jj]).values()))
        H_constraint_value[t - 1, jj] = np.sum(bestx[0:K] * con_coefficient)

    # fourth step, update estimate of parameter
    H_P[t - 1] = price_offered_index # record the index of offered price

    # fourth step, record the optimal solution in this round
    H_bestX[t - 1, :] = bestx

    # fourth step, record the realization of demand
    if price_offered_index < K + 1:
        H_D[t - 1, :] = mybinomial(np.ones(N), theta[price_offered_index - 1, :]) # record the realization of demand
    else: # the demand must be zero
        H_D[t - 1, :] = np.zeros(N)

    # fourth step, record the reward in this period
    if price_offered_index < K + 1:
        H_reward[t - 1] = P_list[price_offered_index - 1, :].dot(H_D[t - 1, :])  # record the realization of demand
    else: # the demand must be zero
        H_reward[t - 1] = 0

    # fourth step, record the remain inventory, size(A) = [N, M]
    H_I[t] = H_I[t - 1] - np.transpose(A).dot(H_D[t - 1, :])
    if not all(H_I[t] >= 0):
        break

    # fourth step, record the new estimate of alpha and beta
    if price_offered_index < K + 1:
        # if demand = 1, then alpha plus 1; if demand = 0, then alpha remain unchanged
        H_alpha[t, :, :] = H_alpha[t - 1, :, :]
        H_alpha[t, price_offered_index - 1, :] = H_alpha[t, price_offered_index - 1, :] + H_D[t - 1, :]

        # if demand = 1, then beta remained unchanged; if demand = 0, then beta plus 1
        H_beta[t, :, :] = H_beta[t - 1, :, :]
        H_beta[t, price_offered_index - 1, :] = H_beta[t - 1, price_offered_index - 1, :] + np.ones(N) - H_D[t - 1, :]
    else: # the demand must be zero, then all the estimate remain unchanged
        H_alpha[t, :, :] = H_alpha[t - 1, :, :]
        H_beta[t, :, :] = H_beta[t - 1, :, :]
print("Total reward of algorithm = {:f}".format(np.sum(H_reward)))
```

    Total reward of algorithm = 4259.000000

```python
# generate the csv data file and restore it into csv file
from datetime import datetime
moment = str(datetime.now()).replace(':', '-').replace(' ', '_')[:-7]
# the value of moment:  '2021-10-21_11-39-24', we used this varibale to name our files

# we firstly input the parameter into txt file
prameter_filename = moment + "_K-{:d}_N-{:d}_M-{:d}_T-{:d}_randomseed-{:d}_ParameterList.txt".format(K, N, M, T, random_seed)
with open(".\\Result_Ferreira et.al. 2018 Algorithm 2\\" + prameter_filename, 'w') as f:
    f.write("K = {:d}\n".format(K))
    f.write("N = {:d}\n".format(N))
    f.write("M = {:d}\n".format(M))
    f.write("randomseed = {:d}\n".format(random_seed))

    f.write("-----Admissible Prive Vector------\n")
    for kindex in range(K):
        f.write("Price vector {:d} : {:s}\n".format(kindex, str(P_list[kindex, :])))

    f.write("-----Initial Inventory------\n")
    f.write("Initial Inventory: {:s}\n".format(str(I_0)))

    f.write("-----Resource Consumption------\n")
    for nindex in range(N):
        f.write("Cost of resource of product {:d}: {:s}\n".format(nindex + 1, str(A[nindex, :])))

    f.write("-----Real theta------\n")
    for kindex in range(K):
        f.write("Given price vector {:d}, the real theta is {:s}\n".format(kindex + 1, str(theta[kindex, :])))

# then we input the experiment into csc file, we use period as index
data = pd.DataFrame({"Pricing_index": H_P, "Reward": H_reward})
data['Period_index'] = range(1, T+1)
# add the demand of each product
data = pd.concat([data, pd.DataFrame(H_D, columns = ["product_{:d}_demand".format(nindex) for nindex in range(1,N+1)])], axis = 1)
# add the remain invetory of each resource
data = pd.concat([data, pd.DataFrame(H_I[0:T, :], columns = ["resouce_{:d}".format(mindex) for mindex in range(1,M+1)])], axis = 1)
# add the constraint value
data = pd.concat([data, pd.DataFrame(H_constraint_value[0:T, :], columns = ["constraint_{:d}".format(conindex) for conindex in range(0,M+1)])], axis = 1)
# add the resource limits
data = pd.concat([data, pd.DataFrame(c[0:T, :], columns = ["c_{:d}".format(conindex) for conindex in range(1,M+1)])], axis = 1)
# add the best solution in each round
data = pd.concat([data,
                  pd.DataFrame(H_bestX,\
                              columns = ["use_price_index_{:d}".format(kindex) for kindex in range(1,K+1)] + ["use_infinite_price"])],
                  axis = 1)
# add the estimation of alpha
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_alpha[0:T, :, :], newshape =(T, K * N)),
                             columns = ["alpha_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
# add the estimation of alpha
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_beta[0:T, :, :], newshape =(T, K * N)),
                             columns = ["beta_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
# add the estimation of theta
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_theta[0:T, :, :], newshape =(T, K * N)),
                             columns = ["theta_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
exp_filename = moment + "_K-{:d}_N-{:d}_M-{:d}_T-{:d}_randomseed-{:d}_Experiment.csv".format(K, N, M, T, random_seed)
data.to_csv(".\\Result_Ferreira et.al. 2018 Algorithm 2\\" + exp_filename, index = False)
```
# clear the memory
H_reward = np.zeros(T) # the reward in each period
H_bestX = np.zeros(shape = (T, K + 1)) # the best solution in each optimization
H_P = np.zeros(shape = T) # the index of pricing vector we used in each period
H_D = np.zeros(shape = (T, N)) # the demand of products in each period
H_I = np.zeros(shape = (T + 1, M)) # avaliable inventory in each period
H_I[0, :] = np.float64(I_0)
c = np.zeros(shape = (T, M))# if we replace step 1 with true theta
for t in range(1, T+1):
    # first step, sample from posterior distribution
    # H_alpha[t-1, :, :], H_beta[t-1, :, :] is the history data from 0 to t
    # H_theta[t-1, :, :] is the sample theta we used in round t
#     H_theta[t-1, :, :] = mybeta(H_alpha[t-1, :, :], H_beta[t-1, :, :])

    # first step, calculate the mean demand given sample theta
    demand_mean = theta

    # second step, optimize a linear function
    model = pyscipopt.Model("Optimization in Rount {:d}".format(t))
    # generate decision variable
    x = {}
    for xindex in range(1, K+1):
        x[xindex] = model.addVar(vtype="C", lb = 0, ub = 1, name="x{:d}".format(xindex))

    # second step, generate object function
    obj_coefficient = np.sum(demand_mean *  P_list, axis = 1)# obj_coefficient[k] = $\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1}$
    model.setObjective(quicksum(x[xindex]*obj_coefficient[xindex-1] for xindex in range(1, K+1)), "maximize")
    # objective = $\sum_{k=1}^K(\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1})x_{k}$

    # second step, add constraint x_1+...+x_k<=1
    model.addCons(quicksum(x[xindex] for xindex in range(1, K+1)) <= 1)

    # second step, for each resources, we require \sum_{k=1}^K\sum_{i=1}^N d_{i,k}a_{i,j}x_l<=c_j
    c[t-1, :] = H_I[t-1, :] / (T - t + 1)
    for jj in range(1, M+1):
        # size(A) = [N, M], size(demand_mean) = [K, N]
        con_coefficient = A[:, jj - 1].dot(np.transpose(demand_mean))# con_coefficient[k] = $\sum_{i=1}^N a_{i,j}d_{i,k+1}$
        model.addCons(quicksum(x[xindex] * con_coefficient[xindex - 1] for xindex in range(1, K+1)) <= c[t - 1, jj - 1])

    # second step, optimize the problem
    model.optimize()
    bestx = np.zeros(K+1) # p_{K+1} would force the demand be zero
    for xindex in range(1,K+1):
        bestx[xindex - 1] = model.getVal(x[xindex])
    bestx[K] = 1 - np.sum(bestx[0:K])
    eliminate_error = lambda x: 0 if np.abs(x) < 1e-10 else x #there would be numerical error in the best solution
    bestx = np.array([eliminate_error(x) for x in bestx])
    bestx = bestx / np.sum(bestx)

    # third step, offer price
    price_offered_index = np.random.choice(np.arange(1, K + 2), p = bestx)

    # fourth step, update estimate of parameter
    H_P[t - 1] = price_offered_index # record the index of offered price

    # fourth step, record the optimal solution in this round
    H_bestX[t - 1, :] = bestx

    # fourth step, record the realization of demand
    if price_offered_index < K + 1:
        H_D[t - 1, :] = mybinomial(np.ones(N), theta[price_offered_index - 1, :]) # record the realization of demand
    else: # the demand must be zero
        H_D[t - 1, :] = np.zeros(N)

    # fourth step, record the reward in this period
    if price_offered_index < K + 1:
        H_reward[t - 1] = P_list[price_offered_index - 1, :].dot(H_D[t - 1, :])  # record the realization of demand
    else: # the demand must be zero
        H_reward[t - 1] = 0

    # fourth step, record the remain inventory, size(A) = [N, M]
    H_I[t, :] = H_I[t - 1, :] - np.transpose(A).dot(H_D[t - 1, :])
    if not all(H_I[t, :] >= 0):
        break

print("Total reward of known distribution: {:f}".format(np.sum(H_reward)))# then input the experiment into csc file, we use period as index
data = pd.DataFrame({"Pricing_index": H_P, "Reward": H_reward})
data['Period_index'] = range(1, T+1)
# add the demand of each product
data = pd.concat([data, pd.DataFrame(H_D, columns = ["product_{:d}_demand".format(nindex) for nindex in range(1,N+1)])], axis = 1)
# add the remain invetory of each resource
data = pd.concat([data, pd.DataFrame(H_I[0:T, :], columns = ["resouce_{:d}".format(mindex) for mindex in range(1,M+1)])], axis = 1)
# add the best solution in each round
data = pd.concat([data,
                  pd.DataFrame(H_bestX,\
                              columns = ["use_price_index_{:d}".format(kindex) for kindex in range(1,K+1)] + ["use_infinite_price"])],
                  axis = 1)
# add the estimation of alpha
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_alpha[0:T, :, :], newshape =(T, K * N)),
                             columns = ["alpha_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
# add the estimation of alpha
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_beta[0:T, :, :], newshape =(T, K * N)),
                             columns = ["beta_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
# add the estimation of theta
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_theta[0:T, :, :], newshape =(T, K * N)),
                             columns = ["theta_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
exp_filename = moment + "_K-{:d}_N-{:d}_M-{:d}_T-{:d}_randomseed-{:d}_RealThetaExperiment.csv".format(K, N, M, T, random_seed)
data.to_csv(".\\Result_Ferreira et.al. 2018 Algorithm 2\\" + exp_filename, index = False)c
## 3rd algotithm

## 4 th alogrithm
We implement the 4th algorithm of Ferreira et.al. 2018

Following is its notation and description

|      Parameter       | description                                                  |
| :------------------: | :----------------------------------------------------------- |
|         $K$          | Total number of available price vectors                      |
|         $N$          | Total number of products                                    |
|         $M$          | Total number of kinds of resource                            |
|         $T$          | Total number of periods                                      |
|        $[x]$         | We define $[x]$ as a set, $[x]=\{1,2,\cdots, x\}$            |
|         $i$          | Index of products                                            |
|         $j$          | Index of resources                                           |
|         $t$          | Index of period                                              |
|     $I_j,I_j(t)$     | $I_j$ is the initial inventory for each resource $j\in [M]$, $I_j(t)$ is the inventory at the end of period $t$. $I_j(0)=I_j$ |
|       $a_{ij}$       | When we produced one unit item $i$, it would consume $a_{ij}$ unit $j$ |
|        $c_j$         | we define $c_j=\frac{I_j}{t}$                                |
|        $p_k$         | We define $\{p_1,p_2,\cdots,p_K\}$ as the admissible price vectors, each $p_k$ is a $N\times 1$ vector, specifying the price of each product, $p_k=(p_{1k},\cdots,p_{Nk})$, where $p_{ik}$ is the price of product $i$, for $i\in [N]$. We define $p_{\infty}$ as a "shut-off" price, such that the demand for any product under this price is zero. |
|        $P(t)$        | We denote by $P(t)=(P_1(t),\cdots,P_N(t))$ the prices chosen by the retailer in this period, and require that $P(t)\in \{p_1,p_2,\cdots,p_K,p_{\infty}\}$ |
|        $D(t)$        | We denote by $D(t) = (D_1(t),\cdots,D_N(t))$ the demand of each product at period $t$. We assume that given $P(t)=p_k$, the demand $D(t)$ is sampled from a probability distribution on $\mathbb{R}^{N}_+$ with joint cumulative distribution function (CDF) $F (x_1,\cdots,x_N, pk, \theta )$. $D(t)$ is independent of the history $\mathcal{H}_{t-1}$, given $P(t)$ |
|       $\theta$       | $\theta$ is the parameter of demand distribution, takes values in the parameter space $\Theta\subset\mathbb{R}^l$. The nature would sample $\theta$ from a prior distribution at the beginning form the process. The distribution is assumed to be subexponential; |
|  $\mathcal{H}_{t }$  | $\mathcal{H}_{t}=(P(1),D(1),\cdots,P(t),D(t))$               |
|       $\xi(t)$       | At the beginning of each period $t\in[T]$, the retailer observes some context $\xi(t)$, $\xi(t)$ belongs to some discrete set $\mathcal{X}$. We assume $\xi(t)$ is sampled i.i.d from a known distribution. |
 | $d_{ik}(\xi|\theta)$ | The mean demand of product $i\in [N]$ under price vector $p_k$, $\forall k\in[K]$, given context $\xi$ and parameter $\theta$ |

We assume the distribution of demand is calculated as follows:
\begin{equation}
(D_1,D_2,\cdots,D_{N}) \sim F(d_1, d_2,\cdots,d_N|P,\xi,\theta)
\end{equation}

$P$ is the price vector we offer, $\xi$ is the observed context variable, $\theta$ is the parameter that we need to estimate, it is sampled from the nature at the beginning of our study. We assume $\theta \sim f_{\Theta}(\theta)$

We furtherly assume the density function of this distribution is $f(d_1, d_2,\cdots,d_N|P,\xi,\theta)$.  Then we will derive the posterior distribution of $ \theta$

$$
\begin{align}
&f(\theta \ | \ (P_1,\xi_1,D^{(1)}), \ (P_2,\xi_2,D^{(2)}),\cdots, \ (P_t,\xi_t,D^{(t)}))\\
=&\frac{f(\theta, \ (P_1,\xi_1,D^{(1)}), \ (P_2,\xi_2,D^{(2)}),\cdots, \ (P_t,\xi_t,D^{(t)}))}{\int_{-\infty}^{\infty}f_{\Theta}(\theta)\prod_{i=1}^tf(d^{(i)}_1, d^{(i)}_2,\cdots,d^{(i)}_N|P_i,\xi_i,\theta)d\theta}\\
=&\frac{f(\ (P_1,\xi_1,D^{(1)}), \ (P_2,\xi_2,D^{(2)}),\cdots, \ (P_t,\xi_t,D^{(t)})\ | \ \theta) f_{\Theta}(\theta)}{\int_{-\infty}^{\infty}f_{\Theta}(\theta)\prod_{i=1}^tf(d^{(i)}_1, d^{(i)}_2,\cdots,d^{(i)}_N|P_i,\xi_i,\theta)d\theta}\\
=&\frac{ f_{\Theta}(\theta)\prod_{i=1}^t f(P_t,\xi_t,D^{(t)}| \ \theta)}{\int_{-\infty}^{\infty}f_{\Theta}(\theta)\prod_{i=1}^tf(d^{(i)}_1, d^{(i)}_2,\cdots,d^{(i)}_N|P_i,\xi_i,\theta)d\theta}\\
=&\frac{ f_{\Theta}(\theta)\prod_{i=1}^t f(D^{(i)}| \ \theta, P_t,\xi_t,)}{\int_{-\infty}^{\infty}f_{\Theta}(\theta)\prod_{i=1}^tf(d^{(i)}_1, d^{(i)}_2,\cdots,d^{(i)}_N|P_i,\xi_i,\theta)d\theta}\\
=&\frac{ f_{\Theta}(\theta)\prod_{i=1}^t f(d^{(i)}_1, d^{(i)}_2,\cdots,d^{(i)}_N| \ \theta, P_t,\xi_t,)}{\int_{-\infty}^{\infty}f_{\Theta}(\theta)\prod_{i=1}^tf(d^{(i)}_1, d^{(i)}_2,\cdots,d^{(i)}_N|P_i,\xi_i,\theta)d\theta}\\
\end{align}
$$

$P_i$ is the pricing vector we choose in the $i^{th}$ period

$\xi_i$ is the context we observed in the $i^{th}$ period

$D^{(i)}$ is the demand vector in $i^{th}$ period

$d^{(i)}_j$ is the demand of $j^{th}$ product in $i^{th}$ period

To implement this procedure, we need to solve 3 problems
1. Given the real demand generator, we need to calculate its density function.
2. sample random number from a customized density function
3. calculate the integral, numerically

We assume the demand follows discrete distribution, that is $d_i$

```python
# Generate parameters
np.random.seed(12345)

K = np.random.randint(low = 1, high = 5) # Total number of available price vectors
N = np.random.randint(low = 1, high = 5) # Total number of products
M = np.random.randint(low = 1, high = 5) # Total number of kinds of resource
T = 1000 # Total number of periods

# generate the demand

# each row represent an admissible pricing strategy
P_list = np.float64(np.random.randint(low = 1, high = 10, size = (K, M)))

# type of context
xi_type_num = np.random.randint(low = 3, high = 5 ) # maximum number of context type
xi_list = np.float64(np.arange(1, xi_type_num, 1))

# initialize inventory
I_0 = np.random.randint(low = 9000, high = 11000, size = M)

# initialize a_ij
A = np.float64(np.random.randint(low = 5, high = 15, size = (N,M)))

# real distribution of theta, we adopt beta distribution here,
# here theta is the parameter of beta dirstribution
# we assume the alpha and beta is independent of pricing vector , observed vector, and product
alpha_real_scalar = np.random.randint(low = 1, high = 3)
beta_real_scalar = np.random.randint(low = 2, high = 5)
alpha_real = alpha_real_scalar * np.ones(shape = (K, xi_type, N))
beta_real = beta_real_scalar * np.ones(shape = (K, xi_type, N))
# alpha_real = np.random.randint(low = 1, high = 3, size = (K, xi_type, N))
# beta_real = np.random.randint(low = 2, high = 5, size = (K, xi_type, N))

# initialize history
H_P = np.zeros(shape = T) # the index of pricing vector we used in each period
H_xi = np.zeros(shape = T)# the index of observed context in each period
H_D = np.zeros(shape = (T, N)) # the demand of products in each period

H_I = np.zeros(shape = (T, M)) # avaliable inventory in each period
H_I[0, :] = np.float64(I_0)

# each realization of price vector, observed context, index of period,
# corresponds to a estimate
H_alpha = np.zeros(shape = (T,K,xi_type_num,N))
H_beta = np.zeros(shape = (T,K,xi_type_num,N))
for kk in range(0, K):
    for xi_index in range(0, xi_type_num):
        H_alpha[0, kk, xi_index, :] = 1*np.ones(N)
        H_beta[0, kk, xi_index, :] = 1*np.ones(N)
```

```python
# Given the pricing vector and parameter, we use this function to generate demand
def GetDemand(P, xi, theta, num = 1):
    # P is the pricing vector, N*1
    # xi is the context observed by the merchant
    # theta is the parameter of beta distribution, a N*2 vector
    # num is the size of return demand, we would return M*num array, each column denote a sample demand

    # return value is a M*num array, each column denote a sample demand
    demand_sample = np.zeros(N, num)
    for ii in range(N):
        demand_sample[ii, :] = np.random.beta(alpha = alpha[ii], beta = beta[ii], size = num) + xi*5 - P[ii]*2
    return demand_sample
```

We derive the update formular here

```python
def UpdatePosteriorDistribution(t, H_P, H_xi, H_D, H_alpha, H_beta):
    # t is the index of period, the history is available from 0-t
    # H_P is the index of pricing vector in each period
    # H_xi is the index of obeserved context in each period

    # alpha_real and beta_real is the estimation of alpha and beta, given
    alpha_real = alpha_real_scalar * np.ones(shape = (K, xi_type, N))
    beta_real = beta_real_scalar * np.ones(shape = (K, xi_type, N))
```

```python
# implementation of algorithm
for tt in range(0, T):
    # step 1, sample a randome parameter theta
    alpha_tt = H_alpha[tt]
    beta_tt = H_beta[tt]
    # step 1, calculate the mean demand
    d_mean_tt = np.zeros(N, K)
    for kk in range(0, K): # traverse all the admissable pricing vector

```

```python

```

```python
np.float64(12)
```

    <ipython-input-23-92a6b828a7ec>:1: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
    Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
      np.float(12)

    12.0

```python
np.ones(5)
```

    array([1., 1., 1., 1., 1.])

```python
theta_real
```

    array([0.25      , 0.33333333])

```python
xi_list
```

    array([1, 2, 3])

```python
P_list
```

    array([[2, 5],
           [6, 3],
           [2, 7]])

```python
P_list[:,0] + 2
```

    array([4, 8, 4])

```python
I_0
```

    array([10339,  9654])

```python
I
```

    array([[10339.,     0.,     0., ...,     0.,     0.,     0.],
           [ 9654.,     0.,     0., ...,     0.,     0.,     0.]])

```python
help(np.zeros)
```

    Help on built-in function zeros in module numpy:

    zeros(...)
        zeros(shape, dtype=float, order='C', *, like=None)

        Return a new array of given shape and type, filled with zeros.

        Parameters
        ----------
        shape : int or tuple of ints
            Shape of the new array, e.g., ``(2, 3)`` or ``2``.
        dtype : data-type, optional
            The desired data-type for the array, e.g., `numpy.int8`.  Default is
            `numpy.float64`.
        order : {'C', 'F'}, optional, default: 'C'
            Whether to store multi-dimensional data in row-major
            (C-style) or column-major (Fortran-style) order in
            memory.
        like : array_like
            Reference object to allow the creation of arrays which are not
            NumPy arrays. If an array-like passed in as ``like`` supports
            the ``__array_function__`` protocol, the result will be defined
            by it. In this case, it ensures the creation of an array object
            compatible with that passed in via this argument.

            .. note::
                The ``like`` keyword is an experimental feature pending on
                acceptance of :ref:`NEP 35 <NEP35>`.

            .. versionadded:: 1.20.0

        Returns
        -------
        out : ndarray
            Array of zeros with the given shape, dtype, and order.

        See Also
        --------
        zeros_like : Return an array of zeros with shape and type of input.
        empty : Return a new uninitialized array.
        ones : Return a new array setting values to one.
        full : Return a new array of given shape filled with value.

        Examples
        --------
        >>> np.zeros(5)
        array([ 0.,  0.,  0.,  0.,  0.])

        >>> np.zeros((5,), dtype=int)
        array([0, 0, 0, 0, 0])

        >>> np.zeros((2, 1))
        array([[ 0.],
               [ 0.]])

        >>> s = (2,2)
        >>> np.zeros(s)
        array([[ 0.,  0.],
               [ 0.,  0.]])

        >>> np.zeros((2,), dtype=[('x', 'i4'), ('y', 'i4')]) # custom dtype
        array([(0, 0), (0, 0)],
              dtype=[('x', '<i4'), ('y', '<i4')])

```python

```