# Ferreira et.al. 2018
Following is its notation and description

|      Parameter       | description                                                  |
| :------------------: | :----------------------------------------------------------- |
|         $K$          | Total number of available price vectors                      |
|         $N$          | Total number of products                                    |
|         $M$          | Total number of kinds of resource                            |
|         $T$          | Total number of periods                                      |
|        $[x]$         | We define $[x]$ as a set, $[x]=\{1,2,\cdots, x\}$            |
|         $i$          | Index of products                                            |
|         $j$          | Index of resources                                           |
|         $t$          | Index of period                                              |
|     $I_j,I_j(t)$     | $I_j$ is the initial inventory for each resource $j\in [M]$, $I_j(t)$ is the inventory at the end of period $t$. $I_j(0)=I_j$ |
|       $a_{ij}$       | When we produced one unit item $i$, it would consume $a_{ij}$ unit $j$ |
|        $c_j$         | we define $c_j=\frac{I_j}{t}$                                |
|        $p_k$         | We define $\{p_1,p_2,\cdots,p_K\}$ as the admissible price vectors, each $p_k$ is a $N\times 1$ vector, specifying the price of each product, $p_k=(p_{1k},\cdots,p_{Nk})$, where $p_{ik}$ is the price of product $i$, for $i\in [N]$. We define $p_{\infty}$ as a "shut-off" price, such that the demand for any product under this price is zero. |
|        $P(t)$        | We denote by $P(t)=(P_1(t),\cdots,P_N(t))$ the prices chosen by the retailer in this period, and require that $P(t)\in \{p_1,p_2,\cdots,p_K,p_{\infty}\}$ |
|        $D(t)$        | We denote by $D(t) = (D_1(t),\cdots,D_N(t))$ the demand of each product at period $t$. We assume that given $P(t)=p_k$, the demand $D(t)$ is sampled from a probability distribution on $\mathbb{R}^{N}_+$ with joint cumulative distribution function (CDF) $F (x_1,\cdots,x_N, pk, \theta )$. $D(t)$ is independent of the history $\mathcal{H}_{t-1}$, given $P(t)$ |
|       $\theta$       | $\theta$ is the parameter of demand distribution, takes values in the parameter space $\Theta\subset\mathbb{R}^l$. The nature would sample $\theta$ from a prior distribution at the beginning form the process. The distribution is assumed to be subexponential; |
|  $\mathcal{H}_{t }$  | $\mathcal{H}_{t}=(P(1),D(1),\cdots,P(t),D(t))$               |
|       $\xi(t)$       | At the beginning of each period $t\in[T]$, the retailer observes some context $\xi(t)$, $\xi(t)$ belongs to some discrete set $\mathcal{X}$. We assume $\xi(t)$ is sampled i.i.d from a known distribution. |
 | $d_{ik}(\xi|\theta)$ | The mean demand of product $i\in [N]$ under price vector $p_k$, $\forall k\in[K]$, given context $\xi$ and parameter $\theta$ |

## 1 st algorithm

We implement the 4th algorithm of Ferreira et.al. 2018

The most difficult task in the implementation of this algorithm is the update rule of posterior. In fact, for most of the prior distribution, it is nearly impossible to derive the explict expression of posterior distribution. Here, we would like to simplify this process.

We generate K * N matrix from beta(1, 1), in other words, uniform distribution in $[0, 1]$. Each entry corresponds to a product in each pricing vector. We let $d_{k,i}$ denote the demand of $i^{th}$ product given $k^{th}$ pricing vector. We assume
$$
\begin{align}
Pr(d_{k,i} = 1) & = \Theta_{k,i}\\
Pr(d_{k,i} = 0) & = 1 - \Theta_{k,i}\\
\end{align}
$$

In this case, it would be quite easy to update the posterior distribution. Assume we adopt $k^{th}$ price vector $p_k$ in period $t_1, t_2, \cdots, t_\tau$,

For each $k\in [K], i\in [N]$, the poseterior density function of $\Theta_{k,i}$ is
$$
\begin{align}
f(\theta_{k,i} | (p^{(t_1)}_{k},d^{(t_1)}_k),\cdots, (p^{(t_\tau)}_{k},d^{(t_\tau)}_k))
& = \frac{Pr(d^{(t_1)}_k,\cdots, ,d^{(t_\tau)}_k \ | \ \theta_{k,i}) * 1}{\int_{0}^{1}Pr(d^{(t_1)}_k,\cdots, ,d^{(t_\tau)}_k \ | \ \theta)*1d\theta}\\
&=\frac{\prod_{j=1}^\tau Pr(d^{(t_j)}_k| \ \theta_{k,i}) * 1}{\int_{0}^{1}\prod_{j=1}^\tau Pr(d^{(t_j)}_k| \ \theta)*1d\theta}\\
&=\frac{\theta_{k,i}^{\sum_{j=1}^\tau d^{(t_j)}_k} (1-\theta_{k,i})^{\tau-\sum_{j=1}^\tau d^{(t_j)}_k}}{\int_{0}^{1}\theta^{\sum_{j=1}^\tau d^{(t_j)}_k} (1-\theta)^{\tau-\sum_{j=1}^\tau d^{(t_j)}_k}d\theta}\\
&=beta(\sum_{j=1}^\tau d^{(t_j)}_k, \tau-\sum_{j=1}^\tau d^{(t_j)}_k)
\end{align}
$$

$d^{(i)}_j$ is the demand of $j^{th}$ product in $i^{th}$ period

Well, there are also tough problems. That is, there would be counter-intuitive random number. For example, **higher price lead to higher demand**. But anyway, I tried my best to find a suitable prior distribution, but I failed to do so. So I decided to just use this easier one.

```python
%reset -f
import numpy as np
import pandas as pd # we would restore our result in csv file .\\Result_Ferreira et.al. 2018 Algorithm 1\\
import datetime

import pyscipopt
from pyscipopt import quicksum

random_seed = 12345
np.random.seed(random_seed)
```

```python
# Generate parameters
K = np.random.randint(low = 1, high = 5) # Total number of available price vectors
N = np.random.randint(low = 1, high = 5) # Total number of products
M = np.random.randint(low = 1, high = 5) # Total number of kinds of resource
T = 1000 # Total number of periods

# each row represent an admissible pricing strategy
P_list = np.float64(np.random.randint(low = 1, high = 10, size = (K, N)))

# initialize inventory
I_0 = np.float64(np.random.randint(low = 9000, high = 11000, size = M))# 如果要改变T的话，这里也要对应改变
# resource 应该和T有线性关系
c = I_0 / T # c is the average cost of each resource

# initialize a_ij
A = np.float64(np.random.randint(low = 5, high = 15, size = (N,M)))

# initialize real parameter theta
theta = np.random.beta(a = 1, b = 1, size = (K,N))

# initialize history
H_P = np.zeros(shape = T) # the index of pricing vector we used in each period
H_D = np.zeros(shape = (T, N)) # the demand of products in each period

H_I = np.zeros(shape = (T + 1, M)) # avaliable inventory in each period
H_I[0, :] = np.float64(I_0)

H_bestX = np.zeros(shape = (T, K + 1)) # the best solution in each optimization

H_reward = np.zeros(T) # the reward in each period

# each realization of price vector, index of period,
# corresponds to a estimate of theta
H_alpha = np.zeros(shape = (T + 1,K,N))
H_beta = np.zeros(shape = (T + 1,K,N))
H_alpha[0, :, :] = 1 * np.ones(shape = (K,N))
H_beta[0, :, :] = 1 * np.ones(shape = (K,N))

# initialize the constraint value in each round
# M kinds of resources correspond to M constraints, and one more constraint is x1 + ... + xN <=1
H_constraint_value = np.zeros(shape = (T, M + 1))

# vectorize beta sample function to accelerate
mybeta = np.vectorize(np.random.beta)
H_theta = np.zeros(shape = (T,K,N))

# vectorize binomial sample function to accelerate
mybinomial = np.vectorize(np.random.binomial)
```

```python
# implementation of algorithm 1
for t in range(1, T+1):
# for t in range(1, 2):
    # first step, sample from posterior distribution
    # H_alpha[t-1, :, :], H_beta[t-1, :, :] is the history data from 0 to t
    # H_theta[t-1, :, :] is the sample theta we used in round t
    H_theta[t-1, :, :] = mybeta(H_alpha[t-1, :, :], H_beta[t-1, :, :])

    # first step, calculate the mean demand given sample theta
    demand_mean = H_theta[t-1, :, :]

    # second step, optimize a linear function
    model = pyscipopt.Model("Optimization in Rount {:d}".format(t))
    # generate decision variable
    x = {}
    for xindex in range(1, K+1):
        x[xindex] = model.addVar(vtype="C", lb = 0, ub = 1, name="x{:d}".format(xindex))

    # second step, generate object function
    obj_coefficient = np.sum(demand_mean *  P_list, axis = 1)# obj_coefficient[k] = $\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1}$
    model.setObjective(quicksum(x[xindex]*obj_coefficient[xindex-1] for xindex in range(1, K+1)), "maximize")
    # objective = $\sum_{k=1}^K(\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1})x_{k}$

    # second step, add constraint x_1+...+x_k<=1
    constraint_index = {}
    constraint_index[0] = model.addCons(quicksum(x[xindex] for xindex in range(1, K+1)) <= 1)

    # second step, for each resources, we require \sum_{k=1}^K\sum_{i=1}^N d_{i,k}a_{i,j}x_l<=c_j
    for jj in range(1, M+1):
        # size(A) = [N, M], size(demand_mean) = [K, N]
        con_coefficient = A[:, jj - 1].dot(np.transpose(demand_mean))# con_coefficient[k] = $\sum_{i=1}^N a_{i,j}d_{i,k+1}$
        constraint_index[jj] = model.addCons(quicksum(x[xindex] * con_coefficient[xindex - 1] for xindex in range(1, K+1)) <= c[jj - 1])

    # second step, optimize the problem
    model.optimize()
    bestx = np.zeros(K+1) # p_{K+1} would force the demand be zero
    for xindex in range(1,K+1):
        bestx[xindex - 1] = model.getVal(x[xindex])
    bestx[K] = 1 - np.sum(bestx[0:K])
    eliminate_error = lambda x: 0 if np.abs(x) < 1e-10 else x #there would be numerical error in the best solution
    bestx = np.array([eliminate_error(x) for x in bestx])
    bestx = bestx / np.sum(bestx)

    # third step, offer price
    price_offered_index = np.random.choice(np.arange(1, K + 2), p = bestx)

    # fourth step, update estimate of parameter
    H_P[t - 1] = price_offered_index # record the index of offered price

    # fourth step, record the constraint value in optimization
    H_constraint_value[t - 1, 0] = np.sum(bestx[0:K])
    for jj in range(1, M+1):
        con_coefficient = np.array(list(model.getValsLinear(constraint_index[jj]).values()))
        H_constraint_value[t - 1, jj] = np.sum(bestx[0:K] * con_coefficient)

    # fourth step, record the optimal solution in this round
    H_bestX[t - 1, :] = bestx

    # fourth step, record the realization of demand
    if price_offered_index < K + 1:
        H_D[t - 1, :] = mybinomial(np.ones(N), theta[price_offered_index - 1, :]) # record the realization of demand
    else: # the demand must be zero
        H_D[t - 1, :] = np.zeros(N)

    # fourth step, record the reward in this period
    if price_offered_index < K + 1:
        H_reward[t - 1] = P_list[price_offered_index - 1, :].dot(H_D[t - 1, :])  # record the realization of demand
    else: # the demand must be zero
        H_reward[t - 1] = 0

    # fourth step, record the remain inventory, size(A) = [N, M]
    H_I[t] = H_I[t - 1] - np.transpose(A).dot(H_D[t - 1, :])
    if not all(H_I[t] >= 0):
        break

    # fourth step, record the new estimate of alpha and beta
    if price_offered_index < K + 1:
        # if demand = 1, then alpha plus 1; if demand = 0, then alpha remain unchanged
        H_alpha[t, :, :] = H_alpha[t - 1, :, :]
        H_alpha[t, price_offered_index - 1, :] = H_alpha[t, price_offered_index - 1, :] + H_D[t - 1, :]

        # if demand = 1, then beta remained unchanged; if demand = 0, then beta plus 1
        H_beta[t, :, :] = H_beta[t - 1, :, :]
        H_beta[t, price_offered_index - 1, :] = H_beta[t - 1, price_offered_index - 1, :] + np.ones(N) - H_D[t - 1, :]
    else: # the demand must be zero, then all the estimate remain unchanged
        H_alpha[t, :, :] = H_alpha[t - 1, :, :]
        H_beta[t, :, :] = H_beta[t - 1, :, :]
print("Total reward of algorithm = {:f}".format(np.sum(H_reward)))
```

    Total reward of algorithm = 3624.000000

t, M, N, K
constraint_index
model.getValsLinear(constraint_index [1]),
model.getValsLinear(constraint_index [2])
x
model.getVal(x[1]), model.getVal(x[2]), model.getVal(x[3])
bestx
0.530018723612461 * 10.376003106593089, 0.530018723612461 * 17.74088269167503,
c
model.getValsLinear(constraint_index [1])
model.getValsLinear(constraint_index [2])
model.getConss()
model.getNConss()
model.getValsLinear(model.getConss()[0])
model.getValsLinear(model.getConss()[1])

```python
c
```

    array([10.393, 10.339])

```python
# generate the csv data file and restore it into csv file
from datetime import datetime
moment = str(datetime.now()).replace(':', '-').replace(' ', '_')[:-7]
# the value of moment:  '2021-10-21_11-39-24', we used this varibale to name our files

# we firstly input the parameter into txt file
prameter_filename = moment + "_K-{:d}_N-{:d}_M-{:d}_T-{:d}_randomseed-{:d}_ParameterList.txt".format(K, N, M, T, random_seed)
with open(".\\Result_Ferreira et.al. 2018 Algorithm 1\\" + prameter_filename, 'w') as f:
    f.write("K = {:d}\n".format(K))
    f.write("N = {:d}\n".format(N))
    f.write("M = {:d}\n".format(M))
    f.write("randomseed = {:d}\n".format(random_seed))

    f.write("-----Admissible Prive Vector------\n")
    for kindex in range(K):
        f.write("Price vector {:d} : {:s}\n".format(kindex, str(P_list[kindex, :])))

    f.write("-----Initial Inventory------\n")
    f.write("Initial Inventory: {:s}\n".format(str(I_0)))

    f.write("-----Resource Consumption------\n")
    for nindex in range(N):
        f.write("Cost of resource of product {:d}: {:s}\n".format(nindex + 1, str(A[nindex, :])))

    f.write("-----Resouce Constraint in each round------\n")
    for nindex in range(N):
        f.write("c : {:s}\n".format(str(c)))

    f.write("-----Real theta------\n")
    for kindex in range(K):
        f.write("Given price vector {:d}, the real theta is {:s}\n".format(kindex + 1, str(theta[kindex, :])))

# then we input the experiment into csc file, we use period as index
data = pd.DataFrame({"Pricing_index": H_P, "Reward": H_reward})
data['Period_index'] = range(1, T+1)
# add the demand of each product
data = pd.concat([data, pd.DataFrame(H_D, columns = ["product_{:d}_demand".format(nindex) for nindex in range(1,N+1)])], axis = 1)
# add the remain invetory of each resource
data = pd.concat([data, pd.DataFrame(H_I[0:T, :], columns = ["resouce_{:d}".format(mindex) for mindex in range(1,M+1)])], axis = 1)
# add the constraint value
data = pd.concat([data, pd.DataFrame(H_constraint_value[0:T, :], columns = ["constraint_{:d}".format(conindex) for conindex in range(0,M+1)])], axis = 1)
# add the best solution in each round
data = pd.concat([data,
                  pd.DataFrame(H_bestX,\
                              columns = ["use_price_index_{:d}".format(kindex) for kindex in range(1,K+1)] + ["use_infinite_price"])],
                  axis = 1)
# add the estimation of alpha
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_alpha[0:T, :, :], newshape =(T, K * N)),
                             columns = ["alpha_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
# add the estimation of alpha
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_beta[0:T, :, :], newshape =(T, K * N)),
                             columns = ["beta_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
# add the estimation of theta
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_theta[0:T, :, :], newshape =(T, K * N)),
                             columns = ["theta_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
exp_filename = moment + "_K-{:d}_N-{:d}_M-{:d}_T-{:d}_randomseed-{:d}_Experiment.csv".format(K, N, M, T, random_seed)
data.to_csv(".\\Result_Ferreira et.al. 2018 Algorithm 1\\" + exp_filename, index = False)
```

```python
# clear the memory
H_reward = np.zeros(T) # the reward in each period
H_bestX = np.zeros(shape = (T, K + 1)) # the best solution in each optimization
H_P = np.zeros(shape = T) # the index of pricing vector we used in each period
H_D = np.zeros(shape = (T, N)) # the demand of products in each period
H_I = np.zeros(shape = (T + 1, M)) # avaliable inventory in each period
H_I[0, :] = np.float64(I_0)
```

```python
# if we replace step 1 with true theta
for t in range(1, T+1):
    # first step, sample from posterior distribution
    # H_alpha[t-1, :, :], H_beta[t-1, :, :] is the history data from 0 to t
    # H_theta[t-1, :, :] is the sample theta we used in round t
#     H_theta[t-1, :, :] = mybeta(H_alpha[t-1, :, :], H_beta[t-1, :, :])

    # first step, calculate the mean demand given sample theta
    demand_mean = theta

    # second step, optimize a linear function
    model = pyscipopt.Model("Optimization in Rount {:d}".format(t))
    # generate decision variable
    x = {}
    for xindex in range(1, K+1):
        x[xindex] = model.addVar(vtype="C", lb = 0, ub = 1, name="x{:d}".format(xindex))

    # second step, generate object function
    obj_coefficient = np.sum(demand_mean *  P_list, axis = 1)# obj_coefficient[k] = $\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1}$
    model.setObjective(quicksum(x[xindex]*obj_coefficient[xindex-1] for xindex in range(1, K+1)), "maximize")
    # objective = $\sum_{k=1}^K(\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1})x_{k}$

#     # second step, add constraint x_1+...+x_k<=1
#     model.addCons(quicksum(x[xindex] for xindex in range(1, K+1)) <= 1)

#     # second step, for each resources, we require \sum_{k=1}^K\sum_{i=1}^N d_{i,k}a_{i,j}x_l<=c_j
#     for jj in range(1, M+1):
#         # size(A) = [N, M], size(demand_mean) = [K, N]
#         con_coefficient = A[:, jj - 1].dot(np.transpose(demand_mean))# con_coefficient[k] = $\sum_{i=1}^N a_{i,j}d_{i,k+1}$
#         model.addCons(quicksum(x[xindex] * con_coefficient[xindex - 1] for xindex in range(1, K+1)) <= c[jj - 1])

#     # second step, optimize the problem
#     model.optimize()
#     bestx = np.zeros(K+1) # p_{K+1} would force the demand be zero
#     for xindex in range(1,K+1):
#         bestx[xindex - 1] = model.getVal(x[xindex])
#     bestx[K] = 1 - np.sum(bestx[0:K])
#     eliminate_error = lambda x: 0 if np.abs(x) < 1e-10 else x #there would be numerical error in the best solution
#     bestx = np.array([eliminate_error(x) for x in bestx])
#     bestx = bestx / np.sum(bestx)

#     # third step, offer price
#     price_offered_index = np.random.choice(np.arange(1, K + 2), p = bestx)

    # second step, add constraint x_1+...+x_k<=1
    constraint_index = {}
    constraint_index[0] = model.addCons(quicksum(x[xindex] for xindex in range(1, K+1)) <= 1)

    # second step, for each resources, we require \sum_{k=1}^K\sum_{i=1}^N d_{i,k}a_{i,j}x_l<=c_j
    for jj in range(1, M+1):
        # size(A) = [N, M], size(demand_mean) = [K, N]
        con_coefficient = A[:, jj - 1].dot(np.transpose(demand_mean))# con_coefficient[k] = $\sum_{i=1}^N a_{i,j}d_{i,k+1}$
        constraint_index[jj] = model.addCons(quicksum(x[xindex] * con_coefficient[xindex - 1] for xindex in range(1, K+1)) <= c[jj - 1])

    # second step, optimize the problem
    model.optimize()
    bestx = np.zeros(K+1) # p_{K+1} would force the demand be zero
    for xindex in range(1,K+1):
        bestx[xindex - 1] = model.getVal(x[xindex])
    bestx[K] = 1 - np.sum(bestx[0:K])
    eliminate_error = lambda x: 0 if np.abs(x) < 1e-10 else x #there would be numerical error in the best solution
    bestx = np.array([eliminate_error(x) for x in bestx])
    bestx = bestx / np.sum(bestx)

    # third step, offer price
    price_offered_index = np.random.choice(np.arange(1, K + 2), p = bestx)

    # fourth step, update estimate of parameter
    H_P[t - 1] = price_offered_index # record the index of offered price

    # fourth step, record the constraint value in optimization
    H_constraint_value[t - 1, 0] = np.sum(bestx[0:K])
    for jj in range(1, M+1):
        con_coefficient = np.array(list(model.getValsLinear(constraint_index[jj]).values()))
        H_constraint_value[t - 1, jj] = np.sum(bestx[0:K] * con_coefficient)

    # fourth step, update estimate of parameter
    H_P[t - 1] = price_offered_index # record the index of offered price

    # fourth step, record the optimal solution in this round
    H_bestX[t - 1, :] = bestx

    # fourth step, record the realization of demand
    if price_offered_index < K + 1:
        H_D[t - 1, :] = mybinomial(np.ones(N), theta[price_offered_index - 1, :]) # record the realization of demand
    else: # the demand must be zero
        H_D[t - 1, :] = np.zeros(N)

    # fourth step, record the reward in this period
    if price_offered_index < K + 1:
        H_reward[t - 1] = P_list[price_offered_index - 1, :].dot(H_D[t - 1, :])  # record the realization of demand
    else: # the demand must be zero
        H_reward[t - 1] = 0

    # fourth step, record the remain inventory, size(A) = [N, M]
    H_I[t, :] = H_I[t - 1, :] - np.transpose(A).dot(H_D[t - 1, :])
    if not all(H_I[t, :] >= 0):
        break

print("Total reward of known distribution: {:f}".format(np.sum(H_reward)))
```

    Total reward of known distribution: 4254.000000

```python
# then input the experiment into csc file, we use period as index
data = pd.DataFrame({"Pricing_index": H_P, "Reward": H_reward})
data['Period_index'] = range(1, T+1)
# add the demand of each product
data = pd.concat([data, pd.DataFrame(H_D, columns = ["product_{:d}_demand".format(nindex) for nindex in range(1,N+1)])], axis = 1)
# add the remain invetory of each resource
data = pd.concat([data, pd.DataFrame(H_I[0:T, :], columns = ["resouce_{:d}".format(mindex) for mindex in range(1,M+1)])], axis = 1)
# add the constraint value
data = pd.concat([data, pd.DataFrame(H_constraint_value[0:T, :], columns = ["constraint_{:d}".format(conindex) for conindex in range(0,M+1)])], axis = 1)
# add the best solution in each round
data = pd.concat([data,
                  pd.DataFrame(H_bestX,\
                              columns = ["use_price_index_{:d}".format(kindex) for kindex in range(1,K+1)] + ["use_infinite_price"])],
                  axis = 1)
# add the estimation of alpha
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_alpha[0:T, :, :], newshape =(T, K * N)),
                             columns = ["alpha_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
# add the estimation of alpha
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_beta[0:T, :, :], newshape =(T, K * N)),
                             columns = ["beta_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
# add the estimation of theta
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_theta[0:T, :, :], newshape =(T, K * N)),
                             columns = ["theta_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
exp_filename = moment + "_K-{:d}_N-{:d}_M-{:d}_T-{:d}_randomseed-{:d}_RealThetaExperiment.csv".format(K, N, M, T, random_seed)
data.to_csv(".\\Result_Ferreira et.al. 2018 Algorithm 1\\" + exp_filename, index = False)
```

## 2 nd algorithm

We implement the 4th algorithm of Ferreira et.al. 2018

$c_j(t) = \frac{I_j(t-1)}{T-t+1}$

```python
%reset -f
import numpy as np
import pandas as pd

import pyscipopt
from pyscipopt import quicksum

random_seed = 12345
np.random.seed(random_seed)
```

```python
# Generate parameters
K = np.random.randint(low = 1, high = 5) # Total number of available price vectors
N = np.random.randint(low = 1, high = 5) # Total number of products
M = np.random.randint(low = 1, high = 5) # Total number of kinds of resource
T = 1000 # Total number of periods

# generate the demand

# each row represent an admissible pricing strategy
P_list = np.float64(np.random.randint(low = 1, high = 10, size = (K, N)))

# initialize inventory
I_0 = np.float64(np.random.randint(low = 9000, high = 11000, size = M))
c = np.zeros(shape = (T, M)) # we would update c in each period

# initialize a_ij
A = np.float64(np.random.randint(low = 5, high = 15, size = (N,M)))

# initialize real parameter theta
theta = np.random.beta(a = 1, b = 1, size = (K,N))

# initialize history
H_P = np.zeros(shape = T) # the index of pricing vector we used in each period
H_D = np.zeros(shape = (T, N)) # the demand of products in each period

H_I = np.zeros(shape = (T + 1, M)) # avaliable inventory in each period
H_I[0, :] = np.float64(I_0)

H_bestX = np.zeros(shape = (T, K + 1)) # the best solution in each optimization

H_reward = np.zeros(T) # the reward in each period

# each realization of price vector, index of period,
# corresponds to a estimate of theta
H_alpha = np.zeros(shape = (T + 1,K,N))
H_beta = np.zeros(shape = (T + 1,K,N))
H_alpha[0, :, :] = 1*np.ones(shape = (K,N))
H_beta[0, :, :] = 1*np.ones(shape = (K,N))

# initialize the constraint value in each round
# M kinds of resources correspond to M constraints, and one more constraint is x1 + ... + xN <=1
H_constraint_value = np.zeros(shape = (T, M + 1))

# vectorize beta sample function to accelerate
mybeta = np.vectorize(np.random.beta)
H_theta = np.zeros(shape = (T,K,N))

# vectorize binomial sample function to accelerate
mybinomial = np.vectorize(np.random.binomial)
```

```python
# implementation of algorithm 2
for t in range(1, T+1):
    # first step, sample from posterior distribution
    # H_alpha[t-1, :, :], H_beta[t-1, :, :] is the history data from 0 to t
    # H_theta[t-1, :, :] is the sample theta we used in round t
    H_theta[t-1, :, :] = mybeta(H_alpha[t-1, :, :], H_beta[t-1, :, :])

    # first step, calculate the mean demand given sample theta
    demand_mean = H_theta[t-1, :, :]

    # second step, optimize a linear function
    model = pyscipopt.Model("Optimization in Rount {:d}".format(t))
    # generate decision variable
    x = {}
    for xindex in range(1, K+1):
        x[xindex] = model.addVar(vtype="C", lb = 0, ub = 1, name="x{:d}".format(xindex))

    # second step, generate object function
    obj_coefficient = np.sum(demand_mean *  P_list, axis = 1)# obj_coefficient[k] = $\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1}$
    model.setObjective(quicksum(x[xindex]*obj_coefficient[xindex-1] for xindex in range(1, K+1)), "maximize")
    # objective = $\sum_{k=1}^K(\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1})x_{k}$

#     # second step, add constraint x_1+...+x_k<=1
#     model.addCons(quicksum(x[xindex] for xindex in range(1, K+1)) <= 1)

#     # second step, for each resources, we require \sum_{k=1}^K\sum_{i=1}^N d_{i,k}a_{i,j}x_l<=c_j
#     c[t - 1, :] = H_I[t - 1, :] / (T - t + 1)
#     for jj in range(1, M+1):
#         # size(A) = [N, M], size(demand_mean) = [K, N]
#         con_coefficient = A[:, jj - 1].dot(np.transpose(demand_mean))# con_coefficient[k] = $\sum_{i=1}^N a_{i,j}d_{i,k+1}$
#         model.addCons(quicksum(x[xindex] * con_coefficient[xindex - 1] for xindex in range(1, K+1)) <= c[t - 1, jj - 1])

#     # second step, optimize the problem
#     model.optimize()
#     bestx = np.zeros(K+1) # p_{K+1} would force the demand be zero
#     for xindex in range(1,K+1):
#         bestx[xindex - 1] = model.getVal(x[xindex])
#     bestx[K] = 1 - np.sum(bestx[0:K])
#     eliminate_error = lambda x: 0 if np.abs(x) < 1e-10 else x #there would be numerical error in the best solution
#     bestx = np.array([eliminate_error(x) for x in bestx])
#     bestx = bestx / np.sum(bestx)

#     # third step, offer price
#     price_offered_index = np.random.choice(np.arange(1, K + 2), p = bestx)

    # second step, add constraint x_1+...+x_k<=1
    constraint_index = {}
    constraint_index[0] = model.addCons(quicksum(x[xindex] for xindex in range(1, K+1)) <= 1)

    # second step, for each resources, we require \sum_{k=1}^K\sum_{i=1}^N d_{i,k}a_{i,j}x_l<=c_j
    c[t - 1, :] = H_I[t - 1, :] / (T - t + 1)
    for jj in range(1, M+1):
        # size(A) = [N, M], size(demand_mean) = [K, N]
        con_coefficient = A[:, jj - 1].dot(np.transpose(demand_mean))# con_coefficient[k] = $\sum_{i=1}^N a_{i,j}d_{i,k+1}$
        constraint_index[jj] = model.addCons(quicksum(x[xindex] * con_coefficient[xindex - 1] for xindex in range(1, K+1)) <= c[t - 1, jj - 1])

    # second step, optimize the problem
    model.optimize()
    bestx = np.zeros(K+1) # p_{K+1} would force the demand be zero
    for xindex in range(1,K+1):
        bestx[xindex - 1] = model.getVal(x[xindex])
    bestx[K] = 1 - np.sum(bestx[0:K])
    eliminate_error = lambda x: 0 if np.abs(x) < 1e-10 else x #there would be numerical error in the best solution
    bestx = np.array([eliminate_error(x) for x in bestx])
    bestx = bestx / np.sum(bestx)

    # third step, offer price
    price_offered_index = np.random.choice(np.arange(1, K + 2), p = bestx)

    # fourth step, update estimate of parameter
    H_P[t - 1] = price_offered_index # record the index of offered price

    # fourth step, record the constraint value in optimization
    H_constraint_value[t - 1, 0] = np.sum(bestx[0:K])
    for jj in range(1, M+1):
        con_coefficient = np.array(list(model.getValsLinear(constraint_index[jj]).values()))
        H_constraint_value[t - 1, jj] = np.sum(bestx[0:K] * con_coefficient)

    # fourth step, update estimate of parameter
    H_P[t - 1] = price_offered_index # record the index of offered price

    # fourth step, record the optimal solution in this round
    H_bestX[t - 1, :] = bestx

    # fourth step, record the realization of demand
    if price_offered_index < K + 1:
        H_D[t - 1, :] = mybinomial(np.ones(N), theta[price_offered_index - 1, :]) # record the realization of demand
    else: # the demand must be zero
        H_D[t - 1, :] = np.zeros(N)

    # fourth step, record the reward in this period
    if price_offered_index < K + 1:
        H_reward[t - 1] = P_list[price_offered_index - 1, :].dot(H_D[t - 1, :])  # record the realization of demand
    else: # the demand must be zero
        H_reward[t - 1] = 0

    # fourth step, record the remain inventory, size(A) = [N, M]
    H_I[t] = H_I[t - 1] - np.transpose(A).dot(H_D[t - 1, :])
    if not all(H_I[t] >= 0):
        break

    # fourth step, record the new estimate of alpha and beta
    if price_offered_index < K + 1:
        # if demand = 1, then alpha plus 1; if demand = 0, then alpha remain unchanged
        H_alpha[t, :, :] = H_alpha[t - 1, :, :]
        H_alpha[t, price_offered_index - 1, :] = H_alpha[t, price_offered_index - 1, :] + H_D[t - 1, :]

        # if demand = 1, then beta remained unchanged; if demand = 0, then beta plus 1
        H_beta[t, :, :] = H_beta[t - 1, :, :]
        H_beta[t, price_offered_index - 1, :] = H_beta[t - 1, price_offered_index - 1, :] + np.ones(N) - H_D[t - 1, :]
    else: # the demand must be zero, then all the estimate remain unchanged
        H_alpha[t, :, :] = H_alpha[t - 1, :, :]
        H_beta[t, :, :] = H_beta[t - 1, :, :]
print("Total reward of algorithm = {:f}".format(np.sum(H_reward)))
```

    Total reward of algorithm = 4259.000000

```python
# generate the csv data file and restore it into csv file
from datetime import datetime
moment = str(datetime.now()).replace(':', '-').replace(' ', '_')[:-7]
# the value of moment:  '2021-10-21_11-39-24', we used this varibale to name our files

# we firstly input the parameter into txt file
prameter_filename = moment + "_K-{:d}_N-{:d}_M-{:d}_T-{:d}_randomseed-{:d}_ParameterList.txt".format(K, N, M, T, random_seed)
with open(".\\Result_Ferreira et.al. 2018 Algorithm 2\\" + prameter_filename, 'w') as f:
    f.write("K = {:d}\n".format(K))
    f.write("N = {:d}\n".format(N))
    f.write("M = {:d}\n".format(M))
    f.write("randomseed = {:d}\n".format(random_seed))

    f.write("-----Admissible Prive Vector------\n")
    for kindex in range(K):
        f.write("Price vector {:d} : {:s}\n".format(kindex, str(P_list[kindex, :])))

    f.write("-----Initial Inventory------\n")
    f.write("Initial Inventory: {:s}\n".format(str(I_0)))

    f.write("-----Resource Consumption------\n")
    for nindex in range(N):
        f.write("Cost of resource of product {:d}: {:s}\n".format(nindex + 1, str(A[nindex, :])))

    f.write("-----Real theta------\n")
    for kindex in range(K):
        f.write("Given price vector {:d}, the real theta is {:s}\n".format(kindex + 1, str(theta[kindex, :])))

# then we input the experiment into csc file, we use period as index
data = pd.DataFrame({"Pricing_index": H_P, "Reward": H_reward})
data['Period_index'] = range(1, T+1)
# add the demand of each product
data = pd.concat([data, pd.DataFrame(H_D, columns = ["product_{:d}_demand".format(nindex) for nindex in range(1,N+1)])], axis = 1)
# add the remain invetory of each resource
data = pd.concat([data, pd.DataFrame(H_I[0:T, :], columns = ["resouce_{:d}".format(mindex) for mindex in range(1,M+1)])], axis = 1)
# add the constraint value
data = pd.concat([data, pd.DataFrame(H_constraint_value[0:T, :], columns = ["constraint_{:d}".format(conindex) for conindex in range(0,M+1)])], axis = 1)
# add the resource limits
data = pd.concat([data, pd.DataFrame(c[0:T, :], columns = ["c_{:d}".format(conindex) for conindex in range(1,M+1)])], axis = 1)
# add the best solution in each round
data = pd.concat([data,
                  pd.DataFrame(H_bestX,\
                              columns = ["use_price_index_{:d}".format(kindex) for kindex in range(1,K+1)] + ["use_infinite_price"])],
                  axis = 1)
# add the estimation of alpha
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_alpha[0:T, :, :], newshape =(T, K * N)),
                             columns = ["alpha_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
# add the estimation of alpha
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_beta[0:T, :, :], newshape =(T, K * N)),
                             columns = ["beta_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
# add the estimation of theta
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_theta[0:T, :, :], newshape =(T, K * N)),
                             columns = ["theta_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
exp_filename = moment + "_K-{:d}_N-{:d}_M-{:d}_T-{:d}_randomseed-{:d}_Experiment.csv".format(K, N, M, T, random_seed)
data.to_csv(".\\Result_Ferreira et.al. 2018 Algorithm 2\\" + exp_filename, index = False)
```
# clear the memory
H_reward = np.zeros(T) # the reward in each period
H_bestX = np.zeros(shape = (T, K + 1)) # the best solution in each optimization
H_P = np.zeros(shape = T) # the index of pricing vector we used in each period
H_D = np.zeros(shape = (T, N)) # the demand of products in each period
H_I = np.zeros(shape = (T + 1, M)) # avaliable inventory in each period
H_I[0, :] = np.float64(I_0)
c = np.zeros(shape = (T, M))# if we replace step 1 with true theta
for t in range(1, T+1):
    # first step, sample from posterior distribution
    # H_alpha[t-1, :, :], H_beta[t-1, :, :] is the history data from 0 to t
    # H_theta[t-1, :, :] is the sample theta we used in round t
#     H_theta[t-1, :, :] = mybeta(H_alpha[t-1, :, :], H_beta[t-1, :, :])

    # first step, calculate the mean demand given sample theta
    demand_mean = theta

    # second step, optimize a linear function
    model = pyscipopt.Model("Optimization in Rount {:d}".format(t))
    # generate decision variable
    x = {}
    for xindex in range(1, K+1):
        x[xindex] = model.addVar(vtype="C", lb = 0, ub = 1, name="x{:d}".format(xindex))

    # second step, generate object function
    obj_coefficient = np.sum(demand_mean *  P_list, axis = 1)# obj_coefficient[k] = $\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1}$
    model.setObjective(quicksum(x[xindex]*obj_coefficient[xindex-1] for xindex in range(1, K+1)), "maximize")
    # objective = $\sum_{k=1}^K(\sum_{i=1}^N d_{i,k+1}(t)p_{i,k+1})x_{k}$

    # second step, add constraint x_1+...+x_k<=1
    model.addCons(quicksum(x[xindex] for xindex in range(1, K+1)) <= 1)

    # second step, for each resources, we require \sum_{k=1}^K\sum_{i=1}^N d_{i,k}a_{i,j}x_l<=c_j
    c[t-1, :] = H_I[t-1, :] / (T - t + 1)
    for jj in range(1, M+1):
        # size(A) = [N, M], size(demand_mean) = [K, N]
        con_coefficient = A[:, jj - 1].dot(np.transpose(demand_mean))# con_coefficient[k] = $\sum_{i=1}^N a_{i,j}d_{i,k+1}$
        model.addCons(quicksum(x[xindex] * con_coefficient[xindex - 1] for xindex in range(1, K+1)) <= c[t - 1, jj - 1])

    # second step, optimize the problem
    model.optimize()
    bestx = np.zeros(K+1) # p_{K+1} would force the demand be zero
    for xindex in range(1,K+1):
        bestx[xindex - 1] = model.getVal(x[xindex])
    bestx[K] = 1 - np.sum(bestx[0:K])
    eliminate_error = lambda x: 0 if np.abs(x) < 1e-10 else x #there would be numerical error in the best solution
    bestx = np.array([eliminate_error(x) for x in bestx])
    bestx = bestx / np.sum(bestx)

    # third step, offer price
    price_offered_index = np.random.choice(np.arange(1, K + 2), p = bestx)

    # fourth step, update estimate of parameter
    H_P[t - 1] = price_offered_index # record the index of offered price

    # fourth step, record the optimal solution in this round
    H_bestX[t - 1, :] = bestx

    # fourth step, record the realization of demand
    if price_offered_index < K + 1:
        H_D[t - 1, :] = mybinomial(np.ones(N), theta[price_offered_index - 1, :]) # record the realization of demand
    else: # the demand must be zero
        H_D[t - 1, :] = np.zeros(N)

    # fourth step, record the reward in this period
    if price_offered_index < K + 1:
        H_reward[t - 1] = P_list[price_offered_index - 1, :].dot(H_D[t - 1, :])  # record the realization of demand
    else: # the demand must be zero
        H_reward[t - 1] = 0

    # fourth step, record the remain inventory, size(A) = [N, M]
    H_I[t, :] = H_I[t - 1, :] - np.transpose(A).dot(H_D[t - 1, :])
    if not all(H_I[t, :] >= 0):
        break

print("Total reward of known distribution: {:f}".format(np.sum(H_reward)))# then input the experiment into csc file, we use period as index
data = pd.DataFrame({"Pricing_index": H_P, "Reward": H_reward})
data['Period_index'] = range(1, T+1)
# add the demand of each product
data = pd.concat([data, pd.DataFrame(H_D, columns = ["product_{:d}_demand".format(nindex) for nindex in range(1,N+1)])], axis = 1)
# add the remain invetory of each resource
data = pd.concat([data, pd.DataFrame(H_I[0:T, :], columns = ["resouce_{:d}".format(mindex) for mindex in range(1,M+1)])], axis = 1)
# add the best solution in each round
data = pd.concat([data,
                  pd.DataFrame(H_bestX,\
                              columns = ["use_price_index_{:d}".format(kindex) for kindex in range(1,K+1)] + ["use_infinite_price"])],
                  axis = 1)
# add the estimation of alpha
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_alpha[0:T, :, :], newshape =(T, K * N)),
                             columns = ["alpha_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
# add the estimation of alpha
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_beta[0:T, :, :], newshape =(T, K * N)),
                             columns = ["beta_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
# add the estimation of theta
data = pd.concat([data,\
                 pd.DataFrame(np.reshape(a = H_theta[0:T, :, :], newshape =(T, K * N)),
                             columns = ["theta_k-{:d}_n-{:d}".format(k, n) for k in range(1, K+1) for n in range(1, N+1) ])],
                axis = 1)
exp_filename = moment + "_K-{:d}_N-{:d}_M-{:d}_T-{:d}_randomseed-{:d}_RealThetaExperiment.csv".format(K, N, M, T, random_seed)
data.to_csv(".\\Result_Ferreira et.al. 2018 Algorithm 2\\" + exp_filename, index = False)c
## 3rd algotithm

## 4 th alogrithm
We implement the 4th algorithm of Ferreira et.al. 2018

Following is its notation and description

|      Parameter       | description                                                  |
| :------------------: | :----------------------------------------------------------- |
|         $K$          | Total number of available price vectors                      |
|         $N$          | Total number of products                                    |
|         $M$          | Total number of kinds of resource                            |
|         $T$          | Total number of periods                                      |
|        $[x]$         | We define $[x]$ as a set, $[x]=\{1,2,\cdots, x\}$            |
|         $i$          | Index of products                                            |
|         $j$          | Index of resources                                           |
|         $t$          | Index of period                                              |
|     $I_j,I_j(t)$     | $I_j$ is the initial inventory for each resource $j\in [M]$, $I_j(t)$ is the inventory at the end of period $t$. $I_j(0)=I_j$ |
|       $a_{ij}$       | When we produced one unit item $i$, it would consume $a_{ij}$ unit $j$ |
|        $c_j$         | we define $c_j=\frac{I_j}{t}$                                |
|        $p_k$         | We define $\{p_1,p_2,\cdots,p_K\}$ as the admissible price vectors, each $p_k$ is a $N\times 1$ vector, specifying the price of each product, $p_k=(p_{1k},\cdots,p_{Nk})$, where $p_{ik}$ is the price of product $i$, for $i\in [N]$. We define $p_{\infty}$ as a "shut-off" price, such that the demand for any product under this price is zero. |
|        $P(t)$        | We denote by $P(t)=(P_1(t),\cdots,P_N(t))$ the prices chosen by the retailer in this period, and require that $P(t)\in \{p_1,p_2,\cdots,p_K,p_{\infty}\}$ |
|        $D(t)$        | We denote by $D(t) = (D_1(t),\cdots,D_N(t))$ the demand of each product at period $t$. We assume that given $P(t)=p_k$, the demand $D(t)$ is sampled from a probability distribution on $\mathbb{R}^{N}_+$ with joint cumulative distribution function (CDF) $F (x_1,\cdots,x_N, pk, \theta )$. $D(t)$ is independent of the history $\mathcal{H}_{t-1}$, given $P(t)$ |
|       $\theta$       | $\theta$ is the parameter of demand distribution, takes values in the parameter space $\Theta\subset\mathbb{R}^l$. The nature would sample $\theta$ from a prior distribution at the beginning form the process. The distribution is assumed to be subexponential; |
|  $\mathcal{H}_{t }$  | $\mathcal{H}_{t}=(P(1),D(1),\cdots,P(t),D(t))$               |
|       $\xi(t)$       | At the beginning of each period $t\in[T]$, the retailer observes some context $\xi(t)$, $\xi(t)$ belongs to some discrete set $\mathcal{X}$. We assume $\xi(t)$ is sampled i.i.d from a known distribution. |
 | $d_{ik}(\xi|\theta)$ | The mean demand of product $i\in [N]$ under price vector $p_k$, $\forall k\in[K]$, given context $\xi$ and parameter $\theta$ |

We assume the distribution of demand is calculated as follows:
\begin{equation}
(D_1,D_2,\cdots,D_{N}) \sim F(d_1, d_2,\cdots,d_N|P,\xi,\theta)
\end{equation}

$P$ is the price vector we offer, $\xi$ is the observed context variable, $\theta$ is the parameter that we need to estimate, it is sampled from the nature at the beginning of our study. We assume $\theta \sim f_{\Theta}(\theta)$

We furtherly assume the density function of this distribution is $f(d_1, d_2,\cdots,d_N|P,\xi,\theta)$.  Then we will derive the posterior distribution of $ \theta$

$$
\begin{align}
&f(\theta \ | \ (P_1,\xi_1,D^{(1)}), \ (P_2,\xi_2,D^{(2)}),\cdots, \ (P_t,\xi_t,D^{(t)}))\\
=&\frac{f(\theta, \ (P_1,\xi_1,D^{(1)}), \ (P_2,\xi_2,D^{(2)}),\cdots, \ (P_t,\xi_t,D^{(t)}))}{\int_{-\infty}^{\infty}f_{\Theta}(\theta)\prod_{i=1}^tf(d^{(i)}_1, d^{(i)}_2,\cdots,d^{(i)}_N|P_i,\xi_i,\theta)d\theta}\\
=&\frac{f(\ (P_1,\xi_1,D^{(1)}), \ (P_2,\xi_2,D^{(2)}),\cdots, \ (P_t,\xi_t,D^{(t)})\ | \ \theta) f_{\Theta}(\theta)}{\int_{-\infty}^{\infty}f_{\Theta}(\theta)\prod_{i=1}^tf(d^{(i)}_1, d^{(i)}_2,\cdots,d^{(i)}_N|P_i,\xi_i,\theta)d\theta}\\
=&\frac{ f_{\Theta}(\theta)\prod_{i=1}^t f(P_t,\xi_t,D^{(t)}| \ \theta)}{\int_{-\infty}^{\infty}f_{\Theta}(\theta)\prod_{i=1}^tf(d^{(i)}_1, d^{(i)}_2,\cdots,d^{(i)}_N|P_i,\xi_i,\theta)d\theta}\\
=&\frac{ f_{\Theta}(\theta)\prod_{i=1}^t f(D^{(i)}| \ \theta, P_t,\xi_t,)}{\int_{-\infty}^{\infty}f_{\Theta}(\theta)\prod_{i=1}^tf(d^{(i)}_1, d^{(i)}_2,\cdots,d^{(i)}_N|P_i,\xi_i,\theta)d\theta}\\
=&\frac{ f_{\Theta}(\theta)\prod_{i=1}^t f(d^{(i)}_1, d^{(i)}_2,\cdots,d^{(i)}_N| \ \theta, P_t,\xi_t,)}{\int_{-\infty}^{\infty}f_{\Theta}(\theta)\prod_{i=1}^tf(d^{(i)}_1, d^{(i)}_2,\cdots,d^{(i)}_N|P_i,\xi_i,\theta)d\theta}\\
\end{align}
$$

$P_i$ is the pricing vector we choose in the $i^{th}$ period

$\xi_i$ is the context we observed in the $i^{th}$ period

$D^{(i)}$ is the demand vector in $i^{th}$ period

$d^{(i)}_j$ is the demand of $j^{th}$ product in $i^{th}$ period

To implement this procedure, we need to solve 3 problems
1. Given the real demand generator, we need to calculate its density function.
2. sample random number from a customized density function
3. calculate the integral, numerically

We assume the demand follows discrete distribution, that is $d_i$

```python
# Generate parameters
np.random.seed(12345)

K = np.random.randint(low = 1, high = 5) # Total number of available price vectors
N = np.random.randint(low = 1, high = 5) # Total number of products
M = np.random.randint(low = 1, high = 5) # Total number of kinds of resource
T = 1000 # Total number of periods

# generate the demand

# each row represent an admissible pricing strategy
P_list = np.float64(np.random.randint(low = 1, high = 10, size = (K, M)))

# type of context
xi_type_num = np.random.randint(low = 3, high = 5 ) # maximum number of context type
xi_list = np.float64(np.arange(1, xi_type_num, 1))

# initialize inventory
I_0 = np.random.randint(low = 9000, high = 11000, size = M)

# initialize a_ij
A = np.float64(np.random.randint(low = 5, high = 15, size = (N,M)))

# real distribution of theta, we adopt beta distribution here,
# here theta is the parameter of beta dirstribution
# we assume the alpha and beta is independent of pricing vector , observed vector, and product
alpha_real_scalar = np.random.randint(low = 1, high = 3)
beta_real_scalar = np.random.randint(low = 2, high = 5)
alpha_real = alpha_real_scalar * np.ones(shape = (K, xi_type, N))
beta_real = beta_real_scalar * np.ones(shape = (K, xi_type, N))
# alpha_real = np.random.randint(low = 1, high = 3, size = (K, xi_type, N))
# beta_real = np.random.randint(low = 2, high = 5, size = (K, xi_type, N))

# initialize history
H_P = np.zeros(shape = T) # the index of pricing vector we used in each period
H_xi = np.zeros(shape = T)# the index of observed context in each period
H_D = np.zeros(shape = (T, N)) # the demand of products in each period

H_I = np.zeros(shape = (T, M)) # avaliable inventory in each period
H_I[0, :] = np.float64(I_0)

# each realization of price vector, observed context, index of period,
# corresponds to a estimate
H_alpha = np.zeros(shape = (T,K,xi_type_num,N))
H_beta = np.zeros(shape = (T,K,xi_type_num,N))
for kk in range(0, K):
    for xi_index in range(0, xi_type_num):
        H_alpha[0, kk, xi_index, :] = 1*np.ones(N)
        H_beta[0, kk, xi_index, :] = 1*np.ones(N)
```

```python
# Given the pricing vector and parameter, we use this function to generate demand
def GetDemand(P, xi, theta, num = 1):
    # P is the pricing vector, N*1
    # xi is the context observed by the merchant
    # theta is the parameter of beta distribution, a N*2 vector
    # num is the size of return demand, we would return M*num array, each column denote a sample demand

    # return value is a M*num array, each column denote a sample demand
    demand_sample = np.zeros(N, num)
    for ii in range(N):
        demand_sample[ii, :] = np.random.beta(alpha = alpha[ii], beta = beta[ii], size = num) + xi*5 - P[ii]*2
    return demand_sample
```

We derive the update formular here

```python
def UpdatePosteriorDistribution(t, H_P, H_xi, H_D, H_alpha, H_beta):
    # t is the index of period, the history is available from 0-t
    # H_P is the index of pricing vector in each period
    # H_xi is the index of obeserved context in each period

    # alpha_real and beta_real is the estimation of alpha and beta, given
    alpha_real = alpha_real_scalar * np.ones(shape = (K, xi_type, N))
    beta_real = beta_real_scalar * np.ones(shape = (K, xi_type, N))
```

```python
# implementation of algorithm
for tt in range(0, T):
    # step 1, sample a randome parameter theta
    alpha_tt = H_alpha[tt]
    beta_tt = H_beta[tt]
    # step 1, calculate the mean demand
    d_mean_tt = np.zeros(N, K)
    for kk in range(0, K): # traverse all the admissable pricing vector

```

```python

```

```python
np.float64(12)
```

    <ipython-input-23-92a6b828a7ec>:1: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
    Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
      np.float(12)

    12.0

```python
np.ones(5)
```

    array([1., 1., 1., 1., 1.])

```python
theta_real
```

    array([0.25      , 0.33333333])

```python
xi_list
```

    array([1, 2, 3])

```python
P_list
```

    array([[2, 5],
           [6, 3],
           [2, 7]])

```python
P_list[:,0] + 2
```

    array([4, 8, 4])

```python
I_0
```

    array([10339,  9654])

```python
I
```

    array([[10339.,     0.,     0., ...,     0.,     0.,     0.],
           [ 9654.,     0.,     0., ...,     0.,     0.,     0.]])

```python
help(np.zeros)
```

    Help on built-in function zeros in module numpy:

    zeros(...)
        zeros(shape, dtype=float, order='C', *, like=None)

        Return a new array of given shape and type, filled with zeros.

        Parameters
        ----------
        shape : int or tuple of ints
            Shape of the new array, e.g., ``(2, 3)`` or ``2``.
        dtype : data-type, optional
            The desired data-type for the array, e.g., `numpy.int8`.  Default is
            `numpy.float64`.
        order : {'C', 'F'}, optional, default: 'C'
            Whether to store multi-dimensional data in row-major
            (C-style) or column-major (Fortran-style) order in
            memory.
        like : array_like
            Reference object to allow the creation of arrays which are not
            NumPy arrays. If an array-like passed in as ``like`` supports
            the ``__array_function__`` protocol, the result will be defined
            by it. In this case, it ensures the creation of an array object
            compatible with that passed in via this argument.

            .. note::
                The ``like`` keyword is an experimental feature pending on
                acceptance of :ref:`NEP 35 <NEP35>`.

            .. versionadded:: 1.20.0

        Returns
        -------
        out : ndarray
            Array of zeros with the given shape, dtype, and order.

        See Also
        --------
        zeros_like : Return an array of zeros with shape and type of input.
        empty : Return a new uninitialized array.
        ones : Return a new array setting values to one.
        full : Return a new array of given shape filled with value.

        Examples
        --------
        >>> np.zeros(5)
        array([ 0.,  0.,  0.,  0.,  0.])

        >>> np.zeros((5,), dtype=int)
        array([0, 0, 0, 0, 0])

        >>> np.zeros((2, 1))
        array([[ 0.],
               [ 0.]])

        >>> s = (2,2)
        >>> np.zeros(s)
        array([[ 0.,  0.],
               [ 0.,  0.]])

        >>> np.zeros((2,), dtype=[('x', 'i4'), ('y', 'i4')]) # custom dtype
        array([(0, 0), (0, 0)],
              dtype=[('x', '<i4'), ('y', '<i4')])

# Neural Contextual Bandits with UCB-based Exploration
Notation:

| Notation                 | Description                                                  |
| :----------------------- | :----------------------------------------------------------- |
| $K$                      | number of arms                                               |
| $T$                      | number of total rounds                                       |
| $t$                      | index of round                                               |
| $x_{t,a}$                | $x_{t,a}\in\mathbb{R}^d$, $a\in [K]$, it is the context, the context consists of $K$ feature vectors $\{x_{t,a}\in\mathbb{R}^d|a\in[K]\}$ |
| $a_t$                    | after observes the context, the agent select an action $a_t$ in round t |
| $r_{t,a_t}$              | the reward after the agent select action $a_t$               |
| $h$                      | we assume that $r_{t,a_t}=h(x_{t,a_t})+\xi_t$, h is an unknown function satisfying $0\le h(x)\le 1$ for any x |
| $\xi_t$                  | $\xi_t$ is v-sub-Gaussian noise conditioned on $x_{1,a_1},\cdots,x_{t-1,a_{t-1}}$, satisfying $\mathbb{E}\xi_t=0$ |
| $L$                      | the depth of neural network                                  |
| $m$                      | number of neural in each layer of network                    |
| $\sigma(x)$              | we define $\sigma(x)=\max\{x,0\}$                            |
| $W_1,\cdots,W_{L-1},W_L$ | the weight in neural network. $W_1\in\mathbb{R}^{m\times d}$, $W_i\in\mathbb{R}^{m\times m}$, $2\le i\le L-1$, $W_L\in\mathbb{R}^{m\times 1}$ |
| $\theta$                 | $\theta=[vec(W_1)^T,\cdots,vec(W_l)^T]\in\mathbb{R}^p$, $p=m+md+m^2(L-1)$ |
| $f(x;\theta)$            | we define $f(x;\theta)=\sqrt{m}W_L\sigma(W_{l-1}\sigma(\cdots\sigma(W_1x)))$ |

Initialization of parameters:

UCB algorithm:

we set $\mathcal{L}(\theta) = \sum_{i=1}^t\frac{(f(x_{i,a_i};\theta)-r_{i,a_i})^2}{2}+\frac{m\lambda||\theta-\theta^{(0)}||^2_2}{2}$

Then the gradient would be
$$
\nabla\mathcal{L}(\theta) = \sum_{i=1}^t(f(x_{i,a_i};\theta)-r_{i,a_i})\nabla f(x_{i,a_i};\theta) + m\lambda(\theta-\theta^{(0)})
$$

Forawar Algorithm of Neural Network
$$
\begin{align}
X_0 &= X\\
X_1 &=\sigma(W_1X_0)\\
X_2 &=\sigma(W_2X_1)\\
\cdots\\
X_{L-1}&=\sigma(W_{L-1}X_{L-2})\\
X_{L} &=W_L X_{L-1}
\end{align}
$$
$f(X) = X_L$

Backward Propagation
$$
\begin{align}
\nabla_{X_L}f &= 1\\
\nabla_{W_L}f &= X_{L-1}\\
\nabla_{X_{L-1}}f &= W_{L}\\
\\
\nabla_{W_{L-1}}f &= \nabla_{W_{L-1}}f(X_{L-1}(W_{L-1}, X_{L-2}), W_L)=\nabla_{X_{L-1}}f \cdot \nabla_{W_{L-1}}X_{L-1}(W_{L-1}, X_{L-2})\\
\nabla_{X_{L-2}}f &= \nabla_{X_{L-2}}f(X_{L-1}(W_{L-1}, X_{L-2}), W_L)=\nabla_{X_{L-1}}f \cdot \nabla_{X_{L-2}}X_{L-1}(W_{L-1}, X_{L-2})\\
\\
\nabla_{W_{L-2}}f &= \nabla_{W_{L-2}}f(X_{L-2}(W_{L-2}, X_{L-3}), W_L,W_{L-1})=\nabla_{X_{L-2}}f \cdot \nabla_{W_{L-2}}X_{L-2}(W_{L-2}, X_{L-3})\\
\nabla_{X_{L-3}}f &= \nabla_{X_{L-3}}f(X_{L-2}(W_{L-2}, X_{L-3}), W_L,W_{L-1})=\nabla_{X_{L-2}}f \cdot \nabla_{X_{L-3}}X_{L-2}(W_{L-2}, X_{L-3})\\
\cdots\\
\nabla_{W_{l}}f &= \nabla_{W_{l}}f(X_{l}(W_{l}, X_{l-1}), W_L,W_{L-1},\cdots,W_{l+1})=\nabla_{X_{l}}f \cdot \nabla_{W_{l}}X_{l}(W_{l}, X_{l-1})\\
\nabla_{X_{l-1}}f &= \nabla_{X_{l-1}}f(X_{l}(W_{l}, X_{l-1}), W_L,W_{L-1},\cdots,W_{l+1})=\nabla_{X_{l}}f \cdot \nabla_{X_{l-1}}X_{l}(W_{l}, X_{l-1})\\
\cdots\\
\nabla_{W_{1}}f &= \nabla_{W_{1}}f(X_{1}(W_{1}, X_{0}), W_L,W_{L-1},\cdots,W_{2})=\nabla_{X_{1}}f \cdot \nabla_{W_{1}}X_{1}(W_{1}, X_{0})\\
\nabla_{X_{0}}f &= \nabla_{X_{0}}f(X_{1}(W_{1}, X_{0}), W_L,W_{L-1},\cdots,W_{2})=\nabla_{X_{1}}f \cdot \nabla_{X_{0}}X_{1}(W_{1}, X_{0})\\
\end{align}
$$

$\nabla_{W_{l}}f$ is a matrix, to be specific, $\nabla_{W_{l}}f=\left[\begin{matrix}\frac{\partial f}{\partial w^{(l)}_{11}}&\cdots &\frac{\partial f}{\partial w^{(l)}_{1m}}\\ \vdots&& \vdots\\ \frac{\partial f}{\partial w^{(l)}_{m1}}&\cdots &\frac{\partial f}{\partial w^{(l)}_{mm}}\end{matrix}\right]$

$$
\begin{align}
\left[\begin{matrix}\frac{\partial f}{\partial w^{(l)}_{11}}&\cdots &\frac{\partial f}{\partial w^{(l)}_{1m}}\\ \vdots&& \vdots\\ \frac{\partial f}{\partial w^{(l)}_{m1}}&\cdots &\frac{\partial f}{\partial w^{(l)}_{mm}}\end{matrix}\right]=&
\left[\begin{matrix}\frac{\partial f}{\partial x^{(l)}_{1}} \frac{\partial x^{(l)}_{1}}{\partial w^{(l)}_{11}}&\cdots &\frac{\partial f}{\partial x^{(l)}_{1}}\frac{\partial x^{(l)}_{1}}{\partial w^{(l)}_{1m}}\\ \vdots&& \vdots\\ \frac{\partial f}{\partial x^{(l)}_{m}}\frac{\partial x^{(l)}_{m}}{\partial w^{(l)}_{m1}}&\cdots &\frac{\partial f}{\partial x^{(l)}_{m}}\frac{\partial x^{(l)}_{m}}{\partial w^{(l)}_{mm}}\end{matrix}\right]\\
=&
\left[\begin{matrix}\frac{\partial f}{\partial x^{(l)}_{1}} \mathbb{1}_{\sigma(w^{(l)}_{11}x^{(l-1)}_1+w^{(l)}_{12}x^{(l-1)}_2+\cdots+w^{(l)}_{1m}x^{(l-1)}_m)>0} x^{(l-1)}_{1}&\cdots &\frac{\partial f}{\partial x^{(l)}_{1}}\mathbb{1}_{\sigma(w^{(l)}_{11}x^{(l-1)}_1+w^{(l)}_{12}x^{(l-1)}_2+\cdots+w^{(l)}_{1m}x^{(l-1)}_m)>0} x^{(l-1)}_{m}\\ \vdots&& \vdots\\ \frac{\partial f}{\partial x^{(l)}_{m}}\mathbb{1}_{\sigma(w^{(l)}_{m1}x^{(l-1)}_1+w^{(l)}_{m2}x^{(l-1)}_2+\cdots+w^{(l)}_{mm}x^{(l-1)}_m)>0} x^{(l-1)}_{1}&\cdots &\frac{\partial f}{\partial x^{(l)}_{m}}\mathbb{1}_{\sigma(w^{(l)}_{m1}x^{(l-1)}_1+w^{(l)}_{m2}x^{(l-1)}_2+\cdots+w^{(l)}_{mm}x^{(l-1)}_m)>0} x^{(l-1)}_{m}\end{matrix}\right]\\
\end{align}
$$

$\nabla_{X_{l-1}}f$ is a vector, to be specific, $\nabla_{X_{l-1}}f=\left[\begin{matrix}\frac{\partial f}{\partial x^{(l-1)}_{1}}\\ \vdots\\ \frac{\partial f}{\partial x^{(l-1)}_{m}}\end{matrix}\right]$
$$
\begin{align}
\left[\begin{matrix}\frac{\partial f}{\partial x^{(l-1)}_{1}}\\ \vdots\\ \frac{\partial f}{\partial x^{(l-1)}_{m}}\end{matrix}\right]=&
\left[\begin{matrix}\frac{\partial f}{\partial x^{(l)}_{1}}\frac{\partial x^{(l)}_1}{\partial x^{(l-1)}_1}+\frac{\partial f}{\partial x^{(l)}_{2}}\frac{\partial x^{(l)}_{2}}{\partial x^{(l-1)}_{1}}+\cdots+\frac{\partial f}{\partial x^{(l)}_{m}}\frac{\partial x^{(l)}_{m}}{\partial x^{(l-1)}_{1}}\\ \vdots\\ \frac{\partial f}{\partial x^{(l)}_{1}}\frac{\partial x^{(l)}_{1}}{\partial x^{(l-1)}_{m}}+\frac{\partial f}{\partial x^{(l)}_{2}}\frac{\partial x^{(l)}_{2}}{\partial x^{(l-1)}_{m}}+\cdots+\frac{\partial f}{\partial x^{(l)}_{m}}\frac{\partial x^{(l)}_{m}}{\partial x^{(l-1)}_{m}}\end{matrix}\right]\\
=&
\left[\begin{matrix}\frac{\partial f}{\partial x^{(l)}_{1}}\mathbb{1}_{\sigma(w^{(l)}_{11}x^{(l-1)}_1+w^{(l)}_{12}x^{(l-1)}_2+\cdots+w^{(l)}_{1m}x^{(l-1)}_m)>0}w^{(l)}_{11}+\frac{\partial f}{\partial x^{(l)}_{2}}\mathbb{1}_{\sigma(w^{(l)}_{21}x^{(l-1)}_1+w^{(l)}_{22}x^{(l-1)}_2+\cdots+w^{(l)}_{2m}x^{(l-1)}_m)>0}w^{(l)}_{21}+\cdots+\frac{\partial f}{\partial x^{(l)}_{m}}\mathbb{1}_{\sigma(w^{(l)}_{m1}x^{(l-1)}_1+w^{(l)}_{m2}x^{(l-1)}_2+\cdots+w^{(l)}_{mm}x^{(l-1)}_m)>0}w^{(l)}_{m1}\\
\vdots\\
\frac{\partial f}{\partial x^{(l)}_{1}}\mathbb{1}_{\sigma(w^{(l)}_{11}x^{(l-1)}_1+w^{(l)}_{12}x^{(l-1)}_2+\cdots+w^{(l)}_{1m}x^{(l-1)}_m)>0}w^{(l)}_{1m}+\frac{\partial f}{\partial x^{(l)}_{2}}\mathbb{1}_{\sigma(w^{(l)}_{21}x^{(l-1)}_1+w^{(l)}_{22}x^{(l-1)}_2+\cdots+w^{(l)}_{2m}x^{(l-1)}_m)>0}w^{(l)}_{2m}+\cdots+\frac{\partial f}{\partial x^{(l)}_{m}}\mathbb{1}_{\sigma(w^{(l)}_{m1}x^{(l-1)}_1+w^{(l)}_{m2}x^{(l-1)}_2+\cdots+w^{(l)}_{mm}x^{(l-1)}_m)>0}w^{(l)}_{mm}\end{matrix}\right ]\\
=&\left[\begin{matrix}w^{(l)}_{11}&\cdots& w^{(l)}_{1m}\\ \vdots&&\vdots\\ w^{(l)}_{m1}&\cdots& w^{(l)}_{mm}\end{matrix}\right]^T
\left[\begin{matrix}\frac{\partial f}{\partial x^{(l)}_{1}}\mathbb{1}_{\sigma(w^{(l)}_{11}x^{(l-1)}_1+w^{(l)}_{12}x^{(l-1)}_2+\cdots+w^{(l)}_{1m}x^{(l-1)}_m)>0}\\ \vdots\\ \frac{\partial f}{\partial x^{(l)}_{m}}\mathbb{1}_{\sigma(w^{(l)}_{m1}x^{(l-1)}_1+w^{(l)}_{m2}x^{(l-1)}_2+\cdots+w^{(l)}_{mm}x^{(l-1)}_m)>0}\end{matrix}\right]\\
\end{align}
$$

```python
%reset -f
import numpy as np
```

```python
# the setting is based on the description of section 7.1 of the papaer

# Set the parameter of the network
L = 2
m = 20

# Set the parameter of the game
K = 2# Total number of actions,
T = 100 # Total number of periods
d = 2 # the dimension of context

# Total number of parameters in the nerual network
# the paper claims that p = m+md+m^2(L-1)
# let L = 2,
# it also claim that f(x,\theta)=\sqrt{m}W_2\sigma(W_1x)
# then the total number of neurals should be m + m * d + m * m * (L - 2)
p = m + m * d + m * m * (L - 2)
```

 We assume the reward follows reward = context^T * A^T * A * context + \xi

 $\xi$ is a random variable following standard normal distribution N(0, 1)

 A is d\*d matrix, randomly generated from N(0, 1)

 We assume the context is independent from the action and round index.

 Given action a and round index t, the context is randomly sample from a unit ball in dimension d

```python
A = np.random.normal(loc=0, scale=1, size=(d, d))
```

```python
# Initialize the hyper parameter of neural network here
# we fix gamma in each round, according to the description of section 3.1
gamma_t = 0.01 #{0.01, 0.1, 1, 10}
v = 0.1 #{0.01, 0.1, 1}
lambda_ = 0.01 #{0.01, 0.1, 1}
delta = 0.01 #{0.01, 0.1, 1}
S = 0.01 #{0.01, 0.1, 1, 10}
eta = 1e-3 #{0.001, 0.01, 0.1}
# we set J equal to round index t
```

```python
# following are functions that are used to play the game
def SampleContext(d, K):
    # according to the description, the context is uniformly distributed on a d-dimension sphere
    # d is the dimension of context, a scalar
    # K is the total number of arts, a scalar

    # this function return context, as an d*K matrix, each column corresponds a context of action

    context = np.random.normal(loc=0, scale=1, size=(d, K))
    length = np.sqrt( np.sum(context * context, axis = 0) )
    length = np.tile(length, (d, 1))
    context = context / length # each column represent a context
    return context

def GetRealReward(context, A):
    # context is the context of arm, a d*1 vector
    # A is the d*d matrix,

    # this function return the reward

    return context.transpose().dot(A.transpose().dot(A)).dot(context) + np.random.normal(loc=0, scale=1)
```

```python
# following are functions that are related to the neural network
def Relu(x):
    # Relu function
    # if x is a scalar, the return value would be a scalar
    # if x is a matrix, the return value would be a matrix
    return np.maximum(x,0)

def ReluDerivative(x):
    # the derivative of Relu function
    # if x is a scalar, the return value would be a scalar
    # if x is a matrix, the return value would be a matrix
#     return np.array([(lambda x: 1 if x > 0 else 0)(xi) for xi in x])
    if x > 0:
        return 1
    else:
        return 0

def NeuralNetwork(X, params, L, m):
    # X is the input, each column correspont to a context
    # params is a dictionay, each key corresponds to the weight in each layer
    # its keys are "w1" , "w2", ..., "w{:d}".format(L).
    # L is the number of layers
    # m is the number of neurals in each layer

    # the return value would be a dictionary, its key would be like "l0", "l1", "l2", ..., "l{:d}".format(L),
    # each key represent the value of a layer, "l0" is X, the last layer is output

    X_layer = {}
    X_layer["x0"] = X
    for l in range(1, L):
        X_layer["x" + str(l)] = Relu(params["w" + str(l)].dot(X_layer["x" + str(l-1)]))
    X_layer["x" + str(L)] = np.sqrt(m) * params["w" + str(L)].dot(X_layer["x" + str(L-1)])
    return X_layer

def GradientNeuralNetwork(X, params, L, m):
    # this function calculate the gradient of each parameter in the neural network
    # X is the context, a 1 dimension vecotr in R^d
    # params is a dictionary that stores the paraemeters of neural network
    # its keys are "w1" , "w2", ..., "w{:d}".format(L).
    # L is the number of layers
    # m is the number of neurals in each layer

    # vectorize the function we used
    myRelu = np.vectorize(Relu)
    myReluDerivative = np.vectorize(ReluDerivative)

    # we firstly calculate the value of each layer
    X_layer = NeuralNetwork(X, params, L, m)

    # then we calculate the gradient of "X_layer" and gradient of parameter
    grad_X_layer = {}
    grad_parameter = {}

    grad_X_layer["x" + str(L)] = 1
    grad_parameter["w" + str(L)] = np.sqrt(m) * X_layer["x" + str(L - 1)]
    grad_X_layer["x" + str(L - 1)] = np.sqrt(m) * params["w" + str(L)][0, :]
    for l in range(L - 1, 0, -1):
        grad_parameter["w" + str(l) ] = grad_X_layer["x" + str(l)] * myReluDerivative(X_layer["x" + str(l)])
        grad_parameter["w" + str(l) ] = np.matmul(np.expand_dims(grad_parameter["w" + str(l) ], axis=1),\
                                                  np.expand_dims(X_layer["x" + str(l - 1)], axis = 0))

        grad_X_layer["x" + str(l - 1)] = grad_X_layer["x" + str(l)] * myReluDerivative(X_layer["x" + str(l)])
        grad_X_layer["x" + str(l - 1)] = np.matmul(params["w" + str(l)].transpose(),\
                                                   np.expand_dims(grad_X_layer["x" + str(l - 1)] , axis = 1))
    return grad_parameter

def FlattenGradient(grad_parameter, L):
    # accroding to the paper, the function f should be a vector in R^p
    # but what we got in GradientNeuralNetwork is a dictionary,
    # we would use this function to convert the dictinary into a vector
    # grad_parameter is the gradient stored in a dictionary
    # L is the total number of layers

    # the return value is the flattern vector = [vec(w1)^T, vec(w2)^T, ..., vec(w_L)^T]^T

    # we firstly generate the order of all the parameter
    para_order = ["w" + str(l) for l in range(1, L + 1)] #e.g. para_order = ["w1", "w2", "w3"]
    # then we can combine all the 1 d arrays together
    gradient = np.concatenate([grad_parameter[para_name].flatten() for para_name in para_order])
    return gradient

def GradientLossFunction(X, params, L, m, r, theta_0, lambda_):
    # this function calculate the gradient of lossfunction
    # X is the matrix of observed contexts, each column represent a context
    # X is required to be a 2-D matrix here, even sometimes it may only contain one column
    # params is a dictionary that stores the paraemeters of neural network
    # its keys are "w1" , "w2", ..., "w{:d}".format(L).
    # L is the number of layers
    # m is the number of neurals in each layer
    # r is the reward in each round
    # r is required to be a vector here, even sometimes it may only contain one column
    # theta_0 is the initalized value of parameters, restored as a disctionary

    #we would repeatedly call GradientNeuralNetwork() to calculate the gradients here

    # firstly, we calculate the shape of X and r
    context_num = len(r)

    # secondly, we calculate the value of each layer
    X_layer = NeuralNetwork(X, params, L, m) # each value in X_layer would be a

    # secondly, we repeatedly call GradientNeuralNetwork() to calculate the gradient of regression part
    grad_loss = {}# apply for space
    for key in params.keys():
        grad_loss[key] = np.zeros(params[key].shape)

    for ii in range(1, context_num + 1):
        new_term = GradientNeuralNetwork(X[:, ii - 1], params, L, m)
        for key in grad_loss.keys():
            grad_loss[key] = grad_loss[key] + new_term[key] * (X_layer["x" + str(L)][0, ii - 1] - r[ii - 1])
#             grad_loss[key] = grad_loss[key] + new_term[key] * (X_layer["x" + str(L)][0, ii - 1] - r[ii - 1]) / context_num

    # thirdly, we calculate the gradient of regularization
    for key in grad_loss.keys():
        grad_loss[key] = grad_loss[key] + m * lambda_ * (params[key] - theta_0[key])

    return grad_loss

# X = np.array([[1, 1], [2, 3]])
# lambda_ = 0
# theta_0 = {"w1": np.zeros((4, 2)),
#            "w2": np.zeros((1,4))}
# r = np.array([0, 0])
# L = 2
# m = 4
# params = {"w1": np.array([[1, 2], [-3, 2], [0, -1], [3, 3]]),
#           "w2": np.array([[1, 2, 3, -1]])}
# grad_loss = GradientLossFunction(X, params, L, m, r, theta_0, lambda_)
# X_layer = NeuralNetwork(X, params, L, m)
# print(X_layer)
# print(grad_loss)

# grad_parameter = GradientNeuralNetwork(X[:, 0], params, L, m)
# print(grad_parameter)
# grad_parameter = GradientNeuralNetwork(X[:, 1], params, L, m)
# print(grad_parameter)
```

```python
# we implement the algorithm here
from copy import deepcopy

#set the random seed
np.random.seed(12345)

# initialize the value of parameter
theta_0 = {}
W = np.random.normal(loc = 0, scale = 4 / m, size=(int(m/2), int(m/2)))
w = np.random.normal(loc = 0, scale = 4 / m, size=(1, int(m/2)))
for key in range(1, L + 1):
    if key == 1:
        # this paper doesn't present the initialization of w1
        # in its setting, d = m, then he let theta_0["w1"]=[W,0;0,W]
        # but in fact d might not equal to m
        tempW = np.random.normal(loc = 0, scale = 4 / m, size=(int(m/2), int(d/2)))
        theta_0["w1"] = np.zeros((m, d))
        theta_0["w1"][0:int(m/2), 0:int(d/2)] = tempW
        theta_0["w1"][int(m/2):, int(d/2):] = tempW
    elif 2 <= key and key <= L - 1:
        theta_0["w" + str(key)] = np.zeros((m, m))
        theta_0["w" + str(key)][0:int(m/2), 0:int(m/2)] = W
        theta_0["w" + str(key)][int(m/2):, 0:int(m/2):] = W
    else:
        theta_0["w" + str(key)] = np.concatenate([w, -w], axis = 1)

params = deepcopy(theta_0)
Z_t_minus1 = lambda_ * np.eye(p)

np.random.seed(12345)

# implement NeuralUCB
reward = np.zeros(T)
X_history = np.zeros((d, T))
params_history = {}
grad_history = {}
for tt in range(1, T + 1):
    # observe \{x_{t,a}\}_{a=1}^{k=1}
    context_list = SampleContext(d, K)

    # compute the upper bound of reward
    U_t_a = np.zeros(K) # the upper bound of K actions
    predict_reward = np.zeros(K)
    for a in range(1, K + 1):
        predict_reward[a - 1] = NeuralNetwork(context_list[:, a - 1], params, L, m)['x' + str(L)][0]
        grad_parameter = GradientNeuralNetwork(context_list[:, a - 1], params, L, m)
        grad_parameter = FlattenGradient(grad_parameter, L)
        Z_t_minus1_inverse = np.linalg.inv(Z_t_minus1)
        U_t_a[a - 1] = predict_reward[a - 1] + gamma_t * np.sqrt(grad_parameter.dot(Z_t_minus1_inverse).dot(grad_parameter)/m)
    ind = np.argmax(U_t_a, axis=None) # ind is the index of action we would play

    # play ind and observe reward
    X_history[:, tt - 1] = context_list[:, ind]
    reward[tt - 1] = GetRealReward(context_list[:, ind], A)
    print("round {:d}, predicted reward {:4f},\
          predicted upper bound {:4f},\
          actual reward{:4f}".format(tt, predict_reward[ind], U_t_a[ind], reward[tt - 1]))
    assert(np.abs(predict_reward[ind]) <= 50)

    # compute Z_t_minus1
    grad_parameter = GradientNeuralNetwork(context_list[:, ind], params, L, m)
    grad_parameter = FlattenGradient(grad_parameter, L)
    grad_parameter = np.expand_dims(grad_parameter, axis = 1)
    Z_t_minus1 = Z_t_minus1 + grad_parameter.dot(grad_parameter.transpose()) / m

    # train neural network
    J = tt
    for j in range(J):
        grad_loss = GradientLossFunction(X_history[:, 0:tt], params, L, m, reward[0:tt], theta_0, lambda_)
        for key in params.keys():
            params[key] = params[key] - eta * grad_loss[key]

    params_history[tt] = params
    grad_history[tt] = grad_loss
```

    round 1, predicted reward -0.000424,          predicted upper bound 0.090640,          actual reward2.272125
    round 2, predicted reward 0.068763,          predicted upper bound 0.114806,          actual reward1.644206
    round 3, predicted reward 0.035864,          predicted upper bound 0.112615,          actual reward2.374053
    round 4, predicted reward 0.133829,          predicted upper bound 0.159915,          actual reward0.621021
    round 5, predicted reward 0.125655,          predicted upper bound 0.175896,          actual reward0.225242
    round 6, predicted reward 0.160420,          predicted upper bound 0.172339,          actual reward1.690004
    round 7, predicted reward 0.777125,          predicted upper bound 0.800167,          actual reward-0.560492
    round 8, predicted reward 0.983655,          predicted upper bound 1.010498,          actual reward0.744462
    round 9, predicted reward 1.065241,          predicted upper bound 1.076667,          actual reward-0.820915
    round 10, predicted reward 0.283184,          predicted upper bound 0.310808,          actual reward1.749863
    round 11, predicted reward 0.703293,          predicted upper bound 0.738148,          actual reward1.670348
    round 12, predicted reward 1.105121,          predicted upper bound 1.141263,          actual reward0.433601
    round 13, predicted reward 1.212571,          predicted upper bound 1.232599,          actual reward1.332337
    round 14, predicted reward 0.474813,          predicted upper bound 0.492311,          actual reward-0.914758
    round 15, predicted reward 1.054616,          predicted upper bound 1.076929,          actual reward2.096148
    round 16, predicted reward 0.316751,          predicted upper bound 0.331554,          actual reward0.559662
    round 17, predicted reward 1.622827,          predicted upper bound 1.635132,          actual reward1.440866
    round 18, predicted reward 0.930713,          predicted upper bound 0.973603,          actual reward1.502516
    round 19, predicted reward 1.353896,          predicted upper bound 1.373078,          actual reward0.416201
    round 20, predicted reward 1.559543,          predicted upper bound 1.568139,          actual reward0.512476
    round 21, predicted reward 0.834536,          predicted upper bound 0.867873,          actual reward0.425917
    round 22, predicted reward 1.317566,          predicted upper bound 1.328963,          actual reward2.076005
    round 23, predicted reward 1.364796,          predicted upper bound 1.393930,          actual reward-0.160265
    round 24, predicted reward 1.417134,          predicted upper bound 1.426571,          actual reward-1.330353
    round 25, predicted reward 1.056596,          predicted upper bound 1.067803,          actual reward0.817492
    round 26, predicted reward 0.295162,          predicted upper bound 0.315495,          actual reward0.331979
    round 27, predicted reward 1.039549,          predicted upper bound 1.048849,          actual reward1.552421
    round 28, predicted reward 1.288948,          predicted upper bound 1.322777,          actual reward1.542083
    round 29, predicted reward 1.139483,          predicted upper bound 1.150642,          actual reward1.281281
    round 30, predicted reward 0.928884,          predicted upper bound 0.943935,          actual reward1.528348
    round 31, predicted reward 0.526687,          predicted upper bound 0.542968,          actual reward1.059209
    round 32, predicted reward 1.145122,          predicted upper bound 1.155893,          actual reward0.838127
    round 33, predicted reward 1.031941,          predicted upper bound 1.057640,          actual reward-0.269146
    round 34, predicted reward 0.979327,          predicted upper bound 0.996686,          actual reward1.154079
    round 35, predicted reward 1.324289,          predicted upper bound 1.338024,          actual reward-0.529172
    round 36, predicted reward 1.217855,          predicted upper bound 1.227487,          actual reward0.535593
    round 37, predicted reward 0.759149,          predicted upper bound 0.770074,          actual reward1.832194
    round 38, predicted reward 1.029710,          predicted upper bound 1.040862,          actual reward0.805005
    round 39, predicted reward 1.251582,          predicted upper bound 1.260370,          actual reward1.802072
    round 40, predicted reward 0.688033,          predicted upper bound 0.701778,          actual reward1.214210
    round 41, predicted reward 1.014587,          predicted upper bound 1.024450,          actual reward-0.445522
    round 42, predicted reward 0.791284,          predicted upper bound 0.803736,          actual reward0.657000
    round 43, predicted reward 1.055941,          predicted upper bound 1.065663,          actual reward0.939589
    round 44, predicted reward 0.729106,          predicted upper bound 0.742272,          actual reward0.658019
    round 45, predicted reward 0.383343,          predicted upper bound 0.403322,          actual reward0.924442
    round 46, predicted reward 1.073350,          predicted upper bound 1.082610,          actual reward1.287910
    round 47, predicted reward 1.051787,          predicted upper bound 1.060078,          actual reward0.485091
    round 48, predicted reward 1.265421,          predicted upper bound 1.279897,          actual reward1.047104
    round 49, predicted reward 1.084986,          predicted upper bound 1.091933,          actual reward2.123704
    round 50, predicted reward 0.876523,          predicted upper bound 0.886180,          actual reward-0.025674
    round 51, predicted reward 1.229329,          predicted upper bound 1.243771,          actual reward1.423720
    round 52, predicted reward 0.492564,          predicted upper bound 0.508513,          actual reward0.822949
    round 53, predicted reward 0.924992,          predicted upper bound 0.933414,          actual reward-0.581617
    round 54, predicted reward 1.301908,          predicted upper bound 1.311082,          actual reward1.544164
    round 55, predicted reward 0.890817,          predicted upper bound 0.899964,          actual reward1.086808
    round 56, predicted reward 0.752403,          predicted upper bound 0.764056,          actual reward0.326341
    round 57, predicted reward 1.402462,          predicted upper bound 1.411922,          actual reward1.975315
    round 58, predicted reward 0.964382,          predicted upper bound 0.974079,          actual reward1.409764
    round 59, predicted reward 0.690136,          predicted upper bound 0.698323,          actual reward0.360301
    round 60, predicted reward 1.255878,          predicted upper bound 1.270911,          actual reward1.927858
    round 61, predicted reward 1.412268,          predicted upper bound 1.421108,          actual reward1.417952
    round 62, predicted reward 0.931399,          predicted upper bound 0.941860,          actual reward0.604083
    round 63, predicted reward 1.266366,          predicted upper bound 1.276277,          actual reward1.276250
    round 64, predicted reward 0.919050,          predicted upper bound 0.948325,          actual reward1.783952
    round 65, predicted reward 1.418305,          predicted upper bound 1.425967,          actual reward0.098886
    round 66, predicted reward 1.158272,          predicted upper bound 1.166320,          actual reward0.272244
    round 67, predicted reward 0.690655,          predicted upper bound 0.711684,          actual reward-0.158905
    round 68, predicted reward 0.483797,          predicted upper bound 0.494244,          actual reward1.397000
    round 69, predicted reward 0.603170,          predicted upper bound 0.633071,          actual reward0.938655
    round 70, predicted reward 1.001356,          predicted upper bound 1.009557,          actual reward-0.160535
    round 71, predicted reward 1.402945,          predicted upper bound 1.426201,          actual reward2.354198
    round 72, predicted reward 0.474606,          predicted upper bound 0.485840,          actual reward2.270110
    round 73, predicted reward 0.624751,          predicted upper bound 0.635042,          actual reward0.276979
    round 74, predicted reward 1.284081,          predicted upper bound 1.294900,          actual reward0.914317
    round 75, predicted reward 1.527126,          predicted upper bound 1.536955,          actual reward1.243799
    round 76, predicted reward 1.427915,          predicted upper bound 1.441402,          actual reward3.202854
    round 77, predicted reward 0.665389,          predicted upper bound 0.685482,          actual reward2.574900
    round 78, predicted reward 1.112521,          predicted upper bound 1.121283,          actual reward1.642484
    round 79, predicted reward 0.397173,          predicted upper bound 0.406823,          actual reward0.404134
    round 80, predicted reward 1.215799,          predicted upper bound 1.239467,          actual reward0.852867
    round 81, predicted reward 0.714865,          predicted upper bound 0.724042,          actual reward2.350419
    round 82, predicted reward 0.387147,          predicted upper bound 0.403358,          actual reward0.107181
    round 83, predicted reward 1.337601,          predicted upper bound 1.345103,          actual reward-0.036781
    round 84, predicted reward 1.699848,          predicted upper bound 1.710142,          actual reward0.361713
    round 85, predicted reward 1.638043,          predicted upper bound 1.645842,          actual reward1.355636
    round 86, predicted reward 0.531219,          predicted upper bound 0.542413,          actual reward-0.570691
    round 87, predicted reward 0.526051,          predicted upper bound 0.543833,          actual reward0.040482
    round 88, predicted reward 2.052348,          predicted upper bound 2.060005,          actual reward3.109648
    round 89, predicted reward 1.538383,          predicted upper bound 1.544247,          actual reward0.032526
    round 90, predicted reward 1.647936,          predicted upper bound 1.653114,          actual reward1.561023
    round 91, predicted reward 1.463811,          predicted upper bound 1.468371,          actual reward0.730551
    round 92, predicted reward 0.569166,          predicted upper bound 0.585575,          actual reward-0.606856
    round 93, predicted reward 0.404037,          predicted upper bound 0.415614,          actual reward-0.218457
    round 94, predicted reward 0.937470,          predicted upper bound 0.949118,          actual reward1.422678
    round 95, predicted reward 1.599067,          predicted upper bound 1.605454,          actual reward0.572199
    round 96, predicted reward 1.302039,          predicted upper bound 1.314894,          actual reward2.442352
    round 97, predicted reward 1.315582,          predicted upper bound 1.325291,          actual reward1.104891
    round 98, predicted reward 1.823110,          predicted upper bound 1.831120,          actual reward0.521556
    round 99, predicted reward 0.701061,          predicted upper bound 0.721147,          actual reward1.124101
    round 100, predicted reward 0.889067,          predicted upper bound 0.901706,          actual reward1.802893

```python
print(np.sum(reward))
```

    93.85659335284052

```python
# benchmark
np.random.seed(12345)

reward = np.zeros(T)
X_history = np.zeros((d, T))
params_history = {}
grad_history = {}
for tt in range(1, T + 1):
    # observe \{x_{t,a}\}_{a=1}^{k=1}
    context_list = SampleContext(d, K)

    # compute the upper bound of reward
    predict_reward = np.zeros(K)
    for a in range(1, K + 1):
        predict_reward[a - 1] = context_list[:, a - 1].transpose().dot(A.transpose().dot(A)).dot(context_list[:, a - 1])

    ind = np.argmax(predict_reward, axis=None) # ind is the index of action we would play

    # play ind and observe reward
    X_history[:, tt - 1] = context_list[:, ind]
    reward[tt - 1] = GetRealReward(context_list[:, ind], A)
    print("round {:d}, predicted reward {:4f},\
          actual reward{:4f}".format(tt, predict_reward[ind], reward[tt - 1]))
    assert(np.abs(predict_reward[ind]) <= 50)
```

```python
print(np.sum(reward))
```

```python
# code for reference
import h5py
import numpy as np
import argparse

def sigmoid(x):
    """
    define scale function
    """
    return np.exp(x)/(1.0+np.exp(x))

def RELU(x):
    return np.np.maximum(x,0)

def reluDerivative(x):
    return np.array([reluDerivativeSingleElement(xi) for xi in x])

def reluDerivativeSingleElement(xi):
    if xi > 0:
        return 1
    elif xi <= 0:
        return 0

def compute_loss(Y,V):
    L_sum = np.sum(np.multiply(Y, np.log(V)))
    m = Y.shape[1]
    L = -(1./m) * L_sum
    return L

def feed_forward(X, params):
    tempt={}
    tempt["Z"]=np.matmul(params["W"], X) + params["b1"]
    tempt["H"]=sigmoid(tempt["Z"])
    #tempt["H"]=RELU(tempt["Z"])
    tempt["U"]=np.matmul(params["C"], tempt["H"]) + params["b2"]
    tempt["V"]=np.exp(tempt["U"]) / np.sum(np.exp(tempt["U"]), axis=0)
    return tempt

def back_propagate(X, Y, params, tempt, m_batch):
    dU=tempt["V"]-Y
    dC=(1. / m_batch) * np.matmul(dU, tempt["H"].T)
    db2=(1. / m_batch) * np.sum(dU, axis=1, keepdims=True)
    dH=np.matmul(params["C"].T, dU)
    dZ = dH * sigmoid(tempt["Z"]) * (1 - sigmoid(tempt["Z"]))
    #dZ=dH*reluDerivative(tempt["Z"])
    dW = (1. / m_batch) * np.matmul(dZ, X.T)
    db1 = (1. / m_batch) * np.sum(dZ, axis=1, keepdims=True)
    grads={"dW":dW, "db1":db1, "dC":dC, "db2":db2}
    return grads

#hyperparameters
epochs=10
batch_size=1
batchs=np.int32(60000/batch_size)
LR=0.01
dh=100#number of hidden nodes

#getting 60000 samples of training data and 10000 samples of testing data
f=h5py.File('MNISTdata.hdf5','r')
x_test_set=np.float32(f['x_test'][:])
y_test_set=np.int32(np.array(f['y_test'][:,0])).reshape(-1,1)
x_train_set=np.float32(f['x_train'][:])
y_train_set=np.int32(np.array(f['y_train'][:,0])).reshape(-1,1)
f.close()
X=np.vstack((x_train_set,x_test_set))
Y=np.vstack((y_train_set,y_test_set))
num_samples=Y.shape[0]
Y=Y.reshape(1,num_samples)
Y_new = np.eye(10)[Y.astype('int32')]
Y_new = Y_new.T.reshape(10, num_samples)
X_train, X_test=X[:60000].T, X[60000:].T
Y_train, Y_test=Y_new[:,:60000], Y_new[:,60000:]

#building fully connected neural network with one hidden layer
#initialization of parameters
params={"b1":np.zeros((dh,1)),
        "W":np.random.randn(dh,784)*np.sqrt(1. / 784),
        "b2":np.zeros((10,1)),
        "C":np.random.randn(10,dh)*np.sqrt(1. / dh)}

#training the network
for num_epoches in range(epochs):
    if (num_epoches > 5):
        LR = 0.001
    if (num_epoches > 10):
        LR = 0.0001
    if (num_epoches > 15):
        LR = 0.00001
    #shuffle the training data
    shuffle_index=np.random.permutation(X_train.shape[1])
    X_train= X_train[:, shuffle_index]
    Y_train=Y_train[:, shuffle_index]

    for num_batch in range(batchs):
        left_index=num_batch*batch_size
        right_index=min(left_index+batch_size,x_train_set.shape[0]-1)
        m_batch=right_index-left_index
        X=X_train[:,left_index:right_index]
        Y=Y_train[:,left_index:right_index]

        tempt=feed_forward(X, params)
        grads = back_propagate(X, Y, params, tempt, 1)

        #gradient descent
        params["W"] = params["W"] - LR * grads["dW"]
        params["b1"] = params["b1"] - LR * grads["db1"]
        params["C"] = params["C"] - LR * grads["dC"]
        params["b2"] = params["b2"] - LR * grads["db2"]

    #compute loss on training data
    tempt = feed_forward(X_train, params)
    train_loss = compute_loss(Y_train, tempt["V"])
    #compute loss on test set
    tempt=feed_forward(X_test, params)
    test_loss = compute_loss(Y_test, tempt["V"])
    total_correct=0
    for n in range(Y_test.shape[1]):
        p = tempt["V"][:,n]
        prediction = np.argmax(p)
        if prediction == np.argmax(Y_test[:,n]):
            total_correct+=1
    accuracy = np.float32(total_correct) / (Y_test.shape[1])
    #print(params)
    print("Epoch {}: training loss = {}, test loss = {}, accuracy={}".format(
            num_epoches + 1, train_loss, test_loss, accuracy))
```