Table of Contents

1  Factorization Machine (FM)

1.1  Introduction

1.2  Implementation

2  Reference

```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir(os.path.join('..', '..', 'notebook_format'))
from formats import load_style
load_style(css_style = 'custom2.css', plot_style = False)
```

```python
os.chdir(path)

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
# 4. magic to enable retina (high resolution) plots
# https://gist.github.com/minrk/3301035
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format = 'retina'

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.metrics import roc_auc_score
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer

%watermark -a 'Ethen' -d -t -v -p numba,numpy,pandas,sklearn,matplotlib
```

    Ethen 2018-02-24 20:09:44

    CPython 3.6.3
    IPython 6.1.0

    numba 0.37.0
    numpy 1.14.1
    pandas 0.22.0
    sklearn 0.19.1
    matplotlib 2.1.0

# Factorization Machine (FM)

**Factorization Machine** type algorithms are a combination of linear regression and matrix factorization, the cool idea behind this type of algorithm is it aims model interactions between features (a.k.a attributes, explanatory variables) using factorized parameters. By doing so it has the ability to estimate all interactions between features even with extremely sparse data.

## Introduction

Normally, when we think of linear regression, we would think of the following formula:

\begin{align}
\hat{y}(\textbf{x}) = w_{0} + \sum_{i=1}^{n} w_{i} x_{i}
\end{align}

Where:

- $w_0$ is the bias term, a.k.a intercept.
- $w_i$ are weights corresponding to each feature vector $x_i$, here we assume we have $n$ total features.

This formula's advantage is that it can computed in linear time, $O(n)$. The drawback, however, is that it does not handle feature interactions. To capture interactions, we could introduce a weight for each feature combination. This is sometimes referred to as a $2_{nd}$ ordered polynomial. The resulting model is shown below:

\begin{align}
\hat{y}(\textbf{x}) = w_{0} + \sum_{i=1}^{n} w_{i} x_{i} +  \sum_{i=1}^n \sum_{j=i+1}^n w_{ij} x_{i} x_{j}
\end{align}

Compared to our previous model, this formulation has the advantages that it can capture feature interactions at least for two features at a time. But we have now ended up with a $O(n^2)$ complexity which means that to train the model, we now require a lot more time and memory. Another issue is that when we have categorical variables with high cardinality, after one-hot encoding them, we would end up with a lot of columns that are sparse, making it harder to actually capture their interactions (not enough data).

To solve this complexity issue, Factorization Machines takes inspiration from [matrix factorization](http://nbviewer.jupyter.org/github/ethen8181/machine-learning/blob/master/recsys/1_ALSWR.ipynb), and models the feature interaction using latent factors. Every feature $f_i$ has a corresponding latent factor $v_i$, and two features' interactions are modelled as $\langle \textbf{v}_i, \textbf{v}_{j} \rangle$, where $\langle \cdot \;,\cdot \rangle$ refers to the dot product of the two feature vector. If we assume its of size $k$ (this is a hyperparameter that we can tune). Then:

\begin{align}
\langle \textbf{v}_i, \textbf{v}_{j} \rangle = \sum_{f=1}^k v_{i,f} v_{j,f}
\end{align}

This leads of our new equation:

\begin{align}
\hat{y}(\textbf{x}) = w_{0} + \sum_{i=1}^{n} w_{i} x_{i} + \sum_{i=1}^{n} \sum_{j=i+1}^n \langle \textbf{v}_i , \textbf{v}_{j} \rangle x_i x_{j}
\end{align}

This is an improvement from our previous model (when we modeled each pair of interaction terms with weight $w_{ij}$) as the number of parameters is reduced from $n^2$ to $n \times k$, since $k \ll n$, which also helps mitigate overfitting issues. Using the naive way of formulating factorization machine results in a complexity of $O(kn^2)$, because all pairwise interactions have to be computed, but we can reformulate it to make it run in $O(kn)$.

\begin{align}
\sum_{i=1}^n \sum_{j=i+1}^n \langle \textbf{v}_i, \textbf{v}_{j} \rangle x_{i} x_{j}
&= \frac{1}{2} \sum_{i=1}^n \sum_{j=1}^n \langle \textbf{v}_i, \textbf{v}_{j} \rangle x_{i} x_{j} - \frac{1}{2} \sum_{i=1}^n \langle \textbf{v}_i , \textbf{v}_{i} \rangle x_{i} x_{i}  \\
&= \frac{1}{2}\left(\sum_{i=1}^n \sum_{j=1}^n \sum_{f=1}^k v_{i,f} v_{j,f} x_{i} x_{j} \right)\frac{1}{2}\left( \sum_{i=1}^n \sum_{f=1}^k v_{i,f} v_{i,f} x_{i} x_{i} \right) \\
&= \frac{1}{2}\left(\sum_{i=1}^n \sum_{j=1}^n \sum_{f=1}^k v_{i,f} v_{j,f} x_{i} x_{j}  -  \sum_{i=1}^n \sum_{f=1}^k v_{i,f} v_{i,f} x_{i} x_{i} \right) \\
&= \frac{1}{2} \sum_{f=1}^{k} \left( \left(\sum_{i=1}^n v_{i,f}x_{i} \right) \left( \sum_{j=1}^n v_{j,f}x_{j} \right) - \sum_{i=1}^{n} v_{i,f}^2 x_{i}^2 \right) \\
&= \frac{1}{2} \sum_{f=1}^{k} \left( \left( \sum_{i}^{n} v_{i,f}x_{i} \right)^2  - \sum_{i=1}^{n} v_{i,f}^2 x_{i}^2 \right)
\end{align}

Note, summing over different pairs is the same as summing over all pairs minus the self-interactions (divided by two). This is the reason why the value 1/2 is introduced from the beginning of the derivation.

This reformulated equation has a linear complexity in both $k$ and $n$, i.e. its computation is in $O(kn)$, substituting this new equation into the existing factorization machine formula, we end up with:

\begin{align}
\hat{y}(\textbf{x}) = w_{0} + \sum_{i=1}^{n} w_{i} x_{i} + \frac{1}{2} \sum_{f=1}^{k} \left( \left( \sum_{i}^{n} v_{i,f}x_{i} \right)^2  - \sum_{i=1}^{n} v_{i,f}^2 x_{i}^2 \right)
\end{align}

In a machine learning setting, factorization machine can be applied to different supervised prediction tasks:

- **Regression:**, in this case $\hat{y}(\textbf{x})$ can be used directly by minimizing the mean squared error between the model prediction and target value, e.g. $\frac{1}{N}\sum^{N}\big(y - \hat{y}(\textbf{x})\big)^2$
- **Classification:**, if we were to use it in a binary classification setting, we could then minimize the log loss, $\ln \big(e^{-y \cdot \hat{y}(\textbf{x})} + 1 \big)$, where $\sigma$ is the sigmoid/logistic function and $y \in {-1, 1}$.

To train factorization machine, we can use a gradient descent based optimization techniques, the parameters to be learned are $(w_0, \mathbf{w},$ and $\mathbf{V}$).

\begin{align}
\frac{\partial}{\partial\theta}\hat{y}(\textbf{x}) =
\begin{cases}
1,  & \text{if $\theta$ is $w_0$} \\
x_i, & \text{if $\theta$ is $w_i$} \\
x_i\sum_{j=1}^{n} v_{j,f}x_j - v_{i,f}x_{i}^2 & \text{if $\theta$ is $v_{i,f}$}
\end{cases}
\end{align}

- Notice that $\sum_{j=1}^n v_{j, f} x_j$ does not depend on $i$, thus it can be computed independently.
- The last formula above, can also be written as $x_i(\sum_{j=1}^{n} v_{j,f}x_j - v_{i,f}x_{i})$.
- In practice, we would throw in some L2 regularization to prevent overfitting.

As the next section contains implementation of the algorithm from scratch, the gradient of the log loss is also provided here for completeness. The predicted value $\hat{y}(\textbf{x})$ is replaced with $x$ for making the notation cleaner.

\begin{align}
\frac{d}{dx}\left[ \ln \big(e^{-yx} + 1 \big) \right]
&= \frac{1}{e^{-yx} + 1} \cdot  \frac{d}{dx}\left[e^{-yx} + 1 \right] \\
&= \frac{\frac{d}{dx}\left[e^{-yx} \right] + \frac{d}{dx}\left[1 \right]}{e^{-yx} + 1} \\
&= \frac{e^{-yx} \cdot \frac{d}{dx}\left[-yx \right] + 0}{e^{-yx} + 1} \\
&= \frac{e^{-yx} \cdot -y}{e^{-yx} + 1} \\
&= -\frac{ye^{-yx}}{e^{-yx} + 1} \\
&= -\frac{y}{e^{yx} + 1}
\end{align}

---

**Advantages:** We'll now wrap up the theoretical section of factorization machine, with some of its advantages:

- We can observe from the model equation that it can be computed in linear time.
- By leveraging ideas from matrix factorization, we can estimate higher order interaction effects even under very sparse data.
- Compared to traditional matrix factorization methods, which is restricted to modeling a user-item matrix, we can leverage other user or item specific features making factorization machine more flexible.

## Implementation

For the implementation of factorization machine, we'll use a for loop based code as I personally find it easier to comprehend for the gradient update section. There are different ways to speed up for loop based code in Python, such as using [Cython or Numba](http://nbviewer.jupyter.org/github/ethen8181/machine-learning/blob/master/python/cython/cython.ipynb), here we'll be using Numba.

```python
# using the example spam dataset
# read it in, extract the input and output columns
label_col = 'label_num'
sms = pd.read_table('sms.tsv', header = None, names = ['label', 'message'])
sms[label_col] = sms['label'].map({'ham': 0, 'spam': 1})
X = sms['message']
y = sms[label_col].values

# split X and y into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size = 0.25, random_state = 1)

# convert both sets' text column to document-term matrix;
# ideally, we would want to perform some preprocessing on
# our text data, but let's be lazy here as that's not
# the goal of this documentation
tfidf = TfidfVectorizer(min_df = 2, max_df = 0.5)
X_train_dtm = tfidf.fit_transform(X_train)
X_test_dtm = tfidf.transform(X_test)
X_train_dtm
```

    <4179x3508 sparse matrix of type '<class 'numpy.float64'>'
    	with 51261 stored elements in Compressed Sparse Row format>

```python
import numpy as np
from numba import njit
from tqdm import trange
from sklearn.base import BaseEstimator, ClassifierMixin

class FactorizationMachineClassifier(BaseEstimator, ClassifierMixin):
    """
    Factorization Machine [1]_ using Stochastic Gradient Descent.
    For binary classification only.

    Parameters
    ----------
    n_iter : int, default 10
        Number of iterations to train the algorithm.

    n_factors : int, default 10
        Number/dimension of features' latent factors.

    learning_rate : float, default 0.1
        Learning rate for the gradient descent optimizer.

    reg_coef : float, default 0.01
        Regularization strength for weights/coefficients.

    reg_factors : float, default 0.01
        Regularization strength for features' latent factors.

    random_state : int, default 1234
        Seed for the randomly initialized features latent factors

    verbose : bool, default True
        Whether to print progress bar while training.

    Attributes
    ----------
    intercept_ : double
        Intercept term, w0 based on the original notations.

    coef_ : 1d ndarray, shape [n_features,]
        Coefficients, w based on the original notations.

    feature_factors_ : 2d ndarray, shape [n_factors, n_features]
        Latent factors for all features. v based on the original
        notations. The learned factors can be viewed as the
        embeddings for each features. If a pair of features tends
        to co-occur often, then their embeddings should be
        close/similar (in terms of cosine similarity) to each other.

    history_ : list
        Loss function's history at each iteration, useful
        for evaluating whether the algorithm converged or not.

    References
    ----------
    .. [1] `S. Rendle Factorization Machines (2010)
            <http://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf>`_
    """

    def __init__(self, n_iter = 10, n_factors = 10,
                 learning_rate = 0.1, reg_coef = 0.01,
                 reg_factors = 0.01, random_state = 1234, verbose = False):
        self.n_iter = n_iter
        self.verbose = verbose
        self.reg_coef = reg_coef
        self.n_factors = n_factors
        self.reg_factors = reg_factors
        self.random_state = random_state
        self.learning_rate = learning_rate

    def fit(self, X, y):
        """
        Fit the model to the input data and label.

        Parameters
        ----------
        X : scipy sparse csr_matrix, shape [n_samples, n_features]
            Data in sparse matrix format.

        y : 1d ndarray, shape [n_samples,]
            Training data's corresponding label.

        Returns
        -------
        self
        """

        n_samples, n_features = X.shape
        self.coef_ = np.zeros(n_features)
        self.intercept_ = 0.0

        # the factors are often initialized with a mean of 0 and standard deviation
        # of 1 / sqrt(number of latent factor specified)
        np.random.seed(self.random_state)
        self.feature_factors_ = np.random.normal(
            scale = 1 / np.sqrt(self.n_factors), size = (self.n_factors, n_features))

        # the gradient is implemented in a way that requires
        # the negative class to be labeled as -1 instead of 0
        y = y.copy().astype(np.int32)
        y[y == 0] = -1

        loop = range(self.n_iter)
        if self.verbose:
            loop = trange(self.n_iter)

        self.history_ = []
        for _ in loop:
            loss = _sgd_update(X.data, X.indptr, X.indices,
                               y, n_samples, n_features,
                               self.intercept_, self.coef_,
                               self.feature_factors_, self.n_factors,
                               self.learning_rate, self.reg_coef, self.reg_factors)
            self.history_.append(loss)

        return self

    def predict_proba(self, X):
        """
        Probability estimates. The returned estimates for
        all classes are ordered by the label of classes.

        Paramters
        ---------
        X : scipy sparse csr_matrix, shape [n_samples, n_features]
            Data in sparse matrix format.

        Returns
        -------
        proba : 2d ndarray, shape [n_samples, n_classes]
            The probability of the sample for each class in the model.
        """
        pred = self._predict(X)
        pred_proba = 1.0 / (1.0 + np.exp(-pred))
        proba = np.vstack((1 - pred_proba, pred_proba)).T
        return proba

    def _predict(self, X):
        """Similar to _predict_instance but vectorized for all samples"""
        linear_output = X * self.coef_
        v = self.feature_factors_.T
        term = (X * v) ** 2 - (X.power(2) * (v ** 2))
        factor_output = 0.5 * np.sum(term, axis = 1)
        return self.intercept_ + linear_output + factor_output

    def predict(self, X):
        """
        Predict class labels for samples in X.

        Parameters
        ----------
        X : scipy sparse csr_matrix, shape [n_samples, n_features]
            Data in sparse matrix format.

        Returns
        -------
        Predicted class label per sample.
        """
        pred_proba = self.predict_proba(X)[:, 1]
        return pred_proba.round().astype(np.int)

@njit
def _sgd_update(data, indptr, indices, y, n_samples, n_features,
                w0, w, v, n_factors, learning_rate, reg_w, reg_v):
    """
    Compute the loss of the current iteration and update
    gradients accordingly.
    """
    loss = 0.0
    for i in range(n_samples):
        pred, summed = _predict_instance(data, indptr, indices, w0, w, v, n_factors, i)

        # calculate loss and its gradient
        loss += _log_loss(pred, y[i])
        loss_gradient = -y[i] / (np.exp(y[i] * pred) + 1.0)

        # update bias/intercept term
        w0 -= learning_rate * loss_gradient

        # update weight
        for index in range(indptr[i], indptr[i + 1]):
            feature = indices[index]
            w[feature] -= learning_rate * (loss_gradient * data[index] + 2 * reg_w * w[feature])

        # update factor
        for factor in range(n_factors):
            for index in range(indptr[i], indptr[i + 1]):
                feature = indices[index]
                term = summed[factor] - v[factor, feature] * data[index]
                v_gradient = loss_gradient * data[index] * term
                v[factor, feature] -= learning_rate * (v_gradient + 2 * reg_v * v[factor, feature])

    loss /= n_samples
    return loss

@njit
def _predict_instance(data, indptr, indices, w0, w, v, n_factors, i):
    """predicting a single instance"""
    summed = np.zeros(n_factors)
    summed_squared = np.zeros(n_factors)

    # linear output w * x
    pred = w0
    for index in range(indptr[i], indptr[i + 1]):
        feature = indices[index]
        pred += w[feature] * data[index]

    # factor output
    for factor in range(n_factors):
        for index in range(indptr[i], indptr[i + 1]):
            feature = indices[index]
            term = v[factor, feature] * data[index]
            summed[factor] += term
            summed_squared[factor] += term * term

        pred += 0.5 * (summed[factor] * summed[factor] - summed_squared[factor])

    # summed is the independent term that can be re-used
    # during the gradient update stage
    return pred, summed

@njit
def _log_loss(pred, y):
    """
    negative log likelihood of the
    current prediction and label, y.
    """
    return np.log(np.exp(-pred * y) + 1.0)
```

```python
fm = FactorizationMachineClassifier(n_iter = 30, learning_rate = 0.1)
fm.fit(X_train_dtm, y_train)
```

    FactorizationMachineClassifier(learning_rate=0.1, n_factors=10, n_iter=30,
                    random_state=1234, reg_coef=0.01, reg_factors=0.01,
                    verbose=False)

```python
# change default style figure and font size
plt.rcParams['figure.figsize'] = 8, 6
plt.rcParams['font.size'] = 12

# one quick way to check that we've implemented
# the gradient descent is to ensure that the loss
# curve is steadily decreasing
plt.plot(fm.history_)
plt.title('Loss Curve Per Iteration')
plt.xlabel('Iterations')
plt.ylabel('Loss')
plt.show()
```

```python
# predict on the test set and output the auc score
y_pred_prob = fm.predict_proba(X_test_dtm)[:, 1]
auc = roc_auc_score(y_test, y_pred_prob)
print('auc', auc)
```

    auc 0.9973867907642742

```python
# we can compare it with a logistic regression,
logreg = LogisticRegression()
logreg.fit(X_train_dtm, y_train)
y_pred_prob = logreg.predict_proba(X_test_dtm)[:, 1]
auc = roc_auc_score(y_test, y_pred_prob)
print('auc', auc)
```

    auc 0.9949615178092

There are various open-sourced implementations floating around the web, here are the links to some of them:

- https://github.com/ibayer/fastFM
- https://github.com/srendle/libfm
- https://github.com/aksnzhy/xlearn
- https://github.com/scikit-learn-contrib/polylearn

I personally haven't tested which one is more efficient, feel free to grab one of them as see if it helps solve your problem.

# Reference

- [Blog: Factorization Machines](http://www.jefkine.com/recsys/2017/03/27/factorization-machines/)
- [Blog: Deep Understanding of FFM Principles and Practices (Chinese)](https://tech.meituan.com/deep-understanding-of-ffm-principles-and-practices.html)
- [Quora: What are the drawbacks of Factorization Machines?](https://www.quora.com/What-are-the-drawbacks-of-Factorization-Machines)
- [Paper: S. Rendle Factorization Machines (2010)](https://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf)


##### Copyright 2020 The TensorFlow Authors.

```
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

# Hello, many worlds

    View on TensorFlow.org

    Run in Google Colab

    View source on GitHub

    Download notebook

This tutorial shows how a classical neural network can learn to correct qubit calibration errors. It introduces Cirq, a Python framework to create, edit, and invoke Noisy Intermediate Scale Quantum (NISQ) circuits, and demonstrates how Cirq interfaces with TensorFlow Quantum.

## Setup

```
!pip install tensorflow==2.3.1
```

    Collecting tensorflow==2.3.1
      Downloading https://files.pythonhosted.org/packages/eb/18/374af421dfbe74379a458e58ab40cf46b35c3206ce8e183e28c1c627494d/tensorflow-2.3.1-cp37-cp37m-manylinux2010_x86_64.whl (320.4MB)
         |████████████████████████████████| 320.4MB 42kB/s
    Requirement already satisfied: gast==0.3.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.1) (0.3.3)
    Requirement already satisfied: astunparse==1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.1) (1.6.3)
    Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.1) (0.12.0)
    Requirement already satisfied: google-pasta>=0.1.8 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.1) (0.2.0)
    Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.1) (3.3.0)
    Requirement already satisfied: keras-preprocessing<1.2,>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.1) (1.1.2)
    Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.1) (0.36.2)
    Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.1) (1.1.0)
    Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.1) (1.15.0)
    Requirement already satisfied: tensorboard<3,>=2.3.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.1) (2.4.1)
    Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.1) (1.12.1)
    Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.1) (3.12.4)
    Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.1) (2.10.0)
    Collecting tensorflow-estimator<2.4.0,>=2.3.0
      Downloading https://files.pythonhosted.org/packages/e9/ed/5853ec0ae380cba4588eab1524e18ece1583b65f7ae0e97321f5ff9dfd60/tensorflow_estimator-2.3.0-py2.py3-none-any.whl (459kB)
         |████████████████████████████████| 460kB 41.5MB/s
    Collecting numpy<1.19.0,>=1.16.0
      Downloading https://files.pythonhosted.org/packages/d6/c6/58e517e8b1fb192725cfa23c01c2e60e4e6699314ee9684a1c5f5c9b27e1/numpy-1.18.5-cp37-cp37m-manylinux1_x86_64.whl (20.1MB)
         |████████████████████████████████| 20.1MB 1.3MB/s
    Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.1) (1.32.0)
    Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (56.0.0)
    Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (1.28.1)
    Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (0.4.4)
    Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2.23.0)
    Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (1.8.0)
    Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (3.3.4)
    Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.1) (1.0.1)
    Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3.6" in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (4.7.2)
    Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (4.2.1)
    Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (0.2.8)
    Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (1.3.0)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2.10)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (2020.12.5)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (3.0.4)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (1.24.3)
    Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (3.10.1)
    Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.7/dist-packages (from rsa<5,>=3.1.4; python_version >= "3.6"->google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (0.4.8)
    Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (3.1.0)
    Requirement already satisfied: typing-extensions>=3.6.4; python_version < "3.8" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < "3.8"->markdown>=2.6.8->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (3.7.4.3)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < "3.8"->markdown>=2.6.8->tensorboard<3,>=2.3.0->tensorflow==2.3.1) (3.4.1)
    ERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.
    ERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.
    Installing collected packages: tensorflow-estimator, numpy, tensorflow
      Found existing installation: tensorflow-estimator 2.4.0
        Uninstalling tensorflow-estimator-2.4.0:
          Successfully uninstalled tensorflow-estimator-2.4.0
      Found existing installation: numpy 1.19.5
        Uninstalling numpy-1.19.5:
          Successfully uninstalled numpy-1.19.5
      Found existing installation: tensorflow 2.4.1
        Uninstalling tensorflow-2.4.1:
          Successfully uninstalled tensorflow-2.4.1
    Successfully installed numpy-1.18.5 tensorflow-2.3.1 tensorflow-estimator-2.3.0

Install TensorFlow Quantum:

```
!pip install tensorflow-quantum
```

    Collecting tensorflow-quantum
      Downloading https://files.pythonhosted.org/packages/53/02/878b2d4e7711f5c7f8dff9ff838e8ed84d218a359154ce06c7c01178a125/tensorflow_quantum-0.4.0-cp37-cp37m-manylinux2010_x86_64.whl (5.9MB)
         |████████████████████████████████| 5.9MB 5.0MB/s
    Collecting sympy==1.5
      Downloading https://files.pythonhosted.org/packages/4d/a7/25d5d6b3295537ab90bdbcd21e464633fb4a0684dd9a065da404487625bb/sympy-1.5-py2.py3-none-any.whl (5.6MB)
         |████████████████████████████████| 5.6MB 38.5MB/s
    Collecting cirq==0.9.1
      Downloading https://files.pythonhosted.org/packages/18/05/39c24828744b91f658fd1e5d105a9d168da43698cfaec006179c7646c71c/cirq-0.9.1-py3-none-any.whl (1.6MB)
         |████████████████████████████████| 1.6MB 36.6MB/s
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy==1.5->tensorflow-quantum) (1.2.1)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from cirq==0.9.1->tensorflow-quantum) (1.4.1)
    Requirement already satisfied: networkx~=2.4 in /usr/local/lib/python3.7/dist-packages (from cirq==0.9.1->tensorflow-quantum) (2.5.1)
    Requirement already satisfied: matplotlib~=3.0 in /usr/local/lib/python3.7/dist-packages (from cirq==0.9.1->tensorflow-quantum) (3.2.2)
    Requirement already satisfied: sortedcontainers~=2.0 in /usr/local/lib/python3.7/dist-packages (from cirq==0.9.1->tensorflow-quantum) (2.3.0)
    Collecting freezegun~=0.3.15
      Downloading https://files.pythonhosted.org/packages/17/5d/1b9d6d3c7995fff473f35861d674e0113a5f0bd5a72fe0199c3f254665c7/freezegun-0.3.15-py2.py3-none-any.whl
    Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from cirq==0.9.1->tensorflow-quantum) (1.1.5)
    Requirement already satisfied: google-api-core[grpc]<2.0.0dev,>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from cirq==0.9.1->tensorflow-quantum) (1.26.3)
    Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from cirq==0.9.1->tensorflow-quantum) (3.7.4.3)
    Requirement already satisfied: numpy~=1.16 in /usr/local/lib/python3.7/dist-packages (from cirq==0.9.1->tensorflow-quantum) (1.18.5)
    Requirement already satisfied: protobuf~=3.12.0 in /usr/local/lib/python3.7/dist-packages (from cirq==0.9.1->tensorflow-quantum) (3.12.4)
    Requirement already satisfied: requests~=2.18 in /usr/local/lib/python3.7/dist-packages (from cirq==0.9.1->tensorflow-quantum) (2.23.0)
    Requirement already satisfied: decorator<5,>=4.3 in /usr/local/lib/python3.7/dist-packages (from networkx~=2.4->cirq==0.9.1->tensorflow-quantum) (4.4.2)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib~=3.0->cirq==0.9.1->tensorflow-quantum) (2.4.7)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib~=3.0->cirq==0.9.1->tensorflow-quantum) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib~=3.0->cirq==0.9.1->tensorflow-quantum) (1.3.1)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib~=3.0->cirq==0.9.1->tensorflow-quantum) (2.8.1)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from freezegun~=0.3.15->cirq==0.9.1->tensorflow-quantum) (1.15.0)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->cirq==0.9.1->tensorflow-quantum) (2018.9)
    Requirement already satisfied: packaging>=14.3 in /usr/local/lib/python3.7/dist-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->cirq==0.9.1->tensorflow-quantum) (20.9)
    Requirement already satisfied: google-auth<2.0dev,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->cirq==0.9.1->tensorflow-quantum) (1.28.1)
    Requirement already satisfied: setuptools>=40.3.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->cirq==0.9.1->tensorflow-quantum) (56.0.0)
    Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->cirq==0.9.1->tensorflow-quantum) (1.53.0)
    Requirement already satisfied: grpcio<2.0dev,>=1.29.0; extra == "grpc" in /usr/local/lib/python3.7/dist-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->cirq==0.9.1->tensorflow-quantum) (1.32.0)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests~=2.18->cirq==0.9.1->tensorflow-quantum) (2020.12.5)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests~=2.18->cirq==0.9.1->tensorflow-quantum) (1.24.3)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests~=2.18->cirq==0.9.1->tensorflow-quantum) (2.10)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests~=2.18->cirq==0.9.1->tensorflow-quantum) (3.0.4)
    Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2.0dev,>=1.21.1->google-api-core[grpc]<2.0.0dev,>=1.14.0->cirq==0.9.1->tensorflow-quantum) (4.2.1)
    Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3.6" in /usr/local/lib/python3.7/dist-packages (from google-auth<2.0dev,>=1.21.1->google-api-core[grpc]<2.0.0dev,>=1.14.0->cirq==0.9.1->tensorflow-quantum) (4.7.2)
    Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2.0dev,>=1.21.1->google-api-core[grpc]<2.0.0dev,>=1.14.0->cirq==0.9.1->tensorflow-quantum) (0.2.8)
    Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.7/dist-packages (from rsa<5,>=3.1.4; python_version >= "3.6"->google-auth<2.0dev,>=1.21.1->google-api-core[grpc]<2.0.0dev,>=1.14.0->cirq==0.9.1->tensorflow-quantum) (0.4.8)
    Installing collected packages: sympy, freezegun, cirq, tensorflow-quantum
      Found existing installation: sympy 1.7.1
        Uninstalling sympy-1.7.1:
          Successfully uninstalled sympy-1.7.1
    Successfully installed cirq-0.9.1 freezegun-0.3.15 sympy-1.5 tensorflow-quantum-0.4.0

Now import TensorFlow and the module dependencies:

```
import tensorflow as tf
import tensorflow_quantum as tfq

import cirq
import sympy
import numpy as np

# visualization tools
%matplotlib inline
import matplotlib.pyplot as plt
from cirq.contrib.svg import SVGCircuit
```

## 1. The Basics

### 1.1 Cirq and parameterized quantum circuits

Before exploring TensorFlow Quantum (TFQ), let's look at some Cirq basics. Cirq is a Python library for quantum computing from Google. You use it to define circuits, including static and parameterized gates.

Cirq uses SymPy symbols to represent free parameters.

```
a, b = sympy.symbols('a b')
```

The following code creates a two-qubit circuit using your parameters:

```
# Create two qubits
q0, q1 = cirq.GridQubit.rect(1, 2)

# Create a circuit on these qubits using the parameters you created above.
circuit = cirq.Circuit(
    cirq.rx(a).on(q0),
    cirq.ry(b).on(q1), cirq.CNOT(control=q0, target=q1))

SVGCircuit(circuit)
```

    findfont: Font family ['Arial'] not found. Falling back to DejaVu Sans.

To evaluate circuits, you can use the `cirq.Simulator` interface. You replace free parameters in a circuit with specific numbers by passing in a `cirq.ParamResolver` object. The following code calculates the raw state vector output of your parameterized circuit:

```
# Calculate a state vector with a=0.5 and b=-0.5.
resolver = cirq.ParamResolver({a: 0.5, b: -0.5})
output_state_vector = cirq.Simulator().simulate(circuit, resolver).final_state_vector
output_state_vector
```

    array([ 0.9387913 +0.j        , -0.23971277+0.j        ,
            0.        +0.06120872j,  0.        -0.23971277j], dtype=complex64)

State vectors are not directly accessible outside of simulation (notice the complex numbers in the output above). To be physically realistic, you must specify a measurement, which converts a state vector into a real number that classical computers can understand. Cirq specifies measurements using combinations of the Pauli operators $\hat{X}$, $\hat{Y}$, and $\hat{Z}$. As illustration, the following code measures $\hat{Z}_0$ and $\frac{1}{2}\hat{Z}_0 + \hat{X}_1$ on the state vector you just simulated:

```
z0 = cirq.Z(q0)

qubit_map={q0: 0, q1: 1}

z0.expectation_from_state_vector(output_state_vector, qubit_map).real
```

    0.8775825500488281

```
z0x1 = 0.5 * z0 + cirq.X(q1)

z0x1.expectation_from_state_vector(output_state_vector, qubit_map).real
```

    -0.04063427448272705

### 1.2 Quantum circuits as tensors

TensorFlow Quantum (TFQ) provides `tfq.convert_to_tensor`, a function that converts Cirq objects into tensors. This allows you to send Cirq objects to our quantum layers and quantum ops. The function can be called on lists or arrays of Cirq Circuits and Cirq Paulis:

```
# Rank 1 tensor containing 1 circuit.
circuit_tensor = tfq.convert_to_tensor([circuit])

print(circuit_tensor.shape)
print(circuit_tensor.dtype)
```

    (1,)
    <dtype: 'string'>

This encodes the Cirq objects as `tf.string` tensors that `tfq` operations decode as needed.

```
# Rank 1 tensor containing 2 Pauli operators.
pauli_tensor = tfq.convert_to_tensor([z0, z0x1])
pauli_tensor.shape
```

    TensorShape([2])

### 1.3 Batching circuit simulation

TFQ provides methods for computing expectation values, samples, and state vectors. For now, let's focus on *expectation values*.

The highest-level interface for calculating expectation values is the `tfq.layers.Expectation` layer, which is a `tf.keras.Layer`. In its simplest form, this layer is equivalent to simulating a parameterized circuit over many `cirq.ParamResolvers`; however, TFQ allows batching following TensorFlow semantics, and circuits are simulated using efficient C++ code.

Create a batch of values to substitute for our `a` and `b` parameters:

```
batch_vals = np.array(np.random.uniform(0, 2 * np.pi, (5, 2)), dtype=np.float32)
```

Batching circuit execution over parameter values in Cirq requires a loop:

```
cirq_results = []
cirq_simulator = cirq.Simulator()

for vals in batch_vals:
    resolver = cirq.ParamResolver({a: vals[0], b: vals[1]})
    final_state_vector = cirq_simulator.simulate(circuit, resolver).final_state_vector
    cirq_results.append(
        [z0.expectation_from_state_vector(final_state_vector, {
            q0: 0,
            q1: 1
        }).real])

print('cirq batch results: \n {}'.format(np.array(cirq_results)))
```

    cirq batch results:
     [[ 0.99998105]
     [-0.99246973]
     [-0.48508084]
     [ 0.95581585]
     [ 0.32338342]]

The same operation is simplified in TFQ:

```
tfq.layers.Expectation()(circuit,
                         symbol_names=[a, b],
                         symbol_values=batch_vals,
                         operators=z0)
```

    <tf.Tensor: shape=(5, 1), dtype=float32, numpy=
    array([[ 0.9999812 ],
           [-0.9924699 ],
           [-0.48508152],
           [ 0.95581585],
           [ 0.32338524]], dtype=float32)>

## 2. Hybrid quantum-classical optimization

Now that you've seen the basics, let's use TensorFlow Quantum to construct a *hybrid quantum-classical neural net*. You will train a classical neural net to control a single qubit. The control will be optimized to correctly prepare the qubit in the `0` or `1` state, overcoming a simulated systematic calibration error. This figure shows the architecture:

Even without a neural network this is a straightforward problem to solve, but the theme is similar to the real quantum control problems you might solve using TFQ. It demonstrates an end-to-end example of a quantum-classical computation using the `tfq.layers.ControlledPQC` (Parametrized Quantum Circuit) layer inside of a `tf.keras.Model`.

For the implementation of this tutorial, this is architecture is split into 3 parts:

- The *input circuit* or *datapoint circuit*: The first three $R$ gates.
- The *controlled circuit*: The other three $R$ gates.
- The *controller*: The classical neural-network setting the parameters of the controlled circuit.

### 2.1 The controlled circuit definition

Define a learnable single bit rotation, as indicated in the figure above. This will correspond to our controlled circuit.

```
# Parameters that the classical NN will feed values into.
control_params = sympy.symbols('theta_1 theta_2 theta_3')

# Create the parameterized circuit.
qubit = cirq.GridQubit(0, 0)
model_circuit = cirq.Circuit(
    cirq.rz(control_params[0])(qubit),
    cirq.ry(control_params[1])(qubit),
    cirq.rx(control_params[2])(qubit))

SVGCircuit(model_circuit)
```

### 2.2 The controller

Now define controller network:

```
# The classical neural network layers.
controller = tf.keras.Sequential([
    tf.keras.layers.Dense(10, activation='elu'),
    tf.keras.layers.Dense(3)
])
```

Given a batch of commands, the controller outputs a batch of control signals for the controlled circuit.

The controller is randomly initialized so these outputs are not useful, yet.

```
controller(tf.constant([[0.0],[1.0]])).numpy()
```

    array([[ 0.        ,  0.        ,  0.        ],
           [-0.93967277, -0.10683127,  0.49653825]], dtype=float32)

### 2.3 Connect the controller to the circuit

Use `tfq` to connect the controller to the controlled circuit, as a single `keras.Model`.

See the [Keras Functional API guide](https://www.tensorflow.org/guide/keras/functional) for more about this style of model definition.

First define the inputs to the model:

```
# This input is the simulated miscalibration that the model will learn to correct.
circuits_input = tf.keras.Input(shape=(),
                                # The circuit-tensor has dtype `tf.string`
                                dtype=tf.string,
                                name='circuits_input')

# Commands will be either `0` or `1`, specifying the state to set the qubit to.
commands_input = tf.keras.Input(shape=(1,),
                                dtype=tf.dtypes.float32,
                                name='commands_input')

```

Next apply operations to those inputs, to define the computation.

```
dense_2 = controller(commands_input)

# TFQ layer for classically controlled circuits.
expectation_layer = tfq.layers.ControlledPQC(model_circuit,
                                             # Observe Z
                                             operators = cirq.Z(qubit))
expectation = expectation_layer([circuits_input, dense_2])
```

Now package this computation as a `tf.keras.Model`:

```
# The full Keras model is built from our layers.
model = tf.keras.Model(inputs=[circuits_input, commands_input],
                       outputs=expectation)
```

The network architecture is indicated by the plot of the model below.
Compare this model plot to the architecture diagram to verify correctness.

Note: May require a system install of the `graphviz` package.

```
tf.keras.utils.plot_model(model, show_shapes=True, dpi=70)
```

This model takes two inputs: The commands for the controller, and the input-circuit whose output the controller is attempting to correct.

### 2.4 The dataset

The model attempts to output the correct correct measurement value of $\hat{Z}$ for each command. The commands and correct values are defined below.

```
# The command input values to the classical NN.
commands = np.array([[0], [1]], dtype=np.float32)

# The desired Z expectation value at output of quantum circuit.
expected_outputs = np.array([[1], [-1]], dtype=np.float32)
```

This is not the entire training dataset for this task.
Each datapoint in the dataset also needs an input circuit.

### 2.4 Input circuit definition

The input-circuit below defines the random miscalibration the model will learn to correct.

```
random_rotations = np.random.uniform(0, 2 * np.pi, 3)
noisy_preparation = cirq.Circuit(
  cirq.rx(random_rotations[0])(qubit),
  cirq.ry(random_rotations[1])(qubit),
  cirq.rz(random_rotations[2])(qubit)
)
datapoint_circuits = tfq.convert_to_tensor([
  noisy_preparation
] * 2)  # Make two copied of this circuit
```

There are two copies of the circuit, one for each datapoint.

```
datapoint_circuits.shape
```

    TensorShape([2])

### 2.5 Training

With the inputs defined you can test-run the `tfq` model.

```
model([datapoint_circuits, commands]).numpy()
```

    array([[0.9181084 ],
           [0.79686123]], dtype=float32)

Now run a standard training process to adjust these values towards the `expected_outputs`.

```
optimizer = tf.keras.optimizers.Adam(learning_rate=0.05)
loss = tf.keras.losses.MeanSquaredError()
model.compile(optimizer=optimizer, loss=loss)
history = model.fit(x=[datapoint_circuits, commands],
                    y=expected_outputs,
                    epochs=30,
                    verbose=0)
```

```
plt.plot(history.history['loss'])
plt.title("Learning to Control a Qubit")
plt.xlabel("Iterations")
plt.ylabel("Error in Control")
plt.show()
```

From this plot you can see that the neural network has learned to overcome the systematic miscalibration.

### 2.6 Verify outputs
Now use the trained model, to correct the qubit calibration errors. With Cirq:

```
def check_error(command_values, desired_values):
  """Based on the value in `command_value` see how well you could prepare
  the full circuit to have `desired_value` when taking expectation w.r.t. Z."""
  params_to_prepare_output = controller(command_values).numpy()
  full_circuit = noisy_preparation + model_circuit

  # Test how well you can prepare a state to get expectation the expectation
  # value in `desired_values`
  for index in [0, 1]:
    state = cirq_simulator.simulate(
        full_circuit,
        {s:v for (s,v) in zip(control_params, params_to_prepare_output[index])}
    ).final_state_vector
    expt = cirq.Z(qubit).expectation_from_state_vector(state, {qubit: 0}).real
    print(f'For a desired output (expectation) of {desired_values[index]} with'
          f' noisy preparation, the controller\nnetwork found the following '
          f'values for theta: {params_to_prepare_output[index]}\nWhich gives an'
          f' actual expectation of: {expt}\n')

check_error(commands, expected_outputs)
```

    For a desired output (expectation) of [1.] with noisy preparation, the controller
    network found the following values for theta: [0.15362751 0.02613623 0.22343476]
    Which gives an actual expectation of: 0.955525279045105

    For a desired output (expectation) of [-1.] with noisy preparation, the controller
    network found the following values for theta: [-2.7526338  0.3802155  3.12327  ]
    Which gives an actual expectation of: -0.9250240325927734

The value of the loss function during training provides a rough idea of how well the model is learning. The lower the loss, the closer the expectation values in the above cell is to `desired_values`. If you aren't as concerned with the parameter values, you can always check the outputs from above using `tfq`:

```
model([datapoint_circuits, commands])
```

    <tf.Tensor: shape=(2, 1), dtype=float32, numpy=
    array([[ 0.9555249 ],
           [-0.92502314]], dtype=float32)>

## 3 Learning to prepare eigenstates of different operators

The choice of the $\pm \hat{Z}$ eigenstates corresponding to 1 and 0 was arbitrary. You could have just as easily wanted 1 to correspond to the $+ \hat{Z}$ eigenstate and 0 to correspond to the $-\hat{X}$ eigenstate. One way to accomplish this is by specifying a different measurement operator for each command, as indicated in the figure below:

This requires use of tfq.layers.Expectation. Now your input has grown to include three objects: circuit, command, and operator. The output is still the expectation value.

### 3.1 New model definition

Lets take a look at the model to accomplish this task:

```
# Define inputs.
commands_input = tf.keras.layers.Input(shape=(1),
                                       dtype=tf.dtypes.float32,
                                       name='commands_input')
circuits_input = tf.keras.Input(shape=(),
                                # The circuit-tensor has dtype `tf.string`
                                dtype=tf.dtypes.string,
                                name='circuits_input')
operators_input = tf.keras.Input(shape=(1,),
                                 dtype=tf.dtypes.string,
                                 name='operators_input')
```

Here is the controller network:

```
# Define classical NN.
controller = tf.keras.Sequential([
    tf.keras.layers.Dense(10, activation='elu'),
    tf.keras.layers.Dense(3)
])
```

Combine the circuit and the controller into a single `keras.Model` using `tfq`:

```
dense_2 = controller(commands_input)

# Since you aren't using a PQC or ControlledPQC you must append
# your model circuit onto the datapoint circuit tensor manually.
full_circuit = tfq.layers.AddCircuit()(circuits_input, append=model_circuit)
expectation_output = tfq.layers.Expectation()(full_circuit,
                                              symbol_names=control_params,
                                              symbol_values=dense_2,
                                              operators=operators_input)

# Contruct your Keras model.
two_axis_control_model = tf.keras.Model(
    inputs=[circuits_input, commands_input, operators_input],
    outputs=[expectation_output])
```

### 3.2 The dataset

Now you will also include the operators you wish to measure for each datapoint you supply for `model_circuit`:

```
# The operators to measure, for each command.
operator_data = tfq.convert_to_tensor([[cirq.X(qubit)], [cirq.Z(qubit)]])

# The command input values to the classical NN.
commands = np.array([[0], [1]], dtype=np.float32)

# The desired expectation value at output of quantum circuit.
expected_outputs = np.array([[1], [-1]], dtype=np.float32)
```

### 3.3 Training

Now that you have your new inputs and outputs you can train once again using keras.

```
optimizer = tf.keras.optimizers.Adam(learning_rate=0.05)
loss = tf.keras.losses.MeanSquaredError()

two_axis_control_model.compile(optimizer=optimizer, loss=loss)

history = two_axis_control_model.fit(
    x=[datapoint_circuits, commands, operator_data],
    y=expected_outputs,
    epochs=30,
    verbose=1)
```

    Epoch 1/30
    1/1 [==============================] - 0s 2ms/step - loss: 2.0590
    Epoch 2/30
    1/1 [==============================] - 0s 1ms/step - loss: 1.1033
    Epoch 3/30
    1/1 [==============================] - 0s 1ms/step - loss: 0.4088
    Epoch 4/30
    1/1 [==============================] - 0s 1ms/step - loss: 0.2251
    Epoch 5/30
    1/1 [==============================] - 0s 2ms/step - loss: 0.1944
    Epoch 6/30
    1/1 [==============================] - 0s 2ms/step - loss: 0.1292
    Epoch 7/30
    1/1 [==============================] - 0s 2ms/step - loss: 0.0619
    Epoch 8/30
    1/1 [==============================] - 0s 7ms/step - loss: 0.0331
    Epoch 9/30
    1/1 [==============================] - 0s 4ms/step - loss: 0.0311
    Epoch 10/30
    1/1 [==============================] - 0s 2ms/step - loss: 0.0383
    Epoch 11/30
    1/1 [==============================] - 0s 2ms/step - loss: 0.0456
    Epoch 12/30
    1/1 [==============================] - 0s 2ms/step - loss: 0.0484
    Epoch 13/30
    1/1 [==============================] - 0s 7ms/step - loss: 0.0449
    Epoch 14/30
    1/1 [==============================] - 0s 2ms/step - loss: 0.0365
    Epoch 15/30
    1/1 [==============================] - 0s 2ms/step - loss: 0.0262
    Epoch 16/30
    1/1 [==============================] - 0s 2ms/step - loss: 0.0170
    Epoch 17/30
    1/1 [==============================] - 0s 1ms/step - loss: 0.0106
    Epoch 18/30
    1/1 [==============================] - 0s 2ms/step - loss: 0.0072
    Epoch 19/30
    1/1 [==============================] - 0s 2ms/step - loss: 0.0061
    Epoch 20/30
    1/1 [==============================] - 0s 2ms/step - loss: 0.0061
    Epoch 21/30
    1/1 [==============================] - 0s 2ms/step - loss: 0.0063
    Epoch 22/30
    1/1 [==============================] - 0s 2ms/step - loss: 0.0062
    Epoch 23/30
    1/1 [==============================] - 0s 2ms/step - loss: 0.0054
    Epoch 24/30
    1/1 [==============================] - 0s 2ms/step - loss: 0.0042
    Epoch 25/30
    1/1 [==============================] - 0s 4ms/step - loss: 0.0030
    Epoch 26/30
    1/1 [==============================] - 0s 2ms/step - loss: 0.0020
    Epoch 27/30
    1/1 [==============================] - 0s 3ms/step - loss: 0.0014
    Epoch 28/30
    1/1 [==============================] - 0s 3ms/step - loss: 0.0011
    Epoch 29/30
    1/1 [==============================] - 0s 1ms/step - loss: 9.7383e-04
    Epoch 30/30
    1/1 [==============================] - 0s 2ms/step - loss: 8.9875e-04

```
plt.plot(history.history['loss'])
plt.title("Learning to Control a Qubit")
plt.xlabel("Iterations")
plt.ylabel("Error in Control")
plt.show()
```

The loss function has dropped to zero.

The `controller` is available as a stand-alone model. Call the controller, and check its response to each command signal. It would take some work to correctly compare these outputs to the contents of `random_rotations`.

```
controller.predict(np.array([0,1]))
```

    array([[-0.93180346,  1.4380494 ,  0.9859159 ],
           [-5.0782895 ,  0.12699541,  3.3510125 ]], dtype=float32)

Success: See if you can adapt the `check_error` function from your first model to work with this new model architecture.