# **Monitoring and Optimizing Quantum Circuits**

```python
import numpy as np

# Importing standard Qiskit libraries
from qiskit import QuantumCircuit, transpile, Aer, IBMQ, execute
from qiskit.tools.jupyter import *
from qiskit.visualization import *
from ibm_quantum_widgets import *
from qiskit.providers.aer import QasmSimulator

# Loading your IBM Quantum account(s)
provider = IBMQ.load_account()
```

## **Monitoring and Tracking Jobs**

```python
# Import the Qiskit Jupyter tools
from qiskit.tools import jupyter
```

```python
# Initialize the job tracker to automatically track all jobs
%qiskit_job_watcher
```

    Accordion(children=(VBox(layout=Layout(max_width='710px', min_width='710px')),), layout=Layout(max_height='500…

    <IPython.core.display.Javascript object>

```python
# Let's run a simple circuit on the least busy quantum device
# and check the job watcher widget.
from qiskit.providers.ibmq import least_busy

backend = least_busy(provider.backends(filters=lambda x: x.configuration().n_qubits >= (2) and
                                       not x.configuration().simulator
                                       and x.status().operational==True))

#Create a simple circuit
qc = QuantumCircuit(1)
qc.h(0)
qc.measure_all()
#Execute the circuit on the backend
job = execute(qc, backend)
```

```python
#Disable the job watcher
%qiskit_disable_job_watcher
```

```python
#Display the list of all available backends and provide
#a brief overview of each
%qiskit_backend_overview
```

    VBox(children=(HTML(value="<h2 style ='color:#ffffff; background-color:#000000;padding-top: 1%; padding-bottom…

## **Transpiling a Circuit**

```python
# Import the transpiler passes object
from qiskit.transpiler import passes
# List out all the passes available
print(dir(passes))
```

    ['ALAPSchedule', 'ASAPSchedule', 'AlignMeasures', 'ApplyLayout', 'BIPMapping', 'BarrierBeforeFinalMeasurements', 'BasicSwap', 'BasisTranslator', 'CSPLayout', 'CXCancellation', 'CXDirection', 'CheckCXDirection', 'CheckGateDirection', 'CheckMap', 'Collect1qRuns', 'Collect2qBlocks', 'CollectMultiQBlocks', 'CommutationAnalysis', 'CommutativeCancellation', 'ConsolidateBlocks', 'ContainsInstruction', 'CountOps', 'CountOpsLongestPath', 'CrosstalkAdaptiveSchedule', 'DAGFixedPoint', 'DAGLongestPath', 'Decompose', 'DenseLayout', 'Depth', 'DynamicalDecoupling', 'EchoRZXWeylDecomposition', 'EnlargeWithAncilla', 'Error', 'FixedPoint', 'FullAncillaAllocation', 'GateDirection', 'GatesInBasis', 'HoareOptimizer', 'InverseCancellation', 'Layout2qDistance', 'LayoutTransformation', 'LookaheadSwap', 'MergeAdjacentBarriers', 'NoiseAdaptiveLayout', 'NumTensorFactors', 'Optimize1qGates', 'Optimize1qGatesDecomposition', 'Optimize1qGatesSimpleCommutation', 'OptimizeSwapBeforeMeasure', 'PulseGates', 'RZXCalibrationBuilder', 'RZXCalibrationBuilderNoEcho', 'RemoveBarriers', 'RemoveDiagonalGatesBeforeMeasure', 'RemoveFinalMeasurements', 'RemoveResetInZeroState', 'ResourceEstimation', 'SabreLayout', 'SabreSwap', 'SetLayout', 'Size', 'StochasticSwap', 'TemplateOptimization', 'TimeUnitConversion', 'TrivialLayout', 'UnitarySynthesis', 'Unroll3qOrMore', 'UnrollCustomDefinitions', 'Unroller', 'VF2Layout', 'ValidatePulseGates', 'Width', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'analysis', 'basis', 'calibration', 'layout', 'optimization', 'routing', 'scheduling', 'synthesis', 'unitary_synthesis_plugin_names', 'utils']

```python
#Basic Toffoli gate,
qc = QuantumCircuit(3)
qc.ccx(0,1,2)
qc.draw()
```

```python
qc_decomposed = qc.decompose()
qc_decomposed.draw()
```

```python
#Basic circuit with a single and multi-qubit gates
qc = QuantumCircuit(4)
qc.h(0)
qc.cx(0,1)
qc.cx(0,2)
qc.cx(0,3)
qc.draw()
```

```python
#Print the depth of both inital and decomposed circuit
print('Initial circuit depth: ', qc.depth())
print('Decomposed circuit depth: ', qc_decomposed.depth())
#Get the number of operators in initial circuit
print('Initial circuit operation count: ', qc.count_ops())
#Get the number of operators in decomposed circuit
print('Decomposed circuit operation count: ', qc_decomposed.count_ops())
```

    Initial circuit depth:  4
    Decomposed circuit depth:  11
    Initial circuit operation count:  OrderedDict([('cx', 3), ('h', 1)])
    Decomposed circuit operation count:  OrderedDict([('cx', 6), ('t', 4), ('tdg', 3), ('h', 2)])

## **Configuration and Optimization**

```python
# Get the backend device: ibmq_santiago
backend_santiago = provider.get_backend('ibmq_santiago')
# Launch backend viewer of ibmq_santiago
backend_santiago
```

    VBox(children=(HTML(value="<h1 style='color:#ffffff;background-color:#000000;padding-top: 1%;padding-bottom: 1…

    <IBMQBackend('ibmq_santiago') from IBMQ(hub='ibm-q', group='open', project='main')>

```python
# Get the backend device: ibmq_lima
backend_lima = provider.get_backend('ibmq_lima')
# Launch backend viewer of ibmq_lima
backend_lima
```

    VBox(children=(HTML(value="<h1 style='color:#ffffff;background-color:#000000;padding-top: 1%;padding-bottom: 1…

    <IBMQBackend('ibmq_lima') from IBMQ(hub='ibm-q', group='open', project='main')>

```python
# Visualize the coupling directional map between the qubits
plot_gate_map(backend_santiago, plot_directed=True)
```

```python
# Visualize the coupling directional map between the qubits
plot_gate_map(backend_lima, plot_directed=True)
```

```python
# Quantum circuit with a single and multi-qubit gates
qc = QuantumCircuit(4)
qc.h(0)
qc.cx(0,1)
qc.cx(0,2)
qc.cx(0,3)
qc.draw()
```

```python
# Transpile the circuit with an optimization level = 0
qc_santiago_0 = transpile(qc, backend_santiago,
seed_transpiler=10258, optimization_level=0)
# Print out the depth of the circuit
print('Depth:', qc_santiago_0.depth())
# Plot the resulting layout of the quantum circuit after Layout
plot_circuit_layout(qc_santiago_0, backend_santiago)
```

```python
# Draw the transpiled circuit pertaining to Santiago
qc_santiago_0.draw()
```

```python
# View the transpiled circuit with an optimization level = 0
qc_lima_0 = transpile(qc, backend_lima, seed_transpiler=10258, optimization_level=0)
print('Depth:', qc_lima_0.depth())
plot_circuit_layout(qc_lima_0, backend_lima)
```

```python
# Draw the transpiled circuit pertaining to Lima
qc_lima_0.draw()
```

```python
# Transpile the circuit with the optimization level = 3
qc_transpiled_santiago = transpile(qc, backend_santiago, optimization_level=3)
# Print the depth of the transpiled circuit
print('Depth:', qc_transpiled_santiago.depth())
# Print the number of operations of the transpiled circuit
print('Ops count: ', qc_transpiled_santiago.count_ops())
# Plot the layout mapping of the transpiled circuit
plot_circuit_layout(qc_transpiled_santiago, backend_santiago)
```

```python
# Redraw the transpiled circuit at new level
qc_transpiled_santiago.draw()
```

```python
# Transpile the quantum circuit with the optimization level = 3
qc_transpiled_lima = transpile(qc, backend_lima, optimization_level=3)
# Get the depth and operation count of the transpiled circuit.
print('Depth:', qc_transpiled_lima.depth())
print('Ops count: ', qc_transpiled_lima.count_ops())
# Print the circuit layout
plot_circuit_layout(qc_transpiled_lima, backend_lima)
```

```python
# View the ibmq_quito backend device configuration and properties
backend = provider.get_backend('ibmq_quito')
backend
```

    VBox(children=(HTML(value="<h1 style='color:#ffffff;background-color:#000000;padding-top: 1%;padding-bottom: 1…

    <IBMQBackend('ibmq_quito') from IBMQ(hub='ibm-q', group='open', project='main')>

```python
# View the backend coupling map, displayed as CNOTs (Control-Target)
backend = provider.get_backend('ibmq_quito')
# Extract the coupling map from the backend
ibmqquito_coupling_map = backend.configuration().coupling_map
# List out the extracted coupling map
ibmqquito_coupling_map
```

    [[0, 1], [1, 0], [1, 2], [1, 3], [2, 1], [3, 1], [3, 4], [4, 3]]

```python
# Transpile a custom circuit using only the coupling map.
# Set the backend to ‘None’ so it will force using the coupling map provided.
qc_custom = transpile(qc, backend=None,
coupling_map=ibmqquito_coupling_map)
# Draw the resulting custom topology circuit.
qc_custom.draw()
```

```python
# Create our own coupling map (custom topology)
custom_linear_topology = [[0,1],[1,2],[2,3],[3,4]]
# Set the coupling map to our custom linear topology
qc_custom = transpile(qc, backend=None, coupling_map=custom_linear_topology)
# Draw the resulting circuit.
qc_custom.draw()
```

```python
# Import the PassManager and a few Passes
from qiskit.transpiler import PassManager, CouplingMap
from qiskit.transpiler.passes import TrivialLayout, BasicSwap
# Create a TrivialLayout based on the ibmqx2 coupling map
trivial = TrivialLayout(CouplingMap(ibmqquito_coupling_map))

pm = PassManager()
# Append the TrivialLayout to the PassManager
pm.append(trivial)
# Run the PassManager and draw the resulting circuit
tv_qc = pm.run(qc)
tv_qc.draw()
```

```python
# Create a BasicSwap based on the ibmq_quito coupling map we used earlier
basic_swap = BasicSwap(CouplingMap(ibmqquito_coupling_map))
#Add the BasicSwap to the PassManager
pm = PassManager(basic_swap)
# Run the PassManager and draw the results
new_qc = pm.run(qc)
new_qc.draw()
```

```python
# Sample quantum circuit
qc = QuantumCircuit(4)
qc.h(0)
qc.cx(0,1)
qc.barrier()
qc.cx(0,2)
qc.cx(0,3)
qc.barrier()
qc.cz(3,0)
qc.h(0)
qc.measure_all()
# Draw the circuit using the default renderer
qc.draw()
```

```python
qc.draw('latex')
```

```python

```

QC with IBM QE/Monitoring and Optimizing Quantum Circuits.ipynb


```python
!pip install econml
```

    Collecting econml
      Downloading econml-0.12.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.1 MB)
         |████████████████████████████████| 3.1 MB 241 kB/s
    Collecting shap<0.40.0,>=0.38.1
      Downloading shap-0.39.0.tar.gz (356 kB)
         |████████████████████████████████| 356 kB 49.0 MB/s
      Preparing metadata (setup.py) ... - \ done
    Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from econml) (1.19.5)
    Requirement already satisfied: statsmodels>=0.10 in /opt/conda/lib/python3.7/site-packages (from econml) (0.12.2)
    Requirement already satisfied: scikit-learn>0.22.0 in /opt/conda/lib/python3.7/site-packages (from econml) (0.23.2)
    Requirement already satisfied: lightgbm in /opt/conda/lib/python3.7/site-packages (from econml) (3.3.1)
    Collecting sparse
      Downloading sparse-0.13.0-py2.py3-none-any.whl (77 kB)
         |████████████████████████████████| 77 kB 4.5 MB/s
    Requirement already satisfied: scipy>1.4.0 in /opt/conda/lib/python3.7/site-packages (from econml) (1.7.2)
    Requirement already satisfied: pandas in /opt/conda/lib/python3.7/site-packages (from econml) (1.3.4)
    Collecting dowhy
      Downloading dowhy-0.7-py3-none-any.whl (152 kB)
         |████████████████████████████████| 152 kB 47.9 MB/s
    Requirement already satisfied: numba!=0.42.1 in /opt/conda/lib/python3.7/site-packages (from econml) (0.54.1)
    Requirement already satisfied: joblib>=0.13.0 in /opt/conda/lib/python3.7/site-packages (from econml) (1.1.0)
    Requirement already satisfied: llvmlite<0.38,>=0.37.0rc1 in /opt/conda/lib/python3.7/site-packages (from numba!=0.42.1->econml) (0.37.0)
    Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from numba!=0.42.1->econml) (59.1.1)
    Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from scikit-learn>0.22.0->econml) (3.0.0)
    Requirement already satisfied: tqdm>4.25.0 in /opt/conda/lib/python3.7/site-packages (from shap<0.40.0,>=0.38.1->econml) (4.62.3)
    Requirement already satisfied: slicer==0.0.7 in /opt/conda/lib/python3.7/site-packages (from shap<0.40.0,>=0.38.1->econml) (0.0.7)
    Requirement already satisfied: cloudpickle in /opt/conda/lib/python3.7/site-packages (from shap<0.40.0,>=0.38.1->econml) (2.0.0)
    Requirement already satisfied: patsy>=0.5 in /opt/conda/lib/python3.7/site-packages (from statsmodels>=0.10->econml) (0.5.2)
    Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from pandas->econml) (2.8.0)
    Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.7/site-packages (from pandas->econml) (2021.3)
    Requirement already satisfied: pydot>=1.4 in /opt/conda/lib/python3.7/site-packages (from dowhy->econml) (1.4.2)
    Requirement already satisfied: networkx>=2.0 in /opt/conda/lib/python3.7/site-packages (from dowhy->econml) (2.6.3)
    Requirement already satisfied: sympy>=1.4 in /opt/conda/lib/python3.7/site-packages (from dowhy->econml) (1.9)
    Requirement already satisfied: wheel in /opt/conda/lib/python3.7/site-packages (from lightgbm->econml) (0.37.0)
    Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from patsy>=0.5->statsmodels>=0.10->econml) (1.16.0)
    Requirement already satisfied: pyparsing>=2.1.4 in /opt/conda/lib/python3.7/site-packages (from pydot>=1.4->dowhy->econml) (3.0.6)
    Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.7/site-packages (from sympy>=1.4->dowhy->econml) (1.2.1)
    Building wheels for collected packages: shap
      Building wheel for shap (setup.py) ... - \ | / - \ | done
      Created wheel for shap: filename=shap-0.39.0-cp37-cp37m-linux_x86_64.whl size=544687 sha256=445ef5154401c3539d7f5e36cd9a367244536c5f3536b77d40b93faca179bc5a
      Stored in directory: /root/.cache/pip/wheels/ca/25/8f/6ae5df62c32651cd719e972e738a8aaa4a87414c4d2b14c9c0
    Successfully built shap
    Installing collected packages: sparse, shap, dowhy, econml
      Attempting uninstall: shap
        Found existing installation: shap 0.40.0
        Uninstalling shap-0.40.0:
          Successfully uninstalled shap-0.40.0
    Successfully installed dowhy-0.7 econml-0.12.0 shap-0.39.0 sparse-0.13.0
    WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

```python
# Some imports to get us started
import warnings
warnings.simplefilter('ignore')

# Utilities
import os
import urllib.request
import numpy as np
import pandas as pd
from networkx.drawing.nx_pydot import to_pydot
from IPython.display import Image, display

# Generic ML imports
from sklearn.preprocessing import PolynomialFeatures
from sklearn.ensemble import GradientBoostingRegressor

# EconML imports
from econml.dml import LinearDML, CausalForestDML
from econml.cate_interpreter import SingleTreeCateInterpreter, SingleTreePolicyInterpreter

import matplotlib.pyplot as plt

%matplotlib inline
```

```python
# Import the sample pricing data
file_url = "https://msalicedatapublic.blob.core.windows.net/datasets/Pricing/pricing_sample.csv"
train_data = pd.read_csv(file_url)
```

```python
train_data.head()
```

      account_age
      age
      avg_hours
      days_visited
      friends_count
      has_membership
      is_US
      songs_purchased
      income
      price
      demand

      0
      3
      53
      1.834234
      2
      8
      1
      1
      4.903237
      0.960863
      1.0
      3.917117

      1
      5
      54
      7.171411
      7
      9
      0
      1
      3.330161
      0.732487
      1.0
      11.585706

      2
      3
      33
      5.351920
      6
      9
      0
      1
      3.036203
      1.130937
      1.0
      24.675960

      3
      2
      34
      6.723551
      0
      8
      0
      1
      7.911926
      0.929197
      1.0
      6.361776

      4
      4
      30
      2.448247
      5
      8
      1
      0
      7.148967
      0.533527
      0.8
      12.624123

```python
#estimator inputs
train_data["log_demand"] = np.log(train_data["demand"])
train_data["log_price"] = np.log(train_data["price"])

Y = train_data["log_demand"].values
T = train_data["log_price"].values
X = train_data[["income"]].values  # features
confounder_names = ["account_age", "age", "avg_hours", "days_visited", "friends_count", "has_membership", "is_US", "songs_purchased"]
W = train_data[confounder_names].values
```

```python
# Get test data
X_test = np.linspace(0, 5, 100).reshape(-1, 1)
X_test_data = pd.DataFrame(X_test, columns=["income"])
```

## Create Causal Model

```python
# initiate an EconML cate estimator
est = LinearDML(model_y=GradientBoostingRegressor(), model_t=GradientBoostingRegressor(),
              featurizer=PolynomialFeatures(degree=2, include_bias=False))
```

```python
# fit through dowhy
est_dw = est.dowhy.fit(Y, T, X=X, W=W, outcome_names=["log_demand"], treatment_names=["log_price"], feature_names=["income"],
               confounder_names=confounder_names, inference="statsmodels")
```

```python
# Visualize causal graph
try:
    # Try pretty printing the graph. Requires pydot and pygraphviz
    display(
        Image(to_pydot(est_dw._graph._graph).create_png())
    )
except:
    # Fall back on default graph view
    est_dw.view_model()
```

```python
identified_estimand = est_dw.identified_estimand_
print(identified_estimand)
```

    Estimand type: nonparametric-ate

    ### Estimand : 1
    Estimand name: backdoor
    Estimand expression:
         d
    ────────────(Expectation(log_demand|is_US,avg_hours,age,songs_purchased,income
    d[log_price]

    ,friends_count,has_membership,account_age,days_visited))

    Estimand assumption 1, Unconfoundedness: If U→{log_price} and U→log_demand then P(log_demand|log_price,is_US,avg_hours,age,songs_purchased,income,friends_count,has_membership,account_age,days_visited,U) = P(log_demand|log_price,is_US,avg_hours,age,songs_purchased,income,friends_count,has_membership,account_age,days_visited)

    ### Estimand : 2
    Estimand name: iv
    No such variable(s) found!

    ### Estimand : 3
    Estimand name: frontdoor
    No such variable(s) found!

```python
# initiate an EconML cate estimator
est_nonparam = CausalForestDML(model_y=GradientBoostingRegressor(), model_t=GradientBoostingRegressor())
# fit through dowhy
est_nonparam_dw = est_nonparam.dowhy.fit(Y, T, X=X, W=W, outcome_names=["log_demand"], treatment_names=["log_price"],
                                         feature_names=["income"], confounder_names=confounder_names, inference="blb")
```

# Test Estimate Robustness with DoWhy

## Add Random Common Cause

How robust are our estimates to adding another confounder?

```python
res_random = est_nonparam_dw.refute_estimate(method_name="random_common_cause")
print(res_random)
```

    Refute: Add a random common cause
    Estimated effect:-0.9573995973779295
    New effect:-0.95832930881448
    p value:0.37

How robust are our estimates to unobserved confounders

```python
res_unobserved = est_nonparam_dw.refute_estimate(
    method_name="add_unobserved_common_cause",
    confounders_effect_on_treatment="linear",
    confounders_effect_on_outcome="linear",
    effect_strength_on_treatment=0.1,
    effect_strength_on_outcome=0.1,
)
print(res_unobserved)
```

    Refute: Add an Unobserved Common Cause
    Estimated effect:-0.9573995973779295
    New effect:-1.010149368734301

## Replace Treatment with a Random (Placebo) Variable

What happens our estimates if we replace the treatment variable with noise?

```python
res_placebo = est_nonparam_dw.refute_estimate(
    method_name="placebo_treatment_refuter", placebo_type="permute",
    num_simulations=3
)
print(res_placebo)
```

    Refute: Use a Placebo Treatment
    Estimated effect:-0.9573995973779295
    New effect:-0.009779517229192933
    p value:0.15574829607146912

## Remove a Random Subset of the Data

Do we recover similar estimates on subsets of the data?

```python
res_subset = est_nonparam_dw.refute_estimate(
    method_name="data_subset_refuter", subset_fraction=0.8,
    num_simulations=3)
print(res_subset)
```

    Refute: Use a subset of data
    Estimated effect:-0.9573995973779295
    New effect:-0.9517490711399156
    p value:0.31210482684964586

```python

```


```python
# %load /Users/facai/Study/book_notes/preconfig.py
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns
sns.set(color_codes=True)
#sns.set(font='SimHei')
plt.rcParams['axes.grid'] = False

#from IPython.display import SVG
def show_image(filename, figsize=None, res_dir=True):
    if figsize:
        plt.figure(figsize=figsize)

    if res_dir:
        filename = './res/{}'.format(filename)

    plt.imshow(plt.imread(filename))
```

Chapter 8 Optimization for Training Deep Models
=========

optimization: finding the parameters $\theta$ of a neural network that significantly reduce a cost function $J(\theta)$.

### 8.1 How Learning Differs from Pure Optimization

expectation is taken across **the data generating distribution** $p_{data}$ rather than just over the finite training set:

\begin{equation}
    J^*(\theta) = \mathcal{E}_{(x, y) \sim p_{data}} L(f(x; \theta), y)
\end{equation}

#### 8.1.1 Empirical Risk Minimization

Rather than optimizing the risk direcly, we optimize the empirical risk, and hope that the risk decreases significantly as well.

#### 8.1.2 Surrogate Loss Functions and Early Stopping

#### 8.1.3 Batch and Minibatch Algorithms

Small batches can offer a regularing effect.

gradient can handle smaller batch size like 100, while second-order methods typically require much large batch sizes like 10,000.

minibatches must be selected randomly. For very large datasets, it is usually sufficient to shuffle the order of the dataset once and then store it in shuffled fashion.

On the second pass, the estimator becomes biased because it is formed by re-sampling values used before.

### 8.2 Challenges in Neural Network Optimization

#### 8.2.1 Ill-Conditioning

To determin whether ill-conditioning, one can monitor the squared gradient norm $g^T g$ and the $g^T H g$ term. In many cases, the gradient norm does not shrink significantly throughout learning, but the $g^T H g$ term grows by more than order of magnitude.

#### 8.2.2 Local Minima

model identifiability problem: models with latent variables are often not identifiable <= weight space symmetry.

#### 8.2.3 Plateaus, Saddle Points and Other Flat Regions

+ saddle point: local minimum along one cross-section, and local maximum along another another cross-section.
  - in higher dimensional spaces, local minima are rare and saddle points are more common.
  - difficult for newton's method, while easy for gradient descent.

+ maxima
+ wide, flat regions of constant value

#### 8.2.4 Cliffs and Exploding Gradients

```python
show_image("fig8_3.png")
```

can be avoided using the *gradient clipping* heuristic

#### 8.2.5 Long-Term Dependencies

when graph becomes extremely deep => vanishing and exploding gradient problem

Vanishing gradients make it difficult to know which direction the parameters should move to improve to the cost function, while exploding gradients can make learning unstable.

#### 8.2.6 Inexact Gradients

#### 8.2.7 Poor Correspondence between Local and Global Structure

Many existing research directions are aimed at finding good initial points, rather than developing algorithms that use non-local moves.

#### 8.2.8 Theoretical Limits of Optimization

### 8.3 Basic Algorithms

#### 8.3.1 Stochastic Gradient Descent

In practice, it is necessary to gradually decrease the learning rate over time.

In practice, it is common to decay the learning rate linearly until iteration $\tau$:

\begin{equation}
    \epsilon_k = (1 - \alpha) \epsilon_0 + \alpha \epsilon_{\tau}
\end{equation}

+ $\tau$: a few hundred passes through the training set
+ $\epsilon_\tau \approx 1\% \, \epsilon_0$
+ $\epsilon_0$: monitor the first several iterations and use a learning rate that is higher than the best-performing learning rate at this time, but not so high that it causes severe instability.

To study the convergence rate of an optimization algorithm, it is common to measure the *excess error* $J(\theta) - \min_\theta J(\theta)$.
+ SGD is applied to a convex problem: $O(\frac{1}{\sqrt{k}}$
+ in the stronly convex case it is $O(\frac{1}{k})$.

#### 8.3.2 Momentum

The momentum algorithm accumulates an exponentially decaying moving average of past gradients and continues to move in their direction.

\begin{align}
    v &\gets \alpha v - \epsilon \Delta_\theta \left ( \frac1{m} \displaystyle \sum^m_{i = 1} L \left ( f(x^{(i)}; \theta), y^{(i)} \right ) \right ) \\
    \theta &\gets \theta + v
\end{align}

The larger $\alpha$ is relative to $\epsilon$, the more previous gradients affect the current direction.

```python
show_image("fig8_5.png")
```

the size of each step is $\frac{\epsilon \| g \|}{1 - \alpha} \implies$ it is thus helpful to think of the momentum hyperparameter in terms of $\frac1{1 - \alpha}$.

+ Common values of $\alpha$ used in practice include 0.5, 0.9 and 0.99.
+ Typically it begins with a small value and is later raised.
+ It is less important to adapt $\alpha$ over time than to shrink $\epsilon$ over time.

#### 8.3.3 Nesterov Momentum

Nesterov momentum: the gradient is evaluated after the current velocity is applied.

\begin{align}
    v &\gets \alpha v - \epsilon \Delta_\theta \left ( \frac1{m} \displaystyle \sum^m_{i = 1} L \left ( f(x^{(i)}; \theta + \color{blue}{\alpha v}), y^{(i)} \right ) \right ) \\
    \theta &\gets \theta + v
\end{align}

考虑了提前量

### 8.4 Parameter Initialization Strategies

Designing improved initialization strategies is a difficult task because neural network optimization is not yet well understood.

A further difficulty is that some initial points may be benefical from the viewpoint of optimization but detrimental from the viewpoint of generalization.

complete certainty: break symmetry between different units

+ initialize each unit to compute a different function from all of the other units.
+ random initialization of the parameters.
  - Typically, set biases for each unit to heuristically chosen constants, and initilize only the weights randomly.

##### weight

We can think of initializing the parameters $\theta$ to $\theta_0$ as being similar to imposing a Gaussian prior $p(\theta)$ with mean $\theta_0$.
$\implies$ choose $\theta_0$ to be near 0 = more likely that units do not interact with each other than that they do interact.

1. normalized initialization: $W_{i, j} \sim U \left ( - \frac{6}{\sqrt{m + n}}, \frac{6}{\sqrt{m + n}} \right )$
2. initializing to random orthogonal matrices
3. perserve norms
4. sparse initialization

A good rule of thumb for choosing the initial scales is to look at the range or standard deviation of activations or gradients on a single minibatch of data.

##### biase

1. Setting the biases to zero is compatible with most weight initialization schemes.
2. a few situations where we may set some biases to non-zero values:
   + for an output unit, often feneficial to initialize the bias to obtain the right marginal statistics of the output.
   + choose the bias to avoid causing too much saturation at initialization.
     eg: set the bias of ReLU hidden unit to 0.1 rather than 0
   + Sometimes a unit controls whether other units are able to participate in a function => all units have a chance to learn.

##### initialize model parameters using machine learning

eg: to initialize a supervised model with the parameters learned by an unsupervised model trained on the same inputs.

### 8.5 Algorithms with Adaptive Learning Rates

the cost is often highly sensitive to some directions in parameter space and insensitive to others. => adapt the learning rates of model parameters.

#### 8.5.1 AdaGrad

gradient accumulation

$\text{rate} =  \frac1{\sum \text{squared gradients}}$

#### 8.5.2 RMSProp

changing the gradient accumulation into an exponentially weighted moving average.

#### 8.5.3 Adam

adaptive moments: combination of RMSProp and momentum

Currently, the most popular optimization algorithms actively in use include
+ SGD,
+ SGD with momentum,
+ RMSProp,
+ RMSProp with momentum,
+ AdaDelta and Adam.

The choice of which algorithm to use, at this point, seems to depend largely on the user’s familiarity with the algorithm (for ease of hyperparameter tuning).

### 8.6 Approximate Second-Order Methods

#### 8.6.1 Newton's Method

a two-step iterative procedure:

+ update or compute the inverse Hessian
+ update the parameters: $\theta^* = \theta_0 - \mathbf{H}^{-1} \Delta_\theta J(\theta_0)$

#### 8.6.2 Conjugate Gradients

#### 8.6.3 BFGS

L-BFGS

### 8.7 Optimization Strategies and Meta-Algorithms

#### 8.7.1 Batch Normalization

adaptive reparametrization => training very deep models

#### 8.7.2 Coordinate Descent

bad: variables are dependent.

#### 8.7.3 Polyak Averaging

averaging points.

#### 8.7.4 Supervised Pretraining

training sample models on simple tasks => then make the model more complex.

#### 8.7.5 Designing Models to Aid Optimization

In practice, it is more important to choose a model family that is easy to optimize than to use a powerful optimization algorithm.

#### 8.7.6 Continuous Methods and Curriculumn Learning

```python

```