# Deep learning hands-on

## What frameworks do

- Tensor math
- Common network operations/layers
- Gradient of common operations
- Backpropagation
- Optimizer
- GPU implementation of the above
- usally: data loading, serialization (saving/loading) of models
- sometime: distributed computing
- why not: production deployment on clusters/end devices/low power nodes

**TensorFlow** is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries and community resources that lets researchers push the state-of-the-art in ML and developers easily build and deploy ML powered applications.

[ref](https://www.tensorflow.org/tutorials): tf documentation

# Tensors

A Tensor is a multi-dimensional array. Similar to NumPy ndarray objects, Tensor objects have a data type and a shape. Additionally, Tensors can reside in accelerator (like GPU) memory. TensorFlow offers a rich library of operations (tf.add, tf.matmul, tf.linalg.inv etc.) that consume and produce Tensors. These operations automatically convert native Python types.

```python
%pylab inline
import tensorflow as tf
```

    Populating the interactive namespace from numpy and matplotlib

```python
print(tf.add(1, 2))
print(tf.add([1, 2], [3, 4]))
print(tf.square(5))
print(tf.reduce_sum([1, 2, 3]))

# Operator overloading is also supported
print(tf.square(2) + tf.square(3))
```

    tf.Tensor(3, shape=(), dtype=int32)
    tf.Tensor([4 6], shape=(2,), dtype=int32)
    tf.Tensor(25, shape=(), dtype=int32)
    tf.Tensor(6, shape=(), dtype=int32)
    tf.Tensor(13, shape=(), dtype=int32)

Each Tensor has a shape and a datatype

```python
x = tf.matmul([[1]], [[2, 3]])
print(x.shape)
print(x.dtype)
```

    (1, 2)
    <dtype: 'int32'>

The most obvious differences between NumPy arrays and TensorFlow Tensors are:

 -  Tensors can be backed by accelerator memory (like GPU, TPU).
 -  Tensors are immutable.

## GPU acceleration

Many TensorFlow operations can be accelerated by using the GPU for computation. Without any annotations, TensorFlow automatically decides whether to use the GPU or CPU for an operation (and copies the tensor between CPU and GPU memory if necessary). Tensors produced by an operation are typically backed by the memory of the device on which the operation executed. For example:

```python
x = tf.random.uniform([3, 3])

print("Is there any GPUs available?"),
print(tf.config.list_physical_devices('GPU'))

print("Is the Tensor on GPU #0:  "),
print(x.device.endswith('GPU:0'))
```

    Is there any GPUs available?
    []
    Is the Tensor on GPU #0:
    False

## Explicit Device Placement

The term "placement" in TensorFlow refers to how individual operations are assigned (placed on) a device for execution. As mentioned above, when there is no explicit guidance provided, TensorFlow automatically decides which device to execute an operation, and copies Tensors to that device if needed. However, TensorFlow operations can be explicitly placed on specific devices using the tf.device context manager. For example:

```python
def time_matmul(x):
  %timeit tf.matmul(x, x)

# Force execution on CPU
print("On CPU:")
with tf.device("CPU:0"):
  x = tf.random.uniform([1000, 1000])
  assert x.device.endswith("CPU:0")
  time_matmul(x)

# Force execution on GPU #0 if available
if tf.test.is_gpu_available():
  with tf.device("GPU:0"): # Or GPU:1 for the 2nd GPU, GPU:2 for the 3rd etc.
    x = tf.random.uniform([1000, 1000])
    assert x.device.endswith("GPU:0")
    time_matmul(x)
```

    On CPU:
    7.62 ms ± 1.12 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)
    WARNING:tensorflow:From <ipython-input-6-750e9424349b>:12: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use `tf.config.list_physical_devices('GPU')` instead.

# Derivatives of a function - Computing gradients

TensorFlow provides APIs for automatic differentiation - computing the derivative of a function.

To differentiate automatically, TensorFlow needs to remember what operations happen in what order during the forward pass. Then, during the backward pass, TensorFlow traverses this list of operations in reverse order to compute gradients.

## Gradient tapes

TensorFlow provides `tf.GradientTape` API for automatic differentiation; that is, computing the gradient of a computation with respect to its input variables. TensorFlow "records" all operations executed inside the context of a `tf.GradientTape` onto a "tape". TensorFlow then uses that tape and the gradients associated with each recorded operation to compute the gradients of a "recorded" computation using reverse mode differentiation.

```python
# With scalars:
x = tf.constant(3.0)
# y = x ^ 2
with tf.GradientTape() as t:
  t.watch(x)
  y = x * x
# dy = 2x
dy_dx = t.gradient(y, x)
dy_dx.numpy()
```

    6.0

```python

```

```python
# Using matrices:
x = tf.constant([3.0, 3.0])

with tf.GradientTape() as t:
  t.watch(x)
  z = tf.multiply(x, x)

print(z)

# Find derivative of z with respect to the original input tensor x
print(t.gradient(z, x))
```

    tf.Tensor([9. 9.], shape=(2,), dtype=float32)
    tf.Tensor([6. 6.], shape=(2,), dtype=float32)

```python
x = tf.constant([3.0, 3.0])

with tf.GradientTape() as t:
  t.watch(x)
  y = tf.multiply(x, x)
  z = tf.multiply(y, y)

# Use the tape to compute the derivative of z with respect to the
# intermediate value y.
# dz_dx = 2 * y, where y = x ^ 2
print(t.gradient(z, y))
```

    tf.Tensor([18. 18.], shape=(2,), dtype=float32)

## Recording control flow

Because tapes record operations as they are executed, Python control flow (using ifs and whiles for example) is naturally handled:

```python
def f(x, y):
  output = 1.0
  for i in range(y):
    if i > 1 and i < 5:
      output *= x
  return output

def grad(x, y):
  with tf.GradientTape() as t:
    t.watch(x)
    out = f(x, y)
  return t.gradient(out, x)

x = tf.constant(2.0)

print(grad(x, 6).numpy())  # 12.0
print(grad(x, 5).numpy())  # 12.0
print(grad(x, 4).numpy())  # 4.0

```

    12.0
    12.0
    4.0

## Higher-order gradients

Operations inside of the `GradientTape` context manager are recorded for automatic differentiation. If gradients are computed in that context, then the gradient computation is recorded as well. As a result, the exact same API works for higher-order gradients as well. For example:

```python
x = tf.Variable(1.0)  # Create a Tensorflow variable initialized to 1.0

with tf.GradientTape() as t:
  with tf.GradientTape() as t2:
    y = x * x * x

  # Compute the gradient inside the 't' context manager
  # which means the gradient computation is differentiable as well.
  dy_dx = t2.gradient(y, x)
d2y_dx2 = t.gradient(dy_dx, x)

print("dy_dx:", dy_dx.numpy())  # 3.0
print("d2y_dx2:", d2y_dx2.numpy())  # 6.0
```

    dy_dx: 3.0
    d2y_dx2: 6.0

```python
def f(x):
  return tf.square(tf.sin(x))

x = tf.Variable(3.1415/2)
with tf.GradientTape() as t:
    y = f(x)

t.gradient(y,x)
```

    <tf.Tensor: shape=(), dtype=float32, numpy=9.265741e-05>

```python
import numpy as np
import matplotlib.pyplot as plt

x = tf.constant(np.linspace(0,2*np.pi,100))
with tf.GradientTape() as t:
    t.watch(x)
    y = f(x)

g = t.gradient(y,x)
plt.plot(x.numpy(), g.numpy(), x.numpy(), y.numpy())
```

# Canny edge

Canny, John. "A computational approach to edge detection." IEEE Transactions on pattern analysis and machine intelligence 6 (1986): 679-698. (27000+ citations)
Canny:
-  assume noisy step edges
-  construct an edge detector using an optimal linear filter

This is actually a simple neural network...

```python
# package loading, helper functions

import os
import pickle
import numpy as np
import scipy.ndimage as ndi
import sklearn
import tensorflow.keras
import tensorflow.keras.backend as K
import tensorflow as tf
import matplotlib.pyplot as plt

#Source("digraph {X->Y; Y->Z;}")
from graphviz import Source
from scipy.stats import kde
from sklearn import decomposition

def tshow(v, ax=None, **keys):
    if isinstance(v, tf.Tensor):
        v = v.float_val
    if v.ndim == 1:
        v = v.reshape(28, 28)
    if v.ndim == 3 and v.shape[0] == 1:
        v = v[0]
    if v.ndim == 3 and v.shape[2] == 1:
        v = v[:, :, 0]
    if v.ndim == 4:
        v = v[0, 0]
    v = v - amin(v)
    v /= amax(v)
    if ax is not None:
        ax.imshow(v, **keys)
    else:
        imshow(v, **keys)

def showrow(*args, **kw):
    figsize(*kw.get("fs", (15, 5)))
    if "fs" in kw:
        del kw["fs"]
    for i, im in enumerate(args):
        subplot(1, len(args), i+1)
        tshow(im, **kw)

def showgrid(images, rows=4, cols=4, cmap=plt.cm.gray, size=(7, 7)):
    if size is not None:
        figsize(*size)
    for i in range(rows*cols):
        subplot(rows, cols, i+1)
        xticks([])
        yticks([])
        tshow(images[i], cmap=cmap)
```

The Process of Canny edge detection algorithm can be broken down to 5 different steps:

-    Apply Gaussian filter to smooth the image in order to remove the noise
-    Find the intensity gradients of the image
-    Apply non-maximum suppression to get rid of spurious response to edge detection
-    Apply double threshold to determine potential edges
-    Track edge by hysteresis: Finalize the detection of edges by suppressing all the other edges that are weak and not connected to strong edges.

```python
length = 50
pos = np.random.randint(length, size=10000)
signal = np.random.rand(10000,length) / 5
gt = np.zeros((10000,length))

for j,i in enumerate(pos):
    signal[j, i:] = signal[j, i:] + 0.5
    gt[j, i] = 1

figure(figsize=(14,4))
plt.plot(signal[15])
plt.plot(gt[15])
```

```python
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv1D, Flatten
from tensorflow.keras.optimizers import SGD
```

```python

```

```python
# Canny model
model = Sequential()
model.add(Conv1D(1, 33, input_shape=(length, 1), activation="sigmoid", padding='same'))
model.add(Flatten())

model.compile(SGD(lr=.1, momentum=0.9, decay=1e-5), loss='categorical_crossentropy')
model.summary()
```

    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    conv1d (Conv1D)              (None, 50, 1)             34
    _________________________________________________________________
    flatten (Flatten)            (None, 50)                0
    =================================================================
    Total params: 34
    Trainable params: 34
    Non-trainable params: 0
    _________________________________________________________________

```python
K.set_value(model.optimizer.lr, 1e-3)
model.fit(x = np.expand_dims(signal, 3), y=gt, batch_size=100, epochs=100)
```

    /usr/local/lib/python3.7/site-packages/ipykernel_launcher.py:2: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future.

    Epoch 1/100
    100/100 [==============================] - 0s 3ms/step - loss: 3.7604
    Epoch 2/100
    100/100 [==============================] - 0s 3ms/step - loss: 3.5679
    Epoch 3/100
    100/100 [==============================] - 0s 3ms/step - loss: 3.3654
    Epoch 4/100
    100/100 [==============================] - 0s 3ms/step - loss: 3.1902
    Epoch 5/100
    100/100 [==============================] - 0s 3ms/step - loss: 3.0376
    Epoch 6/100
    100/100 [==============================] - 0s 3ms/step - loss: 2.8945
    Epoch 7/100
    100/100 [==============================] - 0s 3ms/step - loss: 2.7549
    Epoch 8/100
    100/100 [==============================] - 0s 3ms/step - loss: 2.6178
    Epoch 9/100
    100/100 [==============================] - 0s 3ms/step - loss: 2.4835
    Epoch 10/100
    100/100 [==============================] - 0s 3ms/step - loss: 2.3532
    Epoch 11/100
    100/100 [==============================] - 0s 3ms/step - loss: 2.2279
    Epoch 12/100
    100/100 [==============================] - 0s 3ms/step - loss: 2.1082
    Epoch 13/100
    100/100 [==============================] - 0s 3ms/step - loss: 1.9946
    Epoch 14/100
    100/100 [==============================] - 0s 3ms/step - loss: 1.8873
    Epoch 15/100
    100/100 [==============================] - 0s 3ms/step - loss: 1.7863
    Epoch 16/100
    100/100 [==============================] - 0s 3ms/step - loss: 1.6916
    Epoch 17/100
    100/100 [==============================] - 0s 3ms/step - loss: 1.6029
    Epoch 18/100
    100/100 [==============================] - 0s 3ms/step - loss: 1.5199
    Epoch 19/100
    100/100 [==============================] - 0s 3ms/step - loss: 1.4424
    Epoch 20/100
    100/100 [==============================] - 0s 3ms/step - loss: 1.3701
    Epoch 21/100
    100/100 [==============================] - 0s 3ms/step - loss: 1.3027
    Epoch 22/100
    100/100 [==============================] - 0s 3ms/step - loss: 1.2399
    Epoch 23/100
    100/100 [==============================] - 0s 3ms/step - loss: 1.1813
    Epoch 24/100
    100/100 [==============================] - 0s 3ms/step - loss: 1.1266
    Epoch 25/100
    100/100 [==============================] - 0s 3ms/step - loss: 1.0756
    Epoch 26/100
    100/100 [==============================] - 0s 3ms/step - loss: 1.0281
    Epoch 27/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.9837
    Epoch 28/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.9422
    Epoch 29/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.9035
    Epoch 30/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.8672
    Epoch 31/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.8333
    Epoch 32/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.8016
    Epoch 33/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.7718
    Epoch 34/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.7438
    Epoch 35/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.7176A: 0s - loss: 0.718
    Epoch 36/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.6929
    Epoch 37/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.6697
    Epoch 38/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.6478
    Epoch 39/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.6272
    Epoch 40/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.6077
    Epoch 41/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.5893
    Epoch 42/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.5719
    Epoch 43/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.5554
    Epoch 44/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.5398
    Epoch 45/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.5250
    Epoch 46/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.5109
    Epoch 47/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.4975
    Epoch 48/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.4848
    Epoch 49/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.4727
    Epoch 50/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.4612
    Epoch 51/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.4501
    Epoch 52/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.4396
    Epoch 53/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.4296
    Epoch 54/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.4200
    Epoch 55/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.4108
    Epoch 56/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.4020
    Epoch 57/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.3936
    Epoch 58/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.3855
    Epoch 59/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.3777
    Epoch 60/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.3703
    Epoch 61/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.3631
    Epoch 62/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.3562
    Epoch 63/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.3496
    Epoch 64/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.3432
    Epoch 65/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.3370
    Epoch 66/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.3311
    Epoch 67/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.3254
    Epoch 68/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.3198
    Epoch 69/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.3145
    Epoch 70/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.3093
    Epoch 71/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.3044
    Epoch 72/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2995
    Epoch 73/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2949
    Epoch 74/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2903
    Epoch 75/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2860
    Epoch 76/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2817
    Epoch 77/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2776
    Epoch 78/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2736
    Epoch 79/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2697
    Epoch 80/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2660
    Epoch 81/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2623
    Epoch 82/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2588
    Epoch 83/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2553
    Epoch 84/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2520
    Epoch 85/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2487
    Epoch 86/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2455
    Epoch 87/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2424
    Epoch 88/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2394
    Epoch 89/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2365
    Epoch 90/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2336
    Epoch 91/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2309
    Epoch 92/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2281
    Epoch 93/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2255
    Epoch 94/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2229
    Epoch 95/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2204
    Epoch 96/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2179
    Epoch 97/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2155
    Epoch 98/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2132
    Epoch 99/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2109
    Epoch 100/100
    100/100 [==============================] - 0s 3ms/step - loss: 0.2086

    <tensorflow.python.keras.callbacks.History at 0x1419b55d0>

```python
figure(figsize=(14,4))
ylim(ymax=1)

a = 128
plt.plot(signal[a,:].reshape((length)))
plt.plot(gt[a,:], '-.')
plt.plot(model.predict(signal[a,:].reshape((1,length,1))).reshape(length))
```

## Kanny kernel

```python
plt.plot(model.get_layer(index=0).get_weights( )[0].flatten())
```

There is a large and rich theory of linear filters in signal processing and image processing:

    - mathematical properties: superposition, decomposition, impulse response
    - frequency domain analysis
    - optimal design for given tasks

NB: "optimal linear" is not the same as "optimal"

# No free lunch theorem!

A model, a family of function, a neural net architecture+regularizer, a likelihood model+prior, are all forms of inductive bias.
And we know that there is no free lunch, meaning that:
- There is no learning without inductive bias.
- There is no neural net training without an architecture.
- There is no statistical estimation without a likelihood model.
- There is no non-parametric estimation without regularization.

Without some sort of inductive bias
- There is no estimation of probability distribution.
- There is no estimation of entropy.
- There is no estimation of mutual information.
- There is no estimation of conditional independence.
- There is no measure of complexity
- There is no estimation of minimum description length.

Which means that none of these things are well defined quantities (except perhaps in the asymptotic case of infinite data. But who cares about that).
The estimation of all of these quantities is subjectively dependent upon your choice of model.
You may say: "the entropy of my data is well defined. It's `H = -SUM_x P(x) log P(x)`
Yes, but what is `P(x)`?
You only know `P(x)` through a bunch of samples.

Which mean you need to estimate a model of `P(x)` from your data.
Which means your model will necessarily have some sort of inductive bias, some sort of arbitrariness in it.

Ultimately, all measures of distributions, information, entropy, complexity and dependency are in the eye of the beholder.

The subjectivity of those quantities also exists when applied to physical systems. The entropy of a physical system is also in the eyes of the beholder.

```python
from tensorflow.keras.layers import BatchNormalization, Activation, ZeroPadding1D
```

```python
# DL model
model = Sequential()
model.add(ZeroPadding1D(input_shape=(length, 1), padding=(8,1)))
model.add(Conv1D(4, 7))
model.add(BatchNormalization(momentum=0.1))
model.add(Activation('relu'))
model.add(ZeroPadding1D(padding=(2,1)))
model.add(Conv1D(1, 7))
model.add(Flatten())
model.add(Activation('sigmoid'))

model.compile(SGD(lr=.1, momentum=0.9, decay=1e-5), loss='categorical_crossentropy')
model.summary()
```

    Model: "sequential_1"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    zero_padding1d (ZeroPadding1 (None, 59, 1)             0
    _________________________________________________________________
    conv1d_1 (Conv1D)            (None, 53, 4)             32
    _________________________________________________________________
    batch_normalization (BatchNo (None, 53, 4)             16
    _________________________________________________________________
    activation (Activation)      (None, 53, 4)             0
    _________________________________________________________________
    zero_padding1d_1 (ZeroPaddin (None, 56, 4)             0
    _________________________________________________________________
    conv1d_2 (Conv1D)            (None, 50, 1)             29
    _________________________________________________________________
    flatten_1 (Flatten)          (None, 50)                0
    _________________________________________________________________
    activation_1 (Activation)    (None, 50)                0
    =================================================================
    Total params: 77
    Trainable params: 69
    Non-trainable params: 8
    _________________________________________________________________

```python
K.set_value(model.optimizer.lr, 1e-3)
model.fit(x = np.expand_dims(signal, 3),
          y=gt,
          batch_size=200,
          epochs=50)
```

    /usr/local/lib/python3.7/site-packages/ipykernel_launcher.py:2: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future.

    Epoch 1/50
    100/100 [==============================] - 0s 3ms/step - loss: 3.2969
    Epoch 2/50
    100/100 [==============================] - 0s 4ms/step - loss: 1.1412
    Epoch 3/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.1079
    Epoch 4/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0440
    Epoch 5/50
    100/100 [==============================] - 0s 4ms/step - loss: 0.0274
    Epoch 6/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0197
    Epoch 7/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0155
    Epoch 8/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0126
    Epoch 9/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0107
    Epoch 10/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0093
    Epoch 11/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0084
    Epoch 12/50
    100/100 [==============================] - 0s 4ms/step - loss: 0.0074
    Epoch 13/50
    100/100 [==============================] - 0s 4ms/step - loss: 0.0068
    Epoch 14/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0062
    Epoch 15/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0057
    Epoch 16/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0052
    Epoch 17/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0049
    Epoch 18/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0046
    Epoch 19/50
    100/100 [==============================] - 0s 4ms/step - loss: 0.0043
    Epoch 20/50
    100/100 [==============================] - 0s 4ms/step - loss: 0.0041
    Epoch 21/50
    100/100 [==============================] - 0s 4ms/step - loss: 0.0039
    Epoch 22/50
    100/100 [==============================] - 0s 4ms/step - loss: 0.0036
    Epoch 23/50
    100/100 [==============================] - 0s 4ms/step - loss: 0.0035
    Epoch 24/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0033
    Epoch 25/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0031
    Epoch 26/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0030
    Epoch 27/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0029
    Epoch 28/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0028
    Epoch 29/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0027
    Epoch 30/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0026
    Epoch 31/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0025
    Epoch 32/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0024
    Epoch 33/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0023
    Epoch 34/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0022
    Epoch 35/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0022
    Epoch 36/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0021
    Epoch 37/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0021
    Epoch 38/50
    100/100 [==============================] - 0s 4ms/step - loss: 0.0020
    Epoch 39/50
    100/100 [==============================] - 0s 4ms/step - loss: 0.0020
    Epoch 40/50
    100/100 [==============================] - 0s 4ms/step - loss: 0.0019
    Epoch 41/50
    100/100 [==============================] - 0s 4ms/step - loss: 0.0018
    Epoch 42/50
    100/100 [==============================] - 0s 4ms/step - loss: 0.0018
    Epoch 43/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0018
    Epoch 44/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0017
    Epoch 45/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0017
    Epoch 46/50
    100/100 [==============================] - 0s 3ms/step - loss: 0.0017
    Epoch 47/50
    100/100 [==============================] - 0s 4ms/step - loss: 0.0016
    Epoch 48/50
    100/100 [==============================] - 0s 4ms/step - loss: 0.0016
    Epoch 49/50
    100/100 [==============================] - 0s 4ms/step - loss: 0.0015
    Epoch 50/50
    100/100 [==============================] - 0s 4ms/step - loss: 0.0015

    <tensorflow.python.keras.callbacks.History at 0x143b095d0>

```python
figure(figsize=(14,4))
ylim(ymax=1)
a = 10
plt.plot(signal[a,:].reshape((length)))
plt.plot(gt[a,:], '-.')
plt.plot(model.predict(signal[a,:].reshape((1,length,1))).reshape(length))
```

```python
for i, l in enumerate(model.layers):
    print (i, l.name)
```

    0 zero_padding1d
    1 conv1d_1
    2 batch_normalization
    3 activation
    4 zero_padding1d_1
    5 conv1d_2
    6 flatten_1
    7 activation_1

## How does it works?

```python
figure(figsize=(14,4))

x = signal[a,:].reshape((1,length,1))
layer = model.layers[1]
f = K.function([model.get_input_at(0)], [layer.get_output_at(0)])
plt.plot(f([x])[0][0,:,:])  # your activation tensor
plt.plot(gt[a,:], '-.')
```

```python
figure(figsize=(14,6))

layer = model.layers[2]
f = K.function([model.get_input_at(0)], [layer.get_output_at(0)])
plt.plot(f([x])[0][0,:,:])  # your activation tensor
plt.plot(gt[a,:], '-.')
```

```python
figure(figsize=(14,4))

layer = model.layers[5]
f = K.function([model.get_input_at(0)], [layer.get_output_at(0)])
plt.plot(f([x])[0][0,:,:])  # your activation tensor
plt.plot(gt[a,:], '-.')
```

- Zero padding on the boundary creates spurious edge responses.
- DL can automatically suppress these for you (beware when benchmarking)
- How would you pick weights to get rid of such spurious responses?

```python
# DL model
model = Sequential()
model.add(ZeroPadding1D(input_shape=(length, 1), padding=(8,1)))
model.add(Conv1D(16, 7))
model.add(BatchNormalization(momentum=0.1))
model.add(Activation('relu'))
model.add(ZeroPadding1D(padding=(2,1)))
model.add(Conv1D(1, 7))
model.add(Flatten())
model.add(Activation('sigmoid'))

model.compile(SGD(lr=1e-2, momentum=0.9, decay=1e-5), loss='categorical_crossentropy')

model.fit(x = np.expand_dims(signal, 3), y=gt, batch_size=200, epochs=50)
```

    /usr/local/lib/python3.7/site-packages/ipykernel_launcher.py:14: DeprecationWarning: Both axis > a.ndim and axis < -a.ndim - 1 are deprecated and will raise an AxisError in the future.

    Epoch 1/50
    100/100 [==============================] - 1s 5ms/step - loss: 0.5026
    Epoch 2/50
    100/100 [==============================] - 1s 6ms/step - loss: 0.0022
    Epoch 3/50
    100/100 [==============================] - 0s 5ms/step - loss: 0.0014
    Epoch 4/50
    100/100 [==============================] - 1s 5ms/step - loss: 0.0011
    Epoch 5/50
    100/100 [==============================] - 1s 5ms/step - loss: 8.6485e-04
    Epoch 6/50
    100/100 [==============================] - 0s 5ms/step - loss: 7.2397e-04
    Epoch 7/50
    100/100 [==============================] - 1s 5ms/step - loss: 6.2857e-04
    Epoch 8/50
    100/100 [==============================] - 1s 5ms/step - loss: 5.5634e-04
    Epoch 9/50
    100/100 [==============================] - 1s 5ms/step - loss: 4.9643e-04
    Epoch 10/50
    100/100 [==============================] - 1s 5ms/step - loss: 4.7250e-04
    Epoch 11/50
    100/100 [==============================] - 1s 5ms/step - loss: 4.2360e-04
    Epoch 12/50
    100/100 [==============================] - 1s 5ms/step - loss: 3.8540e-04
    Epoch 13/50
    100/100 [==============================] - 1s 5ms/step - loss: 3.5587e-04
    Epoch 14/50
    100/100 [==============================] - 1s 5ms/step - loss: 3.2902e-04
    Epoch 15/50
    100/100 [==============================] - 1s 5ms/step - loss: 3.1101e-04
    Epoch 16/50
    100/100 [==============================] - 1s 6ms/step - loss: 3.0796e-04
    Epoch 17/50
    100/100 [==============================] - 1s 5ms/step - loss: 2.8252e-04
    Epoch 18/50
    100/100 [==============================] - 1s 5ms/step - loss: 2.6390e-04
    Epoch 19/50
    100/100 [==============================] - 0s 5ms/step - loss: 2.4992e-04
    Epoch 20/50
    100/100 [==============================] - 0s 5ms/step - loss: 2.4596e-04
    Epoch 21/50
    100/100 [==============================] - 1s 5ms/step - loss: 2.3129e-04
    Epoch 22/50
    100/100 [==============================] - 1s 5ms/step - loss: 2.1925e-04
    Epoch 23/50
    100/100 [==============================] - 1s 5ms/step - loss: 2.3514e-04
    Epoch 24/50
    100/100 [==============================] - 1s 7ms/step - loss: 2.0641e-04
    Epoch 25/50
    100/100 [==============================] - 1s 5ms/step - loss: 1.9244e-04
    Epoch 26/50
    100/100 [==============================] - 1s 6ms/step - loss: 1.8930e-04
    Epoch 27/50
    100/100 [==============================] - 1s 5ms/step - loss: 1.8126e-04
    Epoch 28/50
    100/100 [==============================] - 1s 5ms/step - loss: 1.8018e-04
    Epoch 29/50
    100/100 [==============================] - 1s 5ms/step - loss: 1.7223e-04
    Epoch 30/50
    100/100 [==============================] - 1s 5ms/step - loss: 1.6395e-04
    Epoch 31/50
    100/100 [==============================] - 1s 5ms/step - loss: 1.6237e-04
    Epoch 32/50
    100/100 [==============================] - 0s 5ms/step - loss: 1.6275e-04
    Epoch 33/50
    100/100 [==============================] - 0s 5ms/step - loss: 1.5771e-04
    Epoch 34/50
    100/100 [==============================] - 0s 5ms/step - loss: 1.4646e-04
    Epoch 35/50
    100/100 [==============================] - 1s 5ms/step - loss: 1.4616e-04
    Epoch 36/50
    100/100 [==============================] - 1s 7ms/step - loss: 1.4369e-04
    Epoch 37/50
    100/100 [==============================] - 1s 6ms/step - loss: 1.4063e-04
    Epoch 38/50
    100/100 [==============================] - 0s 5ms/step - loss: 1.3573e-04
    Epoch 39/50
    100/100 [==============================] - 1s 5ms/step - loss: 1.3349e-04
    Epoch 40/50
    100/100 [==============================] - 0s 5ms/step - loss: 1.2955e-04
    Epoch 41/50
    100/100 [==============================] - 0s 5ms/step - loss: 1.2204e-04
    Epoch 42/50
    100/100 [==============================] - 0s 5ms/step - loss: 1.1938e-04
    Epoch 43/50
    100/100 [==============================] - 0s 5ms/step - loss: 1.2378e-04
    Epoch 44/50
    100/100 [==============================] - 0s 5ms/step - loss: 1.1779e-04
    Epoch 45/50
    100/100 [==============================] - 0s 4ms/step - loss: 1.1499e-04
    Epoch 46/50
    100/100 [==============================] - 0s 5ms/step - loss: 1.1127e-04
    Epoch 47/50
    100/100 [==============================] - 0s 5ms/step - loss: 1.0822e-04
    Epoch 48/50
    100/100 [==============================] - 1s 5ms/step - loss: 1.0338e-04
    Epoch 49/50
    100/100 [==============================] - 1s 7ms/step - loss: 1.0708e-04
    Epoch 50/50
    100/100 [==============================] - 1s 6ms/step - loss: 1.0640e-04

    <tensorflow.python.keras.callbacks.History at 0x141895790>

```python
figure(figsize=(18,6))

x = np.zeros(length)
x[int(length/2):] = 1
x = x.reshape((1,length,1))

layer = model.layers[4]
f = K.function([model.get_input_at(0)], [layer.get_output_at(0)])
plt.plot(f([x])[0][0,6:,:])  # your activation tensor
plt.plot(7*x.flatten(), '-.', linewidth=4)
```

- DL discovered another neat trick: Instead of a single edge localizer, train two localization filters in the first layer (plus multiple suppression filters again).
- Each localization filter is offset from the desired peak by one pixel.
- The padding on the second convolutional layer means that the spurious
edges on the boundary are only surrounded by one peak.

# SVM

 Let's try to implement a standard L2-regularized support vector machine (SVM)

 where:

```python
from sklearn import svm
from tensorflow.keras.layers import Input, Dense, Activation
from tensorflow.keras.models import Model
from tensorflow.keras.regularizers import L1L2
import tensorflow.keras.backend as K
```

```python
def make_meshgrid(x, y, h=.02):
    """Create a mesh of points to plot in

    Parameters
    ----------
    x: data to base x-axis meshgrid on
    y: data to base y-axis meshgrid on
    h: stepsize for meshgrid, optional

    Returns
    -------
    xx, yy : ndarray
    """
    x_min, x_max = x.min() - 1, x.max() + 1
    y_min, y_max = y.min() - 1, y.max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))
    return xx, yy

def plot_contours(ax, clf, xx, yy, **params):
    """Plot the decision boundaries for a classifier.

    Parameters
    ----------
    ax: matplotlib axes object
    clf: a classifier
    xx: meshgrid ndarray
    yy: meshgrid ndarray
    params: dictionary of params to pass to contourf, optional
    """
    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()]) > .5
    Z = Z.reshape(xx.shape)
    out = ax.contourf(xx, yy, Z, **params)
    return out

```

```python
rng = np.random.RandomState(0)
n_samples_1 = 1000
n_samples_2 = 1000
X = np.r_[1. * rng.randn(n_samples_1, 2),
          1. * rng.randn(n_samples_2, 2) + [2, 2]]
y = [0] * (n_samples_1) + [1] * (n_samples_2)

# we create an instance of SVM and fit out data. We do not scale our
# data since we want to plot the support vectors
C = 1.0  # SVM regularization parameter
model  = svm.LinearSVC(C=C)
clf = model.fit(X, y)

X0, X1 = X[:, 0], X[:, 1]
xx, yy = make_meshgrid(X0, X1)

plot_contours(plt, clf, xx, yy,
              cmap=plt.cm.coolwarm, alpha=0.8)
plt.scatter(X0, X1, c=y, cmap=plt.cm.coolwarm, s=20, edgecolors='k')

plt.show()
```

```python
def hinge(y_true, y_pred):
    return K.mean(K.maximum(1. - y_true * y_pred, 0.), axis=-1)

l2 = 1/float(C)
inputs = Input(shape=(2,))
x = Dense(1, activation=None, kernel_regularizer=L1L2(l1=0,l2=l2))(inputs)
predictions = Activation('linear')(x)

model = Model(inputs=inputs, outputs=predictions)
model.compile(optimizer='sgd',
              loss=hinge,
              metrics=['accuracy'])

model.summary()

# Wx + b
```

    Model: "model"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    input_1 (InputLayer)         [(None, 2)]               0
    _________________________________________________________________
    dense (Dense)                (None, 1)                 3
    _________________________________________________________________
    activation_4 (Activation)    (None, 1)                 0
    =================================================================
    Total params: 3
    Trainable params: 3
    Non-trainable params: 0
    _________________________________________________________________

```python
K.set_value(model.optimizer.lr, 1e-3)
history = model.fit(x=X, y=np.array(y),
                    batch_size=100,
                    epochs=100,
                    shuffle=True,
                    validation_split=.3)
```

    Epoch 1/100
    14/14 [==============================] - 0s 14ms/step - loss: 3.8389 - accuracy: 0.4507 - val_loss: 6.2210 - val_accuracy: 0.0000e+00
    Epoch 2/100
    14/14 [==============================] - 0s 2ms/step - loss: 3.6745 - accuracy: 0.4536 - val_loss: 5.9683 - val_accuracy: 0.0000e+00
    Epoch 3/100
    14/14 [==============================] - 0s 3ms/step - loss: 3.5189 - accuracy: 0.4564 - val_loss: 5.7268 - val_accuracy: 0.0000e+00
    Epoch 4/100
    14/14 [==============================] - 0s 3ms/step - loss: 3.3718 - accuracy: 0.4579 - val_loss: 5.4958 - val_accuracy: 0.0000e+00
    Epoch 5/100
    14/14 [==============================] - 0s 3ms/step - loss: 3.2326 - accuracy: 0.4600 - val_loss: 5.2750 - val_accuracy: 0.0000e+00
    Epoch 6/100
    14/14 [==============================] - 0s 3ms/step - loss: 3.1010 - accuracy: 0.4607 - val_loss: 5.0638 - val_accuracy: 0.0000e+00
    Epoch 7/100
    14/14 [==============================] - 0s 3ms/step - loss: 2.9764 - accuracy: 0.4636 - val_loss: 4.8616 - val_accuracy: 0.0000e+00
    Epoch 8/100
    14/14 [==============================] - 0s 3ms/step - loss: 2.8586 - accuracy: 0.4657 - val_loss: 4.6682 - val_accuracy: 0.0000e+00
    Epoch 9/100
    14/14 [==============================] - 0s 3ms/step - loss: 2.7472 - accuracy: 0.4679 - val_loss: 4.4830 - val_accuracy: 0.0000e+00
    Epoch 10/100
    14/14 [==============================] - 0s 3ms/step - loss: 2.6417 - accuracy: 0.4693 - val_loss: 4.3057 - val_accuracy: 0.0000e+00
    Epoch 11/100
    14/14 [==============================] - 0s 3ms/step - loss: 2.5419 - accuracy: 0.4764 - val_loss: 4.1359 - val_accuracy: 0.0000e+00
    Epoch 12/100
    14/14 [==============================] - 0s 2ms/step - loss: 2.4476 - accuracy: 0.4814 - val_loss: 3.9733 - val_accuracy: 0.0000e+00
    Epoch 13/100
    14/14 [==============================] - 0s 3ms/step - loss: 2.3583 - accuracy: 0.4829 - val_loss: 3.8175 - val_accuracy: 0.0000e+00
    Epoch 14/100
    14/14 [==============================] - 0s 3ms/step - loss: 2.2738 - accuracy: 0.4900 - val_loss: 3.6682 - val_accuracy: 0.0000e+00
    Epoch 15/100
    14/14 [==============================] - 0s 3ms/step - loss: 2.1939 - accuracy: 0.4957 - val_loss: 3.5251 - val_accuracy: 0.0000e+00
    Epoch 16/100
    14/14 [==============================] - 0s 3ms/step - loss: 2.1182 - accuracy: 0.5000 - val_loss: 3.3878 - val_accuracy: 0.0000e+00
    Epoch 17/100
    14/14 [==============================] - 0s 3ms/step - loss: 2.0466 - accuracy: 0.5064 - val_loss: 3.2562 - val_accuracy: 0.0000e+00
    Epoch 18/100
    14/14 [==============================] - 0s 2ms/step - loss: 1.9789 - accuracy: 0.5114 - val_loss: 3.1300 - val_accuracy: 0.0000e+00
    Epoch 19/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.9148 - accuracy: 0.5150 - val_loss: 3.0089 - val_accuracy: 0.0000e+00
    Epoch 20/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.8541 - accuracy: 0.5229 - val_loss: 2.8926 - val_accuracy: 0.0000e+00
    Epoch 21/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.7966 - accuracy: 0.5279 - val_loss: 2.7810 - val_accuracy: 0.0000e+00
    Epoch 22/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.7422 - accuracy: 0.5329 - val_loss: 2.6739 - val_accuracy: 0.0000e+00
    Epoch 23/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.6908 - accuracy: 0.5393 - val_loss: 2.5710 - val_accuracy: 0.0000e+00
    Epoch 24/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.6421 - accuracy: 0.5500 - val_loss: 2.4721 - val_accuracy: 0.0000e+00
    Epoch 25/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.5959 - accuracy: 0.5543 - val_loss: 2.3771 - val_accuracy: 0.0000e+00
    Epoch 26/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.5522 - accuracy: 0.5636 - val_loss: 2.2858 - val_accuracy: 0.0000e+00
    Epoch 27/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.5109 - accuracy: 0.5700 - val_loss: 2.1980 - val_accuracy: 0.0000e+00
    Epoch 28/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.4717 - accuracy: 0.5750 - val_loss: 2.1136 - val_accuracy: 0.0017
    Epoch 29/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.4346 - accuracy: 0.5821 - val_loss: 2.0324 - val_accuracy: 0.0017
    Epoch 30/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.3995 - accuracy: 0.5893 - val_loss: 1.9543 - val_accuracy: 0.0017
    Epoch 31/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.3662 - accuracy: 0.5971 - val_loss: 1.8791 - val_accuracy: 0.0017
    Epoch 32/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.3347 - accuracy: 0.6043 - val_loss: 1.8067 - val_accuracy: 0.0017
    Epoch 33/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.3048 - accuracy: 0.6129 - val_loss: 1.7370 - val_accuracy: 0.0017
    Epoch 34/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.2765 - accuracy: 0.6271 - val_loss: 1.6698 - val_accuracy: 0.0017
    Epoch 35/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.2497 - accuracy: 0.6350 - val_loss: 1.6051 - val_accuracy: 0.0017
    Epoch 36/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.2243 - accuracy: 0.6457 - val_loss: 1.5428 - val_accuracy: 0.0017
    Epoch 37/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.2002 - accuracy: 0.6479 - val_loss: 1.4827 - val_accuracy: 0.0017
    Epoch 38/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.1773 - accuracy: 0.6543 - val_loss: 1.4248 - val_accuracy: 0.0017
    Epoch 39/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.1556 - accuracy: 0.6607 - val_loss: 1.3688 - val_accuracy: 0.0017
    Epoch 40/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.1351 - accuracy: 0.6664 - val_loss: 1.3149 - val_accuracy: 0.0017
    Epoch 41/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.1156 - accuracy: 0.6743 - val_loss: 1.2629 - val_accuracy: 0.0017
    Epoch 42/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.0971 - accuracy: 0.6829 - val_loss: 1.2126 - val_accuracy: 0.0017
    Epoch 43/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.0796 - accuracy: 0.6886 - val_loss: 1.1642 - val_accuracy: 0.0017
    Epoch 44/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.0629 - accuracy: 0.6943 - val_loss: 1.1173 - val_accuracy: 0.0017
    Epoch 45/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.0471 - accuracy: 0.6986 - val_loss: 1.0720 - val_accuracy: 0.0017
    Epoch 46/100
    14/14 [==============================] - 0s 2ms/step - loss: 1.0321 - accuracy: 0.7007 - val_loss: 1.0283 - val_accuracy: 0.0017
    Epoch 47/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.0178 - accuracy: 0.7029 - val_loss: 0.9860 - val_accuracy: 0.0017
    Epoch 48/100
    14/14 [==============================] - 0s 3ms/step - loss: 1.0043 - accuracy: 0.7050 - val_loss: 0.9451 - val_accuracy: 0.0017
    Epoch 49/100
    14/14 [==============================] - 0s 4ms/step - loss: 0.9915 - accuracy: 0.7057 - val_loss: 0.9056 - val_accuracy: 0.0017
    Epoch 50/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.9792 - accuracy: 0.7064 - val_loss: 0.8674 - val_accuracy: 0.0017
    Epoch 51/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.9676 - accuracy: 0.7100 - val_loss: 0.8304 - val_accuracy: 0.0017
    Epoch 52/100
    14/14 [==============================] - 0s 2ms/step - loss: 0.9566 - accuracy: 0.7114 - val_loss: 0.7946 - val_accuracy: 0.0050
    Epoch 53/100
    14/14 [==============================] - 0s 2ms/step - loss: 0.9461 - accuracy: 0.7121 - val_loss: 0.7599 - val_accuracy: 0.0117
    Epoch 54/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.9361 - accuracy: 0.7136 - val_loss: 0.7263 - val_accuracy: 0.0233
    Epoch 55/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.9266 - accuracy: 0.7157 - val_loss: 0.6938 - val_accuracy: 0.0550
    Epoch 56/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.9175 - accuracy: 0.7221 - val_loss: 0.6622 - val_accuracy: 0.0967
    Epoch 57/100
    14/14 [==============================] - 0s 2ms/step - loss: 0.9089 - accuracy: 0.7393 - val_loss: 0.6317 - val_accuracy: 0.1533
    Epoch 58/100
    14/14 [==============================] - 0s 4ms/step - loss: 0.9007 - accuracy: 0.7586 - val_loss: 0.6020 - val_accuracy: 0.2183
    Epoch 59/100
    14/14 [==============================] - 0s 5ms/step - loss: 0.8928 - accuracy: 0.7871 - val_loss: 0.5732 - val_accuracy: 0.3117
    Epoch 60/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8854 - accuracy: 0.8100 - val_loss: 0.5453 - val_accuracy: 0.3967
    Epoch 61/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8782 - accuracy: 0.8300 - val_loss: 0.5183 - val_accuracy: 0.4650
    Epoch 62/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8714 - accuracy: 0.8493 - val_loss: 0.4920 - val_accuracy: 0.5433
    Epoch 63/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8650 - accuracy: 0.8636 - val_loss: 0.4665 - val_accuracy: 0.6183
    Epoch 64/100
    14/14 [==============================] - 0s 2ms/step - loss: 0.8588 - accuracy: 0.8779 - val_loss: 0.4418 - val_accuracy: 0.7017
    Epoch 65/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8528 - accuracy: 0.8893 - val_loss: 0.4179 - val_accuracy: 0.7700
    Epoch 66/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8472 - accuracy: 0.8993 - val_loss: 0.3948 - val_accuracy: 0.8100
    Epoch 67/100
    14/14 [==============================] - ETA: 0s - loss: 0.8673 - accuracy: 0.86 - 0s 3ms/step - loss: 0.8418 - accuracy: 0.9086 - val_loss: 0.3725 - val_accuracy: 0.8433
    Epoch 68/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8367 - accuracy: 0.9086 - val_loss: 0.3515 - val_accuracy: 0.8617
    Epoch 69/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8320 - accuracy: 0.9071 - val_loss: 0.3319 - val_accuracy: 0.8867
    Epoch 70/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8277 - accuracy: 0.9071 - val_loss: 0.3143 - val_accuracy: 0.8933
    Epoch 71/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8239 - accuracy: 0.9036 - val_loss: 0.2980 - val_accuracy: 0.9100
    Epoch 72/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8203 - accuracy: 0.9000 - val_loss: 0.2831 - val_accuracy: 0.9267
    Epoch 73/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8170 - accuracy: 0.9029 - val_loss: 0.2694 - val_accuracy: 0.9383
    Epoch 74/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8142 - accuracy: 0.8971 - val_loss: 0.2576 - val_accuracy: 0.9417
    Epoch 75/100
    14/14 [==============================] - 0s 2ms/step - loss: 0.8119 - accuracy: 0.8886 - val_loss: 0.2474 - val_accuracy: 0.9500
    Epoch 76/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8099 - accuracy: 0.8836 - val_loss: 0.2385 - val_accuracy: 0.9567
    Epoch 77/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8081 - accuracy: 0.8757 - val_loss: 0.2304 - val_accuracy: 0.9633
    Epoch 78/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8065 - accuracy: 0.8714 - val_loss: 0.2233 - val_accuracy: 0.9683
    Epoch 79/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8050 - accuracy: 0.8693 - val_loss: 0.2170 - val_accuracy: 0.9733
    Epoch 80/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8038 - accuracy: 0.8614 - val_loss: 0.2115 - val_accuracy: 0.9783
    Epoch 81/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8026 - accuracy: 0.8579 - val_loss: 0.2066 - val_accuracy: 0.9783
    Epoch 82/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8016 - accuracy: 0.8564 - val_loss: 0.2022 - val_accuracy: 0.9800
    Epoch 83/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.8006 - accuracy: 0.8536 - val_loss: 0.1981 - val_accuracy: 0.9817
    Epoch 84/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.7998 - accuracy: 0.8486 - val_loss: 0.1944 - val_accuracy: 0.9817
    Epoch 85/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.7989 - accuracy: 0.8450 - val_loss: 0.1909 - val_accuracy: 0.9817
    Epoch 86/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.7982 - accuracy: 0.8429 - val_loss: 0.1879 - val_accuracy: 0.9817
    Epoch 87/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.7974 - accuracy: 0.8421 - val_loss: 0.1851 - val_accuracy: 0.9817
    Epoch 88/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.7967 - accuracy: 0.8407 - val_loss: 0.1825 - val_accuracy: 0.9833
    Epoch 89/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.7961 - accuracy: 0.8371 - val_loss: 0.1800 - val_accuracy: 0.9833
    Epoch 90/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.7954 - accuracy: 0.8357 - val_loss: 0.1777 - val_accuracy: 0.9833
    Epoch 91/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.7948 - accuracy: 0.8350 - val_loss: 0.1756 - val_accuracy: 0.9833
    Epoch 92/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.7942 - accuracy: 0.8314 - val_loss: 0.1735 - val_accuracy: 0.9833
    Epoch 93/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.7936 - accuracy: 0.8264 - val_loss: 0.1715 - val_accuracy: 0.9833
    Epoch 94/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.7931 - accuracy: 0.8243 - val_loss: 0.1697 - val_accuracy: 0.9833
    Epoch 95/100
    14/14 [==============================] - 0s 2ms/step - loss: 0.7925 - accuracy: 0.8229 - val_loss: 0.1680 - val_accuracy: 0.9867
    Epoch 96/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.7920 - accuracy: 0.8200 - val_loss: 0.1664 - val_accuracy: 0.9867
    Epoch 97/100
    14/14 [==============================] - ETA: 0s - loss: 0.7437 - accuracy: 0.90 - 0s 3ms/step - loss: 0.7914 - accuracy: 0.8186 - val_loss: 0.1648 - val_accuracy: 0.9883
    Epoch 98/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.7909 - accuracy: 0.8157 - val_loss: 0.1633 - val_accuracy: 0.9883
    Epoch 99/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.7904 - accuracy: 0.8121 - val_loss: 0.1618 - val_accuracy: 0.9883
    Epoch 100/100
    14/14 [==============================] - 0s 3ms/step - loss: 0.7899 - accuracy: 0.8107 - val_loss: 0.1604 - val_accuracy: 0.9883

```python
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper right');
```

```python
plot_contours(plt, model, xx, yy,
              cmap=plt.cm.coolwarm, alpha=0.8)
plt.scatter(X0, X1, c=y, cmap=plt.cm.coolwarm, s=20, edgecolors='k')
```

# IIR Filters and their DL Equivalents

```python
# simple IIR filter

def simple_iir(xs):
    value = xs[0]
    output = zeros(len(xs))
    for i in range(len(xs)):
        output[i] = value
        value = 0.7 * value + 0.3 * xs[i]
    return output
```

```python
xs = rand(100) * (rand(100) < 0.1)

figure(figsize=(14,5))
plot(xs)
plot(simple_iir(xs))
```

## IIR Filters

IIR filters are simple linear filters.

Unlike FIR filters, the output is a linear function of both inputs and past output values.

IIR filters can approximate FIR filters well.

IIR filters are the linear equivalent of *recurrent neural networks*.

```python
inputs = array([(rand(200) < 0.1).astype('f') for i in range(10000)]).reshape(-1, 1, 200)
outputs = array([4.0*ndi.gaussian_filter(s, 3.0) for s in inputs]).reshape(-1, 1, 200)

figure(figsize=(14,5))
plot(inputs[0,0]); plot(outputs[0,0])
```

### Simple NN

```python
length = 200

model = Sequential()
model.add(Conv1D(1, 33, input_shape=(length, 1), activation="sigmoid", padding='same'))
model.add(Flatten())

model.compile(SGD(lr=.1, momentum=0.9, decay=1e-5), loss='mse')
model.summary()
```

```python
#K.set_value(model.optimizer.lr, 1e-2)
model.fit(x=inputs.reshape((-1, 200,1)), y=outputs.reshape((-1, 200)), batch_size=100, epochs=100)
```

```python
pred = model.predict(inputs.reshape((-1, 200,1)))

figure(figsize=(15,5))
plot(inputs[0,0])
plot(pred[0], linewidth=2, color="green")
plot(outputs[0,0], linewidth=8, alpha=0.3, color="red")
```

### Recurrent neural network - LSTM

```python
from tensorflow.keras.layers import LSTM, Bidirectional
```

```python
length = 200

model = Sequential()
model.add(Bidirectional(LSTM(4, input_shape=(length, 1), activation="sigmoid", return_sequences=True)))
model.add(Conv1D(1, 8, padding='same'))

model.compile(SGD(lr=.1, momentum=0.9, decay=1e-5), loss='mse')
model.build(input_shape=(None, 200,1))
```

```python
model.summary()
```

```python
K.set_value(model.optimizer.lr, 1e-1)
model.fit(x=inputs.reshape((-1, 200,1)),
          y=outputs.reshape((-1, 200,1)),
          batch_size=1000,
          epochs=15)
```

```python
pred = model.predict(inputs.reshape((-1, 200,1)))

figure(figsize=(15,5))
plot(inputs[0,0])
plot(pred[0], linewidth=2, color="green")
plot(outputs[0,0], linewidth=8, alpha=0.3, color="red")
```

# Handsome input pipelines with the `tf.data` API

The `tf.data` API offers functions for data pipelining and related operations. We can build pipelines, map preprocessing functions, shuffle or batch a dataset and much more.

## From tensors

```python
dataset = tf.data.Dataset.from_tensor_slices([8, 3, 0, 8, 2, 1])
iter(dataset).next().numpy()
```

    8

## Batch and Shuffle

```python
# Shuffle
dataset = tf.data.Dataset.from_tensor_slices([8, 3, 0, 8, 2, 1]).shuffle(6)
iter(dataset).next().numpy()
```

    2

```python
# Batch
dataset = tf.data.Dataset.from_tensor_slices([8, 3, 0, 8, 2, 1]).batch(2)
iter(dataset).next().numpy()
```

    array([8, 3], dtype=int32)

```python
# Shuffle and Batch
dataset = tf.data.Dataset.from_tensor_slices([8, 3, 0, 8, 2, 1]).shuffle(6).batch(2)
iter(dataset).next().numpy()
```

    array([1, 2], dtype=int32)

## Zipping Two Datsets

```python
dataset0 = tf.data.Dataset.from_tensor_slices([8, 3, 0, 8, 2, 1])
dataset1 = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4, 5, 6])
dataset = tf.data.Dataset.zip((dataset0, dataset1))
iter(dataset).next()
```

    (<tf.Tensor: shape=(), dtype=int32, numpy=8>,
     <tf.Tensor: shape=(), dtype=int32, numpy=1>)

## Mapping External Functions

```python
def into_2(num):
     return num * 2

dataset = tf.data.Dataset.from_tensor_slices([8, 3, 0, 8, 2, 1]).map(into_2)
```

## ImageDataGenerator

This is one of the best features of the `tensorflow.keras` API (in my opinion). The `ImageDataGenerator` is capable of generating dataset slices while batching and preprocessing along with data augmentation in real-time.

The Generator allows data flow directly from directories or from dataframes.

A misconception about data augmentation in `ImageDataGenerator` is that, it adds more data to the existing dataset. Although that is the actual definition of data augmentation, in `ImageDataGenerator`, the images in the dataset are transformed dynamically at different steps in training so that the model can be trained on noisy data it hasn’t seen.

```python
from tensorflow.keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True
)
```

Here, the rescaling is done on all the samples (for normalizing), while the other parameters are for augmentation.

```python
train_generator = train_datagen.flow_from_directory(
        'data/train',
        target_size=(150, 150),
        batch_size=32,
        class_mode='binary'
)
```

# Custom Layers

Neural Nets are known for many layer deep networks wherein the layers can be of different types. TensorFlow contains many predefined layers (like Dense, LSTM, etc.). But for more complex architectures, the logic of a layer is much more complex than a primary layer. For such instances, TensorFlow allows building custom layers. This can be done by subclassing the tf.keras.layers.Layer class.

```python
class CustomDense(tf.keras.layers.Layer):
    def __init__(self, num_outputs):
        super(CustomDense, self).__init__()
        self.num_outputs = num_outputs

    def build(self, input_shape):
        self.kernel = self.add_weight(
            "kernel",
            shape=[int(input_shape[-1]),
            self.num_outputs]
        )

    def call(self, input):
        return tf.matmul(input, self.kernel)
```

As stated in the documentation, The best way to implement your own layer is extending the `tf.keras.Layer` class and implementing:

-    `__init__` , where you can do all input-independent initialization.
-    `build`, where you know the shapes of the input tensors and can do the rest of the initialization.
-    `call`, where you do the forward computation.

Although the kernel initialization can be done in `__init__` itself, it is considered better to be initialized in build as otherwise, you would have to explicitly specify the input_shape on every instance of a new layer creation.

# Custom Training

The `tf.keras` Sequential and the Model API makes training models easier. However, most of the time while training complex models, custom loss functions are used. Moreover, the model training can also differ from the default (for eg. applying gradients separately to different model components).

TensorFlow’s automatic differentiation helps calculating gradients in an efficient way. These primitives are used in defining custom training loops.

```python
def train(model, inputs, outputs, learning_rate):
    with tf.GradientTape() as t:
        # Computing Losses from Model Prediction
        current_loss = loss(outputs, model(inputs))          # Gradients for Trainable Variables with Obtained Losses
    dW, db = t.gradient(current_loss, [model.W, model.b])    # Applying Gradients to Weights
    model.W.assign_sub(learning_rate * dW)
    model.b.assign_sub(learning_rate * db)
```

This loop can be repeated for multiple epochs and with a more customised setting as per the use case.

# Autoencoders

predict the input, given that same input.

\begin{equation}\label{eq:}
F_{W,b}(x) \approx x
\end{equation}

The goal is to learn a compressed representation of the data, thus find structure. This can be done by limiting the number of hidden units in the model. Those kind of autoencoders are called *undercomplete*.


```python
import numpy as np
import pandas as pd

```

## 1. Cálculo pelo processo convencional

Os dados da torre fornecidos para o problema são:
\begin{align}
\gamma_{concreto} &= 2500 kg/m^3\\
F_{ck} &= 30MPa
\end{align}

```python
A_F = np.pi*(5**2 - 4.6**2)/4
A_L = np.pi*(10**2)/4

mi_F = A_F * 2500

m_L = 5* A_L *0.2*2500

print(" Massa por comprimento do fuste:{0:5.3f}kg/m.".format(mi_F))
print(" Massa das 5 lajes :{0:5.3f}kg.".format(m_L))

```

     Massa por comprimento do fuste:7539.822kg/m.
     Massa das 5 lajes :196349.541kg.

As características do vento estão expostas abaixo:

```python
Vo = 46    ## Velocidade básica do vento
S1 = 1.0   ## Fator topográfico
S3 = 1.0   ## Fator estatístico

## Para categoria III e classe C.
b = 0.93
p = 0.115
################################

z = [4,12.5,21.5,30.5,39.5,46,50,54,58] ## discretização da altura da torre
S2 = np.zeros(len(z))
for i in range (len(z)):
    S2[i] = b*0.95*(z[i]/10)**p

Vk = S1*S2*S3*Vo ## Velocidade característica em cada altura

```

Para o cálculo dos esforços que atuarão na torre, é necessário a definição dos coeficientes de arrasto $C_a$( Tabela 10 da NBR 6123/88):

```python
C_F = 0.5 ## Coeficiente de arrasto do fuste
C_K = 0.5 ## Coeficiente de arrasto do Kevlar
Ca = np.ones(9)* C_F
```

O cálculo da pressão dinâmica e da força de arrasto serão calculados, respectivamente, pelas seguintes equações:

$${q} = 0.613 * V_k^2 \\$$
$$ F_a = C_a * q * A $$

```python
q = 0.613*Vk**2
A = [41.25,45,45,45,45,40,40,40,40]
F1 = Ca*q*A
M1 = F1*z
data = {'Vk (m/s)':Vk[::-1],'S2': S2[::-1],'A (m^2)':A[::-1],
        'q (kPa)':q[::-1]/1000,'Ca':Ca[::-1],'F (kN)':F1[::-1]/1000,'M (kN.m)':M1[::-1]/1000}

Plan = pd.DataFrame( data = data,
                   index = z[::-1])
print(Plan)

```

           Vk (m/s)        S2  A (m^2)   q (kPa)   Ca     F (kN)     M (kN.m)
    58.0  49.746050  1.081436    40.00  1.516972  0.5  30.339448  1759.688004
    54.0  49.338923  1.072585    40.00  1.492244  0.5  29.844878  1611.623405
    50.0  48.904175  1.063134    40.00  1.466062  0.5  29.321241  1466.062032
    46.0  48.437480  1.052989    40.00  1.438214  0.5  28.764282  1323.156986
    39.5  47.596284  1.034702    45.00  1.388694  0.5  31.245616  1234.201845
    30.5  46.201797  1.004387    45.00  1.308514  0.5  29.441554   897.967391
    21.5  44.380767  0.964799    45.00  1.207397  0.5  27.166432   584.078297
    12.5  41.697406  0.906465    45.00  1.065807  0.5  23.980656   299.758205
    4.0   36.576427  0.795140    41.25  0.820093  0.5  16.914416    67.657663

O somatório dos esforços são:

```python
print('Esforço cortante ={0:5.3f}N'.format(np.sum(F1)))
print('Mommento fletor ={0:5.3f}N.m'.format(np.sum(M1)))
```

    Esforço cortante =247018.524N
    Mommento fletor =9244193.828N.m

## 2. Solução pelo capítulo 9 da NBR6123/88

### 1. Método simplificado
Para a velocidade média do vento, utiliza-se a seguinte expressão:
$$ V_k = 0.69*V_o*S1*S2 $$
Para uma chaminé de concreto com razão uniforme (Tabela 19), tem-se:
$$ \\ expoente\,da\,Forma \,modal = 1.7\\ $$
$$Razão\, de \,amortecimento = 0.010 \\$$
$$Frequência\, fundamental= 1.11 Hz$$
Já a pressão dinâmica pode ser expressa por:

$$ q = \overline{q_o}*b^2 \left[ \left( \frac{z}{z_r} \right) ^{2p}+\left(\frac{h}{z_r} \right) ^{p}*\left(
\frac{z}{h} \right) ^{\gamma} *\frac{1+2*\gamma}{1+\gamma+p}*\xi\right]$$

$$ \overline{q_o} =  0.613 * V_k^2 $$

Pela interpolação linear da Figura 16, $ \xi = 1.50$ e pela tabela 20, tem-se que $ b = 0.86$ e  $p = 0.185$

```python
Vk = S1*0.69*S3*Vo
q = 0.613*Vk**2
q_d = np.zeros(9)
for n in range (9):

    q_d[n] = q *0.86**2 *((z[n]/10)**(2*0.185) + (60/10)**0.185 *(z[n]/60)**1.7 * (1+2*1.7)*1.5/(1+1.7+0.185))

F2 = Ca*q_d*A
M2 = F2*z
```

Montando agora a tabela para o cálculo dos esforços:

```python
data = {'Vk (m/s)':Vk,'A (m^2)':A[::-1],
        'q (kPa)':q_d[::-1]/1000,'Ca':Ca[::-1],'F (kN)':F2[::-1]/1000,'M (kN.m)':M2[::-1]/1000}

Plan = pd.DataFrame( data = data,
                   index = z[::-1])
print(Plan)
```

          Vk (m/s)  A (m^2)   q (kPa)   Ca     F (kN)     M (kN.m)
    58.0     31.74    40.00  2.249300  0.5  44.986000  2609.188001
    54.0     31.74    40.00  2.069284  0.5  41.385671  2234.826246
    50.0     31.74    40.00  1.896122  0.5  37.922432  1896.121593
    46.0     31.74    40.00  1.729854  0.5  34.597071  1591.465257
    39.5     31.74    45.00  1.474428  0.5  33.174635  1310.398098
    30.5     31.74    45.00  1.150786  0.5  25.892690   789.727054
    21.5     31.74    45.00  0.860562  0.5  19.362651   416.296998
    12.5     31.74    45.00  0.597191  0.5  13.436809   167.960106
    4.0      31.74    41.25  0.339989  0.5   7.012274    28.049095

```python
print('Esforço cortante ={0:5.3f}N'.format(np.sum(F2)))
print('Mommento fletor ={0:5.3f}N.m'.format(np.sum(M2)))
```

    Esforço cortante =257770.233N
    Mommento fletor =11044032.447N.m

### 2. Método discreto

O capítulo 9 da norma estabelece o seguinte roteiro para a análise dinâmica
$$ x_i = \left(\frac{z_i}{h}\right)^\gamma\\ $$
$$ X_i = \overline{X_i} + \hat{X_i}\\ $$
$$ \overline{X_i} = \overline{q_o}*b^2*C_{ai}*A_i*\left(\frac{z_i}{z_r}\right)^{2p}\\ $$
$$ \hat{x_i} = FH *\phi_i* x_i\\ $$
$$ \phi_i = \frac{m_i}{m_o}\\ $$
$$ FH = \overline{q_o}*b^2*C_i*A_O* \frac{\sum_{n=1}^{n}\beta_i * x_i}{\sum_{n=1}^{n}\phi_i * x_i^2}*\xi\\ $$
$$ A_o = \sum A_i \\ $$
$$ m_o = \sum m_i \\ $$
$$ \beta_i = C_{ai} * \frac{A_i}{A_o} \left( \frac{z_i}{z_r}\right)^p $$

Onde $x_i$ é a forma modal, $X_i$ é a velocidade instantânea do vento, $\overline{X_i}$ é a velocidade média e $\hat{X_i}$ é a velocidade flutuante.

```python
xi = np.zeros(9)
Xi_m = np.zeros(9)
beta =np.zeros(9)
mi = [24543.75, 34361.25, 35343,35343, 31346, 89559.5, 55779.3,55779.3, 75814.95]
phi = np.zeros(9)

for i in range(9):

    xi[i] = (z[i]/60)**1.7
    beta[i] = 0.5* A[i]/sum(A)*(z[i]/10)**0.185
    Xi_m[i] = q *0.86**2 * 0.5 * A[i] * (z[i]/10)**(2*0.185)
    phi[i] = mi[i]/sum(mi)

FH = q * 0.86**2 * 0.5 * np.sum(A)*np.sum(beta*xi)/sum(phi*xi**2) * 1.5

xi_f = FH * phi * xi
Xi = Xi_m + xi_f
M3 = (Xi)*z

data = {'A (m^2)':A[::-1],'m(kg)':mi[::-1],'beta':beta[::-1],
        'phi':phi[::-1],'x': xi[::-1],'FH':FH,'X médio (kN)':Xi_m[::-1]/1000,
        'x_f (kN)':xi_f[::-1]/1000,'Xi (kN)':Xi[::-1]/1000,'M (kN.m)':M3[::-1]/1000}

Plan = pd.DataFrame( data = data,
                   index = z[::-1])
print(Plan)

```

          A (m^2)     m(kg)      beta       phi         x            FH  \
    58.0    40.00  75814.95  0.072620  0.173145  0.943997  93106.752898
    54.0    40.00  55779.30  0.071666  0.127388  0.836012  93106.752898
    50.0    40.00  55779.30  0.070653  0.127388  0.733486  93106.752898
    46.0    40.00  89559.50  0.069571  0.204534  0.636548  93106.752898
    39.5    45.00  31346.00  0.076093  0.071587  0.491313  93106.752898
    30.5    45.00  35343.00  0.072538  0.080716  0.316557  93106.752898
    21.5    45.00  35343.00  0.067994  0.080716  0.174698  93106.752898
    12.5    45.00  34361.25  0.061504  0.078474  0.069485  93106.752898
    4.0     41.25  24543.75  0.045663  0.056053  0.010015  93106.752898

          X médio (kN)   x_f (kN)    Xi (kN)     M (kN.m)
    58.0     17.505301  15.218129  32.723430  1897.958912
    54.0     17.048529   9.915652  26.964181  1456.065753
    50.0     16.569909   8.699632  25.269541  1263.477041
    46.0     16.066512  12.122126  28.188638  1296.677363
    39.5     17.084201   3.274734  20.358935   804.177915
    30.5     15.525467   2.378986  17.904454   546.085836
    21.5     13.641307   1.312888  14.954194   321.515179
    12.5     11.161189   0.507686  11.668874   145.860926
    4.0       6.711620   0.052266   6.763887    27.055546

Portanto, os esforços na base são:

```python
print('Esforço cortante ={0:5.3f}N'.format(np.sum(Xi)))
print('Mommento fletor ={0:5.3f}N.m'.format(np.sum(M3)))
```

    Esforço cortante =184796.132N
    Mommento fletor =7758874.472N.m

Comparando os valores obtidos por cada método disponível na NBR 6123\88, monta-se a seguinte tabela:

```python
data =[np.sum(F1)/1000,np.sum(M1)/1000,np.sum(F2)/1000,np.sum(M2)/1000,np.sum(Xi)/1000,np.sum(M3)/1000]
comp = pd.DataFrame( data = data,
                   index = ['Força convencional','Momento convecional','Força simplificada',
                            'Momento simplificado',
                             'Força discreta','Momento discreto'],
                   columns = ['Valores'])
print(comp)
```

                               Valores
    Força convencional      247.018524
    Momento convecional    9244.193828
    Força simplificada      257.770233
    Momento simplificado  11044.032447
    Força discreta          184.796132
    Momento discreto       7758.874472

## 3. Desprendimento de vórtices pela norma canadense
A principio, deve-se calcular a velocidade crítica, que consiste na velocidade do vento em que a estrutura entrará em ressonância com as forças transversais.

$$ V_{cr} = \frac{1}{S_t}*f*d $$

Onde $S_t$ é o número de Strouhal, que pode apresentar os seguintes valores:

$$ S_t = 0.2,\, 10^3 \leq R_e \leq 2 *10^6$$
$$ S_t = 0.28,\, R_e \geq 2 *10^6$$

O cálculo das características da torre, como módulo de elasticidade, momento de inércia e Rigidez estão apresentados abaixo:

```python
E = 6600*(30+3.5)**0.5
I = 0.25 * np.pi * (2.5**4 - 2.3**4)
K = 3* E*I/(60**3)*10**6
print(' Módulo de elasticidade:     {0:5.3f}MPa'.format(E),'\n',
      'Momento de inércia:         {0:5.3f}m^4'.format(I),'\n',
      'Rigidez à flexão:           {0:5.3f}N/m'.format(K))
```

     Módulo de elasticidade:     38200.262MPa
     Momento de inércia:         8.701m^4
     Rigidez à flexão:           4616371.656N/m

Calculando, agora, as frequências naturais da torre e da massa concentrada (Lajes) em cima dela, por meio das equações:

    Para a massa concentrada
$$f = \frac{1}{2\pi}\sqrt{\frac{K}{m}}$$

 Para a massa Distribuída
$$f = \frac{3.52}{2\pi}\sqrt{\frac{EI}{ML^4}}$$

Tendo estes dois valores, é possível estimar a frequência natural da torre junto da massa, por meio da equação:

$$ \frac{1}{f_n^2} = \frac{1}{f_1^2}+\frac{1}{f_2^2}$$

```python
f1 = 1/(2*np.pi)*np.sqrt(K/m_L)
f2 = 3.52/(2*np.pi)*np.sqrt(E*10**6*I/(mi_F*60**4))
inv_fn = 1/(f1**2) + 1/(f2**2)
fn = (1/inv_fn)**0.5
print(' Frequência das lajes:      {0:5.3f}Hz'.format(f1),'\n',
      'Freqência do fuste:        {0:5.3f}Hz'.format(f2),'\n',
      'Frequência da torre:       {0:5.3f}Hz'.format(fn))

```

     Frequência das lajes:      0.772Hz
     Freqência do fuste:        1.033Hz
     Frequência da torre:       0.618Hz

Com o valor do número de Reynolds obtido nos itens anteriores, o número de Strouhal é o Seguinte:
$$ S_t = 0.2$$

Calculando agora a velocidade crítica para os dois diâmetros da torre:

```python
v1 = 1/0.2 * fn * 5
v2 = 1/0.2 * fn * 10
print(' Velocidade crítica no fuste:         {0:5.3f}m/s'.format(v1),'\n',
      'Velocidade crítica no Kevlar:        {0:5.3f}m/s'.format(v2))
```

     Velocidade crítica no fuste:         15.457m/s
     Velocidade crítica no Kevlar:        30.915m/s

Como a velocidade de projeto é maior que as velocidades críticas, pode ocorrer desprendimento de vórtices.

Os efeitos dinâmicos do desprendimento de vórtices podem ser aproximados por uma força estática lateral, atuando no terço
superior, aplicada no ponto de máximo deslocamento da forma modal considerada. A força estática equivalente por unidade de altura, FL, é dada por:

$$ F_L = \frac{C1}{\sqrt{\lambda}\sqrt{\beta- C_2 \frac{\rho D^2}{M}}}q_HD $$

Onde $\beta$ é o amortecimento em razão do crítico (0.010), $\lambda$ é a relação H/D, H é a altura da estrutura, $q_H$ é a pressão dinâmica correspondente a velocidade crítica( $q_H= 0.6*V_C^2$), M é a massa por unidade de altura do terço superior e $\rho$ é a densidade do ar(1.2 kg/m^3). Para a maioria dos casos:

$$C_1 = \frac{3\sqrt{\lambda}}{4}$$

Os cálculos abaixo são referentes a determinação da força transversal equivalente que atuará na estrutura.

```python
rho = 1.2
lam = 60/5
C1 = 3*lam**0.5/4
C2 = 0.6

M_3 = ((60*mi_F)/3 + m_L)/(60/3)

qh1 = 0.6 * v1**2
qh2 = 0.6 * v2**2

FL1 = C1/((lam**0.5) * (0.01-C2*1.2*5**2/M_3)**0.5) * qh1 *5
FL2 = C1/((lam**0.5) * (0.01- C2*1.2*10**2/M_3)**0.5) * qh2 *10

print('  λ:                          {0:5.3f}'.format(lam),'\n',
      ' C1:                          {0:5.3f}'.format(C1),'\n',
      ' C2:                          {0:5.3f}'.format(C2),'\n',
      ' Massa do terço final:        {0:5.3f}kg/m'.format(M_3),'\n',
      ' Pressão dinâmica no Fuste:   {0:5.3f}N/m²'.format(qh1),'\n',
      ' Pressão dinâmica no Kevlar:  {0:5.3f}N/m²'.format(qh2),'\n',
      ' Força lateral no fuste:      {0:5.3f}kN/m'.format(FL1/1000),'\n',
      ' Força lateral no Kevlar:     {0:5.3f}kN/m'.format(FL2/1000))

```

      λ:                          12.000
      C1:                          2.598
      C2:                          0.600
      Massa do terço final:        17357.299kg/m
      Pressão dinâmica no Fuste:   143.356N/m²
      Pressão dinâmica no Kevlar:  573.425N/m²
      Força lateral no fuste:      5.678kN/m
      Força lateral no Kevlar:     56.220kN/m

Se a seguinte relação for satisfeita, não haverá  oscilações acima de um diâmetro:
$$ \beta < C2\frac{\rho D^2}{M} $$

```python
beta = 0.01
X1 = C2*rho*5**2/M_3
X2 = C2*rho*10**2/M_3

print(' beta:         {0:5.3f}'.format(beta),'\n',
      'Fuste:        {0:5.7f}'.format(X1),'\n',
      'Kevlar:       {0:5.7f}'.format(X2))

```

     beta:         0.010
     Fuste:        0.0010370
     Kevlar:       0.0041481

Portanto, não ocorrerá deslocamentos superiores a um diâmetro.

O cálculo dos esforços na base estão expostos a seguir:

```python
V1 = FL1 * 60
V2 = FL2 * 16
V = V1 + V2

M = V * 50

print(' Força adicional na base:                {0:5.3f}kN'.format(V/1000),'\n',
      'Momento fletor adicional na base:        {0:5.3f}kN.m'.format(M/1000))

```

     Força adicional na base:                1240.218kN
     Momento fletor adicional na base:        62010.905kN.m

```python

```