## Subsurface Data Analytics

## Artificial Neural Networks for Prediction in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

#### Honggeun Jo, Graduate Student, The University of Texas at Austin

##### [LinkedIn](https://www.linkedin.com/in/honggeun-jo/?originalSubdomain=kr) | [Twitter](https://twitter.com/HonggeunJ)

### PGE 383 Exercise: Artificial Neural Networks for Subsurface Modeling in Python

Here's a simple workflow, demonstration of artificial neural networks for subsurface modeling workflows. This should help you get started with building subsurface models that use data analytics and machine learning. Here's some basic details about neural networks.

#### Neural Networks

Machine learning method for supervised learning for classification and regression analysis.  Here are some key aspects of support vector machines.

**Basic Design** *"...a computing system made up of a number of simple, highly interconnected processing elements, which process information by their dynamic state response to external inputs."* Caudill (1989).

**Nature-inspire Computing** based on the neuronal structure in the brain, including many interconnected simple, processing units, known as nodes that are capable of complicated emergent pattern detection due to a large number of nodes and interconnectivity.

**Training and Testing** just like and other predictive model (e.g. linear regression, decision trees and support vector machines) we perform training to fit parameters and testing to tune hyperparameters.  Here we observe the error with training and testing datasets, but do not demonstrate tuning of the hyperparameters.

**Parameters** are the weights applied to each connection and a bias term applied to each node.  For a single node in an artificial neural network, this includes the slope terms, $\beta_i$, and the bias term, $\beta_{0}$.

\begin{equation}
Y = \sum_{i=1}^m \beta_i X + \beta_0
\end{equation}

it can be seen that the number of parameters increases rapidly as we increase the number of nodes and the connectivity between the nodes.

**Layers** the typical artificial neural net is structured with an **input layer**, with one node for each $m$ predictor feature, $X_1,\ldots,X_m$. There is an **ouput layer**, with one node for each $r$ response feature, $Y_1,\ldots,Y_r$.  There may be one or more layers of nodes between the input and output layers, known as **hidden layer(s)**.

**Connections** are the linkages between the nodes in adjacent layers.  For example, in a fully connected artificial neural network, all the input nodes are connected to all of the nodes in the first layer, then all of the nodes in the first layer are connected to the next layer and so forth. Each connection includes a weight parameter as indicated above.

**Nodes** receive the weighted signal from the connected previous layer nodes, sum and then apply this result to the **activation** function in the node. Some example activation functions include:

* **Binary** the node fires or not.  This is represented by a Heaviside step function.

* **Identify** the input is passed to the output $f(x) = x$

* **Linear** the node passes a signal that increases linearly with the weighted input.

* **Logistic** also known as sigmoid or soft step $f(x) = \frac{1}{1+e^{-x}}$

the node output is the nonlinear activiation function applied to the linearly weighted inputs. This is fed to all nodes in the next layer.

**Training Cycles** - the presentation of a batch of data, forward application of the current prediction model to make estimates, calculation of error and then backpropagation of error to correct the artificial neural network parameters to reduce the error over all of the batches.

**Batch** is the set of training data for each training cycle of forward prediction and back propagation of error, drawn to train for each iteration. There is a trade-off, a larger batch results in more computational time per iteration, but a more accurate estimate of the error to adjust the weights.  Smaller batches result in a nosier estimate of the error, but faster epochs, this results in faster learning and even possibly more robust models.

**Epochs** - is a set of training cycles, batches covering all available training data.

**Local Minimums** - if one calculated the error hypersurface over the range of model parameters it would be hyparabolic, there is a global minimium error solution.  But this error hyper surface is rough and it is possible to be stuck in a local minimum. **Learning Rate** and **Mommentum** coefficients are introduced to avoid getting stuck in local minimums.

* **Mommentum** is a hyperparameter to control the use of information from the weight update over the last epoch for consideration in the current epoch.  This can be accomplished with an update vector, $v_i$, a mommentum parameter, $\alpha$, to calculate the current weight update, $v_{i+1}$ given the new update $\theta_{i+1}$.

\begin{equation}
v_{i+1} = \alpha v_i + \theta_{i+1}
\end{equation}

* **Learning Rate** is a hyperparameter that controls the adjustment of the weights in response to the gradient indicated by backpropagation of error

##### Applications to subsurface modeling

We demonstrate the estimation of normal score transformed porosity from depth.  This would be useful for building a vertical trend model.

* modeling the complicated relationship between porosity and depth.

#### Limitations of Neural Network Estimation

Since we demonstrate the use of an artificial neural network to estimate porosity from sparsely sampled data over depth, we should comment on limitations of our artificial neural networks for this estimation problem:

* does not honor the well data

* does not honor the histogram of the data

* does not honor spatial correlation

* does not honor the multivariate relationship

* generally low interpretability models

* requires a large number of data for effective training

* high model complexity with high model variance

#### Workflow Goals

Learn the basics of machine learning in python to predict subsurface features. This includes:

* Loading and visualizing sample data
* Trying out neural nets

#### Objective

In the PGE 383: Subsurface Machine Learning Class, I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - 12_sample_data.csv found [here](https://github.com/GeostatsGuy/GeoDataSets/blob/master/12_sample_data.csv).

There are exampled below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

#### Import Required Packages

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import geostatspy.GSLIB as GSLIB          # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats    # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # for plotting
import seaborn as sns                     # for plotting
import warnings                           # supress warnings from seaborn pairplot
from sklearn.model_selection import train_test_split # train / test DatFrame split
```

We will also need the following packages to train and test our artificial neural nets:

* Tensorflow - open source machine learning

* Keras - high level application programing interface (API) to build and train models

More information is available at [tensorflow install](https://www.tensorflow.org/install).

* This workflow was designed with tensorflow version 1.13.1 and does not work with version 2.0.0 alpha.

To check your current version of tensorflow you could run the next block of code.

```python
import tensorflow as tf
tf.__version__                            # check the installed version of tensorflow
```

    '1.13.1'

Let's import all of the tensorflow and keras methods that we will need in our workflow.

```python
import tensorflow as tf
from keras.models import Sequential, load_model
from keras.layers.core import Dense, Dropout, Activation
from keras.utils import np_utils
import keras
from tensorflow.python.keras import backend as k
```

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("C:/PGE383")
```

#### Loading Data
Let's load the provided multivariate, spatial dataset '12_sample_data.csv'.  It is a comma delimited file with:

* Depth ($m$)
* Normal Score Porosity

It is common to transform properties to standard normal for geostatistical workflows.

We load it with the pandas 'read_csv' function into a data frame we called 'df' and then preview it to make sure it loaded correctly.

**Python Tip: using functions from a package** just type the label for the package that we declared at the beginning:

```python
import pandas as pd
```

so we can access the pandas function 'read_csv' with the command:

```python
pd.read_csv()
```

but read csv has required input parameters. The essential one is the name of the file. For our circumstance all the other default parameters are fine. If you want to see all the possible parameters for this function, just go to the docs [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html).

* The docs are always helpful
* There is often a lot of flexibility for Python functions, possible through using various inputs parameters

also, the program has an output, a pandas DataFrame loaded from the data.  So we have to specficy the name / variable representing that new object.

```python
df = pd.read_csv("1D_Porosity.csv")
```

Let's run this command to load the data and then look at the resulting DataFrame to ensure that we loaded it.

```python
df2 = pd.read_csv("1D_Porosity.csv")                    # read a .csv file in as a DataFrame                                     # display first 4 samples in the table as a preview
df2.head()                                                   # we could also use this command for a table preview
```

      Depth
      Nporosity

      0
      0.25
      -1.37

      1
      0.50
      -2.08

      2
      0.75
      -1.67

      3
      1.00
      -1.16

      4
      1.25
      -0.24

#### Data Normalization

We must normalize the features before we apply them in an artificial neural network model. These are the motivations for this normalization:

* remove impact of scale of different types of data (i.e., depth varies between $[0,10]$, but porosity only varies between $[-3,3.0]$).

* activation functions in artificial neural networks are designed to be more sensive to values of nodes closer to 0.0 (i.e., results in higher gradient and improves backpropagation in training)

Let's normalize each feature.

* We apply the min max normalization by-hand to force both the predictor and response features to be bound $[-1,1]$.

* It is easy to backtransform given we keep track of the original min and max values

```python
depth_min = df2['Depth'].values.min(); depth_max = df2['Depth'].values.max()
Npor_min = df2['Nporosity'].values.min(); Npor_max = df2['Nporosity'].values.max()
df2['norm_Depth'] = (df2['Depth'] - depth_min)/(depth_max - depth_min) * 2 - 1
df2['norm_Nporosity'] = (df2['Nporosity'] - Npor_min)/(Npor_max - Npor_min) * 2 - 1
df2.head()
```

      Depth
      Nporosity
      norm_Depth
      norm_Nporosity

      0
      0.25
      -1.37
      -1.000000
      -0.679458

      1
      0.50
      -2.08
      -0.948718
      -1.000000

      2
      0.75
      -1.67
      -0.897436
      -0.814898

      3
      1.00
      -1.16
      -0.846154
      -0.584650

      4
      1.25
      -0.24
      -0.794872
      -0.169300

It is also a good idea to check the summary statistics.

* All normalized features should now range from -1.0 to 1.0

```python
df2.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Depth
      40.0
      5.125000e+00
      2.922613
      0.25
      2.687500
      5.125000e+00
      7.562500
      10.00

      Nporosity
      40.0
      2.225000e-02
      0.992111
      -2.08
      -0.757500
      1.400000e-01
      0.742500
      2.35

      norm_Depth
      40.0
      -5.551115e-18
      0.599510
      -1.00
      -0.500000
      -5.551115e-17
      0.500000
      1.00

      norm_Nporosity
      40.0
      -5.090293e-02
      0.447906
      -1.00
      -0.402935
      2.257336e-03
      0.274266
      1.00

#### Separation of Training and Testing Data

We also need to split our data into training / testing datasets so that we:

* can train our artificial neural networks using the training data

* while testing their performance with the withheld testing (validation) data.

```python
X2 = df2.iloc[:,[0,2]]                        # extract the predictor feature - acoustic impedance
y2 = df2.iloc[:,[1,3]]                        # extract the response feature - porosity
X2_train, X2_test, y2_train, y2_test = train_test_split(X2, y2, test_size=0.2, random_state=73073)
```

#### Visualize the Dataset

Let's visualize the train and test data split on a single scatter plot.

* we want to make sure it is fair

* ensure that the test samples are not clustered or too far away from the training data.

We will look at the original data and normalized data, the input to the neural network.

```python
plt.subplot(211)
plt.plot(X2_train['Depth'].values,y2_train['Nporosity'].values, 'o', markerfacecolor='red', markeredgecolor='black', alpha=0.4, label = "Train")
plt.plot(X2_test['Depth'].values,y2_test['Nporosity'].values, 'o', markerfacecolor='blue', markeredgecolor='black', alpha=0.4, label = "Test")
plt.title('Standard Normal Porosity vs. Depth')
plt.xlabel('Z (m)')
plt.ylabel('NPorosity')
plt.legend()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)

plt.subplot(212)
plt.plot(X2_train['norm_Depth'].values,y2_train['norm_Nporosity'].values, 'o', markerfacecolor='red', markeredgecolor='black', alpha=0.4, label = "Train")
plt.plot(X2_test['norm_Depth'].values,y2_test['norm_Nporosity'].values, 'o', markerfacecolor='blue', markeredgecolor='black', alpha=0.4, label = "Test")
plt.title('Normalized Standard Normal Porosity vs. Normalized Depth')
plt.xlabel('Normalized Z')
plt.ylabel('Normalized NPorosity')
plt.legend()

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.5, wspace=0.2, hspace=0.3)
plt.show()
```

#### Specify the Prediction Locations

Given this training and testing data, let's specify the prediction locations over the range of the observed depths at regularly spaced $nbins$ locations.

```python
# Specify the prediction locations
nbins = 1000
depth_bins = np.linspace(depth_min, depth_max, nbins) # set the bins for prediction
norm_depth_bins = (depth_bins-depth_min)/(depth_max-depth_min)*2-1 # use normalized bins
```

#### Build and Train a Simple Neural Network

For our first model we will build a simple model with:

* 1 predictor feature - depth ($d$)

* 1 responce feature - normal score porosity ($N\{\phi\}$)

we will build a model to predict normal score porosity from depth over all locations in our model $\bf{u} \in AOI$.

\begin{equation}
N\{\phi(\bf{u})\} = \hat{f} (d(\bf{u}))
\end{equation}

and use this model to support the prediction of porosity between the wells.

#### Design, Train and Test a Neural Network

In the following code we use keras / tensorflow to:

1. **Design the Network** - we use a fully connected, feed forward neural network with one node in the input and output layers, to receive the normalized depth and output the normalized (normal score) porosity.  We found by trial and error, given the complexity of the dataset, that we required a significant network width (about 500 nodes) and a network depth of atleast 4 hidden layers.

2. **Select the Optimizer** - we selected the adam optimizer (Kingma and Ba, 2015).  This optimizer is computationally efficient and is well suited to problems with noisy data and sparse gradients. It is an extension of stochastic gradient descent with the addition of adaptive gradient algorithm that calculates per-parameter learning rates to improve learning with sparse gradients, and root mean square propopagation that sets the learning rate based on the recent magnitudes of the gradients for each parameter to improve performance with noisy data.  We include stochastic gradient descent for experimentation.

    * we found a learning rate of 0.01 to 0.001 works well
    * we found the rate of decay parameters of $\beta_1=0.9$ and $\beta_2=0.999$ performed well

3. **Compile the Machine** - specify the optimizer, loss function and the metric for model training.

4. **Train the Network** - fit / train the model parameters over a specified number of batch size within a specified number of epochs.  We specify the train and test normalized predictor and response features.

Then we visualize the model in the original units.

```python
# Design the neural network
model_2 = Sequential([
    Dense(1, activation='linear', input_shape=(1,)),                  # input layer
    Dense(500, activation='relu'),
    Dense(500, activation='relu'),
    Dense(500, activation='relu'),
#    Dense(100, activation='relu'),                                    # uncomment these to add hidden layers
#    Dense(100, activation='relu'),
#    Dense(100, activation='relu'),
#    Dense(100, activation='relu'),
    Dense(1, activation='linear'),                                     # output layer
])

# Select the Optimizer
adam = keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False) # adam optimizer
#sgd = keras.optimizers.SGD(lr=0.001, momentum=0.0, decay = 0.0, nesterov=False)  # stochastic gradient descent

# Compile the Machine
model_2.compile(optimizer=adam,loss='mse',metrics=['accuracy'])

# Train the Network
hist_2 = model_2.fit(X2_train['norm_Depth'], y2_train['norm_Nporosity'],
    batch_size=5, epochs=1000,
    validation_data=(X2_test['norm_Depth'], y2_test['norm_Nporosity']),verbose = 0)

# Predict with the Network
pred_norm_Nporsity = model_2.predict(np.array(norm_depth_bins)) # predict with our ANN
pred_Nporosity = ((pred_norm_Nporsity + 1)/2*(Npor_max - Npor_min)+Npor_min)

# Plot the Model Predictions
plt.subplot(1,1,1)
plt.plot(depth_bins,pred_Nporosity,'black',linewidth=2)
plt.plot(X2_train['Depth'].values,y2_train['Nporosity'].values, 'o', markerfacecolor='red', markeredgecolor='black', alpha=1.0, label = "Train")
plt.plot(X2_test['Depth'].values,y2_test['Nporosity'].values, 'o', markerfacecolor='blue', markeredgecolor='black', alpha=1.0, label = "Test")
plt.xlabel('Depth (m)')
plt.ylabel('Porosity (faction)')
plt.legend()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=0.8, wspace=0.2, hspace=0.2)
plt.show()
```

#### Evaluation of the Model

For my specified artificial neural network design and optimization parameters I have a very flexible model to fit the data.

* artificial neural networks live up to their designation as **Universal Function Approximators**

Let's check the training curve, loss functions for our model over training and testing datasets.

* **square loss** ($L_2$ loss) is the:

\begin{equation}
L_2 = \sum_{\bf{u}_{\alpha} \in AOI} \left(y(\bf{u}_{\alpha}) - \hat{f}(x_1(\bf{u}_{\alpha}),\ldots,x_m(\bf{u}_{\alpha})\right)
\end{equation}

* this is a measure of the inaccuracy over the available data

We can see the progress of the model over epochs in reduction of training and testing error.

* we can observe that the model matches the training data after about 200 epochs, but continues to improve up the 1,000 epochs

```python
# Plot the Loss vs. Training Epoch
plt.subplot(1,1,1)
plt.plot(hist_2.history['loss'])
plt.plot(hist_2.history['val_loss'])
plt.title('Loss function of Artificial Neural Net Example #2')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper right')
plt.ylim(0,0.4)
plt.grid()
plt.tight_layout()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=0.8, wspace=0.2, hspace=0.2)
plt.show()
```

#### Some Observations

We performed a set of experiments and made the following observations that may help you experiment with the design and training of this artificial neural network. Each machine was trained over 1000 epochs with a batch size of 5.

For a 1 $\times$ 100 $\times$ 100 $\times$ 1 neural network:
* Learning Rate of 0.001 still converging at 1000 epochs, missing some features
* Learning Rate of 0.1 - stuck in a local minum as a step function

For a 1 $\times$ 500 $\times$ 500 $\times$ 1 neural network:
* Learning Rate of 0.01 - 0.001 for a close fit to training data
* Learning Rate of 0.1 - stuck in a local minimum as a line
* Learning Rate of $\le$ 0.0001 still converging at 1000 epochs, missing some features

For a 1 $\times$ 500 $\times$ 500 $\times$ 500 $\times$ 1 neural network:
* Learning Rate of 0.01 - 0.001 for a close fit to training data
* Learning Rate of 0.1 - stuck as a line
* Learning Rate of $\le$ 0.0001 still converging at 1000 epochs, missing some features

#### Visualizing the Neural Net

There are some methods available to interogate the artificial neural net.

* neural net summary

* weights

Here's the summary from our neural net.  It lists by layers the number of nodes and number of parameters.

```python
model_2.summary()                                      # artificial neural network design and number of parameters
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_535 (Dense)            (None, 1)                 2
    _________________________________________________________________
    dense_536 (Dense)            (None, 500)               1000
    _________________________________________________________________
    dense_537 (Dense)            (None, 500)               250500
    _________________________________________________________________
    dense_538 (Dense)            (None, 500)               250500
    _________________________________________________________________
    dense_539 (Dense)            (None, 1)                 501
    =================================================================
    Total params: 502,503
    Trainable params: 502,503
    Non-trainable params: 0
    _________________________________________________________________

We can also see the actual trained weights for each node in each layer.

```python
for layer in model_2.layers:                            # weights for the trained artificial neural network
    g = layer.get_config()
    h = layer.get_weights()
    print(g)
    print(h)
    print('\n')
```

    {'name': 'dense_535', 'trainable': True, 'batch_input_shape': (None, 1), 'dtype': 'float32', 'units': 1, 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}
    [array([[-1.324188]], dtype=float32), array([-0.01764077], dtype=float32)]

    {'name': 'dense_536', 'trainable': True, 'units': 500, 'activation': 'relu', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}
    [array([[ 4.37869839e-02,  2.84792539e-02, -5.04958704e-02,
             1.33315518e-01,  3.78291379e-03, -1.06391739e-02,
             9.11150649e-02, -1.02796906e-03, -1.50301322e-01,
            -6.21361472e-02, -2.31712814e-02,  4.74340748e-03,
             8.10861066e-02, -8.89362991e-02,  2.98289418e-01,
             4.41729911e-02,  4.47279736e-02,  2.22163871e-02,
             4.29757684e-02,  6.64823875e-02,  3.37973014e-02,
             2.44364385e-02, -6.22198582e-02,  8.36473256e-02,
            -1.49021158e-02,  8.82090256e-02,  3.54946870e-03,
            -1.29045490e-02,  8.74734744e-02, -1.06779337e-02,
            -5.43587096e-02,  1.33621832e-03,  1.16167612e-01,
            -6.61311031e-04, -9.01760723e-06, -4.26420793e-02,
             7.38960207e-02,  3.16858850e-02,  1.09584540e-01,
             7.82321021e-02,  3.61922011e-02, -1.04301557e-01,
             8.85869563e-02,  9.70736817e-02, -5.80134802e-03,
             8.00451189e-02, -1.30036043e-03,  3.91601166e-03,
             1.01504810e-01,  3.23649906e-02,  5.36048077e-02,
            -8.75119120e-02,  7.06093982e-02,  9.37056392e-02,
             9.51493159e-02, -3.00738541e-03,  7.92166516e-02,
            -4.90556529e-04, -1.29743107e-02, -7.81329200e-02,
             1.32052168e-01, -8.99229273e-02, -6.77816384e-03,
            -1.38963714e-01,  4.36128713e-02, -6.23008721e-02,
            -6.70676082e-02, -1.58895971e-04,  4.52702381e-02,
             6.76621273e-02, -1.48024713e-03,  1.09858453e-01,
            -9.12376912e-04,  6.73749018e-03,  5.63484197e-03,
            -3.38859251e-03,  1.16737701e-01, -7.53363296e-02,
             1.29707232e-01, -1.08739913e-01, -9.22356173e-03,
             1.16427720e-01, -7.81598166e-02, -6.16153656e-03,
             8.80591869e-02, -6.53325766e-02,  3.85513045e-02,
             1.08171615e-03,  9.70692188e-02, -4.51390678e-03,
            -1.91461872e-02, -9.73084494e-02, -2.10419614e-02,
            -8.55254233e-02,  1.29724205e-01, -1.06441669e-01,
             4.56053168e-02, -7.28177875e-02, -7.60430004e-04,
             5.38264588e-02, -1.00965656e-01, -6.52337819e-02,
             1.06089935e-02, -7.60561451e-02, -6.80339411e-02,
            -6.55979514e-02,  1.01904668e-01,  4.16511856e-03,
             4.18214388e-02,  3.36837284e-02, -6.89377934e-02,
            -3.13841109e-03,  7.37283900e-02,  1.30472451e-01,
             3.95321772e-02, -6.40012696e-02, -2.65501030e-02,
             1.35376722e-01,  5.19507155e-02,  3.28973262e-03,
             9.72523764e-02,  6.78595230e-02, -7.61842132e-02,
             5.09470887e-02, -5.81806116e-02,  8.02602246e-02,
            -1.05615310e-01, -7.84106031e-02, -8.20061564e-03,
            -2.09269673e-02, -6.52782246e-02, -3.76430005e-02,
            -4.43258230e-03, -4.43456182e-03, -7.33490521e-03,
             4.39224727e-02,  7.80225322e-02, -8.18999037e-02,
            -3.16527747e-02,  1.35007845e-02, -1.23362601e-01,
             6.97992444e-02, -4.31013806e-03,  2.70868577e-02,
             7.12058842e-02, -1.16116460e-03, -4.60769923e-04,
            -4.54251692e-02,  6.35423064e-02, -2.99977348e-03,
            -1.34880468e-01,  6.31999224e-02, -6.48167804e-02,
            -6.69402704e-02,  6.27936516e-03, -3.10638584e-02,
            -9.25559103e-02, -1.11442834e-01,  3.28482166e-02,
            -1.14701122e-01, -5.21795750e-02, -6.13903143e-02,
            -7.53833354e-02, -3.98605177e-03,  3.29084210e-02,
            -1.16867386e-02, -5.33676371e-02, -4.76962626e-02,
            -1.55353293e-01,  7.93459788e-02,  4.12472570e-03,
            -6.63398355e-02,  4.17984463e-02, -4.13348991e-03,
            -3.65724275e-03,  2.94227749e-02, -6.86958060e-02,
             8.13451335e-02, -1.81635027e-03,  5.97945638e-02,
            -7.60767385e-02, -5.49900644e-02,  1.34785116e-01,
            -1.07490271e-01,  1.29407361e-01, -1.05475530e-03,
             4.20478433e-02, -8.41621831e-02,  9.80847478e-02,
             7.03405812e-02,  8.84528086e-02,  2.63709179e-03,
             9.27885473e-02, -1.14496797e-01,  5.28439842e-02,
            -3.86079773e-02,  4.00377586e-02,  3.54537810e-03,
            -7.69117326e-02, -1.08731799e-02,  6.50035217e-02,
             6.63219616e-02, -4.19028290e-02,  5.35250939e-02,
             8.24268088e-02,  4.13128324e-02,  5.90596795e-02,
            -2.60966900e-03, -6.19906597e-02, -4.01285430e-03,
            -4.59489822e-02,  1.15634732e-01,  9.54596475e-02,
             3.70567292e-02, -6.69994131e-02,  3.28783505e-02,
            -7.60106593e-02, -9.85587314e-02, -1.66328298e-03,
            -5.84144657e-03, -1.33208230e-01, -9.32852104e-02,
             8.55539441e-02,  4.49849386e-03,  4.29949239e-02,
             3.05023566e-02, -9.41249058e-02, -5.18078580e-02,
             9.61371660e-02,  8.67340341e-02, -2.08380865e-03,
            -1.32435605e-01,  3.22007686e-02, -9.46334153e-02,
             1.68362688e-02, -1.11437447e-01, -7.92395920e-02,
            -7.11047575e-02, -2.16089971e-02, -1.10117979e-01,
             6.64046081e-03,  8.86755735e-02, -3.29512381e-03,
            -7.75940418e-02, -1.31690934e-01,  6.67404830e-02,
             8.91118497e-03, -1.77239031e-02, -8.86819586e-02,
             6.95541948e-02,  6.89079836e-02, -1.05990872e-01,
            -6.73175603e-03, -6.64992407e-02,  9.71645042e-02,
             1.55092170e-03, -1.54509470e-01, -7.50039071e-02,
             1.23657798e-02, -1.34944007e-01, -6.35182112e-02,
             1.14698768e-01, -9.12414715e-02, -1.27469366e-02,
            -1.27320290e-01, -6.35384917e-02,  8.69745612e-02,
            -1.01650916e-01, -7.39746392e-02, -1.11858472e-01,
            -4.81140334e-03,  5.06917983e-02, -1.02960981e-01,
            -6.23498410e-02, -1.90573558e-03, -7.14417920e-02,
            -8.64325166e-02,  2.62962803e-02,  7.69598118e-04,
             7.82793504e-04,  8.48663449e-02,  5.03086671e-03,
            -9.86906141e-02, -6.83365166e-02, -8.57298437e-04,
            -7.14763552e-02, -2.47936677e-02,  4.49170657e-02,
             2.82813385e-02,  6.86316267e-02,  2.87176669e-02,
            -6.49383990e-04,  1.10460266e-01,  4.37190309e-02,
            -1.34818971e-01,  2.25369576e-02,  7.59827569e-02,
             8.77600238e-02,  6.10029958e-02,  4.76534851e-02,
            -1.09301433e-01,  9.92182493e-02,  9.33573022e-02,
            -1.11727640e-01,  7.03948066e-02,  6.28180876e-02,
            -1.55644551e-01,  3.15748453e-02,  3.85251343e-02,
             7.26150498e-02, -2.99699232e-02, -1.16008695e-03,
             9.00451466e-02,  3.72605994e-02, -7.57500231e-02,
             1.35603398e-01,  3.50316837e-02,  1.13473386e-01,
            -7.59589225e-02, -6.93566352e-02, -9.32576507e-02,
            -5.85489348e-02, -8.36537704e-02,  6.00345293e-03,
            -8.75586644e-02,  1.75075140e-02,  5.72374538e-02,
            -1.57084502e-02,  3.77423353e-02, -1.36674359e-03,
             3.36039974e-03, -1.06565133e-01, -1.29231559e-02,
            -6.47024736e-02, -5.83822392e-02,  9.99464691e-02,
            -8.59773308e-02, -2.25808448e-03,  1.04079396e-01,
            -3.04516940e-03, -1.45849600e-01, -8.17036256e-02,
            -1.57709438e-02,  1.06820256e-01, -4.14584093e-02,
            -6.64309459e-03, -9.44048446e-03,  8.36263970e-03,
             1.09452441e-01,  7.22158607e-03, -6.70162402e-03,
             1.76369622e-02, -4.00551409e-02,  3.14997211e-02,
            -6.75998926e-02,  5.34251817e-02, -6.50570542e-02,
            -5.72608598e-02, -2.12053079e-02, -6.81426972e-02,
             1.92111805e-02, -2.33714134e-02, -2.11180479e-04,
             1.36419702e-02,  6.11771382e-02, -5.63817704e-03,
             5.86114563e-02,  8.18122774e-02,  8.32198001e-03,
            -8.27094913e-02,  4.07909937e-02,  8.27760771e-02,
             6.55070022e-02, -1.61201786e-02, -8.37071761e-02,
            -1.21987864e-01,  1.02040082e-01,  6.02825582e-02,
            -6.75648451e-02,  7.14957565e-02,  2.08943477e-03,
             4.29181755e-02, -2.25497922e-03,  6.37466386e-02,
            -6.50759786e-02, -9.48397517e-02,  4.51110490e-02,
             1.01239242e-01, -8.45591426e-02,  3.51027623e-02,
            -7.04090968e-02, -6.20335378e-02,  5.60532324e-02,
            -6.05542287e-02,  8.46762434e-02,  1.10914316e-02,
             6.13659620e-02,  2.12732601e-04,  3.68024129e-03,
            -6.93032220e-02, -2.75107697e-02, -1.10175148e-01,
            -2.30825553e-03, -8.83723274e-02, -5.68870977e-02,
            -1.47443516e-02,  5.29175289e-02,  1.02081917e-01,
             1.03125438e-01, -5.90750249e-03, -1.71697848e-02,
             4.59938950e-04, -9.67270415e-03,  1.13146283e-01,
             6.02231696e-02, -1.22713171e-01, -1.56251043e-01,
             9.25789773e-02,  3.61172482e-02, -6.59704860e-03,
             6.58418387e-02, -1.08241461e-01,  4.66979221e-02,
             1.89785305e-02,  1.06163993e-01, -7.98181817e-03,
            -4.21184935e-02, -6.92812204e-02, -8.86960998e-02,
            -6.53993636e-02, -7.85087645e-02, -8.98909271e-02,
             9.31996405e-02,  3.45884226e-02,  8.91835988e-02,
             6.24386482e-02, -4.41202847e-03,  6.15031570e-02,
            -7.02832565e-02,  1.05966171e-02, -1.43899932e-03,
            -7.76754320e-02,  1.02125727e-01,  8.34097266e-02,
            -1.43925697e-01,  8.42536539e-02,  1.04465811e-02,
             2.24737227e-02,  1.08246863e-01, -7.89064392e-02,
            -9.83206779e-02, -1.22306362e-01,  4.55920026e-02,
             1.05681658e-01, -2.09155977e-02,  3.43536325e-02,
            -8.47020000e-02,  5.27732074e-02,  5.98774552e-02,
            -1.04057558e-01, -1.83242336e-02,  7.08967596e-02,
            -6.99171498e-02, -6.37240848e-03, -2.10936777e-02,
             5.89776859e-02,  8.36545303e-02, -1.14212818e-01,
            -1.50420722e-02,  7.64402226e-02,  7.43056163e-02,
             1.21164434e-01, -1.32907426e-03, -4.82582068e-03,
             2.17562038e-02,  2.86421478e-02, -5.22685573e-02,
            -8.04568008e-02,  1.20499551e-01, -1.01798251e-01,
             1.02965400e-01, -9.50997025e-02, -2.22027255e-03,
             7.36168995e-02,  1.10508129e-03, -3.12446966e-03,
             3.03575695e-02, -1.19504379e-02,  6.95811212e-02,
            -6.69124594e-04,  8.70308727e-02,  3.32550704e-02,
            -7.08441362e-02,  4.81908470e-02,  8.01099166e-02,
             1.68886909e-03,  9.67642516e-02, -5.53571247e-02,
             2.30133021e-03,  6.30277395e-02]], dtype=float32), array([-0.06139961, -0.03823963, -0.04796872, -0.05715915, -0.00778893,
           -0.01715333, -0.07307467, -0.00414138, -0.02369456, -0.08908577,
           -0.03416378, -0.01094732, -0.08148513, -0.06651676, -0.04846705,
           -0.06414326, -0.06192571, -0.03751549, -0.0619099 , -0.07249781,
           -0.05136329,  0.01927103, -0.05907007, -0.08442453, -0.02177054,
           -0.06876581, -0.00651649, -0.02827632, -0.06884521, -0.01586316,
           -0.05935439, -0.01289208, -0.05038578, -0.00867271, -0.00318428,
            0.0086913 , -0.08008435, -0.0433035 , -0.04697267, -0.07393831,
           -0.05754452, -0.07528708, -0.06741227, -0.076387  ,  0.03964805,
           -0.07952945, -0.005366  ,  0.02763389, -0.07387943, -0.0549034 ,
           -0.07615414, -0.08007966, -0.08798408, -0.07791902, -0.07160194,
           -0.00560842, -0.08591481, -0.00418956, -0.02050515, -0.05918604,
           -0.07379995, -0.06774189, -0.01120751, -0.01974001,  0.06067615,
           -0.05945587, -0.05904902,  0.00732201, -0.06428739, -0.07017349,
           -0.0086476 , -0.0485802 , -0.00730538, -0.00957363, -0.02154209,
           -0.01012385, -0.0503656 , -0.07115868, -0.06906632,  0.00565668,
           -0.01403928, -0.07305516, -0.07933737, -0.00866085, -0.0698074 ,
           -0.07289582, -0.05830758, -0.00272358, -0.07188021, -0.01362477,
           -0.02701662, -0.07330002, -0.02126301, -0.08914766, -0.08186267,
           -0.07523412, -0.06335444, -0.06751571, -0.00684798, -0.07694793,
           -0.07039765, -0.06945851, -0.02247573, -0.07047946, -0.06347238,
           -0.04554536, -0.04426727, -0.0142412 , -0.05859767, -0.05376985,
           -0.07195932, -0.00791719, -0.07501336, -0.05634447, -0.0550543 ,
           -0.0462756 , -0.04563486, -0.05884227, -0.07370295, -0.00788396,
           -0.04158321, -0.0841307 , -0.07055681, -0.0639838 , -0.05372795,
           -0.08102303, -0.07511374, -0.05606232, -0.02532341, -0.02959966,
           -0.06019864, -0.05242248, -0.0084167 , -0.00598817, -0.01137267,
           -0.06449889, -0.08436804, -0.07252833, -0.0444572 , -0.01972921,
           -0.04350067, -0.08468529, -0.00869756, -0.04184554, -0.08907367,
           -0.0059012 , -0.00231363, -0.06242609, -0.07555081,  0.0111065 ,
           -0.00878023, -0.06532616, -0.06585679, -0.05300143, -0.01283845,
            0.05271176, -0.0663313 , -0.07893655, -0.05373202, -0.08252703,
           -0.07084762, -0.06485971, -0.07689745, -0.01427657, -0.05416669,
           -0.02175184, -0.0562396 , -0.06432167, -0.02325281, -0.06684574,
           -0.00705054, -0.06296429, -0.05777869, -0.00748384, -0.00635496,
           -0.04159978, -0.06685322, -0.08389334, -0.00701761, -0.06432831,
           -0.07090707, -0.06264   , -0.08206632, -0.07564244, -0.05886821,
           -0.0063959 , -0.05845111, -0.06204287, -0.04233636, -0.07327709,
           -0.07834864, -0.00707027, -0.09887744,  0.00626668, -0.06312662,
           -0.02920069, -0.05897494, -0.00603426, -0.05829883, -0.02059165,
           -0.07834075, -0.06683242, -0.05705149, -0.0762232 , -0.08900511,
           -0.05739731, -0.07109588, -0.00592516, -0.05978489, -0.0183938 ,
           -0.05917713, -0.04931708, -0.07244904, -0.05095007, -0.06334067,
           -0.05363588, -0.06938779, -0.07172804, -0.00444791, -0.01391948,
           -0.03241966, -0.07023546, -0.09296511, -0.00707521, -0.06007748,
           -0.05099807, -0.08633383, -0.04671507, -0.07162772, -0.06880689,
           -0.00580111, -0.02292861, -0.05301842, -0.0699292 ,  0.06440304,
           -0.08127811, -0.08215243, -0.06553537, -0.02928443, -0.07802734,
           -0.01139031, -0.07959687,  0.01446193, -0.07216932,  0.00841058,
           -0.06963574, -0.01869061, -0.0314908 , -0.06402976, -0.08380255,
           -0.08380487, -0.07963423, -0.01089369, -0.06037915, -0.07224868,
           -0.00275239, -0.02820986, -0.07644273, -0.01820268, -0.02047534,
           -0.06006921, -0.04915273, -0.0652189 , -0.01797788, -0.00674913,
           -0.06562325, -0.09353849,  0.01572672, -0.06633594, -0.07833622,
           -0.00790852, -0.06969513, -0.07658083, -0.06411042, -0.00743337,
           -0.06299569, -0.06349686,  0.06481276, -0.01390679, -0.00704703,
           -0.09151021, -0.00897039, -0.07108843, -0.07116202, -0.0073884 ,
           -0.07649622, -0.03437965, -0.06293312, -0.04763817, -0.08573087,
           -0.04416954, -0.00300061, -0.07880595, -0.0616023 , -0.01828985,
           -0.0342025 , -0.07971473, -0.08888055, -0.07533889, -0.06609005,
           -0.07731427, -0.07258897, -0.07445047, -0.08132768, -0.06696977,
           -0.07822786, -0.05202796, -0.05250061, -0.05390753, -0.09042215,
           -0.02522271, -0.00187201, -0.08000803, -0.04903469, -0.06784555,
           -0.0589343 , -0.05476155, -0.05010034, -0.05756522, -0.06287936,
           -0.06581006, -0.06560832, -0.07855322, -0.00921494, -0.06341241,
           -0.03245432, -0.08061009, -0.02204132, -0.05269145, -0.00488693,
           -0.00603298,  0.00657092, -0.02193722, -0.06552897, -0.06532465,
           -0.07232498, -0.07811823,  0.02950191, -0.04523056, -0.0042063 ,
           -0.0276743 , -0.08286427, -0.0218486 , -0.04605348, -0.03906796,
           -0.0113944 , -0.0318488 , -0.01176713, -0.04728981, -0.01139557,
           -0.01032085, -0.02429256, -0.05450579, -0.04662224, -0.04982534,
           -0.07490676, -0.06650402, -0.06072304, -0.03004125, -0.06273857,
           -0.03457063, -0.00580207, -0.0016215 , -0.02143812, -0.07632421,
           -0.01378029, -0.07292905, -0.0883406 ,  0.03496349, -0.08809979,
           -0.05501696, -0.07620736, -0.0675406 , -0.0245028 , -0.06243804,
           -0.08766349, -0.06441812, -0.07074441, -0.07693358, -0.07767762,
           -0.00305608,  0.00857855, -0.01002404, -0.06823479, -0.05937157,
           -0.0650647 , -0.06624278, -0.04493498, -0.05777024, -0.04969624,
           -0.06784492, -0.04617   , -0.0673632 , -0.02127056, -0.06717695,
           -0.01669092, -0.0772848 , -0.00140384, -0.00521039, -0.07037175,
           -0.01954671, -0.08139505, -0.00493742, -0.06340863, -0.06051   ,
           -0.020957  , -0.07684536, -0.10320909, -0.07542219, -0.02146349,
           -0.02428293, -0.00908613, -0.02114126, -0.04844717, -0.07506659,
           -0.04155113, -0.02611309, -0.0735454 , -0.0500888 , -0.01164846,
           -0.07853115, -0.0765461 ,  0.06416114, -0.02982672, -0.0457687 ,
           -0.01188941, -0.06042792, -0.07090043, -0.0842066 , -0.06803164,
           -0.08185688, -0.06315702, -0.10026774, -0.049329  , -0.09501091,
           -0.07488233, -0.00600216, -0.07636529, -0.0779068 , -0.01502587,
           -0.00658086, -0.02787442, -0.08036617, -0.06611934,  0.00686911,
           -0.09045056, -0.01738705,  0.08067761, -0.04652357, -0.05989062,
           -0.07276973, -0.04131578, -0.06277292, -0.04539846, -0.03031358,
           -0.05429706, -0.06148601, -0.05069135, -0.06731895, -0.07334394,
           -0.02559559, -0.08864345, -0.07282297, -0.01017689, -0.0348955 ,
           -0.08250091, -0.06658982, -0.00376104, -0.02169728, -0.07390127,
           -0.07335062, -0.05235442, -0.00371226, -0.00705077,  0.0043261 ,
           -0.04688119, -0.05291606,  0.00359649, -0.06370793,  0.00477501,
           -0.04499554, -0.07048909, -0.00963487, -0.07721131, -0.00331341,
           -0.00476873, -0.04947318, -0.01665951, -0.08469487, -0.00331149,
           -0.06853346, -0.0532644 , -0.06600736, -0.07018627, -0.08656831,
           -0.01211547, -0.04298428, -0.06073886, -0.00776081, -0.07503507],
          dtype=float32)]

    {'name': 'dense_537', 'trainable': True, 'units': 500, 'activation': 'relu', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}
    [array([[ 0.02784217, -0.07406053,  0.05686834, ...,  0.05623003,
             0.01275428,  0.01484464],
           [ 0.03530406, -0.00448383,  0.01998237, ..., -0.05364907,
             0.0077819 ,  0.01632909],
           [ 0.06173798, -0.05853317,  0.01228458, ...,  0.03074703,
             0.0537381 , -0.05432716],
           ...,
           [-0.05984282,  0.0863284 , -0.0747195 , ...,  0.04501601,
            -0.08075234, -0.0200423 ],
           [ 0.01415475,  0.06022877, -0.04345228, ...,  0.02989638,
             0.07509039,  0.04304291],
           [ 0.00107982, -0.02591243, -0.04847531, ...,  0.00550783,
            -0.00614908, -0.05075477]], dtype=float32), array([-6.95233513e-03, -9.49798599e-02, -1.33336997e-02, -3.67417037e-02,
           -6.39090762e-02, -4.00668569e-02, -7.67405853e-02, -1.44556565e-02,
            1.54265668e-02, -4.37265225e-02, -5.99962194e-03, -9.15939268e-03,
            6.18348410e-03, -8.51333663e-02, -6.00262405e-03,  2.27350723e-02,
            5.16982637e-02, -3.17630954e-02, -1.35662496e-01,  3.73157300e-03,
           -1.30675584e-02, -1.78997312e-02, -1.53971314e-02, -6.77956343e-02,
            4.70179133e-02, -3.11631840e-02, -1.49019986e-01, -8.36878084e-03,
           -2.78099030e-02, -3.71143185e-02, -9.32724960e-03, -3.37826759e-02,
           -3.79014723e-02, -2.91629825e-02, -3.86101427e-03, -7.27424817e-03,
           -6.86452463e-02, -9.94789694e-03, -3.50516960e-02, -2.95953229e-02,
            8.26359987e-02, -4.75385338e-02, -7.67151415e-02, -9.38281696e-03,
           -3.62711512e-02, -5.38753346e-02, -3.02499514e-02, -1.07286992e-02,
           -2.16431939e-03,  3.67086083e-02, -1.62495617e-02, -1.44256279e-02,
           -3.71650420e-02, -1.26792053e-02, -1.27532529e-02, -9.49865021e-03,
            4.45175096e-02, -3.39104421e-02, -5.70563860e-02,  3.33037088e-03,
           -2.26025432e-02, -1.38645526e-02, -3.65276448e-02, -1.65512338e-02,
           -1.00625893e-02, -7.24285915e-02, -2.13912632e-02, -7.76627362e-02,
           -3.58297080e-02,  3.80251631e-02, -8.07626769e-02, -3.22895646e-02,
           -1.45213887e-01,  4.68335580e-03,  3.82119901e-02, -3.17256562e-02,
           -9.94088780e-03, -2.35514939e-02, -9.64364186e-02, -1.45641372e-01,
           -7.95768797e-02, -5.17611206e-03, -1.63643006e-02,  1.61456168e-02,
           -1.84729416e-02, -1.09371962e-02, -7.18360320e-02, -7.46829249e-03,
           -1.78928915e-02,  3.59388180e-02, -1.47025371e-02,  2.42233276e-03,
           -2.95703448e-02, -4.98019420e-02, -8.16184562e-03, -6.74523488e-02,
            1.65975699e-03, -8.87086149e-03, -3.96498255e-02, -1.48623693e-03,
           -9.46500804e-03,  1.03141610e-02, -1.65678107e-03, -1.91675648e-02,
           -3.45484503e-02, -9.15384293e-02,  4.80955653e-02,  1.21356705e-02,
           -1.17714843e-02,  2.85943281e-02, -3.51708122e-02, -7.99541920e-03,
           -4.03993465e-02, -3.00618615e-02, -1.41016217e-02,  3.44468243e-02,
           -1.74377691e-02, -6.89924881e-03, -2.49248482e-02,  0.00000000e+00,
           -2.37835329e-02, -2.21980680e-02, -5.36532812e-02, -9.76607949e-03,
            1.97958052e-02,  0.00000000e+00, -3.76204178e-02, -1.27833895e-02,
           -9.55983624e-03, -1.50375843e-01, -2.18630265e-02, -2.51890416e-03,
           -3.05114482e-02, -5.55782905e-03, -1.52521674e-02, -8.44046753e-03,
           -2.19571311e-02, -1.17782187e-02,  5.03456704e-02, -1.45259341e-02,
           -5.21913916e-03,  1.36731444e-02, -1.73022952e-02, -2.41058394e-02,
           -6.85190186e-02, -4.13522534e-02,  4.45318334e-02, -6.92730993e-02,
           -8.20065569e-03, -1.27561688e-01, -4.32568789e-02,  1.25388979e-04,
           -8.13521538e-03,  2.98437644e-02,  4.00355980e-02, -4.15736437e-02,
           -6.42505586e-02, -1.10531293e-01,  0.00000000e+00, -1.06326910e-02,
           -3.71056907e-02, -5.07014133e-02, -3.96588780e-02, -3.36712860e-02,
           -3.27645801e-02, -2.71173846e-03, -5.21564297e-02, -1.92840700e-03,
           -3.60453054e-02,  2.65608430e-02, -7.65534639e-02,  3.42699960e-02,
            4.57194430e-04, -1.17753871e-01, -2.85368469e-02, -6.76004495e-03,
            3.95774879e-02, -1.14256956e-01, -9.89448931e-03, -1.03922030e-02,
           -3.17266919e-02, -4.41980995e-02, -8.74052383e-03, -3.10960617e-02,
           -2.37462837e-02, -3.13367695e-02,  6.64173812e-02, -2.34295381e-03,
           -7.74073303e-02, -8.37145373e-02, -2.34686509e-02, -1.34693887e-02,
           -7.55947232e-02, -7.39513040e-02, -9.69363609e-04, -2.66018119e-02,
           -5.97557984e-03, -2.03780588e-02,  2.35178955e-02, -6.92866147e-02,
           -6.17796541e-05, -3.28556125e-05, -2.91844625e-02, -2.37257853e-02,
           -3.75578254e-02, -8.89574364e-03, -8.91298652e-02, -1.06097935e-02,
           -2.84796022e-03, -1.69994030e-02, -8.80667847e-03, -6.13862686e-02,
            1.43223274e-02,  3.81722003e-02, -1.09963670e-01, -3.44158337e-02,
            3.47511172e-02,  1.35471355e-02, -5.40783210e-03, -1.86581165e-02,
           -5.46754431e-03,  3.94724421e-02, -1.84977874e-02,  2.29518395e-02,
           -2.49619242e-02, -7.27421492e-02,  2.36765444e-02, -1.34942934e-01,
           -8.20308737e-03, -6.00252440e-03, -3.28872316e-02, -4.76802699e-02,
           -1.58844627e-02, -1.11152545e-01, -7.16327354e-02, -1.11666117e-02,
           -1.28277754e-02, -6.90681934e-02,  4.01831791e-02, -3.01918089e-02,
           -3.80508672e-03, -5.27284369e-02, -3.84729579e-02, -3.68304104e-02,
            1.58140883e-02, -5.24130985e-02, -2.39391681e-02,  0.00000000e+00,
           -1.60663687e-02,  2.19470002e-02, -2.00617816e-02,  9.37498920e-03,
            2.67217383e-02, -5.35759963e-02,  3.57801728e-02, -8.21993221e-03,
           -1.06495023e-02, -4.88369074e-03, -9.52593051e-03, -2.66528390e-02,
           -5.41356020e-03, -6.67789020e-03, -2.26827972e-02, -3.40861864e-02,
           -1.34489769e-02, -3.76282749e-03, -1.18529871e-02, -1.46370335e-02,
           -5.35538793e-02, -1.16271429e-01, -2.29098089e-03, -2.31637266e-02,
           -3.06563284e-02,  1.64497048e-02,  5.78400902e-02,  3.97981443e-02,
            3.40966992e-02, -6.51395023e-02, -2.94968463e-03, -3.41778435e-02,
           -3.71010369e-03, -3.91309634e-02, -9.52809397e-03, -1.14490157e-02,
           -5.17875329e-02, -3.59569564e-02, -7.62812123e-02, -9.19873193e-02,
           -1.15651898e-01, -6.00365736e-03, -4.39757630e-02, -1.76046565e-02,
           -7.06027262e-03, -2.62328647e-02, -7.17398478e-03, -7.60595873e-02,
           -6.17707893e-03, -5.67219928e-02, -6.00009132e-03, -4.84477617e-02,
           -8.40939283e-02, -6.65136753e-03, -8.32003057e-02, -6.00056443e-03,
           -5.99867478e-03, -1.11537710e-01, -3.50139402e-02, -3.40903141e-02,
           -1.25632122e-01, -5.69362985e-03, -1.90331843e-02,  6.73239902e-02,
           -7.44232303e-03, -3.85426395e-02, -3.07647288e-02, -1.10139716e-02,
           -1.53886322e-02, -5.11052692e-03, -1.05010392e-02, -1.03870407e-02,
           -6.55998811e-02, -3.04169096e-02, -2.42608059e-02, -6.89162165e-02,
           -5.71818613e-02, -2.51361281e-02, -2.13595461e-02, -6.21430725e-02,
           -5.44859245e-02, -4.52793054e-02,  9.83878970e-02, -1.56501830e-02,
           -4.61495901e-03, -7.89528117e-02, -1.96263809e-02, -2.88048852e-03,
           -7.29473829e-02, -5.00567146e-02, -5.54588437e-03, -1.66873168e-02,
           -7.03809923e-03, -1.30783925e-02, -1.48151055e-01,  3.88778374e-02,
           -3.12695988e-02, -1.68193802e-02, -2.52450462e-02,  4.23216745e-02,
           -5.88337705e-03, -5.26328245e-03, -3.85910533e-02, -4.19047065e-02,
            0.00000000e+00, -6.00367086e-03, -6.60821274e-02, -7.94360694e-03,
           -6.46531722e-03, -1.34650283e-02, -3.25251967e-02, -1.24593107e-02,
            2.67037340e-02, -2.27576084e-02, -3.02173104e-02,  4.26626951e-02,
           -6.03526570e-02, -2.65614148e-02, -5.07303961e-02, -1.71549805e-02,
           -8.45424086e-02, -2.80290078e-02, -2.58437600e-02, -2.57371794e-02,
           -5.90659212e-03, -9.94197372e-03,  2.38292175e-03, -1.26581535e-01,
           -8.86483863e-03, -1.53352134e-02, -3.62392589e-02, -9.49859712e-03,
           -3.09597179e-02, -6.00025710e-03,  3.62080075e-02, -1.42664248e-02,
            1.42897451e-02, -3.50790657e-02, -1.75820086e-02,  1.34018194e-02,
           -2.82326061e-02, -2.31988411e-02, -6.00399543e-03,  7.57009983e-02,
           -8.07125494e-02, -3.16572301e-02, -6.74487930e-03, -2.65552551e-02,
           -1.96597595e-02, -9.23418161e-03, -2.79036760e-02, -6.47726841e-03,
            4.83557284e-02, -8.29529576e-03, -2.72810832e-02, -2.85710823e-02,
           -1.05957473e-02, -2.41279043e-02, -6.00360101e-03, -2.99133174e-02,
            3.87762561e-02, -1.62464790e-02, -2.46733837e-02,  3.09612975e-02,
           -5.60390670e-03, -7.87987560e-03, -1.43264523e-02,  3.99085879e-02,
            3.42928544e-02, -1.23866713e-02, -6.03835881e-02, -3.78885642e-02,
           -6.06681183e-02, -8.32387712e-03, -2.11791731e-02, -1.27332741e-02,
            1.55259883e-02, -7.73372799e-02, -4.44846936e-02,  2.31915936e-02,
           -3.46681662e-02, -2.04327609e-02, -3.40632442e-03, -8.34449381e-02,
           -2.05070879e-02, -7.42613897e-02, -2.60449573e-02, -1.37265008e-02,
            5.32010244e-03, -1.03877066e-02, -1.50378812e-02, -3.32382582e-02,
            3.22106294e-02, -2.82431543e-02, -3.16226855e-02,  4.65320647e-02,
           -6.49466366e-03,  4.91120964e-02, -5.92677444e-02, -1.12898964e-02,
           -3.00402399e-02, -1.71631463e-02, -1.85323376e-02, -7.37725273e-02,
            4.11213562e-02, -1.28697753e-01, -2.85055973e-02, -6.65859273e-03,
           -3.03501468e-02, -4.58149090e-02,  9.01982840e-03,  3.98011580e-02,
           -7.17648417e-02, -2.28612553e-02, -2.33365055e-02, -6.00143569e-03,
           -3.11083738e-02, -1.36809079e-02, -1.35687247e-01, -3.35618109e-02,
           -7.83608388e-03, -3.90590504e-02, -4.81185131e-03, -2.41779927e-02,
           -5.27399927e-02,  1.86226554e-02, -4.67931181e-02,  5.72086759e-02,
           -2.74003893e-02, -9.46008600e-03, -1.93378963e-02, -4.96065915e-02,
           -3.97055224e-02, -3.70420120e-03, -2.63398867e-02, -1.89343765e-02,
           -1.91621855e-02, -3.65754366e-02,  4.71006669e-02, -6.56927377e-02,
           -1.09085683e-02, -1.17961224e-02,  1.14631178e-02,  4.70083281e-02,
           -7.15176715e-03, -7.91617483e-03,  1.13567812e-02, -9.80016193e-04,
           -5.14785983e-02, -3.62391360e-02, -9.51683149e-03, -4.73662391e-02],
          dtype=float32)]

    {'name': 'dense_538', 'trainable': True, 'units': 500, 'activation': 'relu', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}
    [array([[-0.05158091,  0.06292234, -0.04473642, ...,  0.02438152,
             0.00296872,  0.02433176],
           [-0.04607611,  0.04711102, -0.05038525, ..., -0.05185428,
            -0.03661033,  0.0596651 ],
           [-0.02838987, -0.08805306, -0.01347287, ...,  0.0637871 ,
            -0.02241666, -0.05965205],
           ...,
           [ 0.06442582,  0.03612294,  0.05890618, ...,  0.03255871,
            -0.07675577,  0.01445901],
           [-0.00227027,  0.01654661, -0.00233193, ...,  0.06985915,
            -0.06665283,  0.02711091],
           [-0.01991448, -0.01399131, -0.07059535, ...,  0.00305194,
             0.02374734,  0.03666957]], dtype=float32), array([-0.022403  ,  0.06091112, -0.02022716,  0.08656859,  0.08302359,
           -0.01096217, -0.03969866, -0.02771206, -0.10080553,  0.00973784,
            0.05442415, -0.0217184 ,  0.01031937, -0.0615599 , -0.01553361,
            0.03560274,  0.04985088,  0.00246698, -0.02606134,  0.00550786,
            0.07301325,  0.06625933,  0.02374047, -0.00518623,  0.06661124,
            0.09400108,  0.04552118,  0.00506337,  0.04468365,  0.11638091,
            0.02007169, -0.00632261, -0.04364461,  0.06702145,  0.03406793,
           -0.01493052, -0.01472639,  0.04228915, -0.00184474,  0.04951611,
           -0.01408686, -0.02901015, -0.02172418, -0.02397021, -0.05808303,
            0.0486505 ,  0.0565637 ,  0.06049923,  0.15338145,  0.0633488 ,
            0.03723553, -0.01605196,  0.0774424 ,  0.05070666, -0.07429504,
            0.12485828,  0.07754181, -0.02910494, -0.00600589,  0.03912723,
           -0.00599591,  0.04837193,  0.05112853,  0.04046722,  0.0994136 ,
            0.03896002,  0.04046327, -0.02777348,  0.00956383, -0.02440702,
           -0.0145189 , -0.00081246,  0.05123315,  0.01767214, -0.03161279,
           -0.00173142, -0.02572826, -0.02175168, -0.00570111, -0.03658815,
           -0.01336835,  0.06392745,  0.09618898,  0.0777472 ,  0.03413285,
           -0.03725872, -0.24420942,  0.04012506, -0.00832113, -0.02905389,
            0.05928825,  0.00454017,  0.1123696 ,  0.07727442, -0.01604208,
           -0.0234054 , -0.00600459, -0.01607908, -0.03114592, -0.01743984,
            0.0297368 , -0.02473682, -0.02116719, -0.01349558,  0.06926607,
           -0.0091477 ,  0.06480066,  0.02137369, -0.03172997, -0.00940717,
            0.03690773,  0.03539316, -0.02380276,  0.07965514, -0.02740989,
           -0.03160607, -0.00843815,  0.13258992, -0.11078158,  0.02046861,
           -0.0268019 , -0.01229542, -0.04935687,  0.01260277, -0.03154194,
            0.06132272, -0.23645484,  0.04156316,  0.08284736, -0.01235683,
           -0.02290781,  0.00158595, -0.02513298,  0.01289123,  0.08116431,
            0.01776399,  0.02546019,  0.03242729, -0.00216383,  0.13183026,
            0.02566557,  0.00863614, -0.02331898,  0.00324581, -0.00832252,
           -0.03467903, -0.02105725,  0.04413798,  0.09112782, -0.00370367,
            0.00515156, -0.05618842, -0.04144564,  0.0112011 ,  0.02588204,
           -0.03174713,  0.04318699,  0.00182303,  0.06265973, -0.02094489,
            0.04831219,  0.06095348,  0.02123662,  0.08186384, -0.04237565,
           -0.03628996, -0.02888155,  0.00109068, -0.0195195 ,  0.12831534,
           -0.01702755, -0.06222031, -0.06499834, -0.01305952, -0.00356514,
           -0.02327176,  0.04482027, -0.03131105, -0.01811083,  0.01723188,
           -0.00644053,  0.04536613, -0.04951569,  0.03988228,  0.01767246,
           -0.03280504,  0.04935803,  0.05464052,  0.06704611, -0.02800834,
           -0.02867517, -0.009508  , -0.00868461,  0.00442285,  0.07874756,
           -0.01024148, -0.03844687,  0.02871529, -0.0060475 , -0.00315342,
           -0.01010822, -0.03732352, -0.03288921, -0.02320266, -0.01708125,
            0.09362532, -0.03997428,  0.01546396, -0.03484274, -0.00915243,
           -0.00933453, -0.01999688,  0.02643666,  0.08061134,  0.06225961,
           -0.00685552,  0.03149572,  0.06167888, -0.01397133, -0.00981241,
            0.05370974, -0.06929442,  0.05942364, -0.01435741, -0.01895568,
            0.05506639, -0.01635997, -0.01488608,  0.03588402,  0.04049686,
           -0.0232477 , -0.02020557,  0.00199605,  0.00077839, -0.03697154,
            0.09601875, -0.01035253,  0.07911516,  0.04356789, -0.01333866,
           -0.00677878, -0.00843714,  0.07219963,  0.03621945, -0.01069775,
            0.06509949,  0.11428723,  0.        , -0.00817383,  0.10996895,
           -0.00215751, -0.00716174,  0.00441128,  0.03855227,  0.07396007,
           -0.03643089, -0.01991947, -0.01787876, -0.01198122,  0.03943063,
            0.06037132, -0.0372134 , -0.09725422,  0.0280073 , -0.02055868,
           -0.05358442,  0.07282444, -0.01031543,  0.06226239, -0.01270738,
           -0.03325559,  0.0435273 , -0.03838419,  0.00801511, -0.00570303,
            0.0687429 ,  0.02683219, -0.0849959 ,  0.08027921,  0.10863355,
           -0.03918127,  0.03841214, -0.01064911, -0.00447536,  0.07574324,
           -0.00936953,  0.09742852,  0.09730981,  0.04625482,  0.03247229,
           -0.01937907,  0.04467069, -0.005786  , -0.00036449, -0.00452357,
            0.10232816, -0.01907213, -0.02241392,  0.03767801, -0.02901874,
           -0.02484521, -0.00260601,  0.10153711, -0.02082514,  0.05418099,
            0.05054125, -0.06810249, -0.00144897, -0.02703675, -0.0999388 ,
           -0.03242527,  0.02807503,  0.00815004, -0.03494754, -0.0330904 ,
           -0.04105118, -0.00411627, -0.0051143 ,  0.01312674,  0.04298246,
            0.04973406,  0.00321672,  0.03603026,  0.11739659,  0.05296002,
            0.03684315, -0.02047105,  0.08960155, -0.03448946,  0.02036469,
            0.00103472, -0.05426643, -0.01910336, -0.02270681, -0.02645096,
           -0.03550863, -0.01630967, -0.08019739, -0.03189943, -0.08879115,
           -0.0294169 , -0.03475292, -0.03540398,  0.05680719,  0.07740591,
            0.00371755, -0.02352498,  0.00978935, -0.03201743, -0.03970942,
            0.0755543 ,  0.0512775 ,  0.05713934, -0.01811147, -0.0816647 ,
            0.03382933, -0.03475771,  0.04381052, -0.033349  ,  0.10967939,
           -0.01196721, -0.01950055, -0.00964809, -0.01309217,  0.02779179,
            0.17604639,  0.03317779, -0.0058139 , -0.07867931,  0.06614728,
           -0.01321349, -0.00636725,  0.10182535,  0.0167211 ,  0.0359881 ,
           -0.00993092, -0.00862644,  0.03756546, -0.02451141, -0.00208025,
            0.00912114, -0.00376116,  0.00615585, -0.00070233, -0.01025288,
            0.03422861,  0.02101139,  0.10294542, -0.05977224,  0.02140064,
            0.06661134, -0.02586062, -0.02399077, -0.02192246,  0.06738047,
           -0.03811343,  0.02306287,  0.17944336,  0.11063078, -0.02291457,
            0.01101209, -0.02534517,  0.02774396, -0.00803331,  0.06392201,
           -0.01295739,  0.07564051,  0.03522624, -0.02527026, -0.01658064,
           -0.04426979, -0.03230238, -0.0080799 ,  0.06983212,  0.03605225,
            0.03835595, -0.02458027, -0.03104986,  0.02202486, -0.01913553,
           -0.02497337, -0.02646584, -0.03686969, -0.03842543, -0.01277018,
           -0.0149267 , -0.00813162,  0.05354927, -0.01188045,  0.02645831,
           -0.00723814, -0.01886762,  0.10376044,  0.0449053 ,  0.07112146,
           -0.03834936,  0.14818403,  0.03915795,  0.04145229, -0.09886721,
            0.07427064, -0.00277187,  0.05929505, -0.0166438 ,  0.16781977,
           -0.02928308, -0.00748292,  0.11304398,  0.07880735, -0.02384592,
           -0.06013665,  0.06691111, -0.00932897, -0.01090147, -0.11338808,
            0.03687331,  0.08492423, -0.02848981,  0.04913461,  0.06889305,
            0.00599025, -0.01031421, -0.00967749, -0.04877385, -0.01717595,
            0.10711349,  0.07950345,  0.00676464,  0.05932263,  0.06609241,
           -0.01884186,  0.07508033, -0.01340634, -0.02536746,  0.06944999,
           -0.02630302,  0.0399214 , -0.01530996,  0.04970135, -0.00461447,
           -0.03620172, -0.00145422, -0.01403974, -0.01209552,  0.05654605,
            0.0650871 , -0.04045568, -0.01447588, -0.02082426, -0.00839227,
           -0.02174668,  0.08036903,  0.01552895, -0.0093033 ,  0.04171906,
            0.00238645,  0.02113066,  0.04602629,  0.04419628, -0.004174  ],
          dtype=float32)]

    {'name': 'dense_539', 'trainable': True, 'units': 1, 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}
    [array([[-5.97795248e-02],
           [ 3.32893543e-02],
           [ 1.38466293e-02],
           [-1.25850976e-01],
           [-1.14663959e-01],
           [-8.73107743e-03],
           [-4.12307493e-02],
           [-1.30233970e-02],
           [-1.67685002e-01],
           [ 1.41143473e-02],
           [ 5.12351319e-02],
           [ 1.09352460e-02],
           [-6.45687729e-02],
           [-9.86425392e-03],
           [ 2.42059529e-02],
           [ 1.39073161e-02],
           [-7.93522820e-02],
           [ 3.03093698e-02],
           [-2.96375976e-04],
           [-7.76297003e-02],
           [ 8.55475813e-02],
           [-1.26120165e-01],
           [ 1.33573050e-02],
           [-2.35000928e-03],
           [-7.90578946e-02],
           [-9.21113640e-02],
           [ 9.59264040e-02],
           [-1.57897118e-02],
           [-3.81845608e-02],
           [ 1.40251905e-01],
           [ 7.11413333e-03],
           [-9.89815965e-02],
           [-1.21244378e-02],
           [ 1.07401542e-01],
           [-5.39769232e-02],
           [ 6.05849996e-02],
           [-3.48877758e-02],
           [ 1.24270752e-01],
           [-9.55839921e-03],
           [-8.19351599e-02],
           [ 7.34181926e-02],
           [-6.75131083e-02],
           [-3.44938487e-02],
           [ 7.38278255e-02],
           [ 1.49071561e-02],
           [ 5.26775457e-02],
           [-8.30736607e-02],
           [-9.65212286e-02],
           [ 2.81013697e-01],
           [ 3.26364040e-02],
           [-2.62570735e-02],
           [-3.83987837e-02],
           [ 3.03612296e-02],
           [ 4.05216441e-02],
           [-1.14706494e-01],
           [ 1.33874610e-01],
           [-1.07885122e-01],
           [-2.71575097e-02],
           [ 5.04954136e-04],
           [ 5.16807958e-02],
           [-8.03470239e-03],
           [ 2.68833209e-02],
           [ 6.17931299e-02],
           [ 1.99518390e-02],
           [ 9.60129425e-02],
           [ 3.99165563e-02],
           [ 8.02393705e-02],
           [-5.49146906e-02],
           [ 1.35432959e-01],
           [-1.47703628e-04],
           [-4.66447026e-02],
           [ 5.45325279e-02],
           [ 6.10857345e-02],
           [-4.56688693e-03],
           [ 1.67826712e-02],
           [ 3.57214957e-02],
           [ 1.41697784e-03],
           [ 4.39690484e-04],
           [ 4.52730916e-02],
           [-2.62229927e-02],
           [ 9.35633555e-02],
           [-1.00873500e-01],
           [-9.08078402e-02],
           [ 1.25863165e-01],
           [ 3.02917827e-02],
           [-1.08824573e-01],
           [ 6.49098992e-01],
           [ 2.98726503e-02],
           [-4.06323336e-02],
           [-3.83957312e-03],
           [ 6.52786717e-02],
           [-2.79243710e-03],
           [ 1.36378691e-01],
           [-2.98040390e-01],
           [ 6.81353286e-02],
           [-6.99849129e-02],
           [ 8.85089636e-02],
           [ 1.72013883e-02],
           [ 6.62234193e-03],
           [-1.74023919e-02],
           [ 4.49526235e-02],
           [ 9.06632340e-05],
           [-1.79871731e-02],
           [-1.08817250e-01],
           [ 5.52476048e-02],
           [ 6.99102357e-02],
           [-8.78168494e-02],
           [ 1.06043722e-02],
           [-1.00614242e-02],
           [ 4.02905345e-02],
           [ 2.54242718e-02],
           [ 8.40900540e-02],
           [ 1.34556917e-02],
           [-1.14452608e-01],
           [-5.99117135e-04],
           [-4.84296903e-02],
           [-3.34403268e-03],
           [-1.56126678e-01],
           [-2.14538530e-01],
           [ 1.35280658e-02],
           [ 2.53702905e-02],
           [ 3.39074363e-03],
           [ 7.16690393e-03],
           [-1.03773311e-01],
           [-1.47592984e-02],
           [-7.40178376e-02],
           [ 4.62606639e-01],
           [ 6.52658194e-02],
           [-1.26582012e-01],
           [-1.16852205e-02],
           [ 3.82933905e-03],
           [ 7.43042752e-02],
           [-1.53916189e-02],
           [-4.87029254e-02],
           [ 7.82405213e-02],
           [-3.31152342e-02],
           [-1.58011727e-02],
           [-3.38475481e-02],
           [ 9.35077220e-02],
           [ 2.86146373e-01],
           [ 3.39649133e-02],
           [-1.33905262e-02],
           [ 1.01588495e-01],
           [-1.43919699e-02],
           [-8.00099224e-02],
           [-5.16032102e-03],
           [-1.38053475e-02],
           [ 5.07606678e-02],
           [-1.90230533e-01],
           [-2.31243540e-02],
           [ 7.28032291e-02],
           [-5.75091271e-03],
           [-1.18440548e-02],
           [ 1.77396357e-03],
           [ 1.10080771e-01],
           [ 3.60744484e-02],
           [-5.89262508e-02],
           [ 3.10318125e-03],
           [-1.75191641e-01],
           [-7.98943266e-02],
           [ 7.18176216e-02],
           [-1.10672429e-01],
           [-2.10392172e-03],
           [-1.42136008e-01],
           [ 1.15281502e-02],
           [ 1.33095998e-02],
           [ 3.09391804e-02],
           [-7.46795163e-03],
           [-9.51148476e-03],
           [ 9.70374718e-02],
           [-1.56210940e-02],
           [ 2.80483048e-02],
           [-1.48996517e-01],
           [-1.02291442e-02],
           [ 3.90667208e-02],
           [ 1.69292390e-02],
           [ 3.91205885e-02],
           [-9.47038457e-03],
           [-8.33574533e-02],
           [-1.50135290e-02],
           [ 1.19299985e-01],
           [-1.40884489e-01],
           [ 1.98735632e-02],
           [-6.46627694e-02],
           [-2.13083066e-02],
           [-1.05920620e-02],
           [ 5.51300235e-02],
           [-6.11069985e-02],
           [ 4.03404087e-02],
           [-1.27308704e-02],
           [ 2.02584751e-02],
           [ 6.98621559e-04],
           [-8.79521016e-03],
           [ 1.66972466e-02],
           [-1.12289347e-01],
           [ 9.62454155e-02],
           [-5.00966869e-02],
           [ 8.98825824e-02],
           [ 6.26172684e-03],
           [ 1.10935594e-03],
           [-1.50969690e-02],
           [ 2.07992997e-02],
           [-9.95592400e-02],
           [-5.64675704e-02],
           [ 4.81194537e-03],
           [ 1.60759419e-01],
           [ 2.93058660e-02],
           [ 1.86510727e-01],
           [-9.02158469e-02],
           [ 5.75589165e-02],
           [ 1.05213723e-03],
           [ 1.49009554e-02],
           [-5.21322414e-02],
           [-1.89259231e-01],
           [-4.92741279e-02],
           [-1.61837544e-02],
           [ 4.29344028e-02],
           [-1.15048811e-01],
           [ 1.50377927e-02],
           [ 6.52811071e-03],
           [-4.45952080e-02],
           [-8.49019960e-02],
           [-3.74241658e-02],
           [ 7.63153657e-02],
           [ 1.84003031e-03],
           [-5.51915057e-02],
           [-2.12719198e-02],
           [ 4.54220019e-04],
           [-4.85681631e-02],
           [-6.43102303e-02],
           [-5.87361194e-02],
           [ 9.62919891e-02],
           [ 5.48761711e-03],
           [-4.87857824e-03],
           [ 6.67708972e-03],
           [ 1.41806275e-01],
           [-2.88131461e-03],
           [-1.07854418e-01],
           [-5.95012717e-02],
           [ 1.95407122e-02],
           [-5.55393985e-03],
           [ 6.37339577e-02],
           [-7.15638697e-02],
           [ 4.24172506e-02],
           [-5.65894647e-04],
           [ 3.35058600e-01],
           [ 1.13755494e-01],
           [-3.97555530e-02],
           [ 6.60827458e-02],
           [-1.13715343e-01],
           [ 9.02267173e-02],
           [-4.20969119e-03],
           [ 6.76902011e-03],
           [-6.29788041e-02],
           [-1.10312767e-01],
           [-4.34141979e-03],
           [-6.90348586e-03],
           [-5.28468378e-03],
           [ 1.89299695e-02],
           [-4.22906578e-02],
           [-8.09409544e-02],
           [ 7.82706216e-03],
           [-1.57129154e-01],
           [-3.15518603e-02],
           [ 8.05409178e-02],
           [ 3.04544754e-02],
           [ 3.06997560e-02],
           [-2.59811903e-04],
           [-6.37530386e-02],
           [-1.30859937e-03],
           [-1.03240468e-01],
           [ 9.63077992e-02],
           [ 6.45405101e-03],
           [-2.71502361e-02],
           [ 1.61501005e-01],
           [-1.12726845e-01],
           [ 5.15356846e-02],
           [-1.73150823e-01],
           [ 4.41640876e-02],
           [ 6.73258528e-02],
           [-4.83816192e-02],
           [-5.33227511e-02],
           [-7.73339123e-02],
           [-6.77613833e-04],
           [ 1.14590012e-01],
           [-8.88266191e-02],
           [-1.80071041e-01],
           [-1.15224674e-01],
           [ 3.69217843e-02],
           [ 2.05066428e-02],
           [ 5.49575761e-02],
           [ 6.75997883e-02],
           [ 1.14445224e-01],
           [-2.35853455e-04],
           [ 9.77189317e-02],
           [ 8.06192160e-02],
           [ 4.80136462e-02],
           [-3.10615636e-02],
           [-2.09903233e-02],
           [ 1.17083732e-02],
           [-5.91057129e-02],
           [ 2.44240668e-02],
           [ 1.00098446e-01],
           [ 8.21149573e-02],
           [-1.32541105e-01],
           [-4.10993025e-02],
           [-1.87600791e-01],
           [ 3.07700653e-02],
           [ 1.59146520e-03],
           [-1.24086760e-01],
           [ 5.49453171e-03],
           [ 1.01256549e-01],
           [-1.57181397e-01],
           [ 2.04171930e-02],
           [-4.06815931e-02],
           [-7.29135796e-03],
           [ 1.14097679e-02],
           [ 4.15298678e-02],
           [ 1.00974485e-01],
           [-3.39207165e-02],
           [-3.11514363e-02],
           [-1.26533387e-02],
           [ 1.56773068e-02],
           [ 1.05069734e-01],
           [-6.01649024e-02],
           [ 3.18197049e-02],
           [ 3.95036582e-03],
           [ 1.37945846e-01],
           [-2.70028822e-02],
           [ 1.29218679e-02],
           [ 1.30027896e-02],
           [ 3.61586013e-03],
           [-1.11052161e-02],
           [-3.33922682e-03],
           [-5.75203274e-04],
           [ 5.88364340e-02],
           [ 9.47039351e-02],
           [-1.63234606e-01],
           [ 4.19403799e-02],
           [-2.42965177e-01],
           [ 3.80616006e-03],
           [-2.93377023e-02],
           [-2.70378236e-02],
           [-7.28814453e-02],
           [-1.17145434e-01],
           [ 7.53150061e-02],
           [-6.00704886e-02],
           [ 2.50010453e-02],
           [-2.98189912e-02],
           [-5.03513366e-02],
           [-8.62855539e-02],
           [-9.20926407e-02],
           [ 8.72706696e-02],
           [ 2.92146392e-02],
           [-1.74918339e-01],
           [ 4.99229692e-02],
           [-1.85692578e-03],
           [-2.83949655e-02],
           [-4.86441366e-02],
           [ 3.56651127e-01],
           [ 4.23480161e-02],
           [ 5.68069331e-02],
           [-2.39024553e-02],
           [ 3.76291387e-03],
           [-6.35475740e-02],
           [ 4.78802621e-01],
           [ 6.07136823e-02],
           [ 9.92011204e-02],
           [-1.04088455e-01],
           [-6.99911192e-02],
           [ 1.39922798e-02],
           [ 7.11802989e-02],
           [ 1.25243500e-01],
           [ 3.56512778e-02],
           [ 5.73225319e-02],
           [-8.02174117e-03],
           [ 4.97192927e-02],
           [ 2.91048251e-02],
           [-6.10351935e-02],
           [ 4.63643111e-03],
           [ 7.22097792e-03],
           [-1.42452102e-02],
           [-2.27106479e-03],
           [ 1.38643384e-03],
           [-5.65934740e-02],
           [ 3.90773416e-02],
           [ 7.00348709e-03],
           [-1.48283675e-01],
           [ 5.42136375e-03],
           [-9.92271490e-03],
           [ 1.68262683e-02],
           [-2.71074381e-02],
           [ 5.29733673e-03],
           [-1.33249229e-02],
           [ 5.27428463e-02],
           [-1.01253986e-02],
           [ 9.23285857e-02],
           [ 2.68849820e-01],
           [ 1.39395326e-01],
           [-1.44382520e-02],
           [-1.12231476e-02],
           [-1.43260546e-02],
           [ 1.35502182e-02],
           [-8.87447316e-03],
           [ 8.99918377e-02],
           [ 8.47581849e-02],
           [ 9.85623747e-02],
           [-3.67529504e-02],
           [-6.81969523e-02],
           [ 1.71280222e-03],
           [-5.78587782e-03],
           [-4.49550292e-03],
           [ 6.11585788e-02],
           [-1.37733757e-01],
           [ 1.68877952e-02],
           [ 1.87899638e-02],
           [-3.95161472e-03],
           [-9.92924720e-03],
           [-4.73048463e-02],
           [ 5.46065643e-02],
           [-4.46980298e-02],
           [-5.14186062e-02],
           [-1.24721108e-02],
           [-4.34140861e-02],
           [ 3.79035249e-02],
           [ 3.42186890e-03],
           [ 5.79573512e-02],
           [-6.61875010e-02],
           [ 5.58791123e-03],
           [-3.09008043e-02],
           [ 8.58744010e-02],
           [ 5.62902018e-02],
           [ 1.23285480e-01],
           [-6.00670613e-02],
           [ 4.58518267e-02],
           [-6.02330491e-02],
           [ 3.27112585e-01],
           [-4.80739996e-02],
           [-6.42874464e-02],
           [ 8.02485421e-02],
           [-2.13302299e-01],
           [ 9.53765288e-02],
           [ 2.94521973e-02],
           [ 3.64266448e-02],
           [ 4.06655550e-01],
           [-8.44813585e-02],
           [ 1.07146837e-02],
           [-1.24913029e-01],
           [ 1.13783047e-01],
           [-7.88702630e-03],
           [-8.62127841e-02],
           [-1.45320192e-01],
           [ 8.23544934e-02],
           [ 4.10105139e-02],
           [-1.36691913e-01],
           [-3.36164869e-02],
           [-7.54454508e-02],
           [-3.19666676e-02],
           [ 2.24916488e-02],
           [-4.80479710e-02],
           [ 1.11300007e-01],
           [ 1.56592354e-02],
           [ 2.72110989e-03],
           [-1.10204220e-02],
           [-1.85542542e-03],
           [-1.10671453e-01],
           [-8.66593942e-02],
           [ 1.62054405e-01],
           [-1.03218362e-01],
           [-6.43826276e-02],
           [-1.31604047e-02],
           [ 5.92641532e-02],
           [ 4.79774773e-02],
           [ 1.65536255e-02],
           [ 6.37630820e-02],
           [ 1.10786539e-02],
           [ 3.46582532e-02],
           [ 8.07502423e-04],
           [-4.46697213e-02],
           [-3.61783504e-02],
           [ 1.29947346e-02],
           [-1.02628274e-02],
           [ 3.89413275e-02],
           [-4.99057723e-03],
           [-8.95637199e-02],
           [-7.88603276e-02],
           [-2.25425945e-04],
           [-9.36929695e-03],
           [ 8.31389800e-03],
           [ 3.21312551e-03],
           [-6.40653120e-03],
           [-8.06082338e-02],
           [ 2.27612741e-02],
           [ 6.39369860e-02],
           [ 5.03055751e-02],
           [-5.09335008e-03],
           [-5.63904690e-03],
           [-7.72839785e-02],
           [-1.03717700e-01],
           [ 1.20895086e-02]], dtype=float32), array([0.01124922], dtype=float32)]

We can observe the weights and bias terms associate with our neural network.

#### Comments

This was a basic demonstration of artificial neural networks.

The Texas Center for Geostatistics has many other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available [here](https://github.com/GeostatsGuy/PythonNumericalDemos), along with a package for geostatistics in Python called [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy).

We hope this was helpful,

*Michael* and *Honggeun*

***

#### More on Michael Pyrcz and the Texas Center for Data Analytics and Geostatistics:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)


# Applied Stochastic Processes HW_12

**11510691 程远星$\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\plim}{plim}
\newcommand{\ffrac}{\displaystyle \frac}
\newcommand{\d}[1]{\displaystyle{#1}}
\newcommand{\space}{\text{ }}
\newcommand{\bspace}{\;\;\;\;}
\newcommand{\bbspace}{\;\;\;\;\;\;\;\;}
\newcommand{\QQQ}{\boxed{?\:}}
\newcommand{\void}{\left.\right.}
\newcommand{\Tran}[1]{{#1}^{\mathrm{T}}}
\newcommand{\CB}[1]{\left\{ #1 \right\}}
\newcommand{\SB}[1]{\left[ #1 \right]}
\newcommand{\P}[1]{\left( #1 \right)}
\newcommand{\abs}[1]{\left| #1 \right|}
\newcommand{\norm}[1]{\left\| #1 \right\|}
\newcommand{\given}[1]{\left. #1 \right|}
\newcommand{\using}[1]{\stackrel{\mathrm{#1}}{=}}
\newcommand{\asim}{\overset{\text{a}}{\sim}}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\EE}{\mathbb{E}}
\newcommand{\II}{\mathbb{I}}
\newcommand{\NN}{\mathbb{N}}
\newcommand{\ZZ}{\mathbb{Z}}
\newcommand{\QQ}{\mathbb{Q}}
\newcommand{\PP}{\mathbb{P}}
\newcommand{\AcA}{\mathcal{A}}
\newcommand{\FcF}{\mathcal{F}}
\newcommand{\AsA}{\mathscr{A}}
\newcommand{\FsF}{\mathscr{F}}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\I}[1]{\mathrm{I}\left( #1 \right)}
\newcommand{\N}[1]{\mathcal{N}\left( #1 \right)}
\newcommand{\Exp}[1]{\mathrm{E}\left[ #1 \right]}
\newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]}
\newcommand{\Avar}[1]{\mathrm{Avar}\left[ #1 \right]}
\newcommand{\Cov}[1]{\mathrm{Cov}\left( #1 \right)}
\newcommand{\Corr}[1]{\mathrm{Corr}\left( #1 \right)}
\newcommand{\ExpH}{\mathrm{E}}
\newcommand{\VarH}{\mathrm{Var}}
\newcommand{\AVarH}{\mathrm{Avar}}
\newcommand{\CovH}{\mathrm{Cov}}
\newcommand{\CorrH}{\mathrm{Corr}}
\newcommand{\ow}{\text{otherwise}}
\newcommand{\FSD}{\text{FSD}}
\newcommand{\SSD}{\text{SSD}}
\void^\dagger$**

## Question 1

$\P{1}$

$\bspace$From $Proposition\space5.1$, we see that $\d{P\CB{T_2 > t} = \int_0^\infty \lambda e^{-m\P{t+s}}\;\dd s}$ and then we guess that they are not independent.

$\P{2}$

$\bspace$No, judging from the previous result.

$\P 3$

$\bspace \d{P\CB{T_1 > t} = e^{-m\P t} \Rightarrow f_{T_1}\P t = m'\P t e^{-m\P t} = \lambda\P t e^{-m\P t}}$

## Question 2

$\bspace N\P 0 = N_0\P{m\P 0} = N_0\P 0 = 0$

$\bspace$The increment $N\P{t+s} - N\P t = N_0\P{m\P{t+s}} - N_0\P{m\P t}$. Since $m\P t$ is increasing, $N_0\P{m\P{t+s}}$ and $N_0\P{m\P{t}}$ can be considered rely only on $\P{t,t+s}$. Thus, independent increment property holds

$\bspace\begin{align}
P\CB{N\P{t+h} - N\P t = 1} &= P\CB{N_0\P{m\P{t+h}} - N_0\P{m\P{t}} = 1}\\
&\bspace\text{using the hint}: m\P{t+h} = m\P t + \lambda\P t \cdot h + o\P h\\
&= P\CB{N_0\P{m\P t + \lambda\P t \cdot h + o\P h} - N_0\P{m\P{t}} = 1}\\
&= P\CB{N_0\P{\lambda\P t \cdot h + o\P h} = 1}\\
&= \P{\lambda\P t \cdot h + o\P h}\cdot 1 + o\P h = \lambda\P t \cdot h + o\P h
\end{align}$

$\bspace\begin{align}
P\CB{N\P{t+h} - N\P t \geq 2} &= P\CB{N_0\P{m\P t + \lambda\P t \cdot h + o\P h} - N_0\P{m\P{t}} \geq 2}\\
&= P\CB{N_0\P{\lambda\P t \cdot h + o\P h} \geq 2}\\
&= o\P{\lambda\P t \cdot h + o\P h} = o\P h
\end{align}$

## Question 3

$\bspace\begin{align}
\Cov{X\P t , X\P{t+s}} &= \Cov{X\P t , X\P{t}} + \Cov{X\P t , X\P{t+s} - X\P t}\\
&= \Cov{X\P t , X\P{t}} + \Cov{X\P t , X\P{s}}\\
&= \Var{X\P t} = \lambda t \Exp{Y^2}
\end{align}$

## Question 4

$\bspace$Let $N\P t$ denote the number of customers in the station at time $t$ then we can consider $\CB{N\P t, t\geq 0}$ as a birth and death process with $\lambda_n = \lambda \alpha_n$ and $\mu_n = \mu$.

## Question 5

$\P 1$

$\bspace \Exp{T_0} = \ffrac{1}{\lambda_0} = \ffrac{1}{\lambda}$, $\Exp{T_1} = \ffrac{1}{\lambda_1} + \ffrac{\mu_1}{\lambda_1}\Exp{T_0} = \ffrac{\lambda + \mu}{2\lambda^2}$, $\Exp{T_2} = \ffrac{1}{\lambda_2} + \ffrac{\mu_2}{\lambda_2}\Exp{T_1} = \ffrac{\lambda^2 + \lambda\mu + \mu^2}{3\lambda^3}$

$\bspace \Exp{T_3} = \ffrac{\lambda^3 + \lambda^2\mu + \lambda\mu^2 + \mu^3}{4\lambda^4}$, $\Exp{T_0} + \Exp{T_1} + \Exp{T_2} + \Exp{T_3} = \ffrac{25\lambda^3 + 13\lambda^2\mu+7\lambda\mu^2+ 3\mu^2}{12\lambda^4}$

$\P 2$

$\bspace \Exp{T_4} = \ffrac{\lambda^4 + \lambda^3\mu + \lambda^2\mu^2 + \lambda\mu^3 +\mu^4}{5\lambda^5}$, $ \Exp{T_2} + \Exp{T_3} + \Exp{T_4} = \ffrac{47\lambda^4 + 47\lambda^3\mu + 47\lambda^2\mu^2 + 27\lambda\mu^3+ 12\mu^4}{60\lambda^5}$

$\P 3$

$\bspace \Var{T_0} = \ffrac{1}{\lambda_0^2} = \ffrac{1}{\lambda^2}$, $\Var{T_1} = \ffrac{1}{\lambda_1\P{\lambda_1 + \mu_1}} + \ffrac{\mu_1}{\lambda_1}\Var{T_0} + \ffrac{\mu_1}{\lambda_1 + \mu_1} \P{\Exp{T_0} + \Exp{T_1}}^2= \cdots$,

$\bspace$Not worth doing this, give up.

***

Probability and Statistics/Applied Random Process/HW/HW_12.ipynb