nnsa.keras package

Submodules

nnsa.keras.callbacks module

This module contains custom Keras callbacks that might be used for training.

Classes:

BetaUpdateCallback(*args, **kwargs)	Callback to update a non trainable parameter beta after each epoch, increasing from initial to final value in a predefined number of steps.
LambdaBetaUpdateCallback(*args, **kwargs)	Callback to update a non trainable parameter beta at the end of each epoch, by executing a function: on_epoch_end(epoch, logs).
MyEarlyStopping(*args, **kwargs)	Early stopping with extra parameter to start monitoring from min_epochs epochs.
StopOnNanLoss(*args, **kwargs)	Callback that checks the loss after each epoch stops training if the loss is nan.

class nnsa.keras.callbacks.BetaUpdateCallback(*args: Any, **kwargs: Any)

Bases: Callback

Callback to update a non trainable parameter beta after each epoch, increasing from initial to final value in a predefined number of steps.

Useful for e.g. applying a warmup strategy, increasing the loss of a regularizer as training progresses.

Parameters:

• initial_value (float) – initial value of beta.

• final_value (float) – final value of beta.

• steps (int) – number of epochs in which to increase beta from initial to final value.

• verbose (int) – verbose level (if 1, prints the updates to beta).

Example

>>> beta_callback = BetaUpdateCallback(initial_value=0, final_value=1/100, steps=5)
>>> loss = loss_term_1 + beta_callback.beta * loss_term_2

Attributes:

beta

Methods:

on_epoch_begin(epoch[, logs])

property beta

on_epoch_begin(epoch, logs=None)

class nnsa.keras.callbacks.LambdaBetaUpdateCallback(*args: Any, **kwargs: Any)

Bases: Callback

Callback to update a non trainable parameter beta at the end of each epoch, by executing a function: on_epoch_end(epoch, logs).

Parameters:

• fun (function) – a function that takes two inputs: epoch and logs (see on_epoch_begin) and returns a float (the new value for beta).

• initial_value (float) – initial value for beta.

• verbose (int) – verbose level (if 1, prints the updates to lambda).

Example

>>> lambda_callback = LambdaBetaUpdateCallback(initial_value=0, fun=lambda epoch, logs: epoch)

Attributes:

beta

Methods:

on_epoch_end(epoch[, logs])

property beta

on_epoch_end(epoch, logs=None)

class nnsa.keras.callbacks.MyEarlyStopping(*args: Any, **kwargs: Any)

Bases: EarlyStopping

Early stopping with extra parameter to start monitoring from min_epochs epochs.

Methods:

on_epoch_end(epoch[, logs])

on_epoch_end(epoch, logs=None)

class nnsa.keras.callbacks.StopOnNanLoss(*args: Any, **kwargs: Any)

Bases: Callback

Callback that checks the loss after each epoch stops training if the loss is nan.

Parameters:

• which (str or list) – which loss to check for nan, e.g. ‘loss’, or [‘loss, ‘val_loss’]. Defaults to [‘loss’, ‘val_loss’].

• raise_error (bool) – if True, raises an error when loss is nan. If False, stops training without raising error.

Methods:

on_epoch_end(epoch[, logs])
on_train_batch_end(batch[, logs])
on_train_begin([logs])
on_train_end([logs])

on_epoch_end(epoch, logs=None)

on_train_batch_end(batch, logs=None)

on_train_begin(logs=None)

on_train_end(logs=None)

nnsa.keras.layers module

This module contains helper functions to create a block of layers of a neural network.

Classes:

Sampling(*args, **kwargs)

Uses (z_mean, z_log_var) to sample z.

Functions:

conv1d_block(filters, kernel_size[, ...])	Helper function that implements layer_block with a Conv1D layer.
conv1dtranspose_block(filters, kernel_size, ...)	Helper function that implements layer_block with a Conv1DTranspose layer.
conv2d_block(filters, kernel_size[, ...])	Helper function that implements layer_block with a Conv2D layer.
conv2dtranspose_block(filters, kernel_size)	Helper function that implements layer_block with a Conv2DTranspose layer.
dense_block(units[, dense_kwargs, ...])	Helper function that implements layer_block with a Dense layer.
inception1d(filters[, kernel_sizes, ...])	Block of layers for inception module for 1D inputs.
inception2d(filters[, kernel_sizes, ...])	Block of layers for inception module for 2D inputs.
inception2dtranspose(filters[, ...])	Block of layers for transpose inception module for 2D inputs.
layer_block(layer, *args[, batchnorm, ...])	Layer with optional batch normalization, activation and dropout regularization.
sinc2d([filters, branches, ...])	Block of layers for shared inception (sinc) module for 2D inputs.

class nnsa.keras.layers.Sampling(*args: Any, **kwargs: Any)

Bases: Layer

Uses (z_mean, z_log_var) to sample z.

Adapted from https://keras.io/examples/generative/vae/

Examples

>>> from artefact_detection.keras_utils.losses import kl_loss
>>> x = Input(shape=[100, 1])
>>> z_mean = Dense(10, name="z_mean")(x)
>>> z_log_var = Dense(10, name="z_log_var")(x)
>>> z = Sampling(name='z')([z_mean, z_log_var])
>>> y = Dense(100)(z)
>>> model = Model(x, y)
>>> model.add_loss(kl_loss(z_mean, z_log_var, beta=1))

Methods:

call(inputs)

call(inputs)

nnsa.keras.layers.conv1d_block(filters, kernel_size, strides=1, padding='same', conv1d_kwargs=None, batchnorm=True, batchnorm_kwargs=None, activation='relu', activation_kwargs=None, dropout=True, dropout_kwargs=None)

Helper function that implements layer_block with a Conv1D layer.

nnsa.keras.layers.conv1dtranspose_block(filters, kernel_size, strides, padding='same', conv1dtranspose_kwargs=None, batchnorm=True, batchnorm_kwargs=None, activation='relu', activation_kwargs=None, dropout=True, dropout_kwargs=None)

Helper function that implements layer_block with a Conv1DTranspose layer.

nnsa.keras.layers.conv2d_block(filters, kernel_size, strides=1, padding='same', conv2d_kwargs=None, batchnorm=True, batchnorm_kwargs=None, activation='relu', activation_kwargs=None, dropout=True, dropout_kwargs=None)

Helper function that implements layer_block with a Conv2D layer.

nnsa.keras.layers.conv2dtranspose_block(filters, kernel_size, strides=(1, 1), padding='same', conv2dtranspose_kwargs=None, batchnorm=True, batchnorm_kwargs=None, activation='relu', activation_kwargs=None, dropout=True, dropout_kwargs=None)

Helper function that implements layer_block with a Conv2DTranspose layer.

nnsa.keras.layers.dense_block(units, dense_kwargs=None, batchnorm=True, batchnorm_kwargs=None, activation='relu', activation_kwargs=None, dropout=True, dropout_kwargs=None)

Helper function that implements layer_block with a Dense layer.

nnsa.keras.layers.inception1d(filters, kernel_sizes=(10, 20, 40), pool_size=None, strides=1, use_bottleneck=True, bottleneck_size=None, activation=None)

Block of layers for inception module for 1D inputs.

nnsa.keras.layers.inception2d(filters, kernel_sizes=((10, 1), (20, 1), (40, 1)), pool_size=None, strides=(1, 1), use_bottleneck=True, bottleneck_size=None, activation=None)

Block of layers for inception module for 2D inputs.

nnsa.keras.layers.inception2dtranspose(filters, kernel_sizes=((10, 1), (20, 1), (40, 1)), strides=(2, 2), activation=None)

Block of layers for transpose inception module for 2D inputs.

nnsa.keras.layers.layer_block(layer, *args, batchnorm=True, batchnorm_kwargs=None, activation='relu', activation_kwargs=None, dropout=False, dropout_kwargs=None, **kwargs)

Layer with optional batch normalization, activation and dropout regularization.

Parameters:

• layer (keras.Layer) – keras layer object. E.g., layer=Conv2D

• *args – optional positional arguments for the keras layer.

• batchnorm (bool) – bool specifying whether to add a BatchNormalization layer.

• batchnorm_kwargs (dict) – optional keyword arguments for BatchNormalization().

• activation (str or None or Activation) – specifies the activation used. See Activation().

• activation_kwargs (dict) – optional keyword arguments for Activation().

• dropout (bool) – bool specifying whether to add a Dropout layer.

• dropout_kwargs (dict) – optional keyword arguments for Dropout().

• **kwargs – optional keyword arguments for layer.

Returns:

block (function) – function that takes in a keras layer and returns the output layer after passing the input thtough the specified layer and BatchNormalization, Activation and Dropout layers.

nnsa.keras.layers.sinc2d(filters=64, branches=5, l2_regularization=0.001, activation='elu', dropout=True, dropout_kwargs=None)

Block of layers for shared inception (sinc) module for 2D inputs.

nnsa.keras.losses module

This module contains Keras implementations of custom loss functions.

Functions:

SADLoss(c[, eta])	Semi-supervised anomaly detection loss (Ruff et al. 2019).
kl_loss(z_mean, z_log_var[, beta])	Tensorflow implementation of Kullback–Leibler divergence loss term for training variational autoencoders.
mean_var_loss(y_true, y_pred)	Mean-variance loss (Eq.
myCategoricalCrossentropy([weights, axis, ...])	Cross-categorical loss that works for batches with unlabeled examples (where the one-hot encoding is all zeros), by rescaling the computation of the mean (ignoring unlabeled examples), such that the loss won't become small due to a large number of unlabeled data.
quantile_loss(qs)	Return a quantile (pinball) loss function.

nnsa.keras.losses.SADLoss(c, eta=1.0)

Semi-supervised anomaly detection loss (Ruff et al. 2019).

References

L. Ruff et al., “Deep Semi-Supervised Anomaly Detection.” arXiv, 2019, doi: 10.48550/ARXIV.1906.02694.

Parameters:

• c (list or np.ndarray) – hypersphere center (n_laten_dims,).

• eta (float) – controlls the weight of the labeled loss. Set >1 to have more weight on labeled loss, set <1 to have less weight on labeled loss (compared to unlabeled).

Examples

>>> model.compile(loss=SADLoss(c=[4.0, 3.0]))

nnsa.keras.losses.kl_loss(z_mean, z_log_var, beta=1)

Tensorflow implementation of Kullback–Leibler divergence loss term for training variational autoencoders.

References

4. 16. Kingma and M. Welling, “Auto-Encoding Variational Bayes.” 2014. https://arxiv.org/pdf/1312.6114.pdf

Implementation: https://keras.io/examples/generative/vae/#define-the-vae-as-a-model-with-a-custom-trainstep

Parameters:

• z_mean – layer with z_mean.

• z_log_var – layer with logarithm of the variance.

• beta – weight parameter of the KL-loss. Can be a float or a tensorflow variable.

Examples

autoencoder.add_loss(kl_loss(z_mean, z_log_var, beta=1))

nnsa.keras.losses.mean_var_loss(y_true, y_pred)

Mean-variance loss (Eq. 5, 8).

References

A. Kendall and Y. Gal, “What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?,”

2017-03-15.

Parameters:

• y_true – the true values for y.

• y_pred – the first axis of the last dimension should contain the mean/point estimate. The second axis of the last dimension corresponds to the log variance (see Eq. 8).

nnsa.keras.losses.myCategoricalCrossentropy(weights=1.0, axis=-1, pseudolabel_weight=None)

Cross-categorical loss that works for batches with unlabeled examples (where the one-hot encoding is all zeros), by rescaling the computation of the mean (ignoring unlabeled examples), such that the loss won’t become small due to a large number of unlabeled data.

Adopted from rom https://gist.github.com/wassname/ce364fddfc8a025bfab4348cf5de852d.

Parameters:

• weights (tuple) – weight for each class, e.g. for 2 class problem (1.0, 10.0).

• axis (int) – axis corresponding to the number of classes.

Examples

>>> model.compile(loss=myCategoricalCrossentropy())

nnsa.keras.losses.quantile_loss(qs)

Return a quantile (pinball) loss function.

References

Koenker, R., Hallock, K.F.: Quantile regression. Journal of Economic Perspectives 15(4), 143-156 (2001). DOI 10.1257/jep.15.4.143

https://www.kaggle.com/ulrich07/quantile-regression-with-keras/notebook https://www.evergreeninnovations.co/blog-quantile-loss-function-for-machine-learning/

Parameters:

qs (list) – target quantiles (values between 0 and 1).

Returns:

loss (function) – function that takes in y_true and y_pred, computing the pinball loss.

nnsa.keras.utils module

Classes:

MonteCarloDropout(*args, **kwargs)

Monte Carlo dropout layer (which drop neurons also at inference time).

Functions:

convert_to_h5(model_path[, filepath, ...])	Convert model saved in directory model_path (SavedModel) to a single hdf5 (.h5) file.
cut_top(model, output_layer)	Cut off top-layer(s) of a keras model.
enable_monte_carlo_dropout(model[, ...])	Replace regular Dropout layers with Monte Carlo dropout layers (which drop neurons also at inference time).
setup_tf([min_gpu, verbose])	Detects GPUs and (currently) sets automatic memory growth.

class nnsa.keras.utils.MonteCarloDropout(*args: Any, **kwargs: Any)

Bases: Dropout

Monte Carlo dropout layer (which drop neurons also at inference time).

https://towardsdatascience.com/monte-carlo-dropout-7fd52f8b6571

Methods:

call(inputs)

call(inputs)

nnsa.keras.utils.convert_to_h5(model_path, filepath=None, overwrite=False, compile=False, check=True, verbose=0)

Convert model saved in directory model_path (SavedModel) to a single hdf5 (.h5) file.

Parameters:

• model_path (str) – directory with saved model (tensorflow’s SavedModel format).

• filepath (str) – destination file to save the model in HDF5 format. If not given, a path will be created by adding “.h5” to model_path.

• overwrite (bool) – whether to overwrite existing .h5 files (True) or raise an error if the file already exists (False).

• compile (bool) – whether to convert the compiled model (True) or the non-compiled model (False).

• check (bool) – whether to check if the saved model is the same as the original (True). This may not work correctly for all types of models (raising errors where it shouldn’t).

• verbose (int) – verbosity level.

nnsa.keras.utils.cut_top(model, output_layer)

Cut off top-layer(s) of a keras model. Useful for re-training purposes or feature extraction.

Parameters:

• model (keras.Model) – keras model.

• output_layer (str) – name of the final layer tp include. All next layers will be cut off.

Returns:

sub_model (keras.Model) – keras model without the top layer(s).

nnsa.keras.utils.enable_monte_carlo_dropout(model, rate_multiplier=1)

Replace regular Dropout layers with Monte Carlo dropout layers (which drop neurons also at inference time).

By specifying a rateMultiplier you can decrease/increase the dropout rate as it was originally.

https://stackoverflow.com/questions/49492255/how-to-replace-or-insert-intermediate-layer-in-keras-model.

nnsa.keras.utils.setup_tf(min_gpu=0, verbose=1)

Detects GPUs and (currently) sets automatic memory growth.

Based on https://github.com/mabhijithn/irregulars-neureka-codebase/blob/master/training/3-DNN/utils.py

Module contents