nnsa.models package
Submodules
nnsa.models.conformal_prediction module
Functions:
|
Build regression prediction intervals using conformal prediction. |
|
Build classification prediction sets using conformal prediction. |
|
Compute classification (non-)conformal scores. |
|
Compute coverage for classification prediction sets. |
|
Compute the threshold q_hat for conformal prediction. |
|
Compute a performance score from predictions sets, including only the sets with size 1. |
|
Compute conformal diagnostics to test the conformal prediction. |
|
Compute regression (non-)conformal scores. |
- nnsa.models.conformal_prediction.build_prediction_interval(y_pred, q_hat, score='absolute_error', scale=None)[source]
Build regression prediction intervals using conformal prediction.
References
https://www.youtube.com/watch?v=nql000Lu_iE
- Parameters:
y_pred (np.ndarray) – 1D array with predictions.
q_hat (float) – threshold obtained from conformal scores on the calibration dataset.
score (str) – which (non-)conformal score to compute. See code for latest options.
- Returns:
y_intervals (np.ndarray) – 2D array shape (len(y_pred), 2), containing lower and upper values of prediction intervals.
- nnsa.models.conformal_prediction.build_prediction_set(p_pred, q_hat, score='probability', signal_quality=None)[source]
Build classification prediction sets using conformal prediction.
References
https://www.youtube.com/watch?v=nql000Lu_iE
- Parameters:
p_pred (np.ndarray) – 2D array with shape (n_samples, n_classes) containing probabilities for each class. I.e., p_pred[i, j] is the probability of the jth class for sample i.
q_hat (float) – threshold obtained from conformal scores on the calibration dataset.
score (str) – which (non-)conformal score to compute. See code for latest options.
- Returns:
y_pred (np.ndarray) – 2D array same shape as p_pred, containing 0s and 1s, indicating the prediction sets. I.e., if y_pred[i, j] == 1, then class j is included in the prediction set for sample i.
- nnsa.models.conformal_prediction.class_conformal_score(y_true, p_pred, score='probability', signal_quality=None)[source]
Compute classification (non-)conformal scores.
- Parameters:
y_true (np.ndarray) – 1D array with true labels for N classes (coded by 0, 1, …, N), or the equavalent 2D array with one-hot encodings with shape (n_samples, n_classes).
p_pred (np.ndarray) – 2D array with same length as y_true containing probabilities for each class, where p_pred[i, j] is the probability of the jth class for sample i.
score (str) – which score to compute. See code for latest options.
References
https://www.youtube.com/watch?v=nql000Lu_iE
- Returns:
scores (np.ndarray) – 1D array with (non-)conformal scores.
- nnsa.models.conformal_prediction.compute_coverage(y_true, y_pred)[source]
Compute coverage for classification prediction sets.
- Parameters:
y_true (np.ndarray) – 1D array with true labels for N classes (coded by 0, 1, …, N), or the equavalent 2D array with one-hot encodings with shape (n_samples, n_classes).
y_pred (np.ndarray) – 2D array same length y_true, containing 0s and 1s, indicating the prediction sets. I.e., if y_pred[i, j] == 1, then class j is included in the prediction set for sample i.
- Returns:
coverage (float) – fraction of prediction sets that contain true label.
- nnsa.models.conformal_prediction.compute_q_hat(scores, alpha)[source]
Compute the threshold q_hat for conformal prediction.
q_hat is the (1 - alpha) quantile (with minor sample size correction) of scores. Values in scores should measure the uncertainty of the predictions.
- Returns:
q_hat (float) – threshold to use for building new prediction sets.
- nnsa.models.conformal_prediction.compute_score(y_true, y_pred, fun)[source]
Compute a performance score from predictions sets, including only the sets with size 1.
- Parameters:
y_true (np.ndarray) – 1D array with true labels for N classes (coded by 0, 1, …, N), or the equavalent 2D array with one-hot encodings with shape (n_samples, n_classes).
y_pred (np.ndarray) – 2D array same length y_true, containing 0s and 1s, indicating the prediction sets. I.e., if y_pred[i, j] == 1, then class j is included in the prediction set for sample i.
fun (function) – function to apply to y_true, y_pred (both their 1D formats with integers specifying the classes) to compute the score, e.g. sklearns classification score functions.
- Returns:
score (float) – score.
drop_rate (float) – fraction of samples dropped (emtpy prediction set or more than 1 prediction in set).
- nnsa.models.conformal_prediction.conformal_diagnostics(y_true, p_pred, score, sample_id=None, cali_split=0.5, alpha=0.1, n_iter=50, metrics_fun=None, seed=None, verbose=1)[source]
Compute conformal diagnostics to test the conformal prediction.
References
https://www.youtube.com/watch?v=TRx4a2u-j7M
- Parameters:
y_true (np.ndarray) – 1D array with true labels for N classes (coded by 0, 1, …, N), or the equavalent 2D array with one-hot encodings with shape (n_samples, n_classes).
p_pred (np.ndarray) – 2D array with same length as y_true containing probabilities for each class, where p_pred[i, j] is the probability of the jth class for sample i.
score (str) – which score to compute. See code for latest options.
sample_id (list or np.ndarray or None) – if provided, this should have length n_samples, where each entry contains the ID of the corresponding sample. This is used when randomly dividing the data into calibration and test folds: all samples with the same ID are put in the same fold, which may be useful if samples are not independent. If not provided, each sample gets a unique ID and thus is considered independent.
cali_split (float) – the fraction of data to put in the calibration set.
alpha (float) – alpha level (e.g. 0.1).
metrics_fun (dict) – dict with functions that compute any user-defined metrics that accept y_true and y_pred, both boolean arrays with shape (n_samples, n_classes). E.g. metrics={‘coverage’: lambda y_true, y_pred: np.sum(np.any(y_true*y_pred, axis=-1))/np.sum(np.any(y_true, axis=-1))}
n_iter (int) – number of iterations (random splits).
seed (int) – seed for random generator.
verbose (int) – verbosity level.
- Returns:
coverage (np.ndarray) – array with shape (n_iter,) containing the coverage of each iteration.
set_sizes (np.ndarray) – array with shape (n_iter*n_samples) containing the size of the prediction set for each sample per iteration.
stratified_coverage (np.ndarray) – array with shape (n_iter, n_classes) containing the coverage per iteration per class.
metrics (dict) – dict with arrays for each metrics with shapes (n_iter,) containing the metrics computed on the test fold per iteration.
- nnsa.models.conformal_prediction.regr_conformal_score(y_true, y_pred, score='absolute_error', scale=None)[source]
Compute regression (non-)conformal scores.
- Parameters:
y_true (np.ndarray) – 1D array with true values.
y_pred (np.ndarray) – 1D array with predicted values (same lentgh as y_true).
score (str) – which score to compute. See code for latest options.
References
https://www.youtube.com/watch?v=nql000Lu_iE
- Returns:
scores (np.ndarray) – 1D array with (non-)conformal scores.
nnsa.models.evaluation module
Code for evaluation of classifiers.
Classes:
|
High-level interface for manipulating the results of the confusion matrix. |
Functions:
|
Return the average silhouette of clustered points. |
|
Compute some scores for a binary classification algorithm by comparing predicted output to true labels. |
|
Return the inertia of clustered points. |
|
Compute scores and pvalues quantifying how well each combination of features in df separate predefined classes. |
|
Compute the F-score (TODO ask Mario for a reference). |
|
Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores. |
|
Compute a score that captures how much two sets/distributions are separated. |
- class nnsa.models.evaluation.ConfusionMatrix(y_true, y_pred, class_mapping=None)[source]
Bases:
objectHigh-level interface for manipulating the results of the confusion matrix.
Samples with np.nan values in y_true or y_pred are ignored upon initialization of the confusion matrix.
Upon initializing a ConfusionMatrix object, the confusion matrix is computed. Subsequently, by calling methods on the object, all sorts of metrics derived from the confusion matrix can be quickly computed (e.g. accuracy, sensitivity, specificity). This works for binary classification as well as multiclass classification. In the case of multiclass classification, a one-vs-all method is applied if neccessary.
A nice overview of most of thos metrics can be found on Wikipedia: https://en.wikipedia.org/wiki/Confusion_matrix
- Parameters:
y_true (np.ndarray) – 1D array with true class numbers 0:N-1, with N the number of classes.
y_pred (np.ndarray) – 1D array with predicted class numbers 0:N-1.
class_mapping (dict, optional) – dictionary that maps a class label to a class number. If None, default class labels are created for the class numbers. Defaults to None.
- Raises:
ValueError – if length of y_true and y_pred are not equal.
ValueError – if not all detected class numbers are linked to a label in the specified class_mapping.
Methods:
accuracy()Compute accuracy from the confusion matrix.
average_accuracy([class_label])Compute average accuracy from the confusion matrix (which is fairer than accuracy for imbalanced data sets).
f1([class_label])false_negatives([class_label])Return the number of false negatives.
false_positives([class_label])Return the number of false positives.
kappa()Compute Cohen's kappa, i.e., inter-rater agreement (accuracy), corrected for chance.
precision([class_label])Compute precision from confusion matrix.
Return some general evaluation metrics (independent of which class is the positive outcome).
Return some evaluation metrics per class (considering one class as the positive outcome at a time).
sensitivity([class_label])Compute sensitivity from confusion matrix.
specificity([class_label])Compute specificity from confusion matrix.
support([class_label])Return the number of samples classified as the specified class_label in y_true.
true_negatives([class_label])Return the number of true negatives.
true_positives([class_label])Return the number of true positives.
- average_accuracy(class_label=None)[source]
Compute average accuracy from the confusion matrix (which is fairer than accuracy for imbalanced data sets).
- Returns:
(float) – average accuracy.
- false_negatives(class_label=None)[source]
Return the number of false negatives.
- Args:
- class_label (str, optional): class label that represents the positive outcome.
If None, the last class is taken
- .
Defaults to None.
- Returns:
(int): the number of false negatives.
- false_positives(class_label=None)[source]
Return the number of false positives.
- Parameters:
class_label (str, optional) – class label that represents the positive outcome. If None, the last class is taken. Defaults to None.
- Returns:
(int) – the number of false positives.
- kappa()[source]
Compute Cohen’s kappa, i.e., inter-rater agreement (accuracy), corrected for chance.
- Returns:
(float) – Cohen’s kappa.
- precision(class_label=None)[source]
Compute precision from confusion matrix.
- Parameters:
class_label (str, optional) – class label that represents the positive outcome. If None, the last class is taken. Defaults to None.
- Returns:
(float) – precision.
- scores_general()[source]
Return some general evaluation metrics (independent of which class is the positive outcome).
- Returns:
scores (dict) – dictionary with scores.
- scores_per_class()[source]
Return some evaluation metrics per class (considering one class as the positive outcome at a time).
- Returns:
df (pd.DataFrame) – pandas DataFrame with scores.
- sensitivity(class_label=None)[source]
Compute sensitivity from confusion matrix.
- Parameters:
class_label (str, optional) – class label that represents the positive outcome. If None, the last class is taken. Defaults to None.
- Returns:
(float) – sensitivity.
- specificity(class_label=None)[source]
Compute specificity from confusion matrix.
- Parameters:
class_label (str, optional) – class label that represents the positive outcome. If None, the last class is taken. Defaults to None.
- Returns:
(float) – specificity.
- support(class_label=None)[source]
Return the number of samples classified as the specified class_label in y_true.
- Parameters:
class_label (str, optional) – class label to count the support of. If None, the last class is taken. Defaults to None.
- Returns:
(int) – the support.
- nnsa.models.evaluation.average_silhouette(x, y)[source]
Return the average silhouette of clustered points.
Compares the average distance of a point to all points within its cluster to the distance of the point to the center of the closest other cluster.
- Parameters:
x (np.ndarray) – (n, m) array of n samples with m features.
y (np.ndarray) – (n, ) array of n samples with cluster/class numbers. May not contain np.nan values.
- Returns:
silhouette (float) – silhouette.
- nnsa.models.evaluation.binary_classification_scores(y_true, y_pred, y_score=None)[source]
Compute some scores for a binary classification algorithm by comparing predicted output to true labels.
- Parameters:
y_true (np.ndarray) – 1D array with true class numbers (0 or 1).
y_pred (np.ndarray, None) – 1D array with predicted class numbers (0 or 1). If None and y_score is given, y_pred is determined from y_score by applying a threshold that maximizes F1 score.
y_score (np.ndarray) – predicted scores/probabilities, see sklearn.metrics.roc_auc_score().
- Returns:
scores_merged (dict) – dictionary containing some scores for the classification.
- nnsa.models.evaluation.cluster_inertia(x, y)[source]
Return the inertia of clustered points.
- Parameters:
x (np.ndarray) – (n, m) array of n samples with m features. May not contain nan.
y (np.ndarray) – (n, ) array of n samples with cluster/class numbers. May not contain np.nan values.
- Returns:
inertia (float) – inertia.
- nnsa.models.evaluation.compute_separation_scores(df, y='y', n_combine=1, subsets=None, normalize=True, score_fun=None, dropna=True, n_permutations=99, min_score=-inf, n_jobs=None, verbose=1, seed=None)[source]
Compute scores and pvalues quantifying how well each combination of features in df separate predefined classes.
The predefined classes are determined by the class label, which is column y in df.
- The pvalue is determined using permutation test. More specifically, it implements Test 1 in:
Ojala and Garriga. Permutation Tests for Studying Classifier Performance. The Journal of Machine Learning Research (2010) vol. 11
- Parameters:
df (pd.DataFrame) – DataFrame with shape (n_samples, n_features + 1), containing the features and class label (y).
y (str, optional) – column in df containing the class labels.
n_combine (int, optional) – number of features to consider when making combinations.
subsets (list, optional) – list with length n_combine. Determines which features are combined. Elements are column names (list of strings) in the df. combinations are made by taking one features out of every list with column names. If specified, n_combine parameter is ignored.
normalize (bool, optional) – normalize the features (zero mean, unit SD).
score_fun (function, optional) – function that takes as input the feature vector X and the class labels y, i.e.: fun(X, y). It should return one number (e.g. how well features in X separate classes y). Higher score computed by score_fun must relate to better clustering.
dropna (bool, optional) – whether to drop samples with nan values. (NOTE: may be different for different combinations of features since nan values may occur only in certain features).
n_permutations (int, optional) – number of permutations to do for computing the pvalues. If 0 or None, no pvalues are computed.
min_score (float, optional) – minimum score for computing pvalue (pruning). If score is lower than min_score, pvalue is not computed and set to NaN.
n_jobs (int, optional) – Number of jobs to run in parallel when computing the pvalue using permutations.
verbose (int, optional) – verbose level. Choices: 0, 1, 2.
seed (int, optional) – seed for random generator for repeatability of permutation test.
- Returns:
df_scores (pd.DataFrame) – dataframe with scores for each possible combination of features, sorted by score (from high to low). The computed score depends on score_fun.
Examples
>>> # Create toy dataset. >>> np.random.seed(43) >>> df = pd.DataFrame() >>> for i in range(5): df['x{}'.format(i+1)] = np.random.rand(20) >>> df['y'] = df.x3 > df.x2 # x3 and x2 are explaining features for class y.
>>> # Compute scores. >>> n_combine = 2 # Number of features to combine (number of predictors). >>> y = 'y' # Column with class labels. >>> df_scores = compute_separation_scores(df, y=y, n_combine=n_combine, verbose=0) # Scores are sorted in decreasing order. >>> print(df_scores) # Note that the feature combination with highest score is features x2 and x3. score pvalue labels 0 0.365212 0.01 [x2, x3] 1 0.279842 0.01 [x1, x2] 2 0.232275 0.01 [x2, x4] 3 0.216710 0.02 [x2, x5] 4 0.106424 0.08 [x1, x3] 5 0.095038 0.03 [x3, x5] 6 0.094238 0.14 [x3, x4] 7 -0.002126 0.36 [x1, x4] 8 -0.031383 0.76 [x1, x5] 9 -0.059352 0.90 [x4, x5]
- nnsa.models.evaluation.fscore_carolina(y_true, y_score)[source]
Compute the F-score (TODO ask Mario for a reference).
Implemented for binary classification using 1 feature. This measure characterizes the distance of the feature values between the two clusters. The distance metric is normalized by the sum of the variance in each class. So intuitively, it is a ratio of the separation over the variance.
- Parameters:
y_true (np.ndarray) – array containing ones and zeros specifying the class per sample.
y_score (np.ndarray) – array containing the feature value of each sample.
- Returns:
f_score (float) – the F-score.
Examples
>>> np.random.seed(0) >>> f0 = np.random.normal(loc=4, scale=1, size=100000) >>> f1 = np.random.normal(loc=2, scale=1, size=100000) >>> y0 = np.zeros(len(f0)) >>> y1 = np.ones(len(f1)) >>> y_true = np.concatenate((y0, y1)) >>> y_score = np.concatenate((f0, f1)) >>> fscore = fscore_carolina(y_true, y_score) >>> print(fscore) # Expected score is 1. 1.0004474426175411
- nnsa.models.evaluation.roc_auc_score(y_true, y_score, remove_nans=True, **kwargs)[source]
Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores.
Wrapper of the sklearn.metrics.roc_auc_score() function that can handle NaN values and negative relation between y_true and y_score.
- Parameters:
y_true (np.ndarray) – true binary labels or binary label indicators, see sklearn.metrics.roc_auc_score().
y_score (np.ndarray) – predicted scores/probabilities, see sklearn.metrics.roc_auc_score().
**kwargs (optional) – keyword arguments for sklearn.metrics.roc_auc_score().
- Returns:
auc (float) – area under the ROC curve, or np.nan if this area is not defined.
- nnsa.models.evaluation.separation_score(f1, f2)[source]
Compute a score that captures how much two sets/distributions are separated.
It is 0.5 * the squared difference between the means of the two sets, normalized by the sum of their variances.
Ignores nan values.
It is equivalent to fscore_coralina if the sizes of both sets are equal. For unequal set sized, this metric is not dependend on the size of the sets, unlike fscore_carolina(). I.e., if two sets have a certain mean and std, increasing the numeber of samples in one of the sets does not significantly affect the score (apart from the influence on the computation of the std).
- Parameters:
f1 (np.ndarray) – array with values of first set (e.g. feature values).
f2 (np.ndarray) – array with values of second set (e.g. feature values).
- Returns:
score (float) – normalized score that increases if the separation of the two sets is larger.
Examples
>>> np.random.seed(43) >>> f1 = np.random.normal(loc=4, scale=1, size=1000) >>> f2 = np.random.normal(loc=2, scale=1, size=100000) >>> separation_score(f1, f2) # Expected score is 1. 1.0086296863621815
nnsa.models.losses module
Module containing loss functions for varying (parts of) models.
Functions:
|
Return the regularization loss of the parameter vector theta. |
- nnsa.models.losses.regularization_loss(theta, regulizer, lamb)[source]
Return the regularization loss of the parameter vector theta.
- Parameters:
theta (np.ndarray) – parameter vector.
regulizer (str) – type of regulizer. Choose from ‘L1’ (or ‘lasso’): loss = lamb * sum(abs(theta)) ‘L2’ (or ‘ridge’): loss = lamb * 1/2*sum(theta**2)
lamb (float) – lambda parameter (scale factor for the regularization loss).
- Returns:
reg_loss (float) – regularization loss for the given parameter vector theta.
nnsa.models.regressors module
High-level interface for several regressor models. Implemented in accordance with sklearn models.
Classes:
|
Class to train a model that predicts y from multiple measurements. |
|
Fit a linear model with custom regularization. |
- class nnsa.models.regressors.DualRegression(intercept=True, regulizer_a=None, regulizer_b=None, lambda_a=0.001, lambda_b=0.001, p_dropout=0.0)[source]
Bases:
objectClass to train a model that predicts y from multiple measurements.
- The model:
h(X, a, b) = np.dot(f(X, a), g(X, b)) = y
- where:
X is an array with shape n_measurements * n_features. y is the value to predict that corresponds to X. f(X, a) is a linear or logistic regression function that transforms X into a vector with length n_samples
such that sum(f(X)) == 1. a is a vector with parameters for the regression.
- g(X, b) is a linear or logistic regression function that transforms X into a vector with length n_samples.
b is a vector with parameters for the regression.
Note that X (n_measurements, n_features) corresponds only to one value y and that n_measurements may vary between train/test data. The model is trained on a list of X matrices and y values.
Intuition: g(X, a) tries to predict the value y from each measurement in X, where y is the same for each measurement. For some good measurements, y can possibly be predicted with high accuracy. For other, bad measurements (e.g. containing artefacts), the prediction may not be reliable. Therefore, a weighted sum is taken over the predictions of all measurements, where the weight for each measurement is computed by f(X, b). This weighted sum is trainable, i.e. the weight for each measurement is simply computed by another linear/logistic regression that maps the features to a weight.
TODO Examples:
Methods:
compute_normalization_pars(X_list[, method])Compute parameters to normalize the input data.
fit(X_list, y_list[, theta_0, seed])Fit the model.
normalize_input(X_list, center, scale)Normalize the input X_list.
predict(X_list)Predict output y from input X.
Attributes:
Theta is a vector containing all coefficients.
- static compute_normalization_pars(X_list, method='standard')[source]
Compute parameters to normalize the input data.
- Parameters:
X_list (list) – list with length n_samples, where entries are arrays with size (n_features, n_measurements).
method (bool, optional) – specify the method for scaling. If ‘standard’, center is the mean and scale is the standard deviation. This is not recommended if you are not using an intercept since the 1/(np.sum(aX)) term in the loss may blow up. If ‘range’, center is the 5th percentile and scale is the 95th percentile, so that scaled data have approximately a range between 0 and 1.
- Returns:
center (float) – the value with which to center the data.
scale (float) – the value with which to scale the data.
- fit(X_list, y_list, theta_0=None, seed=None, **kwargs)[source]
Fit the model.
- Parameters:
X_list (list) – list with train inputs for self.predict(). Shape of X_list[i] must be (n_features, n_measurements). The length of the list is the number of samples, so each entry in the list is one sample.
y_list (list) – list with the same lenght as X_list containing the target output values.
theta_0 (np.ndarray, optional) – initial values for the coefficients. Shape is 2*n_features if self.intercept is False or 2*n_features+2 if self.intercept is True. If None, coefficients are initialized randomly. Defaults to None.
seed (float, optional) – random seed. If None, no seed is used. Defaults to None.
**kwargs (optional) – optional keyword arguments for the optimizer function.
- Returns:
self – the current updated containing the fitted coefficients.
- static normalize_input(X_list, center, scale)[source]
Normalize the input X_list.
- Parameters:
X_list (list) – list with inputs for self.predict(). Shape of X_list[i] must be (n_features, n_measurements). The length of the list is the number of samples, so each entry in the list is one sample.
center (float) – the value with which to center the data.
scale (float) – the value with which to scale the data.
- Returns:
X_list (list) – normalized data.
- predict(X_list)[source]
Predict output y from input X.
- Parameters:
X_list (list) – list with inputs for self.predict(). Shape of X_list[i] must be (n_features, n_measurements). The length of the list is the number of samples, so each entry in the list is one sample. One output value is predicted per sample.
- Returns:
y_list (list) – predicted values for the samples in X_list.
- property theta
Theta is a vector containing all coefficients.
If split into two equal parts, the first half contains the coefficients a for f(X, a) and the second half contains the coefficients b for g(X, b). If intercept is True, the first entry of a and b is the intercept.
- Returns:
self._theta (np.ndarray) – coefficient vector.
- class nnsa.models.regressors.LinReg(regularizer=None, gamma=None, solver='fast')[source]
Bases:
objectFit a linear model with custom regularization.
- Parameters:
regularizer (str or np.ndarray or None, optional) – type of regularization. If None, no regularization is applied. If an array, this array is used as regularizer. If a string, a common regularizer is constructed. Options are ‘smooth’, ‘ridge’, ‘ols’ (see self.create_regularizer). Defaults to None.
gamma (float, optional) – regularization constant.
Attributes:
Return the regularization matrix.
Return the model coefficients.
Methods:
Compute the regularization cost.
create_regularizer(regularizer, n)Create regularization matrix based on the specific type of regularization.
fit(X, y)Fit the model coefficients to training data.
predict(X)Apply a fitted model to input X.
- property G
Return the regularization matrix.
- property coef_
Return the model coefficients.
- compute_reg_cost()[source]
Compute the regularization cost.
- Returns:
cost (float) – regularization cost (L2 norm of G*coef).
- static create_regularizer(regularizer, n)[source]
Create regularization matrix based on the specific type of regularization.
- Parameters:
regularizer – see class definition.
n (int, optional) – size of the square regularization matrix.
- Returns:
G (np.ndarray) – regularization matrix with shape (n, n).
Module contents
This module contains the building blocks for developing and training a classifier.