torchbearer.metrics

Base Classes

The base metric classes exist to enable complex data flow requirements between metrics. All metrics are either instances of Metric or MetricFactory. These can then be collected in a MetricList or a MetricTree. The MetricList simply aggregates calls from a list of metrics, whereas the MetricTree will pass data from its root metric to each child and collect the outputs. This enables complex running metrics and statistics, without needing to compute the underlying values more than once. Typically, constructions of this kind should be handled using the decorator API.

class torchbearer.metrics.metrics.AdvancedMetric(name)

The AdvancedMetric class is a metric which provides different process methods for training and validation. This enables running metrics which do not output intermediate steps during validation.

Parameters:name (str) – The name of the metric.

eval()

Put the metric in eval mode.

process(*args)

Depending on the current mode, return the result of either ‘process_train’ or ‘process_validate’.

Parameters:state (dict) – The current state dict of the Model. Returns:The metric value.

process_final(*args)

Depending on the current mode, return the result of either ‘process_final_train’ or ‘process_final_validate’.

Parameters:state (dict) – The current state dict of the Model. Returns:The final metric value.

process_final_train(*args)

Process the given state and return the final metric value for a training iteration.

Parameters:state – The current state dict of the Model. Returns:The final metric value for a training iteration.

process_final_validate(*args)

Process the given state and return the final metric value for a validation iteration.

Parameters:state (dict) – The current state dict of the Model. Returns:The final metric value for a validation iteration.

process_train(*args)

Process the given state and return the metric value for a training iteration.

Parameters:state – The current state dict of the Model. Returns:The metric value for a training iteration.

process_validate(*args)

Process the given state and return the metric value for a validation iteration.

Parameters:state – The current state dict of the Model. Returns:The metric value for a validation iteration.

train()

Put the metric in train mode.

class torchbearer.metrics.metrics.Metric(name)

Base metric class. Process will be called on each batch, process-final at the end of each epoch. The metric contract allows for metrics to take any args but not kwargs. The initial metric call will be given state, however, subsequent metrics can pass any values desired.

Note

All metrics must extend this class.

Parameters:name (str) – The name of the metric

eval()

Put the metric in eval mode during model validation.

process(*args)

Process the state and update the metric for one iteration.

Parameters:args – Arguments given to the metric. If this is a root level metric, will be given state Returns:None, or the value of the metric for this batch

process_final(*args)

Process the terminal state and output the final value of the metric.

Parameters:args – Arguments given to the metric. If this is a root level metric, will be given state Returns:None or the value of the metric for this epoch

reset(state)

Reset the metric, called before the start of an epoch.

Parameters:state – The current state dict of the Model.

train()

Put the metric in train mode during model training.

class torchbearer.metrics.metrics.MetricFactory

A simple implementation of a factory pattern. Used to enable construction of complex metrics using decorators.

build()

Build and return a usable Metric instance.

Returns:The constructed Metric

class torchbearer.metrics.metrics.MetricList(metric_list)

The MetricList class is a wrapper for a list of metrics which acts as a single metric and produces a dictionary of outputs.

Parameters:metric_list (list) – The list of metrics to be wrapped. If the list contains a MetricList, this will be unwrapped. Any strings in the list will be retrieved from metrics.DEFAULT_METRICS.

eval()

Put each metric in eval mode

process(state)

Process each metric an wrap in a dictionary which maps metric names to values.

Parameters:state – The current state dict of the Model. Returns:dict[str,any] – A dictionary which maps metric names to values.

process_final(state)

Process each metric an wrap in a dictionary which maps metric names to values.

Parameters:state – The current state dict of the Model. Returns:dict[str,any] – A dictionary which maps metric names to values.

reset(state)

Reset each metric with the given state.

Parameters:state – The current state dict of the Model.

train()

Put each metric in train mode.

class torchbearer.metrics.metrics.MetricTree(metric)

A tree structure which has a node Metric and some children. Upon execution, the node is called with the input and its output is passed to each of the children. A dict is updated with the results.

Parameters:metric (Metric) – The metric to act as the root node of the tree / subtree

add_child(child)

Add a child to this node of the tree

Parameters:child (Metric) – The child to add Returns:None

eval()

Put the metric in eval mode during model validation.

process(*args)

Process this node and then pass the output to each child.

Returns:A dict containing all results from the children

process_final(*args)

Process this node and then pass the output to each child.

Returns:A dict containing all results from the children

reset(state)

Reset the metric, called before the start of an epoch.

Parameters:state – The current state dict of the Model.

train()

Put the metric in train mode during model training.

Decorators - The Decorator API

The decorator API is the core way to interact with metrics in torchbearer. All of the classes and functionality handled here can be reproduced by manually interacting with the classes if necessary. Broadly speaking, the decorator API is used to construct a MetricFactory which will build a MetricTree that handles data flow between instances of Mean, RunningMean, Std etc.

torchbearer.metrics.decorators.default_for_key(key)

The default_for_key() decorator will register the given metric in the global metric dict (metrics.DEFAULT_METRICS) so that it can be referenced by name in instances of MetricList such as in the list given to the torchbearer.Model.

Example:

@default_for_key('acc')
class CategoricalAccuracy(metrics.BatchLambda):
    ...

Parameters:key (str) – The key to use when referencing the metric

torchbearer.metrics.decorators.lambda_metric(name, on_epoch=False)

The lambda_metric() decorator is used to convert a lambda function y_pred, y_true into a Metric instance. In fact it return a MetricFactory which is used to build a Metric. This can make things complicated as in the following example:

@metrics.lambda_metric('my_metric')
def my_metric(y_pred, y_true):
    ... # Calculate some metric

model = Model(metrics=[my_metric()]) # Note we have to call `my_metric` in order to instantiate the class

Parameters:

• name – The name of the metric (e.g. ‘loss’)
• on_epoch – If True the metric will be an instance of EpochLambda instead of BatchLambda

Returns:

A decorator which replaces a function with a MetricFactory

torchbearer.metrics.decorators.mean(clazz)

The mean() decorator is used to add a Mean to the MetricTree which will will output a mean value at the end of each epoch. At build time, if the inner class is not a MetricTree, one will be created. The Mean will also be wrapped in a ToDict for simplicity.

Example:

>>> import torch
>>> from torchbearer import metrics

>>> @metrics.mean
... @metrics.lambda_metric('my_metric')
... def my_metric(y_pred, y_true):
...     return y_pred + y_true
...
>>> metric = my_metric().build()
>>> metric.reset({})
>>> metric.process({'y_pred':torch.Tensor([2]), 'y_true':torch.Tensor([2])}) # 4
{}
>>> metric.process({'y_pred':torch.Tensor([3]), 'y_true':torch.Tensor([3])}) # 6
{}
>>> metric.process({'y_pred':torch.Tensor([4]), 'y_true':torch.Tensor([4])}) # 8
{}
>>> metric.process_final()
{'my_metric': 6.0}

Parameters:clazz – The class to decorate Returns:A MetricFactory which can be instantiated and built to append a Mean to the MetricTree

torchbearer.metrics.decorators.running_mean(clazz=None, batch_size=50, step_size=10)

The running_mean() decorator is used to add a RunningMean to the MetricTree. As with the other decorators, a MetricFactory is created which will do this upon the call to MetricFactory.build(). If the inner class is not / does not build a MetricTree then one will be created. The RunningMean will be wrapped in a ToDict (with ‘running_’ prepended to the name) for simplicity.

Note

The decorator function does not need to be called if not desired, both: @running_mean and @running_mean() are acceptable.

Example:

>>> import torch
>>> from torchbearer import metrics

>>> @metrics.running_mean(step_size=2) # Update every 2 steps
... @metrics.lambda_metric('my_metric')
... def my_metric(y_pred, y_true):
...     return y_pred + y_true
...
>>> metric = my_metric().build()
>>> metric.reset({})
>>> metric.process({'y_pred':torch.Tensor([2]), 'y_true':torch.Tensor([2])}) # 4
{'running_my_metric': 4.0}
>>> metric.process({'y_pred':torch.Tensor([3]), 'y_true':torch.Tensor([3])}) # 6
{'running_my_metric': 4.0}
>>> metric.process({'y_pred':torch.Tensor([4]), 'y_true':torch.Tensor([4])}) # 8, triggers update
{'running_my_metric': 6.0}

Parameters:

• clazz – The class to decorate
• batch_size – See RunningMean
• step_size – See RunningMean

Returns:

decorator or MetricFactory

torchbearer.metrics.decorators.std(clazz)

The std() decorator is used to add a Std to the MetricTree which will will output a population standard deviation value at the end of each epoch. At build time, if the inner class is not a MetricTree, one will be created. The Std will also be wrapped in a ToDict (with ‘_std’ appended) for simplicity.

Example:

>>> import torch
>>> from torchbearer import metrics

>>> @metrics.std
... @metrics.lambda_metric('my_metric')
... def my_metric(y_pred, y_true):
...     return y_pred + y_true
...
>>> metric = my_metric().build()
>>> metric.reset({})
>>> metric.process({'y_pred':torch.Tensor([2]), 'y_true':torch.Tensor([2])}) # 4
{}
>>> metric.process({'y_pred':torch.Tensor([3]), 'y_true':torch.Tensor([3])}) # 6
{}
>>> metric.process({'y_pred':torch.Tensor([4]), 'y_true':torch.Tensor([4])}) # 8
{}
>>> '%.4f' % metric.process_final()['my_metric_std']
'1.6330'

Parameters:clazz – The class to decorate Returns:A MetricFactory which can be instantiated and built to append a Mean to the MetricTree

torchbearer.metrics.decorators.to_dict(clazz)

The to_dict() decorator is used to wrap either a Metric or MetricFactory instance with a ToDict instance. The result is that future output will be wrapped in a dict[name, value].

Example:

>>> from torchbearer import metrics

>>> @metrics.lambda_metric('my_metric')
... def my_metric(y_pred, y_true):
...     return y_pred + y_true
...
>>> my_metric().build().process({'y_pred':4, 'y_true':5})
9

>>> @metrics.to_dict
... @metrics.lambda_metric('my_metric')
... def my_metric(y_pred, y_true):
...     return y_pred + y_true
...
>>> my_metric().build().process({'y_pred':4, 'y_true':5})
{'my_metric': 9}

Parameters:clazz – The class to decorate Returns:A MetricFactory which can be instantiated and built to wrap the given class in a ToDict

Metric Wrappers

Metric wrappers are classes which wrap instances of Metric or, in the case of EpochLambda and BatchLambda, functions. Typically, these should not be used directly (although this is entirely possible), but via the decorator API.

class torchbearer.metrics.wrappers.BatchLambda(name, metric_function)

A metric which returns the output of the given function on each batch.

Parameters:

• name (str) – The name of the metric.
• metric_function – A metric function(‘y_pred’, ‘y_true’) to wrap.

process(state)

Return the output of the wrapped function.

Parameters:state (dict) – The torchbearer.Model state. Returns:The value of the metric function(‘y_pred’, ‘y_true’).

class torchbearer.metrics.wrappers.EpochLambda(name, metric_function, running=True, step_size=50)

A metric wrapper which computes the given function for concatenated values of ‘y_true’ and ‘y_pred’ each epoch. Can be used as a running metric which computes the function for batches of outputs with a given step size during training.

Parameters:

• name (str) – The name of the metric.
• metric_function – The function(‘y_pred’, ‘y_true’) to use as the metric.
• running (bool) – True if this should act as a running metric.
• step_size (int) – Step size to use between calls if running=True.

process_final_train(state)

Evaluate the function with the aggregated outputs.

Parameters:state (dict) – The torchbearer.Model state. Returns:The result of the function.

process_final_validate(state)

Evaluate the function with the aggregated outputs.

Parameters:state (dict) – The torchbearer.Model state. Returns:The result of the function.

process_train(state)

Concatenate the ‘y_true’ and ‘y_pred’ from the state along the 0 dimension. If this is a running metric, evaluates the function every number of steps.

Parameters:state (dict) – The torchbearer.Model state. Returns:The current running result.

process_validate(state)

During validation, just concatenate ‘y_true’ and y_pred’.

Parameters:state (dict) – The torchbearer.Model state.

reset(state)

Reset the ‘y_true’ and ‘y_pred’ caches.

Parameters:state (dict) – The torchbearer.Model state.

class torchbearer.metrics.wrappers.ToDict(metric)

The ToDict class is an AdvancedMetric which will put output from the inner Metric in a dict (mapping metric name to value) before returning. When in eval mode, ‘val_’ will be prepended to the metric name.

Example:

>>> from torchbearer import metrics

>>> @metrics.lambda_metric('my_metric')
... def my_metric(y_pred, y_true):
...     return y_pred + y_true
...
>>> metric = metrics.ToDict(my_metric().build())
>>> metric.process({'y_pred': 4, 'y_true': 5})
{'my_metric': 9}
>>> metric.eval()
>>> metric.process({'y_pred': 4, 'y_true': 5})
{'val_my_metric': 9}

Parameters:metric (metrics.Metric) – The Metric instance to wrap.

eval()

Put the metric in eval mode.

process_final_train(*args)

Process the given state and return the final metric value for a training iteration.

Parameters:state – The current state dict of the Model. Returns:The final metric value for a training iteration.

process_final_validate(*args)

Process the given state and return the final metric value for a validation iteration.

Parameters:state (dict) – The current state dict of the Model. Returns:The final metric value for a validation iteration.

process_train(*args)

Process the given state and return the metric value for a training iteration.

Parameters:state – The current state dict of the Model. Returns:The metric value for a training iteration.

process_validate(*args)

Process the given state and return the metric value for a validation iteration.

Parameters:state – The current state dict of the Model. Returns:The metric value for a validation iteration.

reset(state)

Reset the metric, called before the start of an epoch.

Parameters:state – The current state dict of the Model.

train()

Put the metric in train mode.

Metric Aggregators

Aggregators are a special kind of Metric which takes as input, the output from a previous metric or metrics. As a result, via a MetricTree, a series of aggregators can collect statistics such as Mean or Standard Deviation without needing to compute the underlying metric multiple times. This can, however, make the aggregators complex to use. It is therefore typically better to use the decorator API.

class torchbearer.metrics.aggregators.Mean(name)

Metric aggregator which calculates the mean of process outputs between calls to reset.

Parameters:name (str) – The name of this metric.

process(data)

Add the input to the rolling sum.

Parameters:data (torch.Tensor) – The output of some previous call to Metric.process().

process_final(data)

Compute and return the mean of all metric values since the last call to reset.

Parameters:data (torch.Tensor) – The output of some previous call to Metric.process_final(). Returns:The mean of the metric values since the last call to reset.

reset(state)

Reset the running count and total.

Parameters:state (dict) – The model state.

class torchbearer.metrics.aggregators.RunningMean(name, batch_size=50, step_size=10)

A RunningMetric which outputs the mean of a sequence of its input over the course of an epoch.

Parameters:

• name (str) – The name of this running mean.
• batch_size (int) – The size of the deque to store of previous results.
• step_size (int) – The number of iterations between aggregations.

class torchbearer.metrics.aggregators.RunningMetric(name, batch_size=50, step_size=10)

A metric which aggregates batches of results and presents a method to periodically process these into a value.

Note

Running metrics only provide output during training.

Parameters:

• name (str) – The name of the metric.
• batch_size (int) – The size of the deque to store of previous results.
• step_size (int) – The number of iterations between aggregations.

process_train(*args)

Add the current metric value to the cache and call ‘_step’ is needed.

Parameters:args – The output of some Metric Returns:The current metric value.

reset(state)

Reset the step counter. Does not clear the cache.

Parameters:state (dict) – The current model state.

class torchbearer.metrics.aggregators.Std(name)

Metric aggregator which calculates the standard deviation of process outputs between calls to reset.

Parameters:name (str) – The name of this metric.

process(data)

Compute values required for the std from the input.

Parameters:data (torch.Tensor) – The output of some previous call to Metric.process().

process_final(data)

Compute and return the final standard deviation.

Parameters:data (torch.Tensor) – The output of some previous call to Metric.process_final(). Returns:The standard deviation of each observation since the last reset call.

reset(state)

Reset the statistics to compute the next deviation.

Parameters:state (dict) – The model state.

Base Metrics

Base metrics are the base classes which represent the metrics supplied with torchbearer. The all use the default_for_key() decorator so that they can be accessed in the call to torchbearer.Model via the following strings:

• ‘acc’ or ‘accuracy’: The CategoricalAccuracy metric
• ‘loss’: The Loss metric
• ‘epoch’: The Epoch metric
• ‘roc_auc’ or ‘roc_auc_score’: The RocAucScore metric

class torchbearer.metrics.primitives.CategoricalAccuracy

Categorical accuracy metric. Uses torch.max to determine predictions and compares to targets.

class torchbearer.metrics.primitives.Epoch

Returns the ‘epoch’ from the model state.

process(state)

Process the state and update the metric for one iteration.

Parameters:args – Arguments given to the metric. If this is a root level metric, will be given state Returns:None, or the value of the metric for this batch

process_final(state)

Process the terminal state and output the final value of the metric.

Parameters:args – Arguments given to the metric. If this is a root level metric, will be given state Returns:None or the value of the metric for this epoch

class torchbearer.metrics.primitives.Loss

Simply returns the ‘loss’ value from the model state.

process(state)

Process the state and update the metric for one iteration.

Parameters:args – Arguments given to the metric. If this is a root level metric, will be given state Returns:None, or the value of the metric for this batch