mlshell.Metric¶

class mlshell.Metric(scorer=None, oid=None, score_func=None, score_func_vector=None, greater_is_better=True, needs_proba=False, needs_threshold=False, needs_custom_kw_args=False)[source]¶

Bases: object

Unified pipeline interface.

Implements interface to access arbitrary scorer. Interface: pprint and all underlying scorer methods.

scorer¶

Underlying scorer.

Type: callable, optional (default=None)

oid¶

Instance identifier.

Type: str, optional (default=None)

score_func¶

Scorer score function, return scalar value.

Type: callable, optional (default=None)

score_func_vector¶

Scorer vectorized score function, return vector of values for all samples.

Type: callable, optional (default=None)

greater_is_better¶

Whether score_func is a score function (default), meaning high is good, or a loss function, meaning low is good. In the latter case, the scorer object should sign-flip the outcome of the score_func.

Type: bool, optional (default=True)

needs_proba¶

Whether score_func requires predict_proba to get probability estimates out of a classifier.

Type: bool, optional (default=False)

needs_threshold¶

Whether score_func takes a continuous decision certainty. This only works for classification using estimators that have either a decision_function or predict_proba method.

Type: bool, optional (default=False)

needs_custom_kw_args¶

If True, before score evaluation extract scorer kwargs from pipeline ‘pass_custom’ step (if existed).

Type: bool, optional (default=False)

Notes

Extended scorer object:

Additional needs_custom_kw_args kwarg.

Allows to optimize custom scorer kwargs as hyper-parameters.

Additional score_func_vector kwarg.

Allows to evaluate vectorized score for more detailed analyze.

Attributes

kw_args: dict: Additional kwargs passed to score_func.

Methods

`__call__`(estimator, args, *kwargs)	Redirect call to scorer object.
`pprint`(score)	Pretty print metric result.

__init__(scorer=None, oid=None, score_func=None, score_func_vector=None, greater_is_better=True, needs_proba=False, needs_threshold=False, needs_custom_kw_args=False)[source]¶: Initialize self. See help(type(self)) for accurate signature.

Methods

`__init__`([scorer, oid, score_func, …])	Initialize self.
`pprint`(score)	Pretty print metric result.

Attributes

kw_args

dict: Additional kwargs passed to score_func.

property kw_args¶: dict: Additional kwargs passed to score_func.

pprint(score)[source]¶

Pretty print metric result.

Parameters: score (any object) – score_func output.
Returns: score – Input converted to string.
Return type: str