mlshell.Metric

class mlshell.Metric(scorer=None, oid=None, score_func=None, score_func_vector=None, greater_is_better=True, needs_proba=False, needs_threshold=False, needs_custom_kw_args=False)

Bases: object

Unified pipeline interface.

Implements interface to access arbitrary scorer. Interface: pprint and all underlying scorer methods.

scorer

Underlying scorer.

Type

callable, optional (default=None)

oid

Instance identifier.

Type

str, optional (default=None)

score_func

Scorer score function, return scalar value.

Type

callable, optional (default=None)

score_func_vector

Scorer vectorized score function, return vector of values for all samples.

Type

callable, optional (default=None)

greater_is_better

Whether score_func is a score function (default), meaning high is good, or a loss function, meaning low is good. In the latter case, the scorer object should sign-flip the outcome of the score_func.

Type

bool, optional (default=True)

needs_proba

Whether score_func requires predict_proba to get probability estimates out of a classifier.

Type

bool, optional (default=False)

needs_threshold

Whether score_func takes a continuous decision certainty. This only works for classification using estimators that have either a decision_function or predict_proba method.

Type

bool, optional (default=False)

needs_custom_kw_args

If True, before score evaluation extract scorer kwargs from pipeline ‘pass_custom’ step (if existed).

Type

bool, optional (default=False)

Notes

Extended scorer object:

• Additional needs_custom_kw_args kwarg.

Allows to optimize custom scorer kwargs as hyper-parameters.

• Additional score_func_vector kwarg.

Allows to evaluate vectorized score for more detailed analyze.

Attributes

kw_args

dict: Additional kwargs passed to score_func.

Methods

__call__(estimator, *args, **kwargs)	Redirect call to scorer object.
pprint(score)	Pretty print metric result.

__init__(scorer=None, oid=None, score_func=None, score_func_vector=None, greater_is_better=True, needs_proba=False, needs_threshold=False, needs_custom_kw_args=False)

Initialize self. See help(type(self)) for accurate signature.

Methods

__init__([scorer, oid, score_func, …])	Initialize self.
pprint(score)	Pretty print metric result.

Attributes

kw_args

dict: Additional kwargs passed to score_func.

property kw_args

dict: Additional kwargs passed to score_func.

pprint(score)

Pretty print metric result.

Parameters

score (any object) – score_func output.

Returns

score – Input converted to string.

Return type

str