Evaluate#
ragbits.evaluate.evaluator.Evaluator
#
Evaluator(batch_size: int = 10, num_retries: int = 3, backoff_multiplier: int = 1, backoff_max: int = 60)
Bases: WithConstructionConfig
Evaluator class.
Initialize the Evaluator instance.
PARAMETER | DESCRIPTION |
---|---|
batch_size |
batch size for the evaluation pipeline inference.
TYPE:
|
num_retries |
The number of retries per evaluation pipeline inference error.
TYPE:
|
backoff_multiplier |
The base delay multiplier for exponential backoff (in seconds).
TYPE:
|
backoff_max |
The maximum allowed delay (in seconds) between retries.
TYPE:
|
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/evaluator.py
subclass_from_config
classmethod
#
Initializes the class with the provided configuration. May return a subclass of the class, if requested by the configuration.
PARAMETER | DESCRIPTION |
---|---|
config |
A model containing configuration details for the class.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided configuration. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The class can't be found or is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
subclass_from_factory
classmethod
#
Creates the class using the provided factory function. May return a subclass of the class, if requested by the factory. Supports both synchronous and asynchronous factory functions.
PARAMETER | DESCRIPTION |
---|---|
factory_path |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided factory function. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The factory can't be found or the object returned is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
preferred_subclass
classmethod
#
preferred_subclass(config: CoreConfig, factory_path_override: str | None = None, yaml_path_override: Path | None = None) -> Self
Tries to create an instance by looking at project's component preferences, either from YAML or from the factory. Takes optional overrides for both, which takes a higher precedence.
PARAMETER | DESCRIPTION |
---|---|
config |
The CoreConfig instance containing preferred factory and configuration details.
TYPE:
|
factory_path_override |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
yaml_path_override |
A string representing the path to the YAML file containing the Ragstack instance configuration.
TYPE:
|
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
If the default factory or configuration can't be found. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
from_config
classmethod
#
Initializes the class with the provided configuration.
PARAMETER | DESCRIPTION |
---|---|
config |
A dictionary containing configuration details for the class.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided configuration. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
run_from_config
async
classmethod
#
Run the evaluation based on configuration.
PARAMETER | DESCRIPTION |
---|---|
config |
Evaluation config.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
EvaluatorResult
|
The evaluation results. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/evaluator.py
compute
async
#
compute(pipeline: EvaluationPipeline[EvaluationTargetT, EvaluationDataT, EvaluationResultT], dataloader: DataLoader[EvaluationDataT], metricset: MetricSet[EvaluationResultT]) -> EvaluatorResult[EvaluationResultT]
Compute the evaluation results for the given pipeline and data.
PARAMETER | DESCRIPTION |
---|---|
pipeline |
The pipeline to be evaluated.
TYPE:
|
dataloader |
The dataloader to load the data.
TYPE:
|
metricset |
The metrics to be computed.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
EvaluatorResult[EvaluationResultT]
|
The evaluation results. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/evaluator.py
ragbits.evaluate.optimizer.Optimizer
#
Bases: WithConstructionConfig
Optimizer class.
Initialize the pipeline optimizer.
PARAMETER | DESCRIPTION |
---|---|
direction |
Direction of optimization.
TYPE:
|
n_trials |
The number of trials for each process.
TYPE:
|
max_retries_for_trial |
The number of retires for single process.
TYPE:
|
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/optimizer.py
subclass_from_config
classmethod
#
Initializes the class with the provided configuration. May return a subclass of the class, if requested by the configuration.
PARAMETER | DESCRIPTION |
---|---|
config |
A model containing configuration details for the class.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided configuration. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The class can't be found or is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
subclass_from_factory
classmethod
#
Creates the class using the provided factory function. May return a subclass of the class, if requested by the factory. Supports both synchronous and asynchronous factory functions.
PARAMETER | DESCRIPTION |
---|---|
factory_path |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided factory function. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The factory can't be found or the object returned is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
preferred_subclass
classmethod
#
preferred_subclass(config: CoreConfig, factory_path_override: str | None = None, yaml_path_override: Path | None = None) -> Self
Tries to create an instance by looking at project's component preferences, either from YAML or from the factory. Takes optional overrides for both, which takes a higher precedence.
PARAMETER | DESCRIPTION |
---|---|
config |
The CoreConfig instance containing preferred factory and configuration details.
TYPE:
|
factory_path_override |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
yaml_path_override |
A string representing the path to the YAML file containing the Ragstack instance configuration.
TYPE:
|
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
If the default factory or configuration can't be found. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
from_config
classmethod
#
Initializes the class with the provided configuration.
PARAMETER | DESCRIPTION |
---|---|
config |
A dictionary containing configuration details for the class.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided configuration. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
run_from_config
classmethod
#
Run the optimization process configured with a config object.
PARAMETER | DESCRIPTION |
---|---|
config |
Optimizer config.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
list[tuple[dict, float, dict[str, float]]]
|
List of tested configs with associated scores and metrics. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/optimizer.py
optimize
#
optimize(pipeline_class: type[EvaluationPipeline], pipeline_config: dict, dataloader: DataLoader, metricset: MetricSet, callbacks: list[Callable] | None = None) -> list[tuple[dict, float, dict[str, float]]]
Run the optimization process for given parameters.
PARAMETER | DESCRIPTION |
---|---|
pipeline_class |
Pipeline to be optimized.
TYPE:
|
pipeline_config |
Configuration defining the optimization process.
TYPE:
|
dataloader |
Data loader.
TYPE:
|
metricset |
Metrics to be optimized.
TYPE:
|
callbacks |
Experiment callbacks.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
list[tuple[dict, float, dict[str, float]]]
|
List of tested configs with associated scores and metrics. |