Metrics#
ragbits.evaluate.metrics.base.MetricSet
#
MetricSet(*metrics: Metric[EvaluationResultT])
Bases: WithConstructionConfig
, Generic[EvaluationResultT]
Represents a set of metrics.
Initialize the metric set.
PARAMETER | DESCRIPTION |
---|---|
metrics |
The metrics.
TYPE:
|
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/base.py
subclass_from_config
classmethod
#
Initializes the class with the provided configuration. May return a subclass of the class, if requested by the configuration.
PARAMETER | DESCRIPTION |
---|---|
config |
A model containing configuration details for the class.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided configuration. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The class can't be found or is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
subclass_from_factory
classmethod
#
Creates the class using the provided factory function. May return a subclass of the class, if requested by the factory. Supports both synchronous and asynchronous factory functions.
PARAMETER | DESCRIPTION |
---|---|
factory_path |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided factory function. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The factory can't be found or the object returned is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
preferred_subclass
classmethod
#
preferred_subclass(config: CoreConfig, factory_path_override: str | None = None, yaml_path_override: Path | None = None) -> Self
Tries to create an instance by looking at project's component preferences, either from YAML or from the factory. Takes optional overrides for both, which takes a higher precedence.
PARAMETER | DESCRIPTION |
---|---|
config |
The CoreConfig instance containing preferred factory and configuration details.
TYPE:
|
factory_path_override |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
yaml_path_override |
A string representing the path to the YAML file containing the Ragstack instance configuration.
TYPE:
|
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
If the default factory or configuration can't be found. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
from_config
classmethod
#
Create an instance of MetricSet
from a configuration dictionary.
PARAMETER | DESCRIPTION |
---|---|
config |
A dictionary containing configuration settings for the metric set.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the metric set class initialized with the provided configuration. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/base.py
compute
async
#
Compute the metrics.
PARAMETER | DESCRIPTION |
---|---|
results |
The evaluation results.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
dict
|
The computed metrics. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/base.py
ragbits.evaluate.metrics.base.Metric
#
Bases: WithConstructionConfig
, Generic[EvaluationResultT]
, ABC
Base class for metrics.
Initialize the metric.
PARAMETER | DESCRIPTION |
---|---|
weight |
Metric value weight in the final score, used during optimization.
TYPE:
|
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/base.py
subclass_from_config
classmethod
#
Initializes the class with the provided configuration. May return a subclass of the class, if requested by the configuration.
PARAMETER | DESCRIPTION |
---|---|
config |
A model containing configuration details for the class.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided configuration. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The class can't be found or is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
subclass_from_factory
classmethod
#
Creates the class using the provided factory function. May return a subclass of the class, if requested by the factory. Supports both synchronous and asynchronous factory functions.
PARAMETER | DESCRIPTION |
---|---|
factory_path |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided factory function. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The factory can't be found or the object returned is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
preferred_subclass
classmethod
#
preferred_subclass(config: CoreConfig, factory_path_override: str | None = None, yaml_path_override: Path | None = None) -> Self
Tries to create an instance by looking at project's component preferences, either from YAML or from the factory. Takes optional overrides for both, which takes a higher precedence.
PARAMETER | DESCRIPTION |
---|---|
config |
The CoreConfig instance containing preferred factory and configuration details.
TYPE:
|
factory_path_override |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
yaml_path_override |
A string representing the path to the YAML file containing the Ragstack instance configuration.
TYPE:
|
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
If the default factory or configuration can't be found. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
from_config
classmethod
#
Initializes the class with the provided configuration.
PARAMETER | DESCRIPTION |
---|---|
config |
A dictionary containing configuration details for the class.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided configuration. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
compute
abstractmethod
async
#
Compute the metric.
PARAMETER | DESCRIPTION |
---|---|
results |
The evaluation results.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
dict
|
The computed metric. |
ragbits.evaluate.metrics.document_search.DocumentSearchMetric
#
Bases: Metric[DocumentSearchResult]
, ABC
Metric for document search evaluation based on Relari backend. More details can be found here.
Initialize the document search metric.
PARAMETER | DESCRIPTION |
---|---|
matching_strategy |
Matching strategys that determine relevance.
TYPE:
|
weight |
Metric value weight in the final score, used during optimization.
TYPE:
|
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/document_search.py
subclass_from_config
classmethod
#
Initializes the class with the provided configuration. May return a subclass of the class, if requested by the configuration.
PARAMETER | DESCRIPTION |
---|---|
config |
A model containing configuration details for the class.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided configuration. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The class can't be found or is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
subclass_from_factory
classmethod
#
Creates the class using the provided factory function. May return a subclass of the class, if requested by the factory. Supports both synchronous and asynchronous factory functions.
PARAMETER | DESCRIPTION |
---|---|
factory_path |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided factory function. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The factory can't be found or the object returned is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
preferred_subclass
classmethod
#
preferred_subclass(config: CoreConfig, factory_path_override: str | None = None, yaml_path_override: Path | None = None) -> Self
Tries to create an instance by looking at project's component preferences, either from YAML or from the factory. Takes optional overrides for both, which takes a higher precedence.
PARAMETER | DESCRIPTION |
---|---|
config |
The CoreConfig instance containing preferred factory and configuration details.
TYPE:
|
factory_path_override |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
yaml_path_override |
A string representing the path to the YAML file containing the Ragstack instance configuration.
TYPE:
|
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
If the default factory or configuration can't be found. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
from_config
classmethod
#
Create an instance of DocumentSearchMetric
from a configuration dictionary.
PARAMETER | DESCRIPTION |
---|---|
config |
A dictionary containing configuration settings for the metric.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the metric class initialized with the provided configuration. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/document_search.py
compute
async
#
Compute the metric.
PARAMETER | DESCRIPTION |
---|---|
results |
The evaluation results.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
dict
|
The computed metric. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/document_search.py
ragbits.evaluate.metrics.document_search.DocumentSearchPrecisionRecallF1
#
Bases: DocumentSearchMetric
Precision, recall, and F1 score for context retrieval. More details can be found here.
Initialize the document search metric.
PARAMETER | DESCRIPTION |
---|---|
matching_strategy |
Matching strategys that determine relevance.
TYPE:
|
weight |
Metric value weight in the final score, used during optimization.
TYPE:
|
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/document_search.py
subclass_from_config
classmethod
#
Initializes the class with the provided configuration. May return a subclass of the class, if requested by the configuration.
PARAMETER | DESCRIPTION |
---|---|
config |
A model containing configuration details for the class.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided configuration. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The class can't be found or is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
subclass_from_factory
classmethod
#
Creates the class using the provided factory function. May return a subclass of the class, if requested by the factory. Supports both synchronous and asynchronous factory functions.
PARAMETER | DESCRIPTION |
---|---|
factory_path |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided factory function. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The factory can't be found or the object returned is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
preferred_subclass
classmethod
#
preferred_subclass(config: CoreConfig, factory_path_override: str | None = None, yaml_path_override: Path | None = None) -> Self
Tries to create an instance by looking at project's component preferences, either from YAML or from the factory. Takes optional overrides for both, which takes a higher precedence.
PARAMETER | DESCRIPTION |
---|---|
config |
The CoreConfig instance containing preferred factory and configuration details.
TYPE:
|
factory_path_override |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
yaml_path_override |
A string representing the path to the YAML file containing the Ragstack instance configuration.
TYPE:
|
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
If the default factory or configuration can't be found. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
from_config
classmethod
#
Create an instance of DocumentSearchMetric
from a configuration dictionary.
PARAMETER | DESCRIPTION |
---|---|
config |
A dictionary containing configuration settings for the metric.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the metric class initialized with the provided configuration. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/document_search.py
compute
async
#
Compute the metric.
PARAMETER | DESCRIPTION |
---|---|
results |
The evaluation results.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
dict
|
The computed metric. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/document_search.py
ragbits.evaluate.metrics.document_search.DocumentSearchRankedRetrievalMetrics
#
Bases: DocumentSearchMetric
Rank-aware metrics takes into account the order in which the contexts are retrieved. More details can be found here.
Initialize the document search metric.
PARAMETER | DESCRIPTION |
---|---|
matching_strategy |
Matching strategys that determine relevance.
TYPE:
|
weight |
Metric value weight in the final score, used during optimization.
TYPE:
|
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/document_search.py
subclass_from_config
classmethod
#
Initializes the class with the provided configuration. May return a subclass of the class, if requested by the configuration.
PARAMETER | DESCRIPTION |
---|---|
config |
A model containing configuration details for the class.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided configuration. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The class can't be found or is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
subclass_from_factory
classmethod
#
Creates the class using the provided factory function. May return a subclass of the class, if requested by the factory. Supports both synchronous and asynchronous factory functions.
PARAMETER | DESCRIPTION |
---|---|
factory_path |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided factory function. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The factory can't be found or the object returned is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
preferred_subclass
classmethod
#
preferred_subclass(config: CoreConfig, factory_path_override: str | None = None, yaml_path_override: Path | None = None) -> Self
Tries to create an instance by looking at project's component preferences, either from YAML or from the factory. Takes optional overrides for both, which takes a higher precedence.
PARAMETER | DESCRIPTION |
---|---|
config |
The CoreConfig instance containing preferred factory and configuration details.
TYPE:
|
factory_path_override |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
yaml_path_override |
A string representing the path to the YAML file containing the Ragstack instance configuration.
TYPE:
|
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
If the default factory or configuration can't be found. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
from_config
classmethod
#
Create an instance of DocumentSearchMetric
from a configuration dictionary.
PARAMETER | DESCRIPTION |
---|---|
config |
A dictionary containing configuration settings for the metric.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the metric class initialized with the provided configuration. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/document_search.py
compute
async
#
Compute the metric.
PARAMETER | DESCRIPTION |
---|---|
results |
The evaluation results.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
dict
|
The computed metric. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/document_search.py
ragbits.evaluate.metrics.question_answer.QuestionAnswerMetric
#
QuestionAnswerMetric(llm: LLM, batch_size: int = 15, weight: float = 1.0)
Bases: Generic[MetricT]
, Metric[QuestionAnswerResult]
, ABC
Metric for question answer evaluation based on Relari backend. More details can be found here.
Initialize the agent metric.
PARAMETER | DESCRIPTION |
---|---|
llm |
Judge LLM instance.
TYPE:
|
batch_size |
Batch size for metric computation.
TYPE:
|
weight |
Metric value weight in the final score, used during optimization.
TYPE:
|
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/question_answer.py
subclass_from_config
classmethod
#
Initializes the class with the provided configuration. May return a subclass of the class, if requested by the configuration.
PARAMETER | DESCRIPTION |
---|---|
config |
A model containing configuration details for the class.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided configuration. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The class can't be found or is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
subclass_from_factory
classmethod
#
Creates the class using the provided factory function. May return a subclass of the class, if requested by the factory. Supports both synchronous and asynchronous factory functions.
PARAMETER | DESCRIPTION |
---|---|
factory_path |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided factory function. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The factory can't be found or the object returned is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
preferred_subclass
classmethod
#
preferred_subclass(config: CoreConfig, factory_path_override: str | None = None, yaml_path_override: Path | None = None) -> Self
Tries to create an instance by looking at project's component preferences, either from YAML or from the factory. Takes optional overrides for both, which takes a higher precedence.
PARAMETER | DESCRIPTION |
---|---|
config |
The CoreConfig instance containing preferred factory and configuration details.
TYPE:
|
factory_path_override |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
yaml_path_override |
A string representing the path to the YAML file containing the Ragstack instance configuration.
TYPE:
|
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
If the default factory or configuration can't be found. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
from_config
classmethod
#
Create an instance of QuestionAnswerMetric
from a configuration dictionary.
PARAMETER | DESCRIPTION |
---|---|
config |
A dictionary containing configuration settings for the metric.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the metric class initialized with the provided configuration. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/question_answer.py
compute
async
#
Compute the metric.
PARAMETER | DESCRIPTION |
---|---|
results |
The evaluation results.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
dict
|
The computed metric. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/question_answer.py
ragbits.evaluate.metrics.question_answer.QuestionAnswerAnswerCorrectness
#
QuestionAnswerAnswerCorrectness(llm: LLM, batch_size: int = 15, weight: float = 1.0)
Bases: QuestionAnswerMetric[LLMBasedAnswerCorrectness]
Metric checking answer correctness based on LLM. More details can be found here.
Initialize the agent metric.
PARAMETER | DESCRIPTION |
---|---|
llm |
Judge LLM instance.
TYPE:
|
batch_size |
Batch size for metric computation.
TYPE:
|
weight |
Metric value weight in the final score, used during optimization.
TYPE:
|
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/question_answer.py
metric_cls
class-attribute
instance-attribute
#
subclass_from_config
classmethod
#
Initializes the class with the provided configuration. May return a subclass of the class, if requested by the configuration.
PARAMETER | DESCRIPTION |
---|---|
config |
A model containing configuration details for the class.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided configuration. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The class can't be found or is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
subclass_from_factory
classmethod
#
Creates the class using the provided factory function. May return a subclass of the class, if requested by the factory. Supports both synchronous and asynchronous factory functions.
PARAMETER | DESCRIPTION |
---|---|
factory_path |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided factory function. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The factory can't be found or the object returned is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
preferred_subclass
classmethod
#
preferred_subclass(config: CoreConfig, factory_path_override: str | None = None, yaml_path_override: Path | None = None) -> Self
Tries to create an instance by looking at project's component preferences, either from YAML or from the factory. Takes optional overrides for both, which takes a higher precedence.
PARAMETER | DESCRIPTION |
---|---|
config |
The CoreConfig instance containing preferred factory and configuration details.
TYPE:
|
factory_path_override |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
yaml_path_override |
A string representing the path to the YAML file containing the Ragstack instance configuration.
TYPE:
|
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
If the default factory or configuration can't be found. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
from_config
classmethod
#
Create an instance of QuestionAnswerMetric
from a configuration dictionary.
PARAMETER | DESCRIPTION |
---|---|
config |
A dictionary containing configuration settings for the metric.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the metric class initialized with the provided configuration. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/question_answer.py
compute
async
#
Compute the metric.
PARAMETER | DESCRIPTION |
---|---|
results |
The evaluation results.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
dict
|
The computed metric. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/question_answer.py
ragbits.evaluate.metrics.question_answer.QuestionAnswerAnswerFaithfulness
#
QuestionAnswerAnswerFaithfulness(llm: LLM, batch_size: int = 15, weight: float = 1.0)
Bases: QuestionAnswerMetric[LLMBasedFaithfulness]
Metric checking answer faithfulness based on LLM. More details can be found here.
Initialize the agent metric.
PARAMETER | DESCRIPTION |
---|---|
llm |
Judge LLM instance.
TYPE:
|
batch_size |
Batch size for metric computation.
TYPE:
|
weight |
Metric value weight in the final score, used during optimization.
TYPE:
|
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/question_answer.py
metric_cls
class-attribute
instance-attribute
#
subclass_from_config
classmethod
#
Initializes the class with the provided configuration. May return a subclass of the class, if requested by the configuration.
PARAMETER | DESCRIPTION |
---|---|
config |
A model containing configuration details for the class.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided configuration. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The class can't be found or is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
subclass_from_factory
classmethod
#
Creates the class using the provided factory function. May return a subclass of the class, if requested by the factory. Supports both synchronous and asynchronous factory functions.
PARAMETER | DESCRIPTION |
---|---|
factory_path |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided factory function. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The factory can't be found or the object returned is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
preferred_subclass
classmethod
#
preferred_subclass(config: CoreConfig, factory_path_override: str | None = None, yaml_path_override: Path | None = None) -> Self
Tries to create an instance by looking at project's component preferences, either from YAML or from the factory. Takes optional overrides for both, which takes a higher precedence.
PARAMETER | DESCRIPTION |
---|---|
config |
The CoreConfig instance containing preferred factory and configuration details.
TYPE:
|
factory_path_override |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
yaml_path_override |
A string representing the path to the YAML file containing the Ragstack instance configuration.
TYPE:
|
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
If the default factory or configuration can't be found. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
from_config
classmethod
#
Create an instance of QuestionAnswerMetric
from a configuration dictionary.
PARAMETER | DESCRIPTION |
---|---|
config |
A dictionary containing configuration settings for the metric.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the metric class initialized with the provided configuration. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/question_answer.py
compute
async
#
Compute the metric.
PARAMETER | DESCRIPTION |
---|---|
results |
The evaluation results.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
dict
|
The computed metric. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/question_answer.py
ragbits.evaluate.metrics.question_answer.QuestionAnswerAnswerRelevance
#
QuestionAnswerAnswerRelevance(llm: LLM, batch_size: int = 15, weight: float = 1.0)
Bases: QuestionAnswerMetric[LLMBasedAnswerRelevance]
Metric checking answer relevance based on LLM. More details can be found here.
Initialize the agent metric.
PARAMETER | DESCRIPTION |
---|---|
llm |
Judge LLM instance.
TYPE:
|
batch_size |
Batch size for metric computation.
TYPE:
|
weight |
Metric value weight in the final score, used during optimization.
TYPE:
|
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/question_answer.py
metric_cls
class-attribute
instance-attribute
#
subclass_from_config
classmethod
#
Initializes the class with the provided configuration. May return a subclass of the class, if requested by the configuration.
PARAMETER | DESCRIPTION |
---|---|
config |
A model containing configuration details for the class.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided configuration. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The class can't be found or is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
subclass_from_factory
classmethod
#
Creates the class using the provided factory function. May return a subclass of the class, if requested by the factory. Supports both synchronous and asynchronous factory functions.
PARAMETER | DESCRIPTION |
---|---|
factory_path |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided factory function. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The factory can't be found or the object returned is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
preferred_subclass
classmethod
#
preferred_subclass(config: CoreConfig, factory_path_override: str | None = None, yaml_path_override: Path | None = None) -> Self
Tries to create an instance by looking at project's component preferences, either from YAML or from the factory. Takes optional overrides for both, which takes a higher precedence.
PARAMETER | DESCRIPTION |
---|---|
config |
The CoreConfig instance containing preferred factory and configuration details.
TYPE:
|
factory_path_override |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
yaml_path_override |
A string representing the path to the YAML file containing the Ragstack instance configuration.
TYPE:
|
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
If the default factory or configuration can't be found. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
from_config
classmethod
#
Create an instance of QuestionAnswerMetric
from a configuration dictionary.
PARAMETER | DESCRIPTION |
---|---|
config |
A dictionary containing configuration settings for the metric.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the metric class initialized with the provided configuration. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/question_answer.py
compute
async
#
Compute the metric.
PARAMETER | DESCRIPTION |
---|---|
results |
The evaluation results.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
dict
|
The computed metric. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/question_answer.py
ragbits.evaluate.metrics.question_answer.QuestionAnswerAnswerConsistency
#
QuestionAnswerAnswerConsistency(llm: LLM, batch_size: int = 15, weight: float = 1.0)
Bases: QuestionAnswerMetric[LLMBasedStyleConsistency]
Metric checking answer relevance based on LLM. More details can be found here.
Initialize the agent metric.
PARAMETER | DESCRIPTION |
---|---|
llm |
Judge LLM instance.
TYPE:
|
batch_size |
Batch size for metric computation.
TYPE:
|
weight |
Metric value weight in the final score, used during optimization.
TYPE:
|
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/question_answer.py
metric_cls
class-attribute
instance-attribute
#
subclass_from_config
classmethod
#
Initializes the class with the provided configuration. May return a subclass of the class, if requested by the configuration.
PARAMETER | DESCRIPTION |
---|---|
config |
A model containing configuration details for the class.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided configuration. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The class can't be found or is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
subclass_from_factory
classmethod
#
Creates the class using the provided factory function. May return a subclass of the class, if requested by the factory. Supports both synchronous and asynchronous factory functions.
PARAMETER | DESCRIPTION |
---|---|
factory_path |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the class initialized with the provided factory function. |
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
The factory can't be found or the object returned is not a subclass of the current class. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
preferred_subclass
classmethod
#
preferred_subclass(config: CoreConfig, factory_path_override: str | None = None, yaml_path_override: Path | None = None) -> Self
Tries to create an instance by looking at project's component preferences, either from YAML or from the factory. Takes optional overrides for both, which takes a higher precedence.
PARAMETER | DESCRIPTION |
---|---|
config |
The CoreConfig instance containing preferred factory and configuration details.
TYPE:
|
factory_path_override |
A string representing the path to the factory function in the format of "module.submodule:factory_name".
TYPE:
|
yaml_path_override |
A string representing the path to the YAML file containing the Ragstack instance configuration.
TYPE:
|
RAISES | DESCRIPTION |
---|---|
InvalidConfigError
|
If the default factory or configuration can't be found. |
Source code in packages/ragbits-core/src/ragbits/core/utils/config_handling.py
from_config
classmethod
#
Create an instance of QuestionAnswerMetric
from a configuration dictionary.
PARAMETER | DESCRIPTION |
---|---|
config |
A dictionary containing configuration settings for the metric.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Self
|
An instance of the metric class initialized with the provided configuration. |
Source code in packages/ragbits-evaluate/src/ragbits/evaluate/metrics/question_answer.py
compute
async
#
Compute the metric.
PARAMETER | DESCRIPTION |
---|---|
results |
The evaluation results.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
dict
|
The computed metric. |