Evaluator Input and Return Types Supported

Eval Function Inputs

The Eval function input values are supported below:

Parameter name	Description	Example
input	experiment run input	def eval(input): ...
output	experiment run output	def eval(output): ...
dataset_row	the entire row of the data, including every column as dictionary key	def eval(expected): ...
metadata	experiment metadata	def eval(metadata): ...

class ExampleAll(Evaluator):
    def evaluate(self, input, output, dataset_row, metadata, **kwargs) -> EvaluationResult:  
        print("Evaluator Using All Inputs")
        
class ExampleDatasetrow(Evaluator):
    def evaluate(self, dataset_row, **kwargs) -> EvaluationResult:  
        print("Evaluator Using dataset_row ")  
        
class ExampleInput(Evaluator):
    def evaluate(self, input, **kwargs) -> EvaluationResult:  
        print("Evaluator Using Input") 
        
class ExampleOutput(Evaluator):
    def evaluate(self, output, **kwargs) -> EvaluationResult:  
        print("Evaluator Using Output") 

Output: EvaluationResult

The EvaluationResult results can be a score, label, tuple (score, label, explanation) or a Class EvaluationResult

The Eval function input values are supported below:

Return Type	Description
EvaluationResult	Score, label and explanation
float	Score output
string	Label string output

class ExampleResult(Evaluator):
    def evaluate(self, input, output, dataset_row, metadata, **kwargs) -> EvaluationResult:  
        print("Evaluator Using All Inputs")
        return(EvaluationResult(score=score, label=label, explanation=explanation)
        
class ExampleScore(Evaluator):
    def evaluate(self, input, output, dataset_row, metadata, **kwargs) -> EvaluationResult:  
        print("Evaluator Using A float")
        return 1.0
      
class ExampleLabel(Evaluator):
    def evaluate(self, input, output, dataset_row, metadata, **kwargs) -> EvaluationResult:  
        print("Evaluator label")
        return "good"