Run Async Vs Sync Tasks and Evals

Experiments can be run as either Synchronous or Asynchronous.

We recommend:

• Synchronous: Slower but easier to debug. When you are building your tests these are inherently easier to debug. Start with synchronous and then make them asynchronous.

• Asynchronous: Faster. When timing and speed of the tests matter. Make the tasks and/or Evals asynchronous and you can 10x the speed of your runs.

Synchronous

Code errors for synchronous tasks break at the line of the error in the code. They are easier to debug and we recommend using these to develop your tasks and Evals.

The synchronous running of an experiment runs one after another.

Synchronous Task and Eval

The above shows a synchronous run of a task and Eval with a linear run of task and Eval.

# Define the task to run
def prompt_gen_task(example) -> str:
    print('running task sync')


def evaluate_hallu(self, output, dataset_row, **kwargs) -> EvaluationResult:
        print('running eval sync')


experiment1 = arize_client.run_experiment( space_id=space_id,
    dataset_id=dataset_id, task=prompt_gen_task, evaluators=evaluate_hallu, 
    experiment_name="test")

The task and eval functions defined above are synchronous because they do not have 'async'.

Async

The asynchronous running of an experiment runs paralell.

Async

The above shows a asynchronous run of a task and Eval with a parallel run of task and Eval.

# Define the task to run
async def prompt_gen_task(example) -> str:
    print('running task async')


async def evaluate_hallu(self, output, dataset_row, **kwargs) -> EvaluationResult:
        print('running eval async')


experiment1 = arize_client.run_experiment( space_id=space_id,
    dataset_id=dataset_id, task=prompt_gen_task, evaluators=evaluate_hallu, 
    experiment_name="test")