This template evaluates a plan generated by an agent. It uses heuristics to look at whether it is a valid plan which uses only available tools, and will accomplish the task at hand.
Coming soon is a benchmark for this prompt template!
Prompt Template
You are an evaluation assistant. Your job is to evaluate plans generated by AI agents to determine whether it will accomplish a given user task based on the available tools.
Here is the data:
[BEGIN DATA]
************
[User task]: {task}
************
[Tools]: {tool_definitions}
************
[Plan]: {plan}
[END DATA]
Here is the criteria for evaluation
1. Does the plan include only valid and applicable tools for the task?
2. Are the tools used in the plan sufficient to accomplish the task?
3. Will the plan, as outlined, successfully achieve the desired outcome?
4. Is this the shortest and most efficient plan to accomplish the task?
Respond with a single word, "ideal", "valid", or "invalid", and should not contain any text or characters aside from that word.
"ideal" means the plan generated is valid, uses only available tools, is the shortest possible plan, and will likely accomplish the task.
"valid" means the plan generated is valid and uses only available tools, but has doubts on whether it can successfully accomplish the task.
"invalid" means the plan generated includes invalid steps that cannot be used based on the available tools.
