Evaluation Frameworks
- LLM outputs are non deterministic, so verifying them by equality is not an option
from utils import get_model_from_gcp
llm = get_model_from_gcp()
response = llm.invoke("Explain inheritence in python")
DeepEval
-
Refer Here for deepeval docs
-
This solves the problem by testing the quality of an answer, rather than exact text equality
-
setup
uv add deepeval
Three building block
TestCase
- A test case describes what happened during one interaction with your application
Metric
- A metric defines what quality means Refer Here
Evaluation Runner
-
The runner applies the metric to testcase
-
Refer Here for the changes done
Exercise
- Make this a pytest
