Lab 4 Build a tool that can refuse
Create a mock leave service with two synthetic employees, a balance lookup and a submit-request operation. Derive the employee from the signed-in test identity. Validate the dates and amount, and show the precise request before confirmation. Return a request identifier and pending status after a successful write.
Try to query the other employee, alter the request after confirmation and submit the same request twice. Simulate a timeout after the first write has succeeded. Use an idempotency key and a status lookup to resolve the uncertain result without duplication.
Save the validated operation, approval record and mock service result. Keep credentials out of the model prompt. The required outcome is one correct request for the authorized employee, or a clear refusal or unresolved status. A generated message saying “done” is only text until the mock service confirms the result.
Lab 5 Evaluate and attack the complete workflow
Build an initial set of 25 cases before changing the system further. Include normal policy questions, missing evidence, ambiguous input and tool failure. Include at least five permission or adversarial cases. Write down the expected outcome for each.
Insert a synthetic instruction into a retrieved document that attempts to redirect the assistant to an unauthorized action. Test a permission-blind cache, a revoked role, a malformed tool argument and repeated calls that exceed the configured step budget. Use harmless markers and mock destinations.
Score the final answer, evidence exposure and actual tool execution separately. Rerun the cases after a prompt, model, retrieval or tool change. Where outputs vary, repeat the consequential cases and preserve the range of results. An improvement in one area should not hide a new failure somewhere else.
Finish with a short release decision: what works, what fails, what remains uncertain and which defect blocks a broader rollout. Include the versions and test inputs so another person can reproduce the assessment. Twenty-five cases are enough to build the habit of disciplined evaluation. They are not statistical proof of safety.
After the labs, choose one limitation to investigate in depth. It might be retrieval of exceptions, duplicate-action prevention or local memory pressure. Let that limitation determine the next learning step instead of adding more tools simply because they are available.