Navigating the AI WorldPractice labsChapter 27 of 28

Practise with evidence

Use public or synthetic data in an approved environment. Treat the labs as small investigations, not badges. Keep the inputs, configuration, observed results and the failures you found so that another person can understand what happened.

Lab 1 Compare evidence and product behaviour

Choose a public report you understand. Prepare five answerable questions and two questions the report cannot answer. Run the same requests in two approved assistants with the same document and instructions. Record whether search or other tools were enabled, because those features change the comparison.

Check each material claim against the source. Mark correct answers, unsupported additions, missing exceptions and appropriate abstentions. Then request a structured result for one question and inspect whether an unknown field stays unknown. If API use fits your route, repeat that extraction through a direct model call and validate the schema.

Keep a simple comparison sheet with evidence locations and corrections. Stop when you can explain why the systems differed without pretending that you isolated every model variable from the surrounding product.

Lab 2 Measure a local model

Choose a model whose license, format and runtime fit the machine. Use the same public questions from Lab 1. Record the exact model, quantization, runtime, context settings, load time, peak memory and response time. Inspect task quality alongside speed.

If memory permits, compare a smaller model or another quantization. Do not deliberately exhaust the machine. Record the full data path, including tools and telemetry, then write a recommendation for this workload. Loading a model is setup. Useful performance is the result you are trying to measure.

Lab 3 Separate retrieval from generation

Index 10⁠–⁠20 public or synthetic documents with source identifiers, versions and mock access labels. Create ten questions with known supporting passages, including one exception, one outdated source and one question with no answer. Inspect the eligible retrieved passages before calling a model.

Then generate answers with source references and score retrieval and generation separately. Repeat the test as a user who cannot access one marked document. Inspect the actual model input as well as the visible answer. The useful outcome is knowing where an error began and proving that a targeted change fixed that failure.

Report a correction

Corrections go to the editor and are never published automatically. No account needed.