Fine-tuning is often proposed before the failure has been diagnosed. I would first ask what went wrong. If the right evidence never reached the model, fix retrieval. If the output shape is inconsistent, use schema constraints and validation. A training experiment becomes interesting only after a strong baseline still shows a persistent behaviour problem.
| Need | First intervention to test |
|---|---|
| Current or private facts | Authorized retrieval or a live read tool |
| Consistent output fields | Schema constraints and application validation |
| Better task instructions | Clearer prompts and representative examples |
| Persistent specialized behaviour | Fine-tuning with quality data and held-out evaluation |
| Lower serving cost | A smaller model, routing or quantization, tested on the actual task |
Compare models on the task you actually need to run. Quality, context requirements, tool support, latency, licensing, operating effort and the permitted data path all matter. A larger model may win the difficult cases. A smaller model may be entirely adequate for a narrow classification task. General leaderboards are useful context, not a purchasing decision.
Local runtimes make those tradeoffs visible. Ollama provides a local serving workflow, LM Studio provides a desktop interface and local API capabilities, and llama.cpp supports inference with GGUF models. MLX is an array framework for Apple silicon with related model tooling. Hugging Face Transformers supports loading, inference and training across many architectures. These names represent different roles, not interchangeable packages. [10, 15–18]
Check model format, architecture and runtime support before downloading anything. A GGUF quantization label does not make the same file loadable in MLX. Likewise, “open weights” says nothing by itself about licensing, training-data availability or source-code access.
For a useful local comparison, use the same public test set as the hosted baseline. Record the model and quantization versions, context length, memory use, latency and task quality. Verify the complete data path as well, including tools and telemetry, so “local” describes the workflow rather than only the model process.
Hosted and local deployment move cost and responsibility to different places. For local models, count hardware, energy, administration, availability and security work. For hosted models, count provider cost, data-path constraints and dependency risk. Choose against a workload you can measure.