Running open-source LLMs locally gives you privacy, no API costs, and offline capability. Two tools make this easy.

Ollama — Simplest Local LLM Runner

# Install from ollama.com, then:
ollama pull llama3
ollama run llama3
# That's it — interactive chat in your terminal

# Use from Python:
import ollama
response = ollama.chat(model='llama3', messages=[{'role': 'user', 'content': 'Hello'}])

llama.cpp — Maximum Performance

• Highly optimized C++ inference — runs on CPU with acceptable speed
• Quantized models (4-bit, 8-bit) fit models on consumer hardware
• Powers most local AI apps under the hood

Reference:

Ollama

https://ollama.com/