Demystifying local LLMs by getting under the hood

In a nutshell

One for the technically minded. You can run surprisingly capable LLMs locally on everyday Apple Silicon, but what you can do comes down to memory, context and the trade-offs you make to fit everything in.

Eujin Au gets under the hood with Ollama, Unsloth and Gemma models, digging into quantization, KV cache, context windows and hardware performance, before building his own CLI harness to see how the whole thing actually works.

Peeking behind the curtain

If you’ve spent any time working with AI tools lately, it’s easy to feel like you’re interacting with actual black magic. You type a prompt into a slick web interface, wait a beat, and watch a full architecture document or a working Python script stream onto your screen.

A lot of that magic sits behind proprietary cloud services. I wanted to get further under the hood and explore what happens when you run these models entirely on your own hardware. In particular: How do inference engines communicate with client applications? Where do static model weights end and dynamic context memory begin? And how much performance can you really wring out of Apple Silicon without buying a rack of enterprise GPUs?

Here is what I found going further down the local LLM rabbit hole: from configuring local inference engines to building my own custom CLI harness from scratch.

1. Demystifying the AI Harness & the "Magic" socket

For this experiment, my goal was simple: get a model running locally on my Mac (Apple Silicon) and plug it into an interface like OpenCode as a functional dev assistant.

I used Ollama as the starting point for this particular setup. It’s one of the smoothest ways to get a local LLM running and a chat interface up in minutes. But as I wired OpenCode up to Ollama, I wanted to look more closely at what was happening under the hood: How does an interface talk to an engine?

The answer turns out to be remarkably straightforward: it relies almost entirely on standard OpenAI-compatible REST APIs.

Think of the OpenAI API standard as an electrical wall socket. Because OpenAI set the early standard, open-source communities designed their local engines to match that exact plug shape.

This standard provides several distinct advantages:

• Standardized Code: Official OpenAI SDKs work out of the box.
• Configurable Endpoints: Simply point your client at http://localhost:11434 (or a remote server) instead of cloud endpoints.
• Zero Vendor Lock-In: You can swap models, inference engines, or providers without altering application logic.

Once you realise an "AI Harness" is essentially a wrapper sending HTTP POST requests to a /v1/chat/completions endpoint, the magic fades, and standard software engineering takes over.

2. Models, KV Cache, and Quantization

Ollama was a useful way into this particular setup, but using it highlighted the limits of default configurations. Out of the box, Ollama defaults to modest 4K or 8K context windows. Feeding multi-file codebases or architecture specs required expanding those limits to 32K, 64K, and eventually 128K. Pushing these context bounds made the physical mechanics of LLM memory very tangible, specifically the distinction between static model weights and dynamic context memory:

• Static Model Weights: The fixed memory footprint required to load the model parameters into VRAM, remaining constant regardless of request length.
• Dynamic Context Memory (KV Cache): The working memory footprint that scales continuously with sequence length as the conversation grows.

Quantization compresses model parameters and KV cache values from high-precision formats (like 16-bit float) down to 8-bit, 6-bit, or 4-bit integers. This drastically reduces VRAM requirements for large models and extended context windows. However, aggressive quantization drops precision, which can compromise reasoning quality and code accuracy.

I also moved across to Unsloth as an alternative engine. While widely used for fast fine-tuning, Unsloth also serves local inference via an OpenAI-compatible API. Its explicit UI feedback was particularly useful: seeing real-time memory allocations broken down made the hardware constraints transparent. Testing across an M1, M1 Pro, and M4 Mac provided a great hands-on testbed to observe how different parameters behave under load.

I experimented across various Gemma-4 configurations for software architecture design, coding, and troubleshooting. Comparing the smaller E4B model against the larger 12B variant, I tested 6-bit quantization (UD-Q6_K_XL) against 4-bit (UD-Q4_K_XL) on the 12B model with a 128K context window, evaluating both q8_0 and q4_0 KV cache quantization. While 4-bit quantization fits easily within tighter memory limits, using it for architecture design and troubleshooting revealed noticeable drops in logical reasoning and syntax precision. 6-bit quantization proved to be the ideal sweet spot, preserving coherence without maxing out system memory.

Ultimately, I settled on two specific Gemma configurations based on available Apple Silicon memory limits:

• For 16GB RAM Systems (M1 / M1 Pro):
  • unsloth/gemma-4-E4B-it-GGUF:UD-Q6_K_XL: ~10.58 GiB total memory (~8.21 GiB base weights).
  • unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL: ~11.53 GiB total memory (~7.29 GiB base weights).
  • Both sit right at the ~12GB VRAM allocation threshold imposed by macOS on a 16GB system.
• For 24GB RAM Systems (M4):
  • unsloth/gemma-4-12b-it-GGUF:UD-Q6_K_XL: ~17.74 GiB total memory (~10.38 GiB base weights).
  • Fits comfortably within the ~18GB VRAM allocation limit while leaving overhead for macOS.

Speed vs. Complexity

The Gemma-4 E4B model on M1 generates around 10–12 tokens/second, offering faster initial responses than the 12B model on M4. As a lower-parameter model, E4B acts as a fast, tactical operator ideal for tightly defined tasks, whereas the 12B model excels at high-level planning. Hardware bottlenecks were clear across chipsets: E4B on an M1 Pro delivered a 2.5x increase in tokens/second over the base M1 due to higher memory bandwidth. When running the 12B model, quantization choice (6-bit vs. 4-bit) had a far greater impact on output quality than chip generational differences alone.

3. The infrastructure perspective

Running heavy local AI workloads requires a mental shift in how system resources are planned and managed.

The VRAM Equation: Balancing Model Weights, KV Cache & Quantization

Local LLM capacity is defined by a strict VRAM budget dictated by unified memory limits. Every request holds two distinct components in memory:

• Base Model Weights: Static memory needed to load model parameters (e.g., ~10.38 GiB for Gemma-4 12B in 6-bit quantization).
• KV Cache (Context Memory): Dynamic memory footprint that expands as sequence length grows during interaction.

Total VRAM Footprint = Static Model Weights + Dynamic KV Cache

At 128K context windows, an uncompressed KV cache in 16-bit precision (f16) requires up to 12.47 GiB for context alone on a 12B model. On a 16GB or 24GB system, this exceeds physical limits and triggers severe swapping or out-of-memory errors.

This makes KV cache quantization essential. Quantizing stored keys and values down to 8-bit (q8_0) or 4-bit (q4_0) compresses context memory significantly:

• Gemma-4 12B (128K Context): Drops from 12.47 GiB (f16) to 6.62 GiB at q8_0, and 3.51 GiB at q4_0.
• Gemma-4 E4B (128K Context): Drops from 2.66 GiB (f16) to 1.42 GiB at q8_0.

Quantization involves trade-offs. Over-compressing both model weights and KV cache can degrade reasoning and syntax accuracy. Balancing high-precision weights (6-bit UD-Q6_K_XL) with moderate KV cache quantization (q8_0) ensures full context fits within Apple Silicon memory limits without sacrificing logical fidelity.

The unified memory advantage

Unlike traditional PC setups where System RAM and discrete GPU VRAM are separate, Apple Silicon uses Unified Memory Architecture (UMA). macOS dynamically allocates system memory as VRAM (up to ~75% of system RAM by default). A 16GB M1 caps out at ~12GB VRAM, while a 24GB M4 provides ~18GB VRAM, setting strict boundaries on achievable model sizes and context lengths.

Memory bandwidth directly dictates generation speed. Base M1 provides ~68 GB/s, M4 offers ~120 GB/s, and M1 Pro reaches 200 GB/s, delivering a 2.5x speedup for Gemma-4 E4B over base M1 and proving throughput is bound primarily by memory bus speed rather than raw compute.

While Apple Silicon offers an accessible entry point with high unified memory capacity, discrete GPUs like the NVIDIA RTX 5090 reach up to 1,792 GB/s bandwidth, illustrating why dedicated hardware still dominates peak inference performance.

4. Hands-On: Building My Own CLI Harness & Shell Plugin

Looking closely at the interface to an LLM as essentially an HTTP request prompted me to replace third-party clients with a lightweight custom CLI harness.

The setup

I configured Unsloth locally on my M4 Mac and deployed it across remote M1 and M1 Pro machines to benchmark performance across hardware tiers. Unsloth exposes HTTP endpoints protected by API tokens, which I store as environment variables and pass directly in HTTP headers.

I built a custom Oh-My-Zsh plugin with a lightweight AI shell function. Having a direct terminal alias speeds up my experimentation by eliminating context switching to external browser interfaces. I’m able to configure different sessions with different configurations to use the various remote machines I have set up. Right from the prompt, I can run:

> ai "explain the difference between process memory and thread stacks"

simultaneously across the different models I have running across the hardware I have deployed.

Writing my own “cli” harness was a useful way to get right down into what goes on under the hood and powers our daily interactions with Claude Code and co.

Context, state, and the "Aha!" moment

Because HTTP is stateless, inference engines maintain no inherent memory of prior requests. Multi-turn conversations require resending the entire conversation history with each request. I extended my harness to keep track of my entire conversation, which increased its usefulness in long-running activities and made the mechanics of “memory” in an LLM very tangible: if it isn’t in the context, it isn’t remembered.

The following is a sample of how my harness persisted chat history using a structured JSON format:

{
"usage": {
"input_tokens": 20366,
"output_tokens": 26,
"TTFT": 288146,
"interaction_time": 292687
},
"context": [
{
"role": "system",
"content": "You are a Principal Software Architect..."
},
{
"role": "user",
"content": "remember the following number 42"
},
{
"role": "assistant",
"content": "The number **42** is now part of our context."
},
<...many turns later...>
{
"role": "user",
"content": "what was the number i asked you to remember"
},
{
"role": "assistant",
"content": "The number you asked me to remember was **42**."
}
]
}

How does it remember stuff over time?

There is a finite limit to how much can be retained in an interaction with an LLM. The physical hardware constraints of running a local LLM really highlight this. 128K tokens is tiny in comparison to the 1-million-plus token contexts provided by Cloude, Gemini and other cloud providers. When it comes to local LLMs and limited context windows, you have to resort to the process of compaction.

Compaction is the process of summarizing or pruning older conversation history to prevent the cumulative payload from exceeding the model's finite token limit. As my context file grew over time with the interactions I had using my harness, I would have to “restart” by ordering the LLM to:

> ai "generate me a summary of the conversation to date. outline the key next steps that need to be undertaken. your response will be used to inform a new session to conitue the tasks at hand"

which I could then copy and inject into a new session.

The Time-to-First-Token (TTFT) lightbulb

One thing that intrigued me: as the conversation length increased and the token context got bigger and bigger, the interaction time with the LLM was still relatively quick, until I changed models or restarted the Unsloth engine. This made what the KV Cache was actually doing particularly clear.

An LLM has to evaluate the entire input payload before generating its first token. If static prompts or instructions shift position unpredictably, the engine must re-evaluate the full context from scratch.

Keeping system prompts locked at the beginning of the context array and only appending new messages at the end enables inference engines to leverage cached context evaluations. This significantly reduces TTFT and ensures immediate responsiveness.

As persistent histories expand during extended sessions, cumulative state approaches context window limits. Managing memory state requires deliberate context compaction to keep payload size lean and efficient.

5. So, what was behind the curtain?

Replacing pre-packaged wrappers with a custom CLI tool is a useful way to expose the practical mechanics sitting underneath an LLM.

The primary technical takeaways:

1. Standardised API Interfaces: OpenAI API compatibility makes switching backends trivial.
2. VRAM Budget Management: Balancing model quantization against KV cache footprint is critical for remaining within unified memory limits.
3. Structured Context Optimisation: Proper prompt positioning and context caching dictate system responsiveness.

I’m going to continue tinkering with my harness; next steps are tool calling and ingesting content from my local machine and from the web.

Getting this close to the mechanics reinforces just how much the underlying hardware, context and configuration shape what you can actually do.