I was enriching a persistent agentic memory store, 51,000 facts indexed by full text search. Search quality was degrading because the index had no semantic structure. The fix was adding a one-sentence semantic summary to every row. The problem was doing 51,000 LLM calls without a cloud bill that would make the project uneconomical.
In my opinion, that constraint forced me into local inference seriously for the first time. What I found over the following three weeks changed how I think about the cloud-versus-local decision for a whole category of AI tasks.
The Memory Store Problem in Detail
The store is hop.db, a SQLite database that serves as a cross-session memory layer for my agent workflows. It accumulates facts from every session: tool outputs, research findings, client context, system observations. Full text search via SQLite FTS5 worked well for keyword-based retrieval when the index was small. At 51,000 rows, the failure mode became apparent: semantically related facts that used different words were not surfacing together. A query for "invoice processing performance" would miss rows that talked about "accounts payable automation accuracy" even when those rows were directly relevant.
The fix I designed was a one-sentence semantic summary column on every row. The summary would normalize the language, extract the core claim, and use consistent vocabulary across related facts. A retrieval query against the summary column would then surface related rows that the original text search missed. Effective, but requiring 51,000 individual LLM inference calls to generate the summaries.
The rows average about 120 tokens each as input. Adding 40 tokens of output for the summary puts each call at roughly 160 tokens total. At 51,000 calls that is approximately 8.16 million tokens across the batch.
Why Cloud Was Not Viable at This Scale
GPT-4o mini is OpenAI's lowest-cost model at $0.15 per million input tokens and $0.60 per million output tokens as of the current pricing page. For 51,000 calls at 120 tokens input and 40 tokens output: input cost is approximately $0.92, output cost is approximately $1.22, total approximately $2.14. That sounds cheap.
The problem is not the one-time batch cost. The problem is that hop.db is a living store. New facts accumulate daily. The enrichment job needs to run continuously on new rows, and periodically re-run on existing rows as the vocabulary and context of the project evolves. At $2.14 per full pass, monthly re-enrichment across a growing store approaches $25-30 and continues growing as the database grows. For a private tooling project with no direct revenue, that is a recurring cost that needs to justify itself every month.
Gemini Flash 1.5, at $0.075 per million input tokens and $0.30 per million output tokens, cuts that cost roughly in half. The economics improve but the problem does not disappear. More importantly, running 51,000 API calls against an external endpoint introduces rate limiting, retry logic, error handling overhead, and network latency that makes the batch job brittle. A local inference setup eliminates all of that.
The iGPU Setup and Model Selection
My local machine has an Intel Arc GPU with 16GB of shared VRAM split with system RAM. Intel Arc is not the obvious choice for local LLM inference. Most community documentation and benchmark data targets Nvidia. But Arc hardware is what I have, and the llama.cpp project added Intel GPU support via SYCL, which made it viable.
Model selection criteria for this task: fit comfortably within the VRAM budget, produce coherent one-sentence summaries of technical content, run fast enough to complete 51,000 calls in a reasonable wall-clock window. Qwen3-8B-Q4_K_M met all three. The Q4_K_M quantization brings VRAM usage to approximately 5GB, leaving 11GB free for system operation and parallel processes. The 8B parameter size is sufficient for a summarization task that does not require complex reasoning.
The model ran through Ollama, which handles the model loading, context management, and HTTP API interface cleanly. The Ollama model library page for Qwen3 documents the available quantization options and VRAM requirements. Setup from install to first inference took under 20 minutes including the model download.
Total Enrichment Cost for 51,000 Calls (Estimated)
In my experience, cloud costs from OpenAI and Google pricing pages. Local iGPU cost estimated from 6 hours of Arc GPU runtime at approximately 75W draw and $0.20/kWh. One-time batch; recurring enrichment of new rows costs proportionally less.
Measured Throughput and the Real Numbers
Community benchmarks for Qwen3-8B on Intel Arc hardware, documented in llama.cpp GitHub issues and r/LocalLLaMA threads, show a range of 15-28 tokens per second depending on the specific Arc SKU, quantization, and whether the SYCL backend is properly configured. My measured throughput on this task ran at approximately 18 tokens per second for generation, which sits in the middle of community reports for Arc B580-class hardware.
At 18 tokens per second and 40 output tokens per call, each summary generation takes approximately 2.2 seconds. Across 51,000 rows that is roughly 112,000 seconds of pure generation time, or about 31 hours of wall-clock time. The batch job ran in a loop overnight across three nights with a checkpoint that picked up where it left off after each run. Total electricity consumption for the inference work at approximately 75W GPU draw for the Arc card was around 2.3 kWh. At $0.20/kWh that is approximately $0.46 in electricity. Add system baseline draw and the total cost rounds to under $1.00.
The recurring cost for enriching new rows as they accumulate is proportional and even cheaper per row because the batch overhead amortizes over time. I now run enrichment incrementally on new rows each week rather than in large batches. At typical weekly accumulation rates for hop.db, the incremental enrichment job takes under two hours and costs pennies in electricity.
Quality Comparison Against GPT-4o Mini
Before committing to the full local batch, a quality comparison ran on 200 randomly sampled rows. I generated summaries with both Qwen3-8B locally and GPT-4o mini via API and evaluated them blind against three criteria: factual accuracy relative to the source row, vocabulary normalization (did it use consistent language that would aid retrieval), and sentence clarity.
For factual accuracy, both models were comparable. Neither hallucinated claims not present in the source row on this task. Summarization of short, concrete factual statements is not a task that stresses either model's knowledge boundary.
For vocabulary normalization, GPT-4o mini had a slight edge. It was more consistent in choosing canonical terms for technical concepts. Qwen3-8B occasionally used multiple phrasings across similar rows that reduced the retrieval improvement somewhat. This was mitigable with a more specific system prompt that listed preferred terminology. After prompt tuning, the gap narrowed substantially.
For sentence clarity, Qwen3-8B and GPT-4o mini were indistinguishable in my evaluation. Both produced clean, readable one-sentence summaries. For this specific task, local Qwen3-8B was sufficient quality to justify the cost savings without meaningful degradation in the output I cared about.
What Failed and What to Watch For
Temperature sensitivity was the first failure mode I hit. At temperature 0.7 (Ollama's default), consecutive calls on similar rows produced noticeably different phrasings for the same type of fact. Lowering temperature to 0.1 for this task produced more consistent output across similar rows. For a summarization task where retrieval consistency is the goal, near-zero temperature is the right setting.
I have seen this firsthand: quantization artifacts showed up on a small fraction of rows that contained unusual character combinations, code snippets mixed with natural language, or very short source text. In those cases Qwen3-8B-Q4_K_M occasionally produced output with a repeated phrase or a sentence fragment. The rate was low, under 2% of rows, but the batch job needed a validation pass that flagged summaries shorter than 10 words or containing repeated trigrams for human review. GPT-4o mini did not exhibit this pattern on the same rows in my comparison sample.
Context window behavior was also different from cloud models. Qwen3-8B through Ollama defaults to a 4096-token context window unless you explicitly set num_ctx higher. For rows containing long text, I had to set num_ctx to 8192 to avoid truncation. Cloud APIs handle this transparently. Local deployment requires explicit configuration for each model and task.
The Result and What Actually Improved
After enrichment completed and the summary column was indexed with FTS5, retrieval relevance on test queries improved meaningfully. Queries that previously returned 3-5 relevant results from a 20-result set were returning 10-14 relevant results. The semantic normalization that the summaries provide is the mechanism: related facts that used different vocabulary now share a common summary vocabulary and surface together.
The agent workflows that depend on hop.db retrieval showed the change immediately. Context loaded at the start of sessions became more relevant to the current task and less contaminated by distantly related facts that happened to share a keyword with the query.
What Else Runs Well on Local iGPU
The enrichment project clarified which task types belong on local inference. Embedding generation is the strongest fit. Running nomic-embed-text or mxbai-embed-large locally via Ollama produces embeddings at near-zero cost per call, which makes semantic search pipelines economical at any volume. For a project like hop.db where I might want to move from FTS to vector search eventually, local embeddings make that transition cost-free to operate.
Classification tasks with short inputs and categorical outputs also run well locally. Lead scoring, email triage, document categorization, sentiment tagging. These are tasks where you might make thousands of calls per day, where the input is short and the output is a label or a small structured object, and where a 7-8B model has sufficient capability. The economics strongly favor local for sustained high-volume classification.
What does not work locally for my use case: complex multi-step reasoning, synthesis across large contexts, coding tasks that require broad knowledge of library interfaces, and any task where response latency matters for user experience. For those I still route to Claude via API. The local inference layer is not a replacement for frontier models. It is a cost layer that handles the high-volume, defined-scope work so that cloud API budget is reserved for the tasks where frontier capability is genuinely necessary.
If you have a recurring high-volume task that you are currently running on cloud APIs, the question to ask is whether a 7-8B model could do it acceptably. Run a 200-row quality comparison before committing. If the quality holds, the economics of local inference are compelling enough to make the setup overhead worthwhile within the first month of operation.
Setting Up for Local Inference Without a Dedicated GPU
The Intel Arc setup is not the easiest path into local inference compared to an Nvidia card with mature CUDA support. But it is the path I had, and it worked. The practical setup steps: install the Intel GPU drivers and the oneAPI toolkit, build llama.cpp with the SYCL backend enabled, or use Ollama which handles this automatically on supported Intel hardware. Ollama's installation on Linux detected my Arc GPU and routed inference to it without manual configuration. First model pull and first inference took under 25 minutes from a fresh system state.
If you are evaluating local inference for the first time and do not have a discrete GPU, modern CPUs with AVX2 support can run quantized 7-8B models at 3-6 tokens per second. That is too slow for interactive use but entirely acceptable for overnight batch jobs. A batch of 5,000 rows at 40 output tokens each at 4 tokens per second takes roughly 14 hours. For a task that previously cost money every time it ran and can now run while you sleep at the cost of electricity, that trade is often correct even on CPU-only hardware. The decision tree is: identify the task type, estimate the volume and frequency, run a 200-row quality test, calculate the wall-clock time at your hardware's throughput. If the numbers work, local inference is the right answer for that task.