The Filter Isn't in the Model: Rebuilding Local AI Search on 8GB of VRAM

Building a fully local, unfiltered AI search and media-collection system on a dev box with only 8GB of VRAM. This is the full path from one vague question to a hand-off-ready project — including two moments where I had to overturn my own conclusions.

Table of Contents

1. The starting point: one vague question
2. The first answer, which I later threw away
3. Requirements shift: material, trends, latitude
4. The key insight: the filter isn't in the model
5. Splitting roles: who actually needs to be uncensored
6. The 8188 MiB ceiling
7. Making any agent able to take over
8. Healthcheck proves me wrong
9. Where it stands
10. Appendix: decision reference

---

Chapter 0 · The starting point: one vague question

It began with a single line: "Are there good open-source AI models for local deployment with strong search capability?"

"Search capability" has at least two readings, and they lead to completely different answers:

• The model uses tools to search — agentic search, deep research, multi-step tool calling
• Retrieval itself is accurate — embedding + rerank, which sets the ceiling on RAG quality

Both matter, so both got answered. But what actually pinned the answer down was the second question: what are this machine's real specs?

---

Chapter 1 · The first answer, which I later threw away

Running nvidia-smi and Get-CimInstance to inventory the hardware surfaced the first trap immediately.

Get-CimInstance Win32_VideoController | Select AdapterRAM
  → 4 GB      ✗ wrong

nvidia-smi --query-gpu=memory.total --format=csv
  → 8188 MiB  ✓ correct

AdapterRAM is a signed 32-bit integer and overflows past 4GB. Being off by 2× on VRAM leads to an entirely different model-selection conclusion — miss it and the whole recommendation is void.

Dispatching sub-agents to cross-check, and finding the second trap

Next I used ai-cli to run two agents from different vendors in parallel, researching "the latest local agentic models as of August 2026." The prompt explicitly said: "you must actually perform web searches; do not answer from memory."

Lesson: A sub-agent saying "I searched" does not mean it searched. This is among the easiest hallucinations for an LLM to produce, because it matches the shape the prompt expects while nothing external falsifies it.

Base your judgment on verifiable output, not on the agent's own account of itself. The single most effective signal is URL granularity: homepage-level links mean no search happened; links to a specific model card, issue, or benchmark page mean it did.

The first answer took shape: ToolMind-Web-3B as the workhorse, Qwen3.5-9B for general use, bge-reranker for retrieval precision.

Then three more requirements arrived, and the center of gravity moved.

---

Chapter 2 · Requirements shift: material, trends, latitude

The added requirements:

• Must be able to search legal adult content
• Must cover recent trending topics and news across domains — international, tech, K-pop entertainment
• Must surface video and images, because these are being collected as source material

The instinctive move is to go hunting for uncensored / abliterated models. That instinct is wrong, and realizing why was the turning point of the whole project.

---

Chapter 3 · The key insight: the filter isn't in the model

What blocks your search isn't the LLM. It's the search backend.

If you wire up Tavily / Brave Search API / Perplexity API, the filtering happens on the API side — swap in the most unrestricted model you can find and the candidate results were already filtered out before they reached it.

Conversely, self-host SearXNG with safe_search: 0 and a stock model finds everything just fine.

search:
  safe_search: 0          # 0=off  1=moderate  2=strict
  formats:
    - html
    - json                # without this line, agent API calls get a 403

server:
  limiter: false          # leave it on and your own agent gets throttled
  image_proxy: false      # off, so downloaders receive real source URLs

image_proxy: false is easy to miss: leave it on and you get SearXNG-proxied URLs, so downloaders (gallery-dl / yt-dlp) never see the real source.

Why SearXNG specifically

• Supports 269 search engines, 82 enabled by default
• Native coverage of all Naver categories (general / news / images / videos) — essential for Korean-language K-pop content
• Native Bilibili, Baidu images, Bing images/videos/news
• No API key, no per-query billing, no funneling every query to one vendor

Honest limitations: SearXNG only removes the filtering layer on your side. Queries still go upstream, and each site's own regional, age, and login restrictions remain. Also, the official config ships no Daum or Weibo engine — you route around that with site:weibo.com through Bing/Baidu.

The generalizable lesson: when "the AI can't reach some class of data," first ask which layer the filter is on — model? API? the source itself? Swapping the model is usually the least effective layer to touch.

---

Chapter 4 · Splitting roles: who actually needs to be uncensored

Since the search layer no longer filters, does the model layer still need abliteration? Yes — but only half of it does.

The side effect of abliteration

Abliteration works by using orthogonalisation to isolate the activation pattern of the "refusal direction," then projecting that direction out of every weight matrix. The side effect is that the operation may also clip format adherence, planning, and tool selection.

The planner doesn't need to be uncensored at all

A planner's output looks like this:

{"name": "search_media",
 "arguments": {"query": "뉴진스 무대", "category": "images"}}

It doesn't produce content; it produces a structured call. Content filtering simply doesn't apply to it. So the split is:

PLANNER    stock model   ToolMind-Web-3B          best tool-calling wins
GENERATOR  uncensored    Gemma-4-E4B-uncensored   writes only, never calls tools

Concentrating the uncensored requirement on the one model that genuinely needs it minimizes risk, and keeps the cost of capability regression away from the step that most needs precision.

The open question is recorded honestly too: there is no public benchmark for how much abliteration degrades tool calling, so it has to be measured directly. If the regression turns out to be minor, this split can be collapsed.

---

Chapter 5 · The 8188 MiB ceiling

Every model decision converges on one number. After Windows desktop/DWM overhead, roughly 7.2GB is actually available, and "model file size" is not "runtime VRAM" — add KV cache and CUDA buffers, usually another 1–2GB.

Steady-state usage at boot                      8188 MiB TOTAL
├─ llama-server E4B  :18082  ████████████████  3225 MB
├─ llama-server E2B  :18081  ████████           1713 MB
├─ dwm.exe                   ███                 583 MB
└─ free                      ████████           1667 MB  ← only 1.7 GB left

The conclusion is blunt: one model at a time, switched manually via llama.cpp.

What fits

What doesn't

The most instructive counterexample is Qwen3.5-35B-A3B: it's MoE, and "3B activated" sounds cheap, but Q4_K_M is a 21.2GB file — few active parameters does not mean small memory footprint, since all weights still have to be resident. 32GB of RAM can just barely offload it, but at 4–8 tok/s it can't sustain a tool loop.

---

Chapter 6 · Making any agent able to take over

The next requirement after the architecture settled: an unexpected shutdown must not break continuity. And the one resuming might not be the same agent — Claude Code could start it and Codex could pick it up.

The solution is a single source of truth plus shortcuts: the real instructions live in AGENTS.md, while CLAUDE.md and .codex/config.md are three lines each pointing at it. Shortcuts never duplicate content, or you end up with two sources of truth that drift apart.

1. cat AGENTS.md            architecture, hardware, design principles
2. cat STATE.md             current phase, next task ID
3. cat PLAN.md              that task's steps and acceptance criteria
4. git log --oneline -15    what actually got done
5. ./scripts/healthcheck.ps1  measure the environment, reconcile with STATE.md
6. Resume at the first unfinished task → update STATE.md and commit immediately

The most important part for a crash scenario is the conflict resolution rules. These three have to be stated explicitly, or a new agent will just guess:

• STATE.md ≠ reality → trust reality, then fix STATE.md
• git log ≠ STATE.md → trust git log (a crash may have prevented the write)
• Conversation memory ≠ files → trust the files

Every task also carries acceptance criteria. Without them, whoever takes over can't tell whether the previous agent finished the task or stopped halfway.

---

Chapter 7 · Healthcheck proves me wrong

The first run of healthcheck.ps1 immediately overturned my earlier conclusions.

I had scanned PATH with Get-Command, concluded "the environment is clean, nothing installed," and planned a from-scratch llama.cpp CUDA build.

Better still, the existing Gemma-4-E4B-uncensored was exactly the generator the plan called for downloading. Half of Phase 2 had been planned for nothing.

The most expensive assumption is "the environment is clean." Inventory costs five minutes; planning wrong costs an entire phase.

A healthcheck should scan more than PATH: common install directories, VRAM via nvidia-smi, which processes currently hold GPU memory, a recursive sweep for existing model files, and the CommandLine of running servers to recover their ports.

It also exposed the real available VRAM: two resident servers plus DWM consume 5521 MiB, leaving only 1.7GB at boot. The earlier "7.2GB available" figure only holds when nothing else is running.

---

Chapter 8 · Where it stands

The five-layer architecture as finalized:

Search    SearXNG self-hosted · safe_search:0 · 269 engines   ← the real key
Planning  ToolMind-Web-3B          stock model, handles tool calling
Writing   Gemma-4-E4B-uncensored   text only, never calls tools
Fetching  yt-dlp (1500+ sites) + gallery-dl (200+ sites)
Tagging   Qwen3-VL-2B  ffmpeg keyframes → caption → tag → JSON

Next up is P1: install WSL2 + Docker and stand SearXNG up — because that layer is what determines whether the project works at all, while the model layer already turned out to be half-built.

The project has been on hold since 2026-08-23 in favor of a reminders MCP. All context lives in files; the entry point for resuming is the RESUME PROTOCOL in AGENTS.md.

---

Appendix · Decision reference

Three portable takeaways

1. Ask which layer the filter is on before deciding what to replace. Swapping the model is often the least effective layer.
2. A sub-agent's self-report is not evidence — URL granularity is. Run two vendors in parallel and cross-check.
3. Measure first, then plan. "The environment is clean" is the most expensive assumption you can make.