How Smart Design Converts Local Edge LLMs into Gamechangers

I built [Aperio Health](https://medical360.observer/) (formerly Medical 360) as a passion project. The intent was simple, have a system that provides clinical grade information on any drug, disease state, pathogens etc. as an alternative to google search.

It ingests data from PubMed, [ClinicalTrials.gov](http://ClinicalTrials.gov), WHO, CDC Alerts, NICE HTA, FDA Regulatory, FAERS (Adverse Events), and FDA’s Orange & Purple book (for patents)

[Aperio Health](https://medical360.observer/) uses smaller variants for the Gemma 4 family, specifically Gemma E2B and E4B variants for document synthesis and chat. It also uses the Nomic embed model for generating embeddings. Postgres and pgvector form the back-end data backbone for the application.

Recently I built the functionality of a knowledge graph to draw out visually associated comorbidities, side effects, study correlations, treatments (approved and off-label) between the on-boarded terms

In the initial iterations, the knowledge graph significantly suffered from severe relationship extraction inaccuracies.

It frequently captured nonsensical comorbidities (like linking Influenza to Erectile Dysfunction) or made dangerously broad treatment assumptions (like blindly classifying the drug Keytruda as a blanket treatment for all colorectal cancer without capturing critical biomarker constraints).

Instinctively, I built a functionality where users could up vote and down vote a linkage which would then get re-reviewed by the LLM and either accepted or overridden by the application administrator. This would ensure that users could correct system errors and integrate real world intelligence into the application.

While noble, that was not the right way forward. Users typically give any application 2 or 3 tries and give up if they feel that the application is inaccurate and unreliable.

So, I delved in to investigate and redesign the way I was creating linkages and get the graph as accurate as possible.

The Root Cause: The Co-Occurrence Trap

My initial design was a simple linear, single-threaded pipeline.

I fed a chunk of literature to a local **Gemma 4 E2B** model and asked it to simultaneously find medical terms, deduce their relationships, and format the output into a clean JSON schema.

This created an immense cognitive load on a small model.

In a single pass, it had to handle long context parsing, logical reasoning, and strict syntax enforcement. Overwhelmed, the model fell directly into the Co-Occurrence Trap.

To a small model, structural proximity looks exactly like functional causality.

If a clinical trial document notes that a *“50-year-old male with a history of hypertension presented with acute influenza,”* the model sees the terms co-occurring in the same sentence and blindly draws an edge in the graph.

It couldn't distinguish an ambient demographic baseline characteristic from an active pathophysiological link.

When pushed too hard to enforce strict formatting, it would occasionally suffer generation collapse, crashing the ingestion thread altogether.

The Architecture Shift: The Asymmetric Producer-Consumer Pipeline

To solve this, I changed the linear pipeline architecture and rebuilt it as an asymmetric, decoupled Producer-Consumer architecture optimized for the hardware.

Instead of forcing one model to do everything sequentially, I broke the task into a coordinated assembly line with sub-tasks assigned to different models matching their cognitive bandwidth.

Step 1: Deterministic Code-Level Pre-Filtering

Before an LLM is even invoked, the text passes through a case-insensitive TypeScript pre-filter. If **both **target terms aren't actively present in the raw text string, the application bypasses the LLM entirely. This simple programmatic gate instantly eliminates waste, saving valuable local compute cycles.

Step 2: The Permissive Proposer Pool (Gemma 4 E2B)

I designated the Gemma 4 E2B model as "Proposer" running across 3 parallel workers. I stripped it of all complex clinical reasoning requirements. Its sole job is mechanical text-matching: scan the text, check if the terms are near each other, and copy the exact sentence where they co-occur into a draft payload.

By reducing its cognitive friction, I completely eliminated the empty JSON parsing crashes that plagued my initial build.

Step 3: The Decoupled Async Backlog

The Proposers stream their draft payloads into an in-memory asynchronous backlog queue. This decouples the throughput speeds of our models. The fast E2B model can race ahead through raw documents and build a buffer, ensuring the downstream auditing layer is never starved for work.

Step 4: The Hardened Critic Pool (Gemma 4 E4B)

I assigned the relatively heavier **Gemma 4 E4B** model to act as the "Critic," operating across 2 concurrent workers. Because the Critic doesn't have to parse an entire multi-page PubMed abstract—it only evaluates the isolated, targeted sentence draft passed down by the queue—its context window is free to focus entirely on strict logic.

I injected rigid **Clinical Boundary Guards** into the Critic’s prompt. It is instructed to prioritize clinical reality over grammar:

• If a sentence lists a baseline characteristic or cohort profile, it forces a rejection.
• It actively scans for negative markers ("failed to show efficacy") or sub-population criteria (like restricting a drug to its approved biomarkers).

Step 5: The Algorithmic Quote-Match Verification Gate

As a final safety backstop against LLM imagination, the finalized payload hits a strict programmatic code gate before committing to the database. The system takes the verbatim_evidence_quote extracted by the models and runs a string verification check against the original source document. If the quote does not exist in the source, the edge is hard dropped.

Step 6: Disentangling the Feedback Loop

Previously, the application relied on an In-Context Feedback Learning Loop where administrator overrides were fed back into the system. Feeding this flat list of mixed historical decisions confused the Gemma 4 E2B model.

I refactored this loop to separate the historical data cleanly into markdown-labeled blocks: ### APPROVED CLINICAL PATTERNS and ### CORRECTED STRUCTURAL ERRORS.

Furthermore, I programmatically wrapped these examples in a synthetic **Chain-of-Thought (CoT)** template.

By showing the model explicit, step-by-step reasoning scaffolds of *why* the admin suppressed a past link, the edge model quickly learned how to replicate clinical-grade judgment.

Changing the data structure from a flat history to clearly demarcated blocks (### APPROVED CLINICAL PATTERNS vs. ### CORRECTED STRUCTURAL ERRORS) transformed the model's behavior.

The Result

With these architectural changes, the connected knowledge graph accuracy increased significantly and all co-occurrence based hallucinated linkages were dropped

Lessons & Takeaways

The easiest solution for me would have been to plug the application to a frontier large language model and let it handle the network map.

However, I wanted to keep the cost of application development and testing as low as possible using local LLMs.

I chose to work with the smaller Gemma 4 E2B and E4B models instead of the larger Gemma 4 models (12B, 26B, 31B) to get the best possible token generation speed.

My learnings from this process were

• When running an application locally on unified-memory hardware, the biggest constraint isn't VRAM capacity; it is the computational scheduling and memory bandwidth friction that occurs when small models are forced to think too loosely.
• If your local LLM application is generating inaccurate, hallucinated, or low-quality data, **do not immediately assume you need a larger model.** Instead, review your pipeline and ask:
• *Am I forcing a small model to handle text parsing, logical analysis, and formatting constraints all in a single block?*
• *Can I decouple the throughput by assigning a smaller model to "propose" data and a slightly larger model to "criticize" it?*
• *Am I protecting my models with programmatic code gates (like string includes or schema validators) before and after they execute?*

By shifting from a monolithic prompt mentality to a highly structured, decoupled assembly line, I was able to prove that small, local edge models are entirely capable of generating high-fidelity domain intelligence.

Architecture isn't just a delivery vehicle for an LLM—it is the force multiplier that dictates its intelligence.