makas8585/dual-brain-inference

Design of Inference Engine Architecture via Shared KV-Cache Based Real-Time Dual-Attention Ensemble

makas8585 Independent Researcher tjdrldud850@gmail.com

Abstract

Transformer-based large language models (LLMs) are fundamentally vulnerable to self-bias and hallucination due to their reliance on a single autoregressive attention mechanism. The same component that generates tokens also implicitly validates them, creating a closed loop that cannot self-correct probabilistic errors. In this paper, we propose a Dual-Brain Inference Architecture — a real-time token generation framework that operates two independent attention pathways within a single hardware resource. The two pathways, termed Brain A (Symbolic) and Brain B (Bayesian), share a common KV-Cache memory to minimize memory overhead while computing independent logit distributions per decoding step. These distributions are fused in real time via weighted averaging before token sampling. Inspired by Kahneman's dual-process theory (System 1 / System 2) and Domingos' Master Algorithm, the proposed architecture encodes two complementary reasoning paradigms at the inference engine level — specifically targeting implementations such as llama.cpp — rather than relying on expensive post-processing ensemble methods. We argue that this approach constitutes a principled step toward in-flight hallucination suppression and improved reasoning stability.

1. Introduction

1.1 Background

Large language models based on the Transformer architecture generate text through iterative next-token prediction. At each decoding step t, the model computes attention over all previously generated tokens and selects the most probable next token:

$$xt = \arg\max P(xt \mid x_{<t})$$

This process is inherently self-referential: the same attention mechanism that generates a token also conditions all subsequent generation. When the model commits to an erroneous token, that token becomes part of the context and increases the probability of follow-up errors — a phenomenon known as error compounding or hallucination drift.

A concrete manifestation of this failure mode is the token repetition loop: a model generates a subsequence (e.g., "안녕하LALALALALALA"), which then becomes the dominant context signal, driving further generation of the same pattern. No internal corrective mechanism exists because the generator and the validator are identical.

1.2 Limitations of Existing Approaches

Prior work has addressed hallucination through various strategies:

• —Multi-agent verification systems: A secondary model validates outputs from a primary model. However, this requires two full inference passes and introduces substantial latency, making real-time interaction impractical.
• —Post-processing ensemble methods: Multiple models are loaded independently and their outputs are aggregated. This approach is memory-prohibitive on edge and local hardware environments (e.g., single-GPU setups running llama.cpp).
• —Sampling parameter tuning: Techniques such as temperature scaling, top-p, and repetition penalty reduce but do not eliminate hallucination, as they operate on the same single distribution.

None of these methods address the root cause: a single attention pathway cannot escape its own bias at the moment of token generation.

1.3 Proposed Approach

We propose separating the generation pathway into two independent attention modules that share the same KV-Cache memory. Each module computes a distinct logit distribution per decoding step. These distributions are fused in real time — at the lowest level of the inference engine — to produce a final, more robust token selection.

The key design insight is that the KV-Cache (Key-Value tensors derived from the input context) can be shared when the two brains are architecturally designed with identical Key and Value projection weights, while diverging only in their Query projections and upper-layer weights. This eliminates the memory cost of running two independent models while preserving the diversity needed to suppress shared-bias errors.

2. Theoretical Motivation

2.1 Kahneman's Dual-Process Theory

Cognitive science provides a well-established framework for understanding complementary reasoning modes. Kahneman (2011) distinguishes between:

• —System 1 (Fast, Intuitive): Pattern-based, associative, context-driven. Operates automatically with little conscious effort.
• —System 2 (Slow, Deliberate): Rule-based, logical, analytical. Engages structured reasoning at the cost of additional processing time.

Human cognition benefits from the interplay between these systems — System 1 provides rapid pattern recognition while System 2 catches logical inconsistencies. Current LLMs effectively implement only a statistical approximation of System 1.

Our proposed Dual-Brain architecture maps directly onto this framework: Brain A (Symbolic) encodes System 2-style deliberate reasoning, while Brain B (Bayesian) encodes System 1-style pattern recognition.

2.2 The Master Algorithm Framework

Domingos (2015) identifies five tribes of machine learning — Symbolists, Connectionists, Evolutionaries, Bayesians, and Analogizers — and argues that a unifying "Master Algorithm" would integrate all paradigms. The present work can be interpreted as a targeted instantiation of this vision: the fusion of Symbolic and Bayesian reasoning within a single inference engine, operating at token-generation time rather than through offline ensemble construction.

3. System Architecture

3.1 Overview

[Input Context Prompt]
         │
         ▼
┌─────────────────────────────────┐
│      Shared Backbone Layers     │
│   (Embedding + Lower Transformer│
│    Layers with shared W_K, W_V) │
└─────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────┐
│      Shared KV-Cache Memory     │  ← O(N) memory footprint
└─────────────────────────────────┘
         │
    ┌────┴────────────────────────┐
    ▼                             ▼
┌──────────────────────┐    ┌──────────────────────┐
│    Brain A           │    │    Brain B           │
│  (Symbolic)          │    │  (Bayesian)          │
│  W_Q_A, FFN_A,       │    │  W_Q_B, FFN_B,       │
│  LM_head_A           │    │  LM_head_B           │
└──────────────────────┘    └──────────────────────┘
    │                             │
    │ P_A(x_t | x_{<t})           │ P_B(x_t | x_{<t})
    └──────────────┬──────────────┘
                   ▼
┌─────────────────────────────────┐
│    Real-Time Logit Fusion       │
│  P_final = α·P_A + (1-α)·P_B   │
└─────────────────────────────────┘
                   │
                   ▼
     [Final Token Sampling]
     (Greedy / Top-p / Min-p)

3.2 Shared KV-Cache Engine

The input prompt is processed through a shared backbone — the lower transformer layers — using identical Key and Value projection matrices (WK, WV). The resulting Key-Value tensors are stored once in a shared KV-Cache buffer, referenced by both Brain A and Brain B via pointer sharing rather than duplication.

Memory complexity:

This reduction makes the architecture viable on single-GPU hardware that could not otherwise support two independent model instances.

3.3 Dual-Attention Decoding

At each decoding step t, Brain A and Brain B independently compute their next-token probability distributions using their respective Query projections:

$$PA(xt \mid x{<t}) = \text{softmax}\left(\frac{QA K^T}{\sqrt{dk}}\right) V \cdot W{O_A}$$

$$PB(xt \mid x{<t}) = \text{softmax}\left(\frac{QB K^T}{\sqrt{dk}}\right) V \cdot W{O_B}$$

where K and V are the shared KV-Cache tensors.

3.4 Real-Time Logit Fusion

The final token distribution is computed as a weighted average of the two distributions:

$$P{\text{final}}(xt) = \alpha \cdot PA(xt) + (1 - \alpha) \cdot PB(xt)$$

where α ∈ [0, 1] is a learnable or task-adaptive fusion weight. A temperature-adjusted variant can optionally be applied:

$$P{\text{final}}(xt) = \text{softmax}\left(\frac{\log PA(xt)}{\tauA} + \frac{\log PB(xt)}{\tauB}\right)$$

Token sampling (greedy, top-p, or min-p) is then applied to P_final.

4. Training Strategy

4.1 Phase 1: Shared Backbone Pretraining

The shared backbone (embedding layers + lower transformer layers with shared WK, WV) is pretrained using standard next-token prediction on a large, balanced corpus. This phase is identical to conventional LLM pretraining and establishes a common representational foundation for both brains.

4.2 Phase 2: Divergent Fine-Tuning

After backbone pretraining, the model is forked. Brain A and Brain B diverge through differentiated fine-tuning objectives applied to their respective upper layers (WQ, FFN, LMhead).

Brain A — Symbolic Fine-Tuning:

• —Dataset emphasis: formal reasoning (mathematics, formal proofs, code, logic puzzles, structured knowledge graphs)
• —Loss augmentation: a logical consistency penalty term is added to the standard cross-entropy loss, penalizing outputs that violate explicit rule structures present in the training data
• —Curriculum: rule-following tasks precede generalization tasks, reinforcing structured inference patterns

Brain B — Bayesian Fine-Tuning:

• —Dataset emphasis: natural language (narratives, conversation, creative text, contextual inference tasks)
• —Loss function: standard cross-entropy maximum likelihood estimation — inherently Bayesian, maximizing P(xt | x{<t}) over a diverse, context-rich corpus
• —Curriculum: diverse sampling with high-entropy objectives, reinforcing distributional breadth

Diversity guarantee: Because the two brains are fine-tuned on differently weighted data distributions with different loss functions, their logit distributions will diverge for inputs where symbolic structure and statistical pattern differ — precisely the domain where hallucinations occur.

4.3 Phase 3: Fusion Weight Calibration

The fusion weight α is calibrated on a held-out validation set designed to surface hallucination-prone scenarios. Optionally, α can be made dynamic and input-dependent (e.g., higher α when the input is structurally complex, lower α for open-ended creative tasks).

5. Implementation Considerations

5.1 Inference Engine Integration

The proposed architecture is designed for implementation at the C++/CUDA kernel level within llama.cpp or equivalent inference runtimes. The dual-attention decode loop replaces the standard single-pass attention computation. KV-Cache sharing is achieved at the memory pointer level, requiring no additional VRAM allocation for the second pathway's context representation.

5.2 Computational Overhead

The primary computational cost increase is the doubled Query projection and FFN computation during the decode phase. Prefill (KV-Cache construction) remains single-pass. For typical generation workloads where prefill dominates total compute on long contexts, the overhead of the dual-decode phase is relatively modest.

5.3 Relationship to Existing Techniques

6. Expected Contributions

6.1 In-Flight Hallucination Suppression

When Brain A over-concentrates probability mass on an erroneous token (a characteristic self-bias failure), Brain B's independent distribution naturally down-weights that token in the fused output. The token repetition failure mode — where a single attention pathway reinforces its own errors — is structurally interrupted at the logit level before any token is committed.

6.2 Inference-Native Integration

Unlike dialogue-layer or post-processing solutions, the fusion occurs within the token generation kernel. This eliminates the latency overhead of secondary model calls and makes the architecture suitable for real-time interactive applications.

6.3 Memory-Efficient Dual Reasoning

By sharing the KV-Cache, the proposed architecture delivers dual-paradigm reasoning at O(N) KV memory — the same footprint as a single standard model — enabling deployment on resource-constrained hardware.

6.4 A Principled Step Toward the Master Algorithm

By instantiating two of Domingos' five learning tribes (Symbolic and Bayesian) within a single inference pass, this work represents a concrete architectural move toward the integration of complementary machine learning paradigms — not as an offline ensemble, but as a live cognitive process embedded in the generation loop.

7. Limitations and Future Work

Architectural constraints: KV-Cache sharing is valid only when the shared backbone uses identical WK and WV projections. Architectures that use per-head or per-group Key/Value projections (as in standard MHA or GQA) require careful design to maintain the sharing property.

Diversity convergence risk: If Brain A and Brain B are trained on data with high overlap and similar distributions, they may converge toward similar logit distributions, reducing the effective benefit of fusion. Future work should investigate training protocols that explicitly maximize divergence between the two pathways on hallucination-prone inputs.

Empirical validation: This paper presents a theoretical architecture. Empirical benchmarking against standard hallucination evaluation benchmarks (TruthfulQA, HaluEval, etc.) and generation quality metrics remains as future work.

Dynamic fusion: A static α may not optimally serve all input types. Adaptive fusion mechanisms conditioned on input complexity or uncertainty estimates are a promising direction.

Extension to additional paradigms: Future iterations could incorporate additional learning tribes (e.g., Analogizers via retrieval-augmented pathways) as tertiary branches within the same shared-backbone framework.

8. Conclusion

We have proposed a Dual-Brain Inference Architecture that addresses the self-bias limitation of single-attention LLMs by introducing two independent attention pathways — one Symbolic, one Bayesian — operating over a shared KV-Cache. Real-time logit fusion at the inference engine level suppresses hallucination at its source without post-processing overhead or duplicate memory costs. The architecture draws on established frameworks from cognitive science and machine learning theory, and is designed for practical implementation within existing inference runtimes. We release this proposal as an open research direction, inviting empirical investigation and collaborative implementation from the community.

References

1. 1.Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux.
2. 2.Domingos, P. (2015). The Master Algorithm. Basic Books.
3. 3.Li, X., et al. (2022). Contrastive Decoding: Open-ended Text Generation as Optimization. arXiv:2210.15097.
4. 4.Shazeer, N., et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538.
5. 5.Ainslie, J., et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv:2305.13245.
6. 6.Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017.
7. 7.Lin, S., et al. (2022). TruthfulQA: Measuring How Models Mimic Human Falsehoods. ACL 2022.
8. 8.Mündler, N., et al. (2024). Self-contradictory Hallucinations of Large Language Models. arXiv:2305.15852.

Submitted as an independent research proposal. Correspondence: tjdrldud850@gmail.com