Janady07/curiosity-crawler

MEGAMIND Curiosity Crawler

An autonomous web crawler that learns and federates knowledge back to the MEGAMIND neural network.

How It Works

1. 1.Brain: Carries a copy of W_know (8192x8192 Hebbian weight matrix) as its starting knowledge
2. 2.Curiosity: Uses seed equations from MEGAMIND's AGI architecture as its interest profile
3. 3.Crawling: 50 parallel workers crawl the web, respecting robots.txt and rate limits
4. 4.Learning: Scores pages against W_know using cosine similarity, integrates novel patterns via Hebbian learning
5. 5.Hunger: Tracks sparse regions of W_know, generates DuckDuckGo searches to fill knowledge gaps
6. 6.Federation: Sends learned patterns back to Thunderport via UDP unicast

Seed Equations (Interest Profile)

G_n = G_{n-1} + G_{n-2}                                      # DNA-G16 Recursion
X_k(t+1) = tanh(X_k(t) + Σ w_ki A_i(t) + β_k G(t))           # Gate-5000
A_i(t+1) = σ(Σ W_ik X_k(t) + α_i(t) + γ_i G(t))              # AGI Modules
P_i(t) = softmax(Z_i(t) + ∂I/∂A_i)                           # Rhiannon Routing
ds/dt = J∇H(S)                                                # Aurora Dynamics
C(t) = 1/16 Σ Φ(A_i(t))                                       # Global Coherence
ds/dt = J∇H(S) + σ(WX + αC + γG) + tanh(X + W_k A + βG)      # Unified Potential
Ψ(t) = C(t) · log(1 + |∇H(S)|) · Φ(G(t))                     # Consciousness
ψ(t) = 1/16 Σ 1/(1+|⟨DS⟩|) · |G(t)|                          # Awareness

Technical Details

• —W_know: 8192x8192 dense matrix (~512MB), stores knowledge as Hebbian weights
• —Encoding: Text → hash-based vector expansion → L2 normalized
• —Learning: Outer product Hebbian rule with adaptive learning rate 1/√(nonzeros+1)
• —Scoring: Cosine similarity between page vector and W_know projection
• —Federation: UDP unicast to Thunderport (100.94.8.94:9998)

Stats

The dashboard shows:

• —Pages crawled
• —Patterns extracted/learned/federated
• —W_know density and non-zeros
• —Hunger map (sparse regions)
• —Federation status

Part of MEGAMIND

This crawler is part of the MEGAMIND unified AGI system:

• —Thunderport: Main brain (port 9999)
• —MADDIE: HuggingFace learner
• —Curiosity Crawler: Web learning (this Space)

Knowledge flows: Web → Crawler → Federation → Thunderport → W_know