---
name: training-pipeline-state
description: LoRA training pipeline — model trained, GGUF transfer failed over VPN. Must download on home LAN. Then rebuild BMF + reconnect MCP.
type: project
---

## Training Pipeline — State as of 2026-03-16 (evening)

### MODEL TRANSFER — BLOCKED ON LAN ACCESS
- **6.2 GB GGUF** on clasp at `/tmp/lora-david/model-f16.gguf`
- ~30 rsync attempts over VPN from Berlin failed: hash mismatches from partial corruption + 120s timeout too short for 1.4 MB/s VPN throughput
- Partial file at `/tmp/cm-david-model.gguf` is **corrupt** — delete before next attempt
- **Script ready**: `/tmp/pull-model-vpn.sh` (600s timeout, SSH keepalive, auto-registers in Ollama)
- **Action when home (2026-03-17)**: `rm /tmp/cm-david-model.gguf && nohup /tmp/pull-model-vpn.sh > /dev/null 2>&1 &`
- On LAN (~200 MB/s) a clean transfer takes ~30 seconds
- Log: `tail -f /tmp/pull-model.log`

### WHEN MODEL LANDS
1. Verify: `ollama list | grep cm-david`
2. Rebuild BMF: `cd ~/_ Dev/BetterMemories.io && npm run build`
3. Reconnect MCP in Claude Code: `/mcp`
4. This activates Seb's fixes: teacher during ingestion (#49), chat export (#47), pairing fix (#43), graduation fallback (#45)

### Seb's Fixes (pulled to main, 2026-03-16)
- `8dd17a0` #39 — `'auto'` backend default
- `eca8da7` #40 — llama.cpp GGUF fallback (auto-clones)
- `7fec033` #43 — Pairing ID mismatch detection
- `422212f` #45 — Graduation fallback when teacher unavailable
- `6c332b7` #47 — Chat format in export_training_pairs
- `ce03770` #49 — Teacher enrichment during connector ingestion (rate-limited)
- `39915cc` — Flush timer fix for circle pairings

### Second Training Run (COMPLETE)
- **14,199 deduplicated pairs** from David's instance (vault + git + BMF sessions)
- Qwen2.5-3B-Instruct, rank-16, 2,000 iters, 21 min on M4 Pro
- Output: `/tmp/lora-david/` on clasp, registered as `cm-david-all-slots:lora-v1`

### Clasp State (cleaned up this session)
- 122 stale Modelfile models removed
- PATH fixed (#42): node/npm available in SSH sessions via ~/.zshenv
- TypeScript rebuilt with #39 fix
- mlx-fine-tune.py updated with llama.cpp fallback
- BMF v0.48.0 running, healthy

### Ingestion
- Claude transcripts: **idle / needs restart** — was at 7,965/72,938 (11%) but BMF process restart lost the job
- Restart after model lands + BMF rebuild: `connect_service` with `claude_transcripts`
- With Seb's #49 fix + MCP reconnect, remaining transcripts WILL generate training pairs via teacher

### Remaining Issues (not yet resolved by Seb)
- #41 — Qwen3.5 GGUF broken (upstream, informational)
- #42 — Clasp SSH PATH (we fixed manually via .zshenv)
- #44 — Clasp teacher not configured (decided to skip — unnecessary token burn)
- #46 — Stale models (we cleaned manually)
- #48 — GGUF distribution pipeline (future feature)

### Architecture Finding
- Training pairs only generated via sampling/teacher provider in slot chains
- All slots start SIGNALING with rules-only → no sampling → no pairs from ingestion
- Seb's #49 fix adds optional teacher enrichment during ingestion (rate-limited)
- #45 fix helps graduation progress when teacher unavailable

**Why:** Training flywheel = sovereignty. Local models that learn from your data.
**How to apply:** Model landing → reconnect MCP → Chamber testable. Next training round after ingestion completes (more data = better model).