MALI

MALI Platform

Financial Analytics
💼Portfolio
🔔Alerts🚀Roadmap📘User ManualBilling⚡Upgrade
?
Loading…

MALI CODEX

The complete feature definition and interpretation guide — every workspace, every tab, how to read it, how to use it.

37 tabs indexed7 workspaces
MALI💼Portfolioalways-on🔔Alertsalways-on🌐Global View🗺️Market Explorer🧠MALI Lab⚡Trading Hub🏆Community Hub✨AI Intelligence⚙️Workspace Config

Platform Architecture

Four layers that must evolve together for MALI to become a self-optimising, autonomous financial intelligence platform.

Intelligence Layer
Models · Pipelines · Sentiment · Forecasting
  • Sentiment model trained on Bing Custom Search news ingestion
  • Risk model using volatility, liquidity & macro indicators
  • Anomaly detection for sudden market moves
  • Reinforcement learning agents for scenario simulation
  • Vector embeddings for FX pairs, crypto, macro events & news
  • News → sentiment → forecast chains
  • Chart interpretation (AI reads charts like a human analyst)
Explore Intelligence Layer →
Agent Layer
Multi-agent system · Personas · Safety
  • Market Analyst Agent — reads news, charts, sentiment
  • Macro Forecaster Agent — interprets global events
  • Risk Auditor Agent — flags exposure, vol & liquidity
  • Portfolio Optimizer Agent — suggests rebalancing
  • Compliance Agent — ensures safe outputs
  • Agent communication protocols + shared memory
  • Agent personas with specialised reasoning
Explore Agent Layer →
Interaction Layer
Futuristic UI · Voice · Adaptive UX
  • MALI Console — real-time dashboards, animated charts
  • Agent conversation bubbles + context-aware responses
  • Voice input/output (British voice preference)
  • Interactive panels for FX, crypto & macro
  • Memory of user preferences + multi-turn reasoning
  • Visual explanations — charts, diagrams, flows
  • Predictive UX — anticipates your next question
Explore Interaction Layer →
Systems Layer
Microservices · CI/CD · Ingestion
  • Azure Functions for ingestion + agents
  • Neon Postgres for vector + relational data
  • Dragonfly Streams for real-time dashboards
  • Bing Custom Search for news ingestion
  • Versioned, lockfile-driven CI/CD with rollback triggers
  • Modular microservices: Ingestion · Intelligence · Agents · UI · Compliance · Logging
  • Reproducible environments + agent-safe deployment steps
Explore Systems Layer →
Core Takeaway

To make the MALI AI Assistant truly cutting-edge, these four layers must upgrade simultaneously. When Intelligence, Agent, Interaction, and Systems evolve together, MALI becomes a self-optimising, autonomous, futuristic financial intelligence platform.

The non-obvious upgrade: Fable 5 as the autonomous engineer inside MALI — generating new agents, improving pipelines, optimising models, refactoring code, and proposing new features weekly. This turns MALI into a self-improving platform — the true definition of cutting-edge.

Feature Codex
🌐

Global View

Global View: a globe and cross-asset panels over live quotes. Data layer: Market Breadth Bar (counted over the 11 S&P 500 sector ETFs: sectors above their 200-day average and near a 52-week high or low, the VIX, CNN's Fear & Greed index and the average sector move); Futures (5 contracts: E-mini S&P 500, Nasdaq-100, Dow and Russell 2000, and Nikkei 225, with the change from the prior settlement, volume, expiry and session); Global Indices (18 indices in four regions with YTD, 52-week range, market status and a label from the day's move); FX (16 pairs across majors, emerging markets and crypto, with the Dollar Index); Commodities (13 futures across energy, metals and agriculture, with volume); Rates & Spreads (the US Treasury curve and its shape, the only sovereign curve with a free daily source; the 2s10s and 5s30s curve spreads; ICE BofA IG, HY and EM option-adjusted spreads from FRED); Sector Heatmap (11 S&P 500 SPDR and 11 iShares STOXX Europe 600 sector ETFs). Intelligence layer: 3D globe, a cross-asset matrix of Pearson correlations of daily returns, capital flow Sankey, a macro scenario panel of stated assumptions, the AI Regime rule on live readings, the Event Ripple (after one of five regional indices made a large move, how the others moved in their next session, measured from ten years of daily closes against each market's base rate; measured history, not a forecast), vol surface, timeline scrubber, a narrative rail of keyword-scored headlines, and the macro calendar. Clicking an S&P 500 sector tile opens its drill-down: Command Center (price, performance, volatility percentile, fund AUM, P/E, volume against its 20-day average), Technical State, Options from the live chain (near-money IV, IV of ~5% out-of-the-money puts and calls, put/call by volume and by open interest, max pain, unusual contracts), Subsector ETFs, Top Holdings, Relationships (correlations and betas from daily returns) and Google News headlines.

🗺️

Market Explorer

Scan every asset class and information source in one place: equities, forex, crypto, news, sentiment, SEC financials and alt data, with Technical Analysis and the Cross-Asset Desk (both moved here when Chart Studio was retired on 2026-11-17).

🧠

MALI Lab

The forecasting and model-research workspace. The Forecast engine runs a measured default pair, TimesFM and GARCH(1,1); any selectable model can be run instead, and a model must beat a random walk on held-out data to get any weight. The point forecast has measured no reliable directional edge, while its bands are calibrated per asset and horizon. Forecast carries Model Selection, Batch Predictions, Ensemble Forecast and Diagnostics as sections. Forecast Accuracy scores past forecasts once their horizons close; its Fairness section splits a walk-forward backtest by volatility and trend regime, and its ML Registry section lists the registered models with their measured skill. Model Comparison scores two models out of sample, with A/B Testing and the Model Leaderboard as sections. Each entry below says what its panel measures and what it does not.

⚡

Trading Hub

Where strategies are tested and trades are planned: simulated auto-trading and paper trading, backtesting, rebalancing advice and trade sizing. Nothing here sends an order to a broker.

🏆

Community Hub

The social layer: the community feed, with sharing below it.

✨

AI Intelligence

The AI analysis tools over live market data. The Daily Brief (where the workspace opens), Agent Debate, Regime Detection, CrewAI and the Research Desk are tabs here, the last two since Agent Workspace was retired on 2026-11-17; the tools that used to be tabs of their own are sections of the tabs that use them (the NL Screener in Market Scanner, Options AI and Options Positioning in Options, the Stress Test in Risk Assessment, and so on). Each is labelled by how it reaches its answer: model-written (a language model writing from figures it is given), rule-based (a fixed rule) or statistical. Model-written text names the model that wrote it and is produced by DeepSeek with Zhipu GLM as backup (AI_PROVIDER_ORDER can change the order; Google and OpenAI follow where a deployment has keys for them), or by the Python backend's own provider when that answers. When no model answers, the panel says so and shows no generated text; nothing is substituted. A figure with no live source is shown as unavailable.

⚙️

Workspace Config

Your account, language, data and report schedules, and for administrators the traffic counts, user management and the audit log.

📌

Always Available

Reachable from every workspace via the sidebar, independent of whichever workspace is active.

MALI ROADMAP

The production trajectory — what shipped, what's being built, and what's coming. Every item maps to a real model, API, or UX component.

⌨KEYBOARD SHORTCUT DICTIONARY

Platform-wide hotkeys — Live = implemented · Planned = coming soon

Live Planned

Live

Ctrl+K
Open command palette / global search
Ctrl+/
Show this keyboard shortcut reference
G+D
Go → Dashboard home
G+M
Go → Market Explorer
G+A
Go → AI Intelligence workspace
1–9
Jump to tab N in active workspace
Alt+←
Previous workspace
Alt+→
Next workspace
Esc
Close modal / drawer / panel

Planned · Coming Soon

G+P
Go → Portfolio
G+R
Go → Roadmap
30 shortcuts live·16 coming soon·46 total

Phase 1 — Core Intelligence

Q1–Q2 2025
Shipped10/10
100% complete
Chronos-T5-Small: Amazon zero-shot probabilistic transformer (210 MB, no training required)
AutoTheta ensemble (statsforecast): Theta / DynamicTheta / ThetaF — pure Python, no PyTorch
GaussianHMM 5-state regime engine: mean_reverting / calm / trending / volatile / crisis
Kelly Criterion position sizing: f*=μ/σ², fractional Kelly, 4-gate risk wrapper
HRP-v2 portfolio optimizer: Ledoit-Wolf OAS shrinkage + Ward clustering + recursive bisection
DCC-GARCH cross-asset correlation engine (GARCH(1,1) per asset, MLE + EWMA fallback)
Almgren-Chriss execution cost model: permanent + temporary market impact, TWAP routing
FinBERT sentiment + BART-large-MNLI narrative + FinBERT+EVT shockwave detection
SHAP explainability: LinearSHAPSurrogate (n=32 Shapley samples), per-prediction + global
FastAPI ML service: local inference at localhost:8000, CORS for mali-trw.com

Phase 2 — Platform & Onboarding

Q3 2025
Shipped7/7
100% complete
ML service fully containerised: runs server-side in Docker on IONOS, no client install required
Walk-Forward CV + Optuna TPE hyperopt (n=50, 300s timeout, MedianPruner)
Synthetic crisis training: 7 profiles (GFC / COVID / rate-shock / crypto-crash / FX-flash / vol-spike / liquidity-drought)
Forecast Tournament 13-tab hub: Fan Chart / SHAP / Regime Timeline / Sentiment Fusion / Trade Replay / Portfolio
Drift Monitor: ADWIN concept-drift detection, staleness score, retrain queue
Causal Discovery: Granger causality DAG with lag, p-value, and strength filter
MALI Codex: full feature definition guide with 80+ indexed tabs, search, and diagram keys

Phase 3 — Live Data & Execution

Q4 2025 – Q1 2026
In Progress9/12
75% complete
Intraday day trading horizons: 15m / 30m / 1h / 4h bar forecasts with Chronos-T5-Large, conformal bands, and asset-class routing across all 6 asset classes
Advanced Trade Signal Panel: BUY/SELL/HOLD signal with confidence %, quarter-Kelly position sizing, entry/target/stop-loss prices, per-horizon breakdown, risk metrics (R/R, max drawdown, annualized vol), multi-factor reasoning chain (ensemble agreement, CI tightness, direction strength, momentum slope), and rolling session signal log
Real-time SSE price feed wired to Global View panels (indices, FX, commodities, sectors)
User identity: display name shown first then @username (derived from email prefix) in sidebar; case-insensitive unique indexes on username and display_name with DB-level and API-layer duplicate prevention
Execution Simulator: realistic fill rate (95% calm / 70% crisis), partial fills, latency distribution, Almgren-Chriss pre-trade cost estimates
Options chain live feed + Greeks panel (delta / gamma / theta / vega per strike)
Live brokerage integration: Alpaca paper trading → Alpaca live → Interactive Brokers TWS
Auto-Trader live arm/disarm with Almgren-Chriss TWAP routing to connected brokerage
AI Auto-Invest: HRP-v2 portfolio proposal → one-click execution with risk-profile gating
Tax-Loss Harvesting engine: wash-sale rule, lot selection, year-end harvest report
Portfolio Greeks: aggregate delta / gamma / theta / vega for options-carrying portfolios
Report Scheduler: PDF performance summaries, watchlist digests, scheduled email delivery

Phase 3.7 — Market Intelligence Scanner

Q3 2026
Shipped15/15
100% complete
Market Explorer tab: world-class AI opportunity scanner surfacing BUY/SELL/WATCH signals across equities, ETFs, crypto, forex, and futures in a single unified UI
5 scan strategies: Momentum Surge / Breakout Hunter / Mean Reversion / Value Discovery / Earnings Play — each with tuned scoring weights and timeframe defaults
AI Conviction Score (0–100): composite of signal strength, RSI regime, MACD histogram, ADX trend power, volume confirmation, and momentum velocity — ranked and filterable
Entry Strategy Engine: per-signal entry type (limit / market / scaled / breakout), price zone (low / ideal / high), ATR-based volatility stop, Kelly-derived position size %, and risk-per-trade % cap
3-tier Exit Strategy: stop-loss, PT1/PT2/PT3 profit targets with per-target R:R ratio, trailing stop %, break-even trigger %, time stop (days), and max-loss abort threshold
Deep-dive modal per symbol: scenario analysis (bull / base / bear with probabilities and price targets), multi-timeframe confluence (1h / 4h / 1d / 1w signal + strength), Kelly position sizing tiers (conservative / moderate / aggressive / full-Kelly)
Sector Heatmap sidebar: 12-sector momentum heat (hot / warm / cold), momentum score bar, change %, and per-sector signal badge — updates every 60 s
Market Regime panel: Risk-On / Risk-Off / Rotating detection from sector breadth, with actionable regime advice and bullish vs bearish sector count
Live filter panel: asset class, signal type (BUY / SELL / WATCH), timeframe, min-score slider, sort-by (conviction / signal strength / R:R / expected move)
Summary bar: real-time BUY / SELL / WATCH counts, avg portfolio R:R, top sector — updated on every scan
Auto-refresh every 60 s with live countdown ticker and instant manual refresh button; client-side filter re-render is instant (no round trip)
Backend API: /api/market/scanner/signals (ranked opportunities), /api/market/scanner/opportunity/{symbol} (deep dive), /api/market/scanner/sectors/heatmap, /api/market/scanner/strategies/presets — all registered via router_loader
Registered in navigation-loader loaderNames + mali-dashboard-master bundle import; nav item 🔭 Market Explorer with NEW badge added to sidebar
MarketScannerPanel.tsx: full React 'use client' component ported into Next.js mali-next app — opportunity cards, sector heatmap sidebar, deep-dive modal, filter bar, all using MALI dark-glass Tailwind aesthetic (bg-black/40 backdrop-blur)
Wired as first sub-tab in market_explorer workspace: DashboardTab type extended, WORKSPACE_SUBTABS.market_explorer prepended, TAB_META entry, renderTab() case — Market Explorer now opens directly to Scanner

Phase 3.8 — AI Intelligence Suite (20 Features)

Q3 2026
Shipped20/20
100% complete
AI Daily Brief (#5): Claude-generated morning briefing — market snapshot (S&P, Nasdaq, VIX, Gold, BTC), regime signal, watchlist scan, and 3 priority actions
Portfolio Autopilot (#2): drift analysis, HRP-aligned rebalance trades, risk-profile templates (aggressive growth / growth / balanced / conservative / income), AI narrative
Voice Assistant (#3/#11): Coqui XTTS V2 (primary) + edge-tts fallback + Web Speech API final fallback; faster-whisper STT; end-to-end voice query routing to all MALI capabilities
RAG Document Intelligence (#4): pdfplumber + text ingestion, 500-word chunk/50-word overlap, ChromaDB vector store, Claude-powered retrieval-augmented Q&A
Multi-Agent Debate (#6): Bull / Bear / Quant / Macro agents run in parallel round 1, sequential rebuttals rounds 2+, committee verdict: STRONG BUY → STRONG AVOID
Market Regime Detection (#7): 6-factor classifier (volatility, trend, breadth, credit spread, momentum, macro surprise) → 7 regime labels, confidence score, Claude narrative
Natural-Language Stock Screener (#8): Claude parses plain-English query to JSON criteria, screens 70+ tickers via yfinance, returns ranked results with fundamentals
News Causality Engine (#9): 6-point causal chain per headline (direct, second-order, macro signal, portfolio impact, opportunity, timeframe); batch mode ranks up to 20 headlines
Options Strategy AI (#10): yfinance options chain fetch (nearest 2 expirations), 11-strategy selector, Claude recommends specific strikes + expiry + premium target
AI Onboarding (#12): 6-question structured flow + free-form conversational mode; extracts user profile JSON, stores to agent memory, generates personalised welcome
Explainable AI (#13): WHAT/WHY/CONFIDENCE/CAVEAT format for any model prediction; portfolio risk plain-language narrative; user-level adaptation (beginner → expert)
Adaptive UI (#14): records user interactions to ChromaDB, Claude returns personalised layout config per session + regime, suggests next-best action
Synthetic Scenario Backtesting (#15): 7 crisis templates (2008, COVID, rate shock, stagflation, tech bubble, soft landing, China decoupling), 200-path Monte Carlo per scenario
Alternative Data Signals (#16): FRED (10 macro series), SEC EDGAR insider filings, Reddit keyword-based sentiment scoring — combined composite signal per ticker
Compliance Copilot (#17): PDT rule (FINRA 4210), concentration risk, wash sale (IRS §1091), margin maintenance, best execution, suitability — CLEAR/WARNING/BLOCKED verdict
Institutional Flow Tracker (#18): options flow bias (put/call ratio, unusual volume >5× OI), short interest (float %, days to cover, squeeze score 0–10), institutional holders, AI narrative
Ensemble Forecasting (#19): LightGBM + Prophet + naive weighted ensemble, upper/lower uncertainty bands, direction confidence, Plotly forecast chart
Agent Memory (#20): ChromaDB per-user vector memory, memory types (conversation/preference/insight/alert), recall by semantic query, compact context injection into every AI call
Chart Vision AI (#1): Claude multimodal vision analyzes uploaded chart images (comprehensive / quick / patterns / S&R modes); supports drag-and-drop PNG/JPG/WebP
AI Intelligence workspace: 12-tab sidebar section wired end-to-end — frontend panels → Next.js → FastAPI /api/ai/* router (40+ endpoints) → claude-opus-4-8 with adaptive thinking

Phase 3.9 — Full-Stack Stub Elimination

Q3 2026
Shipped10/10
100% complete
VisualizationPanel: live Plotly treemap / sunburst / Sankey / network-graph — 6 switchable chart types all driven by real fetchPortfolioSummary() holdings data
WatchlistPanel: live prices via /api/market-data (30 s auto-refresh), real add/remove via watchlist server actions, alert count from fetchAlerts(), inline price-alert creation
UserManagementPanel: inline role-edit modal calling setUserRole(), real client-side CSV Blob export, activate/deactivate wired to setUserVerified() with optimistic UI revert
ReportSchedulerPanel: Send Now surface error on SMTP failure, next-delivery date computation, delivery history, removed no-cron-runner placeholder
FactorExposureCard: Fama-French 5-factor + Momentum + Quality (7 total) — asset-class keyword lookup, weight-averaged portfolio exposure, bi-directional bar chart
PortfolioGreeksCard: beta-weighted Delta per $1 market move, Gamma (equity-only placeholder), Theta via daily risk-free carry, Vega via per-asset vol-sensitivity table
QuantIntelligenceBar: parametric VaR 95% 1-day, HHI concentration index, AI projection via /api/portfolio/technical-ai-signals, anomaly detection (crypto overweight / HHI spike)
DrawdownAttributionCard: per-position P&L contribution (%), Plotly waterfall sorted worst → best + TOTAL bar, summary table with $ / % / contribution columns
TaxHarvestCard: IRS §1091 wash-sale filter (31-day expiry), 35% tax savings estimate, replacement-security suggestions by asset class, ✅ no-harvest empty-state
ConcentrationHeatmapCard: 21-pair cross-asset correlation matrix (20-year estimates), Plotly heatmap with blue→red colorscale, portfolio weight bar, high-risk pair count badge

Phase 3.5 — NLP Intelligence Layer

Q3 2026
Shipped17/17
100% complete
mali-nlp-common shared library: NLPPayload / NLPResult Pydantic models, Dragonfly Streams logger, Neon + SQLite db writer
nlp-preprocess-service (port 8001): langdetect language detection + sentence chunker (POST /preprocess)
nlp-summarization-service (port 8002): HF BART-large-CNN / GLM / Azure domain-routed via YAML config (POST /summarize)
nlp-entity-service (port 8003): spaCy NER + currency-pair detection + Azure entity linking (POST /entities)
nlp-sentiment-service (port 8004): FinBERT sentiment + GLM tone/volatility classifier (POST /sentiment)
nlp-topic-service (port 8005): zero-shot BART-MNLI topic classifier with Azure fallback (POST /classify)
nlp-signal-service (port 8006): keyword rules engine + GLM signal enhancement (POST /signals)
nlp-aggregator-service (port 8007): fans out to all 6 services, merges, streams to Dragonfly, writes Neon/SQLite (POST /analyze)
NLP agent layer (port 8008): forex / crypto / macro handlers — calls /analyze, emits action lists (POST /agent/analyze)
docker-compose.nlp.yml: full 8-container NLP stack with shared env vars, graceful degradation when keys absent
NewsService.enrich_article / enrich_articles: semaphore-bounded NLP enrichment wired into /api/news and /api/news/sentiment
NewsService.get_reuters_news: delegates to ReutersAPI, normalises response, supports symbol or general market mode
GET /api/reuters: direct REST endpoint with ?symbol= and ?limit= params, NLP-enriched
GET /sse/api/reuters: SSE endpoint (30s interval) in sse_misc.py
GET /api/analytics/reuters: analytics router endpoint delegating to reuters_news handler
NewsPanel Reuters tab: symbol dropdown, 4-column article cards with FinBERT score, tone, volatility, signal chips
SentimentPanel Reuters section: auto-populates symbol from active ticker pod, collapsible 4-col NLP card grid with Reuters orange accent

Phase 3.6 — NLP Deep Integration & Live Panels

Q3 2026
Shipped11/11
100% complete
NLP agent handlers wired end-to-end: forex / crypto / macro call ingest_nlp_signals() + check_nlp_alerts() after every /analyze, with graceful degradation and legacy action-tag preservation
NarrativeForecastPanel → live data: parallel fetch of live quote + Reuters NLP, buildTriggers() maps nlp_signals[] to humanized forecast adjustments, 30s polling, symbol switcher
ExplainToTradePanel → live data: GET /api/v1/ai/explainability/{symbol} for real XGBoost SHAP importances; tradeSignal() + actionHint() derive live trade plan; 60s polling
DriftMonitorPanel → live data: GET /api/v1/ai/nextgen/meta-learning/drift for live regime, drift_index %, active_strategy; model staleness table; 30s polling
RLLivePanel → live data: GET /api/v1/ai/nextgen/intraday-agents/status; PnL history accumulates across 10s poll cycles; action distribution parsed from live log strings
CausalDiscoveryPanel → live data: GET /api/alt-data (VIX, Fear/Greed, Put/Call) + GET /api/v1/ai/signals drives edge strengths; buildEdges() scales VIX/sentiment → causal graph in real time; 60s polling
Kelly Criterion NLP scalar: recession_risk → ×0.5, vol_signal>0.5 → ×(1−vol), sentiment<−0.6 → ×0.7, strong bearish signal → ×0.8; floor at 10% of pre-NLP position
Regime detector NLP blend: mean_return_blended = mean_return×0.8 + nlp_sentiment×0.2; recession_risk clamps blended return to ≤−0.01; all regime branches use blended value
POST /ml/forecast/v2 endpoint: NLP-augmented forecast; blended_signal = daily_move×0.75 + nlp_sentiment×0.01×0.25; confidence adjusted for NLP/price agreement, conflict, recession_risk; returns sentiment_label, regime_label, nlp_topic, nlp_tone, abstain gate
Narrative topic clustering in SentimentPanel Deep Dive: articles grouped by nlp_topic field; keyword-fallback bucket detection for 6 topics; cluster card = dominant headline + avg sentiment + de-duped signal chips + article count
Auto-Trader NLP veto gate removed (2026-11-10): it read an in-memory feature store that was always empty, so it never blocked a trade, and today's news applied to past sessions would be look-ahead. The run now says news is not replayed.

Phase 3.10 — Dual-LLM Engine

Q3 2026
Shipped5/5
100% complete
Gemini 2.0 Flash as primary LLM: all /api/ai/* routes now call Google Gemini 2.0 Flash first via generativelanguage.googleapis.com — 1M token context, sub-second latency, cost-efficient
GPT-4o-mini fallback: if Gemini returns null or errors, every route automatically retries against OpenAI GPT-4o-mini — zero downtime even during Google API outages
Shared _lib.ts gpt() function: single call site for all AI text generation — internally tries geminiChat() then openaiChat(), transparent to all 13 route handlers
Chart Vision AI: /api/ai/chart/analyze uses Gemini 2.0 Flash Vision (inline_data multipart) as primary — GPT-4o vision as fallback; same dual-LLM pattern as all text routes
GOOGLE_API_KEY added to IONOS .env.local; OPENAI_API_KEY retained as hot standby — both keys active simultaneously

Phase 3.11 — Production Infrastructure & Auth

Q3 2026
Shipped8/8
100% complete
Critical event-loop fix: coordinator.py pubsub.listen() was blocking the asyncio main thread, killing both gunicorn workers every 120 s — offloaded to asyncio.to_thread(); workers now stable indefinitely
Nginx routing overhaul: /api/auth/session (404) fixed by routing all /api/* to Next.js (port 3000); /api/python/* rewrites strip /python/ before hitting FastAPI (port 8000)
Added explicit nginx location blocks for /api/python/macro/ and /api/python/sentiment/ → Next.js, preventing FastAPI from intercepting those Next.js routes
yfinance rate-limit hardening: asyncio.Semaphore(4) at YahooFinanceProvider class level + asyncio.to_thread() for all yfinance calls; 429 short-circuit break prevents retry flood; MarketDataService._fetch_semaphore(5) added
Neon PostgreSQL migration: login_attempts table created with full schema (id, user_id FK, identifier, email, success, ip_address, user_agent, location, attempted_at, created_at, updated_at) + 4 indexes via asyncpg direct connection
Real user login returns JWT in <2 s end-to-end via POST /api/python/v1/auth/login; login_history_service.record_login() now inserts successfully
Session handshake: POST /api/auth/session Next.js route sets httpOnly session cookie; hard navigation window.location.href='/dashboard' sends cookie so dashboard loads without redirect loop
gunicorn access log confirmed empty → 0 requests processed pre-fix; post-fix health + login + dashboard all respond; nginx error log stays clean (0 upstream errors)

Phase 3.12 — MALI Superintelligence Layer

Q3 2026
Shipped12/12
100% complete
Strategic Intelligence Brain: 5-min autonomous cycle fusing 7-layer global reasoning (macro, microstructure, liquidity, sentiment, geopolitics, derivatives, contagion) into a Bayesian composite signal
Memory Fusion Engine: cross-agent episodic + semantic memory consolidation into FusedMemorySnapshot injected into every agent's context each cycle
Contagion Modeler Agent (#9): mean absolute pairwise correlation of 30 daily returns across SPY, TLT, GLD, the dollar index, HYG and EEM, with a correlation-regime label and the most correlated pairs — reports unavailable when price history is missing
Rule-Based Position Agent (#10, formerly labelled RL Trading Agent): position recommendation over a regime × volatility × momentum × sentiment state. Runs a numpy Q-network only when trained weights are stored at rl:q_network:weights, which nothing writes yet; until then a fixed rule policy that reports the score it computed and marks unread inputs unavailable — no confidence, Q-values or expected value
Synthetic Data Pipeline: regime-switching GBM + Merton jump-diffusion + Heston stochastic vol + Cholesky cross-asset correlation; 7 market regimes with Markov transition matrix
Research Analyst Agent (#11): Goldman-style institutional reports via Gemini Pro — executive summary, investment thesis, risk factors, valuation framework, price target
Peer Reviewer Agent (#12): adversarial review with quality scoring (0-100), rating vs regime cross-check, APPROVED/NEEDS_REVISION/REJECTED verdicts; up to 2 revision loops
Autonomous Research Desk: full multi-agent pipeline (draft → review → revise → cite → publish) with Neon persistence and async REST API
Self-Improving Architecture: daily + weekly evolution orchestrator with LLM-powered improvement proposals, auto-apply for safe changes, deploy_required safety gate for code changes
DB schema: brain_cycles, research_reports, evolution_cycles, market_intelligence_snapshots — all live on Neon Postgres
Frontend: StrategicBrainPanel (7-layer signal viz), ResearchDeskPanel (async pipeline UI), EvolutionPanel (proposal tracking) — wired into Agent Workspace in DashboardShell
API endpoints: /api/brain/*, /api/research/*, /api/synthetic/*, /api/evolution/* — all auth-protected, live on mali-trw.com

Phase 3.13 — Unified Superintelligence Platform (v4.0.0)

Jul 2026
Shipped13/13
100% complete
Architecture Diagram (§7): interactive 6-column Bloomberg-style system map — data sources → pipelines → intelligence layers → 12 agents → Strategic Brain → terminal, with animated SVG flow paths and hover tooltips
Unified Agent Ecosystem (§8): 12-agent roster with live health monitoring, per-agent heartbeat aggregation, ecosystem score, single dispatch, fan-out broadcast, two-agent adversarial negotiation, SSE real-time stream — /api/python/agents/*
Unified ML Suite (§9): ModelRegistry with 8 ModelCards — Chronos, Theta/AutoTheta, PatchTST, FinBERT, HRP-v2, DCC-GARCH, GaussianHMM, RL Q-Network (staging, untrained) — lifecycle promote/retire. The performance figures, latency targets and cache keys these cards originally shipped with had no evaluation, definition or writer behind them and were removed. The three forecasting cards now read measured skill against a random walk from the stored measurement table; everything else reports unavailable with the reason
Unified Pipelines (§10): UnifiedPipeline with 9 async stages — Ingest → NLP → Regime → Forecast → Portfolio → Contagion → Brain → Agents → Emit — every reading stage has a real source and reports unavailable with the reason when it returns nothing: Ingest pulls daily bars from the shared market data service; NLP reads the FinBERT headline aggregate the news service publishes (sentiment:aggregate:*), scoring the headlines itself when none is stored; Regime reads mali:regime:latest; Forecast reads the forecast engine's newest baseline record in forecast_runs (last 24 h) with each model's measured skill beside it, and never runs the models; Portfolio runs HRP over the signed-in user's holdings from their daily closes; Contagion and Brain read their stored records; conditional agent dispatch when |signal|>0.2, results persisted to Dragonfly + published on pipeline:results channel
Unified UX System (§11): AgentEcosystemPanel (12-agent grid with ping dispatch), MLRegistryPanel (model card browser; performance shown only where a sourced measurement exists), UnifiedPipelinePanel (live 9-stage run with signal readout) — all wired into Agent Workspace in DashboardShell
Unified Governance & Safety (§12): AuditLogger with async drain queue to Neon audit_log + Dragonfly audit:events channel; AuditEvent dataclass covering agent, brain, safety, auth, pipeline categories; severity triage (info/warning/critical); /api/python/governance/* endpoints
audit_log table migrated live on Neon Postgres — 11 columns (event_id UNIQUE, category, action, actor, outcome, severity, resource_id, details JSONB, ts, created_at) + 4 indexes (category, actor, ts DESC, severity partial)
Unified Optimization Package (§13): CacheOptimizer with hot-key detection (>10 accesses/min → 2× TTL boost), category TTL management, stale-key sweep (5-min cycle), pool recommendations, hit-rate tracking — /api/python/optimization/*
Full Code Generation (§14): all __init__.py stubs resolved; 5 new service packages (ml_registry, pipelines, governance, optimization, versioning) with clean imports; 7 new API routers; AuditLogger + CacheOptimizer started/stopped in FastAPI lifespan
Deployment Instructions (§15): DEPLOYMENT.md covering IONOS git-push deploy, Gunicorn config, Nginx routing, Neon migration pattern, GitHub + HF Spaces push, service management, rollback procedure, health check commands
Versioning Strategy (§16): VersionManager with SemVer MAJOR.MINOR.PATCH, changelog ring (100 entries in Dragonfly), rollback history (20 entries), VERSION file = 4.0.0 written at repo root — /api/python/versioning/*
Next Evolution Cycle (§17): Phase 4.1–4.10 roadmap defined — options IV surface, PatchTST promotion, RL paper trading, contagion D3 heatmap, governance UI, FinBERT SEC streaming, HMM alerts, ML A/B testing, versioning UI, MALI SuperAgent
All 13 new endpoints verified 200 OK on mali-trw.com; agents/health KeyError hotfix caught and deployed in same session; platform version bumped to v4.0.0

Phase 4 — Social & Mobile

Q1 2026
In Progress4/7
57% complete
Copy Trading: following another trader’s positions isn’t available yet, and the panel says so. MALI places no orders
Trading Competitions: entering a competition isn’t available yet; the panel names the missing feature rather than offering controls that do nothing
A dedicated mobile layout: the Mobile View tab says it isn’t available yet, and there is no installable app
Community feed: public posts with likes and comments
CrewAI: agents that use MALI’s own market-data and risk services as tools, with web search when a search key is configured
Council Chamber debates a symbol from the regime, shockwave and headline-sentiment readings; its earlier evidence and confidence scores came from a hash of the topic's text and were removed
Voice interface: spoken responses through MALI’s own speech service, with the browser’s speech as the fallback

Phase 5 — Frontier Models

2026+
In Progress5/7
71% complete
Chronos-T5-Large: a higher-capacity zero-shot forecaster
TimeMixer or TimesFM integration after equity-data validation at production depth
Online GaussianHMM refit: continuous regime learning with Baum-Welch incremental EM via /regime/online-refit
Multi-asset Kelly: cross-portfolio Kelly with correlation-adjusted μ/Σ^{-1} formulation via /kelly/multi-asset
Alternative data beyond public sources — satellite imagery, card transactions, earnings-call audio: no such feed is licensed or read today
Quantum portfolio optimisation: not built; no quantum solver is connected
MALI API: public REST API at /api/public/v1 (forecast, regime, signals) with API-key auth and a per-key rate limit

Phase 4.2 — Sector Intelligence Cockpit & Dashboard Data Integrity

Jul 2026
Shipped13/13
100% complete
Sector Intelligence Cockpit: Bloomberg MAP<GO>-class 6-panel drill-down inline to the SP500 heatmap — GET /api/v1/market/sectors/[sectorId]/intelligence fetches 3-month Yahoo Finance daily chart, computes 1D/1W/1M/3M returns, annualized vol (σ×√252), vol regime (calm/normal/elevated/stress), ETF flow estimate, subsector ETF quotes, breadth approximation, and sentiment scores; 90-second in-process cache per sector ticker
SectorDrilldownPanel.tsx — 6-panel cockpit: (1) AI Narrative strip with typewriter animation; (2) Command Center: live price, 4-period strip, breadth bar, vol ring with percentile arc, ETF flow chip, 3-bar sentiment chart; (3) Predictive Signals: 4 traffic-light chips (1H/4H momentum, vol direction, capital flow) + confidence bars + regime shift probability gauge; (4) Options Intelligence: IV rank/percentile rings, put/call ratio bar, gamma exposure, max pain, 25Δ skew split, unusual activity table; (5) Subsector Micro-Heatmap: heat-colored tiles with volume anomaly badges + options pressure icons; (6) Relationship Graph: radial SVG — sector correlations + macro betas (US10Y/VIX/USD/credit/oil) + commodity links, line thickness ∝ strength
SectorHeatmapGrid.tsx upgraded: clicking any SP500 SPDR tile opens the cockpit inline below the grid with purple selection-ring glow; second click or ✕ closes it; STOXX tiles remain informational only
SUBSECTOR_MAP: complete ETF-to-subsector mapping for all 11 SP500 sectors — XLK→[SOXX,IGV,SKYY,BOTZ,QTEC], XLV→[IBB,XPH,IHI], XLF→[KBE,KIE,KCE], XLE→[XOP,AMLP,OIH], XLY→[XRT,PEJ], XLI→[ITA,IYT,XAR], XLB→[PICK,XME,MOO], XLC→[IYC,FIVG,GAMR], XLRE→[REZ,INDS,MORT], XLP→[PBJ,FSTA], XLU→[IDU,PHO]
Static intelligence matrices: CORRELATIONS (11 sectors × 5 peers, rho_30d + rho_1y), MACRO_LINKS (11 sectors × 3 macro factors — US10Y, VIX, USD Index, credit spread, WTI, consumer confidence — each with beta and stability score)
master-demo-injector.js critical fix: populateAllChartContainers() disabled — was blindly overwriting every [id$='-chart'] div with Math.random() scatter/bar/heatmap data 1.5 s after page load, nuking all legitimately rendered API charts; market indices replaced with deterministic sine-modulated trend; 3D scatter replaced with golden-spiral point distribution
market-intelligence-loader.js fixed: GNN relational map now uses real 5×5 1Y correlation matrix (AAPL/MSFT/NVDA/TSLA/GOOGL) as stable base, applies API response matrix when server returns a recognized shape — was calling the GNN API then discarding the response entirely and filling the chart with Math.random()
social-charts-loader.js fixed: community sentiment now tallies bullish/bearish by symbol from live /api/social/communityfeed posts (was hardcoded [85,92,95,45,78]); copy-trading chart calls /api/social/leaderboard and builds real cumulative return curves for top 3 traders from return_30d values (was hardcoded [100,105,112,108,124] with no API call at all)
analytics-charts-loader.js fixed: renderSectorDistribution() data shape bug — was receiving a performance time-series array and calling Object.keys(array), producing '0','1','2'... as pie chart labels; now extracts sector field from array items or calls /api/portfolio/summary as fallback; _plotSectorPie() helper extracted to avoid duplication
visualization-loader.js fixed: load3DMarketMap() now uses /api/v1/dashboard-intelligence/overview response to scale the surface; loadSectorHeatmap() calls /api/v1/market/sectors and builds a real 5-metric matrix (1D%/1W%/1M%/YTD%/Vol) from live sector data — both were previously ignoring their API calls and rendering fully hardcoded static data
Path aliases router: /api/ai/history → 308 redirect → /api/v1/ai/history; /api/v1/dashboard/intelligence/overview → 307 redirect → /api/v1/dashboard-intelligence/overview — eliminates frontend 404s from URL mismatches between legacy HTML routes and Next.js API paths
NGINX routing fix: dedicated location block maps /api/python/ml/model-comparison directly to FastAPI /ml/model-comparison, bypassing the /api/python/* rewrite chain that was producing 404s for the AI charts model-comparison endpoint
Social API router overhaul: RESOURCE_MAP corrected (competitions→/social/tournaments, copy-trading→/social/leaderboard); transformResponse() added to normalize FastAPI tournament/leaderboard shapes into frontend Competition[] and Trader[] interfaces; CompetitionsPanel.tsx and CopyTradingPanel.tsx hardcoded mock arrays eliminated, replaced with live API state

Phase 4.3 — Codex Knowledge Base Sync

Jul 2026
Shipped10/10
100% complete
Sector Intelligence Cockpit codex entry: global_view section summary expanded with full 6-panel Bloomberg MAP<GO> drill-down, all 11 SP500 SPDR tickers, SUBSECTOR_MAP documentation, Yahoo Finance annualized volatility formula (√(Var(log_returns)×252)×100), 90s cache, and API route reference
Global View codex entry rebuilt: whatItShows / howToRead / howToUse now fully document the click-to-drill workflow — purple selection ring, cockpit panel-by-panel guide (AI Narrative → Command Center → Predictive Signals → Options Intelligence → Subsector Micro-Heatmap → Relationship Graph), and 90s auto-refresh cycle
Auto-Trader entry corrected: replaced stale 'PPO-Finance RL Agent signal' with accurate Kelly Criterion Agent description — f*=μ/σ², fractional Kelly 0.5, conviction tiers (1.0/0.8/0.5/0.4×), 4 sequential risk gates (drawdown kill-switch / vol scaling / confidence gate / position cap), 8% kill-switch threshold
News Feed entry expanded: Reuters/NewsAPI sourcing documented, FinBERT-tone enrichment pipeline added, sentiment badge interpretation (confidence score, red/neutral/green), and headline-to-position impact workflow
Sentiment entry rebuilt: FinBERT-tone NLP pipeline detail — pre-trained financial BERT, −1 to +1 confidence gauge, needle drift interpretation, Bayesian Forecast Corrector feedback loop documented (sentiment tilt ±10% on Conditioned Target)
Alt-Data entry expanded: FRED macro series (10 series: Fed Funds/CPI/unemployment/GDP/M2/consumer confidence/housing starts/ISM/credit spreads/10Y yield), SEC EDGAR Form 4 insider buy/sell tracker, Reddit/StockTwits per-symbol bullish scoring with ≥60%=green / ≥40%=amber / else red color tiers
Narrative Forecast entry rebuilt: live Reuters NLP pipeline, FinBERT keyword extraction, BART-large-MNLI macro theme detection (Fed Pivot / rate shock / earnings beat / sector rotation / geopolitical risk), Conditioned Target vs base Fan Chart comparison workflow
AI & ML entry upgraded: Gemini 2.0 Flash + GPT-4o-mini dual-LLM strategy generator documented — Gemini draft (blue) / GPT-4o-mini critique (amber) / final synthesis (green); high draft/critique divergence = strategy uncertainty signal
Community entry upgraded: live /api/social/communityfeed bullish score bar chart documented — tally logic, color tier thresholds (≥60%/≥40%/else), crowd-consensus reading vs contrarian signal zones (>80% or <20%)
Copy Trading entry upgraded: live /api/social/leaderboard with return_30d monotone cumulative curves for top 3 traders; equity curve slope vs smoothness interpretation guide; copy-allocation cap and diversification guidance

Phase 4.4 — Full Platform Audit & Real-Data Restoration

Jul 2026
Shipped14/14
100% complete
Full 31-subtab production audit across Chart Studio (5), MALI Lab (16), and Market Explorer (10) on live mali-trw.com — every displayed price verified against Yahoo Finance ground truth (AAPL $333.43, SPY 741.69, S&P 7437.63, Dow 52208.06, Nasdaq 25122.178, VIX 17.09, EUR/USD 1.1533 — all exact); MSFT +15.51% single-day move independently confirmed real via live news cross-reference
SYSTEMIC FIX — Next.js rewrite pipeline: /api/python/* and /api/market/* proxy rules moved from afterFiles to fallback. afterFiles rewrites evaluate BEFORE dynamic routes, so every [id]-segment handler under those prefixes (multi-stock-forecast-stream, batch-forecast-status/stream, backtesting results/regimes/trades, autotrader run streams) 404'd in production via FastAPI passthrough
SYSTEMIC FIX — nginx: /api/python/ catch-all repointed from FastAPI (with broken /api/$1 path rewrite) to Next.js :3000 — Next.js route handlers now serve every implemented endpoint and its fallback rewrite proxies the rest to Python; dedicated location blocks added for the multi-stock-forecast family (async POST, SSE stream 300s no-buffer)
Forecast tab restored end-to-end: Run Ensemble → real Python job UUID → live SSE stream → real 110-session diagnostics; Model Selection 'Failed to fetch models' fixed — 29-model roster loads from /ml/ensemble-models with real fitted weights
Model Comparison rebuilt: was a hard-coded 3-model array with no fetch and a dead button; now trains two selectable architectures (XGBoost/LightGBM/RF/N-BEATS/Temporal-Attention) live on 120 sessions of real price history per symbol via new /api/python/ml/model-comparison proxy route — real R² accuracies + top price-lag importances
Forecast Tournament rebuilt: static leaderboard with fake pulsing LIVE badge replaced by real validation leaderboard — 10 models trained live per symbol/horizon, real MAPE/RMSE/R² from per_model_metrics, weight bars showing the actual fitted ensemble (XGBoost led AAPL at 96.3%)
Explainability suite de-faked: model-feature-importances, model-local-explanation, and model-fairness-metrics routes were seeded-RNG generators with fabricated indicator names — all three now proxy real Python computations; NEW Python endpoint /ml/deep-explainability added (exact TreeSHAP per-sample via XGBoost pred_contribs, no shap dependency); honest 'close t−Nd' lag labels; synthetic fallbacks retained only for offline dev and tagged source:'synthetic-fallback'
Drift Monitor triple-fix: Python UnboundLocalError (nlp_sentiment referenced in fallback path before assignment) crashed the endpoint; wrong root-shim real_market_data import lacked get_historical_data; silent flat-100.0 placeholder series zeroed volatility when yfinance rate-limited — switched to market_data_service; drift/regime now computed from real SPY history (13.6% annualized vol verified)
Batch Predictions: never called Python — now Python-first (real ensemble job + SSE) with local fallback only when unreachable; generatePredictions business-day cursor fix (weekend spillover collapsed multiple horizon steps onto the same Monday)
AdvancedForecastHub un-demoed: /api/ml/[...path] proxy fixed (FastAPI services live at /api/v1/ml/* behind CSRF header + HTTPBearer — proxy now maps path, sends X-Requested-With, forwards cookies, and supplies bearer token incl. demo-jwt-token); INTERNAL_API_SECRET generated and configured for both services; new /api/ml/forecast/v2 schema adapter maps hub payload ↔ FastAPI ForecastRequest with derived 50/80/95% CI bands from the real NLP-augmented forecast
Options tab real chain restored: Yahoo v7 options API now requires session-cookie + crumb auth — implemented with 25-min cached handshake and 401 retry; AAPL verified live (76 calls · 73 puts · 9 expiries · post-earnings IV crush 47%→32% visible in term structure); synthetic fallback now anchors to live spot from the chart API instead of a stale price table that showed AAPL at $195
News pipeline: /api/reuters treated Python's empty article list as success, blocking the Yahoo RSS fallback — empty now falls through to real headlines; lexicon bullish/bearish sentiment scoring added to RSS articles so Narrative Forecast trigger cards fire on real news (Python FinBERT path unchanged when populated)
Alt Data: Reddit JSON API 403/429s datacenter IPs — RSS fallback parser added plus last-good-result cache (10-min fresh / 60-min stale-ok) so r/stocks hot posts survive throttle windows; CNN Fear & Greed, VIX, StockTwits, and CBOE put/call verified live
Verified-real without changes: Technical Analysis, Analytics (exact index levels), Analytics Zoom, 3D Charts, Market Scanner, Heatmap (24 live large caps), Forex, Crypto, News Feed (live NewsAPI), Sentiment Orchestrator, Real-Time SSE, Quantum AI Desk (state + shock simulate), AI & ML signals, Causal Discovery, RL Live agents, Explain→Trade, AI Optimization (portfolio metrics correctly gated to real accounts)

Phase 4.5 — AI Intelligence Audit & Live LLM Restoration

Jul 2026
Shipped8/8
100% complete
Full 12-subtab AI Intelligence workspace audit on production: Daily Brief, Agent Debate, Regime Detection, NL Screener, News Causality, Options AI, Compliance, Autopilot, Ensemble Forecast, Chart Analysis, Institutional Flow, Scenario Backtest — every endpoint probed, every panel browser-verified
Institutional Flow rebuilt on REAL market data: put/call ratio + total OI computed from the live crumb-authenticated Yahoo options chain (verified AAPL: P/C 1.26 from 162,791 call OI vs 204,305 put OI); unusual activity = real contracts filtered at volume/OI > 2× (5 detected with actual strikes/expiries); real short interest via quoteSummary defaultKeyStatistics (146.5M shares short, 2.3 days-to-cover, 1.0% float); shared lib yahoo-auth.ts extracted (cookie+crumb handshake, 25-min cache, 401 retry) with fetchOptionsChain + fetchShortInterest helpers
NL Screener stale-price fix: hard-coded universe prices were off by 50%+ (AAPL listed $213 vs $333 real) — every match now refreshed with a live Yahoo quote + day change (price_source: live/snapshot), and price criteria re-applied against the live quote so snapshot prices can't leak names past a price cap
Crash fixes from live error reports: AIEnsemblePanel stored 401 error payloads as results and crashed on result.predictions.length; AIFlowPanel crashed on squeeze_potential.toFixed because the API emitted a boolean where the panel renders a 0–10 numeric score (and squeeze_risk as string vs boolean) — both panels now guard response shape and render a friendly error card (session-expired prompt on 401); flow API emits the numeric squeeze score (short-float% + days-to-cover blend, risk badge at >6)
Live LLM restored: server had an invalid OPENAI_API_KEY and no Google key — GOOGLE_API_KEY installed for both services (Next.js + FastAPI); Gemini is the PRIMARY LLM with GPT-4o-mini as automatic backup per direction, and a GEMINI_API_KEY env alias was added
Retired-model fix: Google retired the pinned gemini-2.0-flash (404 as of Jul 2026) — both call sites (_lib.ts narrative engine, chart/analyze vision) switched to the gemini-flash-latest alias which tracks the current flash model across rotations; verified live (real causal-chain analysis and multi-agent debate synthesis now generating on production)
Verified real without changes: Daily Brief (live S&P/VIX/10Y/watchlist quotes feeding the narrative), Regime Detection (live inputs from /api/market/regime-inputs, deterministic 6-factor classifier), Options AI (live price + IV strategy construction), News Causality (live affected-symbol quotes), Compliance (deterministic rule engine), Autopilot (advisory allocator), Scenario Backtest (historical scenario templates + Monte Carlo), Chart Analysis (Gemini vision on uploads)
Codex: new AI Intelligence section added with all 12 entries (whatItShows / howToRead / howToUse per tool) — the workspace was previously undocumented in the codex entirely

Phase 4.6 — Day-Trading Forecast Restoration

Jul 2026
Shipped6/6
100% complete
Root cause of 'Job not found' in AI Forecast Engine day-trading mode traced through a three-link chain: intraday timeframes send fractional-day horizons (15m = 1/96) but the Python request model typed horizon as int → every intraday POST failed validation with a 422; Next.js then minted a local msf- fallback job id and its stream route proxied that id to Python anyway — whose SSE answers unknown ids with an HTTP-200 stream carrying {"error": "Job not found"}, which passed the ok-check and streamed the error verbatim into the UI
Python intraday engine built: MultiStockForecastRequest now accepts float horizons plus an interval field (15m/30m/1h/4h, with legacy fractional payloads inferring their interval); day-trading runs forecast one full US session of bars (26×15m / 13×30m / 7×1h / 2×4h — matching the UI's bars-per-session copy) from REAL Yahoo v8 intraday closes (4h built from every 4th hourly bar), through the same live-trained ensemble, with bar-spaced HH:MM prediction timestamps, per-bar volatility confidence bands, and an interval tag on every result
SYSTEMIC FIX — cross-worker job store: production gunicorn runs 2 workers and async forecast jobs lived in a per-worker in-memory dict, so any SSE/status request landing on the other worker answered 'Job not found' even for valid jobs; jobs are now mirrored to Dragonfly (30-min TTL) with the local dict as fast path, and the stream generator grace-polls up to 8s before declaring a job missing instead of erroring on the first read
Next.js stream route hardened: msf- fallback ids are never proxied to Python (it cannot know them); the local fallback now emits the exact Python result schema (results.forecasts object with metrics/diagnostics/interval) — it previously wrapped forecasts in an array the panel silently couldn't read — and generatePredictions supports intraday intervals with session bar counts, HH:MM stamps, and bar-scaled volatility
Verified on production: 15m AAPL run returns a real Python UUID job streaming to completion with a 10-model ensemble (v2 ensemble + XGBoost + LightGBM + RF + N-BEATS + Temporal Attention + LSTM + GRU/LSTM-attention) trained on real 15-minute bars at the true ~$333.9 spot; three consecutive 1h MSFT runs plus stream re-reads completed with zero 'Job not found'; 7-day daily forecast regression-checked clean
Trading Hub readability audit (9 subtabs): Trade Explorer rewritten off dark MUI-blue #1976d2 labels to the dark-theme palette with purpose-colored sliders; Auto-Trader's textSecondary captions (rgba(0,0,0,0.6) — black on black) and unthemed MUI form controls fixed; Strategy Builder's #475569 captions lifted to #94a3b8 with themed Select dropdowns — all text now matches the platform-wide secondary-text standard

Phase 4.7 — Universal Stack Integration & True Auto Model Selection

Jul 2026
Shipped6/6
100% complete
Forecast audit finding: the Forecast subtab's DNA-scored 'auto-selected' model list was cosmetic — the frontend sent catalog keys (regime_router, patchtst_xl, cst…) the Python engine never recognized, so most supplementary models silently skipped and only the core v2 ensemble ran regardless of what the UI displayed
True auto model selection: in auto mode (default) the UI now sends NO model list — Python runs its entire suite and weights every member by inverse validation RMSE on the requested asset's own price history, so the best-performing models for that particular market earn the blend automatically on every run; hand-picking models in the architecture grid switches to manual mode (only chosen models run), and changing symbol or clicking Re-apply returns to auto — the run button and asset badge state which mode is live
Universal stack wired into the compute path: the real mali_core architectures — Cross-Sectional Transformer, TCN, N-HiTS (hierarchical interpolation), and Factor-Aware Model — now train per request on the asset's own history behind the same train/predict interface as the classic suite, and compete in the RMSE weighting; previously they existed only as catalog display entries
Mamba-TS built from scratch: a selective state-space model (input-dependent step sizes over a diagonal state matrix, SiLU-gated) — no implementation existed anywhere in the codebase; the scan is vectorised via log-space cumulative sums (a tight clamp keeps the drive/P division inside float32 precision — the naive clamp diverged training), and a random-walk head prior (zero weights, unit bias) makes few-epoch per-request fits start from a sane flat forecast instead of a 26% discontinuity
Rollout stability: the sequence models (CST/TCN/Factor-Aware) predict next-bar returns autoregressively — per-bar moves are clamped to ±6% and damped toward a random walk as the horizon extends, stopping a persistent one-bar bias from compounding into 50% drifts over 14 days
Verified at three levels: adapter smoke test (all five train, predict sanely, and raise cleanly on short history); end-to-end on live AAPL data — 15 models in the weighted ensemble including all five new members, with XGBoost earning 94% on that window (the auto-selection doing its job); production API confirms universal_stack_available: true with torch loaded

Phase 4.8 — SOTA Forecast Suite: Foundation Model, Meta-Learner & Conformal Bands

Aug 2026
Shipped6/6
100% complete
Chronos-Bolt foundation model live in production: Amazon's pretrained time-series foundation model forecasts zero-shot — the way GPT works on text — with no per-asset training; it learned universal price dynamics from millions of series and runs a 14-day forecast in 0.04s on CPU. Installed against the pinned transformers 5.4 stack with weights pre-cached server-side so no user request ever pays the download
Four more cutting-edge members join the suite: PatchTST (ICLR 2023 — non-overlapping price patches as attention tokens), iTransformer (ICLR 2024 — attention across variates instead of time steps), DLinear (AAAI 2023 — the decomposition-linear model that famously embarrassed far larger transformers on long-horizon benchmarks), and LightGBM-FE (gradient boosting on 15 engineered features: lag returns, rolling vol, RSI, momentum, drawdown, trend persistence — the quant-desk workhorse family)
Six superseded models retired with recorded rationale: N-BEATS (N-HiTS is its direct successor from the same authors), Temporal Attention (PatchTST/iTransformer are the peer-reviewed versions of the same idea), Keras LSTM + LSTM-Attention (GRU-Attention covers the family; dropping Keras LSTM removes TensorFlow from the request path entirely), Random Forest (weakest of three tree ensembles on identical features), and CST (its cross-sectional design needs many assets attending to each other — the single-asset slice voided its entire advantage). Net result: 13 models, every one architecturally distinct
Stacked meta-learner replaces pure inverse-RMSE weighting: non-negative least squares on the validation window learns which models were actually right — not just least-wrong on average — blended 50/50 with the inverse-RMSE prior so a tiny validation window can't hand everything to one lucky model; falls back to pure inverse-RMSE when scipy is unavailable. Diagnostics expose weighting_method on every run
Split-conformal confidence bands: the uncertainty envelope is now calibrated from the ensemble's own held-out errors (90th-percentile relative validation error, volatility-floored, √t-scaled) instead of assuming Gaussian returns — a 90% band that empirically contained the price 90% of the time on the validation window. conformal_q90 and band_method surface in diagnostics
Production incident found and fixed during verification: a stale uvicorn process from Aug 1 was squatting on port 8000 serving weeks-old code while the real systemd service crash-looped behind it — deploys were building successfully but the API never picked them up. Process killed, mali.service now owns the port; live AAPL forecast confirms all 13 models competing with stacked_nnls+inverse_rmse weighting and conformal bands active

Phase 4.9 — Catalog Honesty Audit, Quant-Desk Models & Diversity Cap

Aug 2026
Shipped7/7
100% complete
Full forecasting-model audit: every catalog entry was traced to its compute path. Finding — 11 entries never computed a prediction: xlstm_ts, regime_router, bayesian_corr, consensus_eng, tft, multi_horizon, haar_dwt, arima, ridge, and elasticnet were display-only (no implementation ran), and finbert's sentiment feeds its own panel but never the price blend. All 11 removed with per-model rationale recorded in the catalog's removed_models block — the catalog now matches the compute path exactly
The reverse problem fixed too: gru_attention and the always-on v2 statistical ensemble compute on every request but had no catalog entry — both added. The v2 engine's internal members were verified real by code inspection (KalmanFilter, ETS, AR(2), RSI-Momentum, GradientBoostedStacker, MultiTimeframeConfluence, PivotPointGate classes all exist in the engine) and are now labeled as engine components rather than separately selectable models
Quant-desk stack added — the econometric models trading desks actually run, pure numpy, under 50ms combined: GARCH(1,1) (Bollerslev's volatility-clustering model, variance-targeted MLE, drift shrunk by signal-to-noise so noisy drift collapses to a random walk), Ornstein-Uhlenbeck (closed-form AR(1) mean reversion on log-price — the stat-arb backbone, degrades to a random walk when no reversion is detected), and the Theta method (M3 forecasting-competition winner: extrapolated trend averaged with SES on the curvature-doubled series)
40% diversity cap: no single model may exceed 40% of the auto blend — water-filling renormalization freezes violators at the cap and redistributes the excess; loosened to 1/n below 3 models; explicit user custom weights are deliberately not capped
Frontend synced end-to-end: new Quant Desk family in the model grid, regime presets rebuilt on real computing keys only (no more phantom-model weights silently ignored by the backend), auto-select anchor moved from the removed regime_router/consensus_eng to Chronos-Bolt, codex Model Selection entry rewritten
Verified live in production: 16 models competing with the catalog matching the compute path exactly; GARCH earned third place on its first AAPL run; XGBoost capped at exactly 0.400 with the excess redistributed across 15 members; quant_stack_available: true confirmed
In-browser verification on mali-trw.com caught a rendering gap the API checks couldn't: the Architecture Selection grid's family list was hardcoded without the new quant family, so the GARCH/Ornstein-Uhlenbeck/Theta cards never rendered, and a stale 'Baselines (4)' collapsible survived despite zero baseline models remaining. Fixed (family list now includes quant; baselines section renders only when baseline models exist, with a dynamic count) and re-verified live: Quant Desk section renders all three cards, every SOTA member present, all 12 retired model labels confirmed absent

Phase 4.9.5 — Global View Live Data, Display Accuracy & ML Calculation Audit

Aug 2026
Shipped6/6
100% complete
11 Global View sub-panel display accuracy fixes: radarSent/radarCorr derived from live hub data (not hardcoded 0.5); MultiModePanel regime label now reflects live SSE state; vol rank uses Low/Average/High labels; performance 52w removes price*1.18/0.72 fakes; shockwave ring uses hub coordinates; CapitalFlowSankey LIVE badge logic corrected; MacroScenarioPanel unit fixed bps→%; MicroHeatmap deterministic arc paths; SectorHeatmapGrid best/worst uses active sort metric; GlobalIndicesTable flash animation fixed; FXRatesPanel bid/ask formatted consistently; VolSurface underlying picker now functional with distinct per-ticker IV params; AssetDrilldownPanel fly 25Δ sign bug fixed
CapitalFlowSankey wired to live SSE hub flow data: accepts flows prop from useDragonflyStreams market:flows channel, scales Sankey link widths by hub flow magnitude and direction every 10 s tick; falls back to 'Modeled' label when flows are empty — LIVE badge only shown on real data
Three ML calculation bugs fixed (audit-driven): ATR OHLC path shape mismatch in adaptive_stops.py (n,)-(n-1,) crash corrected; VaR/CVaR unit mismatch in risk_management.py (var_95 in dollars compared against fractional returns) fixed with separate var_threshold variable; Kalman filter covariance F@P@F.T symmetry restored in forecast_engine.py (was F@P@F, breaking Kalman gains)
Prediction band accuracy pass across MALI Lab: asset-type-specific daily vol (equity 1.2% / crypto 2.8% / forex 0.4% / ETF 1.0% — EURUSD was 43% wide, now ~12%); proper normal quantile z-multipliers (0.674/1.282/1.960 for 50/80/95%); horizon expansion capped at √10 to prevent runaway bands; Python ci_mult regime map tightened (volatile 1.80→1.25, crisis 2.50→1.65, calm 0.85→0.70)
Global View overlay layout fixes: removed inset:0 from .liquidity-surface CSS that created an unintended 600px opaque overlay blocking Globe3D layer buttons; reduced .flow-list max-height 280px→180px; removed redundant position:absolute wrapper divs that created unnecessary stacking contexts
Dashboard z-index fixes: SentimentPanel deep-dive drawer changed from position:fixed top:0/height:100vh to top:57px/calc(100vh-57px) so it no longer covers the dashboard header; MarketScannerPanel modal raised from z-50 to z-[200] so floating action buttons no longer paint over the modal backdrop

Phase 4.10 — Voice Engine Hardening, Market Calendar & Production Stability

Aug 2026
Shipped9/9
100% complete
Coqui XTTS V2 permanent infrastructure: added [program:voice] supervisord block so voice service survives container restarts; isolated /app/voice-venv (TTS==0.22.0 with its own transformers 4.33–4.43 pin) resolves conflict with the main stack's 4.47; libsndfile1 + ffmpeg added for audio I/O; XTTS V2 weights pre-downloaded at Docker build time so no request pays the model download
XTTS V2 cold-start elimination: background pre-warm thread spins up after FastAPI startup, loading the model into RAM before the first user request; Next.js AbortSignal timeout extended from 30 s to 90 s to cover warm-up lag; fallback chain is coqui → edge-tts → Web Speech API (503)
Market Calendar tab in Market Explorer News Feed: third 📅 tab alongside General News and Reuters; 30+ events Aug–Dec 2026 including FOMC decisions (Sep 16, Oct 28, Dec 9), Jackson Hole (Aug 28), monthly CPI/PPI/PCE/NFP, GDP advance estimate (Oct 29), ISM PMI, Q3 earnings season (Oct 13), and December triple witching (Dec 18)
Calendar UI: mini month calendar grid with prev/next navigation and importance-colored event dots (red = high, amber = medium); scrollable upcoming-events list with countdown labels (Today / Tomorrow / In Xd); click-to-expand event cards showing description, prior, and forecast values; 9-type filter pills (Fed, Inflation, Employment, Growth, PMI, Earnings, Treasury, Options, Other)
AI Intelligence null-crash fixes: NL Screener, Ensemble Forecast, and Institutional Flow panels guarded against null reference crashes that were silently killing panel renders on certain API response shapes
AI Flow Panel hardening: squeeze_potential.toFixed() crash fixed (API returned boolean; panel now coerces to numeric); requestFullscreen console noise silenced; squeeze score now emitted as 0–10 numeric (short-float% + days-to-cover blend)
Network error handling across 7 AI Intelligence panels: all silent fetch failures now surface user-facing error cards instead of blank panels — AI Flow, Ensemble Forecast, NL Screener, Institutional Flow, Daily Brief, Regime Detection, News Causality all catch and display network errors
Self-hosted production assets: Google Fonts moved off CDN to /public/fonts with local @font-face declarations; Font Awesome wired via @fortawesome/fontawesome-free npm package replacing cdnjs CDN link — platform assets now fully self-contained with no external CDN dependencies
Forecast Suggested Size: an earlier pass moved the Kelly stop from bandWidth/2 to bandWidth×0.15 and reported 7–15% sizing. That retuned a constant rather than fixing the formula, and it regressed to 0.0% once Phase 4.30’s drift shrinkage made the expected move small. Corrected properly in Phase 4.40, where the binary payoff form was shown to be structurally incapable of a positive answer here

Phase 4.11 — Calculation Accuracy Audit & Backend Hardening

Aug 2026
Shipped8/8
100% complete
Momentum score floor removed: ForecastInterface conviction engine had a +0.4 additive term in momentumScore that gave every trade a guaranteed 0.06 conviction floor even at zero momentum (flat path = flat bars, zero acceleration → still nonzero signal); removed so the momentum component is now truly zero when no momentum is present
Forecast annualized volatility corrected: vol was computed as std(predicted price levels) / entryPrice × √252, which inflates to ~88% for stable stocks because price levels carry price-magnitude variance; fixed to compute log returns of the predicted price path (log(p[i+1]/p[i])) then std × √252 — consistent with industry standard for return-based vol estimation
Kelly position sizing win_prob fixed (risk_management.py): was using 0.5 + volatility × 0.1, which rewarded high-volatility assets with a higher assumed win rate — the opposite of what volatility implies; replaced with neutral prior of 0.5 (actual win probability requires historical trade data not available at sizing time)
Portfolio max drawdown fixed (portfolio_service.py): max drawdown loop iterated over trades but compared peak against the constant current_value scalar, so running_max always equalled current_value and drawdown was always 0.0%; fixed to build a dated equity curve (cumulative cost + cumulative P&L per trade) and track the peak across that curve
Portfolio Sharpe ratio corrected (portfolio_service.py): previous version used per-trade pnl/cost as 'daily return' (wrong denominator and wrong time axis), population variance (N not N-1), and no risk-free deduction; fixed to build a proper equity-curve daily return series, apply sample variance (N-1), subtract daily RF (5.3% / 252), and annualize correctly
ATR close-to-close limitation documented (forecast_engine.py): ATR function uses only close-to-close changes and ignores H/L range and overnight gaps, understating true ATR by 30–50% in volatile sessions; callers now clearly directed to mali_core.features.indicators.TechnicalIndicators.atr() for OHLC-based True Range
Volume imbalance length guard (indicators.py): else branch of volume alignment was passing raw volume array v to np.where against direction (length len(p)−1), causing silent shape mismatch in any case where len(v) ≠ len(p); fixed to truncate or raise ValueError with an explicit length message
Correlation risk labelled as placeholder (risk_management.py): was silently using len(positions) × 0.1 as a proxy with no market data; now clearly labelled so it is never mistaken for a computed covariance-based measure

Phase 4.12 — Market Scanner Filter Hardening

Aug 2026
Shipped3/3
100% complete
Sector dropdown always populated: was built entirely from the live heatmap API response, so it showed only 'All Sectors' until the API returned; added a static STATIC_SECTORS fallback list (11 SPDR ETF sectors) used when heatmap state is empty, replaced seamlessly by live data once it loads
Market Cap size filter: new dropdown with five tiers — All Cap Sizes / Mega Cap (>$200B) / Large Cap ($10–200B) / Mid Cap ($2–10B) / Small Cap (<$2B); backed by a MARKET_CAP_TIER map assigning tiers to all 15 equity symbols in the scanner universe; ETF, crypto, and FX symbols pass through any cap selection
Price Range filter: new dropdown with six bands — Any Price / Under $10 / $10–$50 / $50–$200 / $200–$500 / Over $500; filters on live fetched price from each opportunity; both new filters apply client-side in the opportunities memo alongside the existing signal filter for instant response without re-triggering a full scan

Phase 4.13 — Forecast Tab Deep Calculation Audit

Aug 2026
Shipped14/14
100% complete
annualizedVol intraday fix: √252 hard-coded annualization replaced with horizon-aware factor √(252 × stepsPerDay[horizon]); 15m forecasts were understating volatility by 5×, 1h by 2.65× — now correct across all 5 horizons
Kelly / stop-loss consistency: rawKelly used bandWidth×0.15 as implied stop distance while the displayed stop was bandWidth×0.60 — a 4× mismatch; both now use a shared stopFrac=bandWidth×0.60 so position size and risk/reward reference the same risk distance
Momentum sign inversion guard: for preds.length ≤ 2, midPred equalled lastPred so momentum resolved to −mu (sign-inverted); all 1-day and 2-step forecasts were contributing a backwards momentum score to conviction; now returns 0 when forecast has insufficient steps to measure a path
horizonSignals proportional indexing: step=floor(N/5) caused 'T+14d' to show the day-7 price on a 14-day forecast, and the final label never reached the last prediction; replaced with proportional checkpoints [0, 0.25, 0.5, 0.75, 1.0] so labels always map to the correct forecast bar
Missing CI bands neutralization: when API returns no confidence interval bounds, bandWidth=0 was scoring bandTightness=1.0 (maximum conviction contribution) and maxDrawdownEst=0% (zero risk); hasBands guard now applies neutral bandTightness=0.5 and a vol-derived drawdown fallback instead
HOLD targetPrice: verdict=HOLD with negative mu was setting dir=1 and displaying a target above entry, indistinguishable from a BUY setup; HOLD now sets targetPrice=entryPrice with zero directional bias
pctChange base price alignment: ForecastCard header chip used preds[0] as base price while the Trade Signal panel used last_close — the two percentage displays could show opposite directions for the same forecast; both now use last_close consistently
STRONG HOLD oxymoron fixed: strength was computed independently of verdict, allowing conviction>0.65 + verdict=HOLD to display 'STRONG HOLD SIGNAL'; HOLD verdicts are now clamped to WEAK regardless of conviction
modelStd and logStd population→sample std: both used ÷N (population) instead of ÷N−1 (sample); for a 3-model ensemble modelStd was understated by 18%, inflating ensembleAgreement and cascading into conviction; both corrected to Bessel-corrected sample std
horizonSignals HOLD threshold unified: per-checkpoint HOLD threshold was 0.3% while main verdict used 0.2%; a BUY signal with mu=0.0025 could produce all-HOLD checkpoint chips; both now use 0.2% consistently
ensembleAgreement note label corrected: tooltip said 'cross-model σ/μ' but denominator was entryPrice not modelMean; corrected to 'σ/entry' to match the actual formula
Demo forecast loop rounding: .toFixed(2) was applied inside the price accumulation loop at each step, quantizing p to 2 decimal places before the next multiply — rounding error compounded over 90 steps; .toFixed now applied only at the push step, accumulation stays full precision
Wilder ATR in indicators.ts: atr() was computing a simple arithmetic mean of the last N true ranges; replaced with Wilder's smoothed EMA (seed with simple mean, then iterate (ATR_{n-1}×(period-1)+TR_n)/period) — the industry standard for ATR since 1978
kellyFraction comment disambiguated: indicators.ts uses half-Kelly (×0.5) capped at 20%; ForecastInterface uses quarter-Kelly (×0.25) capped at 25%; comment updated to document the distinction and cross-reference the forecast engine

Phase 4.14 — Dashboard-Wide Calculation Audit

Aug 2026
Shipped24/24
100% complete
VaR /√252 fix in QuantIntelligenceBar: parametric 1-day VaR now divides annual portfolio vol by √252 before applying z-score — was 15.87× too large, causing every equity portfolio to display CRITICAL risk permanently
Portfolio vol quadratic formula in greeks route: vol now computed as √(Σ(wᵢσᵢ)²) instead of Σwᵢσᵢ; the weighted-sum approach assumed ρ=1 (perfect correlation) and systematically overstated diversified portfolio volatility
Annualized return from daily mean in performance-summary: annualized_return now uses mean(dailyRet)×252 (unbiased daily-compounding estimate) instead of raw YTD return, which was neither annualized nor time-consistent
Proportional index off-by-one in performance-summary: i/(spyLen) caused the final SPY bar to map to index spyLen−2 instead of spyLen−1; corrected to i/(spyLen−1) with zero-length guard
Factor exposure column tracking in factor-exposure route: coefficient array shrinks when IWD or MTUM fails to fetch; fixed destructuring [,bm,bsmb,bhml,bmom] was mislabeling remaining factors — now tracks per-factor coefficient indices before OLS so labels survive any ETF fetch failure
NarrativeForecastPanel price-relative fractions: forecastAdjustment was in dollar units (signed×strength×4), causing thousand-dollar swings on $500 stocks; now a decimal fraction of price (×0.04 for NLP signals, ×0.025 for article-level)
NarrativeForecastPanel scenario replaces baseline: finalPrice was displayBase+totalAdj+scenarioAdj — double-counting the baseline adjustment; now finalPrice = displayBase×(1+scenarioAdj) so scenario is the single active interpretation
Backtest max_drawdown_pct formula: Math.min(return, return×1.3)×100 produced a positive drawdown for profitable scenarios and had no statistical basis for the 1.3 multiplier; corrected to a loss-magnitude formula with a small residual for profitable scenarios
AdvancedForecastHub weight array coverage: MODEL_META has 10 models but the weight array had only 8 elements — models 8–9 received weight:undefined→NaN; array extended to 10 elements summing to 1.00 across all three weight-array occurrences
ExplainToTradePanel R/R and stop-loss consistency: displayed stop applied Math.min(rawStop, price×0.97) but R/R denominator used raw rawStop — R/R was calculated against a different risk distance than what was displayed; both now use a shared displayedStop
DriftMonitorPanel vol model recalibration: Math.min(1, vol×3) triggered warning at ann.vol 10% and critical at 20% — the thresholds for a dead-calm market; recalibrated to (vol−0.10)/0.20 so warning starts at 20% and critical at 30% vol
DriftMonitorPanel param scheduler: learning_rate×100 was used as drift score (LR=0.003→score=0.3, always warning); replaced with drift_index×1.1 so the scheduler reflects actual concept drift rather than a fixed hyperparameter
Scanner volume ratio self-reference: volumes.slice(-20) included the current bar in the 20-bar average, making today's volume compare against a baseline that already contains it; corrected to volumes.slice(-21,-1) for the prior 20 bars only
Scanner RSI bonus monotone in value strategy: rsi<35→14, rsi<45→18 — deeper oversold (RSI 32) scored 4 points lower than moderate oversold (RSI 38); corrected to rsi<30→22, rsi<35→18, rsi<45→14, rsi<55→8, rsi<60→3 for strict descending bonus
Scanner SELL entry zone inversion: for SELL signals entryLow=price+0.1×ATR (above price) and entryHigh=price−0.3×ATR (below price) — Low>High; corrected so entryLow=price−0.3×ATR (always below) and entryHigh=price+0.1×ATR for SELL / price+0.2×ATR for BUY (always above)
Scanner T1/T2/T3 R:R from actual prices: targets previously used bt.rr×arbitrary multipliers (0.7, 1.0, 1.5) with no connection to actual stop or target prices; now computed as |targetPrice−entryPrice| / |entryPrice−stopLoss| for each target independently
Kelly sizing labels corrected in opportunity route: kellyFraction() already applies half-Kelly internally; calling code's conservative/moderate/aggressive multipliers (25/50/100) treated kf as full-Kelly, making all labels 2× too large; replaced with full-Kelly from raw winProb/rr and correct multipliers
Scenario probability coherence: Math.max(10, 100−bullP−bearP) allowed sum>100% when bull+bear exceeded 90; corrected to Math.max(0, 100−bullP−bearP)
ModelLeaderboard RMSE sort direction: ascending sort (lowest RMSE = best) was identical to descending sort due to identical sort expressions; corrected so descending=true returns lowest RMSE first (ascending numeric order)
ForecastTournamentPanel weight normalization: ensemble weights displayed as raw percentages from the API without normalizing — if the weight array didn't sum to 1.0 the percentages were wrong; now normalized by the actual sum before display
Autopilot division by zero guard: target[cls]/holdingsInClass.length crashed when no holdings existed in a target asset class (e.g. a cash-only portfolio with an equity target); added length===0 continue guard before division
RiskPanel alpha as percentage: alpha stored as decimal (0.03=3%) was displayed raw (0.03) without ×100 or % suffix, making alpha appear 100× smaller than every other percentage on the panel; corrected to (alpha×100).toFixed(2)%
DrawdownAttributionCard renamed to P&L Attribution: card computes per-position P&L contribution to total portfolio return, not peak-to-trough drawdown attribution; title corrected to eliminate the mislabeling
ConcentrationHeatmapCard colorscale stops: with zmin=−0.3 and zmax=1.0 (range=1.3) the colorscale fraction for ρ=0 is 0.231 not 0.30; all six internal stops recalculated as (ρ+0.3)/1.3 so color transitions now fire at the correct correlation values

Phase 4.15 — Market Scanner Watchlist Integration

Aug 2026
Shipped4/4
100% complete
+ Watch pill on every OpportunityCard in the Market Scanner: a cyan '+ Watch' button appears in each card's footer; clicking it calls the addWatchlistItem() server action and instantly transitions the pill to a green '✓ Watching' badge — no page reload required
Watchlist state pre-populated on scan: immediately after each scan completes, fetchWatchlist() is called server-side and the returned symbol set is stored in React state — symbols the user already watches show '✓ Watching' on first render without requiring a click
WatchlistPanel automatically reflects scanner additions: addWatchlistItem() writes directly to the watchlist_items DB table; the live-refreshing WatchlistPanel (30s auto-refresh) picks up new entries without any extra wiring
Loading state and disabled guard: the pill shows 'Adding…' with opacity-50 while the async server action is in flight, preventing double-adds; per-card adding state is local so multiple cards can be added concurrently

Phase 4.16 — Horizon UI Polish, Forecast Cone Fix & Dragonfly Primary

Aug 2026
Shipped4/4
100% complete
TradeSignalPanel horizon breakdown legibility overhaul: all near-invisible dark slate colors (#334155 / #475569) replaced with readable palette (#94a3b8 for secondary labels, #cbd5e1 for time labels and section headers, #7dd3fc for price values); minimum font sizes enforced at 0.60rem/0.62rem across section headers (HORIZON BREAKDOWN / SIGNAL REASONING / PRICE LEVELS / SIGNAL HISTORY), signal badge strength text, suggested size footer, and percentage change chips — all previously sub-0.60rem and below WCAG AA contrast on the dark background
Sonic brand rebuilt as zero-byte Web Audio API synthesis: the file-based Audio('audio/mali-startup.mp3') path was emitting a 404 on every session load (file never existed in public/); replaced with a pure Web Audio API chime — sub-bass sine punch (82→38 Hz, 0.55s decay), crystalline triangle (880→1108 Hz, 1.5s fade), shimmer overtone (1760 Hz, 1.3s fade); sessionStorage flag is set immediately on the first call so React StrictMode double-mount never replays the chime; zero bytes of audio data, no files, no CDN dependency
ForecastCone 1-day horizon fix: coneData useMemo now prepends a step 0 anchor point at last_close (from data.diagnostics.input_data_summary.last_close) before all prediction steps — guarantees data.length ≥ 2 for any horizon, including 1-day which returns only a single prediction step; null bound fallback changed from ?? 0 (producing zero-price degenerate SVG paths) to ?? anchor so missing bounds inherit the last close price; 'No confidence path data available' no longer fires on valid 1-day forecasts; useMemo dependency array updated to [preds, data.diagnostics] to pick up last_close
Dragonfly promoted to primary for all cross-worker operations: ml_multi_forecast.py _job_set now writes to Dragonfly first (30-min TTL) before falling back to in-memory dict; _job_get reads Dragonfly first before the local dict; dragonfly_client.py publish_message writes Dragonfly XADD + PUBLISH first, then the in-memory history; in-memory retained as same-process SSE fan-out cache and offline fallback; eliminates the cross-worker race condition where IONOS multi-gunicorn-worker deployments had Worker A store a job that Worker B could never find

Phase 4.17 — Auth Resilience, Registration UX & Stripe Promo Fix

Aug 2026
Shipped8/8
100% complete
bcryptjs password hashing in Neon: every new registration stores a bcrypt hash (cost 12) in the users.password_hash column — login now works end-to-end even when the Python FastAPI backend is unreachable
Login fallback chain: /api/auth/login tries Python /api/v1/auth/login first (8s timeout); on any failure (4xx, 5xx, network error) it falls back to Neon hash verification via bcrypt.compare — eliminates accounts that were permanently locked out because Python had no record of them
Home page login fallback wired: handleTryLoginStep1 in page.tsx also falls back to /api/auth/login when Python returns an error or is unreachable — both login paths now share the same resilient fallback chain
password_hash column added to Drizzle schema and Neon DB: ALTER TABLE users ADD COLUMN IF NOT EXISTS password_hash text; registration INSERT and UPDATE paths both write the hash so Neon stays in sync regardless of whether Python handled the registration
onboarding_completed column added to live Neon DB: ALTER TABLE users ADD COLUMN IF NOT EXISTS onboarding_completed boolean DEFAULT false; existing verified users backfilled (eula_accepted = true AND plan IS NOT NULL); was always false previously because the column existed in the Drizzle schema but had never been applied to the production database
Auto-redirect from /onboarding/done to /dashboard: after the finalize POST completes, a 1.8 s setTimeout fires router.push('/dashboard') — users no longer need to click 'Enter MALI →'; the button remains as a manual escape hatch for users who act before the timer
Stripe promo code flow confirmed working: TREESHA and FOAK4EVER both apply 100% off correctly via the Stripe 2026-06-24.dahlia API; promo shape validated (promotion.coupon not promotion.discount)
Register route stores hash on Python-created accounts: COALESCE(password_hash, $hash) UPDATE after Python success ensures the column is populated even for accounts Python created before the column existed in Neon

Phase 4.18 — Internal Direct Messaging System (Full Overhaul)

Aug 2026
Shipped15/15
100% complete
Zustand useMessenger store: full state atom — toggleOpen, setMode (panel|full), openRoom, loadMessages, sendMessage, addReaction, setReplyTo, connectSSE, disconnectSSE, fetchUnreadCounts, markRead, totalUnread(); sseCleanup lifecycle managed in store
SSE live stream: /api/messages/stream polls Neon DB every 2 s, pushes {type:'message'} and {type:'typing'} events to all connected EventSource clients; X-Accel-Buffering: no header for nginx pass-through
Optimistic UI: messages inserted to store immediately on send with optimistic- prefix; SSE deduplicates by checking content + author match within 5 s window so no double-render
Emoji reactions: UNIQUE(message_id, user_id, emoji) in message_reactions table; POST /api/messages/[id]/react toggles — DELETEs if exists, INSERTs if not; optimistic toggle in store; reactions aggregated as JSON in GET response
Reply-to threading: reply_to_id UUID on chat_messages with JOIN to fetch quoted content + author; Composer shows reply-to strip above input with ✕ to cancel; MessageBubble renders purple left-bordered quote block
Unread counts: message_read_state table — GET /api/messages/read returns per-room unread counts; POST marks read via UPSERT; ChannelSidebar shows cyan unread badges per room; DashboardShell header icon + FAB both show animated unread badge via totalUnread()
In-memory typing store: module-level Map<room, Map<userName, timestamp>> with 4 s TTL; POST /api/messages/typing calls setTyping(); SSE tick includes getTypingUsers() so typing indicator appears in MessageList and FullLayout header
GLM-4 → DeepSeek → Gemini Flash Latest → GPT-4o-mini AI chain: /api/messages/ai-analyze with Summarize / Catch Up / Analyze actions; 8 s timeout per provider with AbortSignal; QuickAIBar shows model attribution in response (e.g. 'MALI AI · via GLM-4')
MessageList: message grouping (consecutive same-author within 3 min), date separators, $TICKER cyan chips, @mention purple text, reactions row, reply-to quote block, hover action bar (emoji picker / reply / copy), animated typing dots, avatar color derived from name hash
Composer: $ticker autocomplete dropdown (30 popular tickers, prefix-filtered, 6 shown), /slash command dropdown (/brief /analyze /catchup /price), 20-emoji picker grid, auto-resize textarea capped at 120 px, char counter (red at 1800/2000), animated send button with spinner, debounced typing notify with 2.5 s stop timer
ChannelSidebar: collapsible Market Rooms and DMs sections, unread badges, presence dots per DM user, total DM unread on section header, user search input for large user lists, New DM placeholder button
FullLayout: live ticker price chip (60 s refresh) for ticker rooms and global-markets/SPY, active participant count from last-hour messages, typing indicator in header bar alongside SSE live dot, QuickAIBar in footer
IntelligencePanel: live price chip (60 s auto-refresh), room sentiment (Bullish/Bearish/Neutral derived from last 10 messages via keyword scoring), mentioned ticker chips extracted from last 30 messages, live news feed with source + sentiment labels, activity summary (message count + unique participants)
MessengerShell: total unread badge pill next to MALI HQ brand, pulsing SSE live indicator, SVG icon expand/collapse and close controls; header always-visible even in full-screen mode
DB migrations in ensureAllTables(): ALTER TABLE ADD COLUMN IF NOT EXISTS reply_to_id UUID / edited_at / message_reactions (UNIQUE constraint) / message_read_state (UPSERT-ready) — raw SQL, no schema.ts touch

Phase 4.19 — API Proxy Gateway Hardening

Aug 2026
Shipped9/9
100% complete
Root cause fixed: 13 prefetch endpoints called on login splash were hitting the Next.js fallback rewrite proxy (no matching route.ts), which returned raw 500s when Python was unreachable; all 13 now have dedicated route handlers with graceful JSON fallbacks
Market intelligence handlers: /api/python/market/fear-greed, /market/52week, /market/bid-ask, /market/earnings-calendar, /market/intraday-range — all proxy to FastAPI with query-string passthrough and typed fallback responses
/api/python/market-data: general market data proxy with query-string forwarding; fallback {success:true, data:{}}
Dashboard v1 handlers: /api/python/v1/dashboard/portfolio-health (fallback score:75, label:HEALTHY), /ai-briefing (fallback sentiment:Neutral, score:50), /alerts-timeline (fallback alerts:[]), /opportunities (fallback opportunity:null)
/api/python/dashboard/recent-activity: proxies with user_id/limit query params; fallback {recent_activity:[], activities:[], count:0}
/api/python/news: proxies news with q/limit/category params; fallback {success:true, articles:[], news:[]}
/api/python/business/subscription: proxies to /business/subscription (no /api/ prefix per rewrite rule); fallback {plan:'free', active:true}
All 13 handlers use export const dynamic = 'force-dynamic', try/catch with empty body, and NextResponse.json() fallbacks — Python outage produces 200 with graceful data instead of 500 console error
portfolio_summary NameError fixed in dashboard_intelligence_v1.py: function called at line 63 was never imported, silently returning success:false; replaced with safe market_service.get_portfolio_summary() call guarded by hasattr

Phase 4.20 — Keyboard Shortcut System

Q4 2026
Shipped13/13
100% complete
Keyboard Shortcut Dictionary: 6-category reference (Navigation / Messenger / Market / AI / Trading / General) embedded in Roadmap tab with live/planned status indicators and per-key Kbd chips
Global keydown listener (useKeyboardShortcuts hook): G+D/M/L/T/A/W/C/S chord navigation to workspaces; Alt+←/→ workspace cycling; 1–9 tab jump; auto-suppressed in text fields
Ctrl+K command palette (CommandPalette.tsx): fuzzy-searched list of all workspaces, tabs, messenger rooms, and quick actions — keyboard-navigable with ↑↓/Enter
Ctrl+Shift+M toggle messenger: wired to useMessenger().toggleOpen() — works from any workspace without focusing a button
Ctrl+Shift+F full-screen messenger: setMode('full') + setOpen(true) from anywhere in the app
Ctrl+Shift+1–5 room jump: jump to the 5 main messenger rooms by keyboard without opening the sidebar
Ctrl+R data refresh: dispatches mali:refresh CustomEvent that live panels can subscribe to
? / Ctrl+/ shortcut overlay (ShortcutOverlay.tsx): full-screen modal with category sidebar, Kbd chips, and live/planned status — opens from any non-text context
Escape key closes messenger panel from anywhere outside a text field
Ctrl+F ticker focus: dispatches mali:focus:ticker CustomEvent — panels with a ticker search input subscribe and focus it on fire
Ctrl+Shift+B (AI Brief) / Ctrl+Shift+A (Agent Workspace) / Ctrl+Shift+E (Explain Trade): direct workspace+tab navigation shortcuts
Ctrl+Shift+Z/C/I/T: Chart Studio shortcuts — zoom jump, clone dispatch, indicator picker dispatch, timeframe cycler (1m→5m→…→1W) via mali:chart:* CustomEvents
Quick-record hotkeys (Ctrl+B / Ctrl+S): record a purchase or sale in the purchases ledger, pre-filled with the active ticker; MALI places no orders

Phase 4.21 — Global Event Bus: Platform-Wide Refresh & Ticker Focus

Aug 2026
Shipped7/7
100% complete
useMaliRefresh hook: single-import utility that wires any panel's reload function to the mali:refresh CustomEvent — stable useCallback + addEventListener/removeEventListener pattern, zero per-panel boilerplate
mali:refresh wired to all 75+ data-loading panels: every panel that fetches remote data (AI, ML, trading, analytics, social, agent, and market panels) now reloads live on Ctrl+R — zero missed panels, zero regressions (TypeScript clean compile confirmed)
mali:focus:ticker global DOM handler in DashboardShell: single useEffect queries <main> for the first visible, enabled text/search input and focuses + selects it — covers every panel with no per-panel changes required
mali:focus:ticker per-panel precision handlers: MarketScannerPanel (symbol search bar), WatchlistPanel (add-symbol input), and NewsPanel (ticker filter input) each register their own handler that fires before the global DOM fallback, targeting the specific contextually relevant field
NewsPanel ticker filter: real-time client-side filter across article title, description, and NLP summary — match count display, clear button, and Ctrl+F focus hint in placeholder; allArticles/articles rename for clarity
MarketScannerPanel symbol search: filters opportunities by symbol or name instantly without re-triggering a scan; SVG search icon, clear button, uppercase normalization; mali:refresh re-runs the full scan
WatchlistPanel Ctrl+F focus: focuses the existing Add to Watchlist input with a (Ctrl+F) placeholder hint — no UI change required, pure listener wiring

Phase 4.22 — Messenger Layout & Scroll Hardening

Aug 2026
Shipped6/6
100% complete
custom-scrollbar CSS class defined in globals.css: thin 5px cyan-to-teal gradient scrollbar (scrollbar-width:thin + webkit rules) — was referenced in 6 files but never declared, so no scrollbars appeared in MessageList, ChannelSidebar, or Composer
MessageList flex height chain fixed: messages wrapper in both FullLayout and PanelLayout gained flex flex-col so MessageList's flex-1 had a flex parent to grow into; compact mode missing min-h-0 fixed — without it overflow-y-auto never triggers regardless of content height
Composer textarea scrollbar: added overflow-y-auto custom-scrollbar to the textarea element so text scrolls within the 120px max-height cap instead of overflow:hidden clipping
Autocomplete dropdown z-index raised from z-30 to z-[60] on all three dropdowns (tickers, slash commands, emoji picker) — prevents clipping inside overflow-y-auto stacking contexts created by the message list
QuickAIBar footer trim: removed mt-3 (12px redundant top margin) and the LLM chain attribution line (✦ GLM-4 · DeepSeek · Gemini · GPT-4o-mini) saving ~32px of footer height in panel mode; gap-2 tightened to gap-1
Footer padding tightened in PanelLayout (pb-3 pt-2) and FullLayout (pt-2 pb-4) with border-t border-white/[0.07] added to PanelLayout footer for visual separation

Phase 4.23 — Stripe Billing Hardening & Payment Recovery

Aug 2026
Shipped6/6
100% complete
GET promo code validation fixed: /api/onboarding/payment GET handler used promo.promotion.coupon which threw TypeError when promotion was undefined (2026-06-24.dahlia API shape); now mirrors POST handler — tries pc.coupon (classic) first, falls back to pc.promotion?.coupon with safe optional chaining; valid promo codes no longer show as invalid in the UI
All Stripe price IDs written to IONOS server .env.local: server had zero STRIPE_PRICE_* vars — every activate call fell through to local mock activation, which is why all stripe_subscription_id values in the DB were null; added Pro Monthly/Annual, Edge Monthly/Annual, and public variants
STRIPE_PRICE_ENTERPRISE cleared: placeholder price_3AbCdEfGhIjKlMnO removed from all env files (local, production, IONOS) — enterprise activations now fall back to the local activation path until a real Enterprise price is created in Stripe Dashboard
Stripe secret key rotated and propagated: old sk_live_...JgOv was invalid in Stripe's API (key had been rolled at some point without updating env files); new key sk_live_...vGIs confirmed working; updated in .env.local, .env.production, and IONOS server via SSH; IONOS Next.js rebuilt and restarted
Null subscriber root-cause analysis: 4 real subscribers (tamika.gps, edelmanfarms, 38consuela, lucianondongmba) had stripe_subscription_id = null; found their cus_ IDs belong to a previous Stripe account (now inaccessible); no charges were ever collected; users need to re-complete checkout against the current Stripe account
Stripe readiness audit: confirmed live keys, webhook secret, price IDs, webhook handler at /api/billing/webhooks/stripe (Python), billing portal, payment flow, and promo codes all operational; identified and fixed the two remaining gaps (GET promo bug + missing price IDs on server)

Phase 4.1 — Next Evolution Cycle

Aug 2026
Shipped10/10
100% complete
4.1 — IV Surface panel (MALI Lab): strike×expiry heatmap from live Yahoo options chain — blue→red IV gradient, ATM term-structure line chart, nearest-expiry skew chart; tab iv_surface in mali_lab workspace
4.2 — PatchTST promoted from staging → active in ml_registry/model_registry.py: version bumped to 1.0.0, tag updated; MLRegistryPanel badge reflects active status
4.3 — RL Paper Trading panel (Trading Hub): P&L curve, position summary, trade log, Sharpe/max-drawdown stats, kill-switch toggle; proxies /api/python/v1/ai/nextgen/intraday-agents/status; kill and resume go to the desk's own routes (/api/python/v1/ai/nextgen/intraday-agents/kill, /resume), admin only; 10s auto-refresh
4.4 — Contagion Map panel (Agent Workspace): SVG force-directed graph of 8 assets (SPY/QQQ/TLT/GLD/BTC/EUR/VIX/XLE) with correlation-weighted edges, centrality-scaled nodes, sector color coding, plus 8×8 correlation matrix heatmap; 30s refresh from /api/python/brain/status
4.5 — Audit Log panel (Workspace Config, admin-only): paginated 50/page event table, severity/category filter pills, actor search, CSV export via Blob; proxies /api/python/governance/events with graceful fallback
4.6 — SEC 8-K Feed: /api/sec-filings route fetches SEC EDGAR Atom RSS (public, no key needed), keyword-based sentiment scoring; scrollable collapsible feed section added to ResearchDeskPanel with company/date/sentiment badge/filing link
4.7 — HMM regime alerts → push notifications: detect regime shifts in GaussianHMM → publish to Dragonfly → SSE push to frontend → browser toast + optional email via Resend
4.8 — Model A/B Testing panel (MALI Lab): dual model selectors, parallel forecast fetch, side-by-side MAPE/RMSE/R² metrics, winner badge, auto-promote banner; /api/python/ml/models/promote proxy route
4.9 — Versioning UI panel (Workspace Config): glowing current-version badge, scrollable changelog, rollback button per entry, Create Snapshot button; proxied via /api/python/versioning/ and /api/python/versioning/changelog/
4.10 — MALI SuperAgent panel (Agent Workspace): reads all 12 agent health scores, underperformer detection, active sub-task queue, improvement proposals from /api/python/evolution/proposals, Dispatch SuperAgent button with SSE-style polling progress; 30s auto-refresh

Phase 4.25 — Portfolio Tab & Alerts Tab

Aug 2026
Shipped4/4
100% complete
Portfolio tab (persistent sidebar): complete holdings command center with 8-view tab bar — Overview (live P&L, positions, allocation pie), The Vault (holdings table with real-time prices from FastAPI bridge tables: faPortfolios / faHoldings / faTrades), Factor Exposure (Fama-French 7-factor beta matrix), P&L Attribution waterfall (per-position contribution sorted worst→best), Historical Stress Tests (7 crisis scenarios), Tax-Loss Harvesting (IRS §1091 wash-sale filter, 35% tax savings estimate, replacement suggestions), Concentration Risk heatmap (21-pair cross-asset correlation matrix, Plotly blue→red colorscale), Portfolio Greeks (beta-weighted Delta per $1 move, Gamma, Theta, Vega)
Portfolio data pipeline: FastAPI bridge tables as authoritative source; Next.js server actions query Neon directly when Python is unreachable; live Yahoo Finance price enrichment per holding; QuantIntelligenceBar with parametric VaR 95% 1-day (annual vol ÷ √252 × z-score), HHI concentration index, AI projection signal from /api/portfolio/technical-ai-signals, and anomaly detection (crypto overweight / HHI spike)
Alerts tab (persistent sidebar): dual-layer alert system — (1) Price & condition alerts: ticker alerts with trigger conditions (price above/below, % move, RSI threshold), alert history log, and create-from-chart workflow; (2) HMM Regime Shift Alerts: automatic push notifications on every GaussianHMM state transition (risk_on / risk_off / vol_spike / liq_crunch / macro_window / neutral / crisis) — color-coded toast notifications that persist 9 s for critical alerts and 6 s otherwise; last-20 transition history persisted in localStorage key mali:regime:alert_history; each toast shows the transition label, confidence %, and top 2 regime drivers
Regime alert infrastructure: regime_alert_service monitors market:regime Dragonfly channel and publishes to mali:regime:alerts SSE stream; broadcaster tick() also emits regime:alert events client-side so toasts fire even without a FastAPI connection; optional email delivery via Resend API (RESEND_API_KEY + MALI_ALERT_EMAIL env vars); toast color encodes urgency — green (risk_on/calm), blue (neutral/macro_window), purple (trending), orange (risk_off/volatile), red (vol_spike/liq_crunch/crisis)

Phase 4.24 — Next-Level Direct Messages

Aug 2026
Shipped13/13
100% complete
Canonical DM room IDs: fixed two-directional chat bug where dm-{A} and dm-{B} were separate rooms; new canonical format dm-{sorted|ids} ensures both parties share the same room; helper functions (dmRoomId, isDmRoom, dmParticipants, dmPartnerId) exported from state/messenger.ts
DM unread counting fixed: /api/messages/read dynamically queries all DM rooms the user participates in instead of 5 hardcoded public rooms; DM sidebar unread badges now work correctly
Real presence tracking: user_presence table (heartbeat every 30s via POST /api/messages/presence); /api/messages/users now joins presence to return isOnline + status (online/trading/analyzing/away/offline); sidebar dots animated with glow per status
Agent command system: /chart SYMBOL (live price + technical analysis), /scan [news] SYMBOL (market or news scan), /risk SYMBOL (risk report + stop levels), /summarize (thread summary), /alert SYMBOL CONDITION (creates dm_alerts DB record), /translate LANG [text] — processed by /api/messages/commands via GLM→DeepSeek→Gemini→GPT cascade; result injected as agent-type message
Ticker hover cards in MessageList: hover any $SYMBOL or $TICKER chip to see live price, % change, volume bar, and trend indicator — fetches /api/market/quote/{symbol} inline without leaving the chat
Agent message type (msg_type = agent): markdown-lite rendered with bold headings, bullet points, and formatted content; distinct glassmorphism card style with cyan border — distinguishes AI command results from human messages
DmAIBar: new AI tools bar specific to DM rooms — Smart Replies (3 AI-generated chip replies clickable to send), Translate (10-language dropdown, translates last message), Summarize (2-3 bullet thread summary), Sentiment (color-coded card with urgency flag and key themes) via /api/messages/ai-dm
Vault mode: per-DM-room toggle in header (amber 🔒 VAULT badge + shield icon); amber bubble border on messages; vault mode banner in message list; persisted in Zustand vaultRooms Set
Message expiry countdown: ExpiryBadge component reads expires_at column and shows live HH:MMm countdown ticker; expired messages hidden from view
Enhanced DM header: shows partner's real name, email, and animated presence dot; partner online status ('Online', 'Trading', 'Analyzing', 'Away', 'Offline'); vault toggle button; market session chip (🗼 Tokyo / 🇬🇧 London / 🗽 NY / 🌙 After Hours)
Functional 'New Direct Message' modal in ChannelSidebar: full-screen search overlay with name/email filter, online status badges, click to open canonical DM room
Composer upgrade: full command list with Agent (cyan) and AI (purple) groups; composer border turns cyan for commands and Send button becomes 'Run ▶'; slash autocomplete shows group, command, and description in 3 columns
IntelligencePanel hidden in DM view (market intelligence right panel is public-room-only); DmAIBar replaces QuickAIBar in DM footer; full modular architecture following dm-core / dm-ai / dm-market / dm-commands / dm-presence layer pattern

Phase 4.26 — Global View Globe: 10 Financial Intelligence Enhancements

Aug 2026
Shipped10/10
100% complete
Enhancement 1 — Live Market Hours Terminator Ring: 8 global exchanges (NYSE, LSE, TSE, HKEX, NSE, ASX, Xetra, B3) tracked by UTC fractional hours; open exchanges emit slow continuous colored rings (repeatPeriod 3500ms) on the globe surface; real-time exchange status panel (top-right) shows animated open/closed dot per exchange with regime-colored border; updates every 60 seconds; open exchange hubs tinted gold to distinguish from closed-market hubs in sentiment mode
Enhancement 2 — Dynamic Capital Flow Arc Streams: when market:flows SSE channel carries data, real cross-border flow pairs rendered as weighted teal arcs between hubs (stroke and opacity scale with flow volume); falls back to 8 static major global capital corridors (US→EU, US→JP, EU→HK, HK→JP, US→IN, EU→BR, US→DE, JP→AU) when SSE is empty; distinct from contagion trails — always visible in all layer modes alongside the existing arc stack
Enhancement 3 — Volatility Index Fear Dots (VIX Cousins): hubs with volatility > 0.25 emit pulsing orange/crimson rings; VIX/VSTOXX/VXJ/VHSI metadata mapped per hub (ny→VIX baseline 18, london→VSTOXX baseline 20, tokyo→VXJ baseline 22); fear level normalized 0–1 from volatility/0.5; high fear (>0.6) → rapid crimson ring (repeatPeriod ~500ms, propagationSpeed 8), low fear → slow amber ring; derived vol-index level displayed in hub hover tooltip
Enhancement 4 — GDP & Growth Bubble Layer: new 'gdp' layer mode renders 12 major economies as Three.js SphereGeometry objects floating at altitude 0.15 above the globe; sphere radius scales proportionally to GDP (USA $27.4T → radius 6.5; Mexico $1.3T → radius 1.8); color encodes YoY growth rate (green = positive, red = contraction); GDP data covers USA/China/Germany/Japan/UK/India/France/Brazil/Canada/S.Korea/Australia/Mexico; toggleable via layer button
Enhancement 5 — Cross-Asset Correlation Web: new 'corr web' layer mode replaces contagion trails with static arc connections between all hub pairs with |correlation| ≥ 0.55 from the live correlations prop; positive correlation arcs = gold (rgba(255,215,0,...)), negative = indigo (rgba(150,0,255,...)); arc stroke weight scales with magnitude; de-duplicated with seen-pair tracking; turns the globe into a 3D live visual correlation matrix
Enhancement 6 — Currency Strength FX Rings: Three.js TorusGeometry rings always rendered at each hub (visible in all layer modes) via the custom object layer; torus oriented flat on the globe surface (Z-normal → radial); green torus = currency strengthening (hub.changePct ≥ 0), red = weakening; ring tube radius and opacity both scale with |changePct|/3 (0–1 normalized); merges into unified customObjectsData alongside spikes and GDP spheres without custom layer conflicts
Enhancement 7 — Macro Event Countdown Detonators: calendar items (from the calendar prop / MacroCalendar SSE channel) within 24 hours generate orange rings (color [255,165,0]) with maxR that shrinks from ~20 (24h out) to ~5 (imminent) via hoursAway×0.63+5; repeatPeriod accelerates from 2000ms to ~1000ms as the event approaches; propagationSpeed increases as T-minus shrinks; rings update every 60 seconds; integrates with existing shockwaveBus to fire a detonation ring at event time
Enhancement 8 — Earnings Season Regional Beat/Miss Trails: active in performance mode only; hubs with |changePct| > 0.8% emit 4 radial short arcs at 45°/135°/225°/315° bearings with distance 7+|changePct|×2 degrees; beat (positive changePct) → gold arcs rgba(255,215,0,0.55), miss → crimson rgba(255,50,80,0.45); arc stroke scales with magnitude; visually distinguishes strong regional earnings momentum from weak; part of unified allArcs array alongside capital flows and contagion trails
Enhancement 9 — Sector Rotation Orbital Arc Segments + Labels: HUB_SECTORS static data defines 4 sectors per major hub (ny: Tech 32%/Fin 18%/Health 14%/Energy 8%; london: Fin 28%/Energy 22%/Materials 12%/Consumer 10%; tokyo: Auto 25%/Tech 22%/Fin 18%/Industry 15%; berlin: Industry 28%/Auto 22%/Chemicals 15%/Fin 12%); thin arc segments drawn at ~7° orbital radius around each hub, arc span proportional to sector weight, colored by sector; floating sector label above each hub shows top-mover sector with P&L pct; hub hover tooltip upgraded with full sector breakdown table sorted by magnitude
Enhancement 10 — Sovereign Yield Spike Propagation: macro events whose title matches /rate|yield|bond|boe|ecb|fed|boj|central bank/i trigger an 8-second gold arc burst (rgba(255,220,50,0.55), stroke 0.75) from the originating hub to all other hubs simultaneously; arcs stored in yieldArcsRef.current and included in allArcs via yieldArcTick state trigger; cleared after 8 seconds via setTimeout; de-duplicated by lastYieldEventTsRef to prevent re-firing on re-renders; visually separates rate-shock contagion from correlation-based contagion trails

Phase 4.27 — FastAPI Startup Hardening & Import Resolution

Aug 2026
Shipped11/11
100% complete
FastAPI 0.136.1 Depends conflict fixed across 6 routers (alternative_data, institutional_flow, ml_registry, governance, versioning, ensemble_forecasting): `= Depends(get_current_user)` alongside `Annotated[dict, Depends(get_current_user)]` is rejected by FastAPI 0.136.1 — all 6 default values replaced with `= None`; also fixed single-line variant `= Depends(get_current_user)):` → `= None):`
auth_core import alias in core/feature_gate.py and core/usage_meter.py: app.core.__init__.py namespace bridge makes root core/ resolve as app.core.*, so from auth_core import get_current_user failed (auth_core.py only exports get_current_active_user); both files now alias `from auth_core import get_current_active_user as get_current_user`
trades_v1 and enterprise_auth guarded imports: MarketService, MainLegacyDashboardPortAdapter, and three enterprise service classes (EnterpriseAdminService, LDAPService, SAMLService) wrapped in try/except with stub fallback classes so missing dependencies no longer crash router load
backtesting_v1 slowapi fix: @limiter.limit() requires request: Request as first route parameter — added to compare_strategies endpoint; added Request to the fastapi import line (name was missing, causing NameError on startup)
paper_trader_service raw SQL fix: ORM _load_accounts_from_db selected portfolios.owner_user_id which does not exist in the live DB schema; replaced with raw text() SELECT of only the 4 existing columns (user_id, total_value, cash_balance, created_at)
observability.py bootstrap() added: main.py calls bootstrap() at lifespan startup but the function never existed; added returning a dict of Sentry and PostHog init results (no-DSN / enabled / error states)
database.py get_async_session() added: services/mali_agents/coordinator.py imports get_async_session but database.py only exported get_db_session; added as a separate @asynccontextmanager alias with identical semantics
regime_service f-string fix: f'{ind.previous_value:.2f if ind.previous_value else "N/A"}' — Python evaluates the format spec as the string 'N/A' (an invalid float spec) when previous_value is None; restructured to `f'{ind.previous_value:.2f}' if ind.previous_value is not None else 'N/A'`
real_market_data datetime subtraction fix in get_live_quote(): cache freshness check subtracted cached_data.get('timestamp') (an ISO string field on the quote dict) from datetime.now() instead of the cache wrapper's datetime object — TypeError at runtime; fixed to read cache_ts = self.cache[cache_key].get('timestamp') from the wrapper
agents_registry table created in Neon Postgres: coordinator._sync_registry_to_db() upserts to agents_registry on every startup cycle but the table never existed; DDL applied directly via psql (agent_id TEXT PK, name, role, version, enabled, schedule, trigger, created_at, updated_at)
Next.js ensemble-backtest _lib import path fixed: route.ts imported seededRand from '../../_lib' (two directories up, resolving to api/_lib which doesn't exist); correct path is '../_lib' (one directory up to api/ai/_lib.ts where seededRand is defined at line 118); Next.js build now compiles cleanly

Phase 4.28 — Model Selection Tab: Auto Mode, Feature Parity & UI Hardening

Sep 2026
Shipped10/10
100% complete
Auto mode (NNLS) set as default: Model Selection tab now omits the models payload in auto mode so Python runs all available models with the NNLS stacked meta-learner and asset-type specialist priors — identical to the Forecast Engine's default behavior; Manual mode is the secondary toggle that reveals the checkbox grid
Model count corrected from 22 to 13: previous implementation used data.models from the /ml/ensemble-models API which returns all 22 catalog entries including v2 engine sub-components (Kalman, ETS, AR2, RSI Momentum, GB Stacker) and signal gates (Multi-Timeframe Confluence, Pivot Gate) that are not independently blended; fixed by importing MODEL_CATALOG directly from ForecastInterface as single source of truth
MODEL_CATALOG imported from ForecastInterface: ModelSelectionPanel now shares the same 13-model set (PatchTST, iTransformer, Chronos-Bolt, TimesFM, Mamba-TS, V2 Ensemble, N-HiTS, XGBoost, LightGBM-FE, HAR-RV, GARCH-MIDAS, Neural SDE, Theta) as the Forecast Engine's model picker — both tabs are now guaranteed to reference the same models
Stress scenario dropdown added: Model Selection tab now exposes None (Baseline), Market Crash (−0.6%/day), Bull Run (+0.6%/day), High Volatility, Inflation Shock, and Mean Reversion — matching the full stress_scenario parameter surface already available in the Forecast Engine
Confidence interval selector added: 80%, 90% (default), 95%, 99% CI options now control the conformal prediction band width in the Python backend via the confidence_interval parameter; fetchForecast in api.ts extended to accept the optional confidence_interval argument
Input label and text color hardening: INPUT_SX design token applied to all four inputs (symbols, horizon, stress, confidence) — labels render as #8b93b3 at rest, #00d4ff when focused, and input values are #e2e8f0 white regardless of focus state; fixes dark-text-on-dark-background legibility bug present since the tab was built
Result chips redesigned: each symbol output card now shows an Auto NNLS / Manual mode badge, an orange scenario chip when a stress test is active, a purple CI% chip, and the ensemble model string chip — giving full run-parameter visibility directly on the result
Ensemble weighting panel upgraded: model name labels in the weighting breakdown now use human-readable MODEL_CATALOG labels (e.g. 'LightGBM-FE', 'HAR-RV') instead of raw underscore-separated key strings
Per-model checkbox colors from MODEL_CATALOG: each model checkbox in manual mode renders its brand color (e.g. #00d4ff for V2 Ensemble, #fbbf24 for Chronos-Bolt, #f43f5e for HAR-RV) for visual identification matching the Forecast Engine's model grid
MUI slotProps compatibility fix: MenuProps.PaperProps replaced with slotProps.paper for Select dropdown dark-background styling to comply with MUI v6 API — resolves TypeScript build error on IONOS deploy

Phase 4.29 — Forecast Engine: 15-Point Expert Accuracy Overhaul

Sep 2026
Shipped14/14
100% complete
Return-space NNLS calibration (Finding 1): ensemble weights now calibrated on log(pred/base_price) — a stationary return series — rather than raw price levels; fixes the I(1) non-stationarity that caused models to exploit price scale rather than actual dynamics, correcting a root-cause accuracy issue across all 13 blend members
Preprocessing leakage fix (Finding 2): outlier clipping and z-score normalization now computed exclusively from prices[:-v_days]; future data no longer contaminates the training window's statistics, preventing the validation window from leaking information into preprocessing
Calibration window raised to 90 days (Finding 3): v_days increased from 30 → 90, giving a ≥6:1 observation-to-parameter ratio for stable NNLS weights across 13 models — the previous 2.3:1 ratio was under-determined and produced noise-driven weight estimates
GARCH conditional vol as CI floor (Finding 4): GARCHForecaster.last_var is extracted after fitting and used as the volatility floor for confidence band computation; GARCH reflects current vol-clustering regime rather than a historical average, producing tighter bands in calm markets and wider bands when volatility is elevated
Walk-forward NNLS with 5-fold expanding CV (Finding 5): replacing single-split NNLS; coefficients fitted on each expanding prefix of the validation window and averaged across folds — eliminates regime-lag bias where weights were calibrated to the regime that just passed rather than current conditions
Regime-conditional weight routing (Finding 6): _detect_regime() classifies the last-60-bar regime via Hurst exponent approximation, cumulative return, and vol ratio into 8 states (bull_low_vol, bull_high_vol, bear_low_vol, bear_high_vol, mean_reverting, trending, high_vol, neutral); _apply_regime_boost() multiplies per-model weights by empirically-motivated factors and renormalizes — e.g. trending boosts Mamba-TS/iTransformer, bear_high_vol boosts GARCH-MIDAS/HAR-RV/Neural SDE
Volatility-scaled stress scenarios (Finding 8): crash, bull, high_vol, and inflation_shock drift magnitudes now calibrated to 1.5–2.5× the asset's own daily_vol rather than fixed ±0.6%/day; a BTC crash is 2× BTC's vol, a TLT crash is 2× TLT's vol — each scenario is now proportionally consistent across all asset classes
Drift-separated CI scaling (Finding 9): confidence band margins now combine a deterministic drift component (scales as t, accounts for systematic trend) with a stochastic residual component (scales as sqrt(t), accounts for random uncertainty); previously both components used sqrt(t) which caused artificially wide bands on trending assets
Earnings calendar CI widening (Finding 10): _earnings_dates_near() fetches Finnhub earnings calendar for the forecast horizon; confidence intervals expand 2.5× on dates ±1 day from a scheduled earnings announcement, and earnings_flag is surfaced per prediction step — reflects the empirical ±4–8% earnings-day uncertainty vs ±1.2% on ordinary days
Foundation model return-space calibration (Finding 11): Chronos-Bolt, TimesFM, and Moirai predictions are now weighted in the same log-return NNLS framework as all other models, correcting the previous mismatch where zero-shot foundation models with different price-scale conventions were averaged with train-on-demand models using raw price NNLS
Per-asset model caching (Finding 12): _get_or_train() caches fitted model objects per (symbol, date, model_key) with a 15-minute TTL; repeated forecasts for the same symbol within a session skip model retraining, reducing latency from ~5 seconds to ~200ms on cache hits; bounded at 500 entries to prevent unbounded growth
Cross-asset SPY correlation (Finding 13): log-return correlation between each symbol and SPY is computed over the last 60 bars and surfaced in diagnostics as spy_correlation; provides beta context for the UI and signals whether a stock's forecast is driven by broad market moves or idiosyncratic factors
Extended forecast horizon to 252 days (Finding 14): MultiStockForecastRequest validator raised from 90 → 252 days (1 trading year), enabling strategic planning horizons; previously quarter-length was the maximum regardless of use case
P(direction) and Sharpe signal (Finding 15): each prediction step now includes p_up (probability that the forecast price exceeds the last known price, computed via normal CDF approximation without scipy dependency); diagnostics now include sharpe_signal (horizon expected return divided by total expected vol), regime, garch_daily_vol, spy_correlation, and daily_drift_per_bar

Phase 4.30 — Forecast Core Rewrite, Validation Engine & Deploy-Pipeline Recovery

Sep 2026
Shipped20/20
100% complete
Return-space distributional rewrite (ml_models/forecast_core.py): the ensemble now models a distribution over cumulative log returns anchored at spot rather than averaging price levels. Weights were previously fit to a single 90-step trajectory from one origin — effectively one independent observation for 13 parameters — and scored on price levels where tracking yesterday's close yields R² ≈ 0.99, so nothing was ever benchmarked against a random walk and models measurably worse than doing nothing still received weight
Random-walk skill gating: every model is scored as the fraction of random-walk error removed, in return space. Non-positive skill earns zero weight. Weights are then shrunk toward the asset-class prior in proportion to the number of genuinely independent observations behind them (λ = n/(n+k₀)), so a single-origin estimate correctly lets the prior dominate instead of pretending to 90 observations
Volatility term structure with correct GARCH aggregation: cumulative variance over h steps is h·σ²_L + (σ²₀−σ²_L)·φ(1−φʰ)/(1−φ), blending EWMA, GARCH(1,1) and HAR-RV. The previous σ√t scaling is only valid if volatility does not mean-revert, so it over-stated long-horizon risk when vol was elevated and under-stated it when depressed — exactly backwards
Cornish-Fisher bands with horizon-aggregated moments: skewness decays as 1/√h and excess kurtosis as 1/h across the horizon. Applying single-bar moments at every step treated a 30-day cumulative return as though it were as fat-tailed as one day, which systematically understated the band. Empirical coverage now holds 86–89% against nominal 90% on Gaussian, fat-tailed and skewed returns
Drift shrinkage by c*, the uncentred regression of realised on predicted returns across many origins — the MSE-optimal amount of an extrapolation to keep. Correlation alone was the wrong statistic: on a steadily trending asset the predictable part is the level, not the timing, so correlation reads zero while the forecast is plainly valuable. Unpredictable series shrink to the 0.25 floor; genuinely trending ones keep 0.6–0.8
v2_ensemble look-ahead leak fixed: it was fed rolling_forecast() in-sample fitted values as both its forecast and its validation series. Those fits track realised prices almost exactly, so v2 won the weight optimiser by construction and then contributed a lagged copy of recent history to the forecast path
Four deep models restored to the ensemble: PatchTST, iTransformer, N-HiTS and Mamba-TS raised "not fitted" on every daily request because the 190-bar fetch left their validation instances without a single training window. They had been silently absent from every daily forecast
Outlier handling moved to return space: overwriting an outlier price with the series median destroyed the level and manufactured two enormous fake returns around every real earnings gap — a 30% gap became a −70% day followed by a +230% day. Clipping the return instead caps the shock's leverage while leaving the path continuous, anchored so the final price is preserved exactly
Bands de-double-dipped: the conformal quantile was drawn from the same window the weights were optimised on, making it an in-sample residual and the bands systematically too narrow. Scale now comes from the volatility model, fitted independently, with a robust median-based conformal correction so the 99% band is 2.0× the 80% band rather than 1.3×
Hurst estimator removed: it used a non-standard formula and inverted on test series, labelling a random walk mean-reverting and an AR(0.88) reverting series trending, so the regime weights it selected were close to arbitrary. Persistence now arrives from the out-of-sample forecastability panel measured over years of data
Forecast requests 4× faster (33s → 8s cold, ~3s warm): GradientBoostedStacker._best_split rebuilt and re-summed both residual lists per candidate split (O(n²) per feature per tree — 25.5M generator calls per forecast) and is now a single O(n) prefix-sum sweep; LightGBM-FE rebuilt the whole feature matrix on every rollout step to use only its last row. Both verified identical to the originals
Horizon contract honoured: n_steps was floored at 14, so a 1-day request silently returned a 14-day path with no way for the caller to tell. Requests now return exactly the horizon asked for
Forecast Validation Engine (forecast_runs / forecast_scores + hourly scoring sweep): every forecast is captured at generation and scored once its horizon closes. Generic schema keyed by source/model/symbol so every MALI forecasting module can share it; idempotent scoring via a UNIQUE constraint; capture is fire-and-forget so a live forecast can never fail because the accuracy layer is unavailable
Skill-anchored predictability score: a MAPE-weighted score grades a random walk at ~88/100 and scores SPY and BTC identically despite wholly different predictability, and rises further if you simply widen your bands. The replacement is anchored so 50 is exactly a random walk, ranks on skill, measures calibration as |coverage − nominal|, and uses Winkler's interval score, which cannot be gamed by widening. Measured discrimination: 47-point spread versus 11
Statistical honesty throughout: overlapping forecasts are discounted to an effective sample size (fifty 24h forecasts an hour apart carry ~3 observations, not 50), hit rate is reported against the asset's own base rate with a Wilson interval, and skill carries a standard error and significance flag
Forecast Accuracy dashboard (MALI Lab › Forecast Accuracy): headline skill and calibration tiles, a per-step decay curve showing how far out the edge survives, a band-calibration chart against the nominal target, and a model leaderboard ranked on skill rather than error size. Positive/negative uses a cyan/amber pair (ΔE 25.8 under deuteranopia) instead of green/red (ΔE 4.9), with sign and glyph labels so colour never carries the meaning alone
Deploy pipeline recovered: supervisord (uvicorn) and systemd (gunicorn) were both configured for port 8000. supervisord held it while mali.service crash-looped 13,395 times unable to bind, so the hook's "systemctl restart mali" was a no-op and no Python backend deploy had gone live for ~38 hours. supervisord is now the owner, mali.service is disabled, and the hook calls supervisorctl with an explicit config path — a stray supervisord.conf in the repo root otherwise shadows the real one
Auth error disclosure closed: the login endpoint returned raw driver text to unauthenticated callers, answering with "Your account or project has exceeded the quota" — naming the backing database service and reporting its billing state. A second handler explicitly returned str(e) for debugging. Both now log in full and return a generic message with a trace_id, and the status is corrected from 500 to 503
Database connection pool right-sized: 20 + 10 overflow per process across 2 workers reserved up to 60 connections against a serverless Postgres that bills compute time and suspends when idle, keeping the endpoint awake around the clock. Now 5 + 5 and env-tunable via DB_POOL_SIZE / DB_MAX_OVERFLOW / DB_POOL_RECYCLE
Boot logging cleaned: an unreachable database produced ~30 traceback lines per worker per boot and buried the one line naming the cause. Unreachability is classified by exception type and SQLSTATE class (08 connection, 53 insufficient resources, 57 operator intervention) rather than message text — the same refused connection reads differently on Linux and Windows — and reports a single warning, while genuine DDL faults keep their stack trace

Phase 4.31 — Process Ownership, Venv Drift & Deploy Verification

Sep 2026
Shipped9/9
100% complete
supervisord.conf renamed to supervisord.docker.conf: the file is the container's process-supervision config — Dockerfile copies it to /app/supervisord.conf and entrypoint.sh execs supervisord against it as PID 1 — so it could not simply be deleted without taking the Hugging Face Space down at container start. Sitting at the repo root under the bare name, it shadowed the host's config whenever supervisorctl ran from the checkout, because supervisorctl searches the working directory before /etc/supervisor. Renaming removes the shadowing and matches the existing supervisord.ionos.conf naming; only the COPY source changed, so both entrypoints are untouched
dragonfly and Next.js process ownership settled: both were declared in supervisord and also running as systemd units (dragonfly.service, mali-frontend.service). supervisord kept starting a second copy of each, which could not bind its port and gave up as FATAL — benign only because it stopped retrying. The duplicate blocks are now disabled and documented in place rather than deleted, so the split of ownership is visible in one file: supervisord owns fastapi and redis-backup, systemd owns dragonfly, mali-frontend, mali-ml and mali-voice, one owner per port
mali-ml.service crash loop ended after 857,108 restarts: the ML service had been dying on ModuleNotFoundError for 'redis' — a package its own requirements.txt pins on line 26 — with nothing listening on :8001 and the loop burning CPU continuously. The venv had drifted from its requirements and nothing on the deploy path installed or checked it
timesfm and huggingface_hub pinned to a compatible pair: with the ML venv finally matching requirements, the service swapped one crash loop for another. timesfm was unpinned, so pip took 2.0.2, which calls ModelHubMixin.__init_subclass__(paper_url=...) — an argument absent before huggingface_hub 0.27 — against a pinned 0.24.0. That combination installs without complaint and raises TypeError at import, and timesfm's own metadata does not catch it since it declares only huggingface_hub>=0.23.0. Tested upward from the pin: 0.25.2 and 0.26.2 still fail, 0.34.0 imports cleanly and stays under the <1.0 ceiling transformers 4.40.2 requires
Deploy hook installs and restarts the ML service: it previously touched only the backend venv, so anything added to mali-ml-service/requirements.txt was pinned but never present, and it never restarted mali-ml either — so even a correct deploy left the service running whatever code happened to be on disk when it last crashed
Silent pip failures fixed: the install line was 'pip install -q -r requirements.txt 2>&1 | tail -5 || echo ...', whose exit status belongs to tail rather than pip, making the error branch unreachable and a broken install indistinguishable from a good one. That is the mechanism that let the venv drift hide. Installs now run through a helper that captures output and reports success or failure explicitly without aborting the rest of the deploy
Deploys verify themselves rather than assuming: pip check runs after each install to walk the declared dependency graph and catch incompatible pins that install fine and fail at import, and a health poll runs after each restart so a deploy can no longer report success while a service sits in a restart loop — which is how the ML service stayed down through 857,108 restarts without a single deploy noticing
Dependency checks baselined so only new conflicts raise an alarm: pip check found six long-standing conflicts in the backend venv on its first run, all harmless — tts and gruut pin older numpy, pandas, networkx and dateparser that the main venv never needs because voice runs from its own isolated venv, and chronos-forecasting declares accelerate that sota_adapters.py deliberately avoids by skipping device_map so the model loads on CPU. Reported raw that is six exclamation marks on every deploy until nobody reads the block at all, so it now diffs against a committed baseline and raises only what is new. The ml-service baseline is deliberately empty
Deploy hook moved into the repo at scripts/post-receive.sh: it lives at .git/hooks/post-receive on the host and was previously unversioned, so the deploy path itself was unreviewable — the same blind spot that hid the systemd crash loop. Its header now documents which manager owns which service, since that split is not discoverable from the code and has caused two outages

Phase 4.32 — Forecast Accuracy Measured: Roster Cut to What Earns Its Cost

Sep 2026
Shipped10/11
91% complete
Walk-forward backtest harness (scripts/forecast_backtest.py): runs the production pipeline unchanged at many historical origins across asset classes and scores it with the same scorer the live validation engine uses. Real market data only, cached once per symbol so runs are reproducible. Two details decide whether the numbers mean anything — the per-request model cache keys on today's date, so it is cleared between origins or every origin would score one fit against many futures; and origins are spaced a full horizon apart so each is an independent observation rather than a shifted view of the same window
The 13-model ensemble was measured, and it had no edge: 14 independent origins on each of 7 assets, 5 years of real data, 21-bar horizon, skill +0.0023 against a random walk and a composite score of 48.4 — marginally below the 50 that means 'no better than assuming the price does not move'. Hit rate was exactly 0.500. Not one of the seven assets showed statistically significant skill
Default blend cut from 13 models to 3 (theta, HAR-RV, GARCH-MIDAS): measured skill rose to +0.0169 and the score to 51.8 while runtime fell from ~51s to ~8.6s per asset. Removing ten models made the forecast better and six times faster
The deep stack was the problem, and the tail was the evidence: PatchTST, iTransformer, N-HiTS and Mamba-TS scored -0.0549 together — worse than doing nothing. They earn their keep on trending names (SPY +0.12, AAPL +0.06) and then lose catastrophically on range-bound ones (TLT -0.365, XLE -0.224), because extrapolation is precisely the wrong prior for a mean-reverting asset. Blended in, those blow-ups cancel the quant models' gains; the cheap roster's worst asset is -0.038
Claims kept to what the data supports: no configuration's skill is significant at 14 effective observations, so the stated conclusion is the weaker one — the deep models do not earn their cost and carry a large downside tail. The 6x latency difference needs no statistics. One asset does clear significance under the new default: SPY at +0.093 skill with a 0.857 hit rate
Every catalog model remains selectable by name, which is what the Model Selection tab does; the roster governs only what runs when the caller expresses no preference. FORECAST_DEFAULT_MODELS overrides it without a deploy, and the catalog endpoint now reports the active roster so the UI stops implying all thirteen are blended on every request
Per-horizon skill estimator corrected: the Forecast Accuracy decay curve averaged per-observation skill ratios, but skill is a ratio of sums. On a quiet bar the realised return is near zero, so that observation's denominator collapses and it contributes a huge negative number regardless of forecast quality. On a construct with a genuine 0.4 skill at every horizon the pooled estimator reports +0.39 to +0.44 and the old one reports -1.65 to -1.91 — a properly skilled model would have been displayed as catastrophically bad
A proposed upgrade was measured and rejected: filtered historical simulation was built to replace the closed-form confidence bands, propagating the GARCH recursion and bootstrapping each asset's own standardised residuals rather than assuming returns aggregate independently. Measured on real history it was equal at a 5-bar horizon and slightly worse at 21 and 63 bars, so it was reverted rather than shipped on the strength of the theory
Optional request fields fixed: models, custom_weights and interval were declared as bare annotations with a None default, which under pydantic v2 accepts omission but rejects an explicit null — a client sending "models": null got a 422 while one omitting the key succeeded, the same intent producing two different outcomes
Per-asset band calibration — closed in Phase 4.33: coverage was good in aggregate (0.940 against a 0.900 nominal) but varied sharply by asset and horizon, with GLD covering 0.615 at 63 bars while SPY and TLT over-covered at 1.000
Directional edge needs a different information set, not another architecture: no time-series model beats a random walk on 21-day direction of liquid assets because the signal is not in the price history. Moving skill materially above zero requires cross-sectional momentum, carry, value, flows or options-implied data — a different class of input, and the one genuinely open avenue

Phase 4.33 — Per-Asset Band Calibration & Honest Architecture Selection

Sep 2026
Shipped9/9
100% complete
Root cause found, and it was not the volatility model: comparing the model's predicted half-width against each asset's realised h-step moves returned multipliers near 1.0 for gold (0.945 at 63 bars), so its variance estimate had been accurate all along. The 0.615 coverage was a centring error — drift is deliberately shrunk by its measured out-of-sample predictability, often to the 0.25 floor, and nothing carried the discarded slope as uncertainty
Drift uncertainty now widens the band, in quadrature with diffusion and growing linearly in h. That term matters because the discarded slope is a systematic centring error of (1 - shrink) x drift x h while diffusion grows only as sqrt(h), so it dominates at long horizons and the band simply misses on a trending asset. Carrying it is not a fudge: shrinking the drift in the first place is precisely the statement that we do not know whether it is real
Per-horizon empirical scale replaces a single conformal scalar: the band is compared against the spread of that asset's own realised h-step returns, a non-parametric reference needing no distributional assumption. One scalar could never work because the correction required had opposite signs per asset and per horizon. Overlapping windows are unavoidable at long horizons, so the correction is shrunk toward 1.0 by how many independent windows back it — at 63 bars, five years supplies about twenty
Scale calibration comes from exactly one estimator, never two multiplied together: the empirical per-horizon scale where history allows, otherwise the conformal scalar. Both estimate the same quantity, so stacking them double-corrects — it would have widened SPY, already over-covering at 0.966, by a further 23% at 21 bars
Measured on the same walk-forward harness across 7 assets and 5 years of real data: aggregate calibration error at a 21-bar horizon fell from 0.040 to 0.022, at 63 bars aggregate coverage is 0.897 against a 0.900 target (error 0.003), and GLD at 63 bars rose from 0.615 to 0.812. The point forecast is untouched — this changes only the uncertainty around it
Both candidate designs were measured before one was shipped: stacking the two estimators and using only one came out statistically indistinguishable (0.001 vs 0.003 aggregate error at 63 bars, well inside sampling noise at 12 origins per asset), but the single-estimator version was better on 6 of 7 individual assets. With the data unable to separate them, the principled design was chosen rather than the marginally better aggregate
Architecture Selection tab now reflects what the engine does: a callout names the three models that run when nothing is selected and gives the measurement behind the choice, a Measured Default button restores the engine's own roster once the grid has been touched, and the three roster models carry a DEFAULT badge. The panel previously presented thirteen architectures as equally-endorsed options, inviting the conclusion that selecting more produces a better forecast
The roster is read from the catalog endpoint rather than hardcoded in the UI, so the panel cannot drift from the engine if the roster is retuned or overridden via FORECAST_DEFAULT_MODELS. The header's claim of an '18-model cleaned roster' — a figure that was never accurate — is replaced with the real one, and the auto-mode description no longer says the backend runs its full suite weighted by inverse validation RMSE, which has not been true since the return-space rewrite
Directional edge via a different information set — tested in Phase 4.34: cross-sectional momentum turned out to carry a genuine ranking edge that does not transfer to single-asset forecasting, so it ships as a ranking surface rather than a forecast tilt. Carry, flows and options-implied data remain untested

Phase 4.34 — Cross-Sectional Signals: Where the Factor Edge Actually Pays

Oct 2026
Shipped8/9
89% complete
Cross-sectional signal library (ml_models/cross_sectional.py): momentum 12-1, one-month reversal, volatility-scaled momentum, low volatility and a beta-neutral momentum variant. These are the first signals in the platform that rank an asset against its peers rather than extrapolating its own history — a different information set, which is what the measurement said was needed
Information coefficients measured on a real 50-symbol universe (sectors, large caps across industries, rates, metals) over 6 years. momentum_12_1 is real: IC +0.068 at 21 days with t=2.12, +0.041 at 5 days with t=2.22, sign and magnitude both matching the literature. Reversal, volatility-scaled momentum and beta-neutral momentum did not clear significance and are not used
The strongest signal was deliberately rejected: low_vol measured t=-3.03, -2.54 and -2.24 across three horizons, the most consistent result of any signal. It is negated volatility, so a negative IC means high-volatility assets outperformed — the inverse of the low-volatility anomaly, among the most replicated effects in finance. Over a 2020-2026 sample dominated by an AI-led bull market that is a regime artefact, and trading it amounts to a leveraged beta bet that looks excellent until the first real correction
Ranking skill was measured not to become forecast skill, which decided the whole design. Translated through Grinold's law (alpha = IC x sigma_xs x z) with the IC estimated only from dates before each forecast, momentum scored -0.002 against a random walk at 21 bars with a 0.493 directional hit rate. The gap is structural rather than a defect: a cross-sectional score is centred within each date, so it holds no view on market direction and its sign agreement with absolute returns is about 50% by construction
So the signal ships as a ranking surface, not a forecast tilt. In a long/short structure the market component cancels and an IC of 0.068 across 50 names is a genuinely tradeable edge — Grinold's law puts the information ratio near 0.48. In a single-asset point forecast the market component dominates the error, and that is precisely what the signal is silent about. Wiring a momentum tilt into the forecast would have added a cross-asset data dependency to every request in exchange for measurably nothing
Cross-Sectional Momentum panel (MALI Lab › Cross-Sectional): the universe ranked by 12-1 momentum with per-asset z-scores, realised volatility and percentile. The panel leads with the information coefficient and its t-statistic rather than burying them, and states in its own copy that it says nothing about market direction and why it is not in the forecast — a ranking shown without the evidence for it invites more confidence than it has earned
Universe cached on a 20-hour refresh rather than fetched per request: ranking 50 symbols means 50 price histories, which is background work. 12-1 momentum skips the most recent month by construction so it barely moves intraday. The cache persists to disk so a restart does not trigger a refetch storm, and the surface reports itself unavailable rather than inventing a ranking when data is missing
Proxy allowlist extended rather than opened: the forecast-accuracy route is a catch-all, so passing the path segment straight through would let any caller reach arbitrary FastAPI routes under the proxy's credentials. Only the two new leaves are permitted, and the two-segment refresh path is matched explicitly
Untested information sets remain: carry (needs term-structure data), value (needs fundamentals) and flows. Options-implied data was the most promising of the set and was tested in Phase 4.35 — a usable source exists in the CBOE indices, but it produced no directional edge that survives correction

Phase 4.35 — Options-Implied Data: Measured, and Where It Stops

Oct 2026
Shipped9/10
90% complete
Options-implied data source added (ml_models/options_implied.py): ^VIX, ^VIX3M, ^VIX9D, ^SKEW and ^VVIX, all confirmed with 5 years of real daily history. Individual-name option chains were ruled out deliberately — free sources serve only a current snapshot, so a signal built from them cannot be backtested, and an untestable signal is one this codebase has no way to believe. The CBOE indices are genuinely options-derived and do have the history
Six signals built on them: VIX term structure (VIX/VIX3M), variance risk premium, VIX and SKEW against their own trailing ranges, VVIX/VIX, and near-term stress (VIX9D/VIX). All measured against forward S&P returns on non-overlapping windows, since overlapping ones would inflate the sample and manufacture significance
No directional edge survives the search that found it. Two results looked significant alone — term structure at 5 bars (t=2.40) and VVIX/VIX at 10 bars (t=-2.22, skill +0.080, five times the best forecast model's). But six signals across four horizons is 24 tests, under the null about 1.2 clear |t|>=2 by luck, and exactly 2 did. The Bonferroni threshold is |t|>=3.1 and nothing reaches it at any horizon
The multiple-testing correction is computed by the measurement script itself rather than left to whoever reads the output, because searching many signals and reporting the best one is the single most common way quantitative research fools itself
The headline hit rates were the market's own drift. A 65.2% directional hit rate at 10 bars is impressive until the S&P's base rate over the same windows is measured at 62.4%: the edge is +2.8 points against a standard error of 5.2, and at 21 bars it is +0.8. The signals were reporting that stocks rose between 2021 and 2026 — the same base-rate trap caught earlier in the cross-sectional work, which is why it was checked
Shipped as market-regime context in the forecast diagnostics rather than as a forecast input, with the caveat travelling in the payload beside the numbers so the data cannot be read as a directional claim it does not support. Live output shows VIX, term structure, variance risk premium and a plain-language regime description
Implied vol tested in the confidence band and rejected (scripts/measure_implied_vol_band.py). This was the best-founded remaining idea: VIX is a volatility forecast and the band is built from backward-looking estimators only. Three variants against baseline at nominal 90% — no variant wins consistently, the ranking flips across horizons, and every gap sits inside sampling error
The sharper conditional test also failed: backward-looking estimators are weakest when volatility is changing, so any information implied vol adds should appear in that subset and be absent elsewhere. The effect flips sign between buckets, and where coverage improves at 21 bars it is bought by widening the band from 33.1% to 40.6% — a 23% wider band is not a free improvement. Closest call was 5 bars: coverage 0.918 to 0.897 with narrower bands, genuinely better on both counts, but 1.5 standard errors and reversing at 21
Design note kept for any retry: implied vol is a biased estimator of realised vol, carrying the variance risk premium measured at +5.8 points on this sample, so its level cannot be used raw without over-widening the band. Use it relative to its own trailing mean, or regress realised on implied to remove the premium first. Both approaches are implemented in the script
Re-run as history accumulates. The obstacle on the band question is sample size rather than the idea: five years gives about 70 independent observations at a 21-bar horizon in the volatility-transition bucket, which cannot resolve an effect of the size being looked for. The forecast validation engine is now building exactly that history

Phase 4.36 — Model Selection Ranked by Measured Skill

Oct 2026
Shipped17/18
94% complete
The AI Best 5 button ranked models by ASSET_DNA_WEIGHTS: hand-assigned capability vectors rating each architecture on accuracy, speed, regime awareness, explainability, multi-horizon coverage and cross-asset modelling, weighted per asset type. Every one of those numbers was written by hand and none had ever been tested. They rated the deep architectures highest, and those measure -0.055 skill against a random walk, as bad as -0.365 on TLT. Chronos-Bolt even carried an explicit +0.05 anchor bonus to guarantee it a top-five slot. Unlike the auto path the button marks the selection as touched, so its picks really were sent to the engine — the ranking was not merely unverified, it pointed the wrong way
Replaced with measurement (scripts/measure_model_skill.py): each selectable model run alone through the production pipeline at 10 independent origins, spaced 81-138 bars against a 21-bar horizon so no window overlaps another, on 7 real assets across equities, ETFs and crypto. The always-on v2 engine was also measured by itself, because without that reference line there is no way to separate a model's own contribution from v2 carrying it
The full table, best to worst: timesfm +0.0086, har_rv +0.0068, neural_sde +0.0049, garch_midas +0.0023, garch +0.0020, theta +0.0016, then below a random walk chronos_bolt -0.0019, moirai -0.0276, lightgbm -0.0291, nhits -0.0415, xgboost -0.1136, itransformer -0.3078, patchtst_xl -0.4329 and mamba_ts -1.4437, against v2 alone at -0.0193. The deep figures are the ones measured after Phase 4.37 made those models reproducible; the originals were single draws from an unseeded initialisation and three of them were far kinder. The split is not deep against shallow, which is what the first eleven rows suggested before the foundation models were measured. It is pretrained against trained-from-scratch: TimesFM and Chronos are both zero-shot and both beat the v2 engine, while every architecture trained per-asset here fails to. Moirai is the exception that keeps this from being a rule, and all of it sits inside sampling error anyway
Two hurdles to be recommended, both necessary: beat a random walk, or the model has no business in a forecast at all; and beat v2 running alone, because v2 runs on every request regardless of selection, so a model that merely rides along adds latency and no skill. Six models clear both, so the button takes the top five and the sixth, Theta, sits just below the cut
Three models were never running. chronos_bolt, moirai and timesfm returned results identical to v2 alone to five decimal places across all seven assets, which is only possible if they never executed — an uninstalled model still gets a forecast back, because v2 runs regardless. The harness would have published v2's performance under their names, the same false attribution the whole table exists to remove. The script now asks the catalog what can run before measuring, and a second guard drops any model that ties the reference exactly, since a real model never ties v2 to the last decimal across seven assets
Per-asset-class ranking was built, measured and then removed. On equities the positive models are patchtst_xl +0.022, itransformer +0.018 and mamba_ts +0.013 — the three worst performers overall. The class holds two symbols, both trending large caps, which rewards exactly the trend extrapolation that collapses elsewhere. Ranking per class would have reintroduced the deep-model bias by a different route, so the ranking is one flat list over the whole basket and per-class numbers are served as tooltip context only
The set is resolvable; the order inside it is not. Over the complete fourteen-model table the best-to-worst gap is 0.193 against a typical standard error of 0.040, so the split between the recommended group and the rest is roughly five standard errors. But timesfm’s own mean of +0.0086 sits inside a standard error of 0.0184, and the gaps within the group are a fraction of one. The payload carries order_resolvable false, the strip no longer numbers its entries, and the button reads 5 that beat a random walk rather than Top 5, because a 1-to-5 ordering would claim precision this sample cannot support
On crypto no model beat a random walk at all. The UI says so on crypto symbols and points at the bands and P(up) instead of the point forecast, rather than letting a basket-wide recommendation imply an edge that is not there
The availability probes were lying, which is what let a dead model stay selectable. chronos_available() answered with importlib find_spec, which proves only that a directory of that name is importable by path, not that importing it succeeds — chronos pulls in transformers, which refuses to load against an incompatible huggingface-hub. The catalog advertised the model, the UI offered it, the gate opened, the adapter raised, the router logged a warning and swallowed it, and the caller received v2's forecast believing a 48M-parameter foundation model had contributed
timesfm_available() and moirai_available() had the identical defect, so all three now perform the real import and all three honour their broken flag, meaning a weight-load failure flips availability to false instead of leaving the catalog advertising a model known not to work. The real import costs about four seconds once per process, so startup resolves the probes on a worker thread and logs which optional models are runnable, keeping that cost off the first catalog request. Production was never affected — the server runs the pinned transformers 5.4.0, huggingface_hub 1.8.0 and torch 2.11.0 and imports chronos fine; the broken import was a local environment drifted from requirements.txt. What was wrong everywhere was the probe
Added diagnostics.roster_did_not_run, naming models that were asked for and produced nothing. Each gate catches its own failure and logs it server-side, which kept the response looking successful while quietly returning v2 alone. Naming them in the payload is the difference between a caller knowing their selection did not run and a caller believing it did
timesfm measured too, and it is the best model in the table: +0.0086 overall against a random walk, ahead of HAR-RV at +0.0068, clearing the v2 reference by +0.028. It had been installed all along in the ML service venv on a stack incompatible with the main app, while version 3.0.2 resolves against the pinned stack cleanly. It did not run on the first attempt — timesfm 3.x replaced the entire 2.x entry point, so the adapter raised at predict time while the package imported perfectly, and the measurement came back identical to v2 alone and was dropped automatically. An availability probe cannot catch an API change, because the import genuinely succeeds; the identical-to-reference guard is what caught it. The adapter now supports both APIs
chronos_bolt measured on the production server, the only host where it imports. At -0.0019 it does not beat a random walk and is not recommended, but it clears the v2 reference by +0.017 and is the strongest of the models that fall below the line — second only to TimesFM among the four pretrained models. Comparability was checked rather than assumed: the exact local history file was uploaded and verified by sha256 on both sides, and the v2-only reference was re-measured on the server alongside it, reproducing bit-for-bit
Dependency drift closed as a side effect: timesfm and chronos were both installed on the server and declared in no requirements file, left as commented-out optional lines, so any venv rebuild would have dropped them silently. That stopped being cosmetic the moment timesfm measured best, so both are now declared dependencies
moirai measured too, in a throwaway venv rather than by changing production: uni2ts resolves to a torch downgrade from 2.11.0 to 2.4.1, a full CUDA stack on a CPU-only host, and downgrades of scipy, fsspec and python-dotenv. It scored -0.0276, worse than a random walk and worse than the v2 engine alone at -0.0193, so it would not be recommended even if it were free to install, and uni2ts stays out of requirements — now with the measured number and a reproduction recipe recorded beside it instead of an assumption. Every one of the 14 selectable models now has a measured row
Measuring in a different environment is only meaningful if the rest of the engine behaves identically there, so the v2 reference was re-measured every time. It reproduced bit-for-bit in four separate environments, including on torch 2.4.1 with downgraded scipy and fsspec, which is what lets rows measured on different hosts sit on one scale rather than beside each other with caveats
The Moirai adapter had four bugs, none findable without running it and all masked by the gate swallowing the exception: an import of a symbol uni2ts 2.x removed, a target tensor built as [batch, target, time] where 2.x types it [batch, time, target], a required past_is_pad never passed, and hand-rolled patching that produced a 64-patch context against a 67-patch prediction mask. Each of the last three measurements — Chronos, TimesFM, Moirai — was caught not running by the identical-to-reference guard rather than by a probe, because in every case the package imported cleanly and the adapter failed at call time. An availability check cannot detect an API change; the guard is the only reason three models’ worth of v2 results were not published under other names
Re-measure as the forecast validation engine accumulates real forward data. The current table is a backtest at 10 origins per asset; every recommended model's absolute skill is still inside its own standard error of zero, so the honest claim today is that these five are the best available, not that any of them is reliably better than assuming tomorrow equals today

Phase 4.37 — Reproducibility, and the Configuration Sweep It Made Possible

Oct 2026
Shipped11/12
92% complete
Measuring the same model twice is what exposed it. Neural SDE scored +0.0058 in the skill table and -0.0030 on a re-run at identical origins on identical history — a swing wider than the gap between first and sixth place among the models that beat a random walk. It drew from two unseeded global generators, torch’s for its network weights and numpy’s for the Euler-Maruyama rollout, so the same symbol returned a different forecast on every request. Refreshing the page changed the number
The same defect was in four more models, and worse. PatchTST, iTransformer, N-HiTS and Mamba-TS each build a network in __init__ and train it per request from torch’s unseeded generator. Two consecutive runs on real SPY data differed by up to $47 on a ~$763 price — about 6%, larger than the 21-day moves the forecast is trying to predict. Every one of their skill rows was a single draw rather than a measurement
Seeding the weights was not enough, and stopping there would have shipped a half-fix that looked complete. Three of the four went deterministic; N-HiTS did not, because it carries dropout and dropout masks are drawn from the global generator on every training forward pass. ml_models/determinism.py now holds the seeded state across the whole training run rather than just construction, which covers both and anything similar added later. Prediction was never the problem — it runs in eval mode where dropout is identity
Seeds are derived from the price window rather than fixed, so the same series always trains the same network while different assets stay independent, and they are applied through torch fork_rng so that seeding these models cannot change what any other torch model in the same process draws. A global seed would have made every other model’s output depend on which one happened to run first, trading one reproducibility bug for a subtler one. Verified three consecutive runs identical to eight decimals across all five affected models, and that each still responds to different data rather than being frozen
Re-measured once reproducible, and the deep models got far worse: itransformer -0.0288 to -0.3078, patchtst_xl -0.1432 to -0.4329, mamba_ts -0.1845 to -1.4437. Neither column is a better estimate of some true skill — these models are initialisation-dominated. What changed is that the row now matches what production reproducibly returns, which the old one never did. The practical consequence is nil, since all four were already excluded, but the table no longer reports numbers nobody could reproduce
With the engine deterministic, configurations could finally be compared rather than models. Every pair with the strongest model was swept, ordered by how correlated the partner is with it: timesfm+har_rv +0.0091 worst -0.0548 at r 0.905, timesfm+garch +0.0083 worst -0.0246 at r 0.848, timesfm+garch_midas +0.0082 worst -0.0249 at r 0.849, timesfm+neural_sde +0.0067 worst -0.0344 at r 0.875, timesfm+theta +0.0043 worst -0.0454 at r 0.918, against timesfm alone at +0.0086 worst -0.0463 and all thirteen together at -0.0485
Two things separate in that table and only one is real. Mean skill spans 0.0042 against a standard error near 0.02, so ranking on it fits noise and the leading pair is not evidence of anything. Worst case spans 0.039 and tracks correlation almost monotonically, and dispersion across assets does the same: the partners TimesFM disagrees with most halve both. That is the mechanism ensembling actually provides — averaging imperfectly correlated errors reduces variance, not bias — so it is the column worth optimising and the one that should generalise beyond this sample
Default roster is now TimesFM + GARCH, down from three models. It gives up 0.0008 of mean, deep inside noise, to cut the worst asset from -0.0548 to -0.0246 and nearly halve the spread across assets, and it beats the three-model roster it replaces on mean, worst case and dispersion at once. GARCH over GARCH-MIDAS is a coin toss settled on cost: they correlate at 1.000 and differ in the fourth decimal, and GARCH is simpler and about half the inference time
Not the single best model, which did measure highest. The gap is inside sampling error at t=0.05, winning on 4 of 7 assets, its worst asset is nearly twice as bad, and a single-model default has no fallback — which is not hypothetical, since TimesFM silently stopped running when its vendor replaced the API and Chronos was dead on arrival from a dependency conflict. Verified the degradation is graceful and visible: with TimesFM absent the forecast still runs on GARCH and names the gap in diagnostics
The Model Selection tab needed more than new copy. It defaulted to selecting all 13 models, which is the worst configuration measured at -0.0485, worse than selecting nothing, and it never called the backend at all, so it had no idea what the engine blends, what the deployment can run, or how anything measured. Its auto mode told users it runs all 13 and that this matches the Forecast Engine, which was untrue in both halves
It now reads the catalog, defaults to the measured roster, shows each model’s measured skill in green or red, disables models that cannot run on this deployment, and states what auto mode actually does. Select All survives as a labelled experiment rather than the neutral default. If the backend is unreachable it falls back to the same pair the Forecast tab uses while loading, so the two tabs cannot disagree. Architecture Selection, still quoting the superseded +0.017 at 8.6 seconds, now carries the configuration numbers and says why the roster is not the highest-scoring pair
The whole frontier sits inside its own standard error of zero. The defensible claim is that this is the best available configuration, not that it reliably beats assuming tomorrow equals today. The forecast validation engine is accumulating real forward observations, which is the only thing that will settle it

Phase 4.38 — Analyst Agents: Audited, and One Deleted

Oct 2026
Shipped6/7
86% complete
An audit of the five customer-facing analyst agents — Market Sentiment, Institutional Flow, ESG, Global Macro and Real-Time Risk — found them declared in one file and present nowhere else in any working sense. All eleven of that file’s tools returned hardcoded values: Twitter sentiment was a fixed 0.65 on a fixed volume of 1,250, and ESG a fixed environmental score of 78 for every symbol ever asked. The file was also unreachable — the class was instantiated at module scope and nothing in the repository imported it
Deleted rather than repaired. Nothing imported it, so removing it changed no behaviour, and leaving it meant the next person to search the codebase for an ESG analyst would find a match and conclude the feature existed. Mock values of this kind are also against a standing rule for this platform, which carries no fabricated data and shows missing data as unavailable
Four of the five capabilities turned out to be already built on real data and simply absent from the agent registry, so nothing listing MALI’s agents mentioned them. Market Sentiment runs on real headlines with TextBlob polarity, Institutional Flow on options flow, short interest and institutional holders, Global Macro on regime detection and policy narration, and Real-Time Risk on VaR and stress testing with a regime alert monitor already started at boot. Each was checked for mock or random values before being registered; all four are clean
ESG was the real gap, and it stays a gap honestly rather than being filled dishonestly. Every source available to this deployment was tested: Yahoo’s esgScores module returns 404 since it was removed upstream, Financial Modeling Prep returns 403 because ESG is not on the configured plan, Finnhub 403 as a premium endpoint, Alpha Vantage answers that the function does not exist, and EODHD 403. There is no accessible ESG data at any price currently paid
So the ESG endpoint ships the real provider chain and no scores. /api/esg/score returns available false with each provider named and the reason it declined, and /api/esg/sources probes them live rather than reading configuration — a key being present says nothing about whether the plan behind it includes ESG, and all three refuse with a key-shaped 403 rather than an error that mentions entitlements. If any plan is upgraded the endpoint returns that provider’s data with no code change, and the agent is registered disabled for the same reason
An ESG score is precisely the kind of number a reader cannot sanity-check by eye — it looks equally plausible whether it was measured or invented — which is what made the mocked version worth deleting rather than leaving in place as a placeholder. A working pipe with nothing flowing through it is more useful than a number nobody can verify
ESG is the only one of the five that cannot be delivered by engineering. FMP’s ESG add-on or Finnhub’s premium tier would each light it up immediately, and the endpoint is already written against both. This one is a procurement decision rather than a build

Phase 4.39 — Catalog Drift, One Verdict Vocabulary, and Four Dead Endpoints

Oct 2026
Shipped12/14
86% complete
The model catalog the UI renders from had drifted from the engine it describes: 13 entries against 15 selectable models. The two missing were not obscure — garch was absent entirely and garch is half the current default roster, so the model the engine actually runs had no card, could not be toggled, and the default-blend line printed its raw key. moirai was missing too, despite having a measured row
TimesFM and Chronos moved to a new foundation family. They are pretrained rather than trained per asset, and that is the distinction the measurements showed predicts performance — both beat the engine core while every from-scratch deep model loses to it. Adding the family exposed four hand-maintained lists of families; the compiler caught two, and one of the two it could not catch drives grid rendering, so left alone it would have removed TimesFM from the Architecture Selection grid entirely while typechecking cleanly
Catalog entries added after this carry no DNA vector rather than inventing one. Those vectors were hand-assigned, were measurably wrong about which models work, and were retired as a ranking input in Phase 4.36; the radar now skips models without one instead of plotting a fabricated row
A buy/sell/hold audit then found the platform expressing one concept five ways. Neutral was HOLD in the Forecast tab, WATCH in the Scanner and sector heatmap, NEUTRAL in Technical Analysis and the AI debate; bearish was SELL almost everywhere and AVOID in the debate. A reader comparing a scanner WATCH against a forecast HOLD had no way to know they meant the same thing
The Market Scanner decided its verdict three separate times — inline in the list route, through a shared helper in the detail route, and a third rule in the Python router. Across 765 indicator combinations those three disagreed on roughly half, and a third of cases were flatly opposite: one surface saying BUY where another said SELL for identical inputs. One worked example: RSI 65 with positive MACD and a 1% day returned SELL from the list, HOLD from the detail card and BUY from Python
A real bug sat underneath that. The scanner panel sent the selected strategy to the detail route, the route read it and used it only as a display label, and the verdict came from a helper that takes no strategy at all. Opening a row from a Reversal scan produced a verdict computed by a trend rule and labelled Reversal. Strategies disagreeing with each other is the point of choosing one; two surfaces disagreeing about the same strategy is not
There is now one rule, in src/lib/signals.ts, mirrored in ml_models/scan_signal.py for the FastAPI side. Two copies are only safe if something proves they agree, so scripts/check_signal_parity.py runs both over 8,150 combinations across all five strategies and exits non-zero on any difference. It imports the real TypeScript module through Node’s native type stripping rather than a copy with annotations removed — a stripped copy would let the two drift in exactly the syntax the stripper mangled
Porting the rule exposed a second live bug: the Python side read the day’s change from a technicals dict built before that value existed, so it silently received zero and dropped the move component that the momentum and earnings strategies weight most heavily
Three Universal Stack tiles reported as permanently loading. One fault: the models-summary proxy called a path mounted only under the /api prefix, so every request 404d, the summary stayed null, and the tiles print Loading whenever the summary is null rather than whenever a load is in flight. A dead endpoint looked like a slow one indefinitely, and average accuracy showed no data for the same reason
All 42 proxied upstream paths were then swept against the running server rather than fixing the one reported and waiting for the next. Four more returned 404: two had the same missing prefix and are fixed; two have no counterpart upstream at all. The agent-kill path is left alone deliberately — its only candidate is a toggle whose handler unconditionally raises 503, so pointing at it would turn an honest undelivered-signal message into a confusing one, and guessing at toggle semantics could start agents when the caller means to stop them
The signal feed returns a null confidence for every symbol while the interface typed it as a number, so multiplying by 100 produced zero: every bar rendered empty and the average read a measured-looking 0.0%. Confidence is nullable now, the average covers only symbols that reported one, and a null shows the day’s move rather than a fabricated zero
That panel also described a hardcoded basket of eight tickers as tracked assets. It now reads the user’s watchlist and falls back to the basket only when the watchlist is empty, saying which. Its heading claimed Cross-Model Signal Fusion for what is a single-source read on each asset’s move today, so it now says Live Momentum Signals and states as much
Four surfaces still speak the old vocabulary — the sector heatmap, portfolio technical-ai-signals, the Technical Analysis panel and the AI debate — and are being converted separately. Until that lands the heatmap can show WATCH beside the scanner’s HOLD for the same idea
Neutral bands still differ by an order of magnitude and in units across surfaces: 0.2% expected return in the Forecast tab, 0.5% daily change in the sector heatmap, 1.5 score points in portfolio signals, and 1.5% in the Python heatmap for the same concept as the 0.5% one. That work unifies vocabulary, not thresholds; picking one silently would change what the panels say without anyone deciding to

Phase 4.40 — Position Sizing: A Formula That Could Only Return Zero

Oct 2026
Shipped7/8
88% complete
Suggested Size read 0.0% on essentially every forecast, and the formula behind it could not have done otherwise. Kelly’s binary form needs a win/loss payoff ratio, which this engine does not produce, so both sides were fabricated from the confidence band: the target was the larger of the expected move and a flat 1%, and the stop was 0.15 times the band width
Those two constants make a positive answer impossible at any realistic horizon. Kelly turns positive only when the payoff ratio exceeds (1-p)/p, and with the engine’s own probability of an up close sitting near 0.50 that requires a ratio above 1. Given a 1% target against a stop of 0.15 × band, that needs a band narrower than 6.67%. Measured bands at a 21-day horizon run 10% on SPY to 41% on TSLA, so the fraction was structurally negative and clamped to zero every time
Substituting the engine’s real probability for the invented one does not rescue it. That was the obvious fix and it was measured before being adopted: with p_up in place of the conviction heuristic, the binary form still returns 0.0% for SPY, AAPL, TSLA and GLD, because the defect is in the fabricated payoff ratio rather than in the probability
Replaced with the continuous form, f = mu / sigma² — the fraction that maximises long-run growth for a position with expected return mu and standard deviation sigma over the same period. It invents nothing: the engine already publishes both per bar, and the UI had simply never declared the fields. Direction comes from the sign of the expected move, which already drives the verdict, so the magnitude is what gets staked. Quarter-Kelly and the 25% cap are unchanged, because full Kelly is aggressive and this engine’s measured edge is thin enough that the damping earns its place
Measured after the change: TSLA sizes at 14.6% and GLD at 25% against full-Kelly fractions of -58% and -147%, while SPY and AAPL correctly size at nothing because their verdict is HOLD — the expected move is under 0.2% and the engine is declining to call a direction
A zero now explains itself rather than looking broken. The tile distinguishes the two reasons it can be zero — the verdict is HOLD, or a direction was called but the expected move is negligible beside the spread around it — and the figure carries the inputs behind it: the full Kelly fraction, the expected move, the forecast spread over the horizon, and the probability the engine puts on closing above entry
The earlier roadmap entry claiming this was fixed has been corrected rather than left standing. It reported 7–15% sizing after retuning the stop from band/2 to band × 0.15, which moved the threshold without removing it; the number it quoted was true for the 2% expected move in its worked example and false once Phase 4.30’s drift shrinkage made real expected moves an order of magnitude smaller
The 25% cap binds on assets with a wide spread relative to their expected move, GLD among them, so the displayed size there is the cap rather than the model’s own answer. Worth revisiting as a risk-based cap rather than a flat percentage, which would let the number stay informative where it currently saturates

Phase 4.41 — Four Panels Showing Numbers Nothing Produced

Oct 2026
Shipped10/11
91% complete
The Alerts tab announced that attention was needed and could not say what for, because the numbers were not real. Three of its four tiles were hardcoded literals — Price Alerts 12, Risk Alerts 4 with the subtitle 2 require attention, News Alerts 7 — and only the live stream count was genuine. The feed directly beneath them was frequently empty, which is how a reader ends up unable to find out what was wrong
All three now derive from real sources. Price Alerts counts the user’s configured rules and names how many symbols they cover, distinguishing none configured from rules unavailable. Risk Alerts counts live critical events and carries their messages in the subtitle, because a count with no cause attached is worse than no tile. The News tile had no source feeding it at all and is now Macro Events, counted off the same stream the feed renders so the number and the list cannot disagree
The alert feed had no empty state: an empty array rendered a blank box, which reads as broken rather than quiet. It now says nothing has triggered, and whether that is because no rules exist or because the rules being watched have not fired
Model Comparison had three faults, each sufficient on its own to make the result meaningless. Three of the five models it offered — Random Forest, N-BEATS and Temporal Attention — had been removed from the platform in the Phase 4.9 audit and were still selectable, and the default comparison pitted XGBoost against one of them
Its handler ended the model if/elif chain with a bare else that ran RandomForest for any unrecognised name and returned it labelled with whatever was asked for. Requesting TimesFM against GARCH produced two cards captioned timesfm and garch with accuracy 0.9931 on both and byte-identical feature importances, because both were the same RandomForest and neither model existed in the handler. Unknown ids are now refused by name, and the picker reads the comparable set from the engine so a model that cannot run cannot be offered
Scoring fit each model on every row and then scored it on those same rows, reporting about 99% for anything. It is now an 80/20 split scored on windows the model never saw, with the quant adapters walking forward one step at a time rather than seeing the whole series
The metric itself was the trap. One minus mean absolute percentage error sits near 98% on price levels for anything that returns a number close to today’s close: measured on AAPL, XGBoost scores 97.96% accuracy at a skill of -1.34 against a random walk, so it is 134% worse than assuming no change while looking excellent. Each card now leads with skill, and the pass badge keys off that rather than off accuracy
A/B Testing was reviewed in the same pass and is correct — recorded here because a clean audit is worth as much as a repair. Its Diebold-Mariano test uses the right loss differential, a HAC long-run variance truncated at h-1 which is what handles overlapping multi-step windows, and the Harvey-Leybourne-Newbold small-sample correction. Training windows end at the forecast origin so there is no look-ahead, models refit at every origin, a winner is declared only on significance, and the directional base rate is tracked. The only quibble is that the p-value uses the normal rather than a t distribution, which is marginally anti-conservative and negligible at the thirty-origin minimum it enforces
The Universal Stack tab’s Adaptive Ensemble Weights were not adaptive, not the engine’s, and not the right models. The chart rendered a constants dict built for display and read by nothing in the forecast path — real weights are solved per request by the NNLS meta-learner against measured skill, so there is no fixed split to show. Checked against the live engine it was displaying twelve models that do not run, led by chronos_bolt at 0.12, while omitting garch entirely despite garch being half the current default roster, and the whole thing summed to 1.03
Replaced with the two things that are true and available: the default roster, and each member’s measured skill against a random walk. The panel now states that weights are solved per request rather than implying a fixed split. The catalog grid below it printed the same fictional weight for every model and now shows measured skill and marks roster members. The constants are kept because the Weighting and Tuning sliders use them as a starting position, which is legitimate, and annotated at the source so they are not presented as live weighting again
The pattern across this tab is worth naming: four separate surfaces presented invented or stale numbers in a way that looked measured, and in each case the giveaway was a figure that could not move — a fixed 12, a permanent Loading, a 99% accuracy, a weight for a model that does not run. Anything displaying a number that never changes across symbols or sessions is worth treating as suspect until its source is traced

Phase 4.42 — Alert Maths, Honest Failures, and a Probe That Reaches the Handlers

Oct 2026
Shipped10/12
83% complete
Volatility alerts fired on almost every quote. The comparison multiplied the quote’s percent change by 100 before testing it against the threshold, but that field is already a percentage — every producer computes it as change over previous close times 100. A 1.2% move was therefore tested as 120, so a threshold of 5% was cleared by any movement at all. Verified after the fix: 1.2% no longer trips a 5% threshold and 6.1% does
The two alert evaluators disagreed with each other. One used strict greater-than and less-than while the other used inclusive comparisons, so an alert resting exactly on its threshold fired down one path and not the other. Both are inclusive now
Price alerts re-triggered on every tick. One evaluator deactivated a rule after it fired and the other never did, so for as long as the condition held the same alert kept firing. It is one-shot now, matching the path that already behaved
The alerts panel also counted critical events by filtering on their display colour, which happened to correspond to severity but would have broken silently the first time the palette changed. Severity is carried on the event now
The RL agent’s 503 turned out to be the backend being accurate rather than failing: no intraday agents are deployed, so there is no profit and loss or trade log, and the handler refuses to invent one. The panel rendered that as a bare HTTP 503, so a correct answer read as an outage. It now explains that no agents are running, and keeps real error reporting for every other status
Swept the backend for other endpoints returning 503 unconditionally and found only the intraday-agents pair, both the same deliberate stub; every other 503 is conditional on a real data failure. Probing the 68 endpoints the dashboard panels call then surfaced a versioning snapshot button posting to a route that exists in neither the frontend nor the engine, reporting HTTP 404 as though the service were broken rather than the feature never built, and a version badge pointed at another path that had never existed
Explainability was showing one model’s numbers under another model’s name. The Model Type control was a free-text box that was never sent to the endpoint and was not in the fetch dependencies, so typing a different model re-rendered nothing and left XGBoost’s figures on screen relabelled. That endpoint fits XGBoost and nothing else, which the control now says
LIME was offered in the same panel and has no implementation anywhere; selecting it returned native XGBoost importances labelled lime. Removed rather than left as a false option. SHAP is genuine where the package is installed, which it is not on this deployment, so that fallback now names itself instead of reading as a generic fallback
The feature importances had no labels and the fallback read backwards. The endpoint returned sixty numbers with nothing identifying them, so the chart showed F1 to F60 — and because each row is the preceding sixty closes, F1 was the oldest bar rather than the nearest. Anyone reading the chart left to right drew the opposite conclusion. They are named t-60 through t-1 now, and on AAPL the ranking is finally legible: the most recent close at 0.361 ahead of t-37 at 0.202 and t-2 at 0.195, the nearest bar dominating as a price-level regressor should
An earlier endpoint sweep was reported as covering the dashboard and did not. It hit the Next layer unauthenticated, so twenty three of sixty eight stopped at the auth gate and twenty seven were POST routes answering 405 to a GET — only sixteen handlers actually ran, and handler-level faults like the 503 above were exactly what it could not see. That limitation was stated rather than left implied, and then acted on
A proper authenticated probe now exists and has been run: 594 GET handlers called directly against the engine with the internal service token, returning 382 healthy and 97 failures — 37 server errors, 30 service-unavailable, 17 timeouts and the remainder. Triage is under way in a separate session. It is GET only by design, because this API carries autotrader, paper-trading and order routes where a blind POST would move money
One lead from that probe needs settling before anything else: several handlers fail with a dict where they expect a user object. That may be an artefact of how the probe authenticates, or it may be live — the frontend proxy sends the internal service token whenever one is configured, and one is, so if the two authentication paths produce different shapes then the API keys, billing and usage endpoints are failing for real users right now. Establish which before changing any handler

Phase 4.43 — Shadowed Modules, Unresolvable Imports, and a Polled 503

Oct 2026
Shipped7/8
88% complete
A probe of 594 authenticated GET handlers returned 97 failures, and the first question was whether that was an artefact of how the probe signs in or a live fault. It was live, and structural: app/core/services is a partial copy of core/services with three modules missing and two left as stubs written to preserve import compatibility after the originals moved. api/deps imports auth_service from there, so it could not import at all, and four v1 routers that try api.deps and fall back on ImportError silently took the fallback — which returns a dict where a user object was expected, so handlers died on attribute access
Removing that shadow moved the failure one layer deeper rather than fixing it outright: /api/risk and /api/strategy then failed on a missing get_live_quote, because another six-line stub of the same kind sat in app/infra/providers, accepting the four gateway callables it is constructed with and discarding all of them. The real adapter exposes the methods and imports cleanly, so the stub is deleted and the bridge resolves it
Four more handler faults, all the same shape — a name that does not resolve, surfacing as a 500 rather than as the missing thing it was. The readiness probe imported health_service as a bare top-level module; it lives under app/utils and the repository root is not on sys.path in production, so /ready answered 500 on every call. That is the endpoint an orchestrator polls to decide whether the service should receive traffic at all
Chasing that one import turned it into a class rather than a one-off: fixing health_service revealed error_handler below it, and that revealed observability below that. Rather than keep pulling the thread, the pattern was swept for — bare imports of modules that exist only inside a package — and thirteen matches found. The sweep’s first answer was wrong, checking only root-level files, which is worth recording: the correction is what turned three fixes into a complete one
The Technical Analysis tab was reading a thinner endpoint than the one that exists. Both Next and FastAPI implement /api/portfolio/technical-ai-signals, and Next route handlers beat the fallback rewrites, so the TypeScript one silently shadowed the Python one. Python is a strict superset: the panel reads around thirty indicator fields and the shadowing route supplied ten, so ADX and its DI lines, Donchian, Keltner, Supertrend, Stochastic, Hull MA, Fisher, KAMA and Hurst were all absent, along with a composite-signal card that could never render
SHAP values in the explainability tab are now actually SHAP values. The endpoint had always attempted TreeSHAP and fallen back to native gain importances when the package was missing, which it was — so the tab showed gain and did not say so. shap 0.49.1 installed on the server after a dry run, since installing into this venv has caused trouble before: it resolves to shap and slicer alone with no downgrades, and numpy, torch and transformers are unchanged
Two panels were polling an endpoint that can never succeed. The intraday-agents status route returns 503 because no such agents are deployed — an accurate, permanent answer — and both the RL Live and RL Paper Trading panels re-asked every ten seconds regardless, filling the browser console with failed-resource errors for a question already answered. Handling the 503 in JavaScript silenced the panel but not the console, because the browser logs it at the network layer before any code runs. Both now stop polling once the answer is known and re-check only on an explicit refresh
The handler probe’s remaining failures have not all been triaged. The ones fixed here were chosen because they shared a cause; what is left is a tail of unrelated faults, and the probe itself is the thing worth keeping — it is the only check that exercises handlers behind authentication, where most of these hid

Phase 4.44 — Migrations That Never Ran, Routers That Never Loaded, Prices Never Observed

Oct 2026
Shipped8/9
89% complete
The backtesting endpoints had answered 500 with UndefinedColumn: backtest_runs.run_uid since June. The models matched the designed migration exactly; the migration had simply never taken effect, because every statement in it is CREATE TABLE IF NOT EXISTS and a backtest_runs of a different shape already existed, so it ran clean and did nothing for four months. The tempting fix — rewrite the five model classes to match the live table — was started and abandoned: the live table has no strategy_params, benchmark, scenario or engine_node, so matching it would have deleted a working design in favour of one that cannot carry the feature, for a table holding a single row. The ad-hoc tables were renamed aside and the designed schema created, in one transaction, with the column set verified inside it before commit
The same silent-skip pattern had hidden more: all 28 phase2 tables were declared and none existed, so /api/v2/dynamic/allocations and /api/v2/federated/learning/models failed on relation “advanced_features_p2” does not exist. Creating them needed an explicit table list rather than the module’s own init, which calls create_all over every model in the application — including ones already shown to disagree with the live schema. AdvancedEdgeCacheRecord also described a different record than its service used and was realigned
Two routers had not loaded at all, and the cause was a fix: renaming owner_user_id to user_id in the Portfolio model landed on two call sites that already passed user_id, producing user_id=…, user_id=… twice. A repeated keyword argument is a SyntaxError, so neither module could be imported, and _include_router_safe catches that in its blanket handler and lets the app start — so FastAPI reported healthy with portfolio_v1 and paper_trading_router simply absent. 1493 routes before, 1540 after: 47 endpoints were missing
That one is worth recording for how it survived. A missing route returns 404, which is indistinguishable from “no such record” without checking the route table — and the Portfolio model fix had been reported as verified on exactly that evidence, /api/portfolio/portfolios/1 returning “a correct 404 for an empty table”. The route did not exist. The model fix itself stood; nothing about it had been exercised
ast.parse does not catch a repeated keyword argument — it is raised in the compile phase — so a parse-based sweep of 562 files reported the tree clean while both imports failed. The same sweep using compile() found both files immediately. A second audit, of every entry in the router loader, then reported thirteen routers broken by AttributeError including the one just proven to be serving: each loader tuple carries a second element that reads exactly like the attribute name, and the loop ignores it and always reads `router`. That dead element is now removed, since honouring it would break all thirteen
get_historical_data fell through to fabricated bars on any failure and on an empty result — open 100.0, high 105.0, low 95.0, close 100.0, volume 1000000, repeated for every date requested, under the comment “ensures backtesting doesn’t completely fail”. Every close identical means volatility and every return are exactly zero, so a Sharpe ratio, correlation or backtest computed on it returned a confident number derived from prices never observed, with nothing marking it synthetic. Sixteen call sites reached it, including five in analysis_router and three at days=365
It was not a rare path. Yahoo was rate limiting the server while the fix was being made, which is how often that fallback must have been firing. The callers were already written for an empty result — analysis_router raises 400, dashboard_intelligence_v1 answers “No market data available”, ai_agent_service raises — and the fabricated bars made every one of those guards unreachable, which is why no caller had ever reported missing data
A rate limit is now 503 rather than an empty list, because [] made “the provider refused to answer” identical to “this symbol has no history”, and the callers turn a falsy result into 400 — blaming the caller for Yahoo’s throttling. MarketDataUnavailable subclasses HTTPException so it survives the except HTTPException: raise guards already present and needs no edit at the other call sites. Two callers were swallowing it and were corrected: one returned HTTP 200 with the error in the body, the other caught it and raised its own 503 saying “need 60 points”, losing the reason
feature_sets was deleted rather than fixed — cross_asset returned a hardcoded VIX of 18.5, a yield of 4.2 and a DXY of 104.3, alt_data a news sentiment of 0.12 and fixed insider trades, and router wired all ten into endpoints. It was never registered in main, so none of it was live; the risk was that registering it would have published those constants as measurements. What remains unaudited is whether other modules of that kind exist unregistered elsewhere

Phase 4.45 — Handler Triage to Zero, Accuracy That Was Memorisation, and a Narrative Signal With a Record

Oct 2026
Shipped19/21
90% complete
Three routers each built their own Jinja2Templates with two dirnames where they needed three, so every template path resolved under app/templates — a directory that has never existed. app/core/templates.py had the correct instance all along and says in its first line “import from here, never create a second one”. Cost: /landing answered 500, /forgot-password 500, /admin/analytics 404 “Admin template not found”, and /login returned 200 while serving the 40-byte “MALI Platform - Login” stub instead of the 91KB portal. All five TemplateResponse calls also used the pre-1.0 Starlette argument order, so the context dict was read as the template name and Jinja raised “unhashable type: dict” — that was the /enterprise/admin 500
hasattr(request, “session”) cannot guard anything. Starlette’s session property raises AssertionError when SessionMiddleware is absent, and hasattr only swallows AttributeError, so the assertion propagates. This app installs no SessionMiddleware. In auth_2fa_legacy that was live: a 2FA generate or verify call with no user_id in its body raised instead of returning the “User not authenticated” response three lines below it
Six integration endpoints reported work they had not done. slack/send-alert and discord/send-alert read the webhook URL and the message, discarded both and returned success — nothing was ever sent; Twilio returned a constant SM123456, SendGrid a constant msg_123456, the calendar route built a Google event body and dropped it, and the webhook routes returned a wh_1 pointing at example.com and “deleted” any id. On a notification path that is worse than a 500: the caller is told the alert went out, so nobody looks for the one that never arrived. None of those credentials exist on this deployment. Slack, Discord and Twilio now send for real and answer 402 when unconfigured; SendGrid delegates to the SMTP service that already works; webhooks and calendar answer 501, because there is no webhook store and no OAuth grant to write a calendar with
Two vocabularies for the user id had grown up on either side of the auth dependency — the dependencies build “id”, while build_demo_user, the synthetic-token branch and the raw JWT payload build “user_id”. Of 54 call sites, 34 read one and fall back to the other; 20 did not, and each failed in the direction its author had not tested: 8 portfolio endpoints raised “Authentication required” at 401 against callers who were authenticated, 6 auth endpoints answered 404 “User not found” to the user asking about themselves, and one raised KeyError reported as 500. Fixed at the dependency, additively, after confirming nothing branches on either key being absent
The second half of that: users.id is a UUID, so every int() on a user id raises. One sat inside login_user, after the password had already been verified and with the exception re-raised, so the login attempt died there; another made POST /api/v1/portfolios answer 500 for every real user; a third was swallowed by a bare except, so display_name came back None for everyone who had set one. Handlers that compare an identity to users.id now check its shape first and answer 404, since “service” and DEMO_USER_ID (“0”) both reach them and neither names a row
Handler-level 500s across the 564 authenticated GET endpoints: 37 at the start of the triage, then 26, 20, 6 and now 0 on a clean full probe. 58 503s and 4 timeouts remain untouched; the largest 503 cluster is 14 /admin/* endpoints all reporting “User directory unavailable”, which looks like one cause and so probably one fix
/api/portfolio/technical-ai-signals trained a RandomForest, a GradientBoosting and an MLP on a 3-bar direction label and published clf.score(X, y) — the score on the rows each model was fitted to. Live, it reported Gradient Boosting at 0.9898 accuracy for predicting three-day stock direction. Measured over eight tickers on two years of daily bars with a time-ordered split: 0.5316, 0.5303 and 0.5289 out of sample against a majority-class rate of 0.5395 — all three below the accuracy of always guessing the more common direction. There was no skill to preserve
What replaced them is what the frontend had already asked for: the panel types accuracy as “its measured hit rate on this history”, renders a samples count the endpoint never sent, and titles the card “Rule Consensus · measured hit rates”. It was rendering in-sample classifier fit under that label because the primary and its own fallback disagreed about what they were sending. ml_models/rule_hit_rates.py ports the four measured rules the Next fallback already computed, so both now send the same thing, and check_rule_parity.py holds them together by lifting the declarations verbatim out of route.ts — 56 rules across 14 price series. AAPL reads 0.50/0.50/0.45/0.51 over 86-117 real calls where it previously claimed 0.86/0.99/0.62/0.82
The Forecast Accuracy tab published +0.2162 skill where the pooled figure was -0.0420 — across the random-walk anchor the whole panel is built on, and through predictability_score the difference between roughly 76/100 and 44/100. aggregate_scores took np.mean of per-forecast skill ratios, which is the estimator error pooled_step_skill explains forty lines above it in the same file: the module diagnosed the bug in one place and committed it in another. 8 of the 13 scored forecasts were single-point, where skill is a one-observation ratio, and they averaged +0.3222 against +0.0465 for the rest
Significance was mean ± 1.96·SE on those ratios, with np.std at ddof=0, on a sample whose forecast windows overlap by construction — independence assumed where it does not hold, which makes every interval about √h times too tight. ml_models/forecast_stats.py replaces it with Diebold-Mariano (HAC truncated at h-1, Harvey-Leybourne-Newbold correction, and a declared note when h-1 approaches n/4 and the variance estimate is too noisy to read), a circular block bootstrap for the hit rate, a two-proportion test so the base rate’s own error counts, Mincer-Zarnowitz with HAC errors, the Berkowitz PIT test, Benjamini-Hochberg across the leaderboard, and an effective-n that counts non-overlapping windows instead of calendar spread
Carrying that through to the service found why the numbers were strange in ways the estimator could not explain: 763 of 790 stored forecasts were captured on 2026-09-30 with anchors a median 22.3 sigma from the asset’s own last close — SPY at 656.82 against a realised 762.63, XLE at 44.13 against 61.50, AAPL at 178.18 against 332 — while every other capture day matched to the cent. The anchor is what every metric is measured against, so it was producing 5.5% coverage on a 90% band and a Mincer-Zarnowitz slope of -2.84, which reads as a catastrophically bad model rather than as bad bookkeeping. The predictions track the bad anchors, so they are not salvageable; scoring now rejects an anchor beyond six sigma, judged in the asset’s own volatility because one percentage cannot serve both a bond ETF and a cryptocurrency
Four more faults in the same layer: 13 scores against 790 forecasts was not a broken scorer but 777 unmatured ones, so open forecasts with at least three elapsed bars are now partially scored and upserted as the horizon fills; the UNIQUE constraint was on forecast_id, which does not stop one forecast being stored twice under two ids, and three such pairs were already inflating n; intraday forecasts were aligned on a ten-character date so every hourly point resolved to the same daily close, and one had been scored against a flat line; and summary() read bar_seconds and horizon_steps off the newest row and applied them to a slice spanning six different combinations, pooling 1-step and 21-step squared errors as if they were the same quantity
One finding the new diagnostics produced that the slope could not: predicted cumulative returns have a standard deviation of 0.39% against 8.7% realised. The forecasts are 22x too timid. The MZ slope of -11.14 is exactly corr(-0.4977) × 22.4 and arithmetically correct, but a slope computed over almost no predictor spread says nothing dependable about direction, so the spread ratio is reported instead and slope_reliable goes false beyond 5x. predictability_score also stopped contradicting its own docstring — it claimed skill carried the score, while at the +0.0086 this domain actually produces that term contributed 1.2 points and calibration error alone could subtract 10
The Narrative Signals tab was running on two regexes: three possible values, every row given an identical “strength” of 0.45 and a category of “Market Signal”, and 16 of 21 real headlines discarded for scoring exactly zero. It fell through to that because /api/reuters returned count:0 without a Reuters key, and the enrichment it would have used points at a container hostname that does not resolve here — and never set nlp_sentiment_score, the field the panel requires, so that path could not have worked even on success. ProsusAI/finbert was installed and loading the whole time. Headlines now come from Yahoo RSS and are scored as P(bullish) - P(bearish): eight live NVDA headlines gave eight distinct values spanning -0.841 to +0.663, and the feed is genuinely per-symbol — checked across three tickers
Removing this panel’s fabricated price target was right, but “no price target” had become “no accountability”: nothing recorded what the signal said, so its value was unmeasured by construction. narrative_validation.py records each reading and scores it over five trading sessions on hit rate against the asset’s own base rate and on the information coefficient — Spearman, because the signal ranks rather than predicts a magnitude. Both are withheld below ten independent observations. Validated on 90 deliberately random readings, which returned IC -0.036 with an interval spanning zero and a hit rate on the base rate: it does not manufacture an edge from noise. Those synthetic rows were then deleted
The tab also stopped guessing whose portfolio it was showing. Six hardcoded tickers matched nobody’s positions; it now leads with the signed-in user’s holdings, keeps SPY and QQQ as benchmarks, and accepts any typed ticker validated against the character set Yahoo symbols actually use, so ^VIX, BRK-B, BTC-USD and 7203.T all work. The measured track record is shown in the panel rather than behind an endpoint, because a sentiment readout with no history invites the reader to supply their own confidence
Both accuracy layers now report nothing rather than something unsupported, which is the right answer and an uncomfortable one: the forecast tab withholds its headline below ten independent observations, and 516 forecasts with 21-day horizons generated inside two weeks carry about one. The narrative tables are empty by design and need five trading sessions before the first reading matures. Earning those samples is the work that remains
The MALI Lab Explainability and Local Explanation tabs were removed, along with everything that existed only to serve them: both panels, both Next proxy routes, and the two api.ts helpers — each of which had exactly one caller, the panel being deleted. Registrations had to come out of four places, because the tab list is duplicated: DashboardShell carries the imports, TAB_META, its own mali_lab array, the description map and both switch cases, state/dashboard.ts keeps the TabId union and a second mali_lab array, and CommandPalette holds the labels. A tab left in the union but absent from the switch renders an empty pane rather than failing, which is how one gets half-removed
Their three Python endpoints went with them — POST /ml/model-feature-importances, POST /ml/deep-explainability and GET /ml/model-local-explanation — plus both entries in the capability catalog that ml_multi_forecast.py serves, since a catalog advertising routes that 404 is worse than a shorter one. Checked against the live route table rather than the diff, because the deletions were made by line range and the first handler sat directly above /ml/ensemble-models: 1661 routes before, 1658 after, and the three that went are the three intended. The Fairness tab and /ml/model-fairness-metrics are untouched
Two pieces of surrounding text would have been left describing features that no longer exist. The codex entries for both tabs explained how to read charts that are gone, and the Explainability one told the reader to “jump to Local Explanation for the per-sample view”; the demo tour promised “full explainability for every output” for MALI Lab, which is a tour describing a tab the viewer cannot find. Both now point at the measured accuracy that is actually there. The earlier roadmap item recording these three routes being de-faked is left as written — it was true when it shipped, and rewriting a history entry to hide that the work happened is worse than it reading as dated
shap stays pinned but its justification had expired within the same session that added it: it was declared for /ml/model-feature-importances, which had been falling back to native gain importances silently on this server until the dependency was made explicit. That endpoint is now deleted. The remaining consumers are POST /ai/explain/shap, which answers 503 when the import fails, and enhancements/technical/feature_engineering.py, which imports it at module level and so cannot start without it — named in the requirements comment, because a pin justified by a reason that no longer holds is how a dependency gets dropped by whoever tidies up next

Phase 4.46 — A BUY for Every Symbol, and the Sources That Were Never Read

Oct 2026
Shipped12/12
100% complete
A single endpoint recommended BUY for every symbol. POST /api/v1/ai/v1/analyze returned {"recommendation": "BUY", "confidence": 0.85, "reasoning": "Strong technical signals"} as a literal, echoing the requested symbol back, which is exactly what made the answer look specific to it. Verified on the deployed server before the change: TSLA, GLD and KO each got the same BUY at the same confidence. Of everything in this audit it was the most consequential, because the fabricated field is advice to buy
It now derives the verdict from 90 days of real history through the same technical analysis the rest of the platform uses, returning BUY / SELL / HOLD with a STRONG / MODERATE / WEAK conviction alongside the RSI, MACD, EMAs, ATR and Bollinger bands behind it — TSLA BUY/STRONG, GLD SELL/STRONG, KO HOLD/WEAK on those same three symbols. There is deliberately no numeric confidence in the response: nothing measures one, and 0.85 was precisely that invention
The services underneath were fabricating too, which is why fundamental and sentiment report 501 rather than being wired up to them. fundamental_analysis returned the same twelve numbers for every symbol — a PE of 15.5, a PB of 2.1, a target price of 200.0, a rating of BUY — and sentiment_analysis a score of 0.65 drawn from 150 sources. Both are reached by /api/v1/analysis/fundamental, /sentiment, the combined endpoint and ai_agent_service, so each reported the same company whichever symbol was asked for
Both now measure. Fundamentals come from the provider with missing figures left as null rather than filled in — a missing debt-to-equity is a fact about the data, where 0.5 is a claim about the company — giving AAPL a PE of 38.6, KO 26.4 and TSLA 354.4. Sentiment scores real headlines and reports the number actually scored: AAPL 0.1733, TSLA 0.156, KO 0.2022 from eight headlines each
A consequence of that constant worth stating on its own: the consuming endpoint calls anything above 0.6 "VERY BULLISH" and the service always returned 0.65, so every symbol on the platform was permanently very bullish. The ladder was also partly unreachable — "greater than 0.4" followed by "less than 0.4" left NEUTRAL for the single value 0.4 exactly. The bands are now symmetric about zero on the real polarity scale, every score lands in one, and a missing score reads UNAVAILABLE instead of defaulting to zero and scoring BEARISH
The news source behind all of it was silently empty. Yahoo returns an empty list rather than an error when it is throttling: measured on the server, .news and .get_news() both gave 0 items with no exception while Search raised a rate-limit error outright, so the silence was the throttle, not an absence of news. Nothing downstream could tell those apart, and that fetch is the only source behind both the sentiment service and the live sentiment stream. Finnhub, already configured, now answers when Yahoo will not — 244 articles for one symbol over seven days where Yahoo had none
Fifteen imports named a service that does not exist. They ask app.services.real_market_data for real_market_data_service; that module exports market_data_service, while the name they want lives in a different module of the same name one directory up. Because they are inline imports nothing failed at startup — each raised ImportError only when its function body ran, across analysis_router, dashboard_intelligence_v1, ai_agent_service, mali_router, both copies of mali_service and ml_heavyweights
They were found only because the same mistake was made writing the analyze fix, which shipped broken and answered 500 on every call until it was tested. Reading the diff would never have caught it, and it is the second time in two phases that a duplicated module name produced a bug invisible from the source alone
Testing that fix then exposed five handlers rewrapping a deliberate 503 as a 500: each ends in a blanket handler that also catches HTTPException, so a rate limit arrived as a server fault whose detail read "503: ...". A guard added to the inner fetch block was not enough, because it re-raised into the same outer blanket
Two audits came out clean, which is worth recording as much as the faults. No unregistered module anywhere else serves hardcoded constants the way the deleted feature_sets did — 167 route-declaring modules checked, 22 with routes that are not live, none of them literal stubs. And the "value or number" fallback pattern does not appear on any data-bearing name in live code
The sweep that found the rest produced two false positives worth knowing, both of which cost real time: an audit of the router loader reported thirteen routers broken by AttributeError while they were serving, because each loader tuple carries a second element that reads exactly like an attribute name and the loop ignores it; and a module was reported as declaring zero routes when it declares four, because the same positional guess picked up a function sharing the router’s name. Field(example=...) blocks and explicit demo branches account for most of the remaining noise
Now run, and reported in 4.47. The gap was: it ranks by whether a module declares live routes, so service modules called by live routes were suppressed from its output — which is exactly where the fabricated fundamentals were hiding. They were found by reading, not by the tool. The same sweep over called-but-not-routing modules has not been run

Phase 4.47 — The Catalog That Invented Itself

Oct 2026
Shipped8/9
89% complete
The blind spot was the ranking, not the detector. The earlier sweep asked "does this file declare a live route", which suppressed every service module — and a service module is exactly where the fabricated fundamentals were. Seeded from the 155 modules declaring a live path and closed transitively over their imports, including the ones inside function bodies, 470 modules are reachable and 315 of them declare no route of their own. That is the set the first sweep could never show
272 hardcoded-value hits across 41 of those modules. One was a live fabrication, two looked damning and were clean, and the remaining 36 files were configuration — regime thresholds, stress-scenario definitions, MACD periods — constants that are the parameter rather than a stand-in for a measurement
/api/ai-ml/kpis returned aiHealthIndex 94, forecastIntegrityScore 96 and syncPulse "synced" as literals, beside a modelFleetStatus genuinely counted from the table. That adjacency is the whole trick: three inventions wearing the same shape as a real figure. Both scores are now null, because nothing computes either, and the response carries modelsRegistered so an empty catalog can be told from a populated one
Underneath it, the catalog invented itself. check_and_seed_data was 158 lines that ran whenever ai_models was empty, so the table filled in and then looked populated: three models that do not exist — a Temporal Fusion Transformer at 0.913 accuracy, a relational GNN at 0.935, an LSTM at 0.892 — each with a full metric suite (RMSE 0.0125, hit rate 0.725, per-regime accuracy), a lifecycle claiming 14,200 training samples and a drift status of "stable", plus five asset forecasts, a fusion snapshot and three scenarios
Production held exactly that and nothing else: 18 rows across seven tables whose counts matched the seeding code line for line, which is how it was established that no real training run had ever contributed. The code shipped first so the seeder could not refill, then the rows were backed up and deleted in one transaction after a dry run. Calling get_kpis afterwards — which invokes the seeder — left every table at zero
The Mali Lab registered-model catalog now shows empty. That is the honest state: no model has been registered and scored. It is wired in 4.48, from the measurement that already existed
Two that looked damning and were not, recorded so they are not re-flagged: a market-data module holding 62 hardcoded prices is a DEMO_MARKET_SNAPSHOTS table referenced only by the test suite, and a second module’s demo quotes are reached only after the real path has already returned price null with data_unavailable true and a per-provider refusal list — the honest-failure path working exactly as intended
The fallback pattern stayed absent here too. Across both sweeps, no "value or number" substitution appears on any data-bearing name in live code, and the single random call in the blind spot is a seeded generator for reproducibility
What is still unswept: both sweeps test numeric literals bound to data-bearing names, so a constant wrapped in a call, or returned through a variable assigned far from its use, is invisible to them. Every finding so far has been the plain form

Phase 4.48 — A Catalog of What Was Measured

Oct 2026
Shipped7/7
100% complete
The catalog is served from measured skill. ml_models/model_skill.json already held each model run alone through the production pipeline at ten historical origins over a 21-day horizon on real data, scored against a random walk, and model_skill.py already exposed catalog_fields() documented as "shaped for the model catalog response". Nothing had ever called it from here — the seeder’s three invented models had been sitting on top of a real measurement the whole time
Fourteen models, ordered by measured skill: TimesFM +0.00863, HAR-RV +0.00675, Neural SDE +0.00489, GARCH-MIDAS +0.00225, GARCH +0.00200, Theta +0.00159, then Chronos-Bolt, Moirai, LightGBM, N-HiTS, XGBoost, iTransformer, PatchTST-XL and Mamba-TS, the last at -1.44369. Eight of the fourteen are worse than a random walk on this basket, which is the measurement, not a fault — and it was previously hidden behind three models that did not exist claiming 91 to 94 per cent accuracy
accuracy is null and stays null. Skill against a random walk is signed and frequently negative; carrying it in a field named accuracy would relabel a measurement as a claim nobody made. The figure travels as measuredSkill with its worst single asset, its per-class breakdown, and the horizon, origin count and measurement date beside it, so the number cannot be read without its provenance
"Active" means something checkable: the six models ranking() returns, which clear two hurdles — better than a random walk, and better than the v2 engine running alone. The other eight are labelled measured, which is deliberately a different word from unmeasured. Warming, cooling and retraining stay at zero, because they are lifecycle states nothing here tracks and apportioning models to them would make the fleet look busy without measuring anything
The measurement is read at request time rather than copied into ai_models. A copy is a second source that drifts from the first, and that drift is the failure this catalog already had. The table stays empty and unused, and the KPI fleet count reads the same catalog — leaving it on the table would have reported a fleet of zero beside a catalog of fourteen
Checked rather than assumed, after the catalog went live: the default roster is {timesfm, garch}, both positive, with the measurement cited in the code beside it; ranking() never recommends a negative model; and three frontend panels already read measured skill, so a weak model cannot be selected without its number visible. Nothing in the forecast path uses the eight
Re-measured in 4.49. The confound was: Every figure was recorded while the v2 engine ran on every request regardless of selection, so each number is that model plus v2, and the ranking carries an extra hurdle to compensate. v2 now runs only when the roster names it, so the table is measuring a configuration that no longer exists. Re-measuring could change which models are recommended

Phase 4.49 — Re-measuring, and What Moved

Oct 2026
Shipped7/7
100% complete
The table was re-run at the same settings on the same cached history, so any movement is attributable to the pipeline rather than to different data. It did not drift, it inverted. v2 alone went from -0.01934 to +0.04315: the engine every model is scored against is now substantially better than a random walk where it used to be worse. Eleven of thirteen models beat a random walk, against six of fourteen before
That raises the bar rather than lowering it. Each model is measured as "that model on top of v2", so a stronger v2 is a harder line to clear, and ranking() now returns har_rv alone. timesfm at +0.04078 no longer beats v2 running by itself at +0.04315 — the model that was best on the old table is now below the reference
Which leaves the default roster resting on superseded figures. {timesfm, garch} is justified by a comment citing the old numbers, and on the new table neither model beats v2 alone. It is deliberately unchanged: this script measures single models against v2 and says nothing about which pair is best, so choosing a new roster needs a combination measurement that does not exist yet
A bookkeeping fault in the measurement script, found by it publishing one. moirai is not installed on the measuring machine — uni2ts forces a torch downgrade and a CUDA stack onto a CPU-only host, so it is measured in a throwaway venv instead — and the merge path carried its previous score into a table stamped with the new date, then dropped it from unmeasured_here precisely because it had been carried forward. The two halves contradicted each other
Carrying forward is right for a subset run, which says nothing about models it was not asked for, and wrong for a model that was asked for and could not run. Those are now separated: the stale score is dropped and the model is named as unmeasured, which is what the rest of that module already promised. The committed table is the output of the fixed script, not a hand-edited file
Worth recording how that fault came to be published: a one-model smoke test, run only to check the script worked, overwrote the shipped table. It left thirteen entries from the previous run beside metadata claiming two origins and a new date. The script writes its output on every invocation, subset runs included — a cheap test is not a read-only one
moirai stays out of the table rather than being re-measured. Its -0.02764 was a real measurement, taken deliberately in a throwaway venv with the v2 reference measured alongside to keep it comparable, and the exclusion is documented with a reproduction recipe. But it was measured against a v2 that no longer exists, so republishing it beside today’s rows would be comparing two different baselines

Phase 4.50 — Python 3.11, and a Voice Nobody Could Hear

Oct 2026
Shipped9/9
100% complete
Both model services moved off Python 3.10, which is now removed from the host entirely along with 14 GB of rollback venvs. The ML service needed no dependency work at all: its pinned requirements resolve on 3.11 unchanged, and it came back with the same health payload, the same 24 routes and no restarts
The first cut-over failed, and the reason is worth keeping: a venv is not relocatable. Building at venv311 and renaming it to venv broke every console script in bin/, because each carries an absolute shebang naming the build path. systemd reported the uvicorn executable as missing, where the missing file is really the interpreter named in its shebang — and the smoke test could not have caught it, having run from the un-renamed path. The unit is edited instead
The voice service was harder, and a resolution check was not enough. Coqui TTS 0.22.0 allows 3.11; what failed was the 3.10-era freeze, where numpy==1.22.0 predates 3.11 wheels and pip named eight pins in a ResolutionImpossible. Installing TTS and letting it choose its own versions resolves cleanly
Then it installed, started, answered /health and listed every speaker — and returned 500 on every attempt to speak. TTS 0.22.0 imports BeamSearchScorer from transformers, which 5.x removed, and TTS asks only for >=4.33.0, so the resolver took 5.20.0 and XTTS could not import its own config class. Pinned to 4.42.4, the version the working 3.10 venv ran, it produced a 3.2-second WAV
That is the lesson the deploy now carries in a comment beside its health check: /health and /speakers answer perfectly while the model cannot load, because neither touches it. Only synthesising distinguishes a service that is up from one that can speak
The dependency set is now recorded at all. There had never been a requirements file, which is why a stale freeze was the only specification to migrate from, and each pin carries the reason it exists rather than a version alone
And the voice had never been audible through the app. Every fetch in the voice route carried a 1,000 ms timeout, against a synthesis measured at 4.12 s cold and 2.57 s warm, so the Coqui branch aborted on every request, fell through to edge-tts on the same budget, and ended at the 503 that tells the browser to use its own speech. The service was synthesising correctly throughout; nothing it produced ever reached a user, and the failure was silent because the fallback is a legitimate response. The app now returns audio/wav
The voice service was also in no part of the deploy — no dependency install, no restart, no health check. It serves its module straight from the repository, so every deploy left it running the previous code until something happened to restart it. All three are wired in now
One consequence of that restart, measured rather than assumed: XTTS takes 56 seconds to load, during which /health answers 200 and synthesis blocks. For about a minute after each deploy the voice falls back to browser speech and then recovers on its own. Raising the budget past the prewarm would mean a user waiting ninety seconds for a sentence, so the fallback is the better behaviour and 30 s stands

Phase 4.51 — One Model Order for Every Text Task

Oct 2026
Shipped4/4
100% complete
DeepSeek now answers every text task, with Zhipu GLM as the backup. The Google key is suspended and the OpenAI key is invalid, so a provider order that started with either spent its first attempt on a certain failure before reaching a model that could answer
Removing the dead OpenAI key made a whole services module unimportable: one optional client was built at import time and raised without a key, taking every other service in the file down with it. The client now reports itself unavailable and the module loads
/api/ai/capabilities names the models the features actually call, rather than a list written when the provider order was different. The pricing page says the same: every plan has the same AI features, on DeepSeek with Zhipu as the backup
The answers given during onboarding now calibrate the AI: the AI routes pass the asker’s profile to the model as context, so an answer is pitched to the person asking

Phase 4.52 — A Demo Account That Never Touches Real Data

Oct 2026
Shipped5/5
100% complete
The owner’s rule: a session signed in as the public demo account sees synthetic figures, every one labelled as synthetic, and every other session sees real data or an explicit unavailable. The decision is made per request, from the session, never by falling back to invented numbers when a live source fails
That rule now holds at every source a demo session can reach: quotes and history from every shared backend price source and from the code that calls Yahoo directly, FRED series and the macro indicator rows, news and SEC filings, option chains and fundamentals, sentiment and macro feeds, and the AI text, which is synthetic and labelled on both the Next.js and FastAPI sides
Shared caches keep demo entries apart from real ones, and a demo session works on its own copies of the shared tables, so nothing a demo visitor does reaches a real member’s data. A public login has no real-world effects: it cannot send email, open billing, start a subscription or delete the demo account
Background jobs, stream publishers and the community chat were traced for anything a demo session could still send out or receive from real members, and each crossing was closed. The chat keeps the demo login and real members apart
A guided presentation for the demo account, narrated in a British male voice, walks through the platform, now including MALI Lab’s Forecast Engine. The demo’s simulated portfolio is scaled to about $10 million, and administrators can see how many people visit the presentation

Phase 4.53 — Messaging, Rebuilt

Oct 2026
Shipped4/4
100% complete
The old messenger let any member read or post into any private chat by naming the room, let anyone post under any name, sent every member’s email address to every member, and claimed in a banner that messages were end-to-end encrypted and ephemeral when they were neither
Community chat and private member-to-member messages were rebuilt on one authenticated stream per tab. Private events are addressed only to the conversation’s participants, members never see each other’s email, and messages are stored and shown as plain text
Blocking, reporting, moderation hooks, rate limits and notifications exist for the first time. Streams re-check the session every two minutes, so a revoked session or a deactivated account loses its stream promptly
The Direct Messages pill left the dashboard header; the round messenger toggle stays, with its unread badge

Phase 4.54 — Accounts: Agreements, Two-Factor Sign-In and Real Deactivation

Oct 2026
Shipped6/6
100% complete
The End User Licence Agreement and the Financial Disclaimer are accepted at sign-up, before a verification code is sent. Registration refuses a request without both, in the browser and on the server, so no account exists without them
Two-factor sign-in can be turned on in User Profile. Sign-in then asks for the current six-digit code from an authenticator app
The session bridge issues a session only for the user an access token verifies to. It used to accept any user id in the request body
Deactivate now sets a real disabled state, which sign-in and the messaging streams check. It used to change the email-verification flag, and an administrator cannot deactivate their own account
Administrators can open everything stored about a member from User Management. Password hashes, two-factor codes and direct-message text are never shown. The licence-acceptance log is restricted to administrators
The dashboard opens on the default tab saved in Settings, which it previously ignored

Phase 4.55 — Plans That Mean What the Pricing Page Says

Oct 2026
Shipped6/7
86% complete
The daily AI-query allowances are enforced on both the Next.js and FastAPI AI routes: 10 a day on Starter, 200 on Pro, unlimited on Edge and Institutional. A query is one request that asks a model for an answer; a debate is one query however many agents speak, and answers from a cache or from rules alone count as none
Starter market data is 15 minutes delayed, as the plan card says, and a Starter account is told so where it sees a price. The Market Scanner stays live on every plan by the owner’s decision. Figures with no delayed counterpart — option chains, the bid and ask, intraday and 52-week ranges — come with the real-time plans only
A Starter account holds one portfolio. The features the plan cards sell with Pro and Edge are held to paid plans on both servers, each refusal naming the feature: the backtesting engine, the daily AI brief, the multi-agent debate, portfolio autopilot, the voice assistant, alternative data feeds, the scenario and stress engine, the compliance co-pilot, the options strategy generator and custom AI agent crews
The plan cards show the prices Stripe charges. Pro is $95 a month or $969 a year; the page had shown $29, which Stripe never charged. Edge is $79 a month or $790 a year. Pro and Edge carry a 14-day trial on a first subscription. Institutional is priced to order, and its custom agents, white-label and SSO/RBAC are offered as built to order, because none of them exists until a contract funds it
Sales tax is charged through Stripe Tax on both checkout paths. Checkout collects a full billing address and asks Stripe to validate it, so a subscription is not created with tax silently switched off. Billing offers Pro subscribers the move to Edge
The public API is live at /api/public/v1 for paid plans: keys made and revoked on the API page and shown once, stored only as hashes, up to five per account, 60 requests a minute per key. Every data answer carries the disclaimer and the terms it is governed by
Starter’s 15-minute delay covers the main quote routes; a few of FastAPI’s own quote routes do not apply it yet

Phase 4.56 — Alerts and Reports That Actually Send

Oct 2026
Shipped4/4
100% complete
Saved alert rules were stored and never evaluated. A scheduled run now checks every armed rule each minute against the owner’s own quotes, delayed on Starter as everywhere else, and emails the account’s verified address when one fires
Report schedules were saved with a cadence and only ever sent by hand. They now go out daily, weekly or monthly from the date they were made. Each send claims its row first, so two runs cannot both send it, and a deactivated account gets neither alerts nor reports
Volume-surge and news-mention rules were offered and could never be checked: a delayed quote carries no volume, and nothing reads headlines for a rule. The form no longer offers them, and existing ones are reported as not checked rather than guessed at
The daily brief shows the FRED macro indicators it reads, each with its period and publication day

Phase 4.57 — The Autonomous Financial Intelligence Layer

Oct 2026
Shipped4/5
80% complete
A pipeline that turns news into debated theses, theses into tracked predictions, and scored predictions into agent weights: enrichment, agent analysis, debate, memory retrieval, decision, thesis, prediction, outcome tracking and calibration, each stage a consumer group on Dragonfly streams so any one can run on its own
The numbers come from deterministic scoring of calibrated votes; the language model writes only the narrative. An opportunity’s expected return is the mean realised return of past opportunities in the same score bucket, and null below thirty of them, because the forecast audits found no single-asset directional edge to state a model return from
Predictions refuse an entry price more than five days old. Testing found that a stale close let a prediction be scored against a move that had already happened
Built for how MALI runs rather than for the brief it came from: the self-hosted Postgres, a supervisord worker that is off by default with per-process debate and spending caps, DeepSeek first with Zhipu as the backup, and no new CI workflows
Retrieval is lexical until pgvector is installed on the production database; the health endpoint says which mode is running. The worker spends money on every debate, so it stays off until it is switched on deliberately

Phase 4.58 — Surfaces That Say What They Have

Oct 2026
Shipped6/7
86% complete
The Strategic Brain reported a neutral composite built from fixed priors: +0.05 for liquidity it could not see, −0.05 for geopolitics, zero for the rest. A layer with no live feed is now unavailable and adds nothing, the composite is formed only from layers with real data, and with none there is no composite, regime or recommendation
The market-sentiment record the Brain, the RL agent, memory fusion and the pipeline read now has one scheduled writer. It was written only when someone happened to open the news panel, with no source, so the RL agent rejected it
Intraday paper-trading agents run behind the RL panels, so those panels show positions and results that were actually produced
Quant Stress Lab and Agent Diagnostics measure the holdings on the Portfolio tab, rather than a sample portfolio
Features that are not built say so: the Layout, Theme and Mobile tabs, the Competitions controls, the AI recommendation routes and the v2 routes with no data source now say a feature isn’t available yet, instead of presenting a working control or asking for a plan
The landing page carries no unsourced testimonials, implied customers or feature claims
Four of the Brain’s seven layers — microstructure, liquidity, geopolitics and derivatives flow — still have no producer, and the macro layer reports the yield curve without a regime model behind it. Until those exist, the Brain’s composite rests on sentiment and contagion alone

Phase 4.59 — Errors and Analytics Kept In-House

Oct 2026
Shipped3/3
100% complete
Sentry and PostHog were removed, along with their setup code, the injected key and their packages. PostHog’s identify call had been sending account email addresses out of the platform
Unhandled server errors are recorded in MALI’s own database, and product analytics events go to a table that strips personal fields — email, phone, names, passwords and network addresses — before anything is stored. Administrators read both through the admin interface
One script reports which database migrations deploys have skipped, and applies them

Phase 4.60 — Africa, the Gulf and Emerging Markets

Oct 2026
Shipped5/6
83% complete
The Global View carries 31 indices, adding Singapore, Buenos Aires, Jakarta, Kuala Lumpur, Bangkok, Manila and Ho Chi Minh City, and a Middle East & Africa region: Riyadh, Dubai, Doha, Cairo, Johannesburg, Tel Aviv and Istanbul
The FX panel gained eight emerging-market currencies and a Middle East & Africa tab: the naira, shilling, Egyptian pound, cedi, dirham, riyal and others. The GDP layer covers 22 economies, Nigeria, Kenya, Egypt, South Africa and Saudi Arabia among them
Several of these exchanges get only today’s bar from Yahoo, so a year-to-date or 52-week figure would have been computed from the wrong close. Those cells read — rather than a number nothing measured
The globe opens on a markets layer, with every index and emerging currency at its exchange, coloured by the day’s move. A signals layer places the scanner’s global verdicts at each instrument’s home market, with column height for conviction and a legend that says these are rule-based signals, not a forecast
The AI analyst’s answer now includes how a move ripples to related instruments, from the knowledge graph, within a time limit so a slow graph cannot stall the answer
Nigeria’s and Kenya’s exchanges sell their index data under licence and no free source carries it, so Lagos and Nairobi are drawn from their currencies alone, and say so. A licensed feed would close that

Phase 4.61 — Lint at Zero, and a Billing Cycle That Did Not Stick

Oct 2026
Shipped3/3
100% complete
The web app’s 268 lint errors and 81 warnings are cleared, across 115 files: real types in place of any, loaders that run through one shared hook, and state that resets during render rather than in an effect. One disable remains, on the sign-in page’s one-time handling of the address it opened at, with its reason
Fixing those surfaced a billing fault. Activation bills the plan and cycle recorded on the card setup, and the Monthly/Annual toggle never renewed it, so choosing Annual after the payment form loaded subscribed the user monthly. A plan change also dropped an accepted promo code while the page still showed the discount. The setup now follows the plan, the cycle and the accepted code
Three smaller faults went the same way: the watchlist’s price refresh fetched from inside a state update React may run twice, the article reader could leave its spinner on, and four menus dropped their styling

Phase 4.62 — Direction, Scored Live

Oct 2026
Shipped2/3
67% complete
Forecast Accuracy shows directional edge with its standard error and significance, a Brier skill score for the probability of an up move, and an independence test of whether the up and down calls carry information beyond their own mix
The leaderboard’s direction test counted hits against every scored forecast rather than the ones that made a call, and treated overlapping forecasts as independent. Both are corrected, and direction is scored at each step of the horizon
What it measures so far agrees with every offline test: no significant directional edge. The panel now says so with error bars instead of leaving the question open

Phase 4.63 — Twenty-Nine Hubs

Oct 2026
Shipped5/6
83% complete
The globe’s event hubs were New York, London, Tokyo and Frankfurt. There are now 29, one for each centre whose index the Global View carries, and each hub’s reading, correlations, performance line and notable-move events come from its own index
Eight of those indices, Shanghai’s CSI 300 and the Bangkok, Manila, Ho Chi Minh City, Riyadh, Dubai, Doha and Cairo indices, come from Yahoo as today’s bar only. Their hubs show the day’s move and say there is no history, rather than being dropped or given a volatility nobody measured
A move in a hub’s own index is announced from that hub instead of from a generic global origin. Event arcs reach the six most correlated hubs, and the correlation web draws each hub’s two strongest links, so twenty-nine hubs stay readable
Hub lookups matched by substring, so an event for New York (“ny”) could land on Sydney. They match whole names now, and the market strip, indices table and matrix each focus the hub they name, which none of them did before
Every hub carries a signal on the signals layer. Six had none: New York (now SPY), Paris, Zurich and Madrid (their country funds), Shanghai (a CSI 300 fund) and Cairo (CIB’s dollar shares, since the only Egypt fund is delisted)
A source with daily history for the eight session-only indices would give those hubs volatility, correlations and a performance line

Phase 4.64 — Earnings Surprise: Pre-Registered, and Not Adopted

Oct 2026
Shipped3/4
75% complete
The first non-price signal tested for the forecast, registered in the repository before any data was fetched: earnings surprise, measured on 2,405 reports across 24 large caps from 2000 to 2026, entering after the announcement so the earnings jump itself never counts
It carries real information, the first signal to clear a multiple-test correction: after a beat these stocks rose over the next 21 sessions 60% of the time, against 48% after a miss (t = 3.83), and they also beat the market at 5 and 21 sessions
It still does not earn a place in the forecast. Fitted only on outcomes known at the time, a drift built from the surprise never reduced error against the plain average post-earnings drift. Most reports are beats, so the odds barely move for most stocks, and a one-month gap of about 1% is small beside 8% monthly volatility. Not adopted, by the rule set before the test
A beat-or-miss probability adjustment at 21 sessions came close (t = 1.92) but was found after seeing the data. It needs its own pre-registered test on different stocks before it can be used

Phase 4.65 — One Backend Tree, One Schema Runner

Oct 2026
Shipped4/4
100% complete
Three package bridges let one file load as two modules, app.services.x and services.x, each with its own caches, locks and rate-limiter state. A test that patched one name missed every caller of the other. 417 references in 206 files now use one name, and the bridges are gone
The root services, core and small packages moved into app/, so there is one backend tree. The largest files were split by concern: app/main.py went from 1,913 lines to 547
The app no longer changes the schema when it starts. Its create_all, five DDL statements and a custom migration runner became one alembic revision, safe on a database that already has every table, and alembic is the only thing that migrates
Eight portfolio routes ran on a store stub with no methods and answered 500 to anyone past sign-in. Nothing called them, and they are removed with the chain only they used

Phase 4.66 — MALI in Thirty-Nine Languages

Oct 2026
Shipped2/2
100% complete
Settings → Language picks one of 39 languages, or auto-detect. The AI answers in it on both backends and in the mobile app, switches at once when asked, and keeps names, tickers and output formats unchanged
The interface labels follow the same choice, right-to-left scripts included, and shared text is translated once and stored rather than on every request

Phase 4.67 — Account Rights, and Plans Held on Every Server

Oct–Nov 2026
Shipped4/4
100% complete
Deleting an account never worked on a real database: 25 foreign keys blocked it, and the SQLite tests hid them. Each key now has an action, and Settings → Your Data offers a download of your data and a deletion that asks again for your password or an emailed code
Admins change a plan only with a stated reason, written to the audit record. A plan Stripe is billing is refused, because its next webhook would undo the change
An audit of the membership gates found paid features open to plans without them, gates that trusted cookies the browser sets, and lifecycle gaps. Each feature now checks the verified account’s plan on Next and on FastAPI, and a past-due account gets its grace period from the day it fell due
The rate limiter counts each visitor and each member apart, in one limiter every server shares

Phase 4.68 — The Forecast Band, Measured Again

Nov 2026
Shipped4/5
80% complete
Dropping the empirical scale from the 90% band made it measurably better, on a rule registered before any 2019–2026 figure existed: about 19,000 holdout forecasts per horizon across the 47-symbol basket. The band is now the fitted volatility term structure as it stands
The review found a cost the rule could not see: single stocks were worse at 21 bars. Refitting their volatility without past earnings days failed its own holdout (t −1.97 against the −2.0 required), so it did not ship
What did ship, after a pre-registered test on 24 large caps no earlier work had used: single stocks with a measured earnings jump get the old scale phased in over the horizon, none at 5 bars and all of it from 63. It passed at 63 bars (t −2.67) and improved 21 bars (t −3.75)
Two trade-gate values nobody measured are gone. A one-step forecast reported its steps agreeing in direction, though one step has no direction to agree with, and now abstains. A flat position reported its sentiment as opposing it, and now reports no position
Commodities fall inside the 90% band only 0.876 of the time at 63 bars, mostly in the 2008 and 2014 oil crashes. Eight variants failed in development, and the holdout was left unused for a later attempt

Phase 4.69 — Trading Hub: An Advisor Over Your Own Holdings

Nov 2026
Shipped5/5
100% complete
The AI Portfolio Advisor opens the Trading Hub. It runs seven checks against your risk tolerance, covering largest position, sector, top five, cash, volatility, beta and risk concentration. It grades only when at least four were measured, and suggests trims and adds toward a risk budget, each with the figures behind it
Goal-based Auto-Invest, an Auto-Trader committee, a Scenario Lab and a scored history of past advice. Nothing places an order
The Auto-Trader had ignored the profile, risk and leverage picked in its panel, so every run was the balanced profile at 50% risk and 1x. The run now stores and uses the settings it was given
Import your trades from a CSV: MALI’s own export, or Schwab, Fidelity, Vanguard, Interactive Brokers and Robinhood history, with a preview that marks each row as new, duplicate, error or skipped before anything is saved. Each buy becomes a tax lot, and tax-loss harvesting works lot by lot with a real wash-sale check
Spin-offs and reverse splits are applied to the lots they touch, cash in lieu counts as a sale, and editing an entry across a spin-off’s date applies or undoes it

Phase 4.70 — A Launch List, and Knowing Where Members Come From

Nov 2026
Shipped4/4
100% complete
/launch collects launch-access emails with double opt-in, keeping only a hash of each confirmation token, and records which social platform the link came from
A sign-up’s source travels from the link through email verification to the account and on to Stripe, with no cookie, so each paid subscriber can be traced to its channel. Trials get a sequence of emails instead of one that mostly never arrived
Yearly plans had never received a renewal notice: the hook sat on an endpoint Stripe does not call and imported a function that did not exist. Stripe’s upcoming-invoice event now sends one
The analytics tables had stayed empty because their timestamps were written in a form the columns refused. Fixed, and sign-ins and landing-page visits now appear in the Admin Console

Phase 4.71 — Event Ripple, SEC Financials and Event Alerts

Nov 2026
Shipped4/5
80% complete
Event Ripple on the Global View: when one regional market makes a large move, how the others moved in their next session, measured from about ten years of daily closes. It replaces an animation timed by on-screen distance, and it says it is a historical study, not a forecast
Market Explorer → SEC Financials & 13F: ten fiscal years and twelve quarters of statements from SEC XBRL, each figure linked to its filing. Quarters the SEC never reports on their own, Q4 among them, are derived and marked as derived. A manager’s latest 13F is set against the quarter before
Alerts beyond price: earnings dates and results, 8-K, 10-Q, 10-K, 13D and 13G filings, insider buying above a value you set, news on your holdings and unusual volume. Each has a daily cap and quiet hours, reads a real source, and reaches you once per event
EODHD now serves quotes and history for markets Yahoo covers poorly. A symbol it does not list is skipped without spending a request
The EODHD key in use is the free tier, at 20 requests a day. A paid plan would carry the global figures through a full day of use

Phase 4.72 — Calls on the Record, and a Debate That Shows Its Working

Nov 2026
Shipped5/5
100% complete
/track-record lists every call MALI makes, with no sign-in. Each is scored on its return over SPY across the same sessions, so a rising market no longer looks like skill. Calls are hash-chained as they are made and scored, the page publishes the chain head and a check anyone can run, and it states the limit: someone with database access could rebuild the chain
The AI Debate panel and the mobile Committee screen now run the real debate engine. The judge’s direction, confidence and agreement are computed from calibrated votes, and only the summary is written by a model. The panel shows which evidence was unavailable and why, and what the debate cost
What MALI remembers about you can be seen and deleted, and it is never placed in an agent’s prompt
Every document-retrieval route had failed on an import of a module that does not exist, over a store shared by all users that account erasure could not reach. Documents are now stored per member in Postgres, and erasure removes them
Every model call on both backends fences untrusted text, from members, news, filings, web pages and stored memory, behind one rule shared word for word: treat it as data and never follow instructions inside it

Phase 4.73 — Fewer Tabs, Measured First

Nov 2026
Shipped3/3
100% complete
21 backend routers that provably had no caller were removed, taking the route table from 935 to 886. The Admin Console now measures activation, retention and use of each tab, and those figures informed a written proposal for which tabs to keep, merge or cut
With the owner’s approval, nine tabs were cut: Real-Time Data, 3D Charts, RL Live Agent, Universal Stack, Copy Trading, Competitions, Layout Config, Theme Config and Mobile View. A link to a cut tab opens its successor. RL Paper Trading became admin-only, since it holds the trading agents’ kill switch
Forty tabs became sections of the tab they belong with, each keeping its own panel unchanged and loading only when opened. Chart Studio and Agent Workspace were retired into Market Explorer and AI Intelligence, and old links, shortcuts and the demo tour follow them

Phase 4.74 — Deploys Without a Gap

Nov 2026
Shipped3/4
75% complete
Each deploy used to replace the live containers in place: about 15 seconds of 502s from the web app and 30 of errors from the API. A stand-in now takes traffic until the new container passes its checks, and the 2026-11-17 deploy had no failed request. ML and voice get stand-ins too when the host has the memory for a second copy of their models
One deploy runs at a time, since two at once had pruned each other’s images. The containers run as an unprivileged user, and background jobs can run in a container of their own, so a job that crashes no longer takes the API down with it
A public status page at /status checks the web app, API, ML, voice, database, cache and each data provider every minute, with real requests
The staging stack and point-in-time database recovery with a streaming standby are scripted, with dry runs and a runbook, but have not yet been run on the server

Phase 6 — Open Items

Planned
Planned0/9
0% complete
Institutional plan: custom AI agents, white-label and SSO/RBAC, built for a customer under contract
Producers for the Strategic Brain’s microstructure, liquidity, geopolitics and derivatives-flow layers, and a macro regime model for the macro layer
Semantic retrieval for the intelligence layer (pgvector on the production database), and a deliberate decision on running its worker
Messaging: market channels, voice rooms and AI-assisted conversations, which the rebuilt schema already accommodates
Administrative role and status changes written to the audit log as they are made, as plan changes now are
Licensed index data for Nigeria and Kenya, and daily history for the eight hub indices that arrive as today’s bar only
A fresh pre-registered test of the beat-or-miss probability adjustment, on stocks the first earnings test did not use
A commodity band that holds its coverage at 63 bars through a crash that trends for months, tested on the holdout kept clean for it
Event alerts on the mobile app, and running the staging stack and point-in-time recovery on the server