Aggiunge il meta-modello, il database locale e la pipeline di ricerca della guida
Il bot ora ha un secondo parere prima di ogni ingresso: un classificatore GBDT (scritto in C#, senza dipendenze native) addestrato sugli esiti dei segnali passati con triple-barrier e meta-labeling, validato con CPCV, PBO e Sharpe deflazionato contando tutte le configurazioni provate. Il modello non propone mai operazioni: può solo rifiutarne una sotto la probabilità minima o ridurne la size, e si sospende da solo quando le feature dal vivo derivano da quelle di addestramento. Senza un campione promosso il bot opera come prima. Perché tutto questo serve, e nell'ordine in cui è stato fatto: - I log del giro reale sul testnet mostravano zero barre chiuse in tre giorni: il decodificatore saltava l'oggetto annidato dei kline. Corretto con test di regressione. Lo stesso giro restava a 1499/1500 barre di riscaldamento perché Binance ne serve al massimo 1500 per richiesta: il client ora pagina e il motore chiede quante ne servono davvero. - Il log è diventato una tabella `;` con data, livello, sorgente, evento ed eccezione (grep `;ERR;` trova ogni errore), con rotazione a dimensione impostabile dalla finestra. Anche decisions.csv/executions.csv/trades.csv hanno intestazione stabile, id monotoni e colonna `motivazione`, e vengono scritti anche in SQLite. - La configurazione vive in Documenti\Encelado (con migrazione dal file accanto all'eseguibile), le credenziali restano in LocalAppData, il database in %ProgramData%\Encelado: tre cartelle per tre ruoli diversi. - In modalità demo gli ordini partono davvero sul testnet (dryRun spento di fabbrica): è l'unico modo di provare il percorso di esecuzione come in produzione. - Lo strumento di backtest copre le fasi 0-4 della guida: qualità dei dati, baseline buy&hold/SMA con PSR e DSR, Engle-Granger + Johansen + Kalman con costo di break-even, dataset e addestramento del meta-modello, DQN su molti seed. Ogni tabella è CSV `;` con motivazione, e la promozione a campione avviene solo se il modello supera i criteri della Fase 3. Sui dati disponibili nessuna coppia supera quei criteri, quindi nessun campione è stato promosso: il bot resta sulla sola regola statistica, che a sua volta non regge fuori campione. Il risultato è documentato, non nascosto. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Claude Fable 5.1
@@ -3,6 +3,7 @@
|
||||
<Project Path="src/Encelado.Binance/Encelado.Binance.csproj" />
|
||||
<Project Path="src/Encelado.Bot/Encelado.Bot.csproj" />
|
||||
<Project Path="src/Encelado.Core/Encelado.Core.csproj" />
|
||||
<Project Path="src/Encelado.Storage/Encelado.Storage.csproj" />
|
||||
</Folder>
|
||||
<Folder Name="/tests/">
|
||||
<Project Path="tests/Encelado.Tests/Encelado.Tests.csproj" />
|
||||
|
||||
+285
-218
@@ -1,246 +1,313 @@
|
||||
Continua con le modifiche occupandoti anche della fase di backtesting.
|
||||
Ora leggi la seguente documentazione per eseguire il backtesting:
|
||||
Questo modulo effettua il backtest vettorializzato della strategia di Statistical Arbitrage, includendo la gestione dello stato della posizione (ingresso, take profit, stop loss) e il calcolo dell'impatto delle commissioni di trading (*fee* di Binance).
|
||||
# PROMPT PER CLAUDE CODE — Progetto "QuantBot": bot di trading crypto con ML e statistical arbitrage, paper-trading-first
|
||||
|
||||
### Dipendenze
|
||||
|
||||
Assicurati di aver installato le librerie necessarie:
|
||||
|
||||
```bash
|
||||
pip install pandas numpy requests statsmodels
|
||||
|
||||
```
|
||||
> **Come usare questo documento:** incollalo in Claude Code come specifica master del progetto. È scritto in italiano; codice, identificatori, nomi di librerie, formule e termini tecnici consolidati restano in inglese. Procedi per fasi, nell'ordine, e non passare alla fase successiva finché i criteri di accettazione della fase corrente non sono soddisfatti e documentati.
|
||||
|
||||
---
|
||||
|
||||
### Script Python: Backtest StatArb con Metriche di Performance
|
||||
## 0. Ruolo, obiettivo e vincoli (leggi prima di scrivere codice)
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import requests
|
||||
import statsmodels.api as sm
|
||||
Sei l'ingegnere quant/ML che costruisce un bot di trading crypto **research-grade**, modulare, in Python. L'utente (Alberto) è uno sviluppatore C++/Python esperto, usa Supabase (Postgres) e già gestisce un sistema news-sentiment su eToro tramite eToro Public API. Implementa il sistema descritto sotto, per fasi, con criteri di accettazione espliciti.
|
||||
|
||||
**Aspettative oneste, dichiarate in cima (NIENTE hype):**
|
||||
- L'evidenza accademica sistematica mostra che l'edge di un backtest si erode fortemente su dati mai visti. McLean & Pontiff (2016, *Journal of Finance* 71(1):5–32), su 97 predittori pubblicati: *"Portfolio returns are 26% lower out-of-sample and 58% lower post-publication"* — il calo out-of-sample (26%) è un upper bound dell'effetto di data mining, mentre il ~58% è il decadimento post-pubblicazione dovuto anche all'arbitraggio del segnale una volta reso noto.
|
||||
- Per dichiarare significativo un fattore, Harvey, Liu & Zhu (2016, *Review of Financial Studies* 29(1):5–68), su 316 fattori catalogati, raccomandano: *"A new factor needs to clear a much higher hurdle, with a t-statistic greater than 3.0"* (t=3.0 ≈ p-value 0,27%), proprio a causa del multiple testing. Applica questa soglia.
|
||||
- La previsione pura del prezzo con LSTM/Transformer raramente sopravvive ai costi out-of-sample; il problema centrale (López de Prado) NON è la previsione ma la **validazione**.
|
||||
- Il deep RL (FinRL) soffre di simulation-to-reality gap, instabilità delle policy e overfitting del backtest: è **opzionale/sperimentale**, non core.
|
||||
- L'arbitraggio di latenza cross-exchange contro HFT è **fuori portata** per un retail (finestre di 30-50 ms contro latenza retail di 100-500 ms). NON è core.
|
||||
- Un edge realistico per un solo sviluppatore con capitale modesto è: (a) relative-value/statistical arbitrage a bassa frequenza (minuti/ore) su asset cointegrati; (b) funding-rate / cash-and-carry basis; (c) meta-labeling su segnali semplici per ridurre falsi positivi e costi. Tutti con Sharpe modesti e capacità limitata.
|
||||
|
||||
def fetch_binance_klines(
|
||||
symbol: str, interval: str = "5m", limit: int = 1000
|
||||
) -> pd.Series:
|
||||
"""Scarica lo storico candele per Binance Futures."""
|
||||
url = "https://fapi.binance.com/fapi/v1/klines"
|
||||
params = {"symbol": symbol, "interval": interval, "limit": limit}
|
||||
res = requests.get(url, params=params)
|
||||
res.raise_for_status()
|
||||
|
||||
df = pd.DataFrame(
|
||||
res.json(),
|
||||
columns=[
|
||||
"open_time",
|
||||
"open",
|
||||
"high",
|
||||
"low",
|
||||
"close",
|
||||
"volume",
|
||||
"close_time",
|
||||
"quote_asset_volume",
|
||||
"number_of_trades",
|
||||
"taker_buy_base_asset_volume",
|
||||
"taker_buy_quote_asset_volume",
|
||||
"ignore",
|
||||
],
|
||||
)
|
||||
df["close"] = df["close"].astype(float)
|
||||
df["open_time"] = pd.to_datetime(df["open_time"], unit="ms")
|
||||
df.set_index("open_time", inplace=True)
|
||||
return df["close"]
|
||||
|
||||
|
||||
def run_pairs_backtest(
|
||||
symbol_a: str = "ETHUSDT",
|
||||
symbol_b: str = "BTCUSDT",
|
||||
interval: str = "5m",
|
||||
limit: int = 1000,
|
||||
window: int = 100,
|
||||
entry_threshold: float = 2.0,
|
||||
exit_threshold: float = 0.2,
|
||||
stop_loss_threshold: float = 3.5,
|
||||
taker_fee: float = 0.0004, # Fee standard Binance Futures VIP0 (0.04%)
|
||||
):
|
||||
"""Esegue il backtest vettorializzato/statale di Pairs Trading con calcolo di Sharpe, Max DD e PnL."""
|
||||
price_a = fetch_binance_klines(symbol_a, interval=interval, limit=limit)
|
||||
price_b = fetch_binance_klines(symbol_b, interval=interval, limit=limit)
|
||||
|
||||
df = pd.DataFrame({"price_a": price_a, "price_b": price_b}).dropna()
|
||||
df["log_a"] = np.log(df["price_a"])
|
||||
df["log_b"] = np.log(df["price_b"])
|
||||
|
||||
# 1. Regressione OLS per calcolare Beta (Hedge Ratio) e Alpha
|
||||
X = sm.add_constant(df["log_b"])
|
||||
model = sm.OLS(df["log_a"], X).fit()
|
||||
alpha = model.params["const"]
|
||||
beta = model.params["log_b"]
|
||||
|
||||
# 2. Spread e Z-Score
|
||||
df["spread"] = df["log_a"] - (beta * df["log_b"]) - alpha
|
||||
df["rolling_mean"] = df["spread"].rolling(window=window).mean()
|
||||
df["rolling_std"] = df["spread"].rolling(window=window).std()
|
||||
df["z_score"] = (df["spread"] - df["rolling_mean"]) / df["rolling_std"]
|
||||
|
||||
df.dropna(inplace=True)
|
||||
|
||||
# 3. Simulazione Macchina a Stati per le Posizioni
|
||||
# 1 = Long Spread (Long A, Short B)
|
||||
# -1 = Short Spread (Short A, Long B)
|
||||
# 0 = Flat / Fuori dal mercato
|
||||
positions = np.zeros(len(df))
|
||||
current_pos = 0
|
||||
|
||||
z_vals = df["z_score"].values
|
||||
|
||||
for i in range(1, len(df)):
|
||||
z = z_vals[i]
|
||||
|
||||
if current_pos == 0:
|
||||
if z <= -entry_threshold:
|
||||
current_pos = 1 # Entra Long Spread
|
||||
elif z >= entry_threshold:
|
||||
current_pos = -1 # Entra Short Spread
|
||||
elif current_pos == 1:
|
||||
# Exit Take Profit o Stop Loss
|
||||
if z >= -exit_threshold or z <= -stop_loss_threshold:
|
||||
current_pos = 0
|
||||
elif current_pos == -1:
|
||||
# Exit Take Profit o Stop Loss
|
||||
if z <= exit_threshold or z >= stop_loss_threshold:
|
||||
current_pos = 0
|
||||
|
||||
positions[i] = current_pos
|
||||
|
||||
df["position"] = positions
|
||||
|
||||
# 4. Calcolo dei Rendimenti
|
||||
# Rendimento percentuale dei singoli asset
|
||||
df["ret_a"] = df["price_a"].pct_change()
|
||||
df["ret_b"] = df["price_b"].pct_change()
|
||||
|
||||
# Normalizzazione del capitale pesata per Beta: Asset A ha peso 1, Asset B ha peso Beta
|
||||
total_weight = 1.0 + abs(beta)
|
||||
|
||||
# Rendimento dello spread combinato: (R_a - beta * R_b) / total_weight
|
||||
df["spread_ret"] = (df["ret_a"] - (beta * df["ret_b"])) / total_weight
|
||||
|
||||
# Applicazione della posizione del periodo precedente (Shift per evitare look-ahead bias)
|
||||
df["strat_ret"] = df["position"].shift(1) * df["spread_ret"]
|
||||
|
||||
# Deduzione Commissioni Trading ad ogni cambio di posizione
|
||||
trades = df["position"].diff().fillna(0) != 0
|
||||
# Ogni eseguito coinvolge due leg (A e B)
|
||||
df.loc[trades, "strat_ret"] -= taker_fee * 2.0
|
||||
|
||||
df["strat_ret"].fillna(0, inplace=True)
|
||||
|
||||
# 5. Metriche di Performance Metriche Finanziarie
|
||||
df["equity_curve"] = (1.0 + df["strat_ret"]).cumprod()
|
||||
|
||||
# PnL Cumulativo Totale
|
||||
total_pnl = (df["equity_curve"].iloc[-1] - 1.0) * 100
|
||||
|
||||
# Max Drawdown
|
||||
df["peak"] = df["equity_curve"].cummax()
|
||||
df["drawdown"] = (df["equity_curve"] - df["peak"]) / df["peak"]
|
||||
max_drawdown = df["drawdown"].min() * 100
|
||||
|
||||
# Sharpe Ratio Annualizzato
|
||||
# Calcolo del fattore di annualizzazione in base al timeframe scelto
|
||||
periods_per_day = {
|
||||
"1m": 1440,
|
||||
"5m": 288,
|
||||
"15m": 96,
|
||||
"1h": 24,
|
||||
}.get(interval, 288)
|
||||
annual_factor = np.sqrt(periods_per_day * 365)
|
||||
|
||||
mean_ret = df["strat_ret"].mean()
|
||||
std_ret = df["strat_ret"].std()
|
||||
|
||||
sharpe_ratio = (
|
||||
(mean_ret / std_ret) * annual_factor if std_ret > 0 else 0.0
|
||||
)
|
||||
|
||||
# Conteggio Operazioni
|
||||
num_trades = int((df["position"].diff().abs() > 0).sum() / 2)
|
||||
|
||||
# Stampa dei Risultati
|
||||
print("\n" + "=" * 50)
|
||||
print(f" RISULTATI BACKTEST STATARB ({symbol_a} / {symbol_b})")
|
||||
print("=" * 50)
|
||||
print(f" Periodo / Candele analizzate : {len(df)} su timeframe {interval}")
|
||||
print(f" Hedge Ratio (Beta) : {beta:.4f}")
|
||||
print(f" Operazioni Eseguite Totalità : {num_trades}")
|
||||
print(f" Net PnL Totale (con Fee) : {total_pnl:.2f}%")
|
||||
print(f" Sharpe Ratio Annualizzato : {sharpe_ratio:.2f}")
|
||||
print(f" Max Drawdown (Massimo Picco) : {max_drawdown:.2f}%")
|
||||
print("=" * 50 + "\n")
|
||||
|
||||
return df
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Esecuzione Backtest su 1000 candele da 5 minuti (~3.5 giorni di dati)
|
||||
df_results = run_pairs_backtest(
|
||||
symbol_a="ETHUSDT",
|
||||
symbol_b="BTCUSDT",
|
||||
interval="5m",
|
||||
limit=1000,
|
||||
entry_threshold=2.0,
|
||||
exit_threshold=0.2,
|
||||
stop_loss_threshold=3.5,
|
||||
)
|
||||
|
||||
```
|
||||
**Vincoli di progetto (hard rules):**
|
||||
1. **Paper-trading-first.** Nessun trading live finché non si superano gate di accettazione espliciti (Fasi 5-6).
|
||||
2. **Nessun ordine live senza un flag esplicito** `LIVE_TRADING=true` + conferma interattiva; default = dry-run.
|
||||
3. **Tutti gli output tabellari** (log, journal, report, trade blotter) devono essere **CSV con separatore `;`** e includere una colonna `motivazione`/`reasoning` che spiega in linguaggio naturale ogni decisione. Doppia scrittura: CSV su disco + tabella Supabase.
|
||||
4. **Costi realistici sempre.** Nessun backtest senza fee, spread, slippage e latenza modellati.
|
||||
5. **No look-ahead bias.** Ogni feature/label deve essere calcolabile solo con l'informazione disponibile a quell'istante.
|
||||
6. Segreti (API key) solo da variabili d'ambiente / secrets manager, mai hardcoded.
|
||||
|
||||
---
|
||||
|
||||
### Formula delle Metriche Calcolate
|
||||
## 1. Sintesi delle evidenze dalla ricerca (cosa funziona, cosa no)
|
||||
|
||||
* **Rendimento della Strategia ($R_t$):**
|
||||
**Cosa ha evidenza ragionevole (dopo i costi):**
|
||||
- **Statistical arbitrage / pairs trading su crypto cointegrate.** Fil & Kristoufek, *"Pairs Trading in Cryptocurrency Markets"* (IEEE Access, 2020) applicano i metodi distance e cointegration a **26 crypto liquide su Binance a frequenza 5-min, 1h e daily**, trovando mean-reversion intraday assente nei dati daily. Fischer, Krauss & Deinert (2019, *Journal of Risk and Financial Management* 12(1):31), con random forest su 40 coin (long/short dei top-3/flop-3, orizzonte 120 min), riportano testualmente: *"ranging from 18 June 2018 to 17 September 2018, and after more than 100,000 trades, we find statistically and economically significant returns of 7.1 bps per day, after transaction costs of 15 bps per half-turn."* La mean-reversion è più forte a frequenza intraday che daily.
|
||||
- **Funding-rate / cash-and-carry basis** su perpetual: delta-neutral (long spot + short perp quando il funding è positivo). Rendimenti realistici ~8-18% APR in condizioni normali per le top coin, spike >100% APR in fasi euforiche ma di breve durata; i rendimenti si comprimono verso zero all'aumentare del capitale che affluisce nella strategia.
|
||||
- **Meta-labeling** (López de Prado): un secondo modello che decide "scommetti o no" sul segnale primario riduce i falsi positivi e i costi di transazione. Ottimo rapporto sforzo/beneficio.
|
||||
- **Order flow / order book imbalance** ha relazione quasi-lineare con le variazioni di prezzo a brevissimo orizzonte (Cont-Kukanov-Stoikov). Utile come feature, ma l'edge vive a orizzonti di secondi.
|
||||
- **Sentiment (FinBERT/CryptoBERT)** aggiunge potere predittivo modesto ma reale in alcuni studi. Attenzione a non sovrastimarlo: Shobayo et al. (2024, arXiv:2412.06837 / *Big Data and Cognitive Computing* 8(11):143), su dataset NGX All-Share, misurano per FinBERT **accuracy 63,33%, precision 63,76%, ROC AUC 65,59%** — battuto da una logistic regression tunata (81,83% accuracy, 89,76% ROC AUC). Il sentiment è una feature, non una strategia.
|
||||
|
||||
$$R_t = \text{Posizione}_{t-1} \cdot \left( \frac{R_{A,t} - \beta R_{B,t}}{1 + \vert{}\beta\vert{}} \right) - \text{Fee}_t$$
|
||||
**Cosa NON funziona / dove il retail non compete:**
|
||||
- Previsione pura del prezzo con LSTM/Transformer come sorgente di alpha.
|
||||
- Latency arbitrage cross-exchange vs HFT (finestre 30-50 ms; vedi Alexander, *"Latency Arbitrage in Cryptocurrency Markets"*, SSRN 5143158: la latenza retail di 100-500 ms può sfruttare solo discrepanze che durano minuti, non secondi).
|
||||
- Deep RL come scatola nera "che impara a fare soldi": instabile, overfittato.
|
||||
- Qualsiasi backtest senza purged CV, deflated Sharpe e modello di costi.
|
||||
|
||||
**Failure modes da prevenire attivamente:** look-ahead bias, backtest overfitting (testare N configurazioni e scegliere la migliore), non-stationarity/regime change, survivorship bias, ignorare costi e latenza, p-hacking (contare TUTTI i test fatti, non solo quelli riusciti).
|
||||
|
||||
* **Sharpe Ratio ($S$):**
|
||||
Misura il rendimento corretto per il rischio rispetto alla deviazione standard del portafoglio:
|
||||
---
|
||||
|
||||
$$S = \frac{\mu_{R}}{\sigma_{R}} \times \sqrt{N}$$
|
||||
## 2. Architettura del sistema
|
||||
|
||||
Monorepo Python. Layout proposto:
|
||||
|
||||
```
|
||||
quantbot/
|
||||
config/ # yaml config, profili exchange, soglie di rischio
|
||||
data/
|
||||
ingestion/ # CCXT REST/WS, Binance Vision, eToro adapter, news/sentiment
|
||||
storage/ # Parquet + DuckDB (locale) + Supabase (Postgres) writer
|
||||
quality/ # validazione schema, gap, duplicati, sanity check
|
||||
features/
|
||||
bars.py # time/tick/volume/dollar/imbalance bars
|
||||
fracdiff.py # differenziazione frazionaria (FFD)
|
||||
microstructure.py# OFI, order book imbalance, microprice
|
||||
technical.py # returns, realized vol, indicatori, encoding ciclici tempo
|
||||
sentiment.py # FinBERT/CryptoBERT, LLM scoring, event detection
|
||||
timefeatures.py # sessioni, orari funding, scadenze, macro releases
|
||||
labeling/
|
||||
triple_barrier.py
|
||||
meta_labeling.py
|
||||
sample_weights.py# unicità campioni, pesi
|
||||
models/
|
||||
gbm.py # LightGBM/XGBoost/CatBoost
|
||||
sequence.py # LSTM/GRU/TCN/Transformer (opzionale)
|
||||
rl/ # FinRL/stable-baselines3 (opzionale, Fase 4)
|
||||
validation/
|
||||
cv.py # purged k-fold, embargo, CPCV
|
||||
metrics.py # Sharpe/Sortino/Calmar, PSR, DSR, PBO
|
||||
strategies/
|
||||
statarb.py # cointegrazione, OU, Kalman hedge ratio, z-score
|
||||
basis.py # funding-rate / cash-and-carry
|
||||
ml_signal.py # segnale ML -> probabilita -> meta-label
|
||||
sizing/
|
||||
kelly.py # Kelly e fractional Kelly
|
||||
vol_target.py # volatility targeting
|
||||
execution/
|
||||
broker_base.py # interfaccia astratta
|
||||
ccxt_adapter.py # exchange crypto
|
||||
etoro_adapter.py # eToro Public API
|
||||
cost_model.py # fee, spread, slippage, latenza
|
||||
risk/
|
||||
engine.py # limiti esposizione, max loss/day, kill-switch, VaR/CVaR
|
||||
journal/
|
||||
csv_writer.py # CSV ';' con colonna motivazione
|
||||
supabase_writer.py
|
||||
monitoring/
|
||||
drift.py # concept drift, champion/challenger
|
||||
alerts.py # Telegram/email
|
||||
backtest/
|
||||
engine.py # walk-forward, costi realistici
|
||||
orchestration/
|
||||
scheduler.py # retraining, loop
|
||||
tests/
|
||||
```
|
||||
|
||||
dove $N$ rappresenta il numero di periodi di trading in un anno solare (es. $288 \times 365$ per candele a 5m).
|
||||
* **Maximum Drawdown ($\text{Max DD}$):**
|
||||
Rappresenta la massima perdita percentuale registrata rispetto al picco precedente del capitale:
|
||||
**Flusso dati:** ingestion → quality → storage (Parquet/DuckDB + Supabase) → feature store → labeling → training → validation → signal → sizing → risk engine → execution (dry-run/paper/live) → journaling → monitoring → (retraining loop).
|
||||
|
||||
$$\text{Max DD} = \min_{t} \left( \frac{\text{Equity}_t - \text{Peak}_t}{\text{Peak}_t} \right)$$
|
||||
**Stack raccomandato:**
|
||||
- Dati: `pandas`/`polars`, `numpy`, `duckdb`, Parquet, Supabase Postgres.
|
||||
- Modelli: `scikit-learn`, `LightGBM` (core), `statsmodels`, `arch` (GARCH/vol), `pykalman`, `PyTorch` (solo se servono sequence models).
|
||||
- Backtest: **vectorbt** per lo screening rapido di molte configurazioni; **NautilusTrader** per la validazione event-driven realistica con parità backtest/live (stesso codice in backtest, paper e live). Il pattern raccomandato è "vectorbt → NautilusTrader": scoperta veloce del segnale, poi validazione microstruttura/esecuzione.
|
||||
- Framework bot alternativo integrato: **freqtrade + FreqAI** offre retraining adattivo con finestre scorrevoli, backtest che emula il retraining periodico e supporto a LightGBM/PyTorch/RL; utile come baseline pronta e come riferimento per il loop di retraining.
|
||||
- Exchange abstraction: **CCXT** (unifica Binance, Kraken, Bybit, Coinbase, OKX; supporta `fetch_ohlcv`, `fetch_order_book`, `fetch_funding_rate`, `fetch_funding_rate_history`, `fetchOpenInterestHistory`, `fetchLiquidations`). Ricorda `enableRateLimit: true`; `rateLimit` è in millisecondi e `pausa = rateLimit × cost_endpoint`.
|
||||
- Container: Docker; scheduling: `APScheduler`/cron; monitoring/alert: Telegram.
|
||||
|
||||
E
|
||||
**Scelta del venue di esecuzione (analisi onesta):**
|
||||
- **eToro Public API**: comodo perché l'utente lo usa già. Autenticazione con header `x-api-key` + `x-user-key` (chiavi long-lived), ambienti demo e live **separati** (chiavi distinte). Candele storiche via `GET /market-data/instruments/history/candles` (max 1000 barre, parametri `direction`/`interval`/`limit`). Rate limit ufficiali: **60 req/min** per i GET di dati (market data, portfolio, watchlist read) e **20 req/min** per endpoint che eseguono trade o query pesanti; su 429 rispetta `Retry-After` con backoff esponenziale. eToro ha lanciato "Agent Portfolios" con API key scoped, budget minimo $200, esplicitamente compatibili con agenti come Claude Code. **Limiti per il nostro scopo:** eToro NON espone order book depth, non consente short spot crypto né perp/funding, incorpora spread/mark-up e non dà dati di microstruttura → **inadatto ad arbitrage/relative-value e a strategie microstruttura**. Va bene per esecuzione direzionale sentiment-based e per copy/social.
|
||||
- **Exchange crypto via CCXT** (Binance/Kraken/Bybit/OKX/Coinbase): fee maker/taker ~0,1% o meno, order book completo, perp + funding, WebSocket a bassa latenza, dati storici gratuiti (Binance Vision, con il noto limite di ~1000 candele per richiesta REST da aggirare paginando via `since`). **Adatto** a statistical arbitrage, basis trade e feature di microstruttura. Kraken applica un contatore di rate limit incrementale (+1/+4 per classe di endpoint).
|
||||
- **Raccomandazione:** due adapter dietro l'interfaccia `broker_base`. Strategie statarb/basis → CCXT (valuta Kraken o Bybit per l'operatività italiana post-MiCA). Strategie sentiment direzionali → eToro. Il risk engine è comune a entrambi.
|
||||
|
||||
Lo script può essere eseguito in **qualsiasi ambiente Python 3.9+** e **non richiede l'importazione manuale di file dati**: le funzioni interne scaricano automaticamente le candele storiche interrogando le API REST pubbliche di Binance.
|
||||
---
|
||||
|
||||
**Dove eseguire lo script**
|
||||
## 3. Piano di lavoro a fasi
|
||||
|
||||
* **PC Locale (VS Code / PyCharm / Terminale):** La scelta ideale per lo sviluppo e i primi test. Salva il codice in un file `.py` (es. `backtest_statarb.py`) ed eseguilo direttamente con `python backtest_statarb.py`.
|
||||
* **Google Colab / Jupyter Notebook:** Utile per test e analisi al volo nel browser senza configurare un ambiente di sviluppo locale.
|
||||
* **Server H24 / Container Docker / VPS:** Necessario solo se trasformerai lo script in un bot live che monitora i mercati ed esegue ordini in modo continuativo.
|
||||
### FASE 0 — Ispezione e validazione dei CSV caricati (OBBLIGATORIA, prima di tutto)
|
||||
|
||||
**Su quali dati opera lo script**
|
||||
I file sono in `/mnt/user-data/uploads/`: `BTCUSD.csv, BTCEUR.csv, BTCUSDT.csv, ETHUSDT.csv, SOLUSDT.csv, AVAXUSDT.csv`. Sono verosimilmente candele OHLCV a 1 minuto, ma **schema, unità/timezone del timestamp, ordine e completezza sono IGNOTI**. NON assumere nulla.
|
||||
|
||||
* **Dati in tempo reale tramite API Binance:** Lo script interroga direttamente l'endpoint pubblico dei Futures Binance (`[fapi.binance.com/fapi/v1/klines](https://fapi.binance.com/fapi/v1/klines)`). Non servono API Key o Secret per leggere lo storico.
|
||||
* **Coppie consigliate per il test:**
|
||||
* `ETHUSDT` vs `BTCUSDT` (Grande cap, elevata stabilità della cointegrazione).
|
||||
* `SOLUSDT` vs `AVAXUSDT` (Maggiore volatilità e ampiezza delle deviazioni dello Z-Score).
|
||||
* `OPUSDT` vs `ARBUSDT` (Forte correlazione fondamentale tra Layer-2).
|
||||
Task:
|
||||
1. Carica ogni file rilevando automaticamente il separatore (prova `sep=';'`, fallback `sep=','`). Stampa `df.head()`, `df.dtypes`, numero righe.
|
||||
2. Identifica le colonne timestamp/OHLCV. Determina l'unità del timestamp (s / ms / ISO) e il timezone (assumi UTC ma **verificalo** confrontando i range dei prezzi con eventi noti).
|
||||
3. Ordina per timestamp; verifica monotonicità; conta e logga **duplicati** e **gap** (differenze ≠ 60 s).
|
||||
4. Sanity check prezzi: nessun valore ≤ 0, nessun high<low, nessun salto impossibile (es. |return| oltre soglia), volumi non negativi.
|
||||
5. **Allineamento cross-file:** calcola l'intersezione dei timestamp comuni tra i 6 file; produci un report di copertura.
|
||||
6. Output: `reports/fase0_data_quality.csv` (separatore `;`, colonna `motivazione` per ogni anomalia) + tabella riassuntiva in Supabase.
|
||||
|
||||
**Criterio di accettazione Fase 0:** report generato; almeno una finestra temporale comune ai file usati in ogni strategia; % di gap documentata; nessun prezzo insano non gestito.
|
||||
|
||||
* **Timeframe e profondità dei dati:**
|
||||
* Il parametro di default `limit=1000` con `interval="5m"` recupera le ultime **1000 candele da 5 minuti** (~3,5 giorni di dati).
|
||||
* Puoi variare i parametri all'interno della funzione (es. `interval="15m"` o `interval="1h"`) per estendere la finestra temporale dell'analisi a diverse settimane.
|
||||
### FASE 1 — Data pipeline + backtest baseline con costi realistici
|
||||
|
||||
Puoi trovare i files .csv che contengono i dati delle coppie valutare per eseguire backtest nella cartella C:\Users\alber\Downloads\Binance. Effettua test approfonditi provando diversi parametri e cercando di trovare i parametri migliori per il bot, che diventeranno i parametri di default
|
||||
Task:
|
||||
1. Standardizza i CSV in un formato canonico (UTC; colonne `timestamp, open, high, low, close, volume`); salva in Parquet + DuckDB.
|
||||
2. Implementa il **cost model** (`execution/cost_model.py`): fee maker/taker configurabili per exchange, spread bid-ask stimato, slippage funzione della size vs volume, latenza simulata.
|
||||
3. Backtest baseline **non-ML**: buy&hold e una regola banale (es. SMA crossover) su BTCUSDT, **con costi**, per validare l'engine.
|
||||
4. Implementa le metriche in `validation/metrics.py`: log-return, realized vol, Sharpe, Sortino, Calmar, max drawdown, PSR, DSR.
|
||||
|
||||
**Criterio di accettazione Fase 1:** engine di backtest riproducibile; la baseline con costi mostra performance realistica (tipicamente Sharpe basso/negativo netto costi — è il punto: dimostrare che l'engine non "regala" alpha).
|
||||
|
||||
### FASE 2 — Strategie statistiche cross-asset sui file forniti (il cuore per il retail)
|
||||
|
||||
Qui si realizza la richiesta esplicita dell'utente di "sfruttare piccole differenze naturali di prezzo". Usa i 6 file.
|
||||
|
||||
Idee da testare, ciascuna con verifica empirica delle leggi:
|
||||
1. **BTCUSD vs BTCUSDT** — basis USD/USDT ed episodi di depeg della stablecoin. Testa la stazionarietà dello spread.
|
||||
2. **BTCUSD vs BTCEUR** — EURUSD implicito vs EURUSD reale (se disponibile) → deviazioni triangolari.
|
||||
3. **ETH/SOL/AVAX vs BTC** — lead-lag a 1 minuto, beta hedging, cointegrazione, pairs/basket.
|
||||
|
||||
Procedura obbligatoria per ogni coppia/basket:
|
||||
1. Verifica che ogni serie sia I(1) (ADF/KPSS sui livelli e sulle differenze).
|
||||
2. **Test di cointegrazione Engle-Granger** e **Johansen** (formule in §5).
|
||||
3. Stima l'**hedge ratio** (OLS statico e **Kalman dinamico**).
|
||||
4. Modella lo spread come **Ornstein-Uhlenbeck**, stima la **half-life** (§5). Scarta le coppie con half-life troppo lunga (mean-reversion troppo lenta) o troppo corta (rumore/costi).
|
||||
5. Regola **z-score** entry/exit (§5), con soglie ottimizzate in walk-forward, MAI in-sample.
|
||||
6. Backtest **con costi realistici** e stima della **soglia di break-even costi** (quanti bps di costo azzerano il profitto — confronta con i 15 bps/half-turn di Fischer-Krauss-Deinert come ordine di grandezza).
|
||||
|
||||
**Criterio di accettazione Fase 2:** per almeno una coppia — spread stazionario (cointegrazione p<0.05), half-life ragionevole (minuti-ore), Sharpe out-of-sample dopo costi > 1.0, break-even cost superiore ai costi reali stimati, ≥100 trade.
|
||||
|
||||
### FASE 3 — Modelli ML con triple-barrier, meta-labeling e purged/combinatorial CV
|
||||
|
||||
Task:
|
||||
1. **Financial data structures** (López de Prado): oltre alle time bars, costruisci **dollar bars**/**volume bars**/**imbalance bars** (proprietà statistiche migliori).
|
||||
2. **Fractional differentiation (FFD):** rendi le serie stazionarie preservando memoria; trova il **d\*** minimo che passa l'ADF (§5).
|
||||
3. **Triple-barrier labeling** (§5): barriere profit-take/stop-loss scalate sulla volatilità locale + barriera verticale (tempo).
|
||||
4. **Meta-labeling:** modello primario per il lato (long/short), modello secondario binario per size/bet-or-not.
|
||||
5. **Sample weights** per l'unicità dei campioni (label sovrapposte).
|
||||
6. Modello: **LightGBM** su feature ingegnerizzate (§6). Sequence models solo se giustificati dai dati.
|
||||
7. **Validazione:** **purged k-fold con embargo** e **Combinatorial Purged CV (CPCV)** (§5). Calcola **PBO** e **Deflated Sharpe Ratio** contando TUTTE le configurazioni testate come N.
|
||||
8. **Bet sizing** dalle probabilità predette.
|
||||
|
||||
**Criterio di accettazione Fase 3:** DSR > 0.95 sul modello selezionato; PBO < 0.5; Sharpe out-of-sample dopo costi > 1.0; feature importance (MDI/MDA/SFI) sensata; nessun degrado catastrofico con piccole variazioni dei parametri (test di robustezza — un edge genuino sopravvive a un lookback 19/21 se il valore pubblicato è 20).
|
||||
|
||||
### FASE 4 — Reinforcement Learning (OPZIONALE, sperimentale)
|
||||
|
||||
Solo dopo che le Fasi 2-3 hanno prodotto qualcosa di solido. Usa FinRL/stable-baselines3 (PPO/DQN/SAC) in un env gym-style con **costi, liquidità e frizioni** incluse. Valuta con **molti training run / molti seed**: l'evidenza mostra che i risultati single-run sono inaffidabili e le policy sono instabili. Se non batte in modo stabile la baseline statarb dopo costi, **abbandona**.
|
||||
|
||||
**Criterio di accettazione Fase 4:** performance stabile su ≥15 seed, Sharpe netto costi > baseline Fase 2/3; altrimenti documenta il fallimento e fermati.
|
||||
|
||||
### FASE 5 — Paper trading con dati live
|
||||
|
||||
Task:
|
||||
1. Collega CCXT WebSocket (dati live) e l'adapter eToro demo.
|
||||
2. Esegui la/le strategie promosse in **modalità paper** (dry-run) per un periodo minimo definito.
|
||||
3. Journaling completo (§7) di ogni segnale, ordine simulato, fill, PnL, con colonna `motivazione`.
|
||||
4. Monitora il **drift** (le statistiche live divergono dal training?).
|
||||
|
||||
**Criterio di accettazione Fase 5:** ≥X settimane di paper trading; Sharpe live-paper coerente col backtest (degrado sotto soglia); nessun bug di esecuzione; drift sotto controllo.
|
||||
|
||||
### FASE 6 — Live con guardrail
|
||||
|
||||
Solo con `LIVE_TRADING=true` + conferma. Capitale iniziale minimo. Tutti i guardrail §8 attivi. Champion/challenger: il modello live è il champion; i challenger girano in paper; promozione automatica solo se battono il champion su metriche predefinite; **demozione automatica** e **kill-switch** se si superano le soglie di perdita.
|
||||
|
||||
---
|
||||
|
||||
## 4. Loop di apprendimento autonomo (come il bot "impara da solo" in sicurezza)
|
||||
|
||||
- **Retraining schedulato** (walk-forward): riaddestra su finestra scorrevole a cadenza fissa (modello FreqAI).
|
||||
- **Champion/challenger:** un nuovo candidato deve battere il champion in CPCV + paper prima della promozione.
|
||||
- **Drift monitor:** test statistici (es. KS / PSI su feature e residui); se drift → allerta ed eventuale retraining/demozione.
|
||||
- **Demozione automatica + kill-switch:** se il DSR live crolla o si supera la max-loss/day → stop.
|
||||
- **Estensibilità feature:** il feature store deve permettere di aggiungere nuovi parametri (nuova fonte news, nuovo on-chain metric) senza riscrivere il pipeline — vedi §6.
|
||||
|
||||
---
|
||||
|
||||
## 5. Formule e leggi da implementare (esplicite)
|
||||
|
||||
**Log-return:** `r_t = ln(P_t / P_{t-1})`.
|
||||
|
||||
**Realized volatility** (finestra n): `σ = sqrt(Σ r_t²)`; annualizza con `sqrt(periodi_per_anno)`.
|
||||
|
||||
**Z-score dello spread:** `z_t = (spread_t − media_mobile) / std_mobile`. Entry tipica `|z|>2`, exit vicino a `z=0`. Nota: l'esempio canonico QuantStart esce a `|z|=1`; implementazioni crypto pratiche escono spesso a `|z|≈0.5` con stop a `|z|>3-4`.
|
||||
|
||||
**Ornstein-Uhlenbeck / half-life** (Ernie Chan, *Algorithmic Trading*): regredisci via OLS `Δy_t = α + λ·y_{t-1} + ε_t`; `λ` è la velocità di mean-reversion (deve essere <0). `half-life = −ln(2)/λ`. Attenzione al segno: se parametrizzi `dy = θ(μ−y)dt` con θ>0, allora `half-life = ln(2)/θ` con θ = −λ (stesso risultato positivo). È la stessa regressione dell'ADF.
|
||||
|
||||
**Cointegrazione Engle-Granger (2 step):** (1) verifica I(1); (2) OLS `Y_t = α + β·X_t + u_t` → β = hedge ratio; (3) ADF sui residui `û_t`: `Δû_t = ρ·û_{t-1} + Σγ_p·Δû_{t-p} + ε_t`. H0 = no cointegrazione (unit root); rifiuta se ADF < valore critico residual-based (≈ −3.34 al 5%, NON le tabelle ADF standard). In Python: `statsmodels.tsa.stattools.coint(y,x)` (applica già i valori critici corretti) e `adfuller(spread, autolag="AIC")`.
|
||||
|
||||
**Johansen** (multivariato, permette più vettori di cointegrazione): VECM `Δy_t = Π·y_{t-1} + ΣΓ_i·Δy_{t-i} + ε_t`; rango di Π = n° relazioni di cointegrazione. Trace: `λ_trace(r) = −T·Σ_{i=r+1}^{K} ln(1−λ̂_i)`. Max-eigenvalue: `λ_max(r) = −T·ln(1−λ̂_{r+1})`. In Python: `statsmodels.tsa.vector_ar.vecm.coint_johansen` → `.lr1` (trace), `.lr2` (max-eig), `.cvt`/`.cvm` (valori critici), `.evec` (vettori di cointegrazione = hedge ratio).
|
||||
|
||||
**Kalman filter per hedge ratio dinamico:** stato `[β_t, α_t]` con random walk `β_t = β_{t-1} + w_t`. Predizione covarianza `R = P + Vw`. Predizione osservazione `y_est = x_t·β`. Varianza `Q = x_t·R·x_tᵀ + Ve`. Errore `e = y − y_est`. Kalman gain `K = R·x_tᵀ / Q`. Update: `β_t = β_t + K·e`, `P = R − K·x_t·R`. Usa `pykalman` o implementazione custom; `delta = Vw/(1−Vw)·I` controlla la velocità di variazione di β (valori tipici 1e-4 … 1e-5).
|
||||
|
||||
**Triple-barrier labeling:** per ogni evento definisci barriera superiore (take-profit, es. +k·σ), inferiore (stop-loss, −k·σ) e verticale (max holding time). Label = +1 se tocca prima la superiore, −1 se l'inferiore, 0 (o segno del return) alla scadenza.
|
||||
|
||||
**Fractional differentiation:** pesi con ricorrenza `ω_0=1`, `ω_k = −ω_{k-1}·(d−k+1)/k`; serie `(1−B)^d X_t = Σ ω_k·X_{t-k}`. Usa FFD a finestra fissa; scegli `d* = min{d : la serie passa l'ADF}` per preservare massima memoria (López de Prado riporta d* spesso ben < 1). È solo preprocessing: non crea alpha e non previene l'overfitting.
|
||||
|
||||
**Kelly criterion:** `f* = (b·p − q)/b`, con p prob. di vincita, q=1−p, b = rapporto vincita/perdita medi. Forma equivalente per traders: `Kelly% = W − (1−W)/R` (W win rate, R = avg win/avg loss). Forma continua: `f* = (μ − r)/σ²`. **Fractional Kelly:** usa 0.25-0.5·f\* (half-Kelly cattura ~75% della crescita con circa metà del drawdown; full Kelly ha ~1/3 di probabilità di un drawdown del 50%). Servono ≥50-100 trade prima che gli input siano affidabili.
|
||||
|
||||
**Volatility targeting:** `size = (σ_target / σ_realizzata)·capitale`, con cap di leva.
|
||||
|
||||
**Sharpe / Sortino / Calmar:** `Sharpe = (media_return − rf)/std`; Sortino usa solo la downside deviation; `Calmar = CAGR / |max_drawdown|`.
|
||||
|
||||
**Probabilistic Sharpe Ratio (PSR):** `PSR(SR*) = Φ( (SR_hat − SR*)·sqrt(T−1) / sqrt(1 − g3·SR_hat + ((g4−1)/4)·SR_hat²) )`, con g3 skewness, g4 kurtosis, T n° osservazioni.
|
||||
|
||||
**Deflated Sharpe Ratio (DSR):** `DSR = PSR(SR*)` dove il benchmark è lo Sharpe massimo atteso su N trial senza skill: `SR* = sqrt(V)·[ (1−γ)·Φ⁻¹(1−1/N) + γ·Φ⁻¹(1−1/(N·e)) ]`, con V varianza degli SR dei trial e γ ≈ 0.5772 (Eulero-Mascheroni). DSR>0.95 = evidenza forte contro il caso. Riduce al PSR quando N=1.
|
||||
|
||||
**Combinatorial Purged CV (CPCV):** partiziona la serie in N gruppi, scegli k gruppi come test per split. N° split = `C(N,k)`. N° di backtest path `φ = (k/N)·C(N,k)`. Esempio López de Prado: N=6, k=2 → 15 split, 5 path. **Purging:** rimuovi dal train le osservazioni le cui label si sovrappongono nel tempo a quelle del test. **Embargo:** elimina un buffer di osservazioni subito dopo ogni blocco di test (~1%). Implementazione: `skfolio.model_selection.CombinatorialPurgedCV(n_folds=N, n_test_folds=k, purged_size, embargo_size)`.
|
||||
|
||||
**Cost model / EV netto:** `EV_netto = p·avg_win − (1−p)·avg_loss − costi` (fee + spread + slippage + funding). Non entrare se `EV_netto ≤ 0`.
|
||||
|
||||
**Order book imbalance (queue imbalance):** `imbalance = (V_bid − V_ask)/(V_bid + V_ask) ∈ [−1,+1]`, con V ai best levels.
|
||||
|
||||
**Order Flow Imbalance (OFI, Cont-Kukanov-Stoikov 2014, *Journal of Financial Econometrics* 12(1):47-88):** `OFI = L^b − C^b − M^s − L^s + C^s + M^b` (net signed order flow ai best quotes: L=limit, C=cancellation, M=market; b=bid/buy, s=ask/sell). Impatto lineare: `ΔP_k = β·OFI_k + ε_k`, con `β ≈ 1/(2D)`, D = profondità media (empiricamente R² ≈ 67% su 50 titoli NYSE). **Distingui i due:** la ratio `(V_bid−V_ask)/(V_bid+V_ask)` è "order book imbalance", NON l'OFI canonico (che è il flusso netto firmato).
|
||||
|
||||
**Avellaneda-Stoikov (market making, 2008):** reservation price `r(s,q,t) = s − q·γ·σ²·(T−t)` (s mid, q inventory, γ risk aversion, σ vol, (T−t) tempo residuo). Spread ottimale totale `≈ γ·σ²·(T−t) + (2/γ)·ln(1 + γ/κ)`, con κ parametro di liquidità del book. Bid = `r − spread/2`, Ask = `r + spread/2`. Nota: il market making retail è difficile per adverse selection e latenza — trattalo come sperimentale.
|
||||
|
||||
**Funding-rate / cash-and-carry:** `APR ≈ funding_rate_per_periodo × periodi_anno` (es. 0.01%/8h × 1095 ≈ 11% APR). Delta-neutral: long spot + short perp quando funding>0. `Profitto = Σ funding − fee_entry − fee_exit − costo_capitale`. Rischi: rate flip (il funding diventa negativo), basis risk, custodial risk, compressione dei rate con l'afflusso di capitale.
|
||||
|
||||
---
|
||||
|
||||
## 6. Feature set (copre i parametri dell'utente)
|
||||
|
||||
L'utente ha citato "articolo, orario dell'asta, prezzo attuale, puntate degli utenti, latenza rete". Interpretazione dichiarata come feature set estensibile per un bot di trading:
|
||||
- **"articolo" → news/sentiment:** FinBERT/CryptoBERT su titoli/articoli; LLM scoring; event detection. Fonti: RSS, CryptoPanic, GDELT, Reddit/X API. Feature: sentiment score, volume di news, sorpresa vs baseline. (Ricorda l'accuratezza modesta del sentiment — vedi §1: usalo come feature, non come segnale principale.)
|
||||
- **"orario dell'asta" → tempo/sessione:** encoding ciclici (sin/cos di ora e giorno-settimana), sessioni (Asia/EU/US), **orari di settlement funding (00:00/08:00/16:00 UTC)**, scadenze opzioni, release macro schedulate (CPI, FOMC).
|
||||
- **"prezzo attuale" → prezzo e derivati:** log-return multi-orizzonte, realized vol, indicatori tecnici come input (non come regole), microprice.
|
||||
- **"puntate degli utenti" → order flow / crowd positioning:** order book depth & imbalance, trade flow / CVD, **open interest**, **funding rate**, long/short ratio, **liquidazioni**, exchange net flows. Via CCXT (`fetchOpenInterestHistory`, `fetchFundingRateHistory`, `fetchLiquidations`).
|
||||
- **"latenza rete" → latenza/API:** misura il round-trip time verso l'exchange, l'order-ack time e il timestamp sync (NTP); usala per decidere la **feasibility** della strategia (se la finestra di arbitraggio < latenza → non entrare). Loggala sia come feature sia come gate di esecuzione.
|
||||
|
||||
Il feature store deve essere **estensibile**: aggiungere una feature = aggiungere una funzione che ritorna una serie allineata al timestamp, registrata in un registry; nessuna riscrittura del pipeline.
|
||||
|
||||
---
|
||||
|
||||
## 7. Journaling (CSV `;` con colonna motivazione)
|
||||
|
||||
Ogni tabella prodotta dal bot è CSV con `sep=';'` e almeno queste colonne dove pertinente: `timestamp;asset;evento;azione;prezzo;size;prob_predetta;z_score;funding;costi_stimati;pnl;motivazione`. La colonna `motivazione` spiega in italiano perché l'azione è stata presa (es. *"z=2.3 > soglia 2.0 su coppia ETH/BTC cointegrata p=0.02, half-life 45min, EV netto costi positivo"*). Doppia scrittura: file in `journal/` + tabella Supabase per query/reporting. Report di backtest, risultati CV, log di drift e trade blotter seguono tutti questa convenzione.
|
||||
|
||||
---
|
||||
|
||||
## 8. Guardrail e sicurezza
|
||||
|
||||
- API key SOLO da env / secrets manager; chiavi separate demo/live; mai in git.
|
||||
- `LIVE_TRADING` default false; ordine live richiede flag + conferma interattiva.
|
||||
- **Max loss per day** (hard stop), **max drawdown** limite, **kill-switch** globale.
|
||||
- Limiti di **esposizione** per asset e per correlazione; limiti di leva.
|
||||
- **VaR/CVaR** monitorati; risk of ruin controllato dal fractional Kelly.
|
||||
- Rate limiting rispettato (backoff esponenziale su 429; eToro 60/20 req-min; CCXT `enableRateLimit: true`).
|
||||
- Logging completo di ogni ordine ed errore; idempotenza sugli ordini; riconciliazione posizioni all'avvio.
|
||||
- Test: unit test sulle formule (half-life, Kelly, z-score, cost model, OU); integration test sugli adapter in demo.
|
||||
|
||||
---
|
||||
|
||||
## 9. Riferimenti e fonti
|
||||
|
||||
- López de Prado, *Advances in Financial Machine Learning* (Wiley, 2018) — triple-barrier, meta-labeling, fracdiff, purged/combinatorial CV, sample weights, DSR, PBO, bet sizing.
|
||||
- Bailey & López de Prado, "The Deflated Sharpe Ratio" (*Journal of Portfolio Management*, 2014); "The Probability of Backtest Overfitting" (*Journal of Computational Finance*, 2017); "Pseudo-mathematics and financial charlatanism" (*Notices of the AMS*, 2014).
|
||||
- McLean & Pontiff, "Does Academic Research Destroy Stock Return Predictability?" (*Journal of Finance* 71(1):5-32, 2016).
|
||||
- Harvey, Liu & Zhu, "…and the Cross-Section of Expected Returns" (*Review of Financial Studies* 29(1):5-68, 2016).
|
||||
- Fil & Kristoufek, "Pairs Trading in Cryptocurrency Markets" (*IEEE Access*, 2020); Fischer, Krauss & Deinert, "Statistical Arbitrage in Cryptocurrency Markets" (*Journal of Risk and Financial Management* 12(1):31, 2019).
|
||||
- Cont, Kukanov & Stoikov, "The Price Impact of Order Book Events" (*Journal of Financial Econometrics* 12(1):47-88, 2014; arXiv:1011.6402).
|
||||
- Avellaneda & Stoikov, "High-frequency trading in a limit order book" (2008).
|
||||
- Ernie Chan, *Algorithmic Trading: Winning Strategies and Their Rationale*.
|
||||
- Shobayo et al., "Innovative Sentiment Analysis… FinBERT, GPT-4 and Logistic Regression" (arXiv:2412.06837 / *Big Data and Cognitive Computing* 8(11):143, 2024).
|
||||
- Alexander, "Latency Arbitrage in Cryptocurrency Markets" (SSRN 5143158).
|
||||
- FinRL / FinRL-Meta (Liu et al., arXiv:2011.09607, 2111.09395, 2112.06753) e le critiche su simulation-to-reality gap e instabilità (multiplicity-aware evaluation).
|
||||
- Docs: CCXT (docs.ccxt.com), Freqtrade/FreqAI (docs.freqtrade.io), eToro API (api-portal.etoro.com, builders.etoro.com), Binance Vision, statsmodels, arbitragelab (Hudson & Thames), skfolio.
|
||||
|
||||
---
|
||||
|
||||
## 10. Note EU/Italia (non consulenza)
|
||||
|
||||
- **Tassazione crypto Italia 2026:** l'imposta sostitutiva sulle plusvalenze da cripto-attività (redditi diversi ex art. 67 c.1 lett. c-sexies TUIR) sale al **33% dal 1° gennaio 2026** (era 26% fino al 2025), come fissato dalla **Legge 30 dicembre 2024, n. 207**; senza franchigia, si applica al momento del realizzo indipendentemente dalla data di acquisto. Eccezione al **26%** solo per gli **E-Money Token (EMT) in euro conformi MiCAR**, introdotta dalla **Legge 30 dicembre 2025, n. 199, art. 1, comma 28**. Il journaling deve registrare ogni realizzo ai fini del quadro RT e del monitoraggio RW.
|
||||
- **MiCA/KYC:** usa exchange conformi; verifica l'operatività retail in Italia al momento (alcuni servizi retail italiani di Binance sono stati ridotti/cessati). Scegli l'exchange dell'adapter CCXT di conseguenza.
|
||||
- Il bot NON fornisce consulenza finanziaria; è uno strumento di ricerca personale.
|
||||
|
||||
---
|
||||
|
||||
**Inizia dalla FASE 0. Non passare alla fase successiva finché i criteri di accettazione della fase corrente non sono soddisfatti e documentati in un report CSV `;` con colonna `motivazione`. Ricorda i tre pilastri onesti: il problema è la validazione, non la previsione; conta TUTTI i test che fai; nessun euro reale a rischio prima dei gate di paper trading.**
|
||||
|
||||
@@ -39,9 +39,11 @@ dotnet msbuild build/Release.proj -t:Verifica
|
||||
dotnet msbuild build/Release.proj -t:Pacchetto
|
||||
dotnet msbuild build/Release.proj -t:Rilascia -p:Versione=3.3.0 -p:Note="Cosa cambia"
|
||||
|
||||
# Il backtest vuole un file di dati
|
||||
dotnet msbuild build/Release.proj -t:Backtest -p:Dati="C:\dati\btcusd.csv"
|
||||
dotnet msbuild build/Release.proj -t:Backtest -p:Dati="..." -p:Comando=frequency
|
||||
# Il backtest vuole la cartella con un CSV a un minuto per simbolo
|
||||
dotnet msbuild build/Release.proj -t:Backtest -p:Dati="C:\Users\alber\Downloads\Binance"
|
||||
dotnet msbuild build/Release.proj -t:Backtest -p:Dati="..." -p:Comando=basket
|
||||
dotnet msbuild build/Release.proj -t:Backtest -p:Dati="..." -p:Comando=inspect
|
||||
dotnet msbuild build/Release.proj -t:Backtest -p:Dati="..." -p:Comando=train -p:Extra="--a ETHUSDT --b BTCUSDT --promote"
|
||||
```
|
||||
|
||||
| Proprietà | Predefinito | A cosa serve |
|
||||
@@ -52,9 +54,20 @@ dotnet msbuild build/Release.proj -t:Backtest -p:Dati="..." -p:Comando=frequency
|
||||
| `Sovrascrivi` | `false` | Sostituisci una release Gitea con lo stesso tag. |
|
||||
| `Bozza` | `false` | Crea la release come bozza. |
|
||||
| `ConsentiModifiche` | `false` | Tagga anche con l'albero sporco. Serve saperlo. |
|
||||
| `Dati` | — | Il CSV da rigiocare. Obbligatorio per `Backtest`. |
|
||||
| `Comando` | `split` | `split`, `walk`, `frequency`, `costs`, `sweep`, `explore`. |
|
||||
| `Barre` | `1d` | Ampiezza delle barre nel backtest. |
|
||||
| `Dati` | — | La cartella con i CSV a un minuto (`<SIMBOLO>.csv`). Obbligatoria per `Backtest`. |
|
||||
| `Comando` | `pairs` | Rigiocata: `run`, `pairs`, `explore`, `sweep`, `confirm`, `basket`. Ricerca (le fasi della guida): `inspect`, `import`, `baseline`, `cointegration`, `dataset`, `train`, `rl`, `status`. |
|
||||
| `Barre` | `15m` | Timeframe su cui ripiegare le barre da un minuto. |
|
||||
| `Extra` | vuoto | Altre opzioni passate allo strumento così come sono, es. `--a ETHUSDT --b BTCUSDT --promote`. |
|
||||
|
||||
## La ricerca e il meta-modello
|
||||
|
||||
`backtest train` addestra il meta-modello (GBDT nativo, validazione CPCV, PBO e DSR
|
||||
contando tutte le configurazioni provate) e lo salva nel database locale del bot,
|
||||
`%ProgramData%\Encelado\encelado.db`. Con `--promote` lo rende **campione** della
|
||||
coppia, ma solo se supera i criteri della Fase 3; il bot lo carica alla prima
|
||||
ricalibrazione utile e da quel momento può rifiutare un ingresso o ridurne la size.
|
||||
Ogni tabella prodotta dalla ricerca è un CSV `;` con una colonna `motivazione`, nella
|
||||
cartella `ricerca` accanto allo strumento oppure dove indica `--out`.
|
||||
|
||||
## Chi chiede la versione
|
||||
|
||||
|
||||
@@ -444,7 +444,7 @@
|
||||
</PropertyGroup>
|
||||
|
||||
<Error Condition="'$(Dati)' == ''"
|
||||
Text="Serve la cartella con i CSV: -p:Dati="C:\percorso\Binance", con un file <SIMBOLO>.csv per gamba.%0AComandi disponibili in -p:Comando= : run, pairs, explore, sweep, confirm, basket." />
|
||||
Text="Serve la cartella con i CSV: -p:Dati="C:\percorso\Binance", con un file <SIMBOLO>.csv per gamba.%0AComandi disponibili in -p:Comando= : run, pairs, explore, sweep, confirm, basket; ricerca: inspect, import, baseline, cointegration, dataset, train, rl, status (altre opzioni in -p:Extra=)." />
|
||||
|
||||
<Error Condition="!Exists('$(Dati)')" Text="Cartella dati non trovata: $(Dati)" />
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"_comment": "Encelado — arbitraggio statistico su coppie cointegrate, Binance Futures USDⓈ-M. Il bot non prende posizione sulla direzione del mercato: è long una gamba e short l'altra nel rapporto che il test di cointegrazione produce, e scommette solo sul fatto che la distanza fra le due si richiuda.",
|
||||
|
||||
"_misura": "IMPORTANTE — che cosa dice il backtest, e perché dryRun parte ATTIVO. Misurato su 6,6 anni di barre da un minuto (2020-2026) di ETHUSDT, BTCUSDT, SOLUSDT e AVAXUSDT, ripiegate su 5m, 15m, 30m, 1h e 4h, con commissione taker 4 bps più 1 bp di slippage per lato. Il risultato: su 5m nessuna combinazione di soglie supera nemmeno i filtri di taratura; su 15m e 1h la ricerca a griglia trova combinazioni che rendono in taratura e in verifica, ma NESSUNA delle prime dieci resta positiva sulla terza fetta, quella che non entra mai nella scelta. Con i valori qui sotto, sulle quattro coppie misurabili, la conferma dà tre risultati negativi e uno positivo del +0,40% su una sola operazione. La ragione di fondo è che il test di cointegrazione passa solo l'8-13% del tempo, quindi in sei anni una coppia produce qualche decina di operazioni: troppo poche per distinguere un margine reale da una serie fortunata. Questi valori sono i meglio supportati fra quelli provati, non una strategia dimostrata. Falla girare in dry-run finché non hai visto coi tuoi occhi come si comporta sul tuo conto. Lo strumento è in tools/Encelado.Backtest: 'backtest basket --data <cartella>' rifa la scelta da capo sui tuoi dati.",
|
||||
"_misura": "IMPORTANTE — che cosa dice il backtest, e perché dryRun parte ATTIVO. Misurato su 6,6 anni di barre da un minuto (2020-2026) di ETHUSDT, BTCUSDT, SOLUSDT e AVAXUSDT, ripiegate su 5m, 15m, 30m, 1h e 4h, con commissione taker 4 bps più 1 bp di slippage per lato. Il risultato: su 5m nessuna combinazione di soglie supera nemmeno i filtri di taratura; su 15m e 1h la ricerca a griglia trova combinazioni che rendono in taratura e in verifica, ma NESSUNA delle prime dieci resta positiva sulla terza fetta, quella che non entra mai nella scelta. Con i valori qui sotto, sulle quattro coppie misurabili, la conferma dà tre risultati negativi e uno positivo del +0,40% su una sola operazione. La ragione di fondo è che il test di cointegrazione passa solo l'8-13% del tempo, quindi in sei anni una coppia produce qualche decina di operazioni: troppo poche per distinguere un margine reale da una serie fortunata. Questi valori sono i meglio supportati fra quelli provati, non una strategia dimostrata: per questo la configurazione di fabbrica punta alla TESTNET, dove gli ordini partono davvero ma i soldi sono finti. Lo strumento è in tools/Encelado.Backtest: 'backtest basket --data <cartella>' rifà la scelta da capo sui tuoi dati.",
|
||||
|
||||
"binance": {
|
||||
"_testnet": "true = testnet (denaro finto, stesse API). Metterlo a false opera con denaro reale. Le chiavi dei due ambienti sono diverse e vengono salvate separatamente.",
|
||||
@@ -39,8 +39,8 @@
|
||||
"_postOnly": "Solo maker (GTX). Fa risparmiare la commissione ma un ordine che non esegue lascia una gamba scoperta, che costa molto di più. Lasciare false se non hai misurato il tuo tasso di riempimento.",
|
||||
"postOnlyEntries": false,
|
||||
|
||||
"_dryRun": "ATTIVO DI FABBRICA, e la ragione è nella nota _misura qui sopra. Calcola e registra tutto, non invia nessun ordine. Toglilo solo dopo aver visto in Stato che le tue coppie passano davvero il test di cointegrazione.",
|
||||
"dryRun": true,
|
||||
"_dryRun": "Calcola e registra tutto, non invia nessun ordine. Spento di fabbrica: in testnet i soldi sono finti e la prova più vicina alla produzione è quella con gli ordini veri sul conto di prova — eseguiti, rifiutati, parzialmente riempiti, con le stesse regole di lotto e di margine. Accendilo per osservare una configurazione nuova senza toccare nemmeno il conto di prova. Sul conto reale resta comunque la conferma esplicita all'avvio.",
|
||||
"dryRun": false,
|
||||
|
||||
"_reconcile": "Ogni quanto il bot ricontrolla conto, posizioni e ordini contro Binance. È anche quando recupera le barre che lo stream non ha consegnato e chiude le coppie rimaste con una gamba sola.",
|
||||
"reconcileSeconds": 20,
|
||||
@@ -96,7 +96,7 @@
|
||||
"_level": "trace | debug | info | warn | error | none. Ogni rifiuto che ferma un ordine viene scritto a 'info' o sopra, quindi 'debug' serve per il flusso dati, non per capire perché il bot non ha operato.",
|
||||
"level": "info",
|
||||
|
||||
"_directory": "Dove salvare tutti gli output. Relativa all'eseguibile, oppure un percorso assoluto. Si cambia anche da Impostazioni.",
|
||||
"_directory": "Dove salvare tutti gli output. Relativa alla cartella della configurazione (Documenti\\Encelado), oppure un percorso assoluto. Si cambia anche da Impostazioni.",
|
||||
"directory": "logs",
|
||||
|
||||
"console": false,
|
||||
@@ -104,8 +104,8 @@
|
||||
"maxFileSizeMb": 32,
|
||||
"maxFiles": 10,
|
||||
|
||||
"_analysis": "decisions.csv ha una riga per ogni barra valutata con spread, z-score e calibrazione; executions.csv una riga per ogni segnale arrivato agli ordini, con il verdetto del risk engine. Si uniscono su decisionId.",
|
||||
"tradeJournal": "trades.jsonl",
|
||||
"_analysis": "Tutte le tabelle usano il separatore ; e hanno una colonna motivazione che spiega in italiano il perché della riga. decisions.csv ha una riga per ogni barra valutata con spread, z-score e calibrazione; executions.csv una riga per ogni segnale arrivato agli ordini, con il verdetto del risk engine; trades.csv una riga per ogni evento d'ordine. decisions ed executions si uniscono su decisionId.",
|
||||
"tradeJournal": "trades.csv",
|
||||
"decisionLog": "decisions.csv",
|
||||
"executionLog": "executions.csv",
|
||||
|
||||
@@ -114,6 +114,26 @@
|
||||
"bufferedLines": 5000
|
||||
},
|
||||
|
||||
"storage": {
|
||||
"_note": "Il database locale (SQLite): barre canoniche, dataset, modelli, validazioni, journal e drift. Vuoto = %ProgramData%\\Encelado\\encelado.db, una cartella di sistema separata sia da Documenti (dove sta la configurazione) sia dai dati utente (dove stanno le chiavi).",
|
||||
"enabled": true,
|
||||
"databasePath": ""
|
||||
},
|
||||
|
||||
"ml": {
|
||||
"_note": "Il meta-modello (meta-labeling): un classificatore addestrato sugli esiti dei segnali passati che dice se un ingresso proposto dalla regola statistica ha probabilità di ripagare i costi. Non propone mai operazioni: può solo rifiutarne una o ridurne la size. Si addestra con lo strumento tools/Encelado.Backtest (comandi dataset, train) e vive nel database locale come 'campione' della coppia; senza campione queste voci non fanno nulla.",
|
||||
"enabled": true,
|
||||
"_minProbability": "Sotto questa probabilità predetta l'ingresso viene rifiutato. 0.55 significa: opero solo quando il modello vede un vantaggio, anche piccolo, rispetto a lanciare una moneta.",
|
||||
"minProbability": 0.55,
|
||||
"_sizeByProbability": "Con true la size è proporzionale alla convinzione del modello (2·Φ(z)−1, López de Prado), fra minSizeFraction e 1; con false è tutto o niente.",
|
||||
"sizeByProbability": true,
|
||||
"minSizeFraction": 0.25,
|
||||
"_drift": "Ogni driftCheckBars segnali valutati le feature live vengono confrontate con quelle di addestramento (PSI e Kolmogorov-Smirnov). Una feature con PSI ≥ driftPsiAlert è 'in deriva'; con driftAlertsToSuspend feature in deriva il modello viene sospeso fino alla ricalibrazione successiva e il bot torna alla sola regola statistica. 0 = mai sospendere, solo registrare.",
|
||||
"driftCheckBars": 50,
|
||||
"driftPsiAlert": 0.25,
|
||||
"driftAlertsToSuspend": 3
|
||||
},
|
||||
|
||||
"_pairs": "Il paniere della guida. Ogni coppia si legge come ln(A) − β·ln(B): A è la gamba su cui si misura lo spread, B quella di copertura. Il test di cointegrazione decide da solo, a ogni ricalibrazione, se una coppia è operabile: quelle che non passano restano in elenco e non vengono aperte. Sui dati disponibili passano il test solo l'8-17% del tempo, quindi il bot resta fermo a lungo — è il comportamento previsto.",
|
||||
|
||||
"_parametri": "I valori qui sotto vengono dalla ricerca a griglia (tools/Encelado.Backtest, comando 'basket'), non dalla guida. Le differenze rispetto alla guida sono tre e sono tutte misurate: zWindow 1500 invece di 100, entryZ 2.5 invece di 2.0, stopZ 6.0 invece di 3.5. La prima è la più importante: con una finestra vicina all'emivita del rientro (30-40 barre) la media mobile insegue lo scostamento e lo assorbe, e il margine sparisce prima ancora delle commissioni.",
|
||||
|
||||
@@ -9,4 +9,8 @@
|
||||
<ProjectReference Include="..\Encelado.Core\Encelado.Core.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<InternalsVisibleTo Include="Encelado.Tests" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
||||
@@ -111,12 +111,23 @@ public sealed class BinanceFuturesClient : IDisposable
|
||||
// Market data
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/// <summary>Most klines one request may return.</summary>
|
||||
public const int MaxKlinesPerRequest = 1500;
|
||||
|
||||
/// <summary>
|
||||
/// Historical klines for one symbol, oldest first.
|
||||
/// Historical klines for one symbol, oldest first: the last <paramref name="limit"/>
|
||||
/// <b>closed</b> bars, or everything from <paramref name="startUtc"/> onwards.
|
||||
/// <para>
|
||||
/// The last kline Binance returns is the one still forming. It is dropped here
|
||||
/// rather than at every call site: a strategy that decides on a partial close and
|
||||
/// then decides again when it finishes is a strategy that trades twice on one bar.
|
||||
/// The last kline Binance returns is the one still forming. It is dropped here rather
|
||||
/// than at every call site: a strategy that decides on a partial close and then
|
||||
/// decides again when it finishes is a strategy that trades twice on one bar.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Binance serves at most 1500 klines per request, and a z-score window of 1500 bars
|
||||
/// needs more closed bars than that. Asking for 1500 used to yield 1499 closed bars,
|
||||
/// leaving every strategy one bar short of ready after each calibration — for up to a
|
||||
/// full bar interval. The history is therefore paged: consecutive requests advance a
|
||||
/// cursor from the earliest bar needed until the present.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public async Task<List<Bar>> GetKlinesAsync(
|
||||
@@ -126,33 +137,51 @@ public sealed class BinanceFuturesClient : IDisposable
|
||||
DateTime? startUtc,
|
||||
CancellationToken ct)
|
||||
{
|
||||
StringBuilder query = new(96);
|
||||
query.Append("symbol=").Append(symbol);
|
||||
query.Append("&interval=").Append(timeFrame.ToBinance());
|
||||
query.Append("&limit=").Append(Math.Clamp(limit, 1, 1500).ToString(CultureInfo.InvariantCulture));
|
||||
|
||||
if (startUtc is { } start)
|
||||
{
|
||||
long ms = new DateTimeOffset(DateTime.SpecifyKind(start, DateTimeKind.Utc)).ToUnixTimeMilliseconds();
|
||||
query.Append("&startTime=").Append(ms.ToString(CultureInfo.InvariantCulture));
|
||||
}
|
||||
|
||||
using JsonDocument doc = await _http.GetAsync("fapi/v1/klines", query.ToString(), ct)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
List<Bar> bars = [];
|
||||
if (doc.RootElement.ValueKind != JsonValueKind.Array)
|
||||
{
|
||||
return bars;
|
||||
}
|
||||
|
||||
long bucketMs = timeFrame.Seconds() * 1000L;
|
||||
long nowMs = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds();
|
||||
long currentBucketOpen = nowMs - (nowMs % bucketMs);
|
||||
|
||||
int wanted = Math.Clamp(limit, 1, 50_000);
|
||||
|
||||
// Two extra buckets of slack: one for the forming bar that is dropped, one for
|
||||
// the boundary the clock may have just crossed.
|
||||
long cursor = startUtc is { } start
|
||||
? new DateTimeOffset(DateTime.SpecifyKind(start, DateTimeKind.Utc)).ToUnixTimeMilliseconds()
|
||||
: currentBucketOpen - ((wanted + 2) * bucketMs);
|
||||
|
||||
List<Bar> bars = new(wanted + 2);
|
||||
|
||||
while (cursor < currentBucketOpen)
|
||||
{
|
||||
int batch = Math.Min(MaxKlinesPerRequest, wanted + 2 - bars.Count);
|
||||
if (batch <= 0)
|
||||
{
|
||||
break;
|
||||
}
|
||||
|
||||
StringBuilder query = new(96);
|
||||
query.Append("symbol=").Append(symbol);
|
||||
query.Append("&interval=").Append(timeFrame.ToBinance());
|
||||
query.Append("&startTime=").Append(cursor.ToString(CultureInfo.InvariantCulture));
|
||||
query.Append("&limit=").Append(batch.ToString(CultureInfo.InvariantCulture));
|
||||
|
||||
using JsonDocument doc = await _http.GetAsync("fapi/v1/klines", query.ToString(), ct)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
if (doc.RootElement.ValueKind != JsonValueKind.Array)
|
||||
{
|
||||
break;
|
||||
}
|
||||
|
||||
int received = 0;
|
||||
long lastOpen = cursor;
|
||||
|
||||
foreach (JsonElement k in doc.RootElement.EnumerateArray())
|
||||
{
|
||||
received++;
|
||||
long openMs = k.ArrayInt64(0);
|
||||
lastOpen = openMs;
|
||||
|
||||
if (openMs >= currentBucketOpen)
|
||||
{
|
||||
continue;
|
||||
@@ -161,6 +190,27 @@ public sealed class BinanceFuturesClient : IDisposable
|
||||
bars.Add(ParseKline(k));
|
||||
}
|
||||
|
||||
// A short page means the exchange has nothing further back; a full one means
|
||||
// there may be more, and the cursor moves past the last bar seen.
|
||||
if (received == 0 || lastOpen + bucketMs <= cursor)
|
||||
{
|
||||
break;
|
||||
}
|
||||
|
||||
cursor = lastOpen + bucketMs;
|
||||
|
||||
if (received < batch)
|
||||
{
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Keep only the most recent `wanted` when the slack over-fetched.
|
||||
if (startUtc is null && bars.Count > wanted)
|
||||
{
|
||||
bars.RemoveRange(0, bars.Count - wanted);
|
||||
}
|
||||
|
||||
return bars;
|
||||
}
|
||||
|
||||
|
||||
@@ -131,6 +131,21 @@ public sealed class MarketDataStream : WebSocketChannel
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
Decode(payload);
|
||||
}
|
||||
catch (JsonException ex)
|
||||
{
|
||||
// A truncated or malformed frame costs one message, never the connection.
|
||||
// The receive loop would swallow this too, but the decoder owning its own
|
||||
// failure mode is what lets it be tested without a socket.
|
||||
Log($"[{Name}] frame non decodificabile ({payload.Length} byte): {ex.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
private void Decode(ReadOnlySpan<byte> payload)
|
||||
{
|
||||
// Combined-stream frames wrap the event: {"stream":"…","data":{…}}. Descend to
|
||||
// "data" and decode from there; a bare event (single-stream socket) decodes as is.
|
||||
Utf8JsonReader reader = new(payload, new JsonReaderOptions { AllowTrailingCommas = true });
|
||||
@@ -242,84 +257,125 @@ public sealed class MarketDataStream : WebSocketChannel
|
||||
/// deciding on a partial close — the strategy would then decide again on the real one,
|
||||
/// which is how a single bar becomes two trades.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The bar's fields live inside a nested <c>k</c> object, and the first version of
|
||||
/// this decoder skipped that object along with every other unwanted value. The bot
|
||||
/// ran for days receiving millions of book updates and not one closed bar, and only
|
||||
/// kept deciding because the REST backstop refetched the bars it never saw. The
|
||||
/// nested object is now descended into explicitly, and a regression test feeds the
|
||||
/// decoder a verbatim frame so this cannot happen quietly again.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private void DecodeKline(ref Utf8JsonReader reader)
|
||||
{
|
||||
int symbolId = -1;
|
||||
long openTime = 0;
|
||||
double open = 0, high = 0, low = 0, close = 0, volume = 0, quoteVolume = 0, takerBuy = 0;
|
||||
int trades = 0;
|
||||
bool closed = false;
|
||||
int depth = 0;
|
||||
KlineFields k = default;
|
||||
|
||||
// Positioned on the event's StartObject. Every property is matched by name so
|
||||
// ordering never matters; unwanted values are skipped whole, nested or not.
|
||||
while (reader.Read())
|
||||
{
|
||||
if (reader.TokenType is JsonTokenType.StartObject or JsonTokenType.StartArray)
|
||||
{
|
||||
depth++;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (reader.TokenType is JsonTokenType.EndObject or JsonTokenType.EndArray)
|
||||
{
|
||||
if (depth == 0)
|
||||
if (reader.TokenType == JsonTokenType.EndObject)
|
||||
{
|
||||
break;
|
||||
}
|
||||
|
||||
depth--;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (reader.TokenType != JsonTokenType.PropertyName)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
// Symbol at the top level, everything else inside the "k" object. Both are
|
||||
// matched by name so ordering never matters.
|
||||
if (depth == 0 && reader.ValueTextEquals("s"u8))
|
||||
if (reader.ValueTextEquals("s"u8))
|
||||
{
|
||||
reader.Read();
|
||||
symbolId = _symbols.Resolve(reader.ValueSpan);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (depth != 1)
|
||||
if (reader.ValueTextEquals("k"u8))
|
||||
{
|
||||
reader.Read();
|
||||
if (reader.TokenType == JsonTokenType.StartObject)
|
||||
{
|
||||
DecodeKlineBody(ref reader, ref k);
|
||||
}
|
||||
else
|
||||
{
|
||||
reader.Skip();
|
||||
}
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
if (reader.ValueTextEquals("t"u8)) { reader.Read(); openTime = reader.GetInt64(); }
|
||||
else if (reader.ValueTextEquals("o"u8)) { reader.Read(); open = Number(ref reader); }
|
||||
else if (reader.ValueTextEquals("h"u8)) { reader.Read(); high = Number(ref reader); }
|
||||
else if (reader.ValueTextEquals("l"u8)) { reader.Read(); low = Number(ref reader); }
|
||||
else if (reader.ValueTextEquals("c"u8)) { reader.Read(); close = Number(ref reader); }
|
||||
else if (reader.ValueTextEquals("v"u8)) { reader.Read(); volume = Number(ref reader); }
|
||||
else if (reader.ValueTextEquals("q"u8)) { reader.Read(); quoteVolume = Number(ref reader); }
|
||||
else if (reader.ValueTextEquals("V"u8)) { reader.Read(); takerBuy = Number(ref reader); }
|
||||
else if (reader.ValueTextEquals("n"u8)) { reader.Read(); trades = reader.GetInt32(); }
|
||||
else if (reader.ValueTextEquals("x"u8)) { reader.Read(); closed = reader.TokenType == JsonTokenType.True; }
|
||||
else { reader.Read(); reader.Skip(); }
|
||||
reader.Read();
|
||||
reader.Skip();
|
||||
}
|
||||
|
||||
if (!closed || symbolId < 0 || close <= 0)
|
||||
if (!k.Closed || symbolId < 0 || k.Close <= 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
Bar bar = new(
|
||||
DateTimeOffset.FromUnixTimeMilliseconds(openTime).UtcDateTime,
|
||||
open, high, low, close, volume,
|
||||
volume > 0 ? quoteVolume / volume : 0,
|
||||
trades,
|
||||
takerBuy);
|
||||
DateTimeOffset.FromUnixTimeMilliseconds(k.OpenTime).UtcDateTime,
|
||||
k.Open, k.High, k.Low, k.Close, k.Volume,
|
||||
k.Volume > 0 ? k.QuoteVolume / k.Volume : 0,
|
||||
k.Trades,
|
||||
k.TakerBuy);
|
||||
|
||||
OnBar?.Invoke(symbolId, _symbols.Name(symbolId), bar);
|
||||
}
|
||||
|
||||
/// <summary>The fields of one kline, filled in whatever order Binance sends them.</summary>
|
||||
private struct KlineFields
|
||||
{
|
||||
public long OpenTime;
|
||||
public double Open;
|
||||
public double High;
|
||||
public double Low;
|
||||
public double Close;
|
||||
public double Volume;
|
||||
public double QuoteVolume;
|
||||
public double TakerBuy;
|
||||
public int Trades;
|
||||
public bool Closed;
|
||||
}
|
||||
|
||||
/// <summary>Reads the <c>k</c> object, positioned on its StartObject, through its EndObject.</summary>
|
||||
private static void DecodeKlineBody(ref Utf8JsonReader reader, ref KlineFields k)
|
||||
{
|
||||
while (reader.Read())
|
||||
{
|
||||
if (reader.TokenType == JsonTokenType.EndObject)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
if (reader.TokenType != JsonTokenType.PropertyName)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (reader.ValueTextEquals("t"u8)) { reader.Read(); k.OpenTime = reader.GetInt64(); }
|
||||
else if (reader.ValueTextEquals("o"u8)) { reader.Read(); k.Open = Number(ref reader); }
|
||||
else if (reader.ValueTextEquals("h"u8)) { reader.Read(); k.High = Number(ref reader); }
|
||||
else if (reader.ValueTextEquals("l"u8)) { reader.Read(); k.Low = Number(ref reader); }
|
||||
else if (reader.ValueTextEquals("c"u8)) { reader.Read(); k.Close = Number(ref reader); }
|
||||
else if (reader.ValueTextEquals("v"u8)) { reader.Read(); k.Volume = Number(ref reader); }
|
||||
else if (reader.ValueTextEquals("q"u8)) { reader.Read(); k.QuoteVolume = Number(ref reader); }
|
||||
else if (reader.ValueTextEquals("V"u8)) { reader.Read(); k.TakerBuy = Number(ref reader); }
|
||||
else if (reader.ValueTextEquals("n"u8)) { reader.Read(); k.Trades = reader.GetInt32(); }
|
||||
else if (reader.ValueTextEquals("x"u8)) { reader.Read(); k.Closed = reader.TokenType == JsonTokenType.True; }
|
||||
else { reader.Read(); reader.Skip(); }
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Feeds one frame straight into the decoder, exactly as the socket would. Exists so
|
||||
/// the tests can hand the decoder a verbatim Binance payload without a connection.
|
||||
/// </summary>
|
||||
internal void Feed(ReadOnlySpan<byte> utf8Frame) => OnMessage(utf8Frame, isText: true);
|
||||
|
||||
private void DecodeBookTicker(ref Utf8JsonReader reader)
|
||||
{
|
||||
int symbolId = -1;
|
||||
|
||||
@@ -54,16 +54,52 @@ public partial class App : Application
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The config lives next to the executable. Working directories vary (debugger,
|
||||
/// shortcut, taskbar), so resolving it relative to the assembly is the only choice
|
||||
/// that always finds the file.
|
||||
/// Where the configuration lives, and how it gets there the first time.
|
||||
/// <para>
|
||||
/// <c>Documenti\Encelado\encelado.json</c>. It is the operator's file — their
|
||||
/// thresholds, their pairs, their notes — so it belongs with their documents, where a
|
||||
/// backup catches it and a reinstall cannot overwrite it. The credentials do
|
||||
/// <b>not</b> live here: they stay encrypted in the per-user application data folder,
|
||||
/// because a file in Documents is precisely the kind of file that gets copied to a
|
||||
/// USB stick or synced to a cloud drive.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// On first run the file is seeded from the copy shipped beside the executable when
|
||||
/// there is one (the previous location, so an existing tuning is carried over rather
|
||||
/// than lost) and from the built-in default otherwise.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static string ConfigDirectory =>
|
||||
Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments), "Encelado");
|
||||
|
||||
private static string ResolveConfigPath()
|
||||
{
|
||||
string beside = Path.Combine(AppContext.BaseDirectory, "encelado.json");
|
||||
return File.Exists(beside) ? beside : Path.GetFullPath("encelado.json");
|
||||
string target = Path.Combine(ConfigDirectory, "encelado.json");
|
||||
if (File.Exists(target))
|
||||
{
|
||||
return target;
|
||||
}
|
||||
|
||||
Directory.CreateDirectory(ConfigDirectory);
|
||||
|
||||
string legacy = Path.Combine(AppContext.BaseDirectory, "encelado.json");
|
||||
if (File.Exists(legacy))
|
||||
{
|
||||
File.Copy(legacy, target, overwrite: false);
|
||||
SeedNote = $"configurazione copiata da {legacy} a {target}: da ora si modifica quella in Documenti";
|
||||
}
|
||||
else
|
||||
{
|
||||
File.WriteAllText(target, ConfigDefaults.Json);
|
||||
SeedNote = $"nessuna configurazione trovata: creata quella di fabbrica in {target}";
|
||||
}
|
||||
|
||||
return target;
|
||||
}
|
||||
|
||||
/// <summary>What happened at first run, for the log; null when the file already existed.</summary>
|
||||
public static string? SeedNote { get; private set; }
|
||||
|
||||
private static void OnDispatcherException(object sender, DispatcherUnhandledExceptionEventArgs e)
|
||||
{
|
||||
Log.Error("UI exception", e.Exception);
|
||||
|
||||
@@ -31,6 +31,10 @@ public sealed class BotConfig
|
||||
|
||||
public LoggingOptions Logging { get; set; } = new();
|
||||
|
||||
public StorageOptions Storage { get; set; } = new();
|
||||
|
||||
public MlOptions Ml { get; set; } = new();
|
||||
|
||||
public List<PairConfig> Pairs { get; set; } = [];
|
||||
|
||||
public IEnumerable<PairConfig> EnabledPairs => Pairs.Where(static p => p.Enabled);
|
||||
@@ -69,6 +73,8 @@ public sealed class BotConfig
|
||||
Risk.Validate();
|
||||
Engine.Validate();
|
||||
Logging.Validate();
|
||||
Storage.Validate();
|
||||
Ml.Validate();
|
||||
|
||||
List<PairConfig> enabled = [.. EnabledPairs];
|
||||
if (enabled.Count == 0)
|
||||
@@ -248,9 +254,9 @@ public sealed class EngineOptions
|
||||
throw new InvalidOperationException("engine.entryOrderType deve essere 'limit' oppure 'market'.");
|
||||
}
|
||||
|
||||
if (WarmupBars is < 50 or > 1500)
|
||||
if (WarmupBars is < 50 or > 20_000)
|
||||
{
|
||||
throw new InvalidOperationException("engine.warmupBars deve essere fra 50 e 1500.");
|
||||
throw new InvalidOperationException("engine.warmupBars deve essere fra 50 e 20000.");
|
||||
}
|
||||
|
||||
if (CalibrationBars is < 100 or > 1500)
|
||||
@@ -301,17 +307,24 @@ public sealed class LoggingOptions
|
||||
public bool Console { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Folder that holds every output file. Relative paths resolve against the
|
||||
/// executable's directory, so the app writes to the same place regardless of where
|
||||
/// it was launched from. An absolute path is used as given.
|
||||
/// Folder that holds every output file. Relative paths resolve against the folder
|
||||
/// the configuration file lives in — <c>Documenti\Encelado</c> by default — so
|
||||
/// everything the operator owns sits in one place. An absolute path is used as given.
|
||||
/// </summary>
|
||||
public string Directory { get; set; } = "logs";
|
||||
|
||||
/// <summary>
|
||||
/// Where relative paths are anchored. Set by the loader to the configuration file's
|
||||
/// own folder; falls back to the executable's folder when a configuration was never
|
||||
/// loaded from disk (tests, defaults built in memory).
|
||||
/// </summary>
|
||||
public string BaseDirectory { get; set; } = AppContext.BaseDirectory;
|
||||
|
||||
/// <summary>Application log file name. Empty disables file logging.</summary>
|
||||
public string File { get; set; } = "encelado.log";
|
||||
|
||||
/// <summary>One JSON line per order event. Empty disables it.</summary>
|
||||
public string TradeJournal { get; set; } = "trades.jsonl";
|
||||
/// <summary>One <c>;</c>-separated row per order event. Empty disables it.</summary>
|
||||
public string TradeJournal { get; set; } = "trades.csv";
|
||||
|
||||
/// <summary>
|
||||
/// One CSV row per evaluated bar, per pair, with the spread, the z-score, the
|
||||
@@ -353,7 +366,7 @@ public sealed class LoggingOptions
|
||||
string directory = string.IsNullOrWhiteSpace(Directory) ? "logs" : Directory;
|
||||
return Path.IsPathRooted(directory)
|
||||
? directory
|
||||
: Path.Combine(AppContext.BaseDirectory, directory);
|
||||
: Path.Combine(BaseDirectory, directory);
|
||||
}
|
||||
|
||||
/// <summary>Absolute path of a file inside the log directory, or null when disabled.</summary>
|
||||
@@ -396,3 +409,104 @@ public sealed class LoggingOptions
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// How the meta-model is used at decision time.
|
||||
/// <para>
|
||||
/// The model never proposes a trade: the statistical rule does that. The model may
|
||||
/// only veto an entry whose predicted probability of paying off is too low, and shrink
|
||||
/// the size of one it is lukewarm about. With no champion in the database these
|
||||
/// settings do nothing and the bot trades the rule as before.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class MlOptions
|
||||
{
|
||||
/// <summary>Consult the champion model before an entry. Needs the database.</summary>
|
||||
public bool Enabled { get; set; } = true;
|
||||
|
||||
/// <summary>Entries below this predicted probability are refused.</summary>
|
||||
public double MinProbability { get; set; } = 0.55;
|
||||
|
||||
/// <summary>Scale the position by <c>2Φ(z)−1</c> of the probability instead of all-or-nothing.</summary>
|
||||
public bool SizeByProbability { get; set; } = true;
|
||||
|
||||
/// <summary>The smallest fraction of the risk engine's size the model may leave.</summary>
|
||||
public double MinSizeFraction { get; set; } = 0.25;
|
||||
|
||||
/// <summary>How many scored entries between two drift checks.</summary>
|
||||
public int DriftCheckBars { get; set; } = 50;
|
||||
|
||||
/// <summary>PSI at or above which a feature counts as drifted.</summary>
|
||||
public double DriftPsiAlert { get; set; } = 0.25;
|
||||
|
||||
/// <summary>
|
||||
/// Drifted features needed to suspend the model until the next recalibration reloads
|
||||
/// it. 0 never suspends: drift is logged and recorded but the model keeps its say.
|
||||
/// </summary>
|
||||
public int DriftAlertsToSuspend { get; set; } = 3;
|
||||
|
||||
public void Validate()
|
||||
{
|
||||
if (MinProbability is < 0.5 or > 0.99)
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
"ml.minProbability deve essere fra 0.5 e 0.99: sotto 0.5 il modello direbbe sì a " +
|
||||
"un ingresso che ritiene più probabilmente perdente.");
|
||||
}
|
||||
|
||||
if (MinSizeFraction is < 0.05 or > 1)
|
||||
{
|
||||
throw new InvalidOperationException("ml.minSizeFraction deve essere fra 0.05 e 1.");
|
||||
}
|
||||
|
||||
if (DriftCheckBars < 10)
|
||||
{
|
||||
throw new InvalidOperationException("ml.driftCheckBars deve essere almeno 10.");
|
||||
}
|
||||
|
||||
if (DriftPsiAlert is <= 0 or > 5)
|
||||
{
|
||||
throw new InvalidOperationException("ml.driftPsiAlert deve essere fra 0 e 5.");
|
||||
}
|
||||
|
||||
if (DriftAlertsToSuspend < 0)
|
||||
{
|
||||
throw new InvalidOperationException("ml.driftAlertsToSuspend non può essere negativo.");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Where the local database lives, and whether it is used at all.</summary>
|
||||
public sealed class StorageOptions
|
||||
{
|
||||
/// <summary>
|
||||
/// Write every journal table to the database as well as to CSV, and read models,
|
||||
/// datasets and the champion from it. Off means CSV only and no machine learning.
|
||||
/// </summary>
|
||||
public bool Enabled { get; set; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// Path of the SQLite file. Empty selects <c>%ProgramData%\Encelado\encelado.db</c>:
|
||||
/// a machine-wide folder, deliberately distinct from Documents (where the
|
||||
/// configuration lives and gets synced) and from the per-user application data
|
||||
/// (where the credentials live and nothing else should).
|
||||
/// </summary>
|
||||
public string DatabasePath { get; set; } = string.Empty;
|
||||
|
||||
public string ResolvePath() =>
|
||||
string.IsNullOrWhiteSpace(DatabasePath)
|
||||
? Path.Combine(
|
||||
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
|
||||
"Encelado",
|
||||
"encelado.db")
|
||||
: Path.GetFullPath(DatabasePath);
|
||||
|
||||
public void Validate()
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(DatabasePath) &&
|
||||
DatabasePath.IndexOfAny(Path.GetInvalidPathChars()) >= 0)
|
||||
{
|
||||
throw new InvalidOperationException("storage.databasePath contiene caratteri non validi.");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -115,7 +115,7 @@ public static class ConfigDefaults
|
||||
{
|
||||
"_comment": "Encelado — arbitraggio statistico su coppie cointegrate, Binance Futures USDⓈ-M. Il bot non prende posizione sulla direzione del mercato: è long una gamba e short l'altra nel rapporto che il test di cointegrazione produce, e scommette solo sul fatto che la distanza fra le due si richiuda.",
|
||||
|
||||
"_misura": "IMPORTANTE — che cosa dice il backtest, e perché dryRun parte ATTIVO. Misurato su 6,6 anni di barre da un minuto (2020-2026) di ETHUSDT, BTCUSDT, SOLUSDT e AVAXUSDT, ripiegate su 5m, 15m, 30m, 1h e 4h, con commissione taker 4 bps più 1 bp di slippage per lato. Il risultato: su 5m nessuna combinazione di soglie supera nemmeno i filtri di taratura; su 15m e 1h la ricerca a griglia trova combinazioni che rendono in taratura e in verifica, ma NESSUNA delle prime dieci resta positiva sulla terza fetta, quella che non entra mai nella scelta. Con i valori qui sotto, sulle quattro coppie misurabili, la conferma dà tre risultati negativi e uno positivo del +0,40% su una sola operazione. La ragione di fondo è che il test di cointegrazione passa solo l'8-13% del tempo, quindi in sei anni una coppia produce qualche decina di operazioni: troppo poche per distinguere un margine reale da una serie fortunata. Questi valori sono i meglio supportati fra quelli provati, non una strategia dimostrata. Falla girare in dry-run finché non hai visto coi tuoi occhi come si comporta sul tuo conto. Lo strumento è in tools/Encelado.Backtest: 'backtest basket --data <cartella>' rifa la scelta da capo sui tuoi dati.",
|
||||
"_misura": "IMPORTANTE — che cosa dice il backtest, e perché dryRun parte ATTIVO. Misurato su 6,6 anni di barre da un minuto (2020-2026) di ETHUSDT, BTCUSDT, SOLUSDT e AVAXUSDT, ripiegate su 5m, 15m, 30m, 1h e 4h, con commissione taker 4 bps più 1 bp di slippage per lato. Il risultato: su 5m nessuna combinazione di soglie supera nemmeno i filtri di taratura; su 15m e 1h la ricerca a griglia trova combinazioni che rendono in taratura e in verifica, ma NESSUNA delle prime dieci resta positiva sulla terza fetta, quella che non entra mai nella scelta. Con i valori qui sotto, sulle quattro coppie misurabili, la conferma dà tre risultati negativi e uno positivo del +0,40% su una sola operazione. La ragione di fondo è che il test di cointegrazione passa solo l'8-13% del tempo, quindi in sei anni una coppia produce qualche decina di operazioni: troppo poche per distinguere un margine reale da una serie fortunata. Questi valori sono i meglio supportati fra quelli provati, non una strategia dimostrata: per questo la configurazione di fabbrica punta alla TESTNET, dove gli ordini partono davvero ma i soldi sono finti. Lo strumento è in tools/Encelado.Backtest: 'backtest basket --data <cartella>' rifà la scelta da capo sui tuoi dati.",
|
||||
|
||||
"binance": {
|
||||
"_testnet": "true = testnet (denaro finto, stesse API). Metterlo a false opera con denaro reale. Le chiavi dei due ambienti sono diverse e vengono salvate separatamente.",
|
||||
@@ -153,8 +153,8 @@ public static class ConfigDefaults
|
||||
"_postOnly": "Solo maker (GTX). Fa risparmiare la commissione ma un ordine che non esegue lascia una gamba scoperta, che costa molto di più. Lasciare false se non hai misurato il tuo tasso di riempimento.",
|
||||
"postOnlyEntries": false,
|
||||
|
||||
"_dryRun": "ATTIVO DI FABBRICA, e la ragione è nella nota _misura qui sopra. Calcola e registra tutto, non invia nessun ordine. Toglilo solo dopo aver visto in Stato che le tue coppie passano davvero il test di cointegrazione.",
|
||||
"dryRun": true,
|
||||
"_dryRun": "Calcola e registra tutto, non invia nessun ordine. Spento di fabbrica: in testnet i soldi sono finti e la prova più vicina alla produzione è quella con gli ordini veri sul conto di prova — eseguiti, rifiutati, parzialmente riempiti, con le stesse regole di lotto e di margine. Accendilo per osservare una configurazione nuova senza toccare nemmeno il conto di prova. Sul conto reale resta comunque la conferma esplicita all'avvio.",
|
||||
"dryRun": false,
|
||||
|
||||
"_reconcile": "Ogni quanto il bot ricontrolla conto, posizioni e ordini contro Binance. È anche quando recupera le barre che lo stream non ha consegnato e chiude le coppie rimaste con una gamba sola.",
|
||||
"reconcileSeconds": 20,
|
||||
@@ -210,7 +210,7 @@ public static class ConfigDefaults
|
||||
"_level": "trace | debug | info | warn | error | none. Ogni rifiuto che ferma un ordine viene scritto a 'info' o sopra, quindi 'debug' serve per il flusso dati, non per capire perché il bot non ha operato.",
|
||||
"level": "info",
|
||||
|
||||
"_directory": "Dove salvare tutti gli output. Relativa all'eseguibile, oppure un percorso assoluto. Si cambia anche da Impostazioni.",
|
||||
"_directory": "Dove salvare tutti gli output. Relativa alla cartella della configurazione (Documenti\\Encelado), oppure un percorso assoluto. Si cambia anche da Impostazioni.",
|
||||
"directory": "logs",
|
||||
|
||||
"console": false,
|
||||
@@ -218,8 +218,8 @@ public static class ConfigDefaults
|
||||
"maxFileSizeMb": 32,
|
||||
"maxFiles": 10,
|
||||
|
||||
"_analysis": "decisions.csv ha una riga per ogni barra valutata con spread, z-score e calibrazione; executions.csv una riga per ogni segnale arrivato agli ordini, con il verdetto del risk engine. Si uniscono su decisionId.",
|
||||
"tradeJournal": "trades.jsonl",
|
||||
"_analysis": "Tutte le tabelle usano il separatore ; e hanno una colonna motivazione che spiega in italiano il perché della riga. decisions.csv ha una riga per ogni barra valutata con spread, z-score e calibrazione; executions.csv una riga per ogni segnale arrivato agli ordini, con il verdetto del risk engine; trades.csv una riga per ogni evento d'ordine. decisions ed executions si uniscono su decisionId.",
|
||||
"tradeJournal": "trades.csv",
|
||||
"decisionLog": "decisions.csv",
|
||||
"executionLog": "executions.csv",
|
||||
|
||||
@@ -228,6 +228,26 @@ public static class ConfigDefaults
|
||||
"bufferedLines": 5000
|
||||
},
|
||||
|
||||
"storage": {
|
||||
"_note": "Il database locale (SQLite): barre canoniche, dataset, modelli, validazioni, journal e drift. Vuoto = %ProgramData%\\Encelado\\encelado.db, una cartella di sistema separata sia da Documenti (dove sta la configurazione) sia dai dati utente (dove stanno le chiavi).",
|
||||
"enabled": true,
|
||||
"databasePath": ""
|
||||
},
|
||||
|
||||
"ml": {
|
||||
"_note": "Il meta-modello (meta-labeling): un classificatore addestrato sugli esiti dei segnali passati che dice se un ingresso proposto dalla regola statistica ha probabilità di ripagare i costi. Non propone mai operazioni: può solo rifiutarne una o ridurne la size. Si addestra con lo strumento tools/Encelado.Backtest (comandi dataset, train) e vive nel database locale come 'campione' della coppia; senza campione queste voci non fanno nulla.",
|
||||
"enabled": true,
|
||||
"_minProbability": "Sotto questa probabilità predetta l'ingresso viene rifiutato. 0.55 significa: opero solo quando il modello vede un vantaggio, anche piccolo, rispetto a lanciare una moneta.",
|
||||
"minProbability": 0.55,
|
||||
"_sizeByProbability": "Con true la size è proporzionale alla convinzione del modello (2·Φ(z)−1, López de Prado), fra minSizeFraction e 1; con false è tutto o niente.",
|
||||
"sizeByProbability": true,
|
||||
"minSizeFraction": 0.25,
|
||||
"_drift": "Ogni driftCheckBars segnali valutati le feature live vengono confrontate con quelle di addestramento (PSI e Kolmogorov-Smirnov). Una feature con PSI ≥ driftPsiAlert è 'in deriva'; con driftAlertsToSuspend feature in deriva il modello viene sospeso fino alla ricalibrazione successiva e il bot torna alla sola regola statistica. 0 = mai sospendere, solo registrare.",
|
||||
"driftCheckBars": 50,
|
||||
"driftPsiAlert": 0.25,
|
||||
"driftAlertsToSuspend": 3
|
||||
},
|
||||
|
||||
"_pairs": "Il paniere della guida. Ogni coppia si legge come ln(A) − β·ln(B): A è la gamba su cui si misura lo spread, B quella di copertura. Il test di cointegrazione decide da solo, a ogni ricalibrazione, se una coppia è operabile: quelle che non passano restano in elenco e non vengono aperte. Sui dati disponibili passano il test solo l'8-17% del tempo, quindi il bot resta fermo a lungo — è il comportamento previsto.",
|
||||
|
||||
"_parametri": "I valori qui sotto vengono dalla ricerca a griglia (tools/Encelado.Backtest, comando 'basket'), non dalla guida. Le differenze rispetto alla guida sono tre e sono tutte misurate: zWindow 1500 invece di 100, entryZ 2.5 invece di 2.0, stopZ 6.0 invece di 3.5. La prima è la più importante: con una finestra vicina all'emivita del rientro (30-40 barre) la media mobile insegue lo scostamento e lo assorbe, e il margine sparisce prima ancora delle commissioni.",
|
||||
|
||||
@@ -25,6 +25,14 @@ public static class ConfigLoader
|
||||
warnings = [];
|
||||
BotConfig config = new();
|
||||
|
||||
// Relative output paths anchor to the configuration's own folder, so a config in
|
||||
// Documents keeps its logs beside it instead of beside the executable.
|
||||
string? folder = Path.GetDirectoryName(Path.GetFullPath(path));
|
||||
if (!string.IsNullOrEmpty(folder))
|
||||
{
|
||||
config.Logging.BaseDirectory = folder;
|
||||
}
|
||||
|
||||
if (File.Exists(path))
|
||||
{
|
||||
using FileStream stream = File.OpenRead(path);
|
||||
@@ -81,6 +89,12 @@ public static class ConfigLoader
|
||||
case "pairs":
|
||||
ReadPairs(config, section.Value, warnings);
|
||||
break;
|
||||
case "storage":
|
||||
ReadStorage(config, section.Value, warnings);
|
||||
break;
|
||||
case "ml":
|
||||
ReadMl(config, section.Value, warnings);
|
||||
break;
|
||||
case "$schema":
|
||||
case "_comment":
|
||||
break;
|
||||
@@ -203,6 +217,39 @@ public static class ConfigLoader
|
||||
}
|
||||
}
|
||||
|
||||
private static void ReadStorage(BotConfig config, JsonElement e, List<string> warnings)
|
||||
{
|
||||
StorageOptions o = config.Storage;
|
||||
foreach (JsonProperty p in Properties(e, "storage", warnings))
|
||||
{
|
||||
switch (p.Name.ToLowerInvariant())
|
||||
{
|
||||
case "enabled": o.Enabled = Bool(p); break;
|
||||
case "databasepath": o.DatabasePath = Str(p); break;
|
||||
default: warnings.Add($"chiave sconosciuta 'storage.{p.Name}'"); break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static void ReadMl(BotConfig config, JsonElement e, List<string> warnings)
|
||||
{
|
||||
MlOptions o = config.Ml;
|
||||
foreach (JsonProperty p in Properties(e, "ml", warnings))
|
||||
{
|
||||
switch (p.Name.ToLowerInvariant())
|
||||
{
|
||||
case "enabled": o.Enabled = Bool(p); break;
|
||||
case "minprobability": o.MinProbability = Num(p); break;
|
||||
case "sizebyprobability": o.SizeByProbability = Bool(p); break;
|
||||
case "minsizefraction": o.MinSizeFraction = Num(p); break;
|
||||
case "driftcheckbars": o.DriftCheckBars = Int(p); break;
|
||||
case "driftpsialert": o.DriftPsiAlert = Num(p); break;
|
||||
case "driftalertstosuspend": o.DriftAlertsToSuspend = Int(p); break;
|
||||
default: warnings.Add($"chiave sconosciuta 'ml.{p.Name}'"); break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static void ReadPairs(BotConfig config, JsonElement e, List<string> warnings)
|
||||
{
|
||||
if (e.ValueKind != JsonValueKind.Array)
|
||||
|
||||
@@ -2,6 +2,7 @@ using System.Globalization;
|
||||
using System.Text;
|
||||
using Encelado.Bot.Configuration;
|
||||
using Encelado.Bot.Logging;
|
||||
using Encelado.Core.Journal;
|
||||
using Encelado.Core.Market;
|
||||
using Encelado.Core.Strategies.Pairs;
|
||||
|
||||
@@ -15,59 +16,65 @@ namespace Encelado.Bot.Diagnostics;
|
||||
/// <list type="bullet">
|
||||
/// <item><b>decisions</b> — one row per aligned bar per pair: both closes, the spread,
|
||||
/// the z-score, the calibration that produced them, the position at the time, and the
|
||||
/// signal that came out. This is the dataset to load into pandas when asking "why did it
|
||||
/// do that" or "would a different entry threshold have helped".</item>
|
||||
/// signal that came out. This is the dataset to load when asking "why did it do that" or
|
||||
/// "would a different entry threshold have helped".</item>
|
||||
/// <item><b>executions</b> — one row per signal that reached the order path: the risk
|
||||
/// verdict, the size that survived it, and the exchange's answer for each leg.</item>
|
||||
/// </list>
|
||||
/// <para>
|
||||
/// CSV rather than JSON on purpose: it opens in Excel, loads in one line of pandas, and
|
||||
/// stays readable when a run produces tens of thousands of rows. Writes are buffered and
|
||||
/// flushed on a timer, so the decision path never waits on the disk.
|
||||
/// Every table the bot writes uses <c>;</c> as its separator and carries a
|
||||
/// <c>motivazione</c> column that says in plain language why the row exists. The
|
||||
/// separator is what makes the file open correctly in an Italian-locale spreadsheet
|
||||
/// without an import wizard; the column is what makes the row readable without the
|
||||
/// code that wrote it.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Two things learned from a real run are enforced here. A file whose first line is not
|
||||
/// this header is moved aside rather than appended to — the previous run had left an
|
||||
/// executions file with the old column layout, and a decisions file with no header at
|
||||
/// all. And the decision id is seeded from the clock, because a counter that restarts
|
||||
/// at one on every launch produced two unrelated rows with <c>decisionId = 1</c>.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class AnalyticsLog : IDisposable
|
||||
{
|
||||
private const string DecisionHeader =
|
||||
"barTimeUtc,decisionId,pair,symbolA,symbolB,ready," +
|
||||
"closeA,closeB,volumeA,volumeB,deltaA,deltaB," +
|
||||
"spread,zScore,beta,alpha,pValue,adf,critical5,halfLife,cointegrated," +
|
||||
"qtyA,qtyB,entryZ,barsHeld,signal,reason," +
|
||||
"bidA,askA,spreadPctA,bidB,askB,spreadPctB,quoteAgeSec," +
|
||||
"fundingA,fundingB,equity,halted";
|
||||
public const char Separator = ';';
|
||||
|
||||
private const string ExecutionHeader =
|
||||
"timestampUtc,decisionId,pair,phase,approved,reason,detail," +
|
||||
"sideA,notionalA,quantityA,priceA,sideB,notionalB,quantityB,priceB," +
|
||||
"netFundingRate,equity,availableBalance,grossExposure,openPairs," +
|
||||
"orderIdA,orderIdB,error,latencyMs";
|
||||
public const string DecisionHeader =
|
||||
"barTimeUtc;decisionId;pair;symbolA;symbolB;ready;" +
|
||||
"closeA;closeB;volumeA;volumeB;deltaA;deltaB;" +
|
||||
"spread;zScore;beta;alpha;pValue;adf;critical5;halfLife;cointegrated;" +
|
||||
"qtyA;qtyB;entryZ;barsHeld;signal;" +
|
||||
"bidA;askA;spreadPctA;bidB;askB;spreadPctB;quoteAgeSec;" +
|
||||
"fundingA;fundingB;equity;halted;probPredetta;motivazione";
|
||||
|
||||
public const string ExecutionHeader =
|
||||
"timestampUtc;decisionId;pair;phase;approved;reason;" +
|
||||
"sideA;notionalA;quantityA;priceA;sideB;notionalB;quantityB;priceB;" +
|
||||
"netFundingRate;equity;availableBalance;grossExposure;openPairs;" +
|
||||
"orderIdA;orderIdB;error;latencyMs;motivazione";
|
||||
|
||||
private readonly StreamWriter? _decisions;
|
||||
private readonly StreamWriter? _executions;
|
||||
private readonly IJournalSink? _sink;
|
||||
private readonly Lock _gate = new();
|
||||
private readonly StringBuilder _row = new(768);
|
||||
|
||||
private long _nextId;
|
||||
private DateTime _lastFlush = DateTime.UtcNow;
|
||||
|
||||
public AnalyticsLog(LoggingOptions options)
|
||||
public AnalyticsLog(LoggingOptions options, IJournalSink? sink = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
_sink = sink;
|
||||
|
||||
_decisions = Open(options.ResolvePath(options.DecisionLog));
|
||||
_executions = Open(options.ResolvePath(options.ExecutionLog));
|
||||
_decisions = CsvTable.Open(options.ResolvePath(options.DecisionLog), DecisionHeader);
|
||||
_executions = CsvTable.Open(options.ResolvePath(options.ExecutionLog), ExecutionHeader);
|
||||
|
||||
// Headers are fixed, so they can be written up front rather than deferred until
|
||||
// the first row — which is what forced the old file to guess at its own columns.
|
||||
if (_decisions is not null && _decisions.BaseStream.Length == 0)
|
||||
{
|
||||
_decisions.WriteLine(DecisionHeader);
|
||||
}
|
||||
|
||||
if (_executions is not null && _executions.BaseStream.Length == 0)
|
||||
{
|
||||
_executions.WriteLine(ExecutionHeader);
|
||||
}
|
||||
// Milliseconds since the epoch, times a thousand: unique across restarts, and
|
||||
// still monotonic within one run for anything under a thousand decisions a
|
||||
// millisecond.
|
||||
_nextId = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() * 1000;
|
||||
}
|
||||
|
||||
public bool IsEnabled => _decisions is not null || _executions is not null;
|
||||
@@ -86,19 +93,30 @@ public sealed class AnalyticsLog : IDisposable
|
||||
StatArbStrategy strategy,
|
||||
in PairPositionView position,
|
||||
in PairSignal signal,
|
||||
string explanation,
|
||||
double quoteAgeSeconds,
|
||||
double equity,
|
||||
bool halted)
|
||||
bool halted,
|
||||
double metaProbability = double.NaN)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(strategy);
|
||||
|
||||
PairCalibration c = strategy.Calibration;
|
||||
string motivazione = signal.Reason.Length > 0 ? signal.Reason : explanation;
|
||||
|
||||
_sink?.Decision(new DecisionRow(
|
||||
barA.TimeUtc, decisionId, pair, legA.Symbol, legB.Symbol, strategy.IsReady,
|
||||
barA.Close, barB.Close, strategy.Spread, strategy.ZScore, c.Beta, c.Alpha, c.PValue,
|
||||
c.AdfStatistic, c.HalfLifeBars, c.IsCointegrated, position.QuantityA, position.QuantityB,
|
||||
position.EntryZScore, position.BarsHeld, signal.Kind.ToString(), legA.SpreadPct,
|
||||
legB.SpreadPct, legA.FundingRate, legB.FundingRate, equity, halted, metaProbability,
|
||||
motivazione));
|
||||
|
||||
if (_decisions is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
PairCalibration c = strategy.Calibration;
|
||||
|
||||
lock (_gate)
|
||||
{
|
||||
_row.Clear();
|
||||
@@ -132,7 +150,6 @@ public sealed class AnalyticsLog : IDisposable
|
||||
Add(position.EntryZScore);
|
||||
Add(position.BarsHeld);
|
||||
Add(signal.Kind.ToString());
|
||||
Add(signal.Reason);
|
||||
|
||||
Add(legA.Bid);
|
||||
Add(legA.Ask);
|
||||
@@ -145,7 +162,17 @@ public sealed class AnalyticsLog : IDisposable
|
||||
Add(legA.FundingRate);
|
||||
Add(legB.FundingRate);
|
||||
Add(equity);
|
||||
Add(halted ? 1 : 0, last: true);
|
||||
Add(halted ? 1 : 0);
|
||||
|
||||
// The meta-model's probability that this entry pays off; empty when there is
|
||||
// no model or the row is not an entry.
|
||||
Add(double.IsFinite(metaProbability)
|
||||
? metaProbability.ToString("F4", CultureInfo.InvariantCulture)
|
||||
: string.Empty);
|
||||
|
||||
// The signal's own reason when there is one; otherwise the strategy's
|
||||
// explanation of why it is waiting, so a row of "None" still says something.
|
||||
Add(motivazione, last: true);
|
||||
|
||||
_decisions.WriteLine(_row.ToString());
|
||||
MaybeFlush();
|
||||
@@ -155,6 +182,14 @@ public sealed class AnalyticsLog : IDisposable
|
||||
/// <summary>Records what the order path did with a signal.</summary>
|
||||
public void Execution(in ExecutionRecord record)
|
||||
{
|
||||
_sink?.Execution(new ExecutionRow(
|
||||
DateTime.UtcNow, record.DecisionId, record.Pair, record.Phase, record.Approved, record.Reason,
|
||||
Describe(record.SideA), record.NotionalA, record.QuantityA, record.PriceA,
|
||||
Describe(record.SideB), record.NotionalB, record.QuantityB, record.PriceB,
|
||||
record.NetFundingRate, record.Equity, record.AvailableBalance, record.GrossExposure,
|
||||
record.OpenPairs, record.OrderIdA, record.OrderIdB, record.Error, record.LatencyMs,
|
||||
record.Detail));
|
||||
|
||||
if (_executions is null)
|
||||
{
|
||||
return;
|
||||
@@ -170,7 +205,6 @@ public sealed class AnalyticsLog : IDisposable
|
||||
Add(record.Phase);
|
||||
Add(record.Approved ? 1 : 0);
|
||||
Add(record.Reason);
|
||||
Add(record.Detail);
|
||||
|
||||
Add(Describe(record.SideA));
|
||||
Add(record.NotionalA);
|
||||
@@ -189,7 +223,8 @@ public sealed class AnalyticsLog : IDisposable
|
||||
Add(record.OrderIdA);
|
||||
Add(record.OrderIdB);
|
||||
Add(record.Error);
|
||||
Add(record.LatencyMs, last: true);
|
||||
Add(record.LatencyMs);
|
||||
Add(record.Detail, last: true);
|
||||
|
||||
_executions.WriteLine(_row.ToString());
|
||||
MaybeFlush();
|
||||
@@ -212,7 +247,7 @@ public sealed class AnalyticsLog : IDisposable
|
||||
|
||||
if (!last)
|
||||
{
|
||||
_row.Append(',');
|
||||
_row.Append(Separator);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -221,28 +256,17 @@ public sealed class AnalyticsLog : IDisposable
|
||||
_row.Append(value.ToString(CultureInfo.InvariantCulture));
|
||||
if (!last)
|
||||
{
|
||||
_row.Append(',');
|
||||
_row.Append(Separator);
|
||||
}
|
||||
}
|
||||
|
||||
private void Add(string? value, bool last = false)
|
||||
{
|
||||
if (!string.IsNullOrEmpty(value))
|
||||
{
|
||||
// Quote only when necessary; a reason string routinely contains commas.
|
||||
if (value.AsSpan().IndexOfAny(',', '"', '\n') >= 0)
|
||||
{
|
||||
_row.Append('"').Append(value.Replace("\"", "\"\"", StringComparison.Ordinal)).Append('"');
|
||||
}
|
||||
else
|
||||
{
|
||||
_row.Append(value);
|
||||
}
|
||||
}
|
||||
CsvTable.AppendField(_row, value);
|
||||
|
||||
if (!last)
|
||||
{
|
||||
_row.Append(',');
|
||||
_row.Append(Separator);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -273,28 +297,6 @@ public sealed class AnalyticsLog : IDisposable
|
||||
}
|
||||
}
|
||||
|
||||
private static StreamWriter? Open(string? path)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(path))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
System.IO.Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(path))!);
|
||||
return new StreamWriter(
|
||||
new FileStream(path, FileMode.Append, FileAccess.Write, FileShare.ReadWrite, 16384),
|
||||
Encoding.UTF8)
|
||||
{ AutoFlush = false };
|
||||
}
|
||||
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
Log.Warn($"impossibile aprire il file di analisi {path}: {ex.Message}");
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
lock (_gate)
|
||||
@@ -315,6 +317,111 @@ public sealed class AnalyticsLog : IDisposable
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The one way every <c>;</c>-separated table in the bot is opened and written.
|
||||
/// </summary>
|
||||
public static class CsvTable
|
||||
{
|
||||
/// <summary>
|
||||
/// Opens a table for appending, writing the header when the file is new and moving
|
||||
/// the file aside when its first line is some <i>other</i> header.
|
||||
/// <para>
|
||||
/// Appending rows in one layout under a header in another is worse than losing the
|
||||
/// old rows: the file still opens, every column is silently one place off, and the
|
||||
/// analysis built on it is wrong without a single error. The old file is kept under
|
||||
/// <c>.old</c> so nothing is actually lost.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static StreamWriter? Open(string? path, string header)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(path))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
string full = Path.GetFullPath(path);
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(full)!);
|
||||
|
||||
if (File.Exists(full) && new FileInfo(full).Length > 0 && !HasHeader(full, header))
|
||||
{
|
||||
string aside = Path.Combine(
|
||||
Path.GetDirectoryName(full)!,
|
||||
Path.GetFileNameWithoutExtension(full) + ".old" + Path.GetExtension(full));
|
||||
|
||||
File.Move(full, aside, overwrite: true);
|
||||
Log.Warn($"{Path.GetFileName(full)} aveva colonne di una versione precedente: spostato in {Path.GetFileName(aside)}");
|
||||
}
|
||||
|
||||
FileStream stream = new(full, FileMode.Append, FileAccess.Write, FileShare.ReadWrite, 16384);
|
||||
StreamWriter writer = new(stream, new UTF8Encoding(encoderShouldEmitUTF8Identifier: false))
|
||||
{
|
||||
AutoFlush = false,
|
||||
};
|
||||
|
||||
if (stream.Length == 0)
|
||||
{
|
||||
writer.WriteLine(header);
|
||||
writer.Flush();
|
||||
}
|
||||
|
||||
return writer;
|
||||
}
|
||||
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
Log.Warn($"impossibile aprire la tabella {path}: {ex.Message}");
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private static bool HasHeader(string path, string header)
|
||||
{
|
||||
try
|
||||
{
|
||||
using StreamReader reader = new(path, Encoding.UTF8, detectEncodingFromByteOrderMarks: true);
|
||||
return string.Equals(reader.ReadLine(), header, StringComparison.Ordinal);
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Appends one text field, quoted only when it contains the separator, a quote or a
|
||||
/// line break. Line breaks become spaces: a row is one line, always.
|
||||
/// </summary>
|
||||
public static void AppendField(StringBuilder row, string? value)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(row);
|
||||
|
||||
if (string.IsNullOrEmpty(value))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
if (value.AsSpan().IndexOfAny(";\"\r\n") < 0)
|
||||
{
|
||||
row.Append(value);
|
||||
return;
|
||||
}
|
||||
|
||||
row.Append('"');
|
||||
foreach (char c in value)
|
||||
{
|
||||
if (c == '"')
|
||||
{
|
||||
row.Append('"');
|
||||
}
|
||||
|
||||
row.Append(c is '\r' or '\n' ? ' ' : c);
|
||||
}
|
||||
|
||||
row.Append('"');
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>One leg's market state at the moment of a decision, for the decision log.</summary>
|
||||
public readonly record struct LegSnapshot(
|
||||
string Symbol,
|
||||
@@ -324,9 +431,9 @@ public readonly record struct LegSnapshot(
|
||||
double FundingRate);
|
||||
|
||||
/// <summary>
|
||||
/// One row of the execution log. A record rather than twenty positional parameters:
|
||||
/// the previous signature had eighteen, and a call site that swaps two of them compiles
|
||||
/// silently and corrupts the dataset the strategy is tuned on.
|
||||
/// One row of the execution log. A record rather than twenty positional parameters: a
|
||||
/// call site that swaps two of them compiles silently and corrupts the dataset the
|
||||
/// strategy is tuned on.
|
||||
/// </summary>
|
||||
public readonly record struct ExecutionRecord
|
||||
{
|
||||
@@ -341,6 +448,7 @@ public readonly record struct ExecutionRecord
|
||||
|
||||
public required string Reason { get; init; }
|
||||
|
||||
/// <summary>The plain-language why. Becomes the <c>motivazione</c> column.</summary>
|
||||
public required string Detail { get; init; }
|
||||
|
||||
public Side SideA { get; init; }
|
||||
|
||||
@@ -35,12 +35,17 @@
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\Encelado.Core\Encelado.Core.csproj" />
|
||||
<ProjectReference Include="..\Encelado.Binance\Encelado.Binance.csproj" />
|
||||
<ProjectReference Include="..\Encelado.Storage\Encelado.Storage.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
<!-- DPAPI (System.Security.Cryptography.ProtectedData) ships inside the Windows
|
||||
Desktop framework, so no package reference is needed: the app has zero NuGet
|
||||
dependencies at runtime. -->
|
||||
|
||||
<ItemGroup>
|
||||
<InternalsVisibleTo Include="Encelado.Tests" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<None Include="..\..\config\encelado.json" Link="encelado.json" CopyToOutputDirectory="PreserveNewest" />
|
||||
<None Include="..\..\config\*.json" Exclude="..\..\config\*.local.json" Link="config\%(Filename)%(Extension)" CopyToOutputDirectory="PreserveNewest" />
|
||||
|
||||
@@ -71,6 +71,25 @@ public sealed class LegState(int id, string symbol)
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records the last bar of a downloaded history as already seen, without counting it
|
||||
/// as a live bar. Without this the REST backstop would "recover" the final bars of
|
||||
/// the calibration window on every start, because it compares against a last-closed
|
||||
/// time that was never set.
|
||||
/// </summary>
|
||||
public void SeedHistory(in Bar bar)
|
||||
{
|
||||
if (bar.TimeUtc > LastClosedBar.TimeUtc)
|
||||
{
|
||||
LastClosedBar = bar;
|
||||
}
|
||||
|
||||
if (bar.Close > 0 && LastPrice <= 0)
|
||||
{
|
||||
LastPrice = bar.Close;
|
||||
}
|
||||
}
|
||||
|
||||
public void OnFunding(double markPrice, double fundingRate, DateTime nextFundingUtc)
|
||||
{
|
||||
if (markPrice > 0)
|
||||
|
||||
@@ -0,0 +1,305 @@
|
||||
using System.Globalization;
|
||||
using Encelado.Bot.Configuration;
|
||||
using Encelado.Bot.Logging;
|
||||
using Encelado.Core.Market;
|
||||
using Encelado.Core.Ml;
|
||||
using Encelado.Core.Statistics;
|
||||
using Encelado.Core.Strategies.Pairs;
|
||||
using Encelado.Storage;
|
||||
|
||||
namespace Encelado.Bot.Engine;
|
||||
|
||||
/// <summary>What the meta-model said about one proposed entry.</summary>
|
||||
public readonly record struct MetaVerdict(bool HasModel, double Probability, bool Allowed, double SizeFraction, string Note)
|
||||
{
|
||||
public static readonly MetaVerdict NoModel = new(false, double.NaN, true, 1.0, string.Empty);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The meta-model's seat at the decision table.
|
||||
/// <para>
|
||||
/// For every pair with a promoted champion in the database this keeps the rolling
|
||||
/// history the features need, scores each entry the statistical rule proposes, refuses
|
||||
/// the ones the model rates below the configured probability, sizes the rest by its
|
||||
/// confidence, and watches the live feature distribution for drift against the
|
||||
/// training set. It can only say no or "smaller": the rule still decides what and
|
||||
/// which way, which is what keeps the model from inventing trades nobody validated.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// A pair without a champion passes straight through — the bot then trades exactly as
|
||||
/// it did before the model existed. The champion is re-read at every recalibration, so
|
||||
/// a model promoted by the research tool takes effect without a restart, and a model
|
||||
/// suspended for drift gets another chance when the next one is loaded.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class MetaGate
|
||||
{
|
||||
private readonly MlOptions _options;
|
||||
private readonly StorageDb? _storage;
|
||||
private readonly FeatureRegistry _registry = FeatureRegistry.Default();
|
||||
private readonly Dictionary<string, Slot> _slots = new(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
private sealed class Slot(PairFeatureBuffer buffer)
|
||||
{
|
||||
public PairFeatureBuffer Buffer { get; } = buffer;
|
||||
|
||||
public GbdtModel? Model { get; set; }
|
||||
|
||||
public long ModelId { get; set; }
|
||||
|
||||
public DriftMonitor? Drift { get; set; }
|
||||
|
||||
public bool Suspended { get; set; }
|
||||
|
||||
public int Scored { get; set; }
|
||||
}
|
||||
|
||||
public MetaGate(MlOptions options, StorageDb? storage, IReadOnlyList<PairPipeline> pairs)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(pairs);
|
||||
|
||||
_options = options;
|
||||
_storage = storage;
|
||||
|
||||
foreach (PairPipeline pair in pairs)
|
||||
{
|
||||
_slots[pair.Name] = new Slot(new PairFeatureBuffer(pair.Strategy.Window));
|
||||
}
|
||||
|
||||
if (!options.Enabled)
|
||||
{
|
||||
Log.Info("meta-modello disattivato dalla configurazione: opera la sola regola statistica");
|
||||
}
|
||||
else if (storage is null)
|
||||
{
|
||||
Log.Info("meta-modello non disponibile: senza database locale non c'è un campione da leggere");
|
||||
}
|
||||
}
|
||||
|
||||
public bool IsActive => _options.Enabled && _storage is not null;
|
||||
|
||||
/// <summary>Whether a usable champion is loaded for the pair.</summary>
|
||||
public bool HasModel(string pair) =>
|
||||
_slots.TryGetValue(pair, out Slot? slot) && slot.Model is not null && !slot.Suspended;
|
||||
|
||||
/// <summary>(Re)loads every pair's champion from the database.</summary>
|
||||
public void ReloadChampions()
|
||||
{
|
||||
if (!IsActive)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
foreach ((string pair, Slot slot) in _slots)
|
||||
{
|
||||
try
|
||||
{
|
||||
StorageDb.ModelRecord? champion = _storage!.LoadChampion(pair);
|
||||
if (champion is null)
|
||||
{
|
||||
if (slot.Model is not null)
|
||||
{
|
||||
Log.Warn($"[{pair}] il campione è stato rimosso dal database: torno alla sola regola statistica");
|
||||
}
|
||||
|
||||
slot.Model = null;
|
||||
slot.ModelId = 0;
|
||||
slot.Drift = null;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (champion.Id == slot.ModelId && slot.Model is not null)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
GbdtModel model = GbdtModel.FromBytes(champion.Blob);
|
||||
if (!SameFeatures(model.FeatureNames))
|
||||
{
|
||||
Log.Error($"[{pair}] il campione #{champion.Id} usa feature diverse da quelle del bot: lo ignoro", null);
|
||||
continue;
|
||||
}
|
||||
|
||||
slot.Model = model;
|
||||
slot.ModelId = champion.Id;
|
||||
slot.Suspended = false;
|
||||
slot.Scored = 0;
|
||||
slot.Drift = BuildDrift(champion);
|
||||
|
||||
Log.Info(string.Create(CultureInfo.InvariantCulture,
|
||||
$"[{pair}] campione #{champion.Id} caricato ({model.TreeCount} alberi, " +
|
||||
$"{model.FeatureNames.Length} feature, del {champion.CreatedUtc:yyyy-MM-dd}); " +
|
||||
$"soglia {_options.MinProbability:F2}"));
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
Log.Error($"[{pair}] impossibile caricare il campione: opero senza", ex);
|
||||
slot.Model = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Replaces a pair's feature history with a warm-up download and the strategy's
|
||||
/// z-score after each of its bars.
|
||||
/// </summary>
|
||||
public void Seed(
|
||||
string pair, IReadOnlyList<Bar> barsA, IReadOnlyList<Bar> barsB, in PairCalibration calibration,
|
||||
IReadOnlyList<double> zScores)
|
||||
{
|
||||
if (_slots.TryGetValue(pair, out Slot? slot))
|
||||
{
|
||||
slot.Buffer.Seed(barsA, barsB, calibration, zScores);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Appends the bar just closed with the z-score the strategy computed on it. Call
|
||||
/// once per aligned bar, before <see cref="Judge"/>.
|
||||
/// </summary>
|
||||
public void Observe(string pair, in Bar barA, in Bar barB, in PairCalibration calibration, double zScore)
|
||||
{
|
||||
if (_slots.TryGetValue(pair, out Slot? slot))
|
||||
{
|
||||
slot.Buffer.Append(barA, barB, calibration, zScore);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Scores a proposed entry. Returns <see cref="MetaVerdict.NoModel"/> when there is
|
||||
/// nothing to say, so the caller's path is the same with and without a model.
|
||||
/// </summary>
|
||||
public MetaVerdict Judge(string pair, double latencyMs)
|
||||
{
|
||||
if (!IsActive || !_slots.TryGetValue(pair, out Slot? slot) || slot.Model is null || slot.Suspended)
|
||||
{
|
||||
return MetaVerdict.NoModel;
|
||||
}
|
||||
|
||||
double[]? row = slot.Buffer.Latest(_registry, latencyMs);
|
||||
if (row is null)
|
||||
{
|
||||
return new MetaVerdict(true, double.NaN, true, 1.0,
|
||||
$"storia insufficiente per le feature ({slot.Buffer.Count}/{slot.Buffer.Capacity} barre): passo senza giudizio");
|
||||
}
|
||||
|
||||
double p;
|
||||
try
|
||||
{
|
||||
p = slot.Model.Predict(row);
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
Log.Error($"[{pair}] il meta-modello ha fallito la predizione: passo senza giudizio", ex);
|
||||
return MetaVerdict.NoModel;
|
||||
}
|
||||
|
||||
slot.Scored++;
|
||||
CheckDrift(pair, slot, row);
|
||||
|
||||
if (p < _options.MinProbability)
|
||||
{
|
||||
return new MetaVerdict(true, p, false, 0, string.Create(CultureInfo.InvariantCulture,
|
||||
$"il meta-modello dà {p:P1} di probabilità che l'ingresso ripaghi i costi, " +
|
||||
$"sotto la soglia {_options.MinProbability:P0}"));
|
||||
}
|
||||
|
||||
double fraction = 1.0;
|
||||
if (_options.SizeByProbability)
|
||||
{
|
||||
fraction = Math.Clamp(Performance.BetSizeFromProbability(p), _options.MinSizeFraction, 1.0);
|
||||
}
|
||||
|
||||
string note = fraction < 1
|
||||
? string.Create(CultureInfo.InvariantCulture,
|
||||
$"il meta-modello dà {p:P1}: ingresso ammesso con size al {fraction:P0}")
|
||||
: string.Create(CultureInfo.InvariantCulture, $"il meta-modello dà {p:P1}: ingresso ammesso");
|
||||
|
||||
return new MetaVerdict(true, p, true, fraction, note);
|
||||
}
|
||||
|
||||
private void CheckDrift(string pair, Slot slot, double[] row)
|
||||
{
|
||||
if (slot.Drift is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
slot.Drift.Observe(row);
|
||||
if (!slot.Drift.Due(_options.DriftCheckBars))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
List<DriftReport> reports = slot.Drift.Check();
|
||||
int alerts = 0;
|
||||
DateTime now = DateTime.UtcNow;
|
||||
|
||||
foreach (DriftReport r in reports)
|
||||
{
|
||||
if (r.Alert)
|
||||
{
|
||||
alerts++;
|
||||
Log.Warn($"[{pair}] DERIVA {r.Motivazione}");
|
||||
}
|
||||
|
||||
_storage?.InsertDrift(now, pair, r.Feature, r.Psi, r.Ks, r.Alert, r.Motivazione);
|
||||
}
|
||||
|
||||
Log.Info(string.Create(CultureInfo.InvariantCulture,
|
||||
$"[{pair}] controllo deriva su {slot.Drift.RecentRows} segnali: {alerts} feature su {reports.Count} " +
|
||||
$"oltre PSI {_options.DriftPsiAlert:F2}"));
|
||||
|
||||
if (_options.DriftAlertsToSuspend > 0 && alerts >= _options.DriftAlertsToSuspend)
|
||||
{
|
||||
slot.Suspended = true;
|
||||
Log.Warn($"[{pair}] meta-modello SOSPESO: {alerts} feature in deriva. Fino alla prossima " +
|
||||
"ricalibrazione opera la sola regola statistica, come prima del modello.");
|
||||
}
|
||||
}
|
||||
|
||||
private DriftMonitor? BuildDrift(StorageDb.ModelRecord champion)
|
||||
{
|
||||
if (champion.DatasetId <= 0)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var dataset = _storage!.LoadDataset(champion.DatasetId);
|
||||
if (dataset is null || dataset.Value.Rows.Count < 50)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
double[][] rows = [.. dataset.Value.Rows.Select(static r => r.Features)];
|
||||
return new DriftMonitor(champion.FeatureNames, rows, psiAlert: _options.DriftPsiAlert);
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
Log.Warn($"dataset #{champion.DatasetId} non leggibile: nessun controllo di deriva ({ex.Message})");
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private bool SameFeatures(IReadOnlyList<string> names)
|
||||
{
|
||||
IReadOnlyList<string> mine = _registry.Names;
|
||||
if (names.Count != mine.Count)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
for (int i = 0; i < names.Count; i++)
|
||||
{
|
||||
if (names[i] != mine[i])
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
}
|
||||
@@ -21,7 +21,8 @@ public readonly record struct PairIntent(
|
||||
double ReferenceA,
|
||||
double ReferenceB,
|
||||
long EnqueuedTimestamp,
|
||||
long DecisionId);
|
||||
long DecisionId,
|
||||
double SizeFraction = 1.0);
|
||||
|
||||
/// <summary>What happened to an intent. Every one of these ends up in the log.</summary>
|
||||
public enum IntentOutcome : byte
|
||||
@@ -224,6 +225,18 @@ public sealed class PairRouter(
|
||||
return;
|
||||
}
|
||||
|
||||
// The meta-model's bet size scales the risk engine's notional, never the
|
||||
// other way round: the risk engine sets the ceiling, the model may only
|
||||
// come in under it.
|
||||
if (intent.SizeFraction is > 0 and < 1)
|
||||
{
|
||||
verdict = verdict with
|
||||
{
|
||||
NotionalA = verdict.NotionalA * intent.SizeFraction,
|
||||
NotionalB = verdict.NotionalB * intent.SizeFraction,
|
||||
};
|
||||
}
|
||||
|
||||
// Notional to a sendable size. The exchange's lot step is the last authority:
|
||||
// a quantity that is right to seven decimals and wrong on the step is simply
|
||||
// rejected, and on a two-leg trade a rejection is worse than a refusal.
|
||||
|
||||
@@ -1,33 +1,39 @@
|
||||
using System.Buffers;
|
||||
using System.Globalization;
|
||||
using System.IO;
|
||||
using System.Text.Json;
|
||||
using System.Text;
|
||||
using Encelado.Bot.Diagnostics;
|
||||
using Encelado.Core.Journal;
|
||||
using Encelado.Core.Market;
|
||||
|
||||
namespace Encelado.Bot.Engine;
|
||||
|
||||
/// <summary>
|
||||
/// Append-only JSONL record of everything the engine decided. Separate from the log
|
||||
/// on purpose: this file is meant to be parsed (pandas, jq, a spreadsheet) when
|
||||
/// reviewing why the bot did what it did.
|
||||
/// Append-only record of every order event: entries, fills, exits, rejections and the
|
||||
/// occasional unwind. Separate from the log on purpose — this is meant to be opened in
|
||||
/// a spreadsheet or joined against the decision log, not read line by line.
|
||||
/// <para>
|
||||
/// A <c>;</c>-separated table with a <c>motivazione</c> column, like every other table
|
||||
/// the bot writes. It used to be JSON lines, which no spreadsheet opens and which put
|
||||
/// the one column a human reads last, inside quotes.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class TradeJournal : IDisposable
|
||||
{
|
||||
private readonly FileStream? _stream;
|
||||
public const string Header =
|
||||
"timestampUtc;event;symbol;side;quantity;price;orderId;stop;target;equity;realizedPnl;motivazione";
|
||||
|
||||
private readonly StreamWriter? _writer;
|
||||
private readonly IJournalSink? _sink;
|
||||
private readonly Lock _gate = new();
|
||||
private readonly StringBuilder _row = new(256);
|
||||
|
||||
public TradeJournal(string? path)
|
||||
public TradeJournal(string? path, IJournalSink? sink = null)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(path))
|
||||
{
|
||||
return;
|
||||
_writer = CsvTable.Open(path, Header);
|
||||
_sink = sink;
|
||||
}
|
||||
|
||||
string full = Path.GetFullPath(path);
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(full)!);
|
||||
_stream = new FileStream(full, FileMode.Append, FileAccess.Write, FileShare.ReadWrite, 4096);
|
||||
}
|
||||
|
||||
public bool IsEnabled => _stream is not null;
|
||||
public bool IsEnabled => _writer is not null;
|
||||
|
||||
public void Record(
|
||||
string @event,
|
||||
@@ -42,67 +48,73 @@ public sealed class TradeJournal : IDisposable
|
||||
double? equity = null,
|
||||
double? realizedPnl = null)
|
||||
{
|
||||
if (_stream is null)
|
||||
string sideText = side switch { Side.Buy => "BUY", Side.Sell => "SELL", _ => string.Empty };
|
||||
|
||||
_sink?.Trade(new TradeRow(
|
||||
DateTime.UtcNow, @event, symbol, sideText, quantity, price, orderId,
|
||||
stopPrice, targetPrice, equity, realizedPnl, reason));
|
||||
|
||||
if (_writer is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
ArrayBufferWriter<byte> buffer = new(320);
|
||||
using (Utf8JsonWriter w = new(buffer))
|
||||
{
|
||||
w.WriteStartObject();
|
||||
w.WriteString("ts", DateTime.UtcNow.ToString("O"));
|
||||
w.WriteString("event", @event);
|
||||
w.WriteString("symbol", symbol);
|
||||
w.WriteString("side", side switch { Side.Buy => "buy", Side.Sell => "sell", _ => "none" });
|
||||
w.WriteNumber("qty", Round(quantity));
|
||||
w.WriteNumber("price", Round(price));
|
||||
|
||||
if (stopPrice is { } stop && !double.IsNaN(stop))
|
||||
{
|
||||
w.WriteNumber("stop", Round(stop));
|
||||
}
|
||||
|
||||
if (targetPrice is { } target && !double.IsNaN(target))
|
||||
{
|
||||
w.WriteNumber("target", Round(target));
|
||||
}
|
||||
|
||||
if (equity is { } eq)
|
||||
{
|
||||
w.WriteNumber("equity", Round(eq));
|
||||
}
|
||||
|
||||
if (realizedPnl is { } pnl)
|
||||
{
|
||||
w.WriteNumber("realizedPnl", Round(pnl));
|
||||
}
|
||||
|
||||
if (!string.IsNullOrEmpty(orderId))
|
||||
{
|
||||
w.WriteString("orderId", orderId);
|
||||
}
|
||||
|
||||
w.WriteString("reason", reason);
|
||||
w.WriteEndObject();
|
||||
}
|
||||
|
||||
lock (_gate)
|
||||
{
|
||||
_stream.Write(buffer.WrittenSpan);
|
||||
_stream.WriteByte((byte)'\n');
|
||||
_stream.Flush();
|
||||
_row.Clear();
|
||||
|
||||
Add(DateTime.UtcNow.ToString("O", CultureInfo.InvariantCulture));
|
||||
Add(@event);
|
||||
Add(symbol);
|
||||
Add(sideText);
|
||||
Add(quantity);
|
||||
Add(price);
|
||||
Add(orderId);
|
||||
Add(stopPrice);
|
||||
Add(targetPrice);
|
||||
Add(equity);
|
||||
Add(realizedPnl);
|
||||
CsvTable.AppendField(_row, reason);
|
||||
|
||||
_writer.WriteLine(_row.ToString());
|
||||
|
||||
// Every row is an order event, and there are few of them: flushing each one
|
||||
// costs nothing and means a crash never loses the record of a fill.
|
||||
_writer.Flush();
|
||||
}
|
||||
}
|
||||
|
||||
private static double Round(double value) =>
|
||||
double.IsNaN(value) || double.IsInfinity(value) ? 0 : Math.Round(value, 6);
|
||||
private void Add(string? value)
|
||||
{
|
||||
CsvTable.AppendField(_row, value);
|
||||
_row.Append(AnalyticsLog.Separator);
|
||||
}
|
||||
|
||||
private void Add(double value)
|
||||
{
|
||||
if (double.IsFinite(value))
|
||||
{
|
||||
_row.Append(Math.Round(value, 8).ToString("G10", CultureInfo.InvariantCulture));
|
||||
}
|
||||
|
||||
_row.Append(AnalyticsLog.Separator);
|
||||
}
|
||||
|
||||
private void Add(double? value)
|
||||
{
|
||||
if (value is { } v && double.IsFinite(v))
|
||||
{
|
||||
_row.Append(Math.Round(v, 8).ToString("G10", CultureInfo.InvariantCulture));
|
||||
}
|
||||
|
||||
_row.Append(AnalyticsLog.Separator);
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
_stream?.Dispose();
|
||||
_writer?.Dispose();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -11,6 +11,7 @@ using Encelado.Core.Portfolio;
|
||||
using Encelado.Core.Risk;
|
||||
using Encelado.Core.Statistics;
|
||||
using Encelado.Core.Strategies.Pairs;
|
||||
using Encelado.Storage;
|
||||
|
||||
namespace Encelado.Bot.Engine;
|
||||
|
||||
@@ -30,6 +31,8 @@ public sealed class TradingEngine : IAsyncDisposable
|
||||
private readonly Metrics _metrics = new();
|
||||
private readonly TradeJournal _journal;
|
||||
private readonly AnalyticsLog _analytics;
|
||||
private readonly StorageDb? _storage;
|
||||
private readonly MetaGate _meta;
|
||||
private readonly MarketDataStream _marketData;
|
||||
private readonly UserDataStream _userData;
|
||||
private readonly PairRouter _router;
|
||||
@@ -57,8 +60,9 @@ public sealed class TradingEngine : IAsyncDisposable
|
||||
|
||||
_client = new BinanceFuturesClient(config.Binance);
|
||||
_risk = new RiskEngine(config.Risk);
|
||||
_journal = new TradeJournal(config.Logging.ResolvePath(config.Logging.TradeJournal));
|
||||
_analytics = new AnalyticsLog(config.Logging);
|
||||
_storage = OpenStorage(config.Storage);
|
||||
_journal = new TradeJournal(config.Logging.ResolvePath(config.Logging.TradeJournal), _storage);
|
||||
_analytics = new AnalyticsLog(config.Logging, _storage);
|
||||
_logMarketData = config.Logging.LogMarketData && Log.IsEnabled(Logging.LogLevel.Trace);
|
||||
|
||||
_marketData = new MarketDataStream(config.Binance, _symbols, _timeFrame);
|
||||
@@ -90,13 +94,44 @@ public sealed class TradingEngine : IAsyncDisposable
|
||||
}
|
||||
|
||||
_pairs = [.. pairs];
|
||||
_meta = new MetaGate(config.Ml, _storage, _pairs);
|
||||
|
||||
_router = new PairRouter(
|
||||
_client, _book, _risk, _account, config.Engine, _metrics, _journal, _analytics, _pairs);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Opens the local database, or explains why there is none. Never fatal: a bot that
|
||||
/// cannot write its journal to SQLite still has the CSV, and still trades.
|
||||
/// </summary>
|
||||
private static StorageDb? OpenStorage(StorageOptions options)
|
||||
{
|
||||
if (!options.Enabled)
|
||||
{
|
||||
Log.Info("database locale disattivato: journal solo su CSV, apprendimento automatico spento");
|
||||
return null;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
StorageDb db = new(options.ResolvePath());
|
||||
db.OnError = static (what, ex) => Log.Error($"database: {what}", ex);
|
||||
Log.Info($"database locale: {db.Path}");
|
||||
return db;
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
Log.Error($"impossibile aprire il database locale in {options.ResolvePath()}: continuo con i soli CSV", ex);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
public Metrics Metrics => _metrics;
|
||||
|
||||
public StorageDb? Storage => _storage;
|
||||
|
||||
public MetaGate Meta => _meta;
|
||||
|
||||
public PortfolioBook Book => _book;
|
||||
|
||||
public AccountState Account => _account;
|
||||
@@ -266,9 +301,17 @@ public sealed class TradingEngine : IAsyncDisposable
|
||||
{
|
||||
long t0 = Stopwatch.GetTimestamp();
|
||||
|
||||
// Enough history for both jobs: the regression needs calibrationBars, the
|
||||
// z-score window needs warmupBars, and they are independent settings.
|
||||
int bars = Math.Max(_config.Engine.CalibrationBars, _config.Engine.WarmupBars);
|
||||
// Enough history for every job: the regression needs calibrationBars, the
|
||||
// configured warm-up needs warmupBars, and each strategy's own window needs a
|
||||
// few more than its length before it can report ready. Taking the largest, plus
|
||||
// slack, is what stops a pair from sitting one bar short after every calibration.
|
||||
int widestWindow = 0;
|
||||
foreach (PairPipeline pair in _pairs)
|
||||
{
|
||||
widestWindow = Math.Max(widestWindow, pair.Strategy.WarmupBars + 3);
|
||||
}
|
||||
|
||||
int bars = Math.Max(Math.Max(_config.Engine.CalibrationBars, _config.Engine.WarmupBars), widestWindow);
|
||||
|
||||
Dictionary<string, List<Bar>> history = new(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
@@ -276,9 +319,16 @@ public sealed class TradingEngine : IAsyncDisposable
|
||||
{
|
||||
try
|
||||
{
|
||||
history[symbol] = await _client
|
||||
List<Bar> fetched = await _client
|
||||
.GetKlinesAsync(symbol, _timeFrame, bars, null, ct)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
history[symbol] = fetched;
|
||||
|
||||
if (fetched.Count > 0)
|
||||
{
|
||||
LegFor(_marketData.Symbols.Resolve(symbol))?.SeedHistory(fetched[^1]);
|
||||
}
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
@@ -287,6 +337,7 @@ public sealed class TradingEngine : IAsyncDisposable
|
||||
}
|
||||
|
||||
int calibrated = 0;
|
||||
_meta.ReloadChampions();
|
||||
|
||||
foreach (PairPipeline pair in _pairs)
|
||||
{
|
||||
@@ -327,12 +378,18 @@ public sealed class TradingEngine : IAsyncDisposable
|
||||
|
||||
// Refill the rolling window from the same history the fit used, discarding
|
||||
// the signals: these bars are in the past and were already lived through.
|
||||
// The z-score after each bar is kept for the meta-model's feature history,
|
||||
// so its "z" is the strategy's and not a second opinion.
|
||||
PairPositionView flat = PairPositionView.Flat;
|
||||
double[] zScores = new double[closeA.Length];
|
||||
for (int i = 0; i < closeA.Length; i++)
|
||||
{
|
||||
_ = pair.Strategy.OnBar(closeA[i], closeB[i], flat);
|
||||
zScores[i] = pair.Strategy.IsReady ? pair.Strategy.ZScore : double.NaN;
|
||||
}
|
||||
|
||||
_meta.Seed(pair.Name, barsA, barsB, calibration, zScores);
|
||||
|
||||
// The last historical bar is also the last bar the stream will not resend,
|
||||
// so it must not be decided on again when the first live close arrives.
|
||||
pair.MarkDecided(barsA[^1].TimeUtc);
|
||||
@@ -444,7 +501,16 @@ public sealed class TradingEngine : IAsyncDisposable
|
||||
PairSignal signal = pair.Strategy.OnBar(barA.Close, barB.Close, position);
|
||||
_metrics.BarToSignal.RecordSince(startedAt);
|
||||
|
||||
// The meta-model sees every bar (its features need the history) and speaks only
|
||||
// on entries. Its answer travels with the decision row as probPredetta.
|
||||
_meta.Observe(pair.Name, barA, barB, pair.Strategy.Calibration,
|
||||
pair.Strategy.IsReady ? pair.Strategy.ZScore : double.NaN);
|
||||
MetaVerdict meta = signal.IsEntry
|
||||
? _meta.Judge(pair.Name, Stopwatch.GetElapsedTime(startedAt).TotalMilliseconds)
|
||||
: MetaVerdict.NoModel;
|
||||
|
||||
long decisionId = _analytics.NextDecisionId();
|
||||
string explanation = pair.Strategy.Explain(position);
|
||||
|
||||
_analytics.Decision(
|
||||
decisionId,
|
||||
@@ -456,9 +522,11 @@ public sealed class TradingEngine : IAsyncDisposable
|
||||
pair.Strategy,
|
||||
position,
|
||||
signal,
|
||||
explanation,
|
||||
pair.WorstQuoteAge == TimeSpan.MaxValue ? -1 : pair.WorstQuoteAge.TotalSeconds,
|
||||
_account.Equity,
|
||||
_risk.IsHalted);
|
||||
_risk.IsHalted,
|
||||
meta.Probability);
|
||||
|
||||
// The decision itself, always at info: this is the answer to "why did it (not)
|
||||
// trade", and burying it at debug is how that question became unanswerable.
|
||||
@@ -469,12 +537,23 @@ public sealed class TradingEngine : IAsyncDisposable
|
||||
PairSignalKind.EnterShortSpread =>
|
||||
$"VENDO {pair.LegA.Symbol} / COMPRO {pair.LegB.Symbol} — {signal.Reason}",
|
||||
PairSignalKind.Exit => $"CHIUDO — {signal.Reason}",
|
||||
_ => $"NON FACCIO NULLA — {pair.Strategy.Explain(position)}",
|
||||
_ => $"NON FACCIO NULLA — {explanation}",
|
||||
};
|
||||
|
||||
Log.Info($"[{pair.Name}] {verdict}");
|
||||
|
||||
Publish(pair, signal, position, decisionId);
|
||||
if (meta.HasModel && meta.Note.Length > 0)
|
||||
{
|
||||
Log.Info($"[{pair.Name}] meta-modello: {meta.Note}");
|
||||
}
|
||||
|
||||
if (signal.IsEntry && !meta.Allowed)
|
||||
{
|
||||
Log.Warn($"[{pair.Name}] NON ENTRO — {meta.Note}");
|
||||
return;
|
||||
}
|
||||
|
||||
Publish(pair, signal, position, decisionId, meta.SizeFraction);
|
||||
}
|
||||
|
||||
private static LegSnapshot Snapshot(LegState leg) => new(
|
||||
@@ -492,7 +571,8 @@ public sealed class TradingEngine : IAsyncDisposable
|
||||
/// that makes a bot look broken to the person watching it.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private void Publish(PairPipeline pair, in PairSignal signal, in PairPositionView position, long decisionId)
|
||||
private void Publish(
|
||||
PairPipeline pair, in PairSignal signal, in PairPositionView position, long decisionId, double sizeFraction = 1.0)
|
||||
{
|
||||
if (signal.Kind == PairSignalKind.None)
|
||||
{
|
||||
@@ -544,10 +624,10 @@ public sealed class TradingEngine : IAsyncDisposable
|
||||
}
|
||||
|
||||
_metrics.CountSignal();
|
||||
Enqueue(pair, signal, decisionId, isExit: false);
|
||||
Enqueue(pair, signal, decisionId, isExit: false, sizeFraction);
|
||||
}
|
||||
|
||||
private void Enqueue(PairPipeline pair, in PairSignal signal, long decisionId, bool isExit)
|
||||
private void Enqueue(PairPipeline pair, in PairSignal signal, long decisionId, bool isExit, double sizeFraction = 1.0)
|
||||
{
|
||||
PairIntent intent = new(
|
||||
pair.Id,
|
||||
@@ -556,7 +636,8 @@ public sealed class TradingEngine : IAsyncDisposable
|
||||
pair.LegA.LastPrice,
|
||||
pair.LegB.LastPrice,
|
||||
Stopwatch.GetTimestamp(),
|
||||
decisionId);
|
||||
decisionId,
|
||||
sizeFraction);
|
||||
|
||||
if (_router.Enqueue(intent))
|
||||
{
|
||||
@@ -1032,5 +1113,6 @@ public sealed class TradingEngine : IAsyncDisposable
|
||||
_client.Dispose();
|
||||
_journal.Dispose();
|
||||
_analytics.Dispose();
|
||||
_storage?.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
using System.Diagnostics;
|
||||
using System.Globalization;
|
||||
using System.IO;
|
||||
using System.Runtime.CompilerServices;
|
||||
using System.Text;
|
||||
using System.Threading.Channels;
|
||||
using Encelado.Bot.Configuration;
|
||||
@@ -18,13 +19,32 @@ public enum LogLevel : byte
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Non-blocking logger. Call sites only enqueue; a single background writer does the
|
||||
/// formatting and the I/O, so a burst of ticks never stalls the decode loop on a
|
||||
/// console write.
|
||||
/// Non-blocking, structured logger. Call sites only enqueue; a single background writer
|
||||
/// does the formatting and the I/O, so a burst of ticks never stalls the decode loop on
|
||||
/// a disk write.
|
||||
/// <para>
|
||||
/// The file is a <c>;</c>-separated table with a header, not a stream of prose:
|
||||
/// </para>
|
||||
/// <code>
|
||||
/// timestamp;level;source;subject;event;message;exception;stack
|
||||
/// </code>
|
||||
/// <para>
|
||||
/// This is what makes a problem findable after the fact. The previous format carried a
|
||||
/// time with no date across a file that spanned weeks, no stack trace on errors, and no
|
||||
/// way to isolate one pair — so answering "what went wrong with SOL/AVAX on the 30th"
|
||||
/// meant reading three megabytes. Now it is one filter: <c>;ERR;</c> for every failure,
|
||||
/// a pair name in the <c>subject</c> column for one instrument, and the file opens in a
|
||||
/// spreadsheet as-is. The <c>source</c> is the class that wrote the line, captured from
|
||||
/// the compiler for free; the <c>subject</c> is lifted from the <c>[ETHUSDT/BTCUSDT]</c>
|
||||
/// prefix the code already uses, so no call site had to change to become searchable.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class Log
|
||||
{
|
||||
private const string AnsiReset = "\u001b[0m";
|
||||
private const string AnsiReset = "[0m";
|
||||
|
||||
/// <summary>The columns, in order. Written once at the top of every new file.</summary>
|
||||
public const string Header = "timestamp;level;source;subject;event;message;exception;stack";
|
||||
|
||||
private static readonly Channel<Entry> Queue = Channel.CreateUnbounded<Entry>(
|
||||
new UnboundedChannelOptions { SingleReader = true, SingleWriter = false });
|
||||
@@ -81,9 +101,26 @@ public static class Log
|
||||
try
|
||||
{
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(_path))!);
|
||||
|
||||
// A file left over from the previous, unstructured format is moved aside
|
||||
// rather than appended to: a table whose first thousand rows have no columns
|
||||
// is not a table, and the old lines are still there under the .old name.
|
||||
if (File.Exists(_path) && !HasHeader(_path))
|
||||
{
|
||||
string aside = Path.ChangeExtension(_path, ".old" + Path.GetExtension(_path));
|
||||
File.Move(_path, aside, overwrite: true);
|
||||
}
|
||||
|
||||
FileStream stream = new(_path, FileMode.Append, FileAccess.Write, FileShare.ReadWrite, 8192);
|
||||
_written = stream.Length;
|
||||
_file = new StreamWriter(stream, Encoding.UTF8) { AutoFlush = false };
|
||||
_file = new StreamWriter(stream, new UTF8Encoding(encoderShouldEmitUTF8Identifier: false)) { AutoFlush = false };
|
||||
|
||||
if (_written == 0)
|
||||
{
|
||||
_file.WriteLine(Header);
|
||||
_file.Flush();
|
||||
_written = Header.Length + Environment.NewLine.Length;
|
||||
}
|
||||
}
|
||||
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
@@ -92,10 +129,24 @@ public static class Log
|
||||
}
|
||||
}
|
||||
|
||||
private static bool HasHeader(string path)
|
||||
{
|
||||
try
|
||||
{
|
||||
using StreamReader reader = new(path, Encoding.UTF8, detectEncodingFromByteOrderMarks: true);
|
||||
string? first = reader.ReadLine();
|
||||
return first is null || first.StartsWith("timestamp;level;", StringComparison.Ordinal);
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Rolls <c>encelado.log</c> to <c>encelado.1.log</c>, shifting the older ones up and
|
||||
/// dropping the oldest. Keeps a long-running bot from filling the disk while still
|
||||
/// preserving recent history for analysis.
|
||||
/// preserving recent history for analysis. Both the size and the count are settings.
|
||||
/// </summary>
|
||||
private static void RotateIfNeeded()
|
||||
{
|
||||
@@ -142,18 +193,35 @@ public static class Log
|
||||
}
|
||||
}
|
||||
|
||||
public static void Trace(string message) => Write(LogLevel.Trace, message, null);
|
||||
// -----------------------------------------------------------------------
|
||||
// Call sites
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
public static void Debug(string message) => Write(LogLevel.Debug, message, null);
|
||||
public static void Trace(string message, [CallerFilePath] string caller = "") =>
|
||||
Write(LogLevel.Trace, message, null, null, caller);
|
||||
|
||||
public static void Info(string message) => Write(LogLevel.Info, message, null);
|
||||
public static void Debug(string message, [CallerFilePath] string caller = "") =>
|
||||
Write(LogLevel.Debug, message, null, null, caller);
|
||||
|
||||
public static void Warn(string message) => Write(LogLevel.Warn, message, null);
|
||||
public static void Info(string message, [CallerFilePath] string caller = "") =>
|
||||
Write(LogLevel.Info, message, null, null, caller);
|
||||
|
||||
public static void Error(string message, Exception? exception = null) =>
|
||||
Write(LogLevel.Error, message, exception);
|
||||
public static void Warn(string message, [CallerFilePath] string caller = "") =>
|
||||
Write(LogLevel.Warn, message, null, null, caller);
|
||||
|
||||
private static void Write(LogLevel level, string message, Exception? exception)
|
||||
public static void Error(string message, Exception? exception = null, [CallerFilePath] string caller = "") =>
|
||||
Write(LogLevel.Error, message, exception, null, caller);
|
||||
|
||||
/// <summary>
|
||||
/// A line with an explicit event code — <c>order.submitted</c>, <c>entry.refused</c>,
|
||||
/// <c>kill-switch</c> — so the moments that matter can be counted and filtered
|
||||
/// without matching on prose.
|
||||
/// </summary>
|
||||
public static void Event(LogLevel level, string eventCode, string message, Exception? exception = null,
|
||||
[CallerFilePath] string caller = "") =>
|
||||
Write(level, message, exception, eventCode, caller);
|
||||
|
||||
private static void Write(LogLevel level, string message, Exception? exception, string? eventCode, string caller)
|
||||
{
|
||||
if (level < _minimum)
|
||||
{
|
||||
@@ -162,7 +230,7 @@ public static class Log
|
||||
|
||||
DateTime now = DateTime.Now;
|
||||
|
||||
if (Queue.Writer.TryWrite(new Entry(now, level, message, exception)))
|
||||
if (Queue.Writer.TryWrite(new Entry(now, level, message, exception, eventCode, SourceOf(caller))))
|
||||
{
|
||||
Interlocked.Increment(ref _enqueued);
|
||||
}
|
||||
@@ -185,6 +253,25 @@ public static class Log
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>The class that logged, from the compiler-supplied file path. Free at the call site.</summary>
|
||||
private static string SourceOf(string callerPath)
|
||||
{
|
||||
if (string.IsNullOrEmpty(callerPath))
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
ReadOnlySpan<char> span = callerPath.AsSpan();
|
||||
int slash = span.LastIndexOfAny('\\', '/');
|
||||
if (slash >= 0)
|
||||
{
|
||||
span = span[(slash + 1)..];
|
||||
}
|
||||
|
||||
int dot = span.IndexOf('.');
|
||||
return dot > 0 ? span[..dot].ToString() : span.ToString();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Waits until the writer has caught up with everything enqueued so far. Needed
|
||||
/// before writing to the console directly — an interactive prompt must not be
|
||||
@@ -217,9 +304,13 @@ public static class Log
|
||||
}
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Writer
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
private static async Task WriteLoopAsync()
|
||||
{
|
||||
StringBuilder sb = new(256);
|
||||
StringBuilder sb = new(512);
|
||||
long lastFlush = Stopwatch.GetTimestamp();
|
||||
|
||||
await foreach (Entry entry in Queue.Reader.ReadAllAsync().ConfigureAwait(false))
|
||||
@@ -228,39 +319,16 @@ public static class Log
|
||||
// full disk should cost log lines, not the trading session.
|
||||
try
|
||||
{
|
||||
sb.Clear();
|
||||
sb.Append(entry.Timestamp.ToString("HH:mm:ss.fff", CultureInfo.InvariantCulture))
|
||||
.Append(' ')
|
||||
.Append(Tag(entry.Level))
|
||||
.Append(' ')
|
||||
.Append(entry.Message);
|
||||
|
||||
if (entry.Exception is not null)
|
||||
{
|
||||
sb.Append(" | ").Append(entry.Exception.GetType().Name)
|
||||
.Append(": ").Append(entry.Exception.Message);
|
||||
}
|
||||
|
||||
string line = sb.ToString();
|
||||
|
||||
if (_console)
|
||||
{
|
||||
if (_colors)
|
||||
{
|
||||
Console.Out.Write(Color(entry.Level));
|
||||
Console.Out.Write(line);
|
||||
Console.Out.WriteLine(AnsiReset);
|
||||
}
|
||||
else
|
||||
{
|
||||
Console.Out.WriteLine(line);
|
||||
}
|
||||
WriteConsole(entry, sb);
|
||||
}
|
||||
|
||||
if (_file is not null)
|
||||
{
|
||||
await _file.WriteLineAsync(line).ConfigureAwait(false);
|
||||
_written += line.Length + Environment.NewLine.Length;
|
||||
string row = FormatRow(entry, sb);
|
||||
await _file.WriteLineAsync(row).ConfigureAwait(false);
|
||||
_written += row.Length + Environment.NewLine.Length;
|
||||
|
||||
// Warnings and errors flush immediately; routine lines are batched so
|
||||
// a busy session is not one fsync per entry.
|
||||
@@ -292,6 +360,157 @@ public static class Log
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// One row of the table. The pair or symbol is lifted out of the message's leading
|
||||
/// <c>[…]</c> tag into its own column; everything else is CSV-quoted only when it
|
||||
/// has to be, so the common line stays readable in a plain editor.
|
||||
/// </summary>
|
||||
internal static string FormatRow(in Entry entry, StringBuilder sb)
|
||||
{
|
||||
sb.Clear();
|
||||
|
||||
(string subject, string message) = SplitSubject(entry.Message);
|
||||
|
||||
sb.Append(entry.Timestamp.ToString("yyyy-MM-dd'T'HH:mm:ss.fffzzz", CultureInfo.InvariantCulture)).Append(';');
|
||||
sb.Append(Tag(entry.Level)).Append(';');
|
||||
Quote(sb, entry.Source).Append(';');
|
||||
Quote(sb, subject).Append(';');
|
||||
Quote(sb, entry.EventCode ?? string.Empty).Append(';');
|
||||
Quote(sb, message).Append(';');
|
||||
|
||||
if (entry.Exception is { } ex)
|
||||
{
|
||||
Quote(sb, DescribeException(ex)).Append(';');
|
||||
Quote(sb, FlattenStack(ex));
|
||||
}
|
||||
else
|
||||
{
|
||||
sb.Append(';');
|
||||
}
|
||||
|
||||
return sb.ToString();
|
||||
}
|
||||
|
||||
private static void WriteConsole(in Entry entry, StringBuilder sb)
|
||||
{
|
||||
sb.Clear();
|
||||
sb.Append(entry.Timestamp.ToString("HH:mm:ss.fff", CultureInfo.InvariantCulture))
|
||||
.Append(' ').Append(Tag(entry.Level))
|
||||
.Append(' ').Append(entry.Message);
|
||||
|
||||
if (entry.Exception is not null)
|
||||
{
|
||||
sb.Append(" | ").Append(entry.Exception.GetType().Name)
|
||||
.Append(": ").Append(entry.Exception.Message);
|
||||
}
|
||||
|
||||
if (_colors)
|
||||
{
|
||||
Console.Out.Write(Color(entry.Level));
|
||||
Console.Out.Write(sb.ToString());
|
||||
Console.Out.WriteLine(AnsiReset);
|
||||
}
|
||||
else
|
||||
{
|
||||
Console.Out.WriteLine(sb.ToString());
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Splits <c>[ETHUSDT/BTCUSDT] rest</c> into its tag and the rest.</summary>
|
||||
internal static (string Subject, string Message) SplitSubject(string message)
|
||||
{
|
||||
if (message.Length > 2 && message[0] == '[')
|
||||
{
|
||||
int close = message.IndexOf(']', StringComparison.Ordinal);
|
||||
if (close > 1 && close < 40)
|
||||
{
|
||||
string subject = message[1..close];
|
||||
string rest = message[(close + 1)..].TrimStart();
|
||||
return (subject, rest);
|
||||
}
|
||||
}
|
||||
|
||||
return (string.Empty, message);
|
||||
}
|
||||
|
||||
/// <summary>CSV quoting for a <c>;</c>-separated file: only when the value needs it.</summary>
|
||||
private static StringBuilder Quote(StringBuilder sb, string value)
|
||||
{
|
||||
if (value.Length == 0)
|
||||
{
|
||||
return sb;
|
||||
}
|
||||
|
||||
if (value.AsSpan().IndexOfAny(";\"\r\n") < 0)
|
||||
{
|
||||
return sb.Append(value);
|
||||
}
|
||||
|
||||
sb.Append('"');
|
||||
foreach (char c in value)
|
||||
{
|
||||
if (c == '"')
|
||||
{
|
||||
sb.Append('"');
|
||||
}
|
||||
|
||||
sb.Append(c is '\r' or '\n' ? ' ' : c);
|
||||
}
|
||||
|
||||
return sb.Append('"');
|
||||
}
|
||||
|
||||
/// <summary>Type and message of the exception and every inner one, innermost last.</summary>
|
||||
private static string DescribeException(Exception ex)
|
||||
{
|
||||
StringBuilder sb = new(128);
|
||||
Exception? current = ex;
|
||||
while (current is not null)
|
||||
{
|
||||
if (sb.Length > 0)
|
||||
{
|
||||
sb.Append(" <- ");
|
||||
}
|
||||
|
||||
sb.Append(current.GetType().Name).Append(": ").Append(current.Message);
|
||||
current = current.InnerException;
|
||||
}
|
||||
|
||||
return sb.ToString();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The stack trace on one line, frames separated by <c> | </c>. A stack is what turns
|
||||
/// "execution failed" into a line number, and the old format never wrote one.
|
||||
/// </summary>
|
||||
private static string FlattenStack(Exception ex)
|
||||
{
|
||||
string? stack = ex.StackTrace ?? ex.InnerException?.StackTrace;
|
||||
if (string.IsNullOrWhiteSpace(stack))
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
StringBuilder sb = new(stack.Length);
|
||||
foreach (string line in stack.Split('\n'))
|
||||
{
|
||||
string frame = line.Trim();
|
||||
if (frame.Length == 0)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (sb.Length > 0)
|
||||
{
|
||||
sb.Append(" | ");
|
||||
}
|
||||
|
||||
sb.Append(frame);
|
||||
}
|
||||
|
||||
return sb.ToString();
|
||||
}
|
||||
|
||||
/// <summary>Drains the queue and flushes the file. Call before the process exits.</summary>
|
||||
public static async Task ShutdownAsync()
|
||||
{
|
||||
@@ -356,13 +575,19 @@ public static class Log
|
||||
|
||||
private static string Color(LogLevel level) => level switch
|
||||
{
|
||||
LogLevel.Trace => "\u001b[90m",
|
||||
LogLevel.Debug => "\u001b[36m",
|
||||
LogLevel.Trace => "[90m",
|
||||
LogLevel.Debug => "[36m",
|
||||
LogLevel.Info => AnsiReset,
|
||||
LogLevel.Warn => "\u001b[33m",
|
||||
LogLevel.Error => "\u001b[31m",
|
||||
LogLevel.Warn => "[33m",
|
||||
LogLevel.Error => "[31m",
|
||||
_ => AnsiReset,
|
||||
};
|
||||
|
||||
private readonly record struct Entry(DateTime Timestamp, LogLevel Level, string Message, Exception? Exception);
|
||||
internal readonly record struct Entry(
|
||||
DateTime Timestamp,
|
||||
LogLevel Level,
|
||||
string Message,
|
||||
Exception? Exception,
|
||||
string? EventCode,
|
||||
string Source);
|
||||
}
|
||||
|
||||
@@ -138,6 +138,11 @@ public partial class MainWindow : Window, IUiActions
|
||||
Log.Info($"Encelado avviato — configurazione {App.ConfigPath}");
|
||||
Log.Info($"log in {_config.Logging.ResolveDirectory()}");
|
||||
|
||||
if (App.SeedNote is { } seeded)
|
||||
{
|
||||
Log.Warn(seeded);
|
||||
}
|
||||
|
||||
CredentialLookup lookup = CredentialResolver.Resolve(_config);
|
||||
if (!lookup.Found)
|
||||
{
|
||||
|
||||
@@ -217,7 +217,7 @@ public partial class SettingsPage : UserControl
|
||||
"Cosa viene scritto in questa cartella:",
|
||||
string.Empty,
|
||||
Named(logging.File, "log dell'applicazione"),
|
||||
Named(logging.TradeJournal, "diario delle operazioni"),
|
||||
Named(logging.TradeJournal, "diario delle operazioni (CSV con ;)"),
|
||||
Named(logging.DecisionLog, "una riga per barra valutata"),
|
||||
Named(logging.ExecutionLog, "una riga per segnale arrivato agli ordini"),
|
||||
];
|
||||
|
||||
@@ -96,6 +96,9 @@ public sealed class SettingField : INotifyPropertyChanged
|
||||
/// <summary>Unit suffix shown after the box, e.g. "%" or "secondi".</summary>
|
||||
public string Suffix { get; init; } = string.Empty;
|
||||
|
||||
/// <summary>A text field that means "use the default" when left blank.</summary>
|
||||
public bool AllowEmpty { get; init; }
|
||||
|
||||
public double Minimum { get; init; } = double.NegativeInfinity;
|
||||
|
||||
public double Maximum { get; init; } = double.PositiveInfinity;
|
||||
@@ -217,7 +220,7 @@ public sealed class SettingField : INotifyPropertyChanged
|
||||
|
||||
case SettingKind.Text:
|
||||
default:
|
||||
if (_value.Length == 0)
|
||||
if (_value.Length == 0 && !AllowEmpty)
|
||||
{
|
||||
Error = "non può essere vuoto";
|
||||
}
|
||||
|
||||
@@ -33,6 +33,7 @@ public static class SettingsCatalogue
|
||||
groups.Add(Exchange(config));
|
||||
groups.Add(Engine(config));
|
||||
groups.Add(Logging(config));
|
||||
groups.Add(Learning(config));
|
||||
|
||||
// Validated immediately, not on the first edit. A value that has become invalid
|
||||
// because an update changed the rules has to announce itself when the page opens:
|
||||
@@ -673,6 +674,68 @@ public static class SettingsCatalogue
|
||||
"mostra il mercato muoversi e uno che sembra fermo.",
|
||||
});
|
||||
|
||||
g.Fields.Add(new SettingField
|
||||
{
|
||||
Path = "storage.enabled",
|
||||
Label = "Database locale",
|
||||
Initial = config.Storage.Enabled ? "sì" : "no",
|
||||
Kind = SettingKind.Boolean,
|
||||
Tooltip =
|
||||
"Con 'sì' ogni tabella del journal viene scritta anche nel database SQLite, " +
|
||||
"e da lì il bot legge i modelli e il campione per coppia.\n\n" +
|
||||
"Con 'no' restano solo i file CSV e la parte di apprendimento automatico è " +
|
||||
"disattivata: senza database non c'è dove tenere dataset e modelli.",
|
||||
});
|
||||
|
||||
g.Fields.Add(new SettingField
|
||||
{
|
||||
Path = "storage.databasePath",
|
||||
Label = "Percorso del database",
|
||||
Initial = config.Storage.DatabasePath,
|
||||
Kind = SettingKind.Text,
|
||||
AllowEmpty = true,
|
||||
Tooltip =
|
||||
"Il file SQLite. Vuoto significa %ProgramData%\\Encelado\\encelado.db: una " +
|
||||
"cartella di sistema, separata sia da Documenti (dove sta la configurazione e " +
|
||||
"che viene sincronizzata e copiata) sia dai dati utente (dove stanno le chiavi " +
|
||||
"cifrate).\n\n" +
|
||||
"Un percorso assoluto viene usato così com'è. Il file e la cartella vengono " +
|
||||
"creati al primo avvio.",
|
||||
});
|
||||
|
||||
g.Fields.Add(new SettingField
|
||||
{
|
||||
Path = "logging.maxFileSizeMb",
|
||||
Label = "Dimensione massima del log",
|
||||
Initial = config.Logging.MaxFileSizeMb.ToString(CultureInfo.CurrentCulture),
|
||||
Kind = SettingKind.Integer,
|
||||
Suffix = "MB (0 = nessuna rotazione)",
|
||||
Minimum = 0,
|
||||
Maximum = 4096,
|
||||
Tooltip =
|
||||
"Quando encelado.log supera questa dimensione viene chiuso e rinominato in " +
|
||||
"encelado.1.log, e se ne apre uno nuovo. I file precedenti scalano di un " +
|
||||
"numero: encelado.1.log diventa encelado.2.log e così via.\n\n" +
|
||||
"Con 0 il file cresce senza limite. Un bot lasciato girare per settimane a " +
|
||||
"livello 'debug' produce centinaia di megabyte: la rotazione è quello che " +
|
||||
"impedisce al disco di riempirsi senza perdere la storia recente.",
|
||||
});
|
||||
|
||||
g.Fields.Add(new SettingField
|
||||
{
|
||||
Path = "logging.maxFiles",
|
||||
Label = "File di log conservati",
|
||||
Initial = config.Logging.MaxFiles.ToString(CultureInfo.CurrentCulture),
|
||||
Kind = SettingKind.Integer,
|
||||
Suffix = "file ruotati",
|
||||
Minimum = 1,
|
||||
Maximum = 500,
|
||||
Tooltip =
|
||||
"Quanti file ruotati tenere prima di cancellare il più vecchio.\n\n" +
|
||||
"Moltiplicato per la dimensione massima dà lo spazio totale che i log possono " +
|
||||
"occupare: con 32 MB e 10 file, al massimo 352 MB compreso quello attivo.",
|
||||
});
|
||||
|
||||
g.Fields.Add(new SettingField
|
||||
{
|
||||
Path = "logging.bufferedLines",
|
||||
@@ -690,6 +753,103 @@ public static class SettingsCatalogue
|
||||
return g;
|
||||
}
|
||||
|
||||
private static SettingGroup Learning(BotConfig config)
|
||||
{
|
||||
SettingGroup g = new(
|
||||
"Apprendimento automatico",
|
||||
"Il meta-modello: un classificatore addestrato sugli esiti dei segnali passati che " +
|
||||
"può rifiutare un ingresso o ridurne la size. Non propone mai operazioni. Si addestra " +
|
||||
"con lo strumento di ricerca (backtest dataset / train) e vive nel database locale.");
|
||||
|
||||
g.Fields.Add(new SettingField
|
||||
{
|
||||
Path = "ml.enabled",
|
||||
Label = "Consulta il meta-modello",
|
||||
Initial = config.Ml.Enabled ? "sì" : "no",
|
||||
Kind = SettingKind.Boolean,
|
||||
Tooltip =
|
||||
"Con 'sì', prima di ogni ingresso il bot chiede al campione della coppia la " +
|
||||
"probabilità che l'operazione ripaghi i costi, e rifiuta quelle sotto soglia.\n\n" +
|
||||
"Senza un campione nel database questa voce non cambia nulla: il bot opera la " +
|
||||
"sola regola statistica, come prima.",
|
||||
});
|
||||
|
||||
g.Fields.Add(new SettingField
|
||||
{
|
||||
Path = "ml.minProbability",
|
||||
Label = "Probabilità minima",
|
||||
Initial = SettingField.Format(config.Ml.MinProbability, SettingKind.Number),
|
||||
Kind = SettingKind.Number,
|
||||
Minimum = 0.5,
|
||||
Maximum = 0.99,
|
||||
Tooltip =
|
||||
"Sotto questa probabilità predetta l'ingresso viene rifiutato.\n\n" +
|
||||
"0.55 vuol dire: opero solo quando il modello vede un vantaggio, anche piccolo, " +
|
||||
"rispetto a lanciare una moneta. Alzarla riduce le operazioni e, se il modello " +
|
||||
"è buono, ne migliora la qualità; non può scendere sotto 0.5.",
|
||||
});
|
||||
|
||||
g.Fields.Add(new SettingField
|
||||
{
|
||||
Path = "ml.sizeByProbability",
|
||||
Label = "Size in base alla convinzione",
|
||||
Initial = config.Ml.SizeByProbability ? "sì" : "no",
|
||||
Kind = SettingKind.Boolean,
|
||||
Tooltip =
|
||||
"Con 'sì' la size è proporzionale alla convinzione del modello (la regola " +
|
||||
"2·Φ(z)−1 di López de Prado), mai sopra quella decisa dal risk engine e mai " +
|
||||
"sotto la frazione minima. Con 'no' è tutto o niente.",
|
||||
});
|
||||
|
||||
g.Fields.Add(new SettingField
|
||||
{
|
||||
Path = "ml.minSizeFraction",
|
||||
Label = "Frazione minima di size",
|
||||
Initial = SettingField.Format(config.Ml.MinSizeFraction, SettingKind.Number),
|
||||
Kind = SettingKind.Number,
|
||||
Minimum = 0.05,
|
||||
Maximum = 1,
|
||||
Tooltip =
|
||||
"La size più piccola che il modello può lasciare, come frazione di quella del " +
|
||||
"risk engine. Sotto un quarto le commissioni fisse e i lotti minimi di Binance " +
|
||||
"iniziano a pesare più dell'informazione.",
|
||||
});
|
||||
|
||||
g.Fields.Add(new SettingField
|
||||
{
|
||||
Path = "ml.driftPsiAlert",
|
||||
Label = "Soglia di deriva (PSI)",
|
||||
Initial = SettingField.Format(config.Ml.DriftPsiAlert, SettingKind.Number),
|
||||
Kind = SettingKind.Number,
|
||||
Minimum = 0.01,
|
||||
Maximum = 5,
|
||||
Tooltip =
|
||||
"Ogni tot segnali le feature viste dal vivo vengono confrontate con quelle di " +
|
||||
"addestramento. Un Population Stability Index sopra questa soglia segna la " +
|
||||
"feature come 'in deriva'.\n\n0.25 è la convenzione: sopra, la popolazione è " +
|
||||
"cambiata e le probabilità del modello descrivono un mercato che non c'è più.",
|
||||
});
|
||||
|
||||
g.Fields.Add(new SettingField
|
||||
{
|
||||
Path = "ml.driftAlertsToSuspend",
|
||||
Label = "Feature in deriva per sospendere",
|
||||
Initial = config.Ml.DriftAlertsToSuspend.ToString(CultureInfo.CurrentCulture),
|
||||
Kind = SettingKind.Integer,
|
||||
Suffix = "feature (0 = mai)",
|
||||
Minimum = 0,
|
||||
Maximum = 100,
|
||||
Tooltip =
|
||||
"Con questo numero di feature in deriva il modello viene sospeso fino alla " +
|
||||
"ricalibrazione successiva, e nel frattempo opera la sola regola statistica.\n\n" +
|
||||
"È la demozione automatica della guida: un modello che vede un mercato diverso " +
|
||||
"da quello su cui ha imparato smette di avere voce, senza che nessuno debba " +
|
||||
"accorgersene di notte.",
|
||||
});
|
||||
|
||||
return g;
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------------
|
||||
|
||||
private static string Number(PairConfig pair, string key, double fallback, SettingKind kind)
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
using System.Globalization;
|
||||
using Encelado.Core.Market;
|
||||
using Encelado.Core.Statistics;
|
||||
|
||||
namespace Encelado.Core.Backtest;
|
||||
|
||||
/// <summary>
|
||||
/// The full set of numbers a return series is judged by, computed once and the same
|
||||
/// way for every strategy in the project — baseline, pairs, meta-model, RL — so that a
|
||||
/// comparison between them is a comparison and not an accident of who rounded what.
|
||||
/// </summary>
|
||||
public sealed record ReturnMetrics(
|
||||
int Periods,
|
||||
double PeriodsPerYear,
|
||||
double NetReturn,
|
||||
double Cagr,
|
||||
double AnnualSharpe,
|
||||
double AnnualSortino,
|
||||
double MaxDrawdown,
|
||||
double Calmar,
|
||||
double Skewness,
|
||||
double Kurtosis,
|
||||
double Psr,
|
||||
double Dsr,
|
||||
int Trials,
|
||||
int Trades)
|
||||
{
|
||||
public double Years => PeriodsPerYear > 0 ? Periods / PeriodsPerYear : 0;
|
||||
|
||||
/// <summary>
|
||||
/// Per-period returns in, everything out. <paramref name="trials"/> is how many
|
||||
/// configurations were tried before this one was reported — the deflated Sharpe is
|
||||
/// meaningless without it, and one is only honest when nothing else was looked at.
|
||||
/// </summary>
|
||||
public static ReturnMetrics From(
|
||||
IReadOnlyList<double> periodReturns, double periodsPerYear, int trades, int trials = 1,
|
||||
double trialSharpeVariance = 0)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(periodReturns);
|
||||
|
||||
int n = periodReturns.Count;
|
||||
if (n < 2)
|
||||
{
|
||||
return new ReturnMetrics(n, periodsPerYear, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, trials, trades);
|
||||
}
|
||||
|
||||
double[] equity = new double[n + 1];
|
||||
equity[0] = 1;
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
equity[i + 1] = equity[i] * (1 + periodReturns[i]);
|
||||
}
|
||||
|
||||
double years = periodsPerYear > 0 ? n / periodsPerYear : 0;
|
||||
double net = equity[^1] - 1;
|
||||
double cagr = Performance.Cagr(1, equity[^1], years);
|
||||
double sharpe = Performance.Sharpe(periodReturns);
|
||||
double sortino = Performance.Sortino(periodReturns);
|
||||
double dd = Performance.MaxDrawdown(equity);
|
||||
double skew = Performance.Skewness(periodReturns);
|
||||
double kurt = Performance.Kurtosis(periodReturns);
|
||||
|
||||
// The PSR and DSR are computed on the per-period Sharpe, with the per-period
|
||||
// observation count — annualising first would inflate the confidence by the
|
||||
// square root of the periods per year, which is the classic way to make a noisy
|
||||
// intraday series look certain.
|
||||
double psr = Performance.ProbabilisticSharpe(sharpe, 0, n, skew, kurt);
|
||||
double dsr = Performance.DeflatedSharpe(sharpe, n, skew, kurt, Math.Max(1, trials), trialSharpeVariance);
|
||||
|
||||
return new ReturnMetrics(
|
||||
n, periodsPerYear, net, cagr,
|
||||
Performance.Annualise(sharpe, periodsPerYear),
|
||||
Performance.Annualise(sortino, periodsPerYear),
|
||||
dd, Performance.Calmar(cagr, dd), skew, kurt, psr, dsr, trials, trades);
|
||||
}
|
||||
|
||||
public string Describe() => string.Create(CultureInfo.InvariantCulture,
|
||||
$"netto {NetReturn:P2}, CAGR {Cagr:P2}, Sharpe {AnnualSharpe:F2}, Sortino {AnnualSortino:F2}, " +
|
||||
$"DD {MaxDrawdown:P2}, Calmar {Calmar:F2}, PSR {Psr:F3}, DSR {Dsr:F3} su {Trials} prove, {Trades} op");
|
||||
}
|
||||
|
||||
/// <summary>What a baseline did, and the returns it did it with.</summary>
|
||||
public sealed record BaselineReport(string Name, ReturnMetrics Metrics, double[] PeriodReturns, string Motivazione);
|
||||
|
||||
/// <summary>
|
||||
/// The two strategies every other one has to beat before it is worth discussing.
|
||||
/// <para>
|
||||
/// Buy-and-hold is the null hypothesis of a bull market: any long-biased system that
|
||||
/// cannot beat it after costs has added nothing but fees. The moving-average crossover
|
||||
/// is the null hypothesis of a trend rule: it has been public for sixty years, costs
|
||||
/// nothing, and a "machine learning" model that fails to beat it has learnt nothing
|
||||
/// worth the name. Both pay the same costs the real strategies do.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class Baseline
|
||||
{
|
||||
/// <summary>Long from the first bar to the last, one round trip of costs.</summary>
|
||||
public static BaselineReport BuyAndHold(IReadOnlyList<Bar> bars, double costPerSide, double periodsPerYear)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(bars);
|
||||
|
||||
double[] returns = new double[Math.Max(0, bars.Count - 1)];
|
||||
for (int i = 1; i < bars.Count; i++)
|
||||
{
|
||||
returns[i - 1] = bars[i - 1].Close > 0 ? (bars[i].Close / bars[i - 1].Close) - 1 : 0;
|
||||
}
|
||||
|
||||
if (returns.Length > 0)
|
||||
{
|
||||
returns[0] -= costPerSide;
|
||||
returns[^1] -= costPerSide;
|
||||
}
|
||||
|
||||
ReturnMetrics m = ReturnMetrics.From(returns, periodsPerYear, 1);
|
||||
return new BaselineReport("buy&hold", m, returns, string.Create(CultureInfo.InvariantCulture,
|
||||
$"comprato alla prima barra e tenuto: {m.Describe()}; è il rendimento del mercato stesso, " +
|
||||
$"al netto di un solo giro di costi ({costPerSide:P3} per lato)"));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Long when the fast average is above the slow one, flat otherwise. Decisions are
|
||||
/// taken on the close and applied from the next bar, so nothing is bought at a price
|
||||
/// that was not yet known.
|
||||
/// </summary>
|
||||
public static BaselineReport SmaCrossover(
|
||||
IReadOnlyList<Bar> bars, int fast, int slow, double costPerSide, double periodsPerYear, bool allowShort = false)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(bars);
|
||||
|
||||
if (fast < 1 || slow <= fast)
|
||||
{
|
||||
throw new ArgumentException("la media lenta deve essere più lunga di quella veloce");
|
||||
}
|
||||
|
||||
int n = bars.Count;
|
||||
double[] returns = new double[Math.Max(0, n - 1)];
|
||||
double sumFast = 0, sumSlow = 0;
|
||||
int position = 0;
|
||||
int trades = 0;
|
||||
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
double close = bars[i].Close;
|
||||
|
||||
if (i > 0)
|
||||
{
|
||||
double r = bars[i - 1].Close > 0 ? (close / bars[i - 1].Close) - 1 : 0;
|
||||
returns[i - 1] = position * r;
|
||||
}
|
||||
|
||||
sumFast += close;
|
||||
sumSlow += close;
|
||||
if (i >= fast)
|
||||
{
|
||||
sumFast -= bars[i - fast].Close;
|
||||
}
|
||||
|
||||
if (i >= slow)
|
||||
{
|
||||
sumSlow -= bars[i - slow].Close;
|
||||
}
|
||||
|
||||
if (i < slow - 1)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
double meanFast = sumFast / fast;
|
||||
double meanSlow = sumSlow / slow;
|
||||
int wanted = meanFast > meanSlow ? 1 : allowShort ? -1 : 0;
|
||||
|
||||
if (wanted != position && i > 0)
|
||||
{
|
||||
// Each leg crossed costs one side; a reversal crosses two.
|
||||
returns[i - 1] -= costPerSide * Math.Abs(wanted - position);
|
||||
position = wanted;
|
||||
trades++;
|
||||
}
|
||||
}
|
||||
|
||||
ReturnMetrics m = ReturnMetrics.From(returns, periodsPerYear, trades);
|
||||
return new BaselineReport(
|
||||
string.Create(CultureInfo.InvariantCulture, $"sma {fast}/{slow}{(allowShort ? " long/short" : "")}"),
|
||||
m, returns, string.Create(CultureInfo.InvariantCulture,
|
||||
$"incrocio di medie mobili {fast}/{slow}: {m.Describe()}; regola pubblica da sessant'anni, " +
|
||||
$"se un modello non la batte non ha imparato nulla"));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
using Encelado.Core.Market;
|
||||
using Encelado.Core.Strategies;
|
||||
using Encelado.Core.Strategies.Pairs;
|
||||
|
||||
namespace Encelado.Core.Backtest;
|
||||
|
||||
/// <summary>
|
||||
/// What the primary model said at every bar, and the calibration it said it with.
|
||||
/// <para>
|
||||
/// <see cref="Sides"/> is +1 where the strategy would open a long spread, −1 a short
|
||||
/// one, 0 where it would do nothing — always evaluated <i>as if flat</i>, so a bar
|
||||
/// inside an open position still reports whether the entry rule fires there. That is
|
||||
/// the set meta-labelling needs: every moment the primary would have acted, not only
|
||||
/// the ones where the replay happened to be free to act.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed record PrimaryReplayResult(
|
||||
int[] Sides,
|
||||
double[] Beta,
|
||||
double[] Alpha,
|
||||
double[] HalfLife,
|
||||
double[] ZScore,
|
||||
double[] PValue,
|
||||
int Start)
|
||||
{
|
||||
public int Count => Sides.Length;
|
||||
|
||||
public int SignalCount
|
||||
{
|
||||
get
|
||||
{
|
||||
int n = 0;
|
||||
foreach (int s in Sides)
|
||||
{
|
||||
if (s != 0)
|
||||
{
|
||||
n++;
|
||||
}
|
||||
}
|
||||
|
||||
return n;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Replays the statistical-arbitrage rule as a <i>signal generator</i> rather than as a
|
||||
/// portfolio: same walk-forward calibrations, same warm-ups, same strategy object as
|
||||
/// <see cref="PairBacktest.Run"/>, but the output is the per-bar opinion of the rule
|
||||
/// instead of the equity curve of trading it.
|
||||
/// </summary>
|
||||
public static class PrimaryReplay
|
||||
{
|
||||
public static PrimaryReplayResult Run(
|
||||
IReadOnlyList<Bar> barsA,
|
||||
IReadOnlyList<Bar> barsB,
|
||||
StrategyParameters parameters,
|
||||
PairBacktestSettings settings,
|
||||
PairCalibrationSchedule? schedule = null,
|
||||
CancellationToken ct = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(barsA);
|
||||
ArgumentNullException.ThrowIfNull(barsB);
|
||||
ArgumentNullException.ThrowIfNull(parameters);
|
||||
ArgumentNullException.ThrowIfNull(settings);
|
||||
|
||||
int n = Math.Min(barsA.Count, barsB.Count);
|
||||
int[] sides = new int[n];
|
||||
double[] beta = new double[n];
|
||||
double[] alpha = new double[n];
|
||||
double[] halfLife = new double[n];
|
||||
double[] z = new double[n];
|
||||
double[] pValue = new double[n];
|
||||
Array.Fill(beta, double.NaN);
|
||||
Array.Fill(alpha, double.NaN);
|
||||
Array.Fill(halfLife, double.NaN);
|
||||
Array.Fill(z, double.NaN);
|
||||
Array.Fill(pValue, double.NaN);
|
||||
|
||||
int start = settings.CalibrationBars;
|
||||
if (n < start + 50)
|
||||
{
|
||||
return new PrimaryReplayResult(sides, beta, alpha, halfLife, z, pValue, n);
|
||||
}
|
||||
|
||||
StatArbStrategy strategy = new(parameters);
|
||||
|
||||
double[] closeA = new double[n];
|
||||
double[] closeB = new double[n];
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
closeA[i] = barsA[i].Close;
|
||||
closeB[i] = barsB[i].Close;
|
||||
}
|
||||
|
||||
int stride = Math.Max(1, settings.RecalibrateEveryBars);
|
||||
|
||||
for (int t = start; t < n; t++)
|
||||
{
|
||||
ct.ThrowIfCancellationRequested();
|
||||
|
||||
if ((t - start) % stride == 0)
|
||||
{
|
||||
PairCalibration fit = schedule is not null && schedule.TryGet(t, out PairCalibration cached)
|
||||
? cached
|
||||
: PairBacktest.Calibrate(closeA, closeB, t - settings.CalibrationBars, t);
|
||||
|
||||
strategy.Recalibrate(fit);
|
||||
|
||||
if (!strategy.IsReady)
|
||||
{
|
||||
int from = Math.Max(0, t - settings.CalibrationBars);
|
||||
for (int i = from; i < t; i++)
|
||||
{
|
||||
_ = strategy.OnBar(closeA[i], closeB[i], PairPositionView.Flat);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
PairSignal signal = strategy.OnBar(closeA[t], closeB[t], PairPositionView.Flat);
|
||||
|
||||
sides[t] = signal.Kind switch
|
||||
{
|
||||
PairSignalKind.EnterLongSpread => 1,
|
||||
PairSignalKind.EnterShortSpread => -1,
|
||||
_ => 0,
|
||||
};
|
||||
|
||||
PairCalibration c = strategy.Calibration;
|
||||
if (c.IsValid)
|
||||
{
|
||||
beta[t] = c.Beta;
|
||||
alpha[t] = c.Alpha;
|
||||
halfLife[t] = c.HalfLifeBars;
|
||||
pValue[t] = c.PValue;
|
||||
}
|
||||
|
||||
z[t] = strategy.IsReady ? strategy.ZScore : double.NaN;
|
||||
}
|
||||
|
||||
return new PrimaryReplayResult(sides, beta, alpha, halfLife, z, pValue, start);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
namespace Encelado.Core.Journal;
|
||||
|
||||
/// <summary>
|
||||
/// One evaluated bar for one pair, as the decision log records it.
|
||||
/// </summary>
|
||||
public sealed record DecisionRow(
|
||||
DateTime BarTimeUtc,
|
||||
long DecisionId,
|
||||
string Pair,
|
||||
string SymbolA,
|
||||
string SymbolB,
|
||||
bool Ready,
|
||||
double CloseA,
|
||||
double CloseB,
|
||||
double Spread,
|
||||
double ZScore,
|
||||
double Beta,
|
||||
double Alpha,
|
||||
double PValue,
|
||||
double AdfStatistic,
|
||||
double HalfLife,
|
||||
bool Cointegrated,
|
||||
double QuantityA,
|
||||
double QuantityB,
|
||||
double EntryZ,
|
||||
int BarsHeld,
|
||||
string Signal,
|
||||
double SpreadPctA,
|
||||
double SpreadPctB,
|
||||
double FundingA,
|
||||
double FundingB,
|
||||
double Equity,
|
||||
bool Halted,
|
||||
double MetaProbability,
|
||||
string Motivazione);
|
||||
|
||||
/// <summary>One signal that reached the order path, and what became of it.</summary>
|
||||
public sealed record ExecutionRow(
|
||||
DateTime TimestampUtc,
|
||||
long DecisionId,
|
||||
string Pair,
|
||||
string Phase,
|
||||
bool Approved,
|
||||
string Reason,
|
||||
string SideA,
|
||||
double NotionalA,
|
||||
double QuantityA,
|
||||
double PriceA,
|
||||
string SideB,
|
||||
double NotionalB,
|
||||
double QuantityB,
|
||||
double PriceB,
|
||||
double NetFundingRate,
|
||||
double Equity,
|
||||
double AvailableBalance,
|
||||
double GrossExposure,
|
||||
int OpenPairs,
|
||||
string OrderIdA,
|
||||
string OrderIdB,
|
||||
string Error,
|
||||
double LatencyMs,
|
||||
string Motivazione);
|
||||
|
||||
/// <summary>One order event: entry, fill, exit, rejection, unwind.</summary>
|
||||
public sealed record TradeRow(
|
||||
DateTime TimestampUtc,
|
||||
string Event,
|
||||
string Symbol,
|
||||
string Side,
|
||||
double Quantity,
|
||||
double Price,
|
||||
string? OrderId,
|
||||
double? Stop,
|
||||
double? Target,
|
||||
double? Equity,
|
||||
double? RealizedPnl,
|
||||
string Motivazione);
|
||||
|
||||
/// <summary>
|
||||
/// Somewhere a journal row can be written besides the CSV on disk.
|
||||
/// <para>
|
||||
/// The bot writes every table twice — a <c>;</c>-separated file the operator can open,
|
||||
/// and a database the tools can query — and this is the seam between the two. The CSV
|
||||
/// writers stay ignorant of what the second destination is, and a failure there must
|
||||
/// cost a row, never a decision: implementations swallow their own errors.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public interface IJournalSink
|
||||
{
|
||||
void Decision(DecisionRow row);
|
||||
|
||||
void Execution(ExecutionRow row);
|
||||
|
||||
void Trade(TradeRow row);
|
||||
}
|
||||
@@ -0,0 +1,138 @@
|
||||
using Encelado.Core.Market;
|
||||
|
||||
namespace Encelado.Core.Ml;
|
||||
|
||||
/// <summary>
|
||||
/// Bars sampled by activity instead of by the clock.
|
||||
/// <para>
|
||||
/// A five-minute bar at 04:00 UTC and one at 14:30 during a US data release are the
|
||||
/// same object to a time series and completely different objects to a market: one
|
||||
/// holds a hundred trades, the other twenty thousand. Sampling by dollars traded, by
|
||||
/// volume, or by order-flow imbalance produces bars whose returns are closer to
|
||||
/// independent and closer to normal — which is what every model downstream quietly
|
||||
/// assumes. This is López de Prado's first chapter, and the reason it is first.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class AlternativeBars
|
||||
{
|
||||
/// <summary>One bar per <paramref name="dollarThreshold"/> of quote volume traded.</summary>
|
||||
public static List<Bar> DollarBars(IReadOnlyList<Bar> timeBars, double dollarThreshold) =>
|
||||
Sample(timeBars, static b => b.Volume * (b.Vwap > 0 ? b.Vwap : b.Close), dollarThreshold);
|
||||
|
||||
/// <summary>One bar per <paramref name="volumeThreshold"/> of base volume traded.</summary>
|
||||
public static List<Bar> VolumeBars(IReadOnlyList<Bar> timeBars, double volumeThreshold) =>
|
||||
Sample(timeBars, static b => b.Volume, volumeThreshold);
|
||||
|
||||
/// <summary>
|
||||
/// One bar each time the accumulated order-flow imbalance — taker buys minus taker
|
||||
/// sells — exceeds its own recent typical size.
|
||||
/// <para>
|
||||
/// The threshold is not fixed: it tracks an exponentially weighted average of the
|
||||
/// imbalance at which previous bars closed, so the sampling adapts as the market gets
|
||||
/// louder or quieter. <paramref name="initialThreshold"/> only seeds it.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static List<Bar> ImbalanceBars(IReadOnlyList<Bar> timeBars, double initialThreshold, int span = 20)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(timeBars);
|
||||
|
||||
if (initialThreshold <= 0)
|
||||
{
|
||||
throw new ArgumentOutOfRangeException(nameof(initialThreshold), "deve essere positiva");
|
||||
}
|
||||
|
||||
List<Bar> bars = [];
|
||||
double threshold = initialThreshold;
|
||||
double alpha = 2.0 / (Math.Max(1, span) + 1);
|
||||
double imbalance = 0;
|
||||
Accumulator acc = default;
|
||||
|
||||
foreach (Bar bar in timeBars)
|
||||
{
|
||||
acc.Add(bar);
|
||||
imbalance += bar.HasOrderFlow ? bar.Delta : 0;
|
||||
|
||||
if (Math.Abs(imbalance) >= threshold)
|
||||
{
|
||||
bars.Add(acc.Emit());
|
||||
threshold = ((1 - alpha) * threshold) + (alpha * Math.Abs(imbalance));
|
||||
imbalance = 0;
|
||||
acc = default;
|
||||
}
|
||||
}
|
||||
|
||||
return bars;
|
||||
}
|
||||
|
||||
private static List<Bar> Sample(IReadOnlyList<Bar> timeBars, Func<Bar, double> measure, double threshold)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(timeBars);
|
||||
|
||||
if (threshold <= 0)
|
||||
{
|
||||
throw new ArgumentOutOfRangeException(nameof(threshold), "deve essere positiva");
|
||||
}
|
||||
|
||||
List<Bar> bars = [];
|
||||
double accumulated = 0;
|
||||
Accumulator acc = default;
|
||||
|
||||
foreach (Bar bar in timeBars)
|
||||
{
|
||||
acc.Add(bar);
|
||||
accumulated += measure(bar);
|
||||
|
||||
if (accumulated >= threshold)
|
||||
{
|
||||
bars.Add(acc.Emit());
|
||||
accumulated = 0;
|
||||
acc = default;
|
||||
}
|
||||
}
|
||||
|
||||
return bars;
|
||||
}
|
||||
|
||||
/// <summary>Folds consecutive time bars into one, weighting the VWAP by volume.</summary>
|
||||
private struct Accumulator
|
||||
{
|
||||
private bool _has;
|
||||
private DateTime _time;
|
||||
private double _open;
|
||||
private double _high;
|
||||
private double _low;
|
||||
private double _close;
|
||||
private double _volume;
|
||||
private double _quoteVolume;
|
||||
private int _trades;
|
||||
private double _takerBuy;
|
||||
|
||||
public void Add(in Bar bar)
|
||||
{
|
||||
if (!_has)
|
||||
{
|
||||
_has = true;
|
||||
_time = bar.TimeUtc;
|
||||
_open = bar.Open;
|
||||
_high = bar.High;
|
||||
_low = bar.Low;
|
||||
}
|
||||
else
|
||||
{
|
||||
_high = Math.Max(_high, bar.High);
|
||||
_low = Math.Min(_low, bar.Low);
|
||||
}
|
||||
|
||||
_close = bar.Close;
|
||||
_volume += bar.Volume;
|
||||
_quoteVolume += bar.Volume * (bar.Vwap > 0 ? bar.Vwap : bar.Close);
|
||||
_trades += bar.TradeCount;
|
||||
_takerBuy += bar.TakerBuyVolume;
|
||||
}
|
||||
|
||||
public readonly Bar Emit() => new(
|
||||
_time, _open, _high, _low, _close, _volume,
|
||||
_volume > 0 ? _quoteVolume / _volume : _close,
|
||||
_trades, _takerBuy);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
using System.Globalization;
|
||||
|
||||
namespace Encelado.Core.Ml;
|
||||
|
||||
/// <summary>One feature's distance from the distribution it was trained on.</summary>
|
||||
public readonly record struct DriftReport(string Feature, double Psi, double Ks, bool Alert)
|
||||
{
|
||||
public string Motivazione => Alert
|
||||
? string.Create(CultureInfo.InvariantCulture,
|
||||
$"{Feature}: PSI {Psi:F3} e KS {Ks:F3} — la distribuzione live si è allontanata da quella di addestramento")
|
||||
: string.Create(CultureInfo.InvariantCulture,
|
||||
$"{Feature}: PSI {Psi:F3}, KS {Ks:F3} — entro la tolleranza");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Watches whether the features the model sees in production still look like the ones
|
||||
/// it learnt from.
|
||||
/// <para>
|
||||
/// A model is a bet that the future resembles the training set. This is the instrument
|
||||
/// that says when the bet has stopped being true: for every feature it compares the
|
||||
/// last few hundred live values with the training distribution by population stability
|
||||
/// index and Kolmogorov-Smirnov distance. A PSI above 0.25 is the conventional "the
|
||||
/// population has changed" line; above it the model's probabilities describe a market
|
||||
/// that no longer exists, and the right response is to stop trusting them, not to hope.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class DriftMonitor
|
||||
{
|
||||
private readonly string[] _names;
|
||||
private readonly double[][] _reference;
|
||||
private readonly Queue<double[]> _recent;
|
||||
private readonly int _window;
|
||||
private readonly double _psiAlert;
|
||||
private readonly double _ksAlert;
|
||||
private int _sinceCheck;
|
||||
|
||||
/// <param name="featureNames">Column names, in the model's order.</param>
|
||||
/// <param name="trainingRows">The training feature matrix, one row per example.</param>
|
||||
/// <param name="window">How many live rows to compare.</param>
|
||||
/// <param name="psiAlert">PSI at or above which a feature is flagged.</param>
|
||||
/// <param name="ksAlert">KS distance at or above which a feature is flagged.</param>
|
||||
public DriftMonitor(
|
||||
IReadOnlyList<string> featureNames,
|
||||
IReadOnlyList<double[]> trainingRows,
|
||||
int window = 200,
|
||||
double psiAlert = 0.25,
|
||||
double ksAlert = 0.25)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(featureNames);
|
||||
ArgumentNullException.ThrowIfNull(trainingRows);
|
||||
|
||||
_names = [.. featureNames];
|
||||
_window = Math.Max(20, window);
|
||||
_psiAlert = psiAlert;
|
||||
_ksAlert = ksAlert;
|
||||
_recent = new Queue<double[]>(_window + 1);
|
||||
|
||||
// Reference columns, once.
|
||||
_reference = new double[_names.Length][];
|
||||
for (int f = 0; f < _names.Length; f++)
|
||||
{
|
||||
double[] column = new double[trainingRows.Count];
|
||||
for (int i = 0; i < trainingRows.Count; i++)
|
||||
{
|
||||
column[i] = f < trainingRows[i].Length ? trainingRows[i][f] : double.NaN;
|
||||
}
|
||||
|
||||
_reference[f] = column;
|
||||
}
|
||||
}
|
||||
|
||||
public int ReferenceRows => _reference.Length > 0 ? _reference[0].Length : 0;
|
||||
|
||||
public int RecentRows => _recent.Count;
|
||||
|
||||
/// <summary>Rows observed since the last <see cref="Check"/>.</summary>
|
||||
public int SinceCheck => _sinceCheck;
|
||||
|
||||
public void Observe(double[] row)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(row);
|
||||
|
||||
_recent.Enqueue(row);
|
||||
_sinceCheck++;
|
||||
while (_recent.Count > _window)
|
||||
{
|
||||
_recent.Dequeue();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Whether enough new rows have arrived to be worth testing again.</summary>
|
||||
public bool Due(int everyRows) => _recent.Count >= Math.Min(_window, 50) && _sinceCheck >= Math.Max(1, everyRows);
|
||||
|
||||
/// <summary>Compares every feature and resets the "since check" counter.</summary>
|
||||
public List<DriftReport> Check()
|
||||
{
|
||||
_sinceCheck = 0;
|
||||
List<DriftReport> reports = new(_names.Length);
|
||||
double[][] recent = [.. _recent];
|
||||
|
||||
for (int f = 0; f < _names.Length; f++)
|
||||
{
|
||||
// Constant features (the backtest's latency slot, a flag) cannot drift and
|
||||
// would only produce a spurious alert the first time they take a value.
|
||||
if (IsConstant(_reference[f]))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
double[] live = new double[recent.Length];
|
||||
for (int i = 0; i < recent.Length; i++)
|
||||
{
|
||||
live[i] = f < recent[i].Length ? recent[i][f] : double.NaN;
|
||||
}
|
||||
|
||||
double psi = FeatureRegistry.PopulationStabilityIndex(_reference[f], live);
|
||||
double ks = FeatureRegistry.KolmogorovSmirnov(_reference[f], live);
|
||||
reports.Add(new DriftReport(_names[f], psi, ks, psi >= _psiAlert || ks >= _ksAlert));
|
||||
}
|
||||
|
||||
return reports;
|
||||
}
|
||||
|
||||
private static bool IsConstant(double[] column)
|
||||
{
|
||||
double first = double.NaN;
|
||||
foreach (double v in column)
|
||||
{
|
||||
if (!double.IsFinite(v))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
if (double.IsNaN(first))
|
||||
{
|
||||
first = v;
|
||||
}
|
||||
else if (Math.Abs(v - first) > 1e-12)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,372 @@
|
||||
using Encelado.Core.Statistics;
|
||||
|
||||
namespace Encelado.Core.Ml;
|
||||
|
||||
/// <summary>One column of the feature matrix: a name and a way to compute it for every bar.</summary>
|
||||
public interface IFeature
|
||||
{
|
||||
string Name { get; }
|
||||
|
||||
/// <summary>One value per bar of the series, <see cref="double.NaN"/> where not yet computable.</summary>
|
||||
double[] Compute(PairSeries series);
|
||||
}
|
||||
|
||||
/// <summary>A feature defined by a function, for the ones that are a line or two.</summary>
|
||||
public sealed class LambdaFeature(string name, Func<PairSeries, double[]> compute) : IFeature
|
||||
{
|
||||
public string Name { get; } = name;
|
||||
|
||||
public double[] Compute(PairSeries series) => compute(series);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The extensible list of features. Adding one is registering an <see cref="IFeature"/>;
|
||||
/// nothing downstream — dataset, model, live scoring — has to change, because they all
|
||||
/// address features by the names this registry reports, in the order it reports them.
|
||||
/// <para>
|
||||
/// The default set covers what the strategy note calls "price, order flow, time and
|
||||
/// latency": returns and volatility of both legs, the spread and its z-score, the
|
||||
/// aggressor split of the volume, the hour and weekday as sines and cosines so midnight
|
||||
/// is next to 23:00, the hours to the next funding settlement, and a latency slot that
|
||||
/// the live bot fills with its measured round trip and a backtest leaves at zero.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class FeatureRegistry
|
||||
{
|
||||
private readonly List<IFeature> _features = [];
|
||||
|
||||
public IReadOnlyList<IFeature> Features => _features;
|
||||
|
||||
public IReadOnlyList<string> Names => [.. _features.Select(static f => f.Name)];
|
||||
|
||||
public int Count => _features.Count;
|
||||
|
||||
public FeatureRegistry Register(IFeature feature)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(feature);
|
||||
|
||||
if (_features.Any(f => f.Name == feature.Name))
|
||||
{
|
||||
throw new ArgumentException($"la feature '{feature.Name}' è già registrata", nameof(feature));
|
||||
}
|
||||
|
||||
_features.Add(feature);
|
||||
return this;
|
||||
}
|
||||
|
||||
public FeatureRegistry Register(string name, Func<PairSeries, double[]> compute) =>
|
||||
Register(new LambdaFeature(name, compute));
|
||||
|
||||
/// <summary>The full matrix, one row per bar, one column per feature, in registry order.</summary>
|
||||
public double[][] Compute(PairSeries series)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(series);
|
||||
|
||||
double[][] columns = new double[_features.Count][];
|
||||
for (int f = 0; f < _features.Count; f++)
|
||||
{
|
||||
columns[f] = _features[f].Compute(series);
|
||||
if (columns[f].Length != series.Count)
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"la feature '{_features[f].Name}' ha restituito {columns[f].Length} valori per {series.Count} barre");
|
||||
}
|
||||
}
|
||||
|
||||
double[][] rows = new double[series.Count][];
|
||||
for (int i = 0; i < series.Count; i++)
|
||||
{
|
||||
double[] row = new double[_features.Count];
|
||||
for (int f = 0; f < _features.Count; f++)
|
||||
{
|
||||
row[f] = columns[f][i];
|
||||
}
|
||||
|
||||
rows[i] = row;
|
||||
}
|
||||
|
||||
return rows;
|
||||
}
|
||||
|
||||
/// <summary>The built-in set. See the class summary for what is in it and why.</summary>
|
||||
public static FeatureRegistry Default()
|
||||
{
|
||||
FeatureRegistry r = new();
|
||||
|
||||
foreach (int h in new[] { 1, 4, 16 })
|
||||
{
|
||||
r.Register($"ret_a_{h}", s => LogReturns(s.CloseA, h));
|
||||
r.Register($"ret_b_{h}", s => LogReturns(s.CloseB, h));
|
||||
}
|
||||
|
||||
foreach (int n in new[] { 16, 64 })
|
||||
{
|
||||
r.Register($"vol_a_{n}", s => RollingVolatility(s.CloseA, n));
|
||||
r.Register($"vol_b_{n}", s => RollingVolatility(s.CloseB, n));
|
||||
}
|
||||
|
||||
r.Register("vol_ratio_a", s => Ratio(RollingVolatility(s.CloseA, 16), RollingVolatility(s.CloseA, 64)));
|
||||
|
||||
r.Register("z", static s => (double[])s.ZScore.Clone());
|
||||
r.Register("z_abs", s => s.ZScore.Select(Math.Abs).ToArray());
|
||||
r.Register("z_change_4", s => Difference(s.ZScore, 4));
|
||||
r.Register("spread", static s => (double[])s.Spread.Clone());
|
||||
r.Register("spread_change_4", s => Difference(s.Spread, 4));
|
||||
r.Register("beta", static s => (double[])s.Beta.Clone());
|
||||
r.Register("half_life", static s => (double[])s.HalfLife.Clone());
|
||||
|
||||
r.Register("flow_a", s => FlowImbalance(s.VolumeA, s.TakerBuyA));
|
||||
r.Register("flow_b", s => FlowImbalance(s.VolumeB, s.TakerBuyB));
|
||||
r.Register("flow_a_16", s => Rolling(FlowImbalance(s.VolumeA, s.TakerBuyA), 16));
|
||||
r.Register("flow_b_16", s => Rolling(FlowImbalance(s.VolumeB, s.TakerBuyB), 16));
|
||||
r.Register("volume_a_rel", s => RelativeToRolling(s.VolumeA, 64));
|
||||
r.Register("volume_b_rel", s => RelativeToRolling(s.VolumeB, 64));
|
||||
|
||||
r.Register("hour_sin", s => Cyclic(s.Times, static t => t.Hour + (t.Minute / 60.0), 24, sine: true));
|
||||
r.Register("hour_cos", s => Cyclic(s.Times, static t => t.Hour + (t.Minute / 60.0), 24, sine: false));
|
||||
r.Register("dow_sin", s => Cyclic(s.Times, static t => (int)t.DayOfWeek, 7, sine: true));
|
||||
r.Register("dow_cos", s => Cyclic(s.Times, static t => (int)t.DayOfWeek, 7, sine: false));
|
||||
r.Register("hours_to_funding", s => s.Times.Select(HoursToFunding).ToArray());
|
||||
|
||||
// Filled by the live engine from its measured round trip; a backtest has none.
|
||||
r.Register("latency_ms", s => new double[s.Count]);
|
||||
|
||||
return r;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Building blocks
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
public static double[] LogReturns(IReadOnlyList<double> prices, int horizon)
|
||||
{
|
||||
double[] r = new double[prices.Count];
|
||||
for (int i = 0; i < prices.Count; i++)
|
||||
{
|
||||
r[i] = i >= horizon && prices[i - horizon] > 0 && prices[i] > 0
|
||||
? Math.Log(prices[i] / prices[i - horizon])
|
||||
: double.NaN;
|
||||
}
|
||||
|
||||
return r;
|
||||
}
|
||||
|
||||
public static double[] RollingVolatility(IReadOnlyList<double> prices, int window)
|
||||
{
|
||||
double[] returns = LogReturns(prices, 1);
|
||||
double[] v = new double[prices.Count];
|
||||
double sumSq = 0;
|
||||
int have = 0;
|
||||
|
||||
for (int i = 0; i < prices.Count; i++)
|
||||
{
|
||||
if (double.IsFinite(returns[i]))
|
||||
{
|
||||
sumSq += returns[i] * returns[i];
|
||||
have++;
|
||||
}
|
||||
|
||||
int drop = i - window;
|
||||
if (drop >= 0 && double.IsFinite(returns[drop]))
|
||||
{
|
||||
sumSq -= returns[drop] * returns[drop];
|
||||
have--;
|
||||
}
|
||||
|
||||
v[i] = have >= window ? Math.Sqrt(Math.Max(0, sumSq)) : double.NaN;
|
||||
}
|
||||
|
||||
return v;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Taker-buy share of volume, centred: +1 all buyers, −1 all sellers. Zero — not
|
||||
/// missing — where the bar carries no order-flow split at all, so a history file
|
||||
/// without the column still yields a complete feature row; a model trained on such
|
||||
/// a file simply never splits on it.
|
||||
/// </summary>
|
||||
public static double[] FlowImbalance(IReadOnlyList<double> volume, IReadOnlyList<double> takerBuy)
|
||||
{
|
||||
double[] f = new double[volume.Count];
|
||||
for (int i = 0; i < f.Length; i++)
|
||||
{
|
||||
f[i] = volume[i] > 0 && takerBuy[i] > 0 ? ((2 * takerBuy[i]) - volume[i]) / volume[i] : 0;
|
||||
}
|
||||
|
||||
return f;
|
||||
}
|
||||
|
||||
public static double[] Rolling(IReadOnlyList<double> values, int window)
|
||||
{
|
||||
double[] r = new double[values.Count];
|
||||
double sum = 0;
|
||||
int have = 0;
|
||||
|
||||
for (int i = 0; i < values.Count; i++)
|
||||
{
|
||||
if (double.IsFinite(values[i]))
|
||||
{
|
||||
sum += values[i];
|
||||
have++;
|
||||
}
|
||||
|
||||
int drop = i - window;
|
||||
if (drop >= 0 && double.IsFinite(values[drop]))
|
||||
{
|
||||
sum -= values[drop];
|
||||
have--;
|
||||
}
|
||||
|
||||
r[i] = have > 0 && i >= window - 1 ? sum / have : double.NaN;
|
||||
}
|
||||
|
||||
return r;
|
||||
}
|
||||
|
||||
public static double[] RelativeToRolling(IReadOnlyList<double> values, int window)
|
||||
{
|
||||
double[] mean = Rolling(values, window);
|
||||
double[] r = new double[values.Count];
|
||||
for (int i = 0; i < r.Length; i++)
|
||||
{
|
||||
r[i] = double.IsFinite(mean[i]) && mean[i] > 0 ? values[i] / mean[i] : double.NaN;
|
||||
}
|
||||
|
||||
return r;
|
||||
}
|
||||
|
||||
public static double[] Difference(IReadOnlyList<double> values, int lag)
|
||||
{
|
||||
double[] d = new double[values.Count];
|
||||
for (int i = 0; i < d.Length; i++)
|
||||
{
|
||||
d[i] = i >= lag && double.IsFinite(values[i]) && double.IsFinite(values[i - lag])
|
||||
? values[i] - values[i - lag]
|
||||
: double.NaN;
|
||||
}
|
||||
|
||||
return d;
|
||||
}
|
||||
|
||||
public static double[] Ratio(IReadOnlyList<double> numerator, IReadOnlyList<double> denominator)
|
||||
{
|
||||
double[] r = new double[numerator.Count];
|
||||
for (int i = 0; i < r.Length; i++)
|
||||
{
|
||||
r[i] = double.IsFinite(numerator[i]) && double.IsFinite(denominator[i]) && denominator[i] > 0
|
||||
? numerator[i] / denominator[i]
|
||||
: double.NaN;
|
||||
}
|
||||
|
||||
return r;
|
||||
}
|
||||
|
||||
/// <summary>Sine or cosine of a periodic quantity, so its wrap-around point is not a cliff.</summary>
|
||||
public static double[] Cyclic(IReadOnlyList<DateTime> times, Func<DateTime, double> value, double period, bool sine)
|
||||
{
|
||||
double[] r = new double[times.Count];
|
||||
for (int i = 0; i < r.Length; i++)
|
||||
{
|
||||
double angle = 2 * Math.PI * value(times[i]) / period;
|
||||
r[i] = sine ? Math.Sin(angle) : Math.Cos(angle);
|
||||
}
|
||||
|
||||
return r;
|
||||
}
|
||||
|
||||
/// <summary>Hours until the next funding settlement (00:00, 08:00, 16:00 UTC).</summary>
|
||||
public static double HoursToFunding(DateTime utc)
|
||||
{
|
||||
double hour = utc.Hour + (utc.Minute / 60.0) + (utc.Second / 3600.0);
|
||||
double next = Math.Ceiling(hour / 8.0) * 8.0;
|
||||
if (Math.Abs(next - hour) < 1e-9)
|
||||
{
|
||||
next += 8;
|
||||
}
|
||||
|
||||
return next - hour;
|
||||
}
|
||||
|
||||
/// <summary>Population Stability Index between two samples of one feature, over decile bins of the reference.</summary>
|
||||
public static double PopulationStabilityIndex(IReadOnlyList<double> reference, IReadOnlyList<double> current, int bins = 10)
|
||||
{
|
||||
double[] r = [.. reference.Where(double.IsFinite).Order()];
|
||||
double[] c = [.. current.Where(double.IsFinite)];
|
||||
|
||||
if (r.Length < bins * 2 || c.Length < bins)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
double[] edges = new double[bins - 1];
|
||||
for (int i = 1; i < bins; i++)
|
||||
{
|
||||
edges[i - 1] = r[(int)((long)i * r.Length / bins)];
|
||||
}
|
||||
|
||||
double[] pr = Histogram(r, edges, bins);
|
||||
double[] pc = Histogram(c, edges, bins);
|
||||
|
||||
double psi = 0;
|
||||
for (int b = 0; b < bins; b++)
|
||||
{
|
||||
double a = Math.Max(pr[b], 1e-6);
|
||||
double d = Math.Max(pc[b], 1e-6);
|
||||
psi += (d - a) * Math.Log(d / a);
|
||||
}
|
||||
|
||||
return psi;
|
||||
|
||||
static double[] Histogram(double[] values, double[] edges, int bins)
|
||||
{
|
||||
double[] h = new double[bins];
|
||||
foreach (double v in values)
|
||||
{
|
||||
int b = 0;
|
||||
while (b < edges.Length && v > edges[b])
|
||||
{
|
||||
b++;
|
||||
}
|
||||
|
||||
h[b]++;
|
||||
}
|
||||
|
||||
for (int b = 0; b < bins; b++)
|
||||
{
|
||||
h[b] /= values.Length;
|
||||
}
|
||||
|
||||
return h;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Kolmogorov-Smirnov distance between two samples of one feature.</summary>
|
||||
public static double KolmogorovSmirnov(IReadOnlyList<double> reference, IReadOnlyList<double> current)
|
||||
{
|
||||
double[] r = [.. reference.Where(double.IsFinite).Order()];
|
||||
double[] c = [.. current.Where(double.IsFinite).Order()];
|
||||
|
||||
if (r.Length == 0 || c.Length == 0)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
double d = 0;
|
||||
int i = 0, j = 0;
|
||||
while (i < r.Length && j < c.Length)
|
||||
{
|
||||
if (r[i] <= c[j])
|
||||
{
|
||||
i++;
|
||||
}
|
||||
else
|
||||
{
|
||||
j++;
|
||||
}
|
||||
|
||||
d = Math.Max(d, Math.Abs((i / (double)r.Length) - (j / (double)c.Length)));
|
||||
}
|
||||
|
||||
return d;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
using Encelado.Core.Statistics;
|
||||
|
||||
namespace Encelado.Core.Ml;
|
||||
|
||||
/// <summary>
|
||||
/// Fractional differentiation: make a price series stationary while throwing away as
|
||||
/// little of its memory as possible.
|
||||
/// <para>
|
||||
/// Ordinary returns (<c>d = 1</c>) are stationary and nearly memoryless; log prices
|
||||
/// (<c>d = 0</c>) remember everything and are not stationary. Every model needs the first
|
||||
/// property and every signal lives in the second. The fractional order in between —
|
||||
/// often well below one — is the smallest <c>d</c> at which the Dickey-Fuller test
|
||||
/// passes, and that is what <see cref="MinimumStationaryOrder"/> searches for.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// This is preprocessing, nothing more. It creates no edge and prevents no
|
||||
/// overfitting; it only stops a tree model from learning the price level.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class FractionalDifferentiation
|
||||
{
|
||||
/// <summary>
|
||||
/// The weights of <c>(1 − B)^d</c>, from the recurrence
|
||||
/// <c>ω₀ = 1, ωₖ = −ωₖ₋₁·(d − k + 1)/k</c>, truncated where they fall below
|
||||
/// <paramref name="threshold"/>. Index 0 is the weight on the current value.
|
||||
/// </summary>
|
||||
public static double[] Weights(double d, double threshold = 1e-5, int maxSize = 10_000)
|
||||
{
|
||||
if (d < 0)
|
||||
{
|
||||
throw new ArgumentOutOfRangeException(nameof(d), d, "l'ordine non può essere negativo");
|
||||
}
|
||||
|
||||
List<double> weights = [1.0];
|
||||
for (int k = 1; k < maxSize; k++)
|
||||
{
|
||||
double next = -weights[k - 1] * (d - k + 1) / k;
|
||||
if (Math.Abs(next) < threshold)
|
||||
{
|
||||
break;
|
||||
}
|
||||
|
||||
weights.Add(next);
|
||||
}
|
||||
|
||||
return [.. weights];
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The fixed-window fractionally differenced series. The first <c>len(ω) − 1</c>
|
||||
/// values are <see cref="double.NaN"/>: there is not enough history behind them.
|
||||
/// </summary>
|
||||
public static double[] Apply(IReadOnlyList<double> series, double d, double threshold = 1e-5)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(series);
|
||||
|
||||
double[] w = Weights(d, threshold);
|
||||
int width = w.Length;
|
||||
double[] result = new double[series.Count];
|
||||
|
||||
for (int t = 0; t < series.Count; t++)
|
||||
{
|
||||
if (t < width - 1)
|
||||
{
|
||||
result[t] = double.NaN;
|
||||
continue;
|
||||
}
|
||||
|
||||
double sum = 0;
|
||||
for (int k = 0; k < width; k++)
|
||||
{
|
||||
sum += w[k] * series[t - k];
|
||||
}
|
||||
|
||||
result[t] = sum;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The smallest order, on a grid of <paramref name="step"/>, at which the
|
||||
/// differenced series rejects a unit root at 5%. Returns the order, the series at
|
||||
/// that order, and the test that justified it.
|
||||
/// </summary>
|
||||
public static (double Order, double[] Series, AdfResult Adf) MinimumStationaryOrder(
|
||||
IReadOnlyList<double> series, double step = 0.05, double threshold = 1e-4)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(series);
|
||||
|
||||
if (step <= 0 || step > 1)
|
||||
{
|
||||
throw new ArgumentOutOfRangeException(nameof(step), step, "il passo deve stare in (0, 1]");
|
||||
}
|
||||
|
||||
double[] last = [.. series];
|
||||
AdfResult lastAdf = AdfResult.Invalid;
|
||||
|
||||
for (double d = 0; d <= 1.0 + 1e-9; d += step)
|
||||
{
|
||||
double[] candidate = Apply(series, d, threshold);
|
||||
double[] clean = [.. candidate.Where(static v => double.IsFinite(v))];
|
||||
|
||||
if (clean.Length < 50)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
AdfResult adf = DickeyFuller.Test(clean);
|
||||
last = candidate;
|
||||
lastAdf = adf;
|
||||
|
||||
if (adf.RejectsUnitRoot)
|
||||
{
|
||||
return (Math.Round(d, 4), candidate, adf);
|
||||
}
|
||||
}
|
||||
|
||||
// Not even d = 1 passed: the series is something stranger than a random walk.
|
||||
// Return the fully differenced version and the test that failed on it.
|
||||
return (1.0, last, lastAdf);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,492 @@
|
||||
using System.Text;
|
||||
|
||||
namespace Encelado.Core.Ml;
|
||||
|
||||
/// <summary>The knobs of the gradient boosting model, with the values that work as a start.</summary>
|
||||
public sealed record GbdtParams
|
||||
{
|
||||
public int Trees { get; init; } = 200;
|
||||
|
||||
public double LearningRate { get; init; } = 0.05;
|
||||
|
||||
public int MaxDepth { get; init; } = 4;
|
||||
|
||||
/// <summary>Smallest sum of hessians a leaf may hold: the model's own guard against memorising.</summary>
|
||||
public double MinLeafHessian { get; init; } = 20;
|
||||
|
||||
public int Bins { get; init; } = 32;
|
||||
|
||||
public double Subsample { get; init; } = 0.8;
|
||||
|
||||
public double ColumnSample { get; init; } = 0.8;
|
||||
|
||||
/// <summary>L2 penalty on leaf values.</summary>
|
||||
public double L2 { get; init; } = 1.0;
|
||||
|
||||
public string Describe() =>
|
||||
$"trees={Trees} lr={LearningRate} depth={MaxDepth} minLeaf={MinLeafHessian} bins={Bins} " +
|
||||
$"subsample={Subsample} colsample={ColumnSample} l2={L2}";
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Histogram-based gradient boosted trees for binary classification — the algorithm
|
||||
/// LightGBM runs, in the size this project needs and with no native dependency.
|
||||
/// <para>
|
||||
/// Each feature is bucketed into a few dozen quantile bins once; every split is then
|
||||
/// found by scanning bin histograms of gradients and hessians rather than sorted values,
|
||||
/// which is what makes a round over tens of thousands of rows take milliseconds. Leaves
|
||||
/// take a Newton step on the log-loss with an L2 penalty, and the ensemble is a sum of
|
||||
/// trees scaled by the learning rate. That is the whole method; the rest is bookkeeping.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Missing values are imputed with the training median of their column, and the median
|
||||
/// travels with the model so live scoring imputes the same way.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class Gbdt
|
||||
{
|
||||
public static GbdtModel Train(
|
||||
double[][] x,
|
||||
int[] y,
|
||||
double[]? weights,
|
||||
GbdtParams p,
|
||||
IReadOnlyList<string> featureNames,
|
||||
int seed = 1)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(x);
|
||||
ArgumentNullException.ThrowIfNull(y);
|
||||
ArgumentNullException.ThrowIfNull(p);
|
||||
ArgumentNullException.ThrowIfNull(featureNames);
|
||||
|
||||
int n = x.Length;
|
||||
if (n == 0 || y.Length != n)
|
||||
{
|
||||
throw new ArgumentException("servono righe, e tante etichette quante righe");
|
||||
}
|
||||
|
||||
int features = x[0].Length;
|
||||
if (featureNames.Count != features)
|
||||
{
|
||||
throw new ArgumentException("un nome per ogni colonna");
|
||||
}
|
||||
|
||||
double[] w = weights ?? Enumerable.Repeat(1.0, n).ToArray();
|
||||
Random rng = new(seed);
|
||||
|
||||
// ---- binning --------------------------------------------------------
|
||||
double[] medians = new double[features];
|
||||
double[][] edges = new double[features][];
|
||||
for (int f = 0; f < features; f++)
|
||||
{
|
||||
double[] column = [.. x.Select(row => row[f]).Where(double.IsFinite).Order()];
|
||||
medians[f] = column.Length > 0 ? column[column.Length / 2] : 0;
|
||||
edges[f] = QuantileEdges(column, p.Bins);
|
||||
}
|
||||
|
||||
byte[][] bins = new byte[n][];
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
bins[i] = new byte[features];
|
||||
for (int f = 0; f < features; f++)
|
||||
{
|
||||
bins[i][f] = GbdtModel.BinOf(x[i][f], edges[f], medians[f]);
|
||||
}
|
||||
}
|
||||
|
||||
// ---- boosting -------------------------------------------------------
|
||||
double positive = 0, total = 0;
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
positive += w[i] * y[i];
|
||||
total += w[i];
|
||||
}
|
||||
|
||||
double prior = Math.Clamp(positive / Math.Max(total, 1e-12), 1e-4, 1 - 1e-4);
|
||||
double baseScore = Math.Log(prior / (1 - prior));
|
||||
|
||||
double[] score = new double[n];
|
||||
Array.Fill(score, baseScore);
|
||||
|
||||
double[] g = new double[n];
|
||||
double[] h = new double[n];
|
||||
double[] importance = new double[features];
|
||||
List<GbdtModel.Node[]> trees = [];
|
||||
|
||||
int[] all = Enumerable.Range(0, n).ToArray();
|
||||
|
||||
for (int t = 0; t < p.Trees; t++)
|
||||
{
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
double prob = Sigmoid(score[i]);
|
||||
g[i] = w[i] * (prob - y[i]);
|
||||
h[i] = Math.Max(1e-9, w[i] * prob * (1 - prob));
|
||||
}
|
||||
|
||||
int[] rows = p.Subsample < 1
|
||||
? all.Where(_ => rng.NextDouble() < p.Subsample).ToArray()
|
||||
: all;
|
||||
|
||||
bool[] allowed = new bool[features];
|
||||
int kept = 0;
|
||||
for (int f = 0; f < features; f++)
|
||||
{
|
||||
allowed[f] = p.ColumnSample >= 1 || rng.NextDouble() < p.ColumnSample;
|
||||
kept += allowed[f] ? 1 : 0;
|
||||
}
|
||||
|
||||
if (kept == 0)
|
||||
{
|
||||
allowed[rng.Next(features)] = true;
|
||||
}
|
||||
|
||||
List<GbdtModel.Node> nodes = [];
|
||||
Grow(nodes, rows, bins, g, h, allowed, p, depth: 0, importance);
|
||||
GbdtModel.Node[] tree = [.. nodes];
|
||||
trees.Add(tree);
|
||||
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
score[i] += p.LearningRate * GbdtModel.Evaluate(tree, bins[i]);
|
||||
}
|
||||
}
|
||||
|
||||
return new GbdtModel(features, baseScore, p.LearningRate, edges, medians, [.. featureNames], trees, importance);
|
||||
}
|
||||
|
||||
/// <summary>Grows one tree depth-first; returns the index of the node it created.</summary>
|
||||
private static int Grow(
|
||||
List<GbdtModel.Node> nodes,
|
||||
int[] rows,
|
||||
byte[][] bins,
|
||||
double[] g,
|
||||
double[] h,
|
||||
bool[] allowed,
|
||||
GbdtParams p,
|
||||
int depth,
|
||||
double[] importance)
|
||||
{
|
||||
double sumG = 0, sumH = 0;
|
||||
foreach (int i in rows)
|
||||
{
|
||||
sumG += g[i];
|
||||
sumH += h[i];
|
||||
}
|
||||
|
||||
int index = nodes.Count;
|
||||
nodes.Add(default);
|
||||
|
||||
double leafValue = -sumG / (sumH + p.L2);
|
||||
|
||||
if (depth >= p.MaxDepth || rows.Length < 2 || sumH < 2 * p.MinLeafHessian)
|
||||
{
|
||||
nodes[index] = GbdtModel.Node.Leaf(leafValue);
|
||||
return index;
|
||||
}
|
||||
|
||||
// ---- best split over bin histograms -----------------------------------
|
||||
int features = allowed.Length;
|
||||
int binCount = p.Bins;
|
||||
double bestGain = 0;
|
||||
int bestFeature = -1;
|
||||
int bestBin = -1;
|
||||
|
||||
double parentScore = (sumG * sumG) / (sumH + p.L2);
|
||||
double[] hg = new double[binCount];
|
||||
double[] hh = new double[binCount];
|
||||
|
||||
for (int f = 0; f < features; f++)
|
||||
{
|
||||
if (!allowed[f])
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
Array.Clear(hg);
|
||||
Array.Clear(hh);
|
||||
foreach (int i in rows)
|
||||
{
|
||||
byte b = bins[i][f];
|
||||
hg[b] += g[i];
|
||||
hh[b] += h[i];
|
||||
}
|
||||
|
||||
double leftG = 0, leftH = 0;
|
||||
for (int b = 0; b < binCount - 1; b++)
|
||||
{
|
||||
leftG += hg[b];
|
||||
leftH += hh[b];
|
||||
double rightG = sumG - leftG;
|
||||
double rightH = sumH - leftH;
|
||||
|
||||
if (leftH < p.MinLeafHessian || rightH < p.MinLeafHessian)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
double gain = 0.5 * (((leftG * leftG) / (leftH + p.L2)) + ((rightG * rightG) / (rightH + p.L2)) - parentScore);
|
||||
if (gain > bestGain)
|
||||
{
|
||||
bestGain = gain;
|
||||
bestFeature = f;
|
||||
bestBin = b;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (bestFeature < 0)
|
||||
{
|
||||
nodes[index] = GbdtModel.Node.Leaf(leafValue);
|
||||
return index;
|
||||
}
|
||||
|
||||
importance[bestFeature] += bestGain;
|
||||
|
||||
int[] left = rows.Where(i => bins[i][bestFeature] <= bestBin).ToArray();
|
||||
int[] right = rows.Where(i => bins[i][bestFeature] > bestBin).ToArray();
|
||||
|
||||
int leftIndex = Grow(nodes, left, bins, g, h, allowed, p, depth + 1, importance);
|
||||
int rightIndex = Grow(nodes, right, bins, g, h, allowed, p, depth + 1, importance);
|
||||
|
||||
nodes[index] = GbdtModel.Node.Split(bestFeature, bestBin, leftIndex, rightIndex);
|
||||
return index;
|
||||
}
|
||||
|
||||
private static double[] QuantileEdges(double[] sorted, int bins)
|
||||
{
|
||||
if (sorted.Length == 0)
|
||||
{
|
||||
return [];
|
||||
}
|
||||
|
||||
List<double> edges = [];
|
||||
for (int b = 1; b < bins; b++)
|
||||
{
|
||||
double edge = sorted[(int)Math.Min(sorted.Length - 1, (long)b * sorted.Length / bins)];
|
||||
if (edges.Count == 0 || edge > edges[^1])
|
||||
{
|
||||
edges.Add(edge);
|
||||
}
|
||||
}
|
||||
|
||||
return [.. edges];
|
||||
}
|
||||
|
||||
internal static double Sigmoid(double v) => 1.0 / (1.0 + Math.Exp(-v));
|
||||
}
|
||||
|
||||
/// <summary>A trained ensemble: everything needed to score a row, and nothing else.</summary>
|
||||
public sealed class GbdtModel
|
||||
{
|
||||
private readonly double[][] _edges;
|
||||
private readonly double[] _medians;
|
||||
private readonly List<Node[]> _trees;
|
||||
|
||||
internal GbdtModel(
|
||||
int features, double baseScore, double learningRate, double[][] edges, double[] medians,
|
||||
string[] featureNames, List<Node[]> trees, double[] importance)
|
||||
{
|
||||
FeatureCount = features;
|
||||
BaseScore = baseScore;
|
||||
LearningRate = learningRate;
|
||||
_edges = edges;
|
||||
_medians = medians;
|
||||
FeatureNames = featureNames;
|
||||
_trees = trees;
|
||||
Importance = importance;
|
||||
}
|
||||
|
||||
public int FeatureCount { get; }
|
||||
|
||||
public double BaseScore { get; }
|
||||
|
||||
public double LearningRate { get; }
|
||||
|
||||
public string[] FeatureNames { get; }
|
||||
|
||||
public int TreeCount => _trees.Count;
|
||||
|
||||
/// <summary>Total split gain attributed to each feature: which columns the model actually used.</summary>
|
||||
public double[] Importance { get; }
|
||||
|
||||
/// <summary>The probability that the label is 1.</summary>
|
||||
public double Predict(double[] x)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(x);
|
||||
if (x.Length != FeatureCount)
|
||||
{
|
||||
throw new ArgumentException($"servono {FeatureCount} feature, non {x.Length}");
|
||||
}
|
||||
|
||||
byte[] bins = new byte[FeatureCount];
|
||||
for (int f = 0; f < FeatureCount; f++)
|
||||
{
|
||||
bins[f] = BinOf(x[f], _edges[f], _medians[f]);
|
||||
}
|
||||
|
||||
double score = BaseScore;
|
||||
foreach (Node[] tree in _trees)
|
||||
{
|
||||
score += LearningRate * Evaluate(tree, bins);
|
||||
}
|
||||
|
||||
return Gbdt.Sigmoid(score);
|
||||
}
|
||||
|
||||
public double[] PredictMany(double[][] x)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(x);
|
||||
double[] p = new double[x.Length];
|
||||
for (int i = 0; i < x.Length; i++)
|
||||
{
|
||||
p[i] = Predict(x[i]);
|
||||
}
|
||||
|
||||
return p;
|
||||
}
|
||||
|
||||
internal static byte BinOf(double value, double[] edges, double median)
|
||||
{
|
||||
if (!double.IsFinite(value))
|
||||
{
|
||||
value = median;
|
||||
}
|
||||
|
||||
int lo = 0, hi = edges.Length;
|
||||
while (lo < hi)
|
||||
{
|
||||
int mid = (lo + hi) >> 1;
|
||||
if (value <= edges[mid])
|
||||
{
|
||||
hi = mid;
|
||||
}
|
||||
else
|
||||
{
|
||||
lo = mid + 1;
|
||||
}
|
||||
}
|
||||
|
||||
return (byte)lo;
|
||||
}
|
||||
|
||||
internal static double Evaluate(Node[] tree, byte[] bins)
|
||||
{
|
||||
int i = 0;
|
||||
while (true)
|
||||
{
|
||||
Node node = tree[i];
|
||||
if (node.IsLeaf)
|
||||
{
|
||||
return node.Value;
|
||||
}
|
||||
|
||||
i = bins[node.Feature] <= node.Bin ? node.Left : node.Right;
|
||||
}
|
||||
}
|
||||
|
||||
internal readonly record struct Node(bool IsLeaf, int Feature, int Bin, int Left, int Right, double Value)
|
||||
{
|
||||
public static Node Leaf(double value) => new(true, -1, -1, -1, -1, value);
|
||||
|
||||
public static Node Split(int feature, int bin, int left, int right) => new(false, feature, bin, left, right, 0);
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Serialisation
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
private const int Version = 1;
|
||||
|
||||
public byte[] ToBytes()
|
||||
{
|
||||
using MemoryStream ms = new();
|
||||
using BinaryWriter w = new(ms, Encoding.UTF8, leaveOpen: true);
|
||||
|
||||
w.Write(Version);
|
||||
w.Write(FeatureCount);
|
||||
w.Write(BaseScore);
|
||||
w.Write(LearningRate);
|
||||
|
||||
for (int f = 0; f < FeatureCount; f++)
|
||||
{
|
||||
w.Write(FeatureNames[f]);
|
||||
w.Write(_medians[f]);
|
||||
w.Write(Importance[f]);
|
||||
w.Write(_edges[f].Length);
|
||||
foreach (double e in _edges[f])
|
||||
{
|
||||
w.Write(e);
|
||||
}
|
||||
}
|
||||
|
||||
w.Write(_trees.Count);
|
||||
foreach (Node[] tree in _trees)
|
||||
{
|
||||
w.Write(tree.Length);
|
||||
foreach (Node node in tree)
|
||||
{
|
||||
w.Write(node.IsLeaf);
|
||||
w.Write(node.Feature);
|
||||
w.Write(node.Bin);
|
||||
w.Write(node.Left);
|
||||
w.Write(node.Right);
|
||||
w.Write(node.Value);
|
||||
}
|
||||
}
|
||||
|
||||
w.Flush();
|
||||
return ms.ToArray();
|
||||
}
|
||||
|
||||
public static GbdtModel FromBytes(byte[] bytes)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(bytes);
|
||||
|
||||
using MemoryStream ms = new(bytes);
|
||||
using BinaryReader r = new(ms, Encoding.UTF8);
|
||||
|
||||
int version = r.ReadInt32();
|
||||
if (version != Version)
|
||||
{
|
||||
throw new InvalidDataException($"modello in formato {version}, atteso {Version}");
|
||||
}
|
||||
|
||||
int features = r.ReadInt32();
|
||||
double baseScore = r.ReadDouble();
|
||||
double learningRate = r.ReadDouble();
|
||||
|
||||
string[] names = new string[features];
|
||||
double[] medians = new double[features];
|
||||
double[] importance = new double[features];
|
||||
double[][] edges = new double[features][];
|
||||
|
||||
for (int f = 0; f < features; f++)
|
||||
{
|
||||
names[f] = r.ReadString();
|
||||
medians[f] = r.ReadDouble();
|
||||
importance[f] = r.ReadDouble();
|
||||
int count = r.ReadInt32();
|
||||
edges[f] = new double[count];
|
||||
for (int e = 0; e < count; e++)
|
||||
{
|
||||
edges[f][e] = r.ReadDouble();
|
||||
}
|
||||
}
|
||||
|
||||
int treeCount = r.ReadInt32();
|
||||
List<Node[]> trees = new(treeCount);
|
||||
for (int t = 0; t < treeCount; t++)
|
||||
{
|
||||
int nodeCount = r.ReadInt32();
|
||||
Node[] tree = new Node[nodeCount];
|
||||
for (int i = 0; i < nodeCount; i++)
|
||||
{
|
||||
tree[i] = new Node(r.ReadBoolean(), r.ReadInt32(), r.ReadInt32(), r.ReadInt32(), r.ReadInt32(), r.ReadDouble());
|
||||
}
|
||||
|
||||
trees.Add(tree);
|
||||
}
|
||||
|
||||
return new GbdtModel(features, baseScore, learningRate, edges, medians, names, trees, importance);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,280 @@
|
||||
using System.Globalization;
|
||||
using Encelado.Core.Statistics;
|
||||
|
||||
namespace Encelado.Core.Ml;
|
||||
|
||||
/// <summary>How events are turned into labels.</summary>
|
||||
public sealed record LabelingParams
|
||||
{
|
||||
/// <summary>Profit-take barrier, in local volatilities.</summary>
|
||||
public double ProfitMultiple { get; init; } = 2.0;
|
||||
|
||||
/// <summary>Stop-loss barrier, in local volatilities.</summary>
|
||||
public double StopMultiple { get; init; } = 1.0;
|
||||
|
||||
/// <summary>Vertical barrier, in bars.</summary>
|
||||
public int MaxHoldingBars { get; init; } = 96;
|
||||
|
||||
/// <summary>Span of the exponentially weighted volatility the barriers are scaled by.</summary>
|
||||
public int VolatilitySpan { get; init; } = 100;
|
||||
|
||||
public string Describe() =>
|
||||
string.Create(CultureInfo.InvariantCulture,
|
||||
$"triple-barrier pt={ProfitMultiple}σ sl={StopMultiple}σ max={MaxHoldingBars} barre vol={VolatilitySpan}");
|
||||
}
|
||||
|
||||
/// <summary>One training example.</summary>
|
||||
public sealed record MetaRow(
|
||||
int Index,
|
||||
DateTime TimeUtc,
|
||||
DateTime EndUtc,
|
||||
int EndIndex,
|
||||
int Label,
|
||||
int Side,
|
||||
double Return,
|
||||
double Weight,
|
||||
double[] Features);
|
||||
|
||||
/// <summary>
|
||||
/// A dataset ready to train on: the rows, the names of their columns, and how many
|
||||
/// independent observations the rows really are.
|
||||
/// </summary>
|
||||
public sealed record MetaDataset(IReadOnlyList<string> FeatureNames, IReadOnlyList<MetaRow> Rows, double EffectiveCount)
|
||||
{
|
||||
public int Count => Rows.Count;
|
||||
|
||||
public double PositiveRate => Rows.Count == 0 ? 0 : Rows.Count(static r => r.Label == 1) / (double)Rows.Count;
|
||||
}
|
||||
|
||||
/// <summary>Everything a training run reports.</summary>
|
||||
public sealed record MetaTrainingResult(
|
||||
GbdtModel Model,
|
||||
int Rows,
|
||||
int Paths,
|
||||
double OosAccuracy,
|
||||
double OosLogLoss,
|
||||
double[] PathSharpes,
|
||||
double MeanPathSharpe,
|
||||
double Psr,
|
||||
double Dsr,
|
||||
double[] PathReturns,
|
||||
string Motivazione)
|
||||
{
|
||||
public string Describe() => string.Create(CultureInfo.InvariantCulture,
|
||||
$"righe {Rows}, {Paths} percorsi CPCV, accuratezza OOS {OosAccuracy:P1}, logloss {OosLogLoss:F4}, " +
|
||||
$"Sharpe medio per operazione {MeanPathSharpe:F3}, PSR {Psr:F3}, DSR {Dsr:F3}");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Meta-labelling (López de Prado): a second model that decides whether to act on the
|
||||
/// first one's signal.
|
||||
/// <para>
|
||||
/// The primary model is the statistical arbitrage rule — it says <i>which way</i>. The
|
||||
/// secondary model learns, from what happened after previous signals, <i>whether</i> a
|
||||
/// signal in these conditions tends to pay off, and by how much to size it. It cannot
|
||||
/// invent a trade the primary did not propose, which is what keeps it honest: its only
|
||||
/// power is to say no, and to say it before the fees are paid.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Training is combinatorial purged cross-validation: every out-of-sample statistic here
|
||||
/// comes from predictions on rows the model never saw and whose labels could not have
|
||||
/// leaked into what it did see. The deflated Sharpe is computed against the number of
|
||||
/// configurations the caller says were tried — count them all.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class MetaLabeling
|
||||
{
|
||||
/// <summary>
|
||||
/// Builds the training set: one row per bar where the primary model took a side,
|
||||
/// with the features at that bar, the triple-barrier outcome of that side, and the
|
||||
/// row's uniqueness weight.
|
||||
/// </summary>
|
||||
public static MetaDataset BuildDataset(
|
||||
PairSeries series,
|
||||
IReadOnlyList<int> primarySides,
|
||||
FeatureRegistry registry,
|
||||
LabelingParams labeling)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(series);
|
||||
ArgumentNullException.ThrowIfNull(primarySides);
|
||||
ArgumentNullException.ThrowIfNull(registry);
|
||||
ArgumentNullException.ThrowIfNull(labeling);
|
||||
|
||||
if (primarySides.Count != series.Count)
|
||||
{
|
||||
throw new ArgumentException("un lato per ogni barra (0 dove non c'è segnale)");
|
||||
}
|
||||
|
||||
double[][] matrix = registry.Compute(series);
|
||||
|
||||
// The spread is what the position is actually exposed to, and it is already
|
||||
// dimensionless — but the barriers want a price-like path, so they are drawn on
|
||||
// exp(spread): a spread up 1% is a "price" up 1%.
|
||||
double[] spreadPrice = new double[series.Count];
|
||||
for (int i = 0; i < series.Count; i++)
|
||||
{
|
||||
spreadPrice[i] = double.IsFinite(series.Spread[i]) ? Math.Exp(series.Spread[i]) : double.NaN;
|
||||
}
|
||||
|
||||
double[] volatility = TripleBarrier.RollingVolatility(spreadPrice, labeling.VolatilitySpan);
|
||||
|
||||
List<int> events = [];
|
||||
List<int> sides = [];
|
||||
for (int i = 0; i < series.Count; i++)
|
||||
{
|
||||
if (primarySides[i] != 0 && double.IsFinite(spreadPrice[i]) && Complete(matrix[i]))
|
||||
{
|
||||
events.Add(i);
|
||||
sides.Add(primarySides[i]);
|
||||
}
|
||||
}
|
||||
|
||||
List<BarrierLabel> labels = TripleBarrier.Label(
|
||||
spreadPrice, events, sides, volatility,
|
||||
labeling.ProfitMultiple, labeling.StopMultiple, labeling.MaxHoldingBars);
|
||||
|
||||
double[] weights = SampleWeights.AverageUniqueness(labels, series.Count);
|
||||
double effective = SampleWeights.EffectiveCount(labels, series.Count);
|
||||
|
||||
List<MetaRow> rows = new(labels.Count);
|
||||
for (int k = 0; k < labels.Count; k++)
|
||||
{
|
||||
BarrierLabel label = labels[k];
|
||||
rows.Add(new MetaRow(
|
||||
label.Index,
|
||||
series.Times[label.Index],
|
||||
series.Times[label.EndIndex],
|
||||
label.EndIndex,
|
||||
label.MetaLabel,
|
||||
label.Side,
|
||||
label.Return,
|
||||
weights[k],
|
||||
matrix[label.Index]));
|
||||
}
|
||||
|
||||
return new MetaDataset(registry.Names, rows, effective);
|
||||
|
||||
static bool Complete(double[] row)
|
||||
{
|
||||
foreach (double v in row)
|
||||
{
|
||||
if (!double.IsFinite(v))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Trains and validates. The returned model is fitted on every row; every statistic
|
||||
/// is out-of-sample from the CPCV paths.
|
||||
/// </summary>
|
||||
/// <param name="trials">How many configurations were tried in total, for the deflated Sharpe.</param>
|
||||
/// <param name="trialSharpeVariance">Variance of the trials' Sharpe estimates; 0 when unknown.</param>
|
||||
public static MetaTrainingResult Train(
|
||||
MetaDataset dataset,
|
||||
GbdtParams parameters,
|
||||
int groups = 6,
|
||||
int testGroups = 2,
|
||||
double embargo = 0.01,
|
||||
int trials = 1,
|
||||
double trialSharpeVariance = 0,
|
||||
int seed = 1)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(dataset);
|
||||
ArgumentNullException.ThrowIfNull(parameters);
|
||||
|
||||
int n = dataset.Count;
|
||||
if (n < groups * 10)
|
||||
{
|
||||
throw new ArgumentException($"servono almeno {groups * 10} righe per {groups} gruppi, ne hai {n}");
|
||||
}
|
||||
|
||||
double[][] x = [.. dataset.Rows.Select(static r => r.Features)];
|
||||
int[] y = [.. dataset.Rows.Select(static r => r.Label)];
|
||||
double[] w = [.. dataset.Rows.Select(static r => r.Weight)];
|
||||
int[] starts = [.. dataset.Rows.Select(static r => r.Index)];
|
||||
int[] ends = [.. dataset.Rows.Select(static r => r.EndIndex)];
|
||||
|
||||
List<CpcvSplit> splits = PurgedCv.Combinatorial(n, groups, testGroups, starts, ends, embargo);
|
||||
List<double[]> testPredictions = new(splits.Count);
|
||||
|
||||
foreach (CpcvSplit split in splits)
|
||||
{
|
||||
double[][] trainX = [.. split.Train.Select(i => x[i])];
|
||||
int[] trainY = [.. split.Train.Select(i => y[i])];
|
||||
double[] trainW = [.. split.Train.Select(i => w[i])];
|
||||
|
||||
GbdtModel fold = Gbdt.Train(trainX, trainY, trainW, parameters, dataset.FeatureNames, seed);
|
||||
testPredictions.Add([.. split.Test.Select(i => fold.Predict(x[i]))]);
|
||||
}
|
||||
|
||||
List<double[]> paths = PurgedCv.AssemblePaths(n, groups, splits, testPredictions);
|
||||
|
||||
// ---- out-of-sample statistics ------------------------------------------
|
||||
double correct = 0, counted = 0, logLoss = 0;
|
||||
double[] pathSharpes = new double[paths.Count];
|
||||
List<double> pathReturns = [];
|
||||
|
||||
for (int p = 0; p < paths.Count; p++)
|
||||
{
|
||||
List<double> returns = [];
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
double prob = paths[p][i];
|
||||
if (!double.IsFinite(prob))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
counted++;
|
||||
correct += (prob >= 0.5 ? 1 : 0) == y[i] ? 1 : 0;
|
||||
double clipped = Math.Clamp(prob, 1e-6, 1 - 1e-6);
|
||||
logLoss += -((y[i] * Math.Log(clipped)) + ((1 - y[i]) * Math.Log(1 - clipped)));
|
||||
|
||||
// The bet is the primary's return, scaled by how sure the secondary is,
|
||||
// and nothing at all below a coin flip.
|
||||
double size = Math.Max(0, Performance.BetSizeFromProbability(prob));
|
||||
returns.Add(size * dataset.Rows[i].Return);
|
||||
}
|
||||
|
||||
pathSharpes[p] = Performance.Sharpe(returns);
|
||||
if (p == 0)
|
||||
{
|
||||
pathReturns = returns;
|
||||
}
|
||||
}
|
||||
|
||||
double meanSharpe = pathSharpes.Length > 0 ? pathSharpes.Average() : 0;
|
||||
|
||||
// Every path covers every row once, so a path has as many observations as the
|
||||
// dataset — but overlapping labels are not independent observations, and the
|
||||
// confidence is computed on the effective count, not the row count.
|
||||
int perPath = (int)Math.Round(Math.Min(counted / Math.Max(1, paths.Count), dataset.EffectiveCount));
|
||||
|
||||
double skew = Performance.Skewness(pathReturns);
|
||||
double kurt = Performance.Kurtosis(pathReturns);
|
||||
double psr = Performance.ProbabilisticSharpe(meanSharpe, 0, Math.Max(2, perPath), skew, kurt);
|
||||
double dsr = Performance.DeflatedSharpe(meanSharpe, Math.Max(2, perPath), skew, kurt, Math.Max(1, trials),
|
||||
trialSharpeVariance);
|
||||
|
||||
GbdtModel final = Gbdt.Train(x, y, w, parameters, dataset.FeatureNames, seed);
|
||||
|
||||
string motivazione = string.Create(CultureInfo.InvariantCulture,
|
||||
$"CPCV {groups}/{testGroups} su {n} righe, {perPath} osservazioni indipendenti ({paths.Count} percorsi): accuratezza OOS " +
|
||||
$"{correct / Math.Max(1, counted):P1}, Sharpe medio per operazione {meanSharpe:F3} " +
|
||||
$"(min {pathSharpes.Min():F3}, max {pathSharpes.Max():F3}), PSR {psr:F3}, DSR {dsr:F3} " +
|
||||
$"contando {trials} configurazioni provate; {parameters.Describe()}");
|
||||
|
||||
return new MetaTrainingResult(
|
||||
final, n, paths.Count,
|
||||
counted > 0 ? correct / counted : 0,
|
||||
counted > 0 ? logLoss / counted : 0,
|
||||
pathSharpes, meanSharpe, psr, dsr,
|
||||
[.. pathReturns],
|
||||
motivazione);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
using Encelado.Core.Market;
|
||||
using Encelado.Core.Strategies.Pairs;
|
||||
|
||||
namespace Encelado.Core.Ml;
|
||||
|
||||
/// <summary>
|
||||
/// The rolling history a live pair needs to compute the same features the model was
|
||||
/// trained on.
|
||||
/// <para>
|
||||
/// Training computed every feature from a <see cref="PairSeries"/> over the whole
|
||||
/// file; the bot must produce the identical numbers from the last few hundred bars it
|
||||
/// has in memory. The only way to guarantee "identical" is to compute them the same
|
||||
/// way — so this keeps enough aligned bars for the longest window plus the z-score
|
||||
/// window, rebuilds the series and asks the registry for the last row. At one call per
|
||||
/// closed bar that is a few thousand multiplications: cheap enough to be correct rather
|
||||
/// than clever.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class PairFeatureBuffer
|
||||
{
|
||||
private readonly List<Bar> _a = [];
|
||||
private readonly List<Bar> _b = [];
|
||||
private readonly List<double> _beta = [];
|
||||
private readonly List<double> _alpha = [];
|
||||
private readonly List<double> _halfLife = [];
|
||||
private readonly List<double> _z = [];
|
||||
private readonly int _capacity;
|
||||
private readonly int _zWindow;
|
||||
|
||||
/// <param name="zWindow">The strategy's z-score window, so the rolling z matches the one it trades on.</param>
|
||||
/// <param name="longestFeatureWindow">The longest look-back any feature uses.</param>
|
||||
public PairFeatureBuffer(int zWindow, int longestFeatureWindow = 128)
|
||||
{
|
||||
_zWindow = Math.Max(20, zWindow);
|
||||
_capacity = _zWindow + Math.Max(16, longestFeatureWindow) + 8;
|
||||
}
|
||||
|
||||
public int Count => _a.Count;
|
||||
|
||||
public int Capacity => _capacity;
|
||||
|
||||
/// <summary>Whether enough bars are in hand for every feature to be finite.</summary>
|
||||
public bool IsReady => _a.Count >= _capacity - 8;
|
||||
|
||||
/// <summary>
|
||||
/// Replaces the history with a warm-up download. <paramref name="zScores"/> is the
|
||||
/// strategy's z-score after each of those bars, NaN where it was not ready.
|
||||
/// </summary>
|
||||
public void Seed(
|
||||
IReadOnlyList<Bar> barsA, IReadOnlyList<Bar> barsB, in PairCalibration calibration,
|
||||
IReadOnlyList<double>? zScores = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(barsA);
|
||||
ArgumentNullException.ThrowIfNull(barsB);
|
||||
|
||||
_a.Clear();
|
||||
_b.Clear();
|
||||
_beta.Clear();
|
||||
_alpha.Clear();
|
||||
_halfLife.Clear();
|
||||
_z.Clear();
|
||||
|
||||
int n = Math.Min(barsA.Count, barsB.Count);
|
||||
for (int i = Math.Max(0, n - _capacity); i < n; i++)
|
||||
{
|
||||
Append(barsA[i], barsB[i], calibration, zScores is not null && i < zScores.Count ? zScores[i] : double.NaN);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Adds one aligned closed bar with the calibration in force at that bar and the
|
||||
/// z-score the strategy computed on it.
|
||||
/// </summary>
|
||||
public void Append(in Bar barA, in Bar barB, in PairCalibration calibration, double zScore)
|
||||
{
|
||||
if (_a.Count > 0 && barA.TimeUtc <= _a[^1].TimeUtc)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
_a.Add(barA);
|
||||
_b.Add(barB);
|
||||
_beta.Add(calibration.IsValid ? calibration.Beta : double.NaN);
|
||||
_alpha.Add(calibration.IsValid ? calibration.Alpha : double.NaN);
|
||||
_halfLife.Add(calibration.IsValid ? calibration.HalfLifeBars : double.NaN);
|
||||
_z.Add(zScore);
|
||||
|
||||
if (_a.Count > _capacity)
|
||||
{
|
||||
int drop = _a.Count - _capacity;
|
||||
_a.RemoveRange(0, drop);
|
||||
_b.RemoveRange(0, drop);
|
||||
_beta.RemoveRange(0, drop);
|
||||
_alpha.RemoveRange(0, drop);
|
||||
_halfLife.RemoveRange(0, drop);
|
||||
_z.RemoveRange(0, drop);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The feature row for the most recent bar, in <paramref name="registry"/> order,
|
||||
/// or null when the history is too short for every feature to be finite.
|
||||
/// <paramref name="latencyMs"/> is written into the <c>latency_ms</c> slot.
|
||||
/// </summary>
|
||||
public double[]? Latest(FeatureRegistry registry, double latencyMs)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(registry);
|
||||
|
||||
if (_a.Count < 30)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
PairSeries series = PairSeries.Build(_a, _b, _beta, _alpha, _halfLife, _zWindow, _z);
|
||||
double[][] rows = registry.Compute(series);
|
||||
double[] last = rows[^1];
|
||||
|
||||
int latency = IndexOf(registry.Names, "latency_ms");
|
||||
if (latency >= 0)
|
||||
{
|
||||
last[latency] = latencyMs;
|
||||
}
|
||||
|
||||
foreach (double v in last)
|
||||
{
|
||||
if (!double.IsFinite(v))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
return last;
|
||||
}
|
||||
|
||||
private static int IndexOf(IReadOnlyList<string> names, string name)
|
||||
{
|
||||
for (int i = 0; i < names.Count; i++)
|
||||
{
|
||||
if (names[i] == name)
|
||||
{
|
||||
return i;
|
||||
}
|
||||
}
|
||||
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,153 @@
|
||||
using Encelado.Core.Indicators;
|
||||
using Encelado.Core.Market;
|
||||
|
||||
namespace Encelado.Core.Ml;
|
||||
|
||||
/// <summary>
|
||||
/// An aligned pair with everything a feature can be computed from: both legs' bars,
|
||||
/// the walk-forward hedge ratio at each bar, the spread it implies and the rolling
|
||||
/// z-score of that spread.
|
||||
/// <para>
|
||||
/// Built once and shared by every feature, so no feature recomputes the spread and no
|
||||
/// two features can disagree about what it was. The hedge ratio is a step function of
|
||||
/// time — whatever the last recalibration produced — which is exactly the information
|
||||
/// the live bot has at that bar and not one bar more.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class PairSeries
|
||||
{
|
||||
private PairSeries(int count)
|
||||
{
|
||||
Count = count;
|
||||
Times = new DateTime[count];
|
||||
CloseA = new double[count];
|
||||
CloseB = new double[count];
|
||||
VolumeA = new double[count];
|
||||
VolumeB = new double[count];
|
||||
TakerBuyA = new double[count];
|
||||
TakerBuyB = new double[count];
|
||||
Beta = new double[count];
|
||||
Alpha = new double[count];
|
||||
HalfLife = new double[count];
|
||||
Spread = new double[count];
|
||||
ZScore = new double[count];
|
||||
}
|
||||
|
||||
public int Count { get; }
|
||||
|
||||
public DateTime[] Times { get; }
|
||||
|
||||
public double[] CloseA { get; }
|
||||
|
||||
public double[] CloseB { get; }
|
||||
|
||||
public double[] VolumeA { get; }
|
||||
|
||||
public double[] VolumeB { get; }
|
||||
|
||||
public double[] TakerBuyA { get; }
|
||||
|
||||
public double[] TakerBuyB { get; }
|
||||
|
||||
/// <summary>The hedge ratio in force at each bar.</summary>
|
||||
public double[] Beta { get; }
|
||||
|
||||
public double[] Alpha { get; }
|
||||
|
||||
public double[] HalfLife { get; }
|
||||
|
||||
/// <summary><c>ln(A) − β·ln(B) − α</c>, with the β and α in force at that bar.</summary>
|
||||
public double[] Spread { get; }
|
||||
|
||||
/// <summary>Rolling z-score of the spread; NaN until the window is full.</summary>
|
||||
public double[] ZScore { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Assembles the series. <paramref name="beta"/>, <paramref name="alpha"/> and
|
||||
/// <paramref name="halfLife"/> carry one value per bar — NaN where no calibration
|
||||
/// existed yet.
|
||||
/// <para>
|
||||
/// The z-score is the strategy's own when <paramref name="zScore"/> is given: that is
|
||||
/// the number the rule actually decided on, including its warm-ups after every
|
||||
/// recalibration, and a feature computed from anything else would describe a
|
||||
/// different spread than the one being traded. Without it a rolling z-score over
|
||||
/// <paramref name="zWindow"/> bars is computed here, restarting whenever β moves
|
||||
/// materially, for the same reason the live strategy does it.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static PairSeries Build(
|
||||
IReadOnlyList<Bar> barsA,
|
||||
IReadOnlyList<Bar> barsB,
|
||||
IReadOnlyList<double> beta,
|
||||
IReadOnlyList<double> alpha,
|
||||
IReadOnlyList<double> halfLife,
|
||||
int zWindow,
|
||||
IReadOnlyList<double>? zScore = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(barsA);
|
||||
ArgumentNullException.ThrowIfNull(barsB);
|
||||
ArgumentNullException.ThrowIfNull(beta);
|
||||
ArgumentNullException.ThrowIfNull(alpha);
|
||||
ArgumentNullException.ThrowIfNull(halfLife);
|
||||
|
||||
int n = Math.Min(barsA.Count, barsB.Count);
|
||||
if (beta.Count < n || alpha.Count < n || halfLife.Count < n)
|
||||
{
|
||||
throw new ArgumentException("Le calibrazioni devono coprire ogni barra.");
|
||||
}
|
||||
|
||||
if (zScore is not null && zScore.Count < n)
|
||||
{
|
||||
throw new ArgumentException("Lo z-score, se dato, deve coprire ogni barra.");
|
||||
}
|
||||
|
||||
PairSeries s = new(n);
|
||||
RollingZScore z = new(Math.Max(20, zWindow));
|
||||
double lastBeta = double.NaN;
|
||||
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
Bar a = barsA[i];
|
||||
Bar b = barsB[i];
|
||||
|
||||
s.Times[i] = a.TimeUtc;
|
||||
s.CloseA[i] = a.Close;
|
||||
s.CloseB[i] = b.Close;
|
||||
s.VolumeA[i] = a.Volume;
|
||||
s.VolumeB[i] = b.Volume;
|
||||
s.TakerBuyA[i] = a.TakerBuyVolume;
|
||||
s.TakerBuyB[i] = b.TakerBuyVolume;
|
||||
s.Beta[i] = beta[i];
|
||||
s.Alpha[i] = alpha[i];
|
||||
s.HalfLife[i] = halfLife[i];
|
||||
|
||||
if (!double.IsFinite(beta[i]) || a.Close <= 0 || b.Close <= 0)
|
||||
{
|
||||
s.Spread[i] = double.NaN;
|
||||
s.ZScore[i] = double.NaN;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (double.IsFinite(lastBeta) &&
|
||||
Math.Abs(beta[i] - lastBeta) > Math.Max(1e-4, Math.Abs(lastBeta) * 0.05))
|
||||
{
|
||||
z.Reset();
|
||||
}
|
||||
|
||||
lastBeta = beta[i];
|
||||
|
||||
s.Spread[i] = Math.Log(a.Close) - (beta[i] * Math.Log(b.Close)) - (double.IsFinite(alpha[i]) ? alpha[i] : 0);
|
||||
|
||||
if (zScore is not null)
|
||||
{
|
||||
s.ZScore[i] = zScore[i];
|
||||
continue;
|
||||
}
|
||||
|
||||
double value = z.Update(s.Spread[i]);
|
||||
s.ZScore[i] = z.IsReady ? value : double.NaN;
|
||||
}
|
||||
|
||||
return s;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
namespace Encelado.Core.Ml;
|
||||
|
||||
/// <summary>
|
||||
/// Weights that tell the model how much of each label is really its own.
|
||||
/// <para>
|
||||
/// Two events opened three bars apart, each labelled over the next hundred bars, are
|
||||
/// not two observations: they share ninety-seven of their hundred bars, and the price
|
||||
/// path that resolved one resolved the other. A model shown both at full weight learns
|
||||
/// that path twice. The average uniqueness of a label is the mean, over its own span,
|
||||
/// of one over the number of labels alive at each bar — one for a label alone in time,
|
||||
/// approaching zero for one buried in a crowd.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class SampleWeights
|
||||
{
|
||||
/// <summary>
|
||||
/// Average uniqueness per label, scaled so the weights average one. Zero-length spans
|
||||
/// count as one bar.
|
||||
/// </summary>
|
||||
public static double[] AverageUniqueness(IReadOnlyList<BarrierLabel> labels, int seriesLength)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(labels);
|
||||
|
||||
if (labels.Count == 0)
|
||||
{
|
||||
return [];
|
||||
}
|
||||
|
||||
// Concurrency: how many labels cover each bar.
|
||||
int[] concurrency = new int[Math.Max(seriesLength, 1)];
|
||||
foreach (BarrierLabel label in labels)
|
||||
{
|
||||
int from = Math.Clamp(label.Index, 0, concurrency.Length - 1);
|
||||
int to = Math.Clamp(Math.Max(label.EndIndex, label.Index), 0, concurrency.Length - 1);
|
||||
for (int t = from; t <= to; t++)
|
||||
{
|
||||
concurrency[t]++;
|
||||
}
|
||||
}
|
||||
|
||||
double[] weights = new double[labels.Count];
|
||||
double total = 0;
|
||||
|
||||
for (int i = 0; i < labels.Count; i++)
|
||||
{
|
||||
BarrierLabel label = labels[i];
|
||||
int from = Math.Clamp(label.Index, 0, concurrency.Length - 1);
|
||||
int to = Math.Clamp(Math.Max(label.EndIndex, label.Index), 0, concurrency.Length - 1);
|
||||
|
||||
double sum = 0;
|
||||
for (int t = from; t <= to; t++)
|
||||
{
|
||||
sum += 1.0 / Math.Max(1, concurrency[t]);
|
||||
}
|
||||
|
||||
weights[i] = sum / (to - from + 1);
|
||||
total += weights[i];
|
||||
}
|
||||
|
||||
if (total > 0)
|
||||
{
|
||||
double scale = labels.Count / total;
|
||||
for (int i = 0; i < weights.Length; i++)
|
||||
{
|
||||
weights[i] *= scale;
|
||||
}
|
||||
}
|
||||
|
||||
return weights;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// How many independent observations the labels amount to: the sum of their raw
|
||||
/// average uniqueness. Six hundred signals that come in clusters of ten are sixty
|
||||
/// observations, and any confidence computed on six hundred is a lie.
|
||||
/// </summary>
|
||||
public static double EffectiveCount(IReadOnlyList<BarrierLabel> labels, int seriesLength)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(labels);
|
||||
|
||||
if (labels.Count == 0)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
int[] concurrency = new int[Math.Max(seriesLength, 1)];
|
||||
foreach (BarrierLabel label in labels)
|
||||
{
|
||||
int from = Math.Clamp(label.Index, 0, concurrency.Length - 1);
|
||||
int to = Math.Clamp(Math.Max(label.EndIndex, label.Index), 0, concurrency.Length - 1);
|
||||
for (int t = from; t <= to; t++)
|
||||
{
|
||||
concurrency[t]++;
|
||||
}
|
||||
}
|
||||
|
||||
double total = 0;
|
||||
foreach (BarrierLabel label in labels)
|
||||
{
|
||||
int from = Math.Clamp(label.Index, 0, concurrency.Length - 1);
|
||||
int to = Math.Clamp(Math.Max(label.EndIndex, label.Index), 0, concurrency.Length - 1);
|
||||
double sum = 0;
|
||||
for (int t = from; t <= to; t++)
|
||||
{
|
||||
sum += 1.0 / Math.Max(1, concurrency[t]);
|
||||
}
|
||||
|
||||
total += sum / (to - from + 1);
|
||||
}
|
||||
|
||||
return total;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
namespace Encelado.Core.Ml;
|
||||
|
||||
/// <summary>
|
||||
/// One labelled event: where it started, where its barrier was touched, which barrier,
|
||||
/// and what the position would have made.
|
||||
/// </summary>
|
||||
public readonly record struct BarrierLabel(
|
||||
int Index,
|
||||
int EndIndex,
|
||||
int Label,
|
||||
double Return,
|
||||
double Volatility,
|
||||
int Side)
|
||||
{
|
||||
/// <summary>The meta-label: did the side turn out to be right?</summary>
|
||||
public int MetaLabel => Return > 0 ? 1 : 0;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Triple-barrier labelling (López de Prado): a label is not "did the price go up over
|
||||
/// the next N bars" but "which of three barriers did it hit first" — a profit-take
|
||||
/// above, a stop-loss below, or the clock.
|
||||
/// <para>
|
||||
/// The two horizontal barriers scale with the local volatility, so a label means the
|
||||
/// same thing in a quiet week and a wild one: "moved two standard deviations in our
|
||||
/// favour before moving one against", not "moved two per cent". The vertical barrier is
|
||||
/// what makes every label finite and every sample weightable.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Given a <c>side</c> (+1 long, −1 short) the barriers are mirrored, and the return is
|
||||
/// signed by the side — which is exactly what meta-labelling needs: not "did price go
|
||||
/// up" but "was the strategy's call right".
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class TripleBarrier
|
||||
{
|
||||
/// <summary>
|
||||
/// Exponentially weighted standard deviation of log returns, one value per bar. The
|
||||
/// scale the barriers are measured in.
|
||||
/// </summary>
|
||||
public static double[] RollingVolatility(IReadOnlyList<double> prices, int span = 100)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(prices);
|
||||
|
||||
double alpha = 2.0 / (Math.Max(2, span) + 1);
|
||||
double[] vol = new double[prices.Count];
|
||||
double mean = 0;
|
||||
double variance = 0;
|
||||
bool seeded = false;
|
||||
|
||||
vol[0] = double.NaN;
|
||||
for (int t = 1; t < prices.Count; t++)
|
||||
{
|
||||
double r = prices[t - 1] > 0 && prices[t] > 0 ? Math.Log(prices[t] / prices[t - 1]) : 0;
|
||||
|
||||
if (!seeded)
|
||||
{
|
||||
mean = r;
|
||||
variance = r * r;
|
||||
seeded = true;
|
||||
}
|
||||
else
|
||||
{
|
||||
double delta = r - mean;
|
||||
mean += alpha * delta;
|
||||
variance = ((1 - alpha) * variance) + (alpha * delta * (r - mean));
|
||||
}
|
||||
|
||||
vol[t] = t < 5 ? double.NaN : Math.Sqrt(Math.Max(0, variance));
|
||||
}
|
||||
|
||||
return vol;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Labels every event in <paramref name="events"/>.
|
||||
/// </summary>
|
||||
/// <param name="prices">The price path, oldest first.</param>
|
||||
/// <param name="events">Indices at which a decision was made.</param>
|
||||
/// <param name="sides">The side taken at each event, +1 or −1. Null means long everywhere.</param>
|
||||
/// <param name="volatility">Per-bar volatility, from <see cref="RollingVolatility"/>.</param>
|
||||
/// <param name="profitMultiple">Upper barrier, in volatilities. 0 disables it.</param>
|
||||
/// <param name="stopMultiple">Lower barrier, in volatilities. 0 disables it.</param>
|
||||
/// <param name="maxHoldingBars">The vertical barrier.</param>
|
||||
public static List<BarrierLabel> Label(
|
||||
IReadOnlyList<double> prices,
|
||||
IReadOnlyList<int> events,
|
||||
IReadOnlyList<int>? sides,
|
||||
IReadOnlyList<double> volatility,
|
||||
double profitMultiple,
|
||||
double stopMultiple,
|
||||
int maxHoldingBars)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(prices);
|
||||
ArgumentNullException.ThrowIfNull(events);
|
||||
ArgumentNullException.ThrowIfNull(volatility);
|
||||
|
||||
if (maxHoldingBars < 1)
|
||||
{
|
||||
throw new ArgumentOutOfRangeException(nameof(maxHoldingBars), "serve almeno una barra");
|
||||
}
|
||||
|
||||
List<BarrierLabel> labels = new(events.Count);
|
||||
|
||||
for (int e = 0; e < events.Count; e++)
|
||||
{
|
||||
int i = events[e];
|
||||
if (i < 0 || i >= prices.Count - 1)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
int side = sides is null ? 1 : Math.Sign(sides[e]);
|
||||
if (side == 0)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
double sigma = i < volatility.Count ? volatility[i] : double.NaN;
|
||||
if (!double.IsFinite(sigma) || sigma <= 0 || prices[i] <= 0)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
double upper = profitMultiple > 0 ? profitMultiple * sigma : double.PositiveInfinity;
|
||||
double lower = stopMultiple > 0 ? -stopMultiple * sigma : double.NegativeInfinity;
|
||||
|
||||
int last = Math.Min(prices.Count - 1, i + maxHoldingBars);
|
||||
int label = 0;
|
||||
int end = last;
|
||||
double ret = 0;
|
||||
|
||||
for (int j = i + 1; j <= last; j++)
|
||||
{
|
||||
ret = side * ((prices[j] / prices[i]) - 1);
|
||||
|
||||
if (ret >= upper)
|
||||
{
|
||||
label = 1;
|
||||
end = j;
|
||||
break;
|
||||
}
|
||||
|
||||
if (ret <= lower)
|
||||
{
|
||||
label = -1;
|
||||
end = j;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (label == 0)
|
||||
{
|
||||
// The clock ran out: the sign of what was there, and zero if nothing was.
|
||||
ret = side * ((prices[end] / prices[i]) - 1);
|
||||
label = Math.Abs(ret) < 1e-12 ? 0 : Math.Sign(ret);
|
||||
}
|
||||
|
||||
labels.Add(new BarrierLabel(i, end, label, ret, sigma, side));
|
||||
}
|
||||
|
||||
return labels;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,403 @@
|
||||
using Encelado.Core.Statistics;
|
||||
|
||||
namespace Encelado.Core.Ml;
|
||||
|
||||
/// <summary>One train/test partition of the rows.</summary>
|
||||
public sealed record CvSplit(int[] Train, int[] Test);
|
||||
|
||||
/// <summary>A CPCV partition: which groups are held out, and the rows on each side.</summary>
|
||||
public sealed record CpcvSplit(int[] Train, int[] Test, int[] TestGroups);
|
||||
|
||||
/// <summary>
|
||||
/// Cross-validation that respects time.
|
||||
/// <para>
|
||||
/// A label that spans a hundred bars leaks a hundred bars of the future into any train
|
||||
/// fold that contains it; ordinary k-fold hands the model the answers to the test fold
|
||||
/// through the labels that overlap it. <b>Purging</b> removes from the train set every
|
||||
/// row whose label span touches the test span; the <b>embargo</b> drops a further
|
||||
/// stretch after the test block, because serial correlation leaks even between rows
|
||||
/// that do not overlap. Both are López de Prado's, and both are the difference between
|
||||
/// an out-of-sample number and a story.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class PurgedCv
|
||||
{
|
||||
/// <summary>
|
||||
/// K contiguous folds, each used as the test set once, with purging and embargo.
|
||||
/// <paramref name="starts"/> and <paramref name="ends"/> are the label spans in bar
|
||||
/// indices; rows must be in time order.
|
||||
/// </summary>
|
||||
public static List<CvSplit> KFold(
|
||||
int rows, int folds, IReadOnlyList<int> starts, IReadOnlyList<int> ends, double embargoFraction = 0.01)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(starts);
|
||||
ArgumentNullException.ThrowIfNull(ends);
|
||||
|
||||
if (folds < 2 || rows < folds)
|
||||
{
|
||||
throw new ArgumentException("servono almeno due fold e più righe che fold");
|
||||
}
|
||||
|
||||
List<CvSplit> splits = [];
|
||||
int embargo = EmbargoBars(starts, ends, embargoFraction);
|
||||
|
||||
for (int k = 0; k < folds; k++)
|
||||
{
|
||||
int from = k * rows / folds;
|
||||
int to = (k + 1) * rows / folds;
|
||||
int[] test = Enumerable.Range(from, to - from).ToArray();
|
||||
int[] train = Purge(rows, [test], starts, ends, embargo);
|
||||
splits.Add(new CvSplit(train, test));
|
||||
}
|
||||
|
||||
return splits;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Combinatorial purged cross-validation: every choice of <paramref name="testGroups"/>
|
||||
/// out of <paramref name="groups"/> contiguous blocks becomes a split, and the test
|
||||
/// predictions recombine into <c>k/N·C(N,k)</c> complete out-of-sample paths — with
|
||||
/// N = 6, k = 2, fifteen splits and five paths through the whole history, instead
|
||||
/// of the single path a walk-forward gives. The distribution of results across
|
||||
/// paths is what says whether one lucky stretch carried the number.
|
||||
/// </summary>
|
||||
public static List<CpcvSplit> Combinatorial(
|
||||
int rows, int groups, int testGroups, IReadOnlyList<int> starts, IReadOnlyList<int> ends,
|
||||
double embargoFraction = 0.01)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(starts);
|
||||
ArgumentNullException.ThrowIfNull(ends);
|
||||
|
||||
if (groups < 2 || testGroups < 1 || testGroups >= groups || rows < groups)
|
||||
{
|
||||
throw new ArgumentException("gruppi e gruppi di test non validi");
|
||||
}
|
||||
|
||||
int embargo = EmbargoBars(starts, ends, embargoFraction);
|
||||
int[][] blocks = new int[groups][];
|
||||
for (int g = 0; g < groups; g++)
|
||||
{
|
||||
int from = g * rows / groups;
|
||||
int to = (g + 1) * rows / groups;
|
||||
blocks[g] = Enumerable.Range(from, to - from).ToArray();
|
||||
}
|
||||
|
||||
List<CpcvSplit> splits = [];
|
||||
foreach (int[] chosen in Combinations(groups, testGroups))
|
||||
{
|
||||
int[] test = [.. chosen.SelectMany(g => blocks[g]).Order()];
|
||||
|
||||
// Purged block by block: two held-out groups at opposite ends of the history
|
||||
// are two spans, not one span covering everything in between.
|
||||
int[] train = Purge(rows, [.. chosen.Select(g => blocks[g])], starts, ends, embargo);
|
||||
splits.Add(new CpcvSplit(train, test, chosen));
|
||||
}
|
||||
|
||||
return splits;
|
||||
}
|
||||
|
||||
/// <summary>How many complete out-of-sample paths <see cref="Combinatorial"/> produces.</summary>
|
||||
public static int PathCount(int groups, int testGroups) =>
|
||||
(int)Math.Round(testGroups * (double)Choose(groups, testGroups) / groups);
|
||||
|
||||
/// <summary>
|
||||
/// Recombines per-split test predictions into full paths. Each group appears in the
|
||||
/// test set of several splits; its j-th such appearance lands on path j, so every
|
||||
/// path covers every group exactly once. Rows never predicted stay NaN.
|
||||
/// </summary>
|
||||
public static List<double[]> AssemblePaths(
|
||||
int rows, int groups, IReadOnlyList<CpcvSplit> splits, IReadOnlyList<double[]> testPredictions)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(splits);
|
||||
ArgumentNullException.ThrowIfNull(testPredictions);
|
||||
|
||||
int[] groupOfRow = new int[rows];
|
||||
for (int g = 0; g < groups; g++)
|
||||
{
|
||||
int from = g * rows / groups;
|
||||
int to = (g + 1) * rows / groups;
|
||||
for (int i = from; i < to; i++)
|
||||
{
|
||||
groupOfRow[i] = g;
|
||||
}
|
||||
}
|
||||
|
||||
int[] seen = new int[groups];
|
||||
List<double[]> paths = [];
|
||||
|
||||
for (int s = 0; s < splits.Count; s++)
|
||||
{
|
||||
CpcvSplit split = splits[s];
|
||||
double[] predictions = testPredictions[s];
|
||||
|
||||
foreach (int g in split.TestGroups)
|
||||
{
|
||||
int path = seen[g]++;
|
||||
while (paths.Count <= path)
|
||||
{
|
||||
double[] fresh = new double[rows];
|
||||
Array.Fill(fresh, double.NaN);
|
||||
paths.Add(fresh);
|
||||
}
|
||||
|
||||
for (int t = 0; t < split.Test.Length; t++)
|
||||
{
|
||||
int row = split.Test[t];
|
||||
if (groupOfRow[row] == g)
|
||||
{
|
||||
paths[path][row] = predictions[t];
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return paths;
|
||||
}
|
||||
|
||||
private static int[] Purge(
|
||||
int rows, IReadOnlyList<int[]> testBlocks, IReadOnlyList<int> starts, IReadOnlyList<int> ends, int embargo)
|
||||
{
|
||||
bool[] isTest = new bool[rows];
|
||||
List<(int Start, int End)> spans = [];
|
||||
|
||||
foreach (int[] block in testBlocks)
|
||||
{
|
||||
if (block.Length == 0)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
int spanStart = int.MaxValue;
|
||||
int spanEnd = int.MinValue;
|
||||
foreach (int i in block)
|
||||
{
|
||||
isTest[i] = true;
|
||||
spanStart = Math.Min(spanStart, starts[i]);
|
||||
spanEnd = Math.Max(spanEnd, ends[i]);
|
||||
}
|
||||
|
||||
spans.Add((spanStart, spanEnd));
|
||||
}
|
||||
|
||||
List<int> train = [];
|
||||
for (int i = 0; i < rows; i++)
|
||||
{
|
||||
if (isTest[i])
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
bool keep = true;
|
||||
foreach ((int testStart, int testEnd) in spans)
|
||||
{
|
||||
// Purge: any overlap between the row's label span and a test span.
|
||||
if (ends[i] >= testStart && starts[i] <= testEnd)
|
||||
{
|
||||
keep = false;
|
||||
break;
|
||||
}
|
||||
|
||||
// Embargo: rows that start just after a test span.
|
||||
if (starts[i] > testEnd && starts[i] <= testEnd + embargo)
|
||||
{
|
||||
keep = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (keep)
|
||||
{
|
||||
train.Add(i);
|
||||
}
|
||||
}
|
||||
|
||||
return [.. train];
|
||||
}
|
||||
|
||||
private static int EmbargoBars(IReadOnlyList<int> starts, IReadOnlyList<int> ends, double fraction)
|
||||
{
|
||||
if (starts.Count == 0)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
int span = ends.Max() - starts.Min();
|
||||
return (int)Math.Ceiling(Math.Max(0, fraction) * span);
|
||||
}
|
||||
|
||||
public static long Choose(int n, int k)
|
||||
{
|
||||
if (k < 0 || k > n)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
long result = 1;
|
||||
for (int i = 1; i <= k; i++)
|
||||
{
|
||||
result = result * (n - k + i) / i;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
private static IEnumerable<int[]> Combinations(int n, int k)
|
||||
{
|
||||
int[] current = new int[k];
|
||||
return Recurse(0, 0);
|
||||
|
||||
IEnumerable<int[]> Recurse(int start, int depth)
|
||||
{
|
||||
if (depth == k)
|
||||
{
|
||||
yield return (int[])current.Clone();
|
||||
yield break;
|
||||
}
|
||||
|
||||
for (int i = start; i <= n - (k - depth); i++)
|
||||
{
|
||||
current[depth] = i;
|
||||
foreach (int[] c in Recurse(i + 1, depth + 1))
|
||||
{
|
||||
yield return c;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>The outcome of the overfitting test.</summary>
|
||||
public readonly record struct PboResult(double Probability, double MeanLogit, int Combinations);
|
||||
|
||||
/// <summary>
|
||||
/// The probability of backtest overfitting (Bailey, Borwein, López de Prado, Zhu), by
|
||||
/// combinatorially symmetric cross-validation.
|
||||
/// <para>
|
||||
/// Given the return series of every configuration that was tried, split time into
|
||||
/// blocks, and for each way of choosing half the blocks as "in-sample": find the
|
||||
/// configuration that was best in-sample, then ask where it ranks out-of-sample. If
|
||||
/// the best in-sample choice is below the median out-of-sample more often than not,
|
||||
/// the selection process is picking noise — and that frequency is the PBO. Below 0.5
|
||||
/// is the bar the strategy note sets.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class Pbo
|
||||
{
|
||||
/// <summary>
|
||||
/// <paramref name="returnsByTrial"/> holds one return series per configuration, all
|
||||
/// the same length and aligned in time. Needs at least two trials and enough
|
||||
/// periods for <paramref name="blocks"/> blocks.
|
||||
/// </summary>
|
||||
public static PboResult Compute(IReadOnlyList<double[]> returnsByTrial, int blocks = 8)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(returnsByTrial);
|
||||
|
||||
int trials = returnsByTrial.Count;
|
||||
if (trials < 2)
|
||||
{
|
||||
return new PboResult(double.NaN, double.NaN, 0);
|
||||
}
|
||||
|
||||
int periods = returnsByTrial.Min(static r => r.Length);
|
||||
if (blocks < 2 || blocks % 2 != 0 || periods < blocks * 4)
|
||||
{
|
||||
return new PboResult(double.NaN, double.NaN, 0);
|
||||
}
|
||||
|
||||
int[][] blockRows = new int[blocks][];
|
||||
for (int b = 0; b < blocks; b++)
|
||||
{
|
||||
int from = b * periods / blocks;
|
||||
int to = (b + 1) * periods / blocks;
|
||||
blockRows[b] = Enumerable.Range(from, to - from).ToArray();
|
||||
}
|
||||
|
||||
int half = blocks / 2;
|
||||
int combinations = 0;
|
||||
int overfit = 0;
|
||||
double logitSum = 0;
|
||||
|
||||
foreach (int[] inSample in Combos(blocks, half))
|
||||
{
|
||||
bool[] isIn = new bool[blocks];
|
||||
foreach (int b in inSample)
|
||||
{
|
||||
isIn[b] = true;
|
||||
}
|
||||
|
||||
double[] sharpeIn = new double[trials];
|
||||
double[] sharpeOut = new double[trials];
|
||||
|
||||
for (int t = 0; t < trials; t++)
|
||||
{
|
||||
List<double> inReturns = [];
|
||||
List<double> outReturns = [];
|
||||
for (int b = 0; b < blocks; b++)
|
||||
{
|
||||
foreach (int i in blockRows[b])
|
||||
{
|
||||
(isIn[b] ? inReturns : outReturns).Add(returnsByTrial[t][i]);
|
||||
}
|
||||
}
|
||||
|
||||
sharpeIn[t] = Performance.Sharpe(inReturns);
|
||||
sharpeOut[t] = Performance.Sharpe(outReturns);
|
||||
}
|
||||
|
||||
int best = 0;
|
||||
for (int t = 1; t < trials; t++)
|
||||
{
|
||||
if (sharpeIn[t] > sharpeIn[best])
|
||||
{
|
||||
best = t;
|
||||
}
|
||||
}
|
||||
|
||||
// Relative rank of the in-sample winner among the out-of-sample results.
|
||||
int below = 0;
|
||||
for (int t = 0; t < trials; t++)
|
||||
{
|
||||
if (sharpeOut[t] < sharpeOut[best])
|
||||
{
|
||||
below++;
|
||||
}
|
||||
}
|
||||
|
||||
double omega = Math.Clamp((below + 0.5) / trials, 1e-6, 1 - 1e-6);
|
||||
double logit = Math.Log(omega / (1 - omega));
|
||||
|
||||
combinations++;
|
||||
logitSum += logit;
|
||||
if (logit <= 0)
|
||||
{
|
||||
overfit++;
|
||||
}
|
||||
}
|
||||
|
||||
return new PboResult(overfit / (double)combinations, logitSum / combinations, combinations);
|
||||
}
|
||||
|
||||
private static IEnumerable<int[]> Combos(int n, int k)
|
||||
{
|
||||
int[] current = new int[k];
|
||||
return Recurse(0, 0);
|
||||
|
||||
IEnumerable<int[]> Recurse(int start, int depth)
|
||||
{
|
||||
if (depth == k)
|
||||
{
|
||||
yield return (int[])current.Clone();
|
||||
yield break;
|
||||
}
|
||||
|
||||
for (int i = start; i <= n - (k - depth); i++)
|
||||
{
|
||||
current[depth] = i;
|
||||
foreach (int[] c in Recurse(i + 1, depth + 1))
|
||||
{
|
||||
yield return c;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,255 @@
|
||||
using System.Globalization;
|
||||
using Encelado.Core.Backtest;
|
||||
|
||||
namespace Encelado.Core.Rl;
|
||||
|
||||
/// <summary>The knobs of the agent.</summary>
|
||||
public sealed record DqnParams
|
||||
{
|
||||
public int Hidden { get; init; } = 64;
|
||||
|
||||
public double LearningRate { get; init; } = 5e-4;
|
||||
|
||||
public double Gamma { get; init; } = 0.97;
|
||||
|
||||
public int ReplaySize { get; init; } = 50_000;
|
||||
|
||||
public int Batch { get; init; } = 64;
|
||||
|
||||
/// <summary>Gradient steps between target-network refreshes.</summary>
|
||||
public int TargetEvery { get; init; } = 500;
|
||||
|
||||
public double EpsilonStart { get; init; } = 1.0;
|
||||
|
||||
public double EpsilonEnd { get; init; } = 0.05;
|
||||
|
||||
/// <summary>Environment steps over which ε decays linearly.</summary>
|
||||
public int EpsilonSteps { get; init; } = 20_000;
|
||||
|
||||
/// <summary>Environment steps between gradient steps.</summary>
|
||||
public int TrainEvery { get; init; } = 4;
|
||||
|
||||
public int WarmupSteps { get; init; } = 1_000;
|
||||
|
||||
/// <summary>
|
||||
/// Multiplier on the reward before it reaches the network. A bar's spread return is
|
||||
/// of the order of 1e-4 and a cost 5e-4: at that scale the Q-values sit at zero,
|
||||
/// the gradients vanish and the greedy action is decided by rounding noise, which
|
||||
/// is what a policy that flips on every bar looks like. A thousand puts one bar
|
||||
/// near a tenth, where the network can tell a fee from a return.
|
||||
/// </summary>
|
||||
public double RewardScale { get; init; } = 1_000;
|
||||
|
||||
public string Describe() => string.Create(CultureInfo.InvariantCulture,
|
||||
$"hidden={Hidden} lr={LearningRate} γ={Gamma} replay={ReplaySize} batch={Batch} " +
|
||||
$"target={TargetEvery} ε={EpsilonStart}→{EpsilonEnd}/{EpsilonSteps} scala={RewardScale}");
|
||||
}
|
||||
|
||||
/// <summary>What one seed produced on the held-out span.</summary>
|
||||
public sealed record RlSeedResult(int Seed, int Episodes, ReturnMetrics Test, int Trades, double TrainLoss)
|
||||
{
|
||||
public string Describe() => string.Create(CultureInfo.InvariantCulture,
|
||||
$"seed {Seed}: netto {Test.NetReturn:P2}, Sharpe {Test.AnnualSharpe:F2}, DD {Test.MaxDrawdown:P2}, " +
|
||||
$"{Trades} cambi di posizione, loss finale {TrainLoss:F5}");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Deep Q-learning (Mnih et al. 2015) on the pair environment: replay buffer, target
|
||||
/// network, ε-greedy exploration.
|
||||
/// <para>
|
||||
/// Kept deliberately small — two hidden layers of 64 — because the instability the
|
||||
/// strategy note warns about gets worse with capacity, and because a policy that
|
||||
/// needs a large network to find a signal on eleven inputs has usually found the noise.
|
||||
/// The honest measurement is not one run but the spread across seeds, which is what
|
||||
/// <see cref="Experiment"/> reports.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class DqnAgent
|
||||
{
|
||||
private readonly DqnParams _p;
|
||||
private readonly Mlp _online;
|
||||
private readonly Mlp _target;
|
||||
private readonly Random _rng;
|
||||
private readonly Transition[] _replay;
|
||||
private int _replayCount;
|
||||
private int _replayNext;
|
||||
private int _steps;
|
||||
private int _gradSteps;
|
||||
private double _lastLoss;
|
||||
|
||||
private readonly record struct Transition(double[] State, int Action, double Reward, double[] Next, bool Done);
|
||||
|
||||
public DqnAgent(int stateSize, DqnParams p, int seed)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(p);
|
||||
_p = p;
|
||||
_rng = new Random(seed);
|
||||
_online = new Mlp(stateSize, p.Hidden, PairEnvironment.Actions, seed);
|
||||
_target = _online.Clone();
|
||||
_replay = new Transition[Math.Max(p.Batch * 4, p.ReplaySize)];
|
||||
}
|
||||
|
||||
public double Epsilon => _p.EpsilonEnd + ((_p.EpsilonStart - _p.EpsilonEnd) *
|
||||
Math.Max(0, 1 - (_steps / (double)Math.Max(1, _p.EpsilonSteps))));
|
||||
|
||||
public double LastLoss => _lastLoss;
|
||||
|
||||
public int Act(double[] state, bool explore)
|
||||
{
|
||||
if (explore && _rng.NextDouble() < Epsilon)
|
||||
{
|
||||
return _rng.Next(PairEnvironment.Actions);
|
||||
}
|
||||
|
||||
double[] q = _online.Predict(state);
|
||||
int best = 0;
|
||||
for (int a = 1; a < q.Length; a++)
|
||||
{
|
||||
if (q[a] > q[best])
|
||||
{
|
||||
best = a;
|
||||
}
|
||||
}
|
||||
|
||||
return best;
|
||||
}
|
||||
|
||||
public void Observe(double[] state, int action, double reward, double[] next, bool done)
|
||||
{
|
||||
_replay[_replayNext] = new Transition(state, action, reward * _p.RewardScale, next, done);
|
||||
_replayNext = (_replayNext + 1) % _replay.Length;
|
||||
_replayCount = Math.Min(_replayCount + 1, _replay.Length);
|
||||
_steps++;
|
||||
|
||||
if (_steps < _p.WarmupSteps || _steps % _p.TrainEvery != 0 || _replayCount < _p.Batch)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
Learn();
|
||||
}
|
||||
|
||||
private void Learn()
|
||||
{
|
||||
int b = _p.Batch;
|
||||
double[][] states = new double[b][];
|
||||
int[] actions = new int[b];
|
||||
double[] targets = new double[b];
|
||||
|
||||
for (int i = 0; i < b; i++)
|
||||
{
|
||||
Transition t = _replay[_rng.Next(_replayCount)];
|
||||
states[i] = t.State;
|
||||
actions[i] = t.Action;
|
||||
|
||||
double y = t.Reward;
|
||||
if (!t.Done)
|
||||
{
|
||||
// Double DQN: the online network chooses, the target network evaluates.
|
||||
double[] qOnline = _online.Predict(t.Next);
|
||||
int best = 0;
|
||||
for (int a = 1; a < qOnline.Length; a++)
|
||||
{
|
||||
if (qOnline[a] > qOnline[best])
|
||||
{
|
||||
best = a;
|
||||
}
|
||||
}
|
||||
|
||||
y += _p.Gamma * _target.Predict(t.Next)[best];
|
||||
}
|
||||
|
||||
targets[i] = y;
|
||||
}
|
||||
|
||||
_lastLoss = _online.TrainBatch(states, actions, targets, _p.LearningRate);
|
||||
_gradSteps++;
|
||||
|
||||
if (_gradSteps % _p.TargetEvery == 0)
|
||||
{
|
||||
_target.CopyWeightsFrom(_online);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Trains on one environment for a number of passes over its bars.</summary>
|
||||
public void Train(PairEnvironment env, int episodes)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(env);
|
||||
|
||||
for (int e = 0; e < episodes; e++)
|
||||
{
|
||||
double[] state = env.Reset();
|
||||
while (!env.Done)
|
||||
{
|
||||
int action = Act(state, explore: true);
|
||||
(double[] next, double reward, bool done, _) = env.Step(action);
|
||||
Observe(state, action, reward, next, done);
|
||||
state = next;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Runs the greedy policy through an environment and returns the per-bar rewards.</summary>
|
||||
public (double[] Rewards, int Trades) Evaluate(PairEnvironment env)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(env);
|
||||
|
||||
List<double> rewards = new(env.Length);
|
||||
int trades = 0;
|
||||
double[] state = env.Reset();
|
||||
while (!env.Done)
|
||||
{
|
||||
int action = Act(state, explore: false);
|
||||
(double[] next, double reward, _, bool changed) = env.Step(action);
|
||||
rewards.Add(reward);
|
||||
if (changed)
|
||||
{
|
||||
trades++;
|
||||
}
|
||||
|
||||
state = next;
|
||||
}
|
||||
|
||||
return ([.. rewards], trades);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The many-seed experiment the strategy note requires: the same data, the same agent,
|
||||
/// fifteen or more random seeds, and the distribution of held-out results — because a
|
||||
/// single run of a reinforcement learner is an anecdote.
|
||||
/// </summary>
|
||||
public static class Experiment
|
||||
{
|
||||
public static List<RlSeedResult> Run(
|
||||
PairEnvironment train, PairEnvironment test, DqnParams p, int seeds, int episodes,
|
||||
double periodsPerYear, int firstSeed = 1, Action<RlSeedResult>? progress = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(train);
|
||||
ArgumentNullException.ThrowIfNull(test);
|
||||
ArgumentNullException.ThrowIfNull(p);
|
||||
|
||||
RlSeedResult[] results = new RlSeedResult[seeds];
|
||||
|
||||
Parallel.For(0, seeds, i =>
|
||||
{
|
||||
int seed = firstSeed + i;
|
||||
|
||||
// Each seed walks its own copy of the environments: the data is shared, the
|
||||
// position and the clock are not, and fifteen agents stepping one cursor
|
||||
// would be trading each other's positions.
|
||||
PairEnvironment trainEnv = train.Clone();
|
||||
PairEnvironment testEnv = test.Clone();
|
||||
|
||||
DqnAgent agent = new(trainEnv.StateSize, p, seed);
|
||||
agent.Train(trainEnv, episodes);
|
||||
|
||||
(double[] rewards, int trades) = agent.Evaluate(testEnv);
|
||||
ReturnMetrics m = ReturnMetrics.From(rewards, periodsPerYear, trades, trials: seeds);
|
||||
results[i] = new RlSeedResult(seed, episodes, m, trades, agent.LastLoss);
|
||||
progress?.Invoke(results[i]);
|
||||
});
|
||||
|
||||
return [.. results];
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,261 @@
|
||||
namespace Encelado.Core.Rl;
|
||||
|
||||
/// <summary>
|
||||
/// A two-hidden-layer perceptron with ReLU units, trained by Adam on a squared error —
|
||||
/// the whole of the function approximator a small DQN needs, and nothing the runtime
|
||||
/// would have to load a native library for.
|
||||
/// </summary>
|
||||
public sealed class Mlp
|
||||
{
|
||||
private readonly int _inputs;
|
||||
private readonly int _hidden;
|
||||
private readonly int _outputs;
|
||||
|
||||
// Weights: layer 1 [hidden × inputs], layer 2 [hidden × hidden], layer 3 [outputs × hidden].
|
||||
private readonly double[] _w1;
|
||||
private readonly double[] _b1;
|
||||
private readonly double[] _w2;
|
||||
private readonly double[] _b2;
|
||||
private readonly double[] _w3;
|
||||
private readonly double[] _b3;
|
||||
|
||||
// Adam moments, same shapes.
|
||||
private readonly double[] _m1, _v1, _mb1, _vb1;
|
||||
private readonly double[] _m2, _v2, _mb2, _vb2;
|
||||
private readonly double[] _m3, _v3, _mb3, _vb3;
|
||||
private int _step;
|
||||
|
||||
public Mlp(int inputs, int hidden, int outputs, int seed)
|
||||
{
|
||||
if (inputs < 1 || hidden < 1 || outputs < 1)
|
||||
{
|
||||
throw new ArgumentException("dimensioni non valide");
|
||||
}
|
||||
|
||||
_inputs = inputs;
|
||||
_hidden = hidden;
|
||||
_outputs = outputs;
|
||||
|
||||
Random rng = new(seed);
|
||||
_w1 = Init(hidden * inputs, inputs, rng);
|
||||
_b1 = new double[hidden];
|
||||
_w2 = Init(hidden * hidden, hidden, rng);
|
||||
_b2 = new double[hidden];
|
||||
_w3 = Init(outputs * hidden, hidden, rng);
|
||||
_b3 = new double[outputs];
|
||||
|
||||
_m1 = new double[_w1.Length]; _v1 = new double[_w1.Length]; _mb1 = new double[hidden]; _vb1 = new double[hidden];
|
||||
_m2 = new double[_w2.Length]; _v2 = new double[_w2.Length]; _mb2 = new double[hidden]; _vb2 = new double[hidden];
|
||||
_m3 = new double[_w3.Length]; _v3 = new double[_w3.Length]; _mb3 = new double[outputs]; _vb3 = new double[outputs];
|
||||
}
|
||||
|
||||
private Mlp(Mlp other)
|
||||
{
|
||||
_inputs = other._inputs;
|
||||
_hidden = other._hidden;
|
||||
_outputs = other._outputs;
|
||||
_w1 = (double[])other._w1.Clone();
|
||||
_b1 = (double[])other._b1.Clone();
|
||||
_w2 = (double[])other._w2.Clone();
|
||||
_b2 = (double[])other._b2.Clone();
|
||||
_w3 = (double[])other._w3.Clone();
|
||||
_b3 = (double[])other._b3.Clone();
|
||||
_m1 = new double[_w1.Length]; _v1 = new double[_w1.Length]; _mb1 = new double[_hidden]; _vb1 = new double[_hidden];
|
||||
_m2 = new double[_w2.Length]; _v2 = new double[_w2.Length]; _mb2 = new double[_hidden]; _vb2 = new double[_hidden];
|
||||
_m3 = new double[_w3.Length]; _v3 = new double[_w3.Length]; _mb3 = new double[_outputs]; _vb3 = new double[_outputs];
|
||||
}
|
||||
|
||||
public int Inputs => _inputs;
|
||||
|
||||
public int Outputs => _outputs;
|
||||
|
||||
/// <summary>A copy with the same weights and fresh optimiser state: the target network.</summary>
|
||||
public Mlp Clone() => new(this);
|
||||
|
||||
/// <summary>Copies the weights of <paramref name="source"/> into this network.</summary>
|
||||
public void CopyWeightsFrom(Mlp source)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(source);
|
||||
Array.Copy(source._w1, _w1, _w1.Length);
|
||||
Array.Copy(source._b1, _b1, _b1.Length);
|
||||
Array.Copy(source._w2, _w2, _w2.Length);
|
||||
Array.Copy(source._b2, _b2, _b2.Length);
|
||||
Array.Copy(source._w3, _w3, _w3.Length);
|
||||
Array.Copy(source._b3, _b3, _b3.Length);
|
||||
}
|
||||
|
||||
public double[] Predict(double[] x)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(x);
|
||||
double[] h1 = new double[_hidden];
|
||||
double[] h2 = new double[_hidden];
|
||||
double[] o = new double[_outputs];
|
||||
Forward(x, h1, h2, o);
|
||||
return o;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// One Adam step on the squared error between the network's output for the chosen
|
||||
/// action and the target value — other outputs get no gradient, which is how a Q
|
||||
/// network learns one action at a time.
|
||||
/// </summary>
|
||||
public double TrainBatch(double[][] x, int[] actions, double[] targets, double learningRate)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(x);
|
||||
ArgumentNullException.ThrowIfNull(actions);
|
||||
ArgumentNullException.ThrowIfNull(targets);
|
||||
|
||||
int batch = x.Length;
|
||||
double[] g1 = new double[_w1.Length], gb1 = new double[_hidden];
|
||||
double[] g2 = new double[_w2.Length], gb2 = new double[_hidden];
|
||||
double[] g3 = new double[_w3.Length], gb3 = new double[_outputs];
|
||||
|
||||
double[] h1 = new double[_hidden];
|
||||
double[] h2 = new double[_hidden];
|
||||
double[] o = new double[_outputs];
|
||||
double[] d2 = new double[_hidden];
|
||||
double[] d1 = new double[_hidden];
|
||||
double loss = 0;
|
||||
|
||||
for (int n = 0; n < batch; n++)
|
||||
{
|
||||
Forward(x[n], h1, h2, o);
|
||||
|
||||
int a = actions[n];
|
||||
double error = o[a] - targets[n];
|
||||
|
||||
// Huber-style clipping keeps one absurd target from blowing the weights up.
|
||||
double dOut = Math.Clamp(error, -1, 1);
|
||||
loss += 0.5 * error * error;
|
||||
|
||||
// Layer 3 gradient: only output a.
|
||||
for (int j = 0; j < _hidden; j++)
|
||||
{
|
||||
g3[(a * _hidden) + j] += dOut * h2[j];
|
||||
d2[j] = h2[j] > 0 ? dOut * _w3[(a * _hidden) + j] : 0;
|
||||
}
|
||||
|
||||
gb3[a] += dOut;
|
||||
|
||||
// Layer 2.
|
||||
Array.Clear(d1);
|
||||
for (int j = 0; j < _hidden; j++)
|
||||
{
|
||||
if (d2[j] == 0)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
gb2[j] += d2[j];
|
||||
int row = j * _hidden;
|
||||
for (int k = 0; k < _hidden; k++)
|
||||
{
|
||||
g2[row + k] += d2[j] * h1[k];
|
||||
d1[k] += d2[j] * _w2[row + k];
|
||||
}
|
||||
}
|
||||
|
||||
// Layer 1.
|
||||
for (int k = 0; k < _hidden; k++)
|
||||
{
|
||||
if (h1[k] <= 0 || d1[k] == 0)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
gb1[k] += d1[k];
|
||||
int row = k * _inputs;
|
||||
double[] xi = x[n];
|
||||
for (int i = 0; i < _inputs; i++)
|
||||
{
|
||||
g1[row + i] += d1[k] * xi[i];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
double scale = 1.0 / Math.Max(1, batch);
|
||||
_step++;
|
||||
Adam(_w1, g1, _m1, _v1, scale, learningRate);
|
||||
Adam(_b1, gb1, _mb1, _vb1, scale, learningRate);
|
||||
Adam(_w2, g2, _m2, _v2, scale, learningRate);
|
||||
Adam(_b2, gb2, _mb2, _vb2, scale, learningRate);
|
||||
Adam(_w3, g3, _m3, _v3, scale, learningRate);
|
||||
Adam(_b3, gb3, _mb3, _vb3, scale, learningRate);
|
||||
|
||||
return loss * scale;
|
||||
}
|
||||
|
||||
private void Forward(double[] x, double[] h1, double[] h2, double[] o)
|
||||
{
|
||||
if (x.Length != _inputs)
|
||||
{
|
||||
throw new ArgumentException($"servono {_inputs} ingressi, non {x.Length}");
|
||||
}
|
||||
|
||||
for (int j = 0; j < _hidden; j++)
|
||||
{
|
||||
double s = _b1[j];
|
||||
int row = j * _inputs;
|
||||
for (int i = 0; i < _inputs; i++)
|
||||
{
|
||||
s += _w1[row + i] * x[i];
|
||||
}
|
||||
|
||||
h1[j] = s > 0 ? s : 0;
|
||||
}
|
||||
|
||||
for (int j = 0; j < _hidden; j++)
|
||||
{
|
||||
double s = _b2[j];
|
||||
int row = j * _hidden;
|
||||
for (int k = 0; k < _hidden; k++)
|
||||
{
|
||||
s += _w2[row + k] * h1[k];
|
||||
}
|
||||
|
||||
h2[j] = s > 0 ? s : 0;
|
||||
}
|
||||
|
||||
for (int a = 0; a < _outputs; a++)
|
||||
{
|
||||
double s = _b3[a];
|
||||
int row = a * _hidden;
|
||||
for (int j = 0; j < _hidden; j++)
|
||||
{
|
||||
s += _w3[row + j] * h2[j];
|
||||
}
|
||||
|
||||
o[a] = s;
|
||||
}
|
||||
}
|
||||
|
||||
private void Adam(double[] w, double[] g, double[] m, double[] v, double scale, double lr)
|
||||
{
|
||||
const double beta1 = 0.9, beta2 = 0.999, eps = 1e-8;
|
||||
double c1 = 1 - Math.Pow(beta1, _step);
|
||||
double c2 = 1 - Math.Pow(beta2, _step);
|
||||
|
||||
for (int i = 0; i < w.Length; i++)
|
||||
{
|
||||
double grad = g[i] * scale;
|
||||
m[i] = (beta1 * m[i]) + ((1 - beta1) * grad);
|
||||
v[i] = (beta2 * v[i]) + ((1 - beta2) * grad * grad);
|
||||
double mHat = m[i] / c1;
|
||||
double vHat = v[i] / c2;
|
||||
w[i] -= lr * mHat / (Math.Sqrt(vHat) + eps);
|
||||
}
|
||||
}
|
||||
|
||||
private static double[] Init(int size, int fanIn, Random rng)
|
||||
{
|
||||
// He initialisation for ReLU layers.
|
||||
double limit = Math.Sqrt(6.0 / fanIn);
|
||||
double[] w = new double[size];
|
||||
for (int i = 0; i < size; i++)
|
||||
{
|
||||
w[i] = ((rng.NextDouble() * 2) - 1) * limit;
|
||||
}
|
||||
|
||||
return w;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,226 @@
|
||||
using Encelado.Core.Ml;
|
||||
|
||||
namespace Encelado.Core.Rl;
|
||||
|
||||
/// <summary>
|
||||
/// A pair as a Markov decision process: at every bar the agent sees the market
|
||||
/// features and its own position, chooses flat, long spread or short spread, and is
|
||||
/// paid the spread's return over the next bar minus whatever it cost to change its mind.
|
||||
/// <para>
|
||||
/// The costs are the point. An agent trained without them learns to flip on every
|
||||
/// bar and reports a Sharpe that evaporates the moment a fee is charged. Here every
|
||||
/// change of position pays the same round cost the backtest charges the statarb rule,
|
||||
/// so the two are compared on equal terms.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class PairEnvironment
|
||||
{
|
||||
private readonly double[][] _features;
|
||||
private readonly double[] _spreadReturn;
|
||||
private readonly double _cost;
|
||||
private readonly int _maxHold;
|
||||
private int _t;
|
||||
private int _position;
|
||||
private int _held;
|
||||
|
||||
/// <summary>Actions: 0 flat, 1 long spread, 2 short spread.</summary>
|
||||
public const int Actions = 3;
|
||||
|
||||
/// <param name="features">Standardised market features per bar.</param>
|
||||
/// <param name="spreadReturn">Return of one unit of long spread over bar t → t+1, aligned to t.</param>
|
||||
/// <param name="roundCost">Fractional cost of opening or closing a position.</param>
|
||||
/// <param name="maxHoldBars">Positions older than this are closed by the environment (0 = never).</param>
|
||||
public PairEnvironment(double[][] features, double[] spreadReturn, double roundCost, int maxHoldBars = 0)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(features);
|
||||
ArgumentNullException.ThrowIfNull(spreadReturn);
|
||||
|
||||
if (features.Length != spreadReturn.Length || features.Length < 10)
|
||||
{
|
||||
throw new ArgumentException("feature e rendimenti devono avere la stessa lunghezza, almeno dieci barre");
|
||||
}
|
||||
|
||||
_features = features;
|
||||
_spreadReturn = spreadReturn;
|
||||
_cost = roundCost;
|
||||
_maxHold = maxHoldBars;
|
||||
}
|
||||
|
||||
public int Length => _features.Length;
|
||||
|
||||
public int StateSize => _features[0].Length + 3;
|
||||
|
||||
/// <summary>
|
||||
/// A fresh cursor over the same data. The market arrays are shared and never
|
||||
/// written; the position and the clock are per instance, so every agent — and
|
||||
/// every thread — must walk its own copy.
|
||||
/// </summary>
|
||||
public PairEnvironment Clone() => new(_features, _spreadReturn, _cost, _maxHold);
|
||||
|
||||
public int Position => _position;
|
||||
|
||||
public int Time => _t;
|
||||
|
||||
public bool Done => _t >= _features.Length - 1;
|
||||
|
||||
public double[] Reset()
|
||||
{
|
||||
_t = 0;
|
||||
_position = 0;
|
||||
_held = 0;
|
||||
return State();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Applies <paramref name="action"/> at bar t, earns the return over t → t+1, and
|
||||
/// moves to t+1. Returns the new state, the reward, and whether the episode is over.
|
||||
/// </summary>
|
||||
public (double[] State, double Reward, bool Done, bool Changed) Step(int action)
|
||||
{
|
||||
int wanted = action switch
|
||||
{
|
||||
1 => 1,
|
||||
2 => -1,
|
||||
_ => 0,
|
||||
};
|
||||
|
||||
if (_maxHold > 0 && _position != 0 && _held >= _maxHold)
|
||||
{
|
||||
wanted = 0;
|
||||
}
|
||||
|
||||
double reward = 0;
|
||||
bool changed = wanted != _position;
|
||||
|
||||
if (changed)
|
||||
{
|
||||
// A reversal closes one position and opens another: two costs.
|
||||
reward -= _cost * (_position != 0 && wanted != 0 ? 2 : 1);
|
||||
_position = wanted;
|
||||
_held = 0;
|
||||
}
|
||||
|
||||
reward += _position * _spreadReturn[_t];
|
||||
if (_position != 0)
|
||||
{
|
||||
_held++;
|
||||
}
|
||||
|
||||
_t++;
|
||||
return (State(), reward, Done, changed);
|
||||
}
|
||||
|
||||
private double[] State()
|
||||
{
|
||||
double[] f = _features[Math.Min(_t, _features.Length - 1)];
|
||||
double[] s = new double[f.Length + 3];
|
||||
Array.Copy(f, s, f.Length);
|
||||
s[f.Length] = _position;
|
||||
s[f.Length + 1] = _position != 0 ? Math.Min(1, _held / 100.0) : 0;
|
||||
s[f.Length + 2] = 1;
|
||||
return s;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds the inputs from a pair series: a compact subset of the feature registry,
|
||||
/// standardised on the <i>training</i> span, and the one-bar spread return.
|
||||
/// </summary>
|
||||
public static (double[][] Features, double[] SpreadReturn, string[] Names) Inputs(
|
||||
PairSeries series, FeatureRegistry registry, IReadOnlyList<string> selected,
|
||||
int standardiseUpTo, double[]? means = null, double[]? stds = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(series);
|
||||
ArgumentNullException.ThrowIfNull(registry);
|
||||
ArgumentNullException.ThrowIfNull(selected);
|
||||
|
||||
double[][] all = registry.Compute(series);
|
||||
int[] columns = new int[selected.Count];
|
||||
IReadOnlyList<string> names = registry.Names;
|
||||
for (int i = 0; i < selected.Count; i++)
|
||||
{
|
||||
columns[i] = -1;
|
||||
for (int f = 0; f < names.Count; f++)
|
||||
{
|
||||
if (names[f] == selected[i])
|
||||
{
|
||||
columns[i] = f;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (columns[i] < 0)
|
||||
{
|
||||
throw new ArgumentException($"feature '{selected[i]}' non registrata");
|
||||
}
|
||||
}
|
||||
|
||||
int n = series.Count;
|
||||
double[][] x = new double[n][];
|
||||
for (int t = 0; t < n; t++)
|
||||
{
|
||||
double[] row = new double[columns.Length];
|
||||
for (int i = 0; i < columns.Length; i++)
|
||||
{
|
||||
double v = all[t][columns[i]];
|
||||
row[i] = double.IsFinite(v) ? v : 0;
|
||||
}
|
||||
|
||||
x[t] = row;
|
||||
}
|
||||
|
||||
// Standardise on the training span only.
|
||||
int upTo = Math.Clamp(standardiseUpTo, 1, n);
|
||||
means ??= new double[columns.Length];
|
||||
stds ??= new double[columns.Length];
|
||||
if (stds.All(static s => s == 0))
|
||||
{
|
||||
for (int i = 0; i < columns.Length; i++)
|
||||
{
|
||||
double sum = 0, sumSq = 0;
|
||||
for (int t = 0; t < upTo; t++)
|
||||
{
|
||||
sum += x[t][i];
|
||||
sumSq += x[t][i] * x[t][i];
|
||||
}
|
||||
|
||||
double mean = sum / upTo;
|
||||
double var = Math.Max(0, (sumSq / upTo) - (mean * mean));
|
||||
means[i] = mean;
|
||||
stds[i] = Math.Sqrt(var) > 1e-12 ? Math.Sqrt(var) : 1;
|
||||
}
|
||||
}
|
||||
|
||||
for (int t = 0; t < n; t++)
|
||||
{
|
||||
for (int i = 0; i < columns.Length; i++)
|
||||
{
|
||||
x[t][i] = Math.Clamp((x[t][i] - means[i]) / stds[i], -5, 5);
|
||||
}
|
||||
}
|
||||
|
||||
// Return of one unit of long spread, bar t → t+1: a delta-neutral position with
|
||||
// the β in force, split so the two legs' notionals sum to one.
|
||||
double[] ret = new double[n];
|
||||
for (int t = 0; t < n - 1; t++)
|
||||
{
|
||||
double beta = series.Beta[t];
|
||||
if (!double.IsFinite(beta) || series.CloseA[t] <= 0 || series.CloseB[t] <= 0)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
double ra = (series.CloseA[t + 1] / series.CloseA[t]) - 1;
|
||||
double rb = (series.CloseB[t + 1] / series.CloseB[t]) - 1;
|
||||
ret[t] = (ra - (beta * rb)) / (1 + Math.Abs(beta));
|
||||
}
|
||||
|
||||
return (x, ret, [.. selected]);
|
||||
}
|
||||
|
||||
/// <summary>The features the agent sees by default: the spread's state and the market's mood.</summary>
|
||||
public static readonly string[] DefaultFeatures =
|
||||
[
|
||||
"z", "z_change_4", "spread_change_4", "vol_ratio_a", "vol_a_16", "vol_b_16",
|
||||
"flow_a_16", "flow_b_16", "hour_sin", "hour_cos", "hours_to_funding",
|
||||
];
|
||||
}
|
||||
@@ -0,0 +1,276 @@
|
||||
using System.Globalization;
|
||||
|
||||
namespace Encelado.Core.Statistics;
|
||||
|
||||
/// <summary>What the Johansen procedure found in a set of series.</summary>
|
||||
public sealed record JohansenResult(
|
||||
double[] Eigenvalues,
|
||||
double[] TraceStatistics,
|
||||
double[] MaxEigenStatistics,
|
||||
double[] TraceCritical5,
|
||||
double[] MaxEigenCritical5,
|
||||
int Rank,
|
||||
double[,] Vectors,
|
||||
int Observations,
|
||||
int Lags)
|
||||
{
|
||||
public static JohansenResult Empty(int k) => new(
|
||||
new double[k], new double[k], new double[k], new double[k], new double[k], 0, new double[k, k], 0, 0);
|
||||
|
||||
public bool IsValid => Observations > 0;
|
||||
|
||||
/// <summary>
|
||||
/// The first cointegrating vector, normalised so the first series has weight 1. For
|
||||
/// two series this is <c>[1, −β]</c>: the hedge ratio in the same sense as the
|
||||
/// Engle-Granger regression.
|
||||
/// </summary>
|
||||
public double[] LeadingVector
|
||||
{
|
||||
get
|
||||
{
|
||||
int k = Vectors.GetLength(0);
|
||||
double[] v = new double[k];
|
||||
double head = Vectors[0, 0];
|
||||
|
||||
for (int i = 0; i < k; i++)
|
||||
{
|
||||
v[i] = Math.Abs(head) > 1e-12 ? Vectors[i, 0] / head : Vectors[i, 0];
|
||||
}
|
||||
|
||||
return v;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>The hedge ratio implied by the leading vector, for two series.</summary>
|
||||
public double HedgeRatio =>
|
||||
Vectors.GetLength(0) >= 2 ? -LeadingVector[1] : double.NaN;
|
||||
|
||||
public string Describe()
|
||||
{
|
||||
if (!IsValid)
|
||||
{
|
||||
return "Johansen non calcolabile";
|
||||
}
|
||||
|
||||
List<string> parts = [];
|
||||
for (int r = 0; r < TraceStatistics.Length; r++)
|
||||
{
|
||||
parts.Add(string.Create(CultureInfo.InvariantCulture,
|
||||
$"r≤{r}: trace {TraceStatistics[r]:F2} vs {TraceCritical5[r]:F2}, " +
|
||||
$"λmax {MaxEigenStatistics[r]:F2} vs {MaxEigenCritical5[r]:F2}"));
|
||||
}
|
||||
|
||||
return string.Create(CultureInfo.InvariantCulture,
|
||||
$"rango {Rank} su {Observations} oss. ({Lags} ritardi): {string.Join("; ", parts)}");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The Johansen cointegration test: how many long-run relationships hold among a set of
|
||||
/// series, and what they are.
|
||||
/// <para>
|
||||
/// Engle-Granger asks one question — is <i>this</i> residual stationary — and depends on
|
||||
/// which series was put on the left. Johansen estimates the whole vector error-correction
|
||||
/// model <c>Δyₜ = Π·yₜ₋₁ + ΣΓᵢ·Δyₜ₋ᵢ + ε</c> at once: the rank of <c>Π</c> is the number
|
||||
/// of cointegrating relationships and its eigenvectors are the relationships themselves.
|
||||
/// For a pair the two tests should agree; for a basket of three or four legs only this
|
||||
/// one can say how many spreads are actually there to trade.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The eigenproblem <c>|λ·S₁₁ − S₁₀·S₀₀⁻¹·S₀₁| = 0</c> is solved by whitening with the
|
||||
/// Cholesky factor of <c>S₁₁</c>, which makes it symmetric and lets a Jacobi sweep find
|
||||
/// real eigenvalues in <c>[0, 1)</c>. The critical values are the 5% points for the case
|
||||
/// with an unrestricted constant (MacKinnon, Haug and Michelis), which is the one every
|
||||
/// reference implementation defaults to.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class Johansen
|
||||
{
|
||||
// 5% critical values, indexed by K − r (the number of series still untested).
|
||||
private static readonly double[] TraceCritical = [double.NaN, 3.8415, 15.4943, 29.7961, 47.8545];
|
||||
private static readonly double[] MaxEigenCritical = [double.NaN, 3.8415, 14.2639, 21.1316, 27.5858];
|
||||
|
||||
/// <summary>
|
||||
/// Runs the test on <paramref name="series"/> — one array per variable, all the same
|
||||
/// length, oldest first — with <paramref name="lags"/> lagged differences in the VECM.
|
||||
/// </summary>
|
||||
public static JohansenResult Test(IReadOnlyList<double[]> series, int lags = 1)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(series);
|
||||
|
||||
int k = series.Count;
|
||||
if (k is < 2 or > 4)
|
||||
{
|
||||
throw new ArgumentException("Johansen supporta da 2 a 4 serie.", nameof(series));
|
||||
}
|
||||
|
||||
int t = series[0].Length;
|
||||
foreach (double[] s in series)
|
||||
{
|
||||
if (s.Length != t)
|
||||
{
|
||||
throw new ArgumentException("Le serie devono avere la stessa lunghezza.", nameof(series));
|
||||
}
|
||||
}
|
||||
|
||||
lags = Math.Max(0, lags);
|
||||
int start = lags + 1;
|
||||
int n = t - start;
|
||||
|
||||
if (n < 20 + (k * (lags + 1)))
|
||||
{
|
||||
return JohansenResult.Empty(k);
|
||||
}
|
||||
|
||||
// Regressors shared by both auxiliary regressions: a constant and the lagged
|
||||
// differences of every series.
|
||||
int columns = 1 + (k * lags);
|
||||
double[][] z = new double[n][];
|
||||
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
int time = start + i;
|
||||
double[] row = new double[columns];
|
||||
row[0] = 1;
|
||||
int c = 1;
|
||||
for (int lag = 1; lag <= lags; lag++)
|
||||
{
|
||||
for (int v = 0; v < k; v++)
|
||||
{
|
||||
row[c++] = series[v][time - lag] - series[v][time - lag - 1];
|
||||
}
|
||||
}
|
||||
|
||||
z[i] = row;
|
||||
}
|
||||
|
||||
// R0: residuals of Δyₜ on Z. R1: residuals of yₜ₋₁ on Z.
|
||||
double[][] r0 = new double[k][];
|
||||
double[][] r1 = new double[k][];
|
||||
|
||||
for (int v = 0; v < k; v++)
|
||||
{
|
||||
double[] dy = new double[n];
|
||||
double[] lagged = new double[n];
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
int time = start + i;
|
||||
dy[i] = series[v][time] - series[v][time - 1];
|
||||
lagged[i] = series[v][time - 1];
|
||||
}
|
||||
|
||||
r0[v] = Residuals(z, dy);
|
||||
r1[v] = Residuals(z, lagged);
|
||||
}
|
||||
|
||||
double[,] s00 = Product(r0, r0, n);
|
||||
double[,] s11 = Product(r1, r1, n);
|
||||
double[,] s01 = Product(r0, r1, n);
|
||||
double[,] s10 = Linear.Transpose(s01);
|
||||
|
||||
double[,]? l = Linear.Cholesky(s11);
|
||||
double[,]? s00Inverse = Linear.InvertSpd(s00);
|
||||
if (l is null || s00Inverse is null)
|
||||
{
|
||||
return JohansenResult.Empty(k);
|
||||
}
|
||||
|
||||
double[,] linv = Linear.InvertLowerTriangular(l);
|
||||
|
||||
// M = L⁻¹ · S₁₀ · S₀₀⁻¹ · S₀₁ · L⁻ᵀ, symmetric by construction.
|
||||
double[,] m = Linear.Multiply(
|
||||
Linear.Multiply(Linear.Multiply(Linear.Multiply(linv, s10), s00Inverse), s01),
|
||||
Linear.Transpose(linv));
|
||||
|
||||
// Symmetrise against rounding before the Jacobi sweep.
|
||||
for (int i = 0; i < k; i++)
|
||||
{
|
||||
for (int j = i + 1; j < k; j++)
|
||||
{
|
||||
double avg = (m[i, j] + m[j, i]) / 2;
|
||||
m[i, j] = avg;
|
||||
m[j, i] = avg;
|
||||
}
|
||||
}
|
||||
|
||||
(double[] eigenvalues, double[,] whitened) = Linear.JacobiEigen(m);
|
||||
|
||||
// β = L⁻ᵀ · v, one column per eigenvalue.
|
||||
double[,] vectors = Linear.Multiply(Linear.Transpose(linv), whitened);
|
||||
|
||||
double[] trace = new double[k];
|
||||
double[] maxEigen = new double[k];
|
||||
double[] traceCritical = new double[k];
|
||||
double[] maxCritical = new double[k];
|
||||
|
||||
for (int r = 0; r < k; r++)
|
||||
{
|
||||
double sum = 0;
|
||||
for (int i = r; i < k; i++)
|
||||
{
|
||||
sum += Math.Log(Math.Max(1e-12, 1 - Math.Clamp(eigenvalues[i], 0, 1 - 1e-12)));
|
||||
}
|
||||
|
||||
trace[r] = -n * sum;
|
||||
maxEigen[r] = -n * Math.Log(Math.Max(1e-12, 1 - Math.Clamp(eigenvalues[r], 0, 1 - 1e-12)));
|
||||
traceCritical[r] = TraceCritical[k - r];
|
||||
maxCritical[r] = MaxEigenCritical[k - r];
|
||||
}
|
||||
|
||||
// The rank is the first r whose trace statistic fails to reject "at most r".
|
||||
int rank = k;
|
||||
for (int r = 0; r < k; r++)
|
||||
{
|
||||
if (trace[r] < traceCritical[r])
|
||||
{
|
||||
rank = r;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
return new JohansenResult(eigenvalues, trace, maxEigen, traceCritical, maxCritical, rank, vectors, n, lags);
|
||||
}
|
||||
|
||||
private static double[] Residuals(double[][] design, double[] y)
|
||||
{
|
||||
OlsFit? fit = Ols.Fit(design, y);
|
||||
if (fit is not null)
|
||||
{
|
||||
return fit.Residuals;
|
||||
}
|
||||
|
||||
// A singular design (no variation in the lags) leaves the raw, demeaned series.
|
||||
double mean = Performance.Mean(y);
|
||||
double[] residuals = new double[y.Length];
|
||||
for (int i = 0; i < y.Length; i++)
|
||||
{
|
||||
residuals[i] = y[i] - mean;
|
||||
}
|
||||
|
||||
return residuals;
|
||||
}
|
||||
|
||||
/// <summary><c>AᵀB / n</c> for two sets of residual columns.</summary>
|
||||
private static double[,] Product(double[][] a, double[][] b, int n)
|
||||
{
|
||||
int k = a.Length;
|
||||
double[,] s = new double[k, k];
|
||||
for (int i = 0; i < k; i++)
|
||||
{
|
||||
for (int j = 0; j < k; j++)
|
||||
{
|
||||
double sum = 0;
|
||||
double[] ai = a[i];
|
||||
double[] bj = b[j];
|
||||
for (int t = 0; t < n; t++)
|
||||
{
|
||||
sum += ai[t] * bj[t];
|
||||
}
|
||||
|
||||
s[i, j] = sum / n;
|
||||
}
|
||||
}
|
||||
|
||||
return s;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
namespace Encelado.Core.Statistics;
|
||||
|
||||
/// <summary>
|
||||
/// A hedge ratio that follows the market instead of being refitted once a day.
|
||||
/// <para>
|
||||
/// The state is <c>[β, α]</c> with <c>y = β·x + α</c>, and both are allowed to drift as
|
||||
/// a random walk. Each observation nudges them by a Kalman gain that is large when the
|
||||
/// filter is uncertain and small once it has settled — so the ratio tracks a genuine
|
||||
/// change in the relationship within a handful of bars, without being dragged around
|
||||
/// by every tick of noise the way a short OLS window would be.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// One number governs it: <c>delta</c>, the variance the state is assumed to gain per
|
||||
/// step, expressed through <c>Vw = delta/(1−delta)</c>. Around 1e-4 the ratio moves
|
||||
/// meaningfully over a few hundred bars; at 1e-5 it is nearly static. The innovation
|
||||
/// <c>e = y − ŷ</c> is itself the spread, already centred, which is why this pairs
|
||||
/// naturally with a z-score on the innovation series.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class KalmanHedge
|
||||
{
|
||||
private readonly double _vw;
|
||||
private readonly double _ve;
|
||||
|
||||
// State and its covariance.
|
||||
private double _beta;
|
||||
private double _alpha;
|
||||
private double _p00 = 1;
|
||||
private double _p01;
|
||||
private double _p11 = 1;
|
||||
|
||||
public KalmanHedge(double delta = 1e-4, double observationNoise = 1e-3)
|
||||
{
|
||||
if (delta is <= 0 or >= 1)
|
||||
{
|
||||
throw new ArgumentOutOfRangeException(nameof(delta), delta, "delta deve stare in (0, 1).");
|
||||
}
|
||||
|
||||
if (observationNoise <= 0)
|
||||
{
|
||||
throw new ArgumentOutOfRangeException(nameof(observationNoise), observationNoise, "deve essere positivo.");
|
||||
}
|
||||
|
||||
_vw = delta / (1 - delta);
|
||||
_ve = observationNoise;
|
||||
}
|
||||
|
||||
public double Beta => _beta;
|
||||
|
||||
public double Alpha => _alpha;
|
||||
|
||||
/// <summary>The last innovation: the observation minus what the filter predicted.</summary>
|
||||
public double LastError { get; private set; }
|
||||
|
||||
/// <summary>Predicted variance of the last innovation. Its square root is the spread's scale.</summary>
|
||||
public double LastErrorVariance { get; private set; } = double.NaN;
|
||||
|
||||
public int Updates { get; private set; }
|
||||
|
||||
/// <summary>Enough updates that the gain has settled and β means something.</summary>
|
||||
public bool IsReady => Updates >= 20;
|
||||
|
||||
/// <summary>
|
||||
/// Folds one observation in and returns the innovation. Steps, in the order the
|
||||
/// strategy note gives them: predict the covariance, predict the observation, size
|
||||
/// the gain, correct the state, shrink the covariance.
|
||||
/// </summary>
|
||||
public double Update(double x, double y)
|
||||
{
|
||||
if (!double.IsFinite(x) || !double.IsFinite(y))
|
||||
{
|
||||
return double.NaN;
|
||||
}
|
||||
|
||||
// R = P + Vw·I: the state may have drifted since the last step.
|
||||
double r00 = _p00 + _vw;
|
||||
double r01 = _p01;
|
||||
double r11 = _p11 + _vw;
|
||||
|
||||
// ŷ = h·state with h = [x, 1].
|
||||
double predicted = (_beta * x) + _alpha;
|
||||
|
||||
// Q = h·R·hᵀ + Ve.
|
||||
double hr0 = (r00 * x) + r01;
|
||||
double hr1 = (r01 * x) + r11;
|
||||
double q = (hr0 * x) + hr1 + _ve;
|
||||
|
||||
double e = y - predicted;
|
||||
|
||||
// K = R·hᵀ / Q.
|
||||
double k0 = hr0 / q;
|
||||
double k1 = hr1 / q;
|
||||
|
||||
_beta += k0 * e;
|
||||
_alpha += k1 * e;
|
||||
|
||||
// P = R − K·h·R.
|
||||
_p00 = r00 - (k0 * hr0);
|
||||
_p01 = r01 - (k0 * hr1);
|
||||
_p11 = r11 - (k1 * hr1);
|
||||
|
||||
LastError = e;
|
||||
LastErrorVariance = q;
|
||||
Updates++;
|
||||
|
||||
return e;
|
||||
}
|
||||
|
||||
public void Reset()
|
||||
{
|
||||
_beta = 0;
|
||||
_alpha = 0;
|
||||
_p00 = 1;
|
||||
_p01 = 0;
|
||||
_p11 = 1;
|
||||
LastError = 0;
|
||||
LastErrorVariance = double.NaN;
|
||||
Updates = 0;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,238 @@
|
||||
namespace Encelado.Core.Statistics;
|
||||
|
||||
/// <summary>
|
||||
/// The handful of dense linear-algebra routines the statistical tests need, for matrices
|
||||
/// of two to a few dozen rows. Nothing here is tuned for size: every matrix in this
|
||||
/// codebase is a covariance of a few series or a design of a few regressors.
|
||||
/// </summary>
|
||||
public static class Linear
|
||||
{
|
||||
public static double[,] Multiply(double[,] a, double[,] b)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(a);
|
||||
ArgumentNullException.ThrowIfNull(b);
|
||||
|
||||
int n = a.GetLength(0);
|
||||
int m = a.GetLength(1);
|
||||
int p = b.GetLength(1);
|
||||
|
||||
if (b.GetLength(0) != m)
|
||||
{
|
||||
throw new ArgumentException("Dimensioni incompatibili per il prodotto.");
|
||||
}
|
||||
|
||||
double[,] c = new double[n, p];
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
for (int k = 0; k < m; k++)
|
||||
{
|
||||
double aik = a[i, k];
|
||||
if (aik == 0)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
for (int j = 0; j < p; j++)
|
||||
{
|
||||
c[i, j] += aik * b[k, j];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return c;
|
||||
}
|
||||
|
||||
public static double[,] Transpose(double[,] a)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(a);
|
||||
int n = a.GetLength(0);
|
||||
int m = a.GetLength(1);
|
||||
double[,] t = new double[m, n];
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
for (int j = 0; j < m; j++)
|
||||
{
|
||||
t[j, i] = a[i, j];
|
||||
}
|
||||
}
|
||||
|
||||
return t;
|
||||
}
|
||||
|
||||
/// <summary>Lower-triangular Cholesky factor of a symmetric positive-definite matrix, or null.</summary>
|
||||
public static double[,]? Cholesky(double[,] a)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(a);
|
||||
int n = a.GetLength(0);
|
||||
double[,] l = new double[n, n];
|
||||
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
for (int j = 0; j <= i; j++)
|
||||
{
|
||||
double sum = a[i, j];
|
||||
for (int p = 0; p < j; p++)
|
||||
{
|
||||
sum -= l[i, p] * l[j, p];
|
||||
}
|
||||
|
||||
if (i == j)
|
||||
{
|
||||
if (sum <= 1e-300 || !double.IsFinite(sum))
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
l[i, j] = Math.Sqrt(sum);
|
||||
}
|
||||
else
|
||||
{
|
||||
l[i, j] = sum / l[j, j];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return l;
|
||||
}
|
||||
|
||||
/// <summary>Inverse of a lower-triangular matrix by forward substitution, column by column.</summary>
|
||||
public static double[,] InvertLowerTriangular(double[,] l)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(l);
|
||||
int n = l.GetLength(0);
|
||||
double[,] inv = new double[n, n];
|
||||
|
||||
for (int col = 0; col < n; col++)
|
||||
{
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
double sum = i == col ? 1 : 0;
|
||||
for (int j = 0; j < i; j++)
|
||||
{
|
||||
sum -= l[i, j] * inv[j, col];
|
||||
}
|
||||
|
||||
inv[i, col] = sum / l[i, i];
|
||||
}
|
||||
}
|
||||
|
||||
return inv;
|
||||
}
|
||||
|
||||
/// <summary>Inverse of a symmetric positive-definite matrix via its Cholesky factor, or null.</summary>
|
||||
public static double[,]? InvertSpd(double[,] a)
|
||||
{
|
||||
double[,]? l = Cholesky(a);
|
||||
if (l is null)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
double[,] linv = InvertLowerTriangular(l);
|
||||
return Multiply(Transpose(linv), linv);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Eigen-decomposition of a symmetric matrix by cyclic Jacobi rotations. Returns the
|
||||
/// eigenvalues in descending order with their eigenvectors as the columns of the
|
||||
/// second array. Slow in the abstract and instantaneous at this size, and — unlike a
|
||||
/// general QR — it cannot produce a complex pair on a matrix that is symmetric by
|
||||
/// construction.
|
||||
/// </summary>
|
||||
public static (double[] Values, double[,] Vectors) JacobiEigen(double[,] symmetric)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(symmetric);
|
||||
int n = symmetric.GetLength(0);
|
||||
|
||||
double[,] a = (double[,])symmetric.Clone();
|
||||
double[,] v = new double[n, n];
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
v[i, i] = 1;
|
||||
}
|
||||
|
||||
for (int sweep = 0; sweep < 100; sweep++)
|
||||
{
|
||||
double off = 0;
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
for (int j = i + 1; j < n; j++)
|
||||
{
|
||||
off += a[i, j] * a[i, j];
|
||||
}
|
||||
}
|
||||
|
||||
if (off < 1e-22)
|
||||
{
|
||||
break;
|
||||
}
|
||||
|
||||
for (int p = 0; p < n; p++)
|
||||
{
|
||||
for (int q = p + 1; q < n; q++)
|
||||
{
|
||||
if (Math.Abs(a[p, q]) < 1e-300)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
double theta = (a[q, q] - a[p, p]) / (2 * a[p, q]);
|
||||
double t = Math.Sign(theta) / (Math.Abs(theta) + Math.Sqrt((theta * theta) + 1));
|
||||
if (theta == 0)
|
||||
{
|
||||
t = 1;
|
||||
}
|
||||
|
||||
double c = 1 / Math.Sqrt((t * t) + 1);
|
||||
double s = t * c;
|
||||
|
||||
for (int k = 0; k < n; k++)
|
||||
{
|
||||
double akp = a[k, p];
|
||||
double akq = a[k, q];
|
||||
a[k, p] = (c * akp) - (s * akq);
|
||||
a[k, q] = (s * akp) + (c * akq);
|
||||
}
|
||||
|
||||
for (int k = 0; k < n; k++)
|
||||
{
|
||||
double apk = a[p, k];
|
||||
double aqk = a[q, k];
|
||||
a[p, k] = (c * apk) - (s * aqk);
|
||||
a[q, k] = (s * apk) + (c * aqk);
|
||||
}
|
||||
|
||||
for (int k = 0; k < n; k++)
|
||||
{
|
||||
double vkp = v[k, p];
|
||||
double vkq = v[k, q];
|
||||
v[k, p] = (c * vkp) - (s * vkq);
|
||||
v[k, q] = (s * vkp) + (c * vkq);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
double[] values = new double[n];
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
values[i] = a[i, i];
|
||||
}
|
||||
|
||||
// Descending, carrying the vectors along.
|
||||
int[] order = [.. Enumerable.Range(0, n).OrderByDescending(i => values[i])];
|
||||
double[] sortedValues = new double[n];
|
||||
double[,] sortedVectors = new double[n, n];
|
||||
|
||||
for (int c = 0; c < n; c++)
|
||||
{
|
||||
sortedValues[c] = values[order[c]];
|
||||
for (int r = 0; r < n; r++)
|
||||
{
|
||||
sortedVectors[r, c] = v[r, order[c]];
|
||||
}
|
||||
}
|
||||
|
||||
return (sortedValues, sortedVectors);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,319 @@
|
||||
namespace Encelado.Core.Statistics;
|
||||
|
||||
/// <summary>
|
||||
/// Performance measurement and position sizing — every formula the strategy note
|
||||
/// spells out, in one place, with tests.
|
||||
/// <para>
|
||||
/// The two that matter most are the ones nobody computes by hand: the probabilistic
|
||||
/// and the <b>deflated</b> Sharpe ratio (Bailey & López de Prado). A Sharpe of 1.5
|
||||
/// from a backtest means something entirely different when it was the best of one
|
||||
/// configuration and when it was the best of three hundred, and the deflated version
|
||||
/// is the number that knows the difference. It is what stops a parameter sweep from
|
||||
/// being mistaken for a strategy.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class Performance
|
||||
{
|
||||
/// <summary>Euler-Mascheroni constant, used by the expected-maximum Sharpe bound.</summary>
|
||||
public const double EulerGamma = 0.57721566490153286;
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Moments
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
public static double Mean(IReadOnlyList<double> values)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(values);
|
||||
if (values.Count == 0)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
double sum = 0;
|
||||
foreach (double v in values)
|
||||
{
|
||||
sum += v;
|
||||
}
|
||||
|
||||
return sum / values.Count;
|
||||
}
|
||||
|
||||
/// <summary>Sample standard deviation (n − 1).</summary>
|
||||
public static double StandardDeviation(IReadOnlyList<double> values)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(values);
|
||||
if (values.Count < 2)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
double mean = Mean(values);
|
||||
double sum = 0;
|
||||
foreach (double v in values)
|
||||
{
|
||||
double d = v - mean;
|
||||
sum += d * d;
|
||||
}
|
||||
|
||||
return Math.Sqrt(sum / (values.Count - 1));
|
||||
}
|
||||
|
||||
/// <summary>Population skewness, <c>m₃ / m₂^1.5</c>.</summary>
|
||||
public static double Skewness(IReadOnlyList<double> values)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(values);
|
||||
if (values.Count < 3)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
(double m2, double m3, _) = CentralMoments(values);
|
||||
return m2 > 0 ? m3 / Math.Pow(m2, 1.5) : 0;
|
||||
}
|
||||
|
||||
/// <summary>Population kurtosis, <c>m₄ / m₂²</c>, <b>not</b> excess: a normal gives 3.</summary>
|
||||
public static double Kurtosis(IReadOnlyList<double> values)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(values);
|
||||
if (values.Count < 4)
|
||||
{
|
||||
return 3;
|
||||
}
|
||||
|
||||
(double m2, _, double m4) = CentralMoments(values);
|
||||
return m2 > 0 ? m4 / (m2 * m2) : 3;
|
||||
}
|
||||
|
||||
private static (double M2, double M3, double M4) CentralMoments(IReadOnlyList<double> values)
|
||||
{
|
||||
double mean = Mean(values);
|
||||
double m2 = 0, m3 = 0, m4 = 0;
|
||||
|
||||
foreach (double v in values)
|
||||
{
|
||||
double d = v - mean;
|
||||
double d2 = d * d;
|
||||
m2 += d2;
|
||||
m3 += d2 * d;
|
||||
m4 += d2 * d2;
|
||||
}
|
||||
|
||||
int n = values.Count;
|
||||
return (m2 / n, m3 / n, m4 / n);
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Returns and ratios
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/// <summary>Log return between two prices; zero when either is unusable.</summary>
|
||||
public static double LogReturn(double previous, double current) =>
|
||||
previous > 0 && current > 0 ? Math.Log(current / previous) : 0;
|
||||
|
||||
/// <summary>Realised volatility of a window of returns: <c>sqrt(Σ r²)</c>, un-annualised.</summary>
|
||||
public static double RealizedVolatility(IReadOnlyList<double> returns)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(returns);
|
||||
double sum = 0;
|
||||
foreach (double r in returns)
|
||||
{
|
||||
sum += r * r;
|
||||
}
|
||||
|
||||
return Math.Sqrt(sum);
|
||||
}
|
||||
|
||||
/// <summary>Per-period Sharpe ratio: mean excess return over its standard deviation.</summary>
|
||||
public static double Sharpe(IReadOnlyList<double> returns, double riskFreePerPeriod = 0)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(returns);
|
||||
double sd = StandardDeviation(returns);
|
||||
return sd > 0 ? (Mean(returns) - riskFreePerPeriod) / sd : 0;
|
||||
}
|
||||
|
||||
/// <summary>Annualises a per-period Sharpe by <c>sqrt(periods per year)</c>.</summary>
|
||||
public static double Annualise(double sharpePerPeriod, double periodsPerYear) =>
|
||||
sharpePerPeriod * Math.Sqrt(Math.Max(0, periodsPerYear));
|
||||
|
||||
/// <summary>Sortino: mean return over the deviation of the negative returns only.</summary>
|
||||
public static double Sortino(IReadOnlyList<double> returns, double target = 0)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(returns);
|
||||
if (returns.Count < 2)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
double sum = 0;
|
||||
foreach (double r in returns)
|
||||
{
|
||||
double shortfall = Math.Min(0, r - target);
|
||||
sum += shortfall * shortfall;
|
||||
}
|
||||
|
||||
double downside = Math.Sqrt(sum / returns.Count);
|
||||
return downside > 0 ? (Mean(returns) - target) / downside : 0;
|
||||
}
|
||||
|
||||
/// <summary>Worst peak-to-trough fall of an equity curve, as a fraction of the peak.</summary>
|
||||
public static double MaxDrawdown(IReadOnlyList<double> equity)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(equity);
|
||||
double peak = double.NegativeInfinity;
|
||||
double worst = 0;
|
||||
|
||||
foreach (double e in equity)
|
||||
{
|
||||
if (e > peak)
|
||||
{
|
||||
peak = e;
|
||||
}
|
||||
|
||||
if (peak > 0)
|
||||
{
|
||||
double dd = (peak - e) / peak;
|
||||
if (dd > worst)
|
||||
{
|
||||
worst = dd;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return worst;
|
||||
}
|
||||
|
||||
public static double Cagr(double startEquity, double endEquity, double years) =>
|
||||
startEquity > 0 && endEquity > 0 && years > 0
|
||||
? Math.Pow(endEquity / startEquity, 1 / years) - 1
|
||||
: 0;
|
||||
|
||||
public static double Calmar(double cagr, double maxDrawdown) =>
|
||||
maxDrawdown > 1e-12 ? cagr / maxDrawdown : 0;
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Probabilistic and deflated Sharpe
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/// <summary>
|
||||
/// The probability that the true Sharpe exceeds <paramref name="benchmark"/>, given an
|
||||
/// estimate of <paramref name="sharpe"/> from <paramref name="observations"/> returns
|
||||
/// with the given skewness and (non-excess) kurtosis.
|
||||
/// <para>
|
||||
/// <c>PSR = Φ[(SR − SR*)·√(T−1) / √(1 − γ₃·SR + (γ₄−1)/4·SR²)]</c>. All three inputs
|
||||
/// are at the sampling frequency, not annualised: the formula's <c>T</c> is a count
|
||||
/// of the returns the Sharpe was computed from.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static double ProbabilisticSharpe(
|
||||
double sharpe, double benchmark, int observations, double skewness, double kurtosis)
|
||||
{
|
||||
if (observations < 2 || !double.IsFinite(sharpe))
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
double variance = 1 - (skewness * sharpe) + (((kurtosis - 1) / 4) * sharpe * sharpe);
|
||||
if (variance <= 0)
|
||||
{
|
||||
return sharpe > benchmark ? 1 : 0;
|
||||
}
|
||||
|
||||
double z = (sharpe - benchmark) * Math.Sqrt(observations - 1) / Math.Sqrt(variance);
|
||||
return Normal.Cdf(z);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The Sharpe the <i>best</i> of <paramref name="trials"/> unskilled strategies would be
|
||||
/// expected to show, given the variance of the trials' Sharpe estimates:
|
||||
/// <c>SR* = √V · [(1−γ)·Φ⁻¹(1−1/N) + γ·Φ⁻¹(1−1/(N·e))]</c>.
|
||||
/// </summary>
|
||||
public static double ExpectedMaximumSharpe(int trials, double varianceOfTrialSharpes)
|
||||
{
|
||||
if (trials <= 1 || varianceOfTrialSharpes <= 0)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
double n = trials;
|
||||
double a = Normal.Quantile(1 - (1 / n));
|
||||
double b = Normal.Quantile(1 - (1 / (n * Math.E)));
|
||||
|
||||
return Math.Sqrt(varianceOfTrialSharpes) * (((1 - EulerGamma) * a) + (EulerGamma * b));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The deflated Sharpe ratio: the probabilistic Sharpe against the benchmark that pure
|
||||
/// selection among <paramref name="trials"/> configurations would have produced.
|
||||
/// <para>
|
||||
/// <paramref name="trials"/> must count <b>every</b> configuration that was tried, not
|
||||
/// the ones that were kept. Counting only the survivors is the whole mistake this
|
||||
/// number exists to catch. Above 0.95 is strong evidence the result is not selection.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static double DeflatedSharpe(
|
||||
double sharpe, int observations, double skewness, double kurtosis,
|
||||
int trials, double varianceOfTrialSharpes)
|
||||
{
|
||||
double benchmark = ExpectedMaximumSharpe(trials, varianceOfTrialSharpes);
|
||||
return ProbabilisticSharpe(sharpe, benchmark, observations, skewness, kurtosis);
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Sizing
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/// <summary>Kelly fraction from a win rate and the ratio of average win to average loss.</summary>
|
||||
public static double Kelly(double winRate, double winLossRatio)
|
||||
{
|
||||
if (winLossRatio <= 0 || winRate is < 0 or > 1)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
return Math.Max(0, winRate - ((1 - winRate) / winLossRatio));
|
||||
}
|
||||
|
||||
/// <summary>The same in betting terms: <c>f* = (b·p − q)/b</c>.</summary>
|
||||
public static double KellyFromOdds(double probability, double odds) =>
|
||||
Kelly(probability, odds);
|
||||
|
||||
/// <summary>Continuous Kelly: <c>f* = (μ − r)/σ²</c>.</summary>
|
||||
public static double KellyContinuous(double meanReturn, double variance, double riskFree = 0) =>
|
||||
variance > 0 ? (meanReturn - riskFree) / variance : 0;
|
||||
|
||||
/// <summary>
|
||||
/// Volatility targeting: the exposure that makes the realised volatility land on the
|
||||
/// target, capped by the maximum leverage. Zero realised volatility means zero size,
|
||||
/// not infinite.
|
||||
/// </summary>
|
||||
public static double VolatilityTargetSize(
|
||||
double targetVolatility, double realizedVolatility, double capital, double maxLeverage)
|
||||
{
|
||||
if (realizedVolatility <= 0 || targetVolatility <= 0 || capital <= 0)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
double size = targetVolatility / realizedVolatility * capital;
|
||||
return Math.Min(size, capital * Math.Max(0, maxLeverage));
|
||||
}
|
||||
|
||||
/// <summary>Expected value after costs: <c>p·win − (1−p)·loss − costs</c>. Below zero, do not enter.</summary>
|
||||
public static double NetExpectedValue(double probability, double averageWin, double averageLoss, double costs) =>
|
||||
(probability * averageWin) - ((1 - probability) * averageLoss) - costs;
|
||||
|
||||
/// <summary>
|
||||
/// Bet size from a predicted probability (López de Prado): <c>2·Φ(z) − 1</c> with
|
||||
/// <c>z = (p − ½)/√(p(1−p))</c>. A coin flip sizes to nothing; certainty to one.
|
||||
/// </summary>
|
||||
public static double BetSizeFromProbability(double probability)
|
||||
{
|
||||
if (probability is <= 0 or >= 1 || !double.IsFinite(probability))
|
||||
{
|
||||
return probability >= 1 ? 1 : 0;
|
||||
}
|
||||
|
||||
double z = (probability - 0.5) / Math.Sqrt(probability * (1 - probability));
|
||||
return Math.Clamp((2 * Normal.Cdf(z)) - 1, -1, 1);
|
||||
}
|
||||
}
|
||||
@@ -102,6 +102,9 @@ public sealed class StatArbStrategy
|
||||
/// </summary>
|
||||
public bool RequireCointegration { get; }
|
||||
|
||||
/// <summary>Length of the rolling z-score window, in bars.</summary>
|
||||
public int Window => _window;
|
||||
|
||||
/// <summary>Bars needed before the z-score is meaningful.</summary>
|
||||
public int WarmupBars => _window + 2;
|
||||
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<!--
|
||||
Il database locale del sistema ML: dataset, feature, etichette, modelli, risultati
|
||||
di validazione e journal. È l'unico progetto con una dipendenza NuGet a runtime —
|
||||
SQLite porta un binario nativo — quindi è tenuto separato da Core, che resta
|
||||
AOT-compatibile e senza dipendenze, e viene referenziato solo dal bot e dagli
|
||||
strumenti.
|
||||
-->
|
||||
<PropertyGroup>
|
||||
<RootNamespace>Encelado.Storage</RootNamespace>
|
||||
<AssemblyName>Encelado.Storage</AssemblyName>
|
||||
<IsAotCompatible>false</IsAotCompatible>
|
||||
<IsTrimmable>false</IsTrimmable>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Data.Sqlite.Core" Version="10.0.0" />
|
||||
<!-- Pinned above the version Microsoft.Data.Sqlite would pull on its own: the 2.1.x
|
||||
native bundle carries a known high-severity advisory (GHSA-2m69-gcr7-jv3q),
|
||||
and the build treats that as an error rather than a warning. -->
|
||||
<PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3" Version="3.0.5" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\Encelado.Core\Encelado.Core.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
@@ -0,0 +1,760 @@
|
||||
using System.Globalization;
|
||||
using Encelado.Core.Journal;
|
||||
using Encelado.Core.Market;
|
||||
using Microsoft.Data.Sqlite;
|
||||
|
||||
namespace Encelado.Storage;
|
||||
|
||||
/// <summary>
|
||||
/// The local database: canonical bars, the data-quality report, every journal table,
|
||||
/// the datasets the models train on, the models themselves, their validation results,
|
||||
/// drift measurements and the champion per pair.
|
||||
/// <para>
|
||||
/// One SQLite file, by default under <c>%ProgramData%\Encelado</c>: a machine-wide
|
||||
/// folder that is neither the user's documents (where the configuration lives, and
|
||||
/// which gets synced and copied) nor the per-user application data (where the
|
||||
/// credentials live, and which nothing else should share). The path is a setting.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Plain SQL, no ORM. Every write goes through one lock: a
|
||||
/// <see cref="SqliteConnection"/> is not thread-safe, and the journal is written from
|
||||
/// the market-data thread, the order thread and the reconcile loop at once. Every
|
||||
/// journal write also swallows its own failure and reports it through
|
||||
/// <see cref="OnError"/> — a database that cannot be written must cost a row, never a
|
||||
/// trading decision.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class StorageDb : IJournalSink, IDisposable
|
||||
{
|
||||
private readonly SqliteConnection _connection;
|
||||
private readonly Lock _gate = new();
|
||||
|
||||
public StorageDb(string path)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(path);
|
||||
|
||||
string full = System.IO.Path.GetFullPath(path);
|
||||
Directory.CreateDirectory(System.IO.Path.GetDirectoryName(full)!);
|
||||
|
||||
Path = full;
|
||||
|
||||
_connection = new SqliteConnection(new SqliteConnectionStringBuilder
|
||||
{
|
||||
DataSource = full,
|
||||
Mode = SqliteOpenMode.ReadWriteCreate,
|
||||
Cache = SqliteCacheMode.Private,
|
||||
}.ToString());
|
||||
|
||||
_connection.Open();
|
||||
|
||||
// Write-ahead logging lets the tools read while the bot writes, and NORMAL
|
||||
// synchronisation is the standard trade: a crash loses at most the last
|
||||
// transaction, never the database.
|
||||
Execute("PRAGMA journal_mode=WAL;");
|
||||
Execute("PRAGMA synchronous=NORMAL;");
|
||||
Execute("PRAGMA foreign_keys=ON;");
|
||||
|
||||
EnsureSchema();
|
||||
}
|
||||
|
||||
/// <summary>Absolute path of the database file.</summary>
|
||||
public string Path { get; }
|
||||
|
||||
/// <summary>Raised on a write that failed. Wired to the log by the host.</summary>
|
||||
public Action<string, Exception>? OnError { get; set; }
|
||||
|
||||
/// <summary>The default location: <c>%ProgramData%\Encelado\encelado.db</c>.</summary>
|
||||
public static string DefaultPath =>
|
||||
System.IO.Path.Combine(
|
||||
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
|
||||
"Encelado",
|
||||
"encelado.db");
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Schema
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
private void EnsureSchema()
|
||||
{
|
||||
const string ddl = """
|
||||
CREATE TABLE IF NOT EXISTS bars(
|
||||
symbol TEXT NOT NULL, timeframe TEXT NOT NULL, time_utc INTEGER NOT NULL,
|
||||
open REAL, high REAL, low REAL, close REAL, volume REAL, vwap REAL,
|
||||
trades INTEGER, taker_buy REAL,
|
||||
PRIMARY KEY(symbol, timeframe, time_utc)) WITHOUT ROWID;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS data_quality(
|
||||
id INTEGER PRIMARY KEY, run_utc INTEGER, file TEXT, symbol TEXT, rows INTEGER,
|
||||
first_utc INTEGER, last_utc INTEGER, duplicates INTEGER, gaps INTEGER,
|
||||
gap_minutes INTEGER, bad_prices INTEGER, motivazione TEXT);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS decisions(
|
||||
bar_time_utc INTEGER, decision_id INTEGER, pair TEXT, symbol_a TEXT, symbol_b TEXT,
|
||||
ready INTEGER, close_a REAL, close_b REAL, spread REAL, z_score REAL, beta REAL,
|
||||
alpha REAL, p_value REAL, adf REAL, half_life REAL, cointegrated INTEGER,
|
||||
qty_a REAL, qty_b REAL, entry_z REAL, bars_held INTEGER, signal TEXT,
|
||||
spread_pct_a REAL, spread_pct_b REAL, funding_a REAL, funding_b REAL,
|
||||
equity REAL, halted INTEGER, meta_probability REAL, motivazione TEXT,
|
||||
PRIMARY KEY(pair, bar_time_utc, decision_id)) WITHOUT ROWID;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS executions(
|
||||
id INTEGER PRIMARY KEY, timestamp_utc INTEGER, decision_id INTEGER, pair TEXT,
|
||||
phase TEXT, approved INTEGER, reason TEXT, side_a TEXT, notional_a REAL,
|
||||
quantity_a REAL, price_a REAL, side_b TEXT, notional_b REAL, quantity_b REAL,
|
||||
price_b REAL, net_funding_rate REAL, equity REAL, available_balance REAL,
|
||||
gross_exposure REAL, open_pairs INTEGER, order_id_a TEXT, order_id_b TEXT,
|
||||
error TEXT, latency_ms REAL, motivazione TEXT);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS trades(
|
||||
id INTEGER PRIMARY KEY, timestamp_utc INTEGER, event TEXT, symbol TEXT, side TEXT,
|
||||
quantity REAL, price REAL, order_id TEXT, stop REAL, target REAL, equity REAL,
|
||||
realized_pnl REAL, motivazione TEXT);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS datasets(
|
||||
id INTEGER PRIMARY KEY, created_utc INTEGER, pair TEXT, timeframe TEXT,
|
||||
from_utc INTEGER, to_utc INTEGER, rows INTEGER, feature_names TEXT,
|
||||
label_spec TEXT, motivazione TEXT);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS dataset_rows(
|
||||
dataset_id INTEGER NOT NULL, time_utc INTEGER NOT NULL, end_utc INTEGER,
|
||||
label INTEGER, side INTEGER, ret REAL, weight REAL, features BLOB,
|
||||
PRIMARY KEY(dataset_id, time_utc)) WITHOUT ROWID;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS models(
|
||||
id INTEGER PRIMARY KEY, created_utc INTEGER, pair TEXT, kind TEXT,
|
||||
dataset_id INTEGER, params TEXT, metrics TEXT, feature_names TEXT, blob BLOB,
|
||||
motivazione TEXT);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS validations(
|
||||
id INTEGER PRIMARY KEY, created_utc INTEGER, model_id INTEGER, method TEXT,
|
||||
trials INTEGER, sharpe_oos REAL, psr REAL, dsr REAL, pbo REAL, accuracy REAL,
|
||||
motivazione TEXT);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS champions(
|
||||
pair TEXT PRIMARY KEY, model_id INTEGER, promoted_utc INTEGER, motivazione TEXT);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS drift(
|
||||
id INTEGER PRIMARY KEY, time_utc INTEGER, pair TEXT, feature TEXT, psi REAL,
|
||||
ks REAL, alert INTEGER, motivazione TEXT);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS rl_runs(
|
||||
id INTEGER PRIMARY KEY, created_utc INTEGER, pair TEXT, seed INTEGER,
|
||||
episodes INTEGER, net_return REAL, sharpe REAL, max_drawdown REAL,
|
||||
baseline_return REAL, motivazione TEXT);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_decisions_time ON decisions(bar_time_utc);
|
||||
CREATE INDEX IF NOT EXISTS ix_executions_time ON executions(timestamp_utc);
|
||||
CREATE INDEX IF NOT EXISTS ix_trades_time ON trades(timestamp_utc);
|
||||
""";
|
||||
|
||||
Execute(ddl);
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Journal sink
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
public void Decision(DecisionRow row)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(row);
|
||||
|
||||
Guarded("decisions", () =>
|
||||
{
|
||||
using SqliteCommand cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
INSERT OR REPLACE INTO decisions VALUES(
|
||||
$t,$id,$pair,$a,$b,$ready,$ca,$cb,$spread,$z,$beta,$alpha,$p,$adf,$hl,$coint,
|
||||
$qa,$qb,$ez,$bars,$signal,$spa,$spb,$fa,$fb,$equity,$halted,$meta,$why)
|
||||
""";
|
||||
|
||||
P(cmd, "$t", Ms(row.BarTimeUtc));
|
||||
P(cmd, "$id", row.DecisionId);
|
||||
P(cmd, "$pair", row.Pair);
|
||||
P(cmd, "$a", row.SymbolA);
|
||||
P(cmd, "$b", row.SymbolB);
|
||||
P(cmd, "$ready", row.Ready ? 1 : 0);
|
||||
P(cmd, "$ca", row.CloseA);
|
||||
P(cmd, "$cb", row.CloseB);
|
||||
P(cmd, "$spread", row.Spread);
|
||||
P(cmd, "$z", row.ZScore);
|
||||
P(cmd, "$beta", row.Beta);
|
||||
P(cmd, "$alpha", row.Alpha);
|
||||
P(cmd, "$p", row.PValue);
|
||||
P(cmd, "$adf", row.AdfStatistic);
|
||||
P(cmd, "$hl", row.HalfLife);
|
||||
P(cmd, "$coint", row.Cointegrated ? 1 : 0);
|
||||
P(cmd, "$qa", row.QuantityA);
|
||||
P(cmd, "$qb", row.QuantityB);
|
||||
P(cmd, "$ez", row.EntryZ);
|
||||
P(cmd, "$bars", row.BarsHeld);
|
||||
P(cmd, "$signal", row.Signal);
|
||||
P(cmd, "$spa", row.SpreadPctA);
|
||||
P(cmd, "$spb", row.SpreadPctB);
|
||||
P(cmd, "$fa", row.FundingA);
|
||||
P(cmd, "$fb", row.FundingB);
|
||||
P(cmd, "$equity", row.Equity);
|
||||
P(cmd, "$halted", row.Halted ? 1 : 0);
|
||||
P(cmd, "$meta", double.IsFinite(row.MetaProbability) ? row.MetaProbability : DBNull.Value);
|
||||
P(cmd, "$why", row.Motivazione);
|
||||
cmd.ExecuteNonQuery();
|
||||
});
|
||||
}
|
||||
|
||||
public void Execution(ExecutionRow row)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(row);
|
||||
|
||||
Guarded("executions", () =>
|
||||
{
|
||||
using SqliteCommand cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
INSERT INTO executions(timestamp_utc,decision_id,pair,phase,approved,reason,side_a,
|
||||
notional_a,quantity_a,price_a,side_b,notional_b,quantity_b,price_b,net_funding_rate,
|
||||
equity,available_balance,gross_exposure,open_pairs,order_id_a,order_id_b,error,
|
||||
latency_ms,motivazione)
|
||||
VALUES($t,$id,$pair,$phase,$ok,$reason,$sa,$na,$qa,$pa,$sb,$nb,$qb,$pb,$fund,
|
||||
$equity,$avail,$gross,$open,$oa,$ob,$err,$lat,$why)
|
||||
""";
|
||||
|
||||
P(cmd, "$t", Ms(row.TimestampUtc));
|
||||
P(cmd, "$id", row.DecisionId);
|
||||
P(cmd, "$pair", row.Pair);
|
||||
P(cmd, "$phase", row.Phase);
|
||||
P(cmd, "$ok", row.Approved ? 1 : 0);
|
||||
P(cmd, "$reason", row.Reason);
|
||||
P(cmd, "$sa", row.SideA);
|
||||
P(cmd, "$na", row.NotionalA);
|
||||
P(cmd, "$qa", row.QuantityA);
|
||||
P(cmd, "$pa", row.PriceA);
|
||||
P(cmd, "$sb", row.SideB);
|
||||
P(cmd, "$nb", row.NotionalB);
|
||||
P(cmd, "$qb", row.QuantityB);
|
||||
P(cmd, "$pb", row.PriceB);
|
||||
P(cmd, "$fund", row.NetFundingRate);
|
||||
P(cmd, "$equity", row.Equity);
|
||||
P(cmd, "$avail", row.AvailableBalance);
|
||||
P(cmd, "$gross", row.GrossExposure);
|
||||
P(cmd, "$open", row.OpenPairs);
|
||||
P(cmd, "$oa", row.OrderIdA);
|
||||
P(cmd, "$ob", row.OrderIdB);
|
||||
P(cmd, "$err", row.Error);
|
||||
P(cmd, "$lat", row.LatencyMs);
|
||||
P(cmd, "$why", row.Motivazione);
|
||||
cmd.ExecuteNonQuery();
|
||||
});
|
||||
}
|
||||
|
||||
public void Trade(TradeRow row)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(row);
|
||||
|
||||
Guarded("trades", () =>
|
||||
{
|
||||
using SqliteCommand cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
INSERT INTO trades(timestamp_utc,event,symbol,side,quantity,price,order_id,stop,target,
|
||||
equity,realized_pnl,motivazione)
|
||||
VALUES($t,$event,$symbol,$side,$qty,$price,$order,$stop,$target,$equity,$pnl,$why)
|
||||
""";
|
||||
|
||||
P(cmd, "$t", Ms(row.TimestampUtc));
|
||||
P(cmd, "$event", row.Event);
|
||||
P(cmd, "$symbol", row.Symbol);
|
||||
P(cmd, "$side", row.Side);
|
||||
P(cmd, "$qty", row.Quantity);
|
||||
P(cmd, "$price", row.Price);
|
||||
P(cmd, "$order", row.OrderId ?? string.Empty);
|
||||
P(cmd, "$stop", row.Stop);
|
||||
P(cmd, "$target", row.Target);
|
||||
P(cmd, "$equity", row.Equity);
|
||||
P(cmd, "$pnl", row.RealizedPnl);
|
||||
P(cmd, "$why", row.Motivazione);
|
||||
cmd.ExecuteNonQuery();
|
||||
});
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Bars
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/// <summary>Stores bars in one transaction, replacing any already held for the same instant.</summary>
|
||||
public int UpsertBars(string symbol, string timeframe, IReadOnlyList<Bar> bars)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(bars);
|
||||
|
||||
lock (_gate)
|
||||
{
|
||||
using SqliteTransaction tx = _connection.BeginTransaction();
|
||||
using SqliteCommand cmd = _connection.CreateCommand();
|
||||
cmd.Transaction = tx;
|
||||
cmd.CommandText = """
|
||||
INSERT OR REPLACE INTO bars VALUES($s,$tf,$t,$o,$h,$l,$c,$v,$vwap,$n,$tb)
|
||||
""";
|
||||
|
||||
SqliteParameter s = cmd.Parameters.Add("$s", SqliteType.Text);
|
||||
SqliteParameter tf = cmd.Parameters.Add("$tf", SqliteType.Text);
|
||||
SqliteParameter t = cmd.Parameters.Add("$t", SqliteType.Integer);
|
||||
SqliteParameter o = cmd.Parameters.Add("$o", SqliteType.Real);
|
||||
SqliteParameter h = cmd.Parameters.Add("$h", SqliteType.Real);
|
||||
SqliteParameter l = cmd.Parameters.Add("$l", SqliteType.Real);
|
||||
SqliteParameter c = cmd.Parameters.Add("$c", SqliteType.Real);
|
||||
SqliteParameter v = cmd.Parameters.Add("$v", SqliteType.Real);
|
||||
SqliteParameter vwap = cmd.Parameters.Add("$vwap", SqliteType.Real);
|
||||
SqliteParameter n = cmd.Parameters.Add("$n", SqliteType.Integer);
|
||||
SqliteParameter tb = cmd.Parameters.Add("$tb", SqliteType.Real);
|
||||
|
||||
s.Value = symbol;
|
||||
tf.Value = timeframe;
|
||||
|
||||
int written = 0;
|
||||
foreach (Bar bar in bars)
|
||||
{
|
||||
t.Value = Ms(bar.TimeUtc);
|
||||
o.Value = bar.Open;
|
||||
h.Value = bar.High;
|
||||
l.Value = bar.Low;
|
||||
c.Value = bar.Close;
|
||||
v.Value = bar.Volume;
|
||||
vwap.Value = bar.Vwap;
|
||||
n.Value = bar.TradeCount;
|
||||
tb.Value = bar.TakerBuyVolume;
|
||||
written += cmd.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
tx.Commit();
|
||||
return written;
|
||||
}
|
||||
}
|
||||
|
||||
public List<Bar> LoadBars(string symbol, string timeframe, DateTime? fromUtc = null, DateTime? toUtc = null)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
using SqliteCommand cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
SELECT time_utc,open,high,low,close,volume,vwap,trades,taker_buy FROM bars
|
||||
WHERE symbol=$s AND timeframe=$tf AND time_utc>=$from AND time_utc<$to
|
||||
ORDER BY time_utc
|
||||
""";
|
||||
P(cmd, "$s", symbol);
|
||||
P(cmd, "$tf", timeframe);
|
||||
P(cmd, "$from", fromUtc is { } f ? Ms(f) : 0L);
|
||||
P(cmd, "$to", toUtc is { } u ? Ms(u) : long.MaxValue);
|
||||
|
||||
List<Bar> bars = [];
|
||||
using SqliteDataReader reader = cmd.ExecuteReader();
|
||||
while (reader.Read())
|
||||
{
|
||||
bars.Add(new Bar(
|
||||
FromMs(reader.GetInt64(0)),
|
||||
reader.GetDouble(1), reader.GetDouble(2), reader.GetDouble(3), reader.GetDouble(4),
|
||||
reader.GetDouble(5), reader.GetDouble(6), reader.GetInt32(7), reader.GetDouble(8)));
|
||||
}
|
||||
|
||||
return bars;
|
||||
}
|
||||
}
|
||||
|
||||
public long CountBars(string symbol, string timeframe)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
using SqliteCommand cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = "SELECT COUNT(*) FROM bars WHERE symbol=$s AND timeframe=$tf";
|
||||
P(cmd, "$s", symbol);
|
||||
P(cmd, "$tf", timeframe);
|
||||
return (long)(cmd.ExecuteScalar() ?? 0L);
|
||||
}
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Data quality
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
public void InsertDataQuality(
|
||||
DateTime runUtc, string file, string symbol, long rows, DateTime firstUtc, DateTime lastUtc,
|
||||
long duplicates, long gaps, long gapMinutes, long badPrices, string motivazione)
|
||||
{
|
||||
Guarded("data_quality", () =>
|
||||
{
|
||||
using SqliteCommand cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
INSERT INTO data_quality(run_utc,file,symbol,rows,first_utc,last_utc,duplicates,gaps,
|
||||
gap_minutes,bad_prices,motivazione)
|
||||
VALUES($run,$file,$symbol,$rows,$first,$last,$dup,$gaps,$gapmin,$bad,$why)
|
||||
""";
|
||||
P(cmd, "$run", Ms(runUtc));
|
||||
P(cmd, "$file", file);
|
||||
P(cmd, "$symbol", symbol);
|
||||
P(cmd, "$rows", rows);
|
||||
P(cmd, "$first", Ms(firstUtc));
|
||||
P(cmd, "$last", Ms(lastUtc));
|
||||
P(cmd, "$dup", duplicates);
|
||||
P(cmd, "$gaps", gaps);
|
||||
P(cmd, "$gapmin", gapMinutes);
|
||||
P(cmd, "$bad", badPrices);
|
||||
P(cmd, "$why", motivazione);
|
||||
cmd.ExecuteNonQuery();
|
||||
});
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Datasets, models, validation, champions, drift, RL
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/// <summary>One training example: the features at an instant and what happened next.</summary>
|
||||
public sealed record DatasetRowRecord(
|
||||
DateTime TimeUtc, DateTime EndUtc, int Label, int Side, double Return, double Weight, double[] Features);
|
||||
|
||||
public long SaveDataset(
|
||||
string pair, string timeframe, IReadOnlyList<string> featureNames, string labelSpec,
|
||||
IReadOnlyList<DatasetRowRecord> rows, string motivazione)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(rows);
|
||||
ArgumentNullException.ThrowIfNull(featureNames);
|
||||
|
||||
lock (_gate)
|
||||
{
|
||||
using SqliteTransaction tx = _connection.BeginTransaction();
|
||||
|
||||
long id;
|
||||
using (SqliteCommand head = _connection.CreateCommand())
|
||||
{
|
||||
head.Transaction = tx;
|
||||
head.CommandText = """
|
||||
INSERT INTO datasets(created_utc,pair,timeframe,from_utc,to_utc,rows,feature_names,
|
||||
label_spec,motivazione)
|
||||
VALUES($c,$pair,$tf,$from,$to,$rows,$names,$label,$why);
|
||||
SELECT last_insert_rowid();
|
||||
""";
|
||||
P(head, "$c", Ms(DateTime.UtcNow));
|
||||
P(head, "$pair", pair);
|
||||
P(head, "$tf", timeframe);
|
||||
P(head, "$from", rows.Count > 0 ? Ms(rows[0].TimeUtc) : 0L);
|
||||
P(head, "$to", rows.Count > 0 ? Ms(rows[^1].TimeUtc) : 0L);
|
||||
P(head, "$rows", (long)rows.Count);
|
||||
P(head, "$names", string.Join(';', featureNames));
|
||||
P(head, "$label", labelSpec);
|
||||
P(head, "$why", motivazione);
|
||||
id = (long)(head.ExecuteScalar() ?? 0L);
|
||||
}
|
||||
|
||||
using (SqliteCommand cmd = _connection.CreateCommand())
|
||||
{
|
||||
cmd.Transaction = tx;
|
||||
cmd.CommandText = """
|
||||
INSERT OR REPLACE INTO dataset_rows VALUES($id,$t,$end,$label,$side,$ret,$w,$f)
|
||||
""";
|
||||
SqliteParameter pid = cmd.Parameters.Add("$id", SqliteType.Integer);
|
||||
SqliteParameter pt = cmd.Parameters.Add("$t", SqliteType.Integer);
|
||||
SqliteParameter pend = cmd.Parameters.Add("$end", SqliteType.Integer);
|
||||
SqliteParameter plabel = cmd.Parameters.Add("$label", SqliteType.Integer);
|
||||
SqliteParameter pside = cmd.Parameters.Add("$side", SqliteType.Integer);
|
||||
SqliteParameter pret = cmd.Parameters.Add("$ret", SqliteType.Real);
|
||||
SqliteParameter pw = cmd.Parameters.Add("$w", SqliteType.Real);
|
||||
SqliteParameter pf = cmd.Parameters.Add("$f", SqliteType.Blob);
|
||||
|
||||
pid.Value = id;
|
||||
foreach (DatasetRowRecord row in rows)
|
||||
{
|
||||
pt.Value = Ms(row.TimeUtc);
|
||||
pend.Value = Ms(row.EndUtc);
|
||||
plabel.Value = row.Label;
|
||||
pside.Value = row.Side;
|
||||
pret.Value = row.Return;
|
||||
pw.Value = row.Weight;
|
||||
pf.Value = Pack(row.Features);
|
||||
cmd.ExecuteNonQuery();
|
||||
}
|
||||
}
|
||||
|
||||
tx.Commit();
|
||||
return id;
|
||||
}
|
||||
}
|
||||
|
||||
public (string Pair, string TimeFrame, string[] FeatureNames, List<DatasetRowRecord> Rows)? LoadDataset(long id)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
string pair;
|
||||
string timeframe;
|
||||
string[] names;
|
||||
|
||||
using (SqliteCommand head = _connection.CreateCommand())
|
||||
{
|
||||
head.CommandText = "SELECT pair,timeframe,feature_names FROM datasets WHERE id=$id";
|
||||
P(head, "$id", id);
|
||||
using SqliteDataReader r = head.ExecuteReader();
|
||||
if (!r.Read())
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
pair = r.GetString(0);
|
||||
timeframe = r.GetString(1);
|
||||
names = r.GetString(2).Split(';', StringSplitOptions.RemoveEmptyEntries);
|
||||
}
|
||||
|
||||
List<DatasetRowRecord> rows = [];
|
||||
using (SqliteCommand cmd = _connection.CreateCommand())
|
||||
{
|
||||
cmd.CommandText = """
|
||||
SELECT time_utc,end_utc,label,side,ret,weight,features FROM dataset_rows
|
||||
WHERE dataset_id=$id ORDER BY time_utc
|
||||
""";
|
||||
P(cmd, "$id", id);
|
||||
using SqliteDataReader r = cmd.ExecuteReader();
|
||||
while (r.Read())
|
||||
{
|
||||
rows.Add(new DatasetRowRecord(
|
||||
FromMs(r.GetInt64(0)), FromMs(r.GetInt64(1)), r.GetInt32(2), r.GetInt32(3),
|
||||
r.GetDouble(4), r.GetDouble(5), Unpack((byte[])r.GetValue(6))));
|
||||
}
|
||||
}
|
||||
|
||||
return (pair, timeframe, names, rows);
|
||||
}
|
||||
}
|
||||
|
||||
public long SaveModel(
|
||||
string pair, string kind, long datasetId, string paramsJson, string metricsJson,
|
||||
IReadOnlyList<string> featureNames, byte[] blob, string motivazione)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(featureNames);
|
||||
|
||||
lock (_gate)
|
||||
{
|
||||
using SqliteCommand cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
INSERT INTO models(created_utc,pair,kind,dataset_id,params,metrics,feature_names,blob,motivazione)
|
||||
VALUES($c,$pair,$kind,$ds,$params,$metrics,$names,$blob,$why);
|
||||
SELECT last_insert_rowid();
|
||||
""";
|
||||
P(cmd, "$c", Ms(DateTime.UtcNow));
|
||||
P(cmd, "$pair", pair);
|
||||
P(cmd, "$kind", kind);
|
||||
P(cmd, "$ds", datasetId);
|
||||
P(cmd, "$params", paramsJson);
|
||||
P(cmd, "$metrics", metricsJson);
|
||||
P(cmd, "$names", string.Join(';', featureNames));
|
||||
cmd.Parameters.Add("$blob", SqliteType.Blob).Value = blob;
|
||||
P(cmd, "$why", motivazione);
|
||||
return (long)(cmd.ExecuteScalar() ?? 0L);
|
||||
}
|
||||
}
|
||||
|
||||
public sealed record ModelRecord(
|
||||
long Id, string Pair, string Kind, long DatasetId, string ParamsJson, string MetricsJson,
|
||||
string[] FeatureNames, byte[] Blob, DateTime CreatedUtc);
|
||||
|
||||
public ModelRecord? LoadModel(long id)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
using SqliteCommand cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
SELECT id,pair,kind,dataset_id,params,metrics,feature_names,blob,created_utc
|
||||
FROM models WHERE id=$id
|
||||
""";
|
||||
P(cmd, "$id", id);
|
||||
using SqliteDataReader r = cmd.ExecuteReader();
|
||||
return r.Read() ? Read(r) : null;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>The model currently promoted for a pair, or null when none has been.</summary>
|
||||
public ModelRecord? LoadChampion(string pair)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
using SqliteCommand cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
SELECT m.id,m.pair,m.kind,m.dataset_id,m.params,m.metrics,m.feature_names,m.blob,m.created_utc
|
||||
FROM champions c JOIN models m ON m.id=c.model_id WHERE c.pair=$pair
|
||||
""";
|
||||
P(cmd, "$pair", pair);
|
||||
using SqliteDataReader r = cmd.ExecuteReader();
|
||||
return r.Read() ? Read(r) : null;
|
||||
}
|
||||
}
|
||||
|
||||
public void SetChampion(string pair, long modelId, string motivazione)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
using SqliteCommand cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
INSERT OR REPLACE INTO champions(pair,model_id,promoted_utc,motivazione)
|
||||
VALUES($pair,$id,$t,$why)
|
||||
""";
|
||||
P(cmd, "$pair", pair);
|
||||
P(cmd, "$id", modelId);
|
||||
P(cmd, "$t", Ms(DateTime.UtcNow));
|
||||
P(cmd, "$why", motivazione);
|
||||
cmd.ExecuteNonQuery();
|
||||
}
|
||||
}
|
||||
|
||||
public long SaveValidation(
|
||||
long modelId, string method, int trials, double sharpeOos, double psr, double dsr, double pbo,
|
||||
double accuracy, string motivazione)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
using SqliteCommand cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
INSERT INTO validations(created_utc,model_id,method,trials,sharpe_oos,psr,dsr,pbo,accuracy,motivazione)
|
||||
VALUES($c,$m,$method,$trials,$sharpe,$psr,$dsr,$pbo,$acc,$why);
|
||||
SELECT last_insert_rowid();
|
||||
""";
|
||||
P(cmd, "$c", Ms(DateTime.UtcNow));
|
||||
P(cmd, "$m", modelId);
|
||||
P(cmd, "$method", method);
|
||||
P(cmd, "$trials", trials);
|
||||
P(cmd, "$sharpe", sharpeOos);
|
||||
P(cmd, "$psr", psr);
|
||||
P(cmd, "$dsr", dsr);
|
||||
P(cmd, "$pbo", pbo);
|
||||
P(cmd, "$acc", accuracy);
|
||||
P(cmd, "$why", motivazione);
|
||||
return (long)(cmd.ExecuteScalar() ?? 0L);
|
||||
}
|
||||
}
|
||||
|
||||
public void InsertDrift(DateTime timeUtc, string pair, string feature, double psi, double ks, bool alert, string motivazione)
|
||||
{
|
||||
Guarded("drift", () =>
|
||||
{
|
||||
using SqliteCommand cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
INSERT INTO drift(time_utc,pair,feature,psi,ks,alert,motivazione)
|
||||
VALUES($t,$pair,$f,$psi,$ks,$alert,$why)
|
||||
""";
|
||||
P(cmd, "$t", Ms(timeUtc));
|
||||
P(cmd, "$pair", pair);
|
||||
P(cmd, "$f", feature);
|
||||
P(cmd, "$psi", psi);
|
||||
P(cmd, "$ks", ks);
|
||||
P(cmd, "$alert", alert ? 1 : 0);
|
||||
P(cmd, "$why", motivazione);
|
||||
cmd.ExecuteNonQuery();
|
||||
});
|
||||
}
|
||||
|
||||
public void InsertRlRun(
|
||||
string pair, int seed, int episodes, double netReturn, double sharpe, double maxDrawdown,
|
||||
double baselineReturn, string motivazione)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
using SqliteCommand cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
INSERT INTO rl_runs(created_utc,pair,seed,episodes,net_return,sharpe,max_drawdown,baseline_return,motivazione)
|
||||
VALUES($c,$pair,$seed,$ep,$net,$sharpe,$dd,$base,$why)
|
||||
""";
|
||||
P(cmd, "$c", Ms(DateTime.UtcNow));
|
||||
P(cmd, "$pair", pair);
|
||||
P(cmd, "$seed", seed);
|
||||
P(cmd, "$ep", episodes);
|
||||
P(cmd, "$net", netReturn);
|
||||
P(cmd, "$sharpe", sharpe);
|
||||
P(cmd, "$dd", maxDrawdown);
|
||||
P(cmd, "$base", baselineReturn);
|
||||
P(cmd, "$why", motivazione);
|
||||
cmd.ExecuteNonQuery();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Row count of any table, for the tools' status output.</summary>
|
||||
public long Count(string table)
|
||||
{
|
||||
if (!table.All(static c => char.IsLetterOrDigit(c) || c == '_'))
|
||||
{
|
||||
throw new ArgumentException("Nome di tabella non valido.", nameof(table));
|
||||
}
|
||||
|
||||
lock (_gate)
|
||||
{
|
||||
using SqliteCommand cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = $"SELECT COUNT(*) FROM {table}";
|
||||
return (long)(cmd.ExecuteScalar() ?? 0L);
|
||||
}
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Plumbing
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
private static ModelRecord Read(SqliteDataReader r) => new(
|
||||
r.GetInt64(0), r.GetString(1), r.GetString(2), r.GetInt64(3), r.GetString(4), r.GetString(5),
|
||||
r.GetString(6).Split(';', StringSplitOptions.RemoveEmptyEntries), (byte[])r.GetValue(7),
|
||||
FromMs(r.GetInt64(8)));
|
||||
|
||||
private void Guarded(string table, Action write)
|
||||
{
|
||||
try
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
write();
|
||||
}
|
||||
}
|
||||
catch (Exception ex) when (ex is SqliteException or InvalidOperationException or IOException)
|
||||
{
|
||||
OnError?.Invoke($"scrittura nella tabella {table} fallita", ex);
|
||||
}
|
||||
}
|
||||
|
||||
private void Execute(string sql)
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
using SqliteCommand cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = sql;
|
||||
cmd.ExecuteNonQuery();
|
||||
}
|
||||
}
|
||||
|
||||
private static void P(SqliteCommand cmd, string name, object? value) =>
|
||||
cmd.Parameters.AddWithValue(name, value switch
|
||||
{
|
||||
null => DBNull.Value,
|
||||
double d when !double.IsFinite(d) => DBNull.Value,
|
||||
_ => value,
|
||||
});
|
||||
|
||||
private static long Ms(DateTime utc) =>
|
||||
utc == DateTime.MinValue ? 0 : new DateTimeOffset(DateTime.SpecifyKind(utc, DateTimeKind.Utc)).ToUnixTimeMilliseconds();
|
||||
|
||||
private static DateTime FromMs(long ms) =>
|
||||
ms <= 0 ? DateTime.MinValue : DateTimeOffset.FromUnixTimeMilliseconds(ms).UtcDateTime;
|
||||
|
||||
/// <summary>Doubles to little-endian bytes: eight per feature, no header.</summary>
|
||||
public static byte[] Pack(double[] values)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(values);
|
||||
byte[] bytes = new byte[values.Length * sizeof(double)];
|
||||
Buffer.BlockCopy(values, 0, bytes, 0, bytes.Length);
|
||||
return bytes;
|
||||
}
|
||||
|
||||
public static double[] Unpack(byte[] bytes)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(bytes);
|
||||
double[] values = new double[bytes.Length / sizeof(double)];
|
||||
Buffer.BlockCopy(bytes, 0, values, 0, values.Length * sizeof(double));
|
||||
return values;
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
lock (_gate)
|
||||
{
|
||||
_connection.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Formats a timestamp the way every table's <c>motivazione</c> column does.</summary>
|
||||
public static string Stamp(DateTime utc) => utc.ToString("yyyy-MM-dd HH:mm", CultureInfo.InvariantCulture);
|
||||
}
|
||||
@@ -68,22 +68,23 @@ public class ConfigDefaultsTests
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The backtest could not find a threshold set that survived on data it had not been
|
||||
/// chosen on, so the shipped configuration must not send orders until a human decides
|
||||
/// it should.
|
||||
/// The shipped configuration points at the testnet and sends real orders there.
|
||||
/// <para>
|
||||
/// The backtest found no threshold set that survived out of sample, so the safety
|
||||
/// comes from the venue, not from suppressing orders: on the testnet the money is
|
||||
/// fake and the fills, rejections and lot rules are real, which is the closest thing
|
||||
/// to production a dry run can never be.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void TheDefaultShipsInDryRun()
|
||||
public void TheDefaultSendsOrdersToTheTestnet()
|
||||
{
|
||||
Assert.True(ConfigDefaults.Parse().Engine.DryRun,
|
||||
"il default deve partire in dry-run: nessuna combinazione di soglie ha retto " +
|
||||
"sulla fetta di conferma, quindi inviare ordini di fabbrica non è difendibile");
|
||||
}
|
||||
BotConfig config = ConfigDefaults.Parse();
|
||||
|
||||
[Fact]
|
||||
public void TheDefaultShipsOnTestnet()
|
||||
{
|
||||
Assert.True(ConfigDefaults.Parse().Binance.Testnet);
|
||||
Assert.True(config.Binance.Testnet);
|
||||
Assert.False(config.Engine.DryRun,
|
||||
"in testnet gli ordini devono partire davvero: è l'unico modo di provare " +
|
||||
"esecuzioni, rifiuti e regole di lotto senza rischiare niente");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
|
||||
@@ -38,6 +38,7 @@
|
||||
<ProjectReference Include="..\..\src\Encelado.Core\Encelado.Core.csproj" />
|
||||
<ProjectReference Include="..\..\src\Encelado.Binance\Encelado.Binance.csproj" />
|
||||
<ProjectReference Include="..\..\src\Encelado.Bot\Encelado.Bot.csproj" />
|
||||
<ProjectReference Include="..\..\src\Encelado.Storage\Encelado.Storage.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
@@ -0,0 +1,159 @@
|
||||
using System.Text;
|
||||
using Encelado.Bot.Logging;
|
||||
|
||||
namespace Encelado.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The structured log format: one <c>;</c>-separated row per entry, searchable by level,
|
||||
/// component and pair.
|
||||
/// <para>
|
||||
/// The previous format was checked by nobody and it showed: a time with no date across
|
||||
/// a file that spanned weeks, and errors with no stack trace. These tests pin the shape
|
||||
/// of a row so a future "small change" to the writer cannot quietly take the file back
|
||||
/// to prose.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public class LogFormatTests
|
||||
{
|
||||
private static readonly DateTime When = new(2026, 9, 8, 10, 15, 2, 113, DateTimeKind.Local);
|
||||
|
||||
private static string Row(LogLevel level, string message, Exception? ex = null, string? code = null) =>
|
||||
Log.FormatRow(new Log.Entry(When, level, message, ex, code, "PairRouter"), new StringBuilder());
|
||||
|
||||
[Fact]
|
||||
public void ARowHasEveryColumnInHeaderOrder()
|
||||
{
|
||||
string row = Row(LogLevel.Info, "[ETHUSDT/BTCUSDT] NON ENTRO — book troppo largo", code: "entry.refused");
|
||||
|
||||
string[] cells = row.Split(';');
|
||||
|
||||
Assert.Equal(Log.Header.Split(';').Length, cells.Length);
|
||||
Assert.StartsWith("2026-09-08T10:15:02.113", cells[0], StringComparison.Ordinal);
|
||||
Assert.Equal("INF", cells[1]);
|
||||
Assert.Equal("PairRouter", cells[2]);
|
||||
Assert.Equal("ETHUSDT/BTCUSDT", cells[3]);
|
||||
Assert.Equal("entry.refused", cells[4]);
|
||||
Assert.Equal("NON ENTRO — book troppo largo", cells[5]);
|
||||
Assert.Equal(string.Empty, cells[6]);
|
||||
Assert.Equal(string.Empty, cells[7]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TheTimestampCarriesTheDateAndTheOffset()
|
||||
{
|
||||
// A time without a date is what made a three-week file unsearchable.
|
||||
string cell = Row(LogLevel.Info, "x").Split(';')[0];
|
||||
|
||||
Assert.Matches(@"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}[+-]\d{2}:\d{2}$", cell);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ThePairIsLiftedOutOfTheMessageTag()
|
||||
{
|
||||
(string subject, string message) = Log.SplitSubject("[SOLUSDT/AVAXUSDT] z=1.48");
|
||||
|
||||
Assert.Equal("SOLUSDT/AVAXUSDT", subject);
|
||||
Assert.Equal("z=1.48", message);
|
||||
|
||||
(subject, message) = Log.SplitSubject("motore avviato");
|
||||
Assert.Equal(string.Empty, subject);
|
||||
Assert.Equal("motore avviato", message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AnErrorCarriesTheExceptionAndItsStack()
|
||||
{
|
||||
Exception caught;
|
||||
try
|
||||
{
|
||||
throw new InvalidOperationException("gamba rifiutata", new TimeoutException("timeout"));
|
||||
}
|
||||
catch (InvalidOperationException ex)
|
||||
{
|
||||
caught = ex;
|
||||
}
|
||||
|
||||
string row = Row(LogLevel.Error, "[ETHUSDT/BTCUSDT] ingresso fallito", caught);
|
||||
string[] cells = Split(row);
|
||||
|
||||
Assert.Equal("ERR", cells[1]);
|
||||
Assert.Contains("InvalidOperationException: gamba rifiutata", cells[6], StringComparison.Ordinal);
|
||||
Assert.Contains("TimeoutException: timeout", cells[6], StringComparison.Ordinal);
|
||||
Assert.Contains(nameof(AnErrorCarriesTheExceptionAndItsStack), cells[7], StringComparison.Ordinal);
|
||||
Assert.DoesNotContain('\n', cells[7]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ErrorsAreFindableWithOneFilter()
|
||||
{
|
||||
string row = Row(LogLevel.Error, "qualcosa");
|
||||
|
||||
// The whole point: `grep ";ERR;"` and nothing else.
|
||||
Assert.Contains(";ERR;", row, StringComparison.Ordinal);
|
||||
Assert.DoesNotContain(";ERR;", Row(LogLevel.Warn, "qualcosa"), StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ASemicolonInsideAMessageIsQuotedNotSplit()
|
||||
{
|
||||
string row = Row(LogLevel.Info, "risk engine [SpreadTooWide]: book largo 0,09%; massimo 0,06%");
|
||||
|
||||
Assert.Equal(Log.Header.Split(';').Length, Split(row).Length);
|
||||
Assert.Contains("\"risk engine [SpreadTooWide]: book largo 0,09%; massimo 0,06%\"", row, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ANewlineInsideAMessageStaysOnOneLine()
|
||||
{
|
||||
string row = Row(LogLevel.Warn, "prima riga\nseconda riga");
|
||||
|
||||
Assert.DoesNotContain('\n', row);
|
||||
Assert.DoesNotContain('\r', row);
|
||||
}
|
||||
|
||||
/// <summary>Splits a CSV row honouring quotes, the way a spreadsheet would.</summary>
|
||||
private static string[] Split(string row)
|
||||
{
|
||||
List<string> cells = [];
|
||||
StringBuilder cell = new();
|
||||
bool quoted = false;
|
||||
|
||||
for (int i = 0; i < row.Length; i++)
|
||||
{
|
||||
char c = row[i];
|
||||
|
||||
if (quoted)
|
||||
{
|
||||
if (c == '"' && i + 1 < row.Length && row[i + 1] == '"')
|
||||
{
|
||||
cell.Append('"');
|
||||
i++;
|
||||
}
|
||||
else if (c == '"')
|
||||
{
|
||||
quoted = false;
|
||||
}
|
||||
else
|
||||
{
|
||||
cell.Append(c);
|
||||
}
|
||||
}
|
||||
else if (c == '"')
|
||||
{
|
||||
quoted = true;
|
||||
}
|
||||
else if (c == ';')
|
||||
{
|
||||
cells.Add(cell.ToString());
|
||||
cell.Clear();
|
||||
}
|
||||
else
|
||||
{
|
||||
cell.Append(c);
|
||||
}
|
||||
}
|
||||
|
||||
cells.Add(cell.ToString());
|
||||
return [.. cells];
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,490 @@
|
||||
using Encelado.Core.Market;
|
||||
using Encelado.Core.Ml;
|
||||
|
||||
namespace Encelado.Tests;
|
||||
|
||||
public class FractionalDifferentiationTests
|
||||
{
|
||||
[Fact]
|
||||
public void OrderOneIsAFirstDifference()
|
||||
{
|
||||
double[] w = FractionalDifferentiation.Weights(1.0);
|
||||
|
||||
Assert.Equal(2, w.Length);
|
||||
Assert.Equal(1, w[0], 10);
|
||||
Assert.Equal(-1, w[1], 10);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void OrderZeroLeavesTheSeriesAlone()
|
||||
{
|
||||
double[] w = FractionalDifferentiation.Weights(0.0);
|
||||
|
||||
Assert.Single(w);
|
||||
Assert.Equal(1, w[0], 10);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void WeightsAlternateAndShrink()
|
||||
{
|
||||
double[] w = FractionalDifferentiation.Weights(0.5, threshold: 1e-3);
|
||||
|
||||
Assert.True(w.Length > 5);
|
||||
Assert.Equal(1, w[0], 10);
|
||||
Assert.Equal(-0.5, w[1], 10);
|
||||
Assert.Equal(-0.125, w[2], 10);
|
||||
for (int k = 1; k < w.Length; k++)
|
||||
{
|
||||
Assert.True(w[k] < 0);
|
||||
Assert.True(Math.Abs(w[k]) <= Math.Abs(w[k - 1]));
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FindsAnOrderBelowOneForARandomWalk()
|
||||
{
|
||||
Random rng = new(3);
|
||||
double[] walk = new double[1500];
|
||||
walk[0] = 100;
|
||||
for (int i = 1; i < walk.Length; i++)
|
||||
{
|
||||
walk[i] = walk[i - 1] + ((rng.NextDouble() - 0.5) * 0.5);
|
||||
}
|
||||
|
||||
(double d, double[] series, _) = FractionalDifferentiation.MinimumStationaryOrder(walk, step: 0.1);
|
||||
|
||||
// A random walk is I(1): a fraction of a difference suffices, and the point of
|
||||
// the whole method is that the fraction is well below one.
|
||||
Assert.True(d < 1.0, $"d* = {d}");
|
||||
Assert.True(d > 0.0);
|
||||
Assert.Equal(walk.Length, series.Length);
|
||||
}
|
||||
}
|
||||
|
||||
public class TripleBarrierTests
|
||||
{
|
||||
private static readonly double[] Flat = Enumerable.Repeat(100.0, 50).ToArray();
|
||||
|
||||
[Fact]
|
||||
public void HitsTheProfitBarrierFirst()
|
||||
{
|
||||
double[] prices = (double[])Flat.Clone();
|
||||
prices[3] = 103;
|
||||
double[] vol = Enumerable.Repeat(0.01, prices.Length).ToArray();
|
||||
|
||||
List<BarrierLabel> labels = TripleBarrier.Label(prices, [0], null, vol, 2, 2, 20);
|
||||
|
||||
BarrierLabel label = Assert.Single(labels);
|
||||
Assert.Equal(1, label.Label);
|
||||
Assert.Equal(3, label.EndIndex);
|
||||
Assert.Equal(1, label.MetaLabel);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void HitsTheStopFirstAndMirrorsForAShort()
|
||||
{
|
||||
double[] prices = (double[])Flat.Clone();
|
||||
prices[2] = 97;
|
||||
double[] vol = Enumerable.Repeat(0.01, prices.Length).ToArray();
|
||||
|
||||
// Long: the fall is a stop. Short: the same fall is a profit.
|
||||
BarrierLabel asLong = Assert.Single(TripleBarrier.Label(prices, [0], [1], vol, 2, 2, 20));
|
||||
BarrierLabel asShort = Assert.Single(TripleBarrier.Label(prices, [0], [-1], vol, 2, 2, 20));
|
||||
|
||||
Assert.Equal(-1, asLong.Label);
|
||||
Assert.Equal(1, asShort.Label);
|
||||
Assert.True(asShort.Return > 0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TheClockRunsOutOnAQuietPath()
|
||||
{
|
||||
double[] prices = (double[])Flat.Clone();
|
||||
prices[10] = 100.5;
|
||||
double[] vol = Enumerable.Repeat(0.01, prices.Length).ToArray();
|
||||
|
||||
BarrierLabel label = Assert.Single(TripleBarrier.Label(prices, [0], null, vol, 2, 2, 10));
|
||||
|
||||
Assert.Equal(10, label.EndIndex);
|
||||
Assert.Equal(1, label.Label);
|
||||
Assert.Equal(0.005, label.Return, 8);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RollingVolatilityIsPositiveAndFinite()
|
||||
{
|
||||
Random rng = new(4);
|
||||
double[] prices = new double[300];
|
||||
prices[0] = 100;
|
||||
for (int i = 1; i < prices.Length; i++)
|
||||
{
|
||||
prices[i] = prices[i - 1] * (1 + ((rng.NextDouble() - 0.5) * 0.02));
|
||||
}
|
||||
|
||||
double[] vol = TripleBarrier.RollingVolatility(prices, 50);
|
||||
|
||||
Assert.True(double.IsNaN(vol[0]));
|
||||
Assert.All(vol.Skip(10), static v => Assert.True(v > 0 && double.IsFinite(v)));
|
||||
}
|
||||
}
|
||||
|
||||
public class SampleWeightTests
|
||||
{
|
||||
[Fact]
|
||||
public void ALabelAloneInTimeWeighsMoreThanOneInACrowd()
|
||||
{
|
||||
List<BarrierLabel> labels =
|
||||
[
|
||||
new(0, 10, 1, 0.01, 0.01, 1),
|
||||
new(100, 110, 1, 0.01, 0.01, 1),
|
||||
new(101, 111, 1, 0.01, 0.01, 1),
|
||||
new(102, 112, 1, 0.01, 0.01, 1),
|
||||
];
|
||||
|
||||
double[] w = SampleWeights.AverageUniqueness(labels, 200);
|
||||
|
||||
Assert.Equal(4, w.Length);
|
||||
Assert.True(w[0] > w[1]);
|
||||
Assert.Equal(1.0, w.Average(), 8);
|
||||
}
|
||||
}
|
||||
|
||||
public class GbdtTests
|
||||
{
|
||||
private static (double[][] X, int[] Y) Toy(int n, int seed)
|
||||
{
|
||||
Random rng = new(seed);
|
||||
double[][] x = new double[n][];
|
||||
int[] y = new int[n];
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
double a = rng.NextDouble() * 2 - 1;
|
||||
double b = rng.NextDouble() * 2 - 1;
|
||||
double noise = (rng.NextDouble() - 0.5) * 0.1;
|
||||
x[i] = [a, b, rng.NextDouble()];
|
||||
y[i] = (a * b) + noise > 0 ? 1 : 0;
|
||||
}
|
||||
|
||||
return (x, y);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void LearnsANonLinearRuleAndIgnoresTheNoiseColumn()
|
||||
{
|
||||
(double[][] x, int[] y) = Toy(3000, 7);
|
||||
GbdtModel model = Gbdt.Train(x, y, null, new GbdtParams { Trees = 150, MaxDepth = 3 }, ["a", "b", "noise"]);
|
||||
|
||||
(double[][] tx, int[] ty) = Toy(1000, 8);
|
||||
int correct = 0;
|
||||
for (int i = 0; i < tx.Length; i++)
|
||||
{
|
||||
correct += (model.Predict(tx[i]) >= 0.5 ? 1 : 0) == ty[i] ? 1 : 0;
|
||||
}
|
||||
|
||||
// The rule is learnable to about 94%: the rest is the noise term flipping labels
|
||||
// near the a·b = 0 boundary.
|
||||
Assert.True(correct / (double)tx.Length > 0.85, $"accuratezza {correct / (double)tx.Length:P1}");
|
||||
|
||||
// The XOR-like rule uses a and b; the third column is pure noise.
|
||||
Assert.True(model.Importance[0] > model.Importance[2]);
|
||||
Assert.True(model.Importance[1] > model.Importance[2]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SurvivesSerialisation()
|
||||
{
|
||||
(double[][] x, int[] y) = Toy(800, 9);
|
||||
GbdtModel model = Gbdt.Train(x, y, null, new GbdtParams { Trees = 40 }, ["a", "b", "noise"]);
|
||||
|
||||
GbdtModel back = GbdtModel.FromBytes(model.ToBytes());
|
||||
|
||||
Assert.Equal(model.TreeCount, back.TreeCount);
|
||||
Assert.Equal(model.FeatureNames, back.FeatureNames);
|
||||
for (int i = 0; i < 20; i++)
|
||||
{
|
||||
Assert.Equal(model.Predict(x[i]), back.Predict(x[i]), 12);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ImputesMissingValuesWithTheTrainingMedian()
|
||||
{
|
||||
(double[][] x, int[] y) = Toy(500, 10);
|
||||
GbdtModel model = Gbdt.Train(x, y, null, new GbdtParams { Trees = 20 }, ["a", "b", "noise"]);
|
||||
|
||||
double p = model.Predict([double.NaN, 0.5, double.NaN]);
|
||||
Assert.InRange(p, 0, 1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RefusesAMismatchedFeatureCount()
|
||||
{
|
||||
(double[][] x, int[] y) = Toy(200, 11);
|
||||
GbdtModel model = Gbdt.Train(x, y, null, new GbdtParams { Trees = 5 }, ["a", "b", "noise"]);
|
||||
|
||||
Assert.Throws<ArgumentException>(() => model.Predict([1, 2]));
|
||||
}
|
||||
}
|
||||
|
||||
public class PurgedCvTests
|
||||
{
|
||||
[Fact]
|
||||
public void PurgingRemovesTrainRowsWhoseLabelsOverlapTheTestSpan()
|
||||
{
|
||||
// Ten rows every ten bars, each labelled over the next fifteen bars.
|
||||
int n = 10;
|
||||
int[] starts = Enumerable.Range(0, n).Select(i => i * 10).ToArray();
|
||||
int[] ends = starts.Select(s => s + 15).ToArray();
|
||||
|
||||
// The second fold tests rows 2 and 3, whose labels span 20..45. Row 1 (10..25)
|
||||
// overlaps the start of that span and row 4 (40..55) its end: both leak and both
|
||||
// go. Row 0 (0..15) and row 5 (50..65) touch nothing and stay.
|
||||
List<CvSplit> splits = PurgedCv.KFold(n, 5, starts, ends, embargoFraction: 0);
|
||||
CvSplit second = splits[1];
|
||||
|
||||
Assert.Equal([2, 3], second.Test);
|
||||
Assert.DoesNotContain(1, second.Train);
|
||||
Assert.DoesNotContain(4, second.Train);
|
||||
Assert.Contains(0, second.Train);
|
||||
Assert.Contains(5, second.Train);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TheEmbargoDropsRowsJustAfterTheTestBlock()
|
||||
{
|
||||
int n = 20;
|
||||
int[] starts = Enumerable.Range(0, n).Select(i => i * 10).ToArray();
|
||||
int[] ends = starts.Select(s => s + 1).ToArray();
|
||||
|
||||
// Total span 0..191; a 10% embargo is 20 bars, i.e. two rows after the block.
|
||||
List<CvSplit> splits = PurgedCv.KFold(n, 4, starts, ends, embargoFraction: 0.10);
|
||||
CvSplit first = splits[0];
|
||||
|
||||
Assert.Equal(Enumerable.Range(0, 5).ToArray(), first.Test);
|
||||
Assert.DoesNotContain(5, first.Train);
|
||||
Assert.DoesNotContain(6, first.Train);
|
||||
Assert.Contains(7, first.Train);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void HoldingOutBothEndsKeepsTheMiddleForTraining()
|
||||
{
|
||||
// Sixty rows, labels five bars long. The split that tests groups 0 and 5 must
|
||||
// still train on groups 1-4: two spans, not one span from start to finish.
|
||||
int n = 60;
|
||||
int[] starts = Enumerable.Range(0, n).Select(i => i * 10).ToArray();
|
||||
int[] ends = starts.Select(s => s + 5).ToArray();
|
||||
|
||||
CpcvSplit ends0and5 = PurgedCv.Combinatorial(n, 6, 2, starts, ends, 0)
|
||||
.Single(s => s.TestGroups[0] == 0 && s.TestGroups[1] == 5);
|
||||
|
||||
Assert.Equal(20, ends0and5.Test.Length);
|
||||
Assert.Equal(40, ends0and5.Train.Length);
|
||||
Assert.Contains(30, ends0and5.Train);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CombinatorialSplitsCountAndPathsMatchTheFormula()
|
||||
{
|
||||
int n = 60;
|
||||
int[] starts = Enumerable.Range(0, n).ToArray();
|
||||
int[] ends = starts.Select(s => s + 1).ToArray();
|
||||
|
||||
List<CpcvSplit> splits = PurgedCv.Combinatorial(n, 6, 2, starts, ends, 0);
|
||||
|
||||
// C(6,2) = 15 splits; φ = 2/6·15 = 5 paths.
|
||||
Assert.Equal(15, splits.Count);
|
||||
Assert.Equal(5, PurgedCv.PathCount(6, 2));
|
||||
|
||||
List<double[]> predictions = [.. splits.Select(s => s.Test.Select(static i => (double)i).ToArray())];
|
||||
List<double[]> paths = PurgedCv.AssemblePaths(n, 6, splits, predictions);
|
||||
|
||||
// Every path covers every row exactly once, with the row's own value.
|
||||
Assert.Equal(5, paths.Count);
|
||||
foreach (double[] path in paths)
|
||||
{
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
Assert.Equal(i, path[i], 10);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public class PboTests
|
||||
{
|
||||
[Fact]
|
||||
public void PureNoiseIsOverfitAboutHalfTheTime()
|
||||
{
|
||||
Random rng = new(12);
|
||||
List<double[]> trials = [];
|
||||
for (int t = 0; t < 12; t++)
|
||||
{
|
||||
trials.Add(Enumerable.Range(0, 400).Select(_ => rng.NextDouble() - 0.5).ToArray());
|
||||
}
|
||||
|
||||
PboResult result = Pbo.Compute(trials, blocks: 8);
|
||||
|
||||
Assert.Equal(70, result.Combinations);
|
||||
Assert.InRange(result.Probability, 0.25, 0.75);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AGenuineEdgeIsRarelyOverfit()
|
||||
{
|
||||
Random rng = new(13);
|
||||
List<double[]> trials = [];
|
||||
for (int t = 0; t < 8; t++)
|
||||
{
|
||||
double edge = t == 0 ? 0.3 : 0;
|
||||
trials.Add(Enumerable.Range(0, 400).Select(_ => rng.NextDouble() - 0.5 + edge).ToArray());
|
||||
}
|
||||
|
||||
PboResult result = Pbo.Compute(trials, blocks: 8);
|
||||
|
||||
Assert.True(result.Probability < 0.2, $"PBO {result.Probability}");
|
||||
}
|
||||
}
|
||||
|
||||
public class AlternativeBarTests
|
||||
{
|
||||
private static List<Bar> Minutes(int n)
|
||||
{
|
||||
DateTime t = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
List<Bar> bars = [];
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
double p = 100 + i;
|
||||
bars.Add(new Bar(t.AddMinutes(i), p, p + 1, p - 1, p + 0.5, 10, p, 5, i % 3 == 0 ? 8 : 2));
|
||||
}
|
||||
|
||||
return bars;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void VolumeBarsCloseAtTheThreshold()
|
||||
{
|
||||
List<Bar> bars = AlternativeBars.VolumeBars(Minutes(10), 30);
|
||||
|
||||
// 10 volume per minute: a bar every three minutes, the tenth minute left over.
|
||||
Assert.Equal(3, bars.Count);
|
||||
Assert.Equal(30, bars[0].Volume, 8);
|
||||
Assert.Equal(100, bars[0].Open, 8);
|
||||
Assert.Equal(102.5, bars[0].Close, 8);
|
||||
Assert.Equal(103, bars[0].High, 8);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ImbalanceBarsAdaptTheirThreshold()
|
||||
{
|
||||
List<Bar> bars = AlternativeBars.ImbalanceBars(Minutes(200), initialThreshold: 10, span: 5);
|
||||
|
||||
Assert.True(bars.Count > 5);
|
||||
Assert.True(bars.Count < 200);
|
||||
}
|
||||
}
|
||||
|
||||
public class FeatureRegistryTests
|
||||
{
|
||||
private static PairSeries Series(int n)
|
||||
{
|
||||
DateTime t = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
Random rng = new(14);
|
||||
List<Bar> a = [];
|
||||
List<Bar> b = [];
|
||||
double pa = 100, pb = 50;
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
pa *= 1 + ((rng.NextDouble() - 0.5) * 0.01);
|
||||
pb *= 1 + ((rng.NextDouble() - 0.5) * 0.01);
|
||||
a.Add(new Bar(t.AddMinutes(15 * i), pa, pa, pa, pa, 100, pa, 10, 60));
|
||||
b.Add(new Bar(t.AddMinutes(15 * i), pb, pb, pb, pb, 200, pb, 10, 90));
|
||||
}
|
||||
|
||||
double[] beta = Enumerable.Repeat(1.0, n).ToArray();
|
||||
double[] alpha = Enumerable.Repeat(0.0, n).ToArray();
|
||||
double[] hl = Enumerable.Repeat(30.0, n).ToArray();
|
||||
return PairSeries.Build(a, b, beta, alpha, hl, zWindow: 30);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TheDefaultRegistryComputesEveryFeatureForEveryBar()
|
||||
{
|
||||
PairSeries s = Series(300);
|
||||
FeatureRegistry registry = FeatureRegistry.Default();
|
||||
|
||||
double[][] matrix = registry.Compute(s);
|
||||
|
||||
Assert.Equal(300, matrix.Length);
|
||||
Assert.Equal(registry.Count, matrix[0].Length);
|
||||
Assert.All(matrix[^1], static v => Assert.True(double.IsFinite(v)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RegisteringAFeatureTwiceIsRefused()
|
||||
{
|
||||
FeatureRegistry registry = FeatureRegistry.Default();
|
||||
Assert.Throws<ArgumentException>(() => registry.Register("z", static s => new double[s.Count]));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void HoursToFundingCountsDownToTheNextSettlement()
|
||||
{
|
||||
Assert.Equal(8, FeatureRegistry.HoursToFunding(new DateTime(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc)), 6);
|
||||
Assert.Equal(1, FeatureRegistry.HoursToFunding(new DateTime(2026, 1, 1, 7, 0, 0, DateTimeKind.Utc)), 6);
|
||||
Assert.Equal(0.5, FeatureRegistry.HoursToFunding(new DateTime(2026, 1, 1, 15, 30, 0, DateTimeKind.Utc)), 6);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DriftStatisticsAreZeroOnTheSameDistributionAndLargeOnAShift()
|
||||
{
|
||||
Random rng = new(15);
|
||||
double[] reference = Enumerable.Range(0, 2000).Select(_ => rng.NextDouble()).ToArray();
|
||||
double[] same = Enumerable.Range(0, 2000).Select(_ => rng.NextDouble()).ToArray();
|
||||
double[] shifted = Enumerable.Range(0, 2000).Select(_ => rng.NextDouble() + 0.5).ToArray();
|
||||
|
||||
Assert.True(FeatureRegistry.PopulationStabilityIndex(reference, same) < 0.05);
|
||||
Assert.True(FeatureRegistry.PopulationStabilityIndex(reference, shifted) > 0.5);
|
||||
Assert.True(FeatureRegistry.KolmogorovSmirnov(reference, same) < 0.06);
|
||||
Assert.True(FeatureRegistry.KolmogorovSmirnov(reference, shifted) > 0.4);
|
||||
}
|
||||
}
|
||||
|
||||
public class MetaLabelingTests
|
||||
{
|
||||
[Fact]
|
||||
public void BuildsRowsOnlyWhereThePrimaryTookASide()
|
||||
{
|
||||
DateTime t = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
Random rng = new(16);
|
||||
int n = 600;
|
||||
List<Bar> a = [];
|
||||
List<Bar> b = [];
|
||||
double pa = 100, pb = 50;
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
pa *= 1 + ((rng.NextDouble() - 0.5) * 0.01);
|
||||
pb *= 1 + ((rng.NextDouble() - 0.5) * 0.01);
|
||||
a.Add(new Bar(t.AddMinutes(15 * i), pa, pa, pa, pa, 100, pa, 10, 60));
|
||||
b.Add(new Bar(t.AddMinutes(15 * i), pb, pb, pb, pb, 200, pb, 10, 90));
|
||||
}
|
||||
|
||||
PairSeries series = PairSeries.Build(a, b,
|
||||
Enumerable.Repeat(1.0, n).ToArray(), Enumerable.Repeat(0.0, n).ToArray(),
|
||||
Enumerable.Repeat(30.0, n).ToArray(), zWindow: 30);
|
||||
|
||||
int[] sides = new int[n];
|
||||
for (int i = 150; i < n; i += 7)
|
||||
{
|
||||
sides[i] = i % 14 == 0 ? 1 : -1;
|
||||
}
|
||||
|
||||
MetaDataset dataset = MetaLabeling.BuildDataset(series, sides, FeatureRegistry.Default(),
|
||||
new LabelingParams { MaxHoldingBars = 20 });
|
||||
|
||||
Assert.True(dataset.Count > 40);
|
||||
Assert.All(dataset.Rows, r => Assert.NotEqual(0, r.Side));
|
||||
Assert.All(dataset.Rows, r => Assert.True(r.EndIndex > r.Index));
|
||||
Assert.All(dataset.Rows, r => Assert.Equal(FeatureRegistry.Default().Count, r.Features.Length));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,250 @@
|
||||
using Encelado.Core.Statistics;
|
||||
|
||||
namespace Encelado.Tests;
|
||||
|
||||
/// <summary>The performance formulas, checked against cases whose answer is known.</summary>
|
||||
public class PerformanceTests
|
||||
{
|
||||
[Fact]
|
||||
public void MaxDrawdownIsThePeakToTroughFall()
|
||||
{
|
||||
Assert.Equal(0.5, Performance.MaxDrawdown([100, 120, 60, 90, 130, 117]), 10);
|
||||
Assert.Equal(0, Performance.MaxDrawdown([1, 2, 3]), 10);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void KellyMatchesTheClosedForm()
|
||||
{
|
||||
// 60% winners, wins equal to losses: f* = 0.6 − 0.4/1 = 0.2.
|
||||
Assert.Equal(0.2, Performance.Kelly(0.6, 1.0), 10);
|
||||
|
||||
// Wins twice the losses at 40%: 0.4 − 0.6/2 = 0.1.
|
||||
Assert.Equal(0.1, Performance.Kelly(0.4, 2.0), 10);
|
||||
|
||||
// A losing proposition sizes to nothing, never to a short.
|
||||
Assert.Equal(0, Performance.Kelly(0.3, 1.0), 10);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void VolatilityTargetingIsCappedByLeverage()
|
||||
{
|
||||
// Target 20% on 10% realised wants 2× the capital; the cap allows 1.5×.
|
||||
Assert.Equal(15_000, Performance.VolatilityTargetSize(0.20, 0.10, 10_000, 1.5), 8);
|
||||
Assert.Equal(0, Performance.VolatilityTargetSize(0.20, 0, 10_000, 3), 8);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ProbabilisticSharpeIsOneHalfAtTheBenchmark()
|
||||
{
|
||||
// With normal returns the statistic is centred on the benchmark.
|
||||
Assert.Equal(0.5, Performance.ProbabilisticSharpe(0.1, 0.1, 500, 0, 3), 6);
|
||||
Assert.True(Performance.ProbabilisticSharpe(0.2, 0.0, 500, 0, 3) > 0.99);
|
||||
Assert.True(Performance.ProbabilisticSharpe(-0.2, 0.0, 500, 0, 3) < 0.01);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DeflatedSharpeReducesToProbabilisticWithOneTrial()
|
||||
{
|
||||
double psr = Performance.ProbabilisticSharpe(0.15, 0, 400, 0, 3);
|
||||
double dsr = Performance.DeflatedSharpe(0.15, 400, 0, 3, trials: 1, varianceOfTrialSharpes: 0.01);
|
||||
|
||||
Assert.Equal(psr, dsr, 10);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MoreTrialsDeflateTheSameSharpe()
|
||||
{
|
||||
double few = Performance.DeflatedSharpe(0.15, 400, 0, 3, trials: 5, varianceOfTrialSharpes: 0.005);
|
||||
double many = Performance.DeflatedSharpe(0.15, 400, 0, 3, trials: 300, varianceOfTrialSharpes: 0.005);
|
||||
|
||||
// The same backtest result, chosen out of three hundred instead of five, is
|
||||
// weaker evidence — that is the whole point of the number.
|
||||
Assert.True(many < few);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ExpectedMaximumSharpeGrowsWithTheNumberOfTrials()
|
||||
{
|
||||
double ten = Performance.ExpectedMaximumSharpe(10, 0.01);
|
||||
double thousand = Performance.ExpectedMaximumSharpe(1000, 0.01);
|
||||
|
||||
Assert.True(ten > 0);
|
||||
Assert.True(thousand > ten);
|
||||
Assert.Equal(0, Performance.ExpectedMaximumSharpe(1, 0.01), 10);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BetSizeIsZeroAtACoinFlipAndGrowsWithConfidence()
|
||||
{
|
||||
Assert.Equal(0, Performance.BetSizeFromProbability(0.5), 10);
|
||||
Assert.True(Performance.BetSizeFromProbability(0.7) > 0.2);
|
||||
Assert.True(Performance.BetSizeFromProbability(0.9) > Performance.BetSizeFromProbability(0.7));
|
||||
Assert.True(Performance.BetSizeFromProbability(0.3) < 0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void KurtosisOfANormalSampleIsNearThree()
|
||||
{
|
||||
Random rng = new(5);
|
||||
double[] sample = new double[20_000];
|
||||
for (int i = 0; i < sample.Length; i++)
|
||||
{
|
||||
// Box-Muller.
|
||||
double u1 = 1 - rng.NextDouble();
|
||||
double u2 = rng.NextDouble();
|
||||
sample[i] = Math.Sqrt(-2 * Math.Log(u1)) * Math.Cos(2 * Math.PI * u2);
|
||||
}
|
||||
|
||||
Assert.InRange(Performance.Kurtosis(sample), 2.8, 3.2);
|
||||
Assert.InRange(Performance.Skewness(sample), -0.1, 0.1);
|
||||
}
|
||||
}
|
||||
|
||||
public class JohansenTests
|
||||
{
|
||||
private static (double[] A, double[] B) Cointegrated(int n, double beta, int seed)
|
||||
{
|
||||
Random rng = new(seed);
|
||||
double[] b = new double[n];
|
||||
double[] a = new double[n];
|
||||
b[0] = 10;
|
||||
double wobble = 0;
|
||||
|
||||
for (int i = 1; i < n; i++)
|
||||
{
|
||||
b[i] = b[i - 1] + ((rng.NextDouble() - 0.5) * 0.05);
|
||||
wobble = (wobble * 0.85) + ((rng.NextDouble() - 0.5) * 0.02);
|
||||
a[i] = 0.5 + (beta * b[i]) + wobble;
|
||||
}
|
||||
|
||||
a[0] = 0.5 + (beta * b[0]);
|
||||
return (a, b);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FindsOneRelationshipInACointegratedPair()
|
||||
{
|
||||
(double[] a, double[] b) = Cointegrated(1500, beta: 1.3, seed: 21);
|
||||
|
||||
JohansenResult result = Johansen.Test([a, b], lags: 1);
|
||||
|
||||
Assert.True(result.IsValid);
|
||||
Assert.Equal(1, result.Rank);
|
||||
Assert.True(result.TraceStatistics[0] > result.TraceCritical5[0]);
|
||||
Assert.True(result.TraceStatistics[1] < result.TraceCritical5[1]);
|
||||
|
||||
// The leading vector normalises to [1, −β]: the same hedge ratio Engle-Granger gives.
|
||||
Assert.InRange(result.HedgeRatio, 1.2, 1.4);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FindsNoRelationshipBetweenIndependentWalks()
|
||||
{
|
||||
Random rng = new(22);
|
||||
double[] a = new double[1500];
|
||||
double[] b = new double[1500];
|
||||
|
||||
for (int i = 1; i < a.Length; i++)
|
||||
{
|
||||
a[i] = a[i - 1] + ((rng.NextDouble() - 0.5) * 0.05);
|
||||
b[i] = b[i - 1] + ((rng.NextDouble() - 0.5) * 0.05);
|
||||
}
|
||||
|
||||
JohansenResult result = Johansen.Test([a, b], lags: 1);
|
||||
|
||||
Assert.Equal(0, result.Rank);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void EigenvaluesLieInTheUnitInterval()
|
||||
{
|
||||
(double[] a, double[] b) = Cointegrated(800, beta: 0.8, seed: 23);
|
||||
|
||||
JohansenResult result = Johansen.Test([a, b], lags: 2);
|
||||
|
||||
Assert.All(result.Eigenvalues, static v => Assert.InRange(v, 0, 1));
|
||||
Assert.True(result.Eigenvalues[0] >= result.Eigenvalues[1]);
|
||||
}
|
||||
}
|
||||
|
||||
public class LinearTests
|
||||
{
|
||||
[Fact]
|
||||
public void CholeskyInverseRecoversTheIdentity()
|
||||
{
|
||||
double[,] a = { { 4, 2 }, { 2, 3 } };
|
||||
double[,]? inverse = Linear.InvertSpd(a);
|
||||
|
||||
Assert.NotNull(inverse);
|
||||
double[,] identity = Linear.Multiply(a, inverse!);
|
||||
|
||||
Assert.Equal(1, identity[0, 0], 10);
|
||||
Assert.Equal(1, identity[1, 1], 10);
|
||||
Assert.Equal(0, identity[0, 1], 10);
|
||||
Assert.Equal(0, identity[1, 0], 10);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void JacobiFindsTheEigenvaluesOfASymmetricMatrix()
|
||||
{
|
||||
double[,] a = { { 2, 1 }, { 1, 2 } };
|
||||
(double[] values, double[,] vectors) = Linear.JacobiEigen(a);
|
||||
|
||||
Assert.Equal(3, values[0], 10);
|
||||
Assert.Equal(1, values[1], 10);
|
||||
|
||||
// A·v = λ·v for the leading pair.
|
||||
double[,] v = { { vectors[0, 0] }, { vectors[1, 0] } };
|
||||
double[,] av = Linear.Multiply(a, v);
|
||||
Assert.Equal(3 * v[0, 0], av[0, 0], 10);
|
||||
Assert.Equal(3 * v[1, 0], av[1, 0], 10);
|
||||
}
|
||||
}
|
||||
|
||||
public class KalmanHedgeTests
|
||||
{
|
||||
[Fact]
|
||||
public void ConvergesToAFixedHedgeRatio()
|
||||
{
|
||||
Random rng = new(31);
|
||||
KalmanHedge kalman = new(delta: 1e-4, observationNoise: 1e-3);
|
||||
|
||||
for (int i = 0; i < 2000; i++)
|
||||
{
|
||||
double x = 100 + (rng.NextDouble() * 20);
|
||||
double y = 5 + (1.5 * x) + ((rng.NextDouble() - 0.5) * 0.2);
|
||||
kalman.Update(x, y);
|
||||
}
|
||||
|
||||
Assert.True(kalman.IsReady);
|
||||
Assert.InRange(kalman.Beta, 1.45, 1.55);
|
||||
Assert.InRange(kalman.Alpha, 3, 7);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FollowsAChangeInTheRatioWithinAFewHundredBars()
|
||||
{
|
||||
Random rng = new(32);
|
||||
KalmanHedge kalman = new(delta: 1e-3, observationNoise: 1e-3);
|
||||
|
||||
for (int i = 0; i < 1500; i++)
|
||||
{
|
||||
double x = 100 + (rng.NextDouble() * 20);
|
||||
double beta = i < 750 ? 1.0 : 2.0;
|
||||
double y = (beta * x) + ((rng.NextDouble() - 0.5) * 0.2);
|
||||
kalman.Update(x, y);
|
||||
}
|
||||
|
||||
// A daily OLS refit would still be reporting something near 1.5 here; the filter
|
||||
// has moved to the new regime.
|
||||
Assert.InRange(kalman.Beta, 1.85, 2.15);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RefusesInvalidParameters()
|
||||
{
|
||||
Assert.Throws<ArgumentOutOfRangeException>(() => new KalmanHedge(delta: 0));
|
||||
Assert.Throws<ArgumentOutOfRangeException>(() => new KalmanHedge(delta: 1));
|
||||
Assert.Throws<ArgumentOutOfRangeException>(() => new KalmanHedge(observationNoise: 0));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,382 @@
|
||||
using Encelado.Bot.Configuration;
|
||||
using Encelado.Core.Backtest;
|
||||
using Encelado.Core.Market;
|
||||
using Encelado.Core.Ml;
|
||||
using Encelado.Core.Rl;
|
||||
using Encelado.Core.Strategies;
|
||||
using Encelado.Core.Strategies.Pairs;
|
||||
|
||||
namespace Encelado.Tests;
|
||||
|
||||
internal static class Synthetic
|
||||
{
|
||||
/// <summary>Two cointegrated legs: B a random walk, A = B plus a mean-reverting spread.</summary>
|
||||
public static (List<Bar> A, List<Bar> B) CointegratedPair(int n, int seed, double spreadSigma = 0.01, double theta = 0.1)
|
||||
{
|
||||
Random rng = new(seed);
|
||||
DateTime t = new(2025, 1, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
List<Bar> a = [];
|
||||
List<Bar> b = [];
|
||||
double logB = Math.Log(50_000);
|
||||
double spread = 0;
|
||||
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
logB += (rng.NextDouble() - 0.5) * 0.004;
|
||||
spread += (-theta * spread) + ((rng.NextDouble() - 0.5) * spreadSigma);
|
||||
double pb = Math.Exp(logB);
|
||||
double pa = Math.Exp((0.8 * logB) + spread + 1.0);
|
||||
double vol = 100 + (rng.NextDouble() * 50);
|
||||
a.Add(new Bar(t.AddMinutes(15 * i), pa, pa * 1.001, pa * 0.999, pa, vol, pa, 20, vol * (0.4 + (rng.NextDouble() * 0.2))));
|
||||
b.Add(new Bar(t.AddMinutes(15 * i), pb, pb * 1.001, pb * 0.999, pb, vol * 2, pb, 40, vol * (0.9 + (rng.NextDouble() * 0.2))));
|
||||
}
|
||||
|
||||
return (a, b);
|
||||
}
|
||||
|
||||
public static StrategyParameters Parameters() =>
|
||||
new StrategyParameters().Set("entryZ", 2.0).Set("exitZ", 0.5).Set("stopZ", 4.0).Set("zWindow", 60)
|
||||
.Set("maxPValue", 0.05).Set("minHalfLife", 1).Set("maxHalfLife", 400);
|
||||
|
||||
public static PairBacktestSettings Settings() => new()
|
||||
{
|
||||
CalibrationBars = 300,
|
||||
RecalibrateEveryBars = 96,
|
||||
TimeFrame = TimeFrame.FifteenMinutes,
|
||||
};
|
||||
}
|
||||
|
||||
public class BaselineTests
|
||||
{
|
||||
private static List<Bar> Trend(int n, double drift, int seed)
|
||||
{
|
||||
Random rng = new(seed);
|
||||
DateTime t = new(2025, 1, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
List<Bar> bars = [];
|
||||
double p = 100;
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
p *= 1 + drift + ((rng.NextDouble() - 0.5) * 0.002);
|
||||
bars.Add(new Bar(t.AddMinutes(15 * i), p, p, p, p, 1, p, 1));
|
||||
}
|
||||
|
||||
return bars;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BuyAndHoldEarnsTheDriftMinusTwoSides()
|
||||
{
|
||||
List<Bar> bars = Trend(2000, 0.0005, 1);
|
||||
BaselineReport r = Baseline.BuyAndHold(bars, 0.0005, 35_040);
|
||||
|
||||
double expected = (bars[^1].Close / bars[0].Close) - 1;
|
||||
Assert.InRange(r.Metrics.NetReturn, expected - 0.01, expected);
|
||||
Assert.Equal(1, r.Metrics.Trades);
|
||||
Assert.Contains("comprato", r.Motivazione, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SmaCrossoverStaysFlatInAFallingMarket()
|
||||
{
|
||||
List<Bar> bars = Trend(3000, -0.0005, 2);
|
||||
BaselineReport r = Baseline.SmaCrossover(bars, 20, 50, 0.0005, 35_040);
|
||||
|
||||
// Never long for more than the initial ambiguity, so it loses far less than the market.
|
||||
Assert.True(r.Metrics.NetReturn > -0.2, $"netto {r.Metrics.NetReturn:P2}");
|
||||
Assert.True(r.Metrics.NetReturn > (bars[^1].Close / bars[0].Close) - 1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MetricsAgreeWithTheFormulas()
|
||||
{
|
||||
double[] returns = [0.01, -0.005, 0.02, 0.0, -0.01, 0.015];
|
||||
ReturnMetrics m = ReturnMetrics.From(returns, 252, trades: 3, trials: 4);
|
||||
|
||||
double equity = returns.Aggregate(1.0, static (e, r) => e * (1 + r));
|
||||
Assert.Equal(equity - 1, m.NetReturn, 10);
|
||||
Assert.Equal(4, m.Trials);
|
||||
Assert.Equal(3, m.Trades);
|
||||
Assert.InRange(m.Psr, 0, 1);
|
||||
Assert.InRange(m.Dsr, 0, 1);
|
||||
Assert.True(m.Dsr <= m.Psr, "il DSR non può superare il PSR");
|
||||
}
|
||||
}
|
||||
|
||||
public class PrimaryReplayTests
|
||||
{
|
||||
[Fact]
|
||||
public void ReportsSidesOnlyWhereTheRuleWouldEnter()
|
||||
{
|
||||
(List<Bar> a, List<Bar> b) = Synthetic.CointegratedPair(3000, 3);
|
||||
PairBacktestSettings settings = Synthetic.Settings();
|
||||
|
||||
PrimaryReplayResult r = PrimaryReplay.Run(a, b, Synthetic.Parameters(), settings);
|
||||
|
||||
Assert.Equal(3000, r.Count);
|
||||
Assert.Equal(300, r.Start);
|
||||
Assert.True(r.SignalCount > 10, $"segnali {r.SignalCount}");
|
||||
Assert.All(Enumerable.Range(0, r.Start), i => Assert.Equal(0, r.Sides[i]));
|
||||
Assert.All(Enumerable.Range(0, r.Start), i => Assert.True(double.IsNaN(r.Beta[i])));
|
||||
Assert.True(r.Beta.Skip(r.Start).All(double.IsFinite));
|
||||
|
||||
// Where it says long, the z-score was below −entry; where short, above +entry.
|
||||
for (int i = r.Start; i < r.Count; i++)
|
||||
{
|
||||
if (r.Sides[i] > 0) { Assert.True(r.ZScore[i] <= -2.0); }
|
||||
if (r.Sides[i] < 0) { Assert.True(r.ZScore[i] >= 2.0); }
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TheDatasetBuiltFromItHasOneRowPerSignal()
|
||||
{
|
||||
(List<Bar> a, List<Bar> b) = Synthetic.CointegratedPair(3000, 4);
|
||||
PairBacktestSettings settings = Synthetic.Settings();
|
||||
PrimaryReplayResult r = PrimaryReplay.Run(a, b, Synthetic.Parameters(), settings);
|
||||
PairSeries series = PairSeries.Build(a, b, r.Beta, r.Alpha, r.HalfLife, 60, r.ZScore);
|
||||
|
||||
MetaDataset ds = MetaLabeling.BuildDataset(series, r.Sides, FeatureRegistry.Default(),
|
||||
new LabelingParams { MaxHoldingBars = 48 });
|
||||
|
||||
Assert.True(ds.Count > 0);
|
||||
Assert.True(ds.Count <= r.SignalCount);
|
||||
Assert.InRange(ds.PositiveRate, 0.2, 1.0);
|
||||
}
|
||||
}
|
||||
|
||||
public class PairFeatureBufferTests
|
||||
{
|
||||
[Fact]
|
||||
public void TheLiveRowMatchesTheOfflineMatrix()
|
||||
{
|
||||
(List<Bar> a, List<Bar> b) = Synthetic.CointegratedPair(600, 5);
|
||||
PairCalibration c = new(1.0, 0.8, 0.01, -4, -3.4, 30, true, DateTime.UtcNow, 300);
|
||||
FeatureRegistry registry = FeatureRegistry.Default();
|
||||
|
||||
// Offline: the whole history at once, with a constant calibration.
|
||||
double[] beta = Enumerable.Repeat(0.8, a.Count).ToArray();
|
||||
double[] alpha = Enumerable.Repeat(1.0, a.Count).ToArray();
|
||||
double[] hl = Enumerable.Repeat(30.0, a.Count).ToArray();
|
||||
PairSeries offlineSeries = PairSeries.Build(a, b, beta, alpha, hl, 60);
|
||||
double[][] offline = registry.Compute(offlineSeries);
|
||||
|
||||
// Live: seeded with the first part, then fed bar by bar, with the same z-scores
|
||||
// the strategy would have handed over.
|
||||
PairFeatureBuffer buffer = new(60, 128);
|
||||
buffer.Seed(a.Take(400).ToList(), b.Take(400).ToList(), c, offlineSeries.ZScore.Take(400).ToArray());
|
||||
for (int i = 400; i < a.Count; i++)
|
||||
{
|
||||
buffer.Append(a[i], b[i], c, offlineSeries.ZScore[i]);
|
||||
}
|
||||
|
||||
double[]? live = buffer.Latest(registry, latencyMs: 12);
|
||||
Assert.NotNull(live);
|
||||
|
||||
int latency = registry.Names.ToList().IndexOf("latency_ms");
|
||||
for (int f = 0; f < live.Length; f++)
|
||||
{
|
||||
if (f == latency)
|
||||
{
|
||||
Assert.Equal(12, live[f]);
|
||||
continue;
|
||||
}
|
||||
|
||||
Assert.Equal(offline[^1][f], live[f], 9);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RefusesARowWhileTheHistoryIsShort()
|
||||
{
|
||||
(List<Bar> a, List<Bar> b) = Synthetic.CointegratedPair(50, 6);
|
||||
PairCalibration c = new(1.0, 0.8, 0.01, -4, -3.4, 30, true, DateTime.UtcNow, 300);
|
||||
PairFeatureBuffer buffer = new(60);
|
||||
buffer.Seed(a, b, c, null);
|
||||
|
||||
Assert.Null(buffer.Latest(FeatureRegistry.Default(), 0));
|
||||
}
|
||||
}
|
||||
|
||||
public class DriftMonitorTests
|
||||
{
|
||||
[Fact]
|
||||
public void FlagsAShiftedFeatureAndNotAStableOne()
|
||||
{
|
||||
Random rng = new(7);
|
||||
double[][] reference = Enumerable.Range(0, 1000)
|
||||
.Select(_ => new[] { rng.NextDouble(), rng.NextDouble(), 0.0 })
|
||||
.ToArray();
|
||||
|
||||
DriftMonitor monitor = new(["a", "b", "latency_ms"], reference, window: 200);
|
||||
|
||||
for (int i = 0; i < 200; i++)
|
||||
{
|
||||
monitor.Observe([rng.NextDouble(), rng.NextDouble() + 2, 15]);
|
||||
}
|
||||
|
||||
Assert.True(monitor.Due(50));
|
||||
List<DriftReport> reports = monitor.Check();
|
||||
|
||||
// The constant column is skipped, not flagged.
|
||||
Assert.Equal(2, reports.Count);
|
||||
Assert.False(reports.Single(static r => r.Feature == "a").Alert);
|
||||
Assert.True(reports.Single(static r => r.Feature == "b").Alert);
|
||||
Assert.Equal(0, monitor.SinceCheck);
|
||||
}
|
||||
}
|
||||
|
||||
public class MlpTests
|
||||
{
|
||||
[Fact]
|
||||
public void LearnsASimpleQTarget()
|
||||
{
|
||||
Mlp net = new(2, 32, 3, seed: 1);
|
||||
Random rng = new(2);
|
||||
|
||||
// Action 0 is worth x0, action 1 worth x1, action 2 worth −(x0+x1).
|
||||
for (int step = 0; step < 1500; step++)
|
||||
{
|
||||
double[][] x = new double[32][];
|
||||
int[] actions = new int[32];
|
||||
double[] targets = new double[32];
|
||||
for (int i = 0; i < 32; i++)
|
||||
{
|
||||
double a = rng.NextDouble() * 2 - 1;
|
||||
double b = rng.NextDouble() * 2 - 1;
|
||||
x[i] = [a, b];
|
||||
actions[i] = rng.Next(3);
|
||||
targets[i] = actions[i] switch { 0 => a, 1 => b, _ => -(a + b) };
|
||||
}
|
||||
|
||||
net.TrainBatch(x, actions, targets, 1e-3);
|
||||
}
|
||||
|
||||
double[] q = net.Predict([0.5, -0.3]);
|
||||
Assert.Equal(0.5, q[0], 1);
|
||||
Assert.Equal(-0.3, q[1], 1);
|
||||
Assert.Equal(-0.2, q[2], 1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CloneCopiesWeightsAndNotTraining()
|
||||
{
|
||||
Mlp net = new(2, 8, 2, seed: 3);
|
||||
Mlp copy = net.Clone();
|
||||
Assert.Equal(net.Predict([1, 2]), copy.Predict([1, 2]));
|
||||
|
||||
net.TrainBatch([[1, 2]], [0], [5], 0.01);
|
||||
Assert.NotEqual(net.Predict([1, 2])[0], copy.Predict([1, 2])[0]);
|
||||
|
||||
copy.CopyWeightsFrom(net);
|
||||
Assert.Equal(net.Predict([1, 2]), copy.Predict([1, 2]));
|
||||
}
|
||||
}
|
||||
|
||||
public class PairEnvironmentTests
|
||||
{
|
||||
[Fact]
|
||||
public void PaysTheCostOnEveryChangeOfPosition()
|
||||
{
|
||||
double[][] x = Enumerable.Range(0, 20).Select(_ => new double[] { 0 }).ToArray();
|
||||
double[] ret = Enumerable.Repeat(0.01, 20).ToArray();
|
||||
PairEnvironment env = new(x, ret, roundCost: 0.001);
|
||||
|
||||
double[] s = env.Reset();
|
||||
Assert.Equal(4, s.Length);
|
||||
|
||||
(_, double r1, _, bool c1) = env.Step(1); // open long: cost, then +1%
|
||||
Assert.True(c1);
|
||||
Assert.Equal(0.01 - 0.001, r1, 10);
|
||||
|
||||
(_, double r2, _, bool c2) = env.Step(1); // hold: +1%, no cost
|
||||
Assert.False(c2);
|
||||
Assert.Equal(0.01, r2, 10);
|
||||
|
||||
(_, double r3, _, _) = env.Step(2); // reverse: two costs, −1%
|
||||
Assert.Equal(-0.01 - 0.002, r3, 10);
|
||||
|
||||
(double[] s4, double r4, _, _) = env.Step(0); // flat: one cost, nothing earned
|
||||
Assert.Equal(-0.001, r4, 10);
|
||||
Assert.Equal(0, s4[1]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AGreedyAgentOnAKnownEdgeBeatsFlat()
|
||||
{
|
||||
// The feature is the sign of next bar's spread return: a trivially learnable edge.
|
||||
Random rng = new(11);
|
||||
int n = 4000;
|
||||
double[] ret = new double[n];
|
||||
double[][] x = new double[n][];
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
ret[i] = (rng.NextDouble() - 0.5) * 0.02;
|
||||
x[i] = [Math.Sign(ret[i])];
|
||||
}
|
||||
|
||||
PairEnvironment train = new(x[..3000], ret[..3000], roundCost: 0.0005);
|
||||
PairEnvironment test = new(x[3000..], ret[3000..], roundCost: 0.0005);
|
||||
|
||||
DqnAgent agent = new(train.StateSize, new DqnParams { EpsilonSteps = 4000, WarmupSteps = 200, Hidden = 32 }, seed: 1);
|
||||
agent.Train(train, episodes: 2);
|
||||
(double[] rewards, int trades) = agent.Evaluate(test);
|
||||
|
||||
Assert.True(rewards.Sum() > 0, $"totale {rewards.Sum():F4} su {trades} cambi");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SeedsRunInParallelOnTheirOwnCursors()
|
||||
{
|
||||
Random rng = new(12);
|
||||
int n = 1500;
|
||||
double[] ret = new double[n];
|
||||
double[][] x = new double[n][];
|
||||
for (int i = 0; i < n; i++)
|
||||
{
|
||||
ret[i] = (rng.NextDouble() - 0.5) * 0.02;
|
||||
x[i] = [Math.Sign(ret[i])];
|
||||
}
|
||||
|
||||
PairEnvironment train = new(x[..1000], ret[..1000], roundCost: 0.0005);
|
||||
PairEnvironment test = new(x[1000..], ret[1000..], roundCost: 0.0005);
|
||||
DqnParams p = new() { EpsilonSteps = 1000, WarmupSteps = 100, Hidden = 16, ReplaySize = 2000 };
|
||||
|
||||
List<RlSeedResult> results = Experiment.Run(train, test, p, seeds: 6, episodes: 1, periodsPerYear: 35_040);
|
||||
|
||||
Assert.Equal(6, results.Count);
|
||||
Assert.All(results, r => Assert.True(double.IsFinite(r.Test.NetReturn)));
|
||||
Assert.All(results, r => Assert.True(r.Trades <= test.Length));
|
||||
Assert.Equal(0, train.Time);
|
||||
}
|
||||
}
|
||||
|
||||
public class MlConfigTests
|
||||
{
|
||||
[Fact]
|
||||
public void TheDefaultsLoadAndValidate()
|
||||
{
|
||||
string path = Path.Combine(Path.GetTempPath(), $"encelado-ml-{Guid.NewGuid():N}.json");
|
||||
File.WriteAllText(path, ConfigDefaults.Json);
|
||||
try
|
||||
{
|
||||
BotConfig config = ConfigLoader.Load(path, out List<string> warnings);
|
||||
Assert.DoesNotContain(warnings, static w => w.Contains("ml.", StringComparison.Ordinal));
|
||||
Assert.True(config.Ml.Enabled);
|
||||
Assert.Equal(0.55, config.Ml.MinProbability);
|
||||
Assert.True(config.Ml.SizeByProbability);
|
||||
Assert.Equal(3, config.Ml.DriftAlertsToSuspend);
|
||||
config.Ml.Validate();
|
||||
}
|
||||
finally
|
||||
{
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RefusesAThresholdBelowACoinFlip()
|
||||
{
|
||||
MlOptions o = new() { MinProbability = 0.4 };
|
||||
Assert.Throws<InvalidOperationException>(o.Validate);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,168 @@
|
||||
using Encelado.Core.Journal;
|
||||
using Encelado.Core.Market;
|
||||
using Encelado.Storage;
|
||||
|
||||
namespace Encelado.Tests;
|
||||
|
||||
/// <summary>The local database: every table round-trips, and a failure never escapes.</summary>
|
||||
public class StorageTests
|
||||
{
|
||||
private static StorageDb Open(out string path)
|
||||
{
|
||||
path = Path.Combine(Path.GetTempPath(), $"encelado-db-{Guid.NewGuid():N}", "encelado.db");
|
||||
return new StorageDb(path);
|
||||
}
|
||||
|
||||
private static void Cleanup(string path)
|
||||
{
|
||||
try
|
||||
{
|
||||
Directory.Delete(Path.GetDirectoryName(path)!, recursive: true);
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
// Windows may still hold the WAL for a moment; a leftover temp folder is not a failure.
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CreatesTheFileAndItsFolder()
|
||||
{
|
||||
using StorageDb db = Open(out string path);
|
||||
|
||||
Assert.True(File.Exists(path));
|
||||
Assert.Equal(0, db.Count("bars"));
|
||||
|
||||
db.Dispose();
|
||||
Cleanup(path);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BarsRoundTripAndReplaceOnTheSameInstant()
|
||||
{
|
||||
using StorageDb db = Open(out string path);
|
||||
DateTime t = new(2026, 9, 8, 10, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
List<Bar> bars =
|
||||
[
|
||||
new(t, 1, 2, 0.5, 1.5, 100, 1.2, 10, 60),
|
||||
new(t.AddMinutes(15), 1.5, 2.5, 1, 2, 200, 1.7, 20, 120),
|
||||
];
|
||||
|
||||
Assert.Equal(2, db.UpsertBars("ETHUSDT", "15m", bars));
|
||||
|
||||
// The same instant again with a different close: one row, the newer value.
|
||||
db.UpsertBars("ETHUSDT", "15m", [new Bar(t, 1, 2, 0.5, 1.75, 100, 1.2, 10, 60)]);
|
||||
|
||||
List<Bar> loaded = db.LoadBars("ETHUSDT", "15m");
|
||||
|
||||
Assert.Equal(2, loaded.Count);
|
||||
Assert.Equal(1.75, loaded[0].Close, 8);
|
||||
Assert.Equal(60, loaded[0].TakerBuyVolume, 8);
|
||||
Assert.Equal(DateTimeKind.Utc, loaded[0].TimeUtc.Kind);
|
||||
Assert.Equal(t, loaded[0].TimeUtc);
|
||||
|
||||
db.Dispose();
|
||||
Cleanup(path);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void JournalRowsLandInTheirTables()
|
||||
{
|
||||
using StorageDb db = Open(out string path);
|
||||
DateTime now = DateTime.UtcNow;
|
||||
|
||||
db.Decision(new DecisionRow(now, 42, "ETHUSDT/BTCUSDT", "ETHUSDT", "BTCUSDT", true,
|
||||
3000, 60_000, 0.01, -2.6, 1.15, 0.3, 0.02, -3.9, 38, true, 0, 0, 0, 0, "EnterLongSpread",
|
||||
0.0001, 0.0001, 0.0001, 0.0001, 5000, false, 0.62, "spread basso: z=-2.60"));
|
||||
|
||||
db.Execution(new ExecutionRow(now, 42, "ETHUSDT/BTCUSDT", "order", true, "None",
|
||||
"BUY", 750, 0.25, 3000, "SELL", 750, 0.0125, 60_000, 0.0001, 5000, 4000, 1500, 1,
|
||||
"1", "2", string.Empty, 41.5, "ingresso eseguito"));
|
||||
|
||||
db.Trade(new TradeRow(now, "entry", "ETHUSDT", "BUY", 0.25, 3000, "1", null, null, 5000, null,
|
||||
"spread basso"));
|
||||
|
||||
Assert.Equal(1, db.Count("decisions"));
|
||||
Assert.Equal(1, db.Count("executions"));
|
||||
Assert.Equal(1, db.Count("trades"));
|
||||
|
||||
db.Dispose();
|
||||
Cleanup(path);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DatasetsAndModelsRoundTrip()
|
||||
{
|
||||
using StorageDb db = Open(out string path);
|
||||
DateTime t = new(2026, 9, 8, 0, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
List<StorageDb.DatasetRowRecord> rows =
|
||||
[
|
||||
new(t, t.AddHours(1), 1, 1, 0.004, 0.8, [0.1, -0.2, 3.5]),
|
||||
new(t.AddMinutes(15), t.AddHours(2), -1, -1, -0.002, 0.5, [0.2, -0.1, 2.5]),
|
||||
];
|
||||
|
||||
long datasetId = db.SaveDataset("ETHUSDT/BTCUSDT", "15m", ["r1", "r2", "z"], "triple-barrier", rows, "prova");
|
||||
Assert.True(datasetId > 0);
|
||||
|
||||
var loaded = db.LoadDataset(datasetId);
|
||||
Assert.NotNull(loaded);
|
||||
Assert.Equal(["r1", "r2", "z"], loaded!.Value.FeatureNames);
|
||||
Assert.Equal(2, loaded.Value.Rows.Count);
|
||||
Assert.Equal(3.5, loaded.Value.Rows[0].Features[2], 10);
|
||||
Assert.Equal(-1, loaded.Value.Rows[1].Label);
|
||||
|
||||
long modelId = db.SaveModel("ETHUSDT/BTCUSDT", "gbdt", datasetId, "{}", "{}", ["r1", "r2", "z"],
|
||||
[1, 2, 3, 4], "prova");
|
||||
|
||||
Assert.Null(db.LoadChampion("ETHUSDT/BTCUSDT"));
|
||||
|
||||
db.SetChampion("ETHUSDT/BTCUSDT", modelId, "primo modello");
|
||||
StorageDb.ModelRecord? champion = db.LoadChampion("ETHUSDT/BTCUSDT");
|
||||
|
||||
Assert.NotNull(champion);
|
||||
Assert.Equal(modelId, champion!.Id);
|
||||
Assert.Equal([1, 2, 3, 4], champion.Blob);
|
||||
Assert.Equal("gbdt", champion.Kind);
|
||||
|
||||
db.Dispose();
|
||||
Cleanup(path);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FeatureBlobsPackAndUnpackExactly()
|
||||
{
|
||||
double[] values = [0, -1.5, double.MaxValue, 1e-300, Math.PI];
|
||||
|
||||
Assert.Equal(values, StorageDb.Unpack(StorageDb.Pack(values)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ARejectedTableNameIsRefusedNotInterpolated()
|
||||
{
|
||||
using StorageDb db = Open(out string path);
|
||||
|
||||
Assert.Throws<ArgumentException>(() => db.Count("bars; DROP TABLE bars"));
|
||||
|
||||
db.Dispose();
|
||||
Cleanup(path);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AJournalFailureIsReportedNotThrown()
|
||||
{
|
||||
using StorageDb db = Open(out string path);
|
||||
string? reported = null;
|
||||
db.OnError = (what, _) => reported = what;
|
||||
|
||||
// Dispose the connection underneath, then write: the row is lost, the caller is
|
||||
// not. A journal that could take the trading path down is worse than no journal.
|
||||
db.Dispose();
|
||||
|
||||
db.Trade(new TradeRow(DateTime.UtcNow, "fill", "ETHUSDT", "BUY", 1, 1, null, null, null, null, null, "x"));
|
||||
|
||||
Assert.NotNull(reported);
|
||||
Cleanup(path);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,148 @@
|
||||
using System.Text;
|
||||
using Encelado.Binance;
|
||||
using Encelado.Binance.Streaming;
|
||||
using Encelado.Core.Market;
|
||||
|
||||
namespace Encelado.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The market-data decoder, fed verbatim Binance frames.
|
||||
/// <para>
|
||||
/// This suite exists because of a production run in which the bot received 2.3 million
|
||||
/// book updates and not one closed bar: the kline fields live inside a nested object,
|
||||
/// and the decoder skipped it with everything else it did not want. The bot kept
|
||||
/// deciding only because the REST backstop refetched the bars the stream never delivered.
|
||||
/// Every frame here is the shape Binance actually sends, wrapper included.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public class StreamDecodingTests
|
||||
{
|
||||
private static MarketDataStream Stream(params string[] symbols) => new(
|
||||
new BinanceOptions { ApiKey = new string('k', 64), ApiSecret = new string('s', 64) },
|
||||
symbols,
|
||||
TimeFrame.FifteenMinutes);
|
||||
|
||||
private static void Feed(MarketDataStream stream, string frame) =>
|
||||
stream.Feed(Encoding.UTF8.GetBytes(frame));
|
||||
|
||||
private const string ClosedKline = """
|
||||
{"stream":"ethusdt@kline_15m","data":{"e":"kline","E":1756379700123,"s":"ETHUSDT","k":{"t":1756378800000,"T":1756379699999,"s":"ETHUSDT","i":"15m","f":100,"L":200,"o":"2500.10","c":"2504.64","h":"2506.00","l":"2498.50","v":"1234.567","n":4321,"x":true,"q":"3090000.5","V":"700.25","Q":"1752000.1","B":"0"}}}
|
||||
""";
|
||||
|
||||
[Fact]
|
||||
public void AClosedKlineFrameProducesOneBar()
|
||||
{
|
||||
MarketDataStream stream = Stream("ETHUSDT", "BTCUSDT");
|
||||
List<(int Id, string Symbol, Bar Bar)> bars = [];
|
||||
stream.OnBar = (int id, string symbol, in Bar bar) => bars.Add((id, symbol, bar));
|
||||
|
||||
Feed(stream, ClosedKline);
|
||||
|
||||
(int id, string symbol, Bar bar) = Assert.Single(bars);
|
||||
Assert.Equal(0, id);
|
||||
Assert.Equal("ETHUSDT", symbol);
|
||||
Assert.Equal(2500.10, bar.Open, 8);
|
||||
Assert.Equal(2506.00, bar.High, 8);
|
||||
Assert.Equal(2498.50, bar.Low, 8);
|
||||
Assert.Equal(2504.64, bar.Close, 8);
|
||||
Assert.Equal(1234.567, bar.Volume, 8);
|
||||
Assert.Equal(700.25, bar.TakerBuyVolume, 8);
|
||||
Assert.Equal(4321, bar.TradeCount);
|
||||
Assert.Equal(DateTimeOffset.FromUnixTimeMilliseconds(1756378800000).UtcDateTime, bar.TimeUtc);
|
||||
Assert.Equal(3090000.5 / 1234.567, bar.Vwap, 8);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AFormingKlineProducesNothing()
|
||||
{
|
||||
MarketDataStream stream = Stream("ETHUSDT");
|
||||
int bars = 0;
|
||||
stream.OnBar = (int _, string _, in Bar _) => bars++;
|
||||
|
||||
// Binance re-sends the forming bar several times a second; acting on any of
|
||||
// those would be deciding on a partial close.
|
||||
Feed(stream, ClosedKline.Replace("\"x\":true", "\"x\":false", StringComparison.Ordinal));
|
||||
|
||||
Assert.Equal(0, bars);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FieldOrderInsideTheKlineDoesNotMatter()
|
||||
{
|
||||
MarketDataStream stream = Stream("ETHUSDT");
|
||||
Bar? seen = null;
|
||||
stream.OnBar = (int _, string _, in Bar bar) => seen = bar;
|
||||
|
||||
// The close flag first, the symbol last: a decoder that depends on ordering
|
||||
// would miss one or the other.
|
||||
Feed(stream, """
|
||||
{"stream":"ethusdt@kline_15m","data":{"k":{"x":true,"c":"10.5","o":"10.0","h":"11.0","l":"9.5","v":"5","q":"52","n":3,"V":"2","t":1756378800000},"e":"kline","s":"ETHUSDT"}}
|
||||
""");
|
||||
|
||||
Assert.NotNull(seen);
|
||||
Assert.Equal(10.5, seen!.Value.Close, 8);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AnUnsubscribedSymbolIsIgnored()
|
||||
{
|
||||
MarketDataStream stream = Stream("BTCUSDT");
|
||||
int bars = 0;
|
||||
stream.OnBar = (int _, string _, in Bar _) => bars++;
|
||||
|
||||
Feed(stream, ClosedKline);
|
||||
|
||||
Assert.Equal(0, bars);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ABookTickerFrameProducesAQuote()
|
||||
{
|
||||
MarketDataStream stream = Stream("BTCUSDT");
|
||||
Quote? seen = null;
|
||||
stream.OnQuote = (int _, string _, in Quote quote) => seen = quote;
|
||||
|
||||
Feed(stream, """
|
||||
{"stream":"btcusdt@bookTicker","data":{"e":"bookTicker","u":400900217,"E":1756379700123,"T":1756379700120,"s":"BTCUSDT","b":"79618.90","B":"1.250","a":"79627.60","A":"0.800"}}
|
||||
""");
|
||||
|
||||
Assert.NotNull(seen);
|
||||
Assert.Equal(79618.90, seen!.Value.BidPrice, 8);
|
||||
Assert.Equal(79627.60, seen.Value.AskPrice, 8);
|
||||
Assert.Equal(1.25, seen.Value.BidSize, 8);
|
||||
Assert.True(seen.Value.IsValid);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AMarkPriceFrameCarriesTheFundingRate()
|
||||
{
|
||||
MarketDataStream stream = Stream("BTCUSDT");
|
||||
(double Mark, double Rate, DateTime Next)? seen = null;
|
||||
stream.OnFunding = (_, _, mark, rate, next) => seen = (mark, rate, next);
|
||||
|
||||
Feed(stream, """
|
||||
{"stream":"btcusdt@markPrice@1s","data":{"e":"markPriceUpdate","E":1756379700123,"s":"BTCUSDT","p":"79620.12","i":"79615.00","P":"79600.00","r":"0.00010000","T":1756396800000}}
|
||||
""");
|
||||
|
||||
Assert.NotNull(seen);
|
||||
Assert.Equal(79620.12, seen!.Value.Mark, 8);
|
||||
Assert.Equal(0.0001, seen.Value.Rate, 10);
|
||||
Assert.Equal(DateTimeOffset.FromUnixTimeMilliseconds(1756396800000).UtcDateTime, seen.Value.Next);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AMalformedFrameDoesNotThrow()
|
||||
{
|
||||
MarketDataStream stream = Stream("BTCUSDT");
|
||||
int bars = 0;
|
||||
stream.OnBar = (int _, string _, in Bar _) => bars++;
|
||||
|
||||
// A frame the channel would otherwise choke on must cost one message, not the
|
||||
// whole connection.
|
||||
Feed(stream, "{\"stream\":\"x\",\"data\":{\"e\":\"kline\",\"s\":\"BTCUSDT\",\"k\":{\"x\":true,\"c\":\"1\"");
|
||||
Feed(stream, "not json at all");
|
||||
Feed(stream, "{}");
|
||||
|
||||
Assert.Equal(0, bars);
|
||||
}
|
||||
}
|
||||
@@ -24,6 +24,7 @@
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\src\Encelado.Core\Encelado.Core.csproj" />
|
||||
<ProjectReference Include="..\..\src\Encelado.Storage\Encelado.Storage.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
||||
@@ -59,6 +59,39 @@ public sealed record Options
|
||||
|
||||
public double[]? GridMaxP { get; init; }
|
||||
|
||||
// ---- research commands ------------------------------------------------
|
||||
|
||||
/// <summary>Single symbol for <c>baseline</c>; defaults to <see cref="SymbolA"/>.</summary>
|
||||
public string Symbol { get; init; } = string.Empty;
|
||||
|
||||
/// <summary>SQLite path; empty means the bot's own database in %ProgramData%.</summary>
|
||||
public string DatabasePath { get; init; } = string.Empty;
|
||||
|
||||
public bool NoDb { get; init; }
|
||||
|
||||
/// <summary>Where the CSV tables go; empty means a "ricerca" folder beside the tool.</summary>
|
||||
public string OutputDirectory { get; init; } = string.Empty;
|
||||
|
||||
public int SmaFast { get; init; }
|
||||
|
||||
public int SmaSlow { get; init; }
|
||||
|
||||
public double ProfitMultiple { get; init; } = 2.0;
|
||||
|
||||
public double StopMultiple { get; init; } = 1.0;
|
||||
|
||||
/// <summary>Vertical barrier in bars; 0 means one day at the timeframe.</summary>
|
||||
public int LabelMaxBars { get; init; }
|
||||
|
||||
/// <summary>How many GBDT configurations to try; 0 means the whole small grid.</summary>
|
||||
public int Trials { get; init; }
|
||||
|
||||
public bool Promote { get; init; }
|
||||
|
||||
public int Seeds { get; init; } = 15;
|
||||
|
||||
public int Episodes { get; init; } = 2;
|
||||
|
||||
public string TimeFrameText => TimeFrame.ToBinance();
|
||||
|
||||
public PairBacktestSettings Settings() => new()
|
||||
@@ -99,6 +132,8 @@ public sealed record Options
|
||||
{
|
||||
case "fit-once": o = o with { FitOnce = true }; continue;
|
||||
case "no-coint": o = o with { RequireCointegration = false }; continue;
|
||||
case "no-db": o = o with { NoDb = true }; continue;
|
||||
case "promote": o = o with { Promote = true }; continue;
|
||||
}
|
||||
|
||||
if (i + 1 >= args.Length)
|
||||
@@ -133,11 +168,23 @@ public sealed record Options
|
||||
"grid-window" => o with { GridWindow = Integers(value) },
|
||||
"grid-maxbars" => o with { GridMaxBars = Integers(value) },
|
||||
"grid-maxp" => o with { GridMaxP = Numbers(value) },
|
||||
"symbol" => o with { Symbol = value.ToUpperInvariant() },
|
||||
"db" => o with { DatabasePath = value },
|
||||
"out" => o with { OutputDirectory = value },
|
||||
"fast" => o with { SmaFast = (int)Number(value) },
|
||||
"slow" => o with { SmaSlow = (int)Number(value) },
|
||||
"pt" => o with { ProfitMultiple = Number(value) },
|
||||
"sl" => o with { StopMultiple = Number(value) },
|
||||
"label-max" => o with { LabelMaxBars = (int)Number(value) },
|
||||
"trials" => o with { Trials = (int)Number(value) },
|
||||
"seeds" => o with { Seeds = (int)Number(value) },
|
||||
"episodes" => o with { Episodes = (int)Number(value) },
|
||||
_ => throw new ArgumentException($"opzione sconosciuta --{flag}"),
|
||||
};
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(o.DataDirectory))
|
||||
bool needsData = args.Length == 0 || !args[0].Equals("status", StringComparison.OrdinalIgnoreCase);
|
||||
if (needsData && string.IsNullOrWhiteSpace(o.DataDirectory))
|
||||
{
|
||||
throw new ArgumentException("--data è obbligatoria: indica la cartella con i file CSV.");
|
||||
}
|
||||
|
||||
@@ -48,6 +48,14 @@ public static class Program
|
||||
"pairs" => CommandPairs(options),
|
||||
"explore" => CommandExplore(options),
|
||||
"basket" => CommandBasket(options),
|
||||
"inspect" => Research.Inspect(options),
|
||||
"import" => Research.Import(options),
|
||||
"baseline" => Research.BaselineCommand(options),
|
||||
"cointegration" => Research.Cointegration(options),
|
||||
"dataset" => Research.Dataset(options),
|
||||
"train" => Research.Train(options),
|
||||
"rl" => Research.Rl(options),
|
||||
"status" => Research.Status(options),
|
||||
_ => Usage(),
|
||||
};
|
||||
}
|
||||
@@ -82,6 +90,26 @@ public static class Program
|
||||
|
||||
La classifica dello sweep è ordinata sulla fetta di VERIFICA. La fetta di
|
||||
CONFERMA non entra mai nella scelta: serve solo a dire quanto è sopravvissuto.
|
||||
|
||||
Ricerca (le fasi della guida; ogni tabella è CSV ';' con colonna motivazione)
|
||||
backtest inspect --data <cartella> FASE 0: qualità dei file a un minuto
|
||||
backtest import --data <cartella> [--tf 15m] barre canoniche nel database
|
||||
backtest baseline --symbol BTCUSDT [--fast 20 --slow 50] FASE 1: buy&hold e medie mobili
|
||||
backtest cointegration --a ETHUSDT --b BTCUSDT FASE 2: Engle-Granger, Johansen, Kalman, break-even
|
||||
backtest dataset --a ETHUSDT --b BTCUSDT FASE 3: dataset di meta-labeling (triple barrier)
|
||||
backtest train --a ETHUSDT --b BTCUSDT [--promote] FASE 3: GBDT + CPCV, PBO, DSR, campione
|
||||
backtest rl --a ETHUSDT --b BTCUSDT [--seeds 15 --episodes 2] FASE 4: DQN multi-seed
|
||||
backtest status conteggi delle tabelle del database
|
||||
|
||||
--db <file> database SQLite (default: quello del bot in %ProgramData%\Encelado)
|
||||
--no-db non toccare il database
|
||||
--out <cartella> dove scrivere le tabelle (default: cartella 'ricerca' accanto allo strumento)
|
||||
--pt, --sl barriere di profitto e stop in volatilità locali (default 2 e 1)
|
||||
--label-max <barre> barriera verticale (default: un giorno)
|
||||
--trials <n> quante configurazioni GBDT provare (default: tutte e sei)
|
||||
|
||||
Un modello diventa campione solo con --promote e solo se supera i criteri della
|
||||
Fase 3 (PBO < 0.5, DSR > 0.95 contando tutte le prove, Sharpe fuori campione > 0).
|
||||
""");
|
||||
return 2;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,907 @@
|
||||
using System.Globalization;
|
||||
using System.Text;
|
||||
using Encelado.Core.Backtest;
|
||||
using Encelado.Core.Market;
|
||||
using Encelado.Core.Ml;
|
||||
using Encelado.Core.Rl;
|
||||
using Encelado.Core.Statistics;
|
||||
using Encelado.Core.Strategies;
|
||||
using Encelado.Storage;
|
||||
|
||||
namespace Encelado.Backtest;
|
||||
|
||||
/// <summary>
|
||||
/// The research pipeline the strategy note lays out, one command per phase: data
|
||||
/// quality, baselines, the cointegration study, the meta-model, reinforcement
|
||||
/// learning. Every table it writes is <c>;</c>-separated with a <c>motivazione</c>
|
||||
/// column, and everything worth keeping also goes into the SQLite database the bot
|
||||
/// reads its champion from.
|
||||
/// </summary>
|
||||
public static class Research
|
||||
{
|
||||
private const char Sep = ';';
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// FASE 0 — data quality
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/// <summary>
|
||||
/// Reads every CSV in the data folder at minute resolution and reports what is
|
||||
/// wrong with it: duplicates, gaps, impossible prices, impossible jumps.
|
||||
/// </summary>
|
||||
public static int Inspect(Options o)
|
||||
{
|
||||
string[] files = Directory.GetFiles(o.DataDirectory, "*.csv");
|
||||
if (files.Length == 0)
|
||||
{
|
||||
Console.Error.WriteLine($"nessun CSV in {o.DataDirectory}");
|
||||
return 1;
|
||||
}
|
||||
|
||||
using StorageDb? db = OpenDb(o);
|
||||
string outPath = OutPath(o, "data_quality.csv");
|
||||
using StreamWriter w = Table(outPath,
|
||||
"file;simbolo;righe;prima;ultima;duplicati;buchi;minuti_mancanti;prezzi_errati;salti_impossibili;copertura;motivazione");
|
||||
|
||||
DateTime run = DateTime.UtcNow;
|
||||
Console.WriteLine();
|
||||
Console.WriteLine(" FASE 0 — qualità dei dati (barre da un minuto)");
|
||||
Console.WriteLine();
|
||||
|
||||
foreach (string file in files.Order(StringComparer.OrdinalIgnoreCase))
|
||||
{
|
||||
string symbol = Path.GetFileNameWithoutExtension(file).ToUpperInvariant();
|
||||
IReadOnlyList<Bar> bars;
|
||||
try
|
||||
{
|
||||
bars = CsvBarSource.Load(file);
|
||||
}
|
||||
catch (Exception ex) when (ex is IOException or InvalidDataException)
|
||||
{
|
||||
Console.WriteLine($" {symbol,-10} illeggibile: {ex.Message}");
|
||||
continue;
|
||||
}
|
||||
|
||||
long duplicates = 0, gaps = 0, missingMinutes = 0, badPrices = 0, jumps = 0;
|
||||
DateTime previous = DateTime.MinValue;
|
||||
double previousClose = 0;
|
||||
|
||||
foreach (Bar b in bars)
|
||||
{
|
||||
if (b.Open <= 0 || b.High <= 0 || b.Low <= 0 || b.Close <= 0 || b.High < b.Low || b.Volume < 0)
|
||||
{
|
||||
badPrices++;
|
||||
}
|
||||
|
||||
if (previous != DateTime.MinValue)
|
||||
{
|
||||
double minutes = (b.TimeUtc - previous).TotalMinutes;
|
||||
if (minutes <= 0)
|
||||
{
|
||||
duplicates++;
|
||||
}
|
||||
else if (minutes > 1.5)
|
||||
{
|
||||
gaps++;
|
||||
missingMinutes += (long)Math.Round(minutes) - 1;
|
||||
}
|
||||
|
||||
if (previousClose > 0 && b.Close > 0 && Math.Abs((b.Close / previousClose) - 1) > 0.20)
|
||||
{
|
||||
jumps++;
|
||||
}
|
||||
}
|
||||
|
||||
previous = b.TimeUtc;
|
||||
previousClose = b.Close;
|
||||
}
|
||||
|
||||
DateTime first = bars[0].TimeUtc;
|
||||
DateTime last = bars[^1].TimeUtc;
|
||||
double expected = Math.Max(1, (last - first).TotalMinutes + 1);
|
||||
double coverage = Math.Min(1, bars.Count / expected);
|
||||
|
||||
string motivazione = Motivate(duplicates, gaps, missingMinutes, badPrices, jumps, coverage);
|
||||
|
||||
Row(w, Path.GetFileName(file), symbol, bars.Count, Stamp(first), Stamp(last), duplicates, gaps,
|
||||
missingMinutes, badPrices, jumps, coverage, motivazione);
|
||||
|
||||
db?.InsertDataQuality(run, Path.GetFileName(file), symbol, bars.Count, first, last, duplicates, gaps,
|
||||
missingMinutes, badPrices, motivazione);
|
||||
|
||||
Console.WriteLine(string.Create(CultureInfo.InvariantCulture,
|
||||
$" {symbol,-10} {bars.Count,10:N0} righe {first:yyyy-MM-dd} → {last:yyyy-MM-dd} " +
|
||||
$"copertura {coverage,7:P2} duplicati {duplicates,4} buchi {gaps,5} ({missingMinutes:N0} min) " +
|
||||
$"prezzi errati {badPrices} salti>20% {jumps}"));
|
||||
Console.WriteLine($" {motivazione}");
|
||||
}
|
||||
|
||||
Console.WriteLine();
|
||||
Console.WriteLine($" tabella: {outPath}");
|
||||
return 0;
|
||||
|
||||
static string Motivate(long dup, long gaps, long missing, long bad, long jumps, double coverage)
|
||||
{
|
||||
List<string> parts = [];
|
||||
parts.Add(coverage >= 0.995 ? "copertura praticamente completa" :
|
||||
coverage >= 0.98 ? "copertura buona: i buchi sono manutenzioni dell'exchange" :
|
||||
"copertura insufficiente: verificare il file prima di usarlo");
|
||||
if (dup > 0) { parts.Add($"{dup} timestamp ripetuti, tenuto l'ultimo"); }
|
||||
if (bad > 0) { parts.Add($"{bad} righe con prezzi o volumi impossibili, da scartare"); }
|
||||
if (jumps > 0) { parts.Add($"{jumps} salti oltre il 20% in un minuto, da controllare a mano"); }
|
||||
if (gaps > 0 && missing > 0) { parts.Add($"{gaps} buchi per {missing} minuti: l'allineamento per timestamp li salta"); }
|
||||
return string.Join("; ", parts);
|
||||
}
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Canonical import
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/// <summary>Folds the minute files to the working timeframe and stores them as canonical bars.</summary>
|
||||
public static int Import(Options o)
|
||||
{
|
||||
using StorageDb? db = OpenDb(o);
|
||||
if (db is null)
|
||||
{
|
||||
Console.Error.WriteLine("import ha bisogno del database: passa --db oppure lascia il predefinito");
|
||||
return 1;
|
||||
}
|
||||
|
||||
string[] files = Directory.GetFiles(o.DataDirectory, "*.csv");
|
||||
TimeSpan bucket = TimeSpan.FromSeconds(o.TimeFrame.Seconds());
|
||||
|
||||
Console.WriteLine();
|
||||
Console.WriteLine($" importo le barre {o.TimeFrameText} in {db.Path}");
|
||||
Console.WriteLine();
|
||||
|
||||
foreach (string file in files.Order(StringComparer.OrdinalIgnoreCase))
|
||||
{
|
||||
string symbol = Path.GetFileNameWithoutExtension(file).ToUpperInvariant();
|
||||
IReadOnlyList<Bar> bars = CsvBarSource.LoadAggregated(file, bucket);
|
||||
int written = db.UpsertBars(symbol, o.TimeFrameText, bars);
|
||||
Console.WriteLine(string.Create(CultureInfo.InvariantCulture,
|
||||
$" {symbol,-10} {bars.Count,8:N0} barre {o.TimeFrameText}, {written,8:N0} scritte; " +
|
||||
$"in tabella ora {db.CountBars(symbol, o.TimeFrameText):N0}"));
|
||||
}
|
||||
|
||||
Console.WriteLine();
|
||||
return 0;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// FASE 1 — baselines
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
public static int BaselineCommand(Options o)
|
||||
{
|
||||
string symbol = string.IsNullOrEmpty(o.Symbol) ? o.SymbolA : o.Symbol;
|
||||
string path = Path.Combine(o.DataDirectory, symbol + ".csv");
|
||||
if (!File.Exists(path))
|
||||
{
|
||||
Console.Error.WriteLine($"file dati non trovato: {path}");
|
||||
return 1;
|
||||
}
|
||||
|
||||
TimeSpan bucket = TimeSpan.FromSeconds(o.TimeFrame.Seconds());
|
||||
IReadOnlyList<Bar> bars = CsvBarSource.LoadAggregated(path, bucket);
|
||||
double perYear = 365.0 * 24 * 3600 / o.TimeFrame.Seconds();
|
||||
double costPerSide = (o.FeeBps + o.SlippageBps) / 10_000.0;
|
||||
|
||||
// Three consecutive slices, like everywhere else: the last one is the only one
|
||||
// that says anything about a rule with a parameter.
|
||||
int n = bars.Count;
|
||||
(int tr, int va, int co) = o.SplitWeights;
|
||||
int total = Math.Max(1, tr + va + co);
|
||||
int a = n * tr / total;
|
||||
int b = a + (n * va / total);
|
||||
|
||||
(string Label, IReadOnlyList<Bar> Bars)[] slices =
|
||||
[
|
||||
("intero", bars),
|
||||
("taratura", bars.Take(a).ToArray()),
|
||||
("verifica", bars.Skip(a).Take(b - a).ToArray()),
|
||||
("conferma", bars.Skip(b).ToArray()),
|
||||
];
|
||||
|
||||
(int Fast, int Slow)[] smas = o.SmaFast > 0 && o.SmaSlow > o.SmaFast
|
||||
? [(o.SmaFast, o.SmaSlow)]
|
||||
: [(20, 50), (50, 200), (10, 100)];
|
||||
|
||||
string outPath = OutPath(o, $"baseline_{symbol}_{o.TimeFrameText}.csv");
|
||||
using StreamWriter w = Table(outPath,
|
||||
"simbolo;timeframe;fetta;strategia;barre;anni;netto;cagr;sharpe;sortino;max_dd;calmar;psr;dsr;prove;operazioni;motivazione");
|
||||
|
||||
Console.WriteLine();
|
||||
Console.WriteLine($" FASE 1 — baseline su {symbol} {o.TimeFrameText}, costo per lato {costPerSide:P3}");
|
||||
Console.WriteLine();
|
||||
Console.WriteLine(" fetta strategia barre netto CAGR Sharpe Sortino maxDD Calmar PSR DSR op");
|
||||
Console.WriteLine(" " + new string('-', 108));
|
||||
|
||||
foreach ((string label, IReadOnlyList<Bar> slice) in slices)
|
||||
{
|
||||
if (slice.Count < 300)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
List<BaselineReport> reports = [Core.Backtest.Baseline.BuyAndHold(slice, costPerSide, perYear)];
|
||||
foreach ((int fast, int slow) in smas)
|
||||
{
|
||||
reports.Add(Core.Backtest.Baseline.SmaCrossover(slice, fast, slow, costPerSide, perYear));
|
||||
}
|
||||
|
||||
// Every SMA pair is a trial; the deflated Sharpe of each is computed against
|
||||
// all of them, with the variance of their per-period Sharpes as the guide's V.
|
||||
int trials = smas.Length;
|
||||
double[] trialSharpes = [.. reports.Skip(1).Select(static r => Performance.Sharpe(r.PeriodReturns))];
|
||||
double sd = Performance.StandardDeviation(trialSharpes);
|
||||
double variance = sd * sd;
|
||||
foreach (BaselineReport r in reports)
|
||||
{
|
||||
ReturnMetrics m = r.Name.StartsWith("sma", StringComparison.Ordinal)
|
||||
? ReturnMetrics.From(r.PeriodReturns, perYear, r.Metrics.Trades, trials, variance)
|
||||
: r.Metrics;
|
||||
|
||||
Row(w, symbol, o.TimeFrameText, label, r.Name, slice.Count, m.Years, m.NetReturn, m.Cagr,
|
||||
m.AnnualSharpe, m.AnnualSortino, m.MaxDrawdown, m.Calmar, m.Psr, m.Dsr, m.Trials, m.Trades,
|
||||
r.Motivazione);
|
||||
|
||||
Console.WriteLine(string.Create(CultureInfo.InvariantCulture,
|
||||
$" {label,-10} {r.Name,-16} {slice.Count,7:N0} {m.NetReturn,9:P1} {m.Cagr,8:P1} {m.AnnualSharpe,7:F2} " +
|
||||
$"{m.AnnualSortino,8:F2} {m.MaxDrawdown,7:P1} {m.Calmar,7:F2} {m.Psr,6:F3} {m.Dsr,6:F3} {m.Trades,4}"));
|
||||
}
|
||||
|
||||
Console.WriteLine();
|
||||
}
|
||||
|
||||
Console.WriteLine($" tabella: {outPath}");
|
||||
Console.WriteLine();
|
||||
return 0;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// FASE 2 — cointegration study
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/// <summary>
|
||||
/// Engle-Granger and Johansen on rolling windows, the Kalman hedge ratio beside the
|
||||
/// OLS one, the OU half-life, and the break-even cost of the rule on each slice.
|
||||
/// </summary>
|
||||
public static int Cointegration(Options o)
|
||||
{
|
||||
PairData data = PairData.Load(o);
|
||||
PairBacktestSettings settings = o.Settings();
|
||||
double perYear = settings.BarsPerYear;
|
||||
|
||||
Console.WriteLine();
|
||||
Console.WriteLine($" FASE 2 — {o.SymbolA} / {o.SymbolB} {data.Count:N0} barre {o.TimeFrameText} " +
|
||||
$"{data.From:yyyy-MM-dd} → {data.To:yyyy-MM-dd}");
|
||||
Console.WriteLine();
|
||||
|
||||
// ---- whole-history tests, for orientation only ---------------------
|
||||
double[] logA = [.. data.CloseA.Select(static v => Math.Log(v))];
|
||||
double[] logB = [.. data.CloseB.Select(static v => Math.Log(v))];
|
||||
CointegrationResult eg = Core.Statistics.Cointegration.Test(logA, logB);
|
||||
JohansenResult jo = Johansen.Test([logA, logB], lags: 1);
|
||||
|
||||
Console.WriteLine(" Sull'intera storia (orientativo: un test su sei anni dice poco su oggi):");
|
||||
Console.WriteLine($" Engle-Granger {eg.Describe()}");
|
||||
Console.WriteLine($" Johansen {jo.Describe()}");
|
||||
Console.WriteLine();
|
||||
|
||||
// ---- rolling windows -----------------------------------------------
|
||||
int window = settings.CalibrationBars;
|
||||
int stride = settings.RecalibrateEveryBars;
|
||||
KalmanHedge kalman = new();
|
||||
List<(DateTime Time, CointegrationResult Eg, JohansenResult Jo, double KalmanBeta)> fits = [];
|
||||
|
||||
int kalmanIndex = 0;
|
||||
for (int t = window; t < data.Count; t += stride)
|
||||
{
|
||||
for (; kalmanIndex < t; kalmanIndex++)
|
||||
{
|
||||
kalman.Update(logB[kalmanIndex], logA[kalmanIndex]);
|
||||
}
|
||||
|
||||
double[] wa = logA[(t - window)..t];
|
||||
double[] wb = logB[(t - window)..t];
|
||||
fits.Add((data.BarsA[t].TimeUtc, Core.Statistics.Cointegration.Test(wa, wb), Johansen.Test([wa, wb]), kalman.Beta));
|
||||
}
|
||||
|
||||
int egPass = fits.Count(static f => f.Eg.IsCointegrated);
|
||||
int joPass = fits.Count(static f => f.Jo.Rank >= 1);
|
||||
int both = fits.Count(static f => f.Eg.IsCointegrated && f.Jo.Rank >= 1);
|
||||
double[] halfLives = [.. fits.Where(static f => double.IsFinite(f.Eg.HalfLifeBars)).Select(static f => f.Eg.HalfLifeBars)];
|
||||
double medianHl = halfLives.Length > 0 ? Median(halfLives) : double.NaN;
|
||||
double minutesPerBar = o.TimeFrame.Seconds() / 60.0;
|
||||
|
||||
Console.WriteLine($" Finestre mobili di {window} barre ogni {stride}: {fits.Count} calibrazioni");
|
||||
Console.WriteLine(string.Create(CultureInfo.InvariantCulture,
|
||||
$" Engle-Granger passa il 5% nel {egPass / (double)Math.Max(1, fits.Count):P1} dei casi, " +
|
||||
$"Johansen (rango ≥ 1) nel {joPass / (double)Math.Max(1, fits.Count):P1}, entrambi nel {both / (double)Math.Max(1, fits.Count):P1}"));
|
||||
Console.WriteLine(string.Create(CultureInfo.InvariantCulture,
|
||||
$" emivita OU mediana {medianHl:F0} barre = {medianHl * minutesPerBar / 60:F1} ore"));
|
||||
Console.WriteLine();
|
||||
|
||||
string fitsPath = OutPath(o, $"cointegrazione_{o.SymbolA}_{o.SymbolB}_{o.TimeFrameText}.csv");
|
||||
using (StreamWriter w = Table(fitsPath,
|
||||
"data;coppia;timeframe;beta_ols;alpha;adf;critico_5;p_value;cointegrata_eg;emivita_barre;emivita_ore;" +
|
||||
"johansen_traccia;johansen_critico;johansen_rango;beta_johansen;beta_kalman;motivazione"))
|
||||
{
|
||||
foreach ((DateTime time, CointegrationResult f, JohansenResult j, double kb) in fits)
|
||||
{
|
||||
string why = f.IsCointegrated && j.Rank >= 1
|
||||
? "cointegrata per entrambi i test: operabile"
|
||||
: f.IsCointegrated ? "solo Engle-Granger passa: operabile con cautela"
|
||||
: j.Rank >= 1 ? "solo Johansen passa: la regola (che usa Engle-Granger) resta ferma"
|
||||
: "nessun test passa: nessuna media a cui tornare";
|
||||
|
||||
Row(w, Stamp(time), $"{o.SymbolA}/{o.SymbolB}", o.TimeFrameText, f.Beta, f.Alpha, f.AdfStatistic,
|
||||
f.CriticalValue5, f.PValue, f.IsCointegrated ? 1 : 0, f.HalfLifeBars,
|
||||
f.HalfLifeBars * minutesPerBar / 60, j.TraceStatistics.Length > 0 ? j.TraceStatistics[0] : double.NaN,
|
||||
j.TraceCritical5.Length > 0 ? j.TraceCritical5[0] : double.NaN, j.Rank, j.HedgeRatio, kb, why);
|
||||
}
|
||||
}
|
||||
|
||||
// ---- the rule on the three slices, with its break-even cost ----------
|
||||
(Slice train, Slice validate, Slice confirm) = data.Split(o.SplitWeights, settings.CalibrationBars);
|
||||
string rulePath = OutPath(o, $"regola_{o.SymbolA}_{o.SymbolB}_{o.TimeFrameText}.csv");
|
||||
using StreamWriter rw = Table(rulePath,
|
||||
"coppia;timeframe;fetta;da;a;barre;operazioni;netto;lordo;sharpe;max_dd;calmar;psr;in_mercato;cointegrata_pct;" +
|
||||
"costo_per_lato_bps;break_even_bps;motivazione");
|
||||
|
||||
Console.WriteLine(" La regola statistica con i parametri correnti, e il costo che la azzera:");
|
||||
Console.WriteLine();
|
||||
|
||||
foreach (Slice slice in new[] { train, validate, confirm })
|
||||
{
|
||||
PairCalibrationSchedule schedule = PairCalibrationSchedule.Precompute(slice.CloseA, slice.CloseB, settings);
|
||||
PairBacktestReport r = PairBacktest.Run(slice.Label, slice.BarsA, slice.BarsB, o.Parameters(), settings, schedule);
|
||||
|
||||
double breakEven = BreakEvenBps(slice, o, settings, schedule);
|
||||
double[] eqReturns = EquityReturns(r.EquityCurve);
|
||||
ReturnMetrics m = ReturnMetrics.From(eqReturns, perYear, r.Trades.Count);
|
||||
|
||||
string costVerdict = breakEven > o.FeeBps + o.SlippageBps ? "il margine copre i costi" : "i costi mangiano il margine";
|
||||
string why = r.Trades.Count == 0
|
||||
? "nessuna operazione: la coppia non ha mai passato il filtro con questi parametri"
|
||||
: string.Create(CultureInfo.InvariantCulture,
|
||||
$"{r.Trades.Count} operazioni, netto {r.NetReturn:P2}, Sharpe {m.AnnualSharpe:F2}; " +
|
||||
$"break-even {breakEven:F1} bps per lato contro {o.FeeBps + o.SlippageBps:F1} stimati: {costVerdict}");
|
||||
|
||||
Row(rw, $"{o.SymbolA}/{o.SymbolB}", o.TimeFrameText, slice.Label, Stamp(r.FromUtc), Stamp(r.ToUtc), r.Bars,
|
||||
r.Trades.Count, r.NetReturn, r.GrossReturn, m.AnnualSharpe, r.MaxDrawdown, r.Calmar, m.Psr, r.TimeInMarket,
|
||||
r.CointegratedFraction, o.FeeBps + o.SlippageBps, breakEven, why);
|
||||
|
||||
Console.WriteLine($" {slice.Label,-9} {r.Describe()}");
|
||||
Console.WriteLine(string.Create(CultureInfo.InvariantCulture,
|
||||
$" PSR {m.Psr:F3} break-even {breakEven:F1} bps/lato (stimati {o.FeeBps + o.SlippageBps:F1})"));
|
||||
}
|
||||
|
||||
Console.WriteLine();
|
||||
Console.WriteLine($" tabelle: {fitsPath}");
|
||||
Console.WriteLine($" {rulePath}");
|
||||
Console.WriteLine();
|
||||
return 0;
|
||||
}
|
||||
|
||||
/// <summary>The per-side cost, in basis points, at which the rule's net return crosses zero.</summary>
|
||||
private static double BreakEvenBps(Slice slice, Options o, PairBacktestSettings settings, PairCalibrationSchedule schedule)
|
||||
{
|
||||
double NetAt(double bps)
|
||||
{
|
||||
PairBacktestSettings s = settings with { TakerFeeBps = bps, SlippageBps = 0 };
|
||||
return PairBacktest.Run("be", slice.BarsA, slice.BarsB, o.Parameters(), s, schedule).NetReturn;
|
||||
}
|
||||
|
||||
if (NetAt(0) <= 0)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
double lo = 0, hi = 200;
|
||||
if (NetAt(hi) > 0)
|
||||
{
|
||||
return hi;
|
||||
}
|
||||
|
||||
for (int i = 0; i < 20; i++)
|
||||
{
|
||||
double mid = (lo + hi) / 2;
|
||||
if (NetAt(mid) > 0)
|
||||
{
|
||||
lo = mid;
|
||||
}
|
||||
else
|
||||
{
|
||||
hi = mid;
|
||||
}
|
||||
}
|
||||
|
||||
return (lo + hi) / 2;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// FASE 3 — meta-labelling dataset and model
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/// <summary>Builds and stores the meta-labelling dataset for a pair.</summary>
|
||||
public static int Dataset(Options o)
|
||||
{
|
||||
using StorageDb? db = OpenDb(o);
|
||||
(MetaDataset dataset, PairData data, string labelSpec) = BuildDataset(o);
|
||||
|
||||
if (dataset.Count == 0)
|
||||
{
|
||||
Console.WriteLine(" nessuna riga: la regola non ha mai proposto un ingresso su questi dati");
|
||||
return 1;
|
||||
}
|
||||
|
||||
long id = 0;
|
||||
if (db is not null)
|
||||
{
|
||||
id = db.SaveDataset($"{o.SymbolA}/{o.SymbolB}", o.TimeFrameText, dataset.FeatureNames, labelSpec,
|
||||
[.. dataset.Rows.Select(static r => new StorageDb.DatasetRowRecord(r.TimeUtc, r.EndUtc, r.Label, r.Side, r.Return, r.Weight, r.Features))],
|
||||
string.Create(CultureInfo.InvariantCulture,
|
||||
$"{dataset.Count} segnali della regola su {data.Count} barre {o.TimeFrameText}, " +
|
||||
$"{dataset.PositiveRate:P1} andati a buon fine; {labelSpec}"));
|
||||
}
|
||||
|
||||
string path = OutPath(o, $"dataset_{o.SymbolA}_{o.SymbolB}_{o.TimeFrameText}.csv");
|
||||
using (StreamWriter w = Table(path,
|
||||
"data;fine;lato;etichetta;rendimento;peso;" + string.Join(Sep, dataset.FeatureNames) + ";motivazione"))
|
||||
{
|
||||
foreach (MetaRow r in dataset.Rows)
|
||||
{
|
||||
object[] cells = new object[6 + r.Features.Length + 1];
|
||||
cells[0] = Stamp(r.TimeUtc);
|
||||
cells[1] = Stamp(r.EndUtc);
|
||||
cells[2] = r.Side;
|
||||
cells[3] = r.Label;
|
||||
cells[4] = r.Return;
|
||||
cells[5] = r.Weight;
|
||||
for (int f = 0; f < r.Features.Length; f++)
|
||||
{
|
||||
cells[6 + f] = r.Features[f];
|
||||
}
|
||||
|
||||
cells[^1] = string.Create(CultureInfo.InvariantCulture,
|
||||
$"segnale {(r.Side > 0 ? "long" : "short")} spread: {(r.Label == 1 ? "ha reso" : "ha perso")} " +
|
||||
$"{r.Return:P2} entro {r.EndIndex - r.Index} barre; peso {r.Weight:F3} per la sovrapposizione con altri segnali");
|
||||
Row(w, cells);
|
||||
}
|
||||
}
|
||||
|
||||
Console.WriteLine();
|
||||
string saved = id > 0 ? string.Create(CultureInfo.InvariantCulture, $", salvato come #{id}") : string.Empty;
|
||||
Console.WriteLine(string.Create(CultureInfo.InvariantCulture,
|
||||
$" dataset: {dataset.Count} righe ({dataset.EffectiveCount:F0} indipendenti), {dataset.FeatureNames.Count} feature, " +
|
||||
$"positivi {dataset.PositiveRate:P1}{saved}"));
|
||||
Console.WriteLine($" tabella: {path}");
|
||||
Console.WriteLine();
|
||||
return 0;
|
||||
}
|
||||
|
||||
private static (MetaDataset Dataset, PairData Data, string LabelSpec) BuildDataset(Options o)
|
||||
{
|
||||
PairData data = PairData.Load(o);
|
||||
PairBacktestSettings settings = o.Settings();
|
||||
|
||||
Console.WriteLine();
|
||||
Console.WriteLine($" FASE 3 — {o.SymbolA} / {o.SymbolB} {data.Count:N0} barre {o.TimeFrameText}");
|
||||
|
||||
PairCalibrationSchedule schedule = PairCalibrationSchedule.Precompute(data.CloseA, data.CloseB, settings);
|
||||
PrimaryReplayResult primary = PrimaryReplay.Run(data.BarsA, data.BarsB, o.Parameters(), settings, schedule);
|
||||
|
||||
Console.WriteLine($" la regola propone un ingresso su {primary.SignalCount:N0} barre");
|
||||
|
||||
PairSeries series = PairSeries.Build(data.BarsA, data.BarsB, primary.Beta, primary.Alpha, primary.HalfLife, o.Window, primary.ZScore);
|
||||
LabelingParams labeling = new()
|
||||
{
|
||||
ProfitMultiple = o.ProfitMultiple,
|
||||
StopMultiple = o.StopMultiple,
|
||||
MaxHoldingBars = o.LabelMaxBars > 0 ? o.LabelMaxBars : Math.Max(24, 86_400 / o.TimeFrame.Seconds()),
|
||||
};
|
||||
|
||||
MetaDataset dataset = MetaLabeling.BuildDataset(series, primary.Sides, FeatureRegistry.Default(), labeling);
|
||||
return (dataset, data, labeling.Describe());
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Trains the meta-model over a small grid, validates each configuration by CPCV,
|
||||
/// reports PBO and the deflated Sharpe counting every configuration, and — only if
|
||||
/// asked and only if the bar is cleared — promotes the winner to champion.
|
||||
/// </summary>
|
||||
public static int Train(Options o)
|
||||
{
|
||||
using StorageDb? db = OpenDb(o);
|
||||
(MetaDataset dataset, PairData data, string labelSpec) = BuildDataset(o);
|
||||
string pair = $"{o.SymbolA}/{o.SymbolB}";
|
||||
|
||||
if (dataset.Count < 120)
|
||||
{
|
||||
Console.WriteLine($" {dataset.Count} righe non bastano: servono almeno 120 segnali per sei gruppi CPCV");
|
||||
return 1;
|
||||
}
|
||||
|
||||
long datasetId = db?.SaveDataset(pair, o.TimeFrameText, dataset.FeatureNames, labelSpec,
|
||||
[.. dataset.Rows.Select(static r => new StorageDb.DatasetRowRecord(r.TimeUtc, r.EndUtc, r.Label, r.Side, r.Return, r.Weight, r.Features))],
|
||||
$"dataset per l'addestramento: {dataset.Count} segnali, {labelSpec}") ?? 0;
|
||||
|
||||
GbdtParams[] grid = o.Trials > 0 && o.Trials < 6
|
||||
? [.. Grid().Take(o.Trials)]
|
||||
: [.. Grid()];
|
||||
|
||||
Console.WriteLine($" {grid.Length} configurazioni, CPCV 6/2 con embargo 1%: ogni numero qui sotto è fuori campione");
|
||||
Console.WriteLine();
|
||||
|
||||
MetaTrainingResult[] results = new MetaTrainingResult[grid.Length];
|
||||
for (int i = 0; i < grid.Length; i++)
|
||||
{
|
||||
results[i] = MetaLabeling.Train(dataset, grid[i], trials: grid.Length, seed: 1 + i);
|
||||
Console.WriteLine(string.Create(CultureInfo.InvariantCulture,
|
||||
$" #{i + 1} {results[i].Describe()}"));
|
||||
Console.WriteLine($" {grid[i].Describe()}");
|
||||
}
|
||||
|
||||
// PBO across the configurations: each one's out-of-sample path returns are one
|
||||
// trial. The deflated Sharpe is recomputed with the variance of the trials'
|
||||
// Sharpes now that it is known.
|
||||
List<double[]> paths = [.. results.Select(static r => r.PathReturns)];
|
||||
int minLen = paths.Min(static p => p.Length);
|
||||
PboResult pbo = Pbo.Compute([.. paths.Select(p => p[..minLen])], blocks: 8);
|
||||
double[] sharpes = [.. results.Select(static r => r.MeanPathSharpe)];
|
||||
double variance = sharpes.Length > 1 ? Performance.StandardDeviation(sharpes) * Performance.StandardDeviation(sharpes) : 0;
|
||||
|
||||
int best = 0;
|
||||
for (int i = 1; i < results.Length; i++)
|
||||
{
|
||||
if (results[i].MeanPathSharpe > results[best].MeanPathSharpe)
|
||||
{
|
||||
best = i;
|
||||
}
|
||||
}
|
||||
|
||||
MetaTrainingResult winner = results[best];
|
||||
|
||||
// Confidence on the independent observations, not on the row count: signals come
|
||||
// in clusters, and a cluster is one observation however many bars it spans.
|
||||
int perPath = (int)Math.Max(2, Math.Round(Math.Min(dataset.Count, dataset.EffectiveCount)));
|
||||
double dsr = Performance.DeflatedSharpe(winner.MeanPathSharpe, perPath,
|
||||
Performance.Skewness(winner.PathReturns), Performance.Kurtosis(winner.PathReturns), grid.Length, variance);
|
||||
|
||||
// Robustness: the winner's neighbours in the grid must not collapse.
|
||||
double neighbourMin = results.Where((_, i) => i != best).Select(static r => r.MeanPathSharpe).DefaultIfEmpty(winner.MeanPathSharpe).Min();
|
||||
bool robust = neighbourMin > 0 || winner.MeanPathSharpe <= 0;
|
||||
|
||||
Console.WriteLine();
|
||||
Console.WriteLine(string.Create(CultureInfo.InvariantCulture,
|
||||
$" PBO {pbo.Probability:F3} su {pbo.Combinations} combinazioni DSR del migliore {dsr:F3} " +
|
||||
$"(contando {grid.Length} configurazioni, varianza degli Sharpe {variance:F4})"));
|
||||
|
||||
double years = Math.Max(0.1, data.Count / o.Settings().BarsPerYear);
|
||||
double annualSharpe = winner.MeanPathSharpe * Math.Sqrt(Math.Max(1, dataset.EffectiveCount / years));
|
||||
bool passes = pbo.Probability < 0.5 && dsr > 0.95 && winner.OosAccuracy > 0.5 && winner.MeanPathSharpe > 0 && robust;
|
||||
|
||||
string verdict = passes
|
||||
? "supera i criteri della Fase 3 (PBO < 0.5, DSR > 0.95, Sharpe OOS > 0, vicini della griglia non negativi)"
|
||||
: "NON supera i criteri della Fase 3: " + string.Join(", ", Reasons(pbo.Probability, dsr, winner, robust));
|
||||
|
||||
Console.WriteLine($" {verdict}");
|
||||
Console.WriteLine(string.Create(CultureInfo.InvariantCulture,
|
||||
$" Sharpe per segnale {winner.MeanPathSharpe:F3} ≈ {annualSharpe:F2} annualizzato: {dataset.Count} segnali " +
|
||||
$"ma {dataset.EffectiveCount:F0} osservazioni indipendenti in {years:F1} anni"));
|
||||
|
||||
// Feature importance of the winner: which columns did the work.
|
||||
Console.WriteLine();
|
||||
Console.WriteLine(" Importanza delle feature (guadagno di split, prime dieci):");
|
||||
var importance = winner.Model.FeatureNames
|
||||
.Select((name, i) => (Name: name, Gain: winner.Model.Importance[i]))
|
||||
.OrderByDescending(static x => x.Gain)
|
||||
.Take(10);
|
||||
double totalGain = Math.Max(1e-12, winner.Model.Importance.Sum());
|
||||
foreach ((string name, double gain) in importance)
|
||||
{
|
||||
Console.WriteLine(string.Create(CultureInfo.InvariantCulture, $" {name,-18} {gain / totalGain,7:P1}"));
|
||||
}
|
||||
|
||||
// ---- tables and database ---------------------------------------------
|
||||
string path = OutPath(o, $"validazione_{o.SymbolA}_{o.SymbolB}_{o.TimeFrameText}.csv");
|
||||
using (StreamWriter w = Table(path,
|
||||
"coppia;timeframe;configurazione;righe;percorsi;accuratezza_oos;logloss_oos;sharpe_oos_medio;sharpe_oos_min;sharpe_oos_max;psr;dsr;pbo;prove;migliore;motivazione"))
|
||||
{
|
||||
for (int i = 0; i < results.Length; i++)
|
||||
{
|
||||
MetaTrainingResult r = results[i];
|
||||
Row(w, pair, o.TimeFrameText, grid[i].Describe(), r.Rows, r.Paths, r.OosAccuracy, r.OosLogLoss,
|
||||
r.MeanPathSharpe, r.PathSharpes.Min(), r.PathSharpes.Max(), r.Psr, i == best ? dsr : r.Dsr,
|
||||
pbo.Probability, grid.Length, i == best ? 1 : 0,
|
||||
i == best ? r.Motivazione + "; " + verdict : r.Motivazione);
|
||||
}
|
||||
}
|
||||
|
||||
long modelId = 0;
|
||||
if (db is not null)
|
||||
{
|
||||
string metrics = string.Create(CultureInfo.InvariantCulture,
|
||||
$"{{\"oosAccuracy\":{winner.OosAccuracy:F4},\"oosLogLoss\":{winner.OosLogLoss:F4}," +
|
||||
$"\"meanPathSharpe\":{winner.MeanPathSharpe:F4},\"psr\":{winner.Psr:F4},\"dsr\":{dsr:F4}," +
|
||||
$"\"pbo\":{pbo.Probability:F4},\"trials\":{grid.Length},\"rows\":{dataset.Count}}}");
|
||||
string parameters = "{\"gbdt\":\"" + grid[best].Describe() + "\",\"labels\":\"" + labelSpec + "\"}";
|
||||
|
||||
modelId = db.SaveModel(pair, "gbdt-meta", datasetId, parameters, metrics, winner.Model.FeatureNames,
|
||||
winner.Model.ToBytes(), winner.Motivazione + "; " + verdict);
|
||||
db.SaveValidation(modelId, "cpcv-6-2", grid.Length, winner.MeanPathSharpe, winner.Psr, dsr, pbo.Probability,
|
||||
winner.OosAccuracy, verdict);
|
||||
|
||||
Console.WriteLine();
|
||||
Console.WriteLine($" modello #{modelId} salvato (dataset #{datasetId})");
|
||||
|
||||
if (o.Promote)
|
||||
{
|
||||
if (passes)
|
||||
{
|
||||
db.SetChampion(pair, modelId, $"promosso da 'train': {verdict}");
|
||||
Console.WriteLine($" PROMOSSO a campione di {pair}: il bot lo caricherà alla prossima calibrazione");
|
||||
}
|
||||
else
|
||||
{
|
||||
Console.WriteLine(" non promosso: --promote vale solo per un modello che supera i criteri");
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
Console.WriteLine(" non promosso (aggiungi --promote per farlo diventare il campione, se supera i criteri)");
|
||||
}
|
||||
}
|
||||
|
||||
Console.WriteLine($" tabella: {path}");
|
||||
Console.WriteLine();
|
||||
return passes ? 0 : 3;
|
||||
|
||||
static IEnumerable<GbdtParams> Grid()
|
||||
{
|
||||
yield return new GbdtParams { Trees = 150, LearningRate = 0.05, MaxDepth = 3, MinLeafHessian = 20 };
|
||||
yield return new GbdtParams { Trees = 300, LearningRate = 0.03, MaxDepth = 4, MinLeafHessian = 20 };
|
||||
yield return new GbdtParams { Trees = 200, LearningRate = 0.05, MaxDepth = 2, MinLeafHessian = 30 };
|
||||
yield return new GbdtParams { Trees = 400, LearningRate = 0.02, MaxDepth = 3, MinLeafHessian = 40 };
|
||||
yield return new GbdtParams { Trees = 100, LearningRate = 0.10, MaxDepth = 4, MinLeafHessian = 10 };
|
||||
yield return new GbdtParams { Trees = 250, LearningRate = 0.04, MaxDepth = 3, MinLeafHessian = 20, Subsample = 0.6, ColumnSample = 0.6 };
|
||||
}
|
||||
|
||||
static IEnumerable<string> Reasons(double pbo, double dsr, MetaTrainingResult w, bool robust)
|
||||
{
|
||||
if (!(pbo < 0.5)) { yield return string.Create(CultureInfo.InvariantCulture, $"PBO {pbo:F2} ≥ 0.5 (la selezione sceglie rumore)"); }
|
||||
if (!(dsr > 0.95)) { yield return string.Create(CultureInfo.InvariantCulture, $"DSR {dsr:F2} ≤ 0.95 (lo Sharpe non si distingue dal caso, contate le prove)"); }
|
||||
if (!(w.OosAccuracy > 0.5)) { yield return string.Create(CultureInfo.InvariantCulture, $"accuratezza OOS {w.OosAccuracy:P1} non sopra il 50%"); }
|
||||
if (!(w.MeanPathSharpe > 0)) { yield return "Sharpe OOS non positivo"; }
|
||||
if (!robust) { yield return "una configurazione vicina nella griglia è negativa: l'edge non è stabile"; }
|
||||
}
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// FASE 4 — reinforcement learning
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
public static int Rl(Options o)
|
||||
{
|
||||
using StorageDb? db = OpenDb(o);
|
||||
PairData data = PairData.Load(o);
|
||||
PairBacktestSettings settings = o.Settings();
|
||||
string pair = $"{o.SymbolA}/{o.SymbolB}";
|
||||
int seeds = Math.Max(1, o.Seeds);
|
||||
int episodes = Math.Max(1, o.Episodes);
|
||||
|
||||
Console.WriteLine();
|
||||
Console.WriteLine($" FASE 4 — DQN su {pair} {o.TimeFrameText}, {seeds} seed × {episodes} episodi");
|
||||
|
||||
PairCalibrationSchedule schedule = PairCalibrationSchedule.Precompute(data.CloseA, data.CloseB, settings);
|
||||
PrimaryReplayResult primary = PrimaryReplay.Run(data.BarsA, data.BarsB, o.Parameters(), settings, schedule);
|
||||
PairSeries series = PairSeries.Build(data.BarsA, data.BarsB, primary.Beta, primary.Alpha, primary.HalfLife, o.Window, primary.ZScore);
|
||||
|
||||
(int tr, int va, int co) = o.SplitWeights;
|
||||
int total = Math.Max(1, tr + va + co);
|
||||
int trainEnd = data.Count * tr / total;
|
||||
int testStart = trainEnd;
|
||||
int testEnd = trainEnd + (data.Count * va / total);
|
||||
|
||||
(double[][] x, double[] ret, string[] names) = PairEnvironment.Inputs(
|
||||
series, FeatureRegistry.Default(), PairEnvironment.DefaultFeatures, standardiseUpTo: trainEnd);
|
||||
|
||||
// Rows before the first calibration carry no spread: drop them from both spans.
|
||||
int first = primary.Start + Math.Max(20, o.Window);
|
||||
if (first >= trainEnd - 100 || testEnd - testStart < 200)
|
||||
{
|
||||
Console.Error.WriteLine(" troppo poche barre per addestrare e verificare");
|
||||
return 1;
|
||||
}
|
||||
|
||||
PairEnvironment train = new(x[first..trainEnd], ret[first..trainEnd], settings.RoundCost, o.MaxBars);
|
||||
PairEnvironment test = new(x[testStart..testEnd], ret[testStart..testEnd], settings.RoundCost, o.MaxBars);
|
||||
|
||||
// The baseline on the same held-out span, with the same costs.
|
||||
Slice validate = new("verifica", data.BarsA.GetRange(Math.Max(0, testStart - settings.CalibrationBars), testEnd - Math.Max(0, testStart - settings.CalibrationBars)),
|
||||
data.BarsB.GetRange(Math.Max(0, testStart - settings.CalibrationBars), testEnd - Math.Max(0, testStart - settings.CalibrationBars)));
|
||||
PairBacktestReport baseline = PairBacktest.Run("baseline", validate.BarsA, validate.BarsB, o.Parameters(), settings,
|
||||
PairCalibrationSchedule.Precompute(validate.CloseA, validate.CloseB, settings));
|
||||
ReturnMetrics baselineMetrics = ReturnMetrics.From(EquityReturns(baseline.EquityCurve), settings.BarsPerYear, baseline.Trades.Count);
|
||||
|
||||
Console.WriteLine(string.Create(CultureInfo.InvariantCulture,
|
||||
$" addestramento {train.Length:N0} barre, verifica {test.Length:N0} barre, {names.Length} ingressi + posizione"));
|
||||
Console.WriteLine(string.Create(CultureInfo.InvariantCulture,
|
||||
$" baseline (regola statistica) sulla verifica: netto {baseline.NetReturn:P2}, Sharpe {baselineMetrics.AnnualSharpe:F2}, " +
|
||||
$"DD {baseline.MaxDrawdown:P2}, {baseline.Trades.Count} op"));
|
||||
Console.WriteLine();
|
||||
|
||||
DqnParams p = new();
|
||||
object gate = new();
|
||||
List<RlSeedResult> results = Experiment.Run(train, test, p, seeds, episodes, settings.BarsPerYear, progress: r =>
|
||||
{
|
||||
lock (gate)
|
||||
{
|
||||
Console.WriteLine(" " + r.Describe());
|
||||
}
|
||||
});
|
||||
|
||||
results.Sort(static (a, b) => a.Seed.CompareTo(b.Seed));
|
||||
|
||||
double[] nets = [.. results.Select(static r => r.Test.NetReturn)];
|
||||
double[] sharpes = [.. results.Select(static r => r.Test.AnnualSharpe)];
|
||||
int beating = results.Count(r => r.Test.AnnualSharpe > baselineMetrics.AnnualSharpe && r.Test.NetReturn > 0);
|
||||
double medianSharpe = Median(sharpes);
|
||||
double worstSharpe = sharpes.Min();
|
||||
bool stable = beating >= (int)Math.Ceiling(seeds * 0.8) && worstSharpe > 0 && seeds >= 15;
|
||||
|
||||
string verdict = stable
|
||||
? $"stabile: {beating}/{seeds} seed battono la baseline dopo i costi e nessuno è negativo"
|
||||
: $"NON stabile: {beating}/{seeds} seed battono la baseline, Sharpe mediano {medianSharpe:F2}, " +
|
||||
$"peggiore {worstSharpe:F2}{(seeds < 15 ? " (servono almeno 15 seed per un giudizio)" : string.Empty)} — " +
|
||||
"la guida dice di abbandonare e tenere la regola statistica";
|
||||
|
||||
Console.WriteLine();
|
||||
Console.WriteLine(string.Create(CultureInfo.InvariantCulture,
|
||||
$" netto mediano {Median(nets):P2} (min {nets.Min():P2}, max {nets.Max():P2}) " +
|
||||
$"Sharpe mediano {medianSharpe:F2} (min {worstSharpe:F2}, max {sharpes.Max():F2})"));
|
||||
Console.WriteLine($" {verdict}");
|
||||
|
||||
string path = OutPath(o, $"rl_{o.SymbolA}_{o.SymbolB}_{o.TimeFrameText}.csv");
|
||||
using (StreamWriter w = Table(path,
|
||||
"coppia;timeframe;seed;episodi;netto;sharpe;max_dd;cambi_posizione;baseline_netto;baseline_sharpe;batte_baseline;motivazione"))
|
||||
{
|
||||
foreach (RlSeedResult r in results)
|
||||
{
|
||||
bool beats = r.Test.AnnualSharpe > baselineMetrics.AnnualSharpe && r.Test.NetReturn > 0;
|
||||
string why = beats
|
||||
? "batte la baseline sulla verifica dopo i costi"
|
||||
: "non batte la baseline sulla verifica dopo i costi";
|
||||
Row(w, pair, o.TimeFrameText, r.Seed, r.Episodes, r.Test.NetReturn, r.Test.AnnualSharpe, r.Test.MaxDrawdown,
|
||||
r.Trades, baseline.NetReturn, baselineMetrics.AnnualSharpe, beats ? 1 : 0, why + "; " + p.Describe());
|
||||
|
||||
db?.InsertRlRun(pair, r.Seed, r.Episodes, r.Test.NetReturn, r.Test.AnnualSharpe, r.Test.MaxDrawdown,
|
||||
baseline.NetReturn, why + "; " + verdict);
|
||||
}
|
||||
}
|
||||
|
||||
Console.WriteLine($" tabella: {path}");
|
||||
Console.WriteLine();
|
||||
return stable ? 0 : 3;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Status
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
public static int Status(Options o)
|
||||
{
|
||||
using StorageDb? db = OpenDb(o);
|
||||
if (db is null)
|
||||
{
|
||||
Console.WriteLine(" nessun database");
|
||||
return 1;
|
||||
}
|
||||
|
||||
Console.WriteLine();
|
||||
Console.WriteLine($" database: {db.Path}");
|
||||
foreach (string table in new[] { "bars", "data_quality", "decisions", "executions", "trades", "datasets", "dataset_rows", "models", "validations", "champions", "drift", "rl_runs" })
|
||||
{
|
||||
Console.WriteLine(string.Create(CultureInfo.InvariantCulture, $" {table,-14} {db.Count(table),10:N0}"));
|
||||
}
|
||||
|
||||
Console.WriteLine();
|
||||
return 0;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Helpers
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
private static StorageDb? OpenDb(Options o)
|
||||
{
|
||||
if (o.NoDb)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
string path = string.IsNullOrWhiteSpace(o.DatabasePath) ? StorageDb.DefaultPath : Path.GetFullPath(o.DatabasePath);
|
||||
try
|
||||
{
|
||||
StorageDb db = new(path);
|
||||
db.OnError = static (what, ex) => Console.Error.WriteLine($" database ({what}): {ex.Message}");
|
||||
return db;
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
Console.Error.WriteLine($" database non apribile in {path}: {ex.Message} — continuo con i soli CSV");
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private static string OutPath(Options o, string file)
|
||||
{
|
||||
string dir = string.IsNullOrWhiteSpace(o.OutputDirectory)
|
||||
? Path.Combine(AppContext.BaseDirectory, "ricerca")
|
||||
: Path.GetFullPath(o.OutputDirectory);
|
||||
Directory.CreateDirectory(dir);
|
||||
return Path.Combine(dir, file);
|
||||
}
|
||||
|
||||
private static StreamWriter Table(string path, string header)
|
||||
{
|
||||
StreamWriter w = new(path, false, new UTF8Encoding(false));
|
||||
w.WriteLine(header);
|
||||
return w;
|
||||
}
|
||||
|
||||
private static void Row(StreamWriter w, params object[] cells)
|
||||
{
|
||||
StringBuilder sb = new(256);
|
||||
for (int i = 0; i < cells.Length; i++)
|
||||
{
|
||||
if (i > 0)
|
||||
{
|
||||
sb.Append(Sep);
|
||||
}
|
||||
|
||||
string text = cells[i] switch
|
||||
{
|
||||
double d => double.IsFinite(d) ? d.ToString("0.########", CultureInfo.InvariantCulture) : string.Empty,
|
||||
float f => f.ToString("0.######", CultureInfo.InvariantCulture),
|
||||
int n => n.ToString(CultureInfo.InvariantCulture),
|
||||
long l => l.ToString(CultureInfo.InvariantCulture),
|
||||
bool b => b ? "1" : "0",
|
||||
null => string.Empty,
|
||||
_ => cells[i].ToString() ?? string.Empty,
|
||||
};
|
||||
|
||||
sb.Append(text.Replace(Sep, ',').Replace('\n', ' ').Replace('\r', ' '));
|
||||
}
|
||||
|
||||
w.WriteLine(sb.ToString());
|
||||
}
|
||||
|
||||
private static string Stamp(DateTime utc) => utc.ToString("yyyy-MM-dd HH:mm", CultureInfo.InvariantCulture);
|
||||
|
||||
private static double[] EquityReturns(IReadOnlyList<double> equity)
|
||||
{
|
||||
double[] r = new double[Math.Max(0, equity.Count - 1)];
|
||||
for (int i = 1; i < equity.Count; i++)
|
||||
{
|
||||
r[i - 1] = equity[i - 1] > 0 ? (equity[i] / equity[i - 1]) - 1 : 0;
|
||||
}
|
||||
|
||||
return r;
|
||||
}
|
||||
|
||||
private static double Median(IReadOnlyList<double> values)
|
||||
{
|
||||
double[] s = [.. values.Order()];
|
||||
if (s.Length == 0)
|
||||
{
|
||||
return double.NaN;
|
||||
}
|
||||
|
||||
int mid = s.Length / 2;
|
||||
return s.Length % 2 == 1 ? s[mid] : (s[mid - 1] + s[mid]) / 2;
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user