Fase 2 — Dati di mercato #88

Merged
Alby96 merged 11 commits from fase-2 into main 2026-11-10 17:53:14 +01:00
11 Commits
Author SHA1 Message Date
Alby96andClaude Opus 5.5 1a8d48212f Give test databases a longer PostgreSQL connect timeout (refs #20 #25)
ci / test (postgres) (push) Successful in 2m37s
ci / test (sqlite) (push) Successful in 2m4s
ci / test (postgres) (pull_request) Successful in 3m17s
ci / test (sqlite) (pull_request) Successful in 2m22s
ci / image (push) Successful in 35s
ci / image (pull_request) Successful in 11s
On the CI runner twelve test workers open PostgreSQL connections at once
and some waited longer than the application's 5 seconds, failing two
collector tests with a connection timeout that has nothing to do with
the code under test. The connect timeout is now a DbSettings field that
keeps the 5-second default for the application, and the test databases
ask for 30 seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 17:40:12 +01:00
Alby96andClaude Opus 5.5 094a527ba2 Make two tests steady on a busy CI runner (refs #20 #25)
ci / test (postgres) (push) Successful in 2m48s
ci / test (sqlite) (push) Successful in 2m10s
ci / test (postgres) (pull_request) Failing after 3m37s
ci / test (sqlite) (pull_request) Successful in 2m20s
ci / image (pull_request) Skipped
ci / image (push) Successful in 21s
- The skipped-minute test waited for the recovered bar, but the issue is
  closed a moment after the bar is stored; on PostgreSQL in CI (run 181) it
  read the issue in between and saw it still open. It now waits for the
  issue's final state.
- Creating a PostgreSQL test database retries on a connection timeout: CI
  run 179 on main failed because twelve workers created their databases at
  once and one connection missed the application's 5-second timeout. The
  application's timeout is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 17:23:58 +01:00
Alby96andClaude Opus 5.5 de2cc0eedf Fix the phase 2 review findings (refs #20 #23 #25)
ci / test (postgres) (push) Successful in 2m10s
ci / test (sqlite) (push) Successful in 2m14s
ci / image (push) Successful in 31s
ci / test (postgres) (pull_request) Failing after 2m50s
ci / test (sqlite) (pull_request) Successful in 2m27s
ci / image (pull_request) Skipped
- Gaps between the last stored bar and the first new one were never found,
  so a minute the live feed skipped, or a day missing right after the stored
  history, was lost for good. A gap is now new when it ends after the last
  bar held, and is downloaded again like any other.
- When one OKX connection dropped while the other kept writing, the write to
  the closed feed threw out of the live loop and left it marked connected.
  Writes now stop quietly, and the loop treats any failure (database
  included) as a drop: marked down, back off, reconnect.
- The background worker now survives a failed round instead of stopping for
  the life of the process.
- Alpaca gives adjusted prices on the basis of the download day, so a later
  split looked like a crash. Each download re-reads the last stored day and,
  if its adjusted/real ratio changed, rescales all older adjusted prices.
- Stock days are fetched up to the last closed New York session, so the job
  after the close collects that day; missing stock days are checked against
  the exchange calendar; delisted stocks with history are not asked again.
- Smaller: a lock in the freshness monitor, no NaN mood readings (SQLite
  stored them as NULL), and the network proof runs only with
  ANUBIS_LIVE_PROOF=1. The test wait is 30 s for busy machines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 17:06:39 +01:00
Alby96andClaude Opus 5.5 68fdd0baee Record the Phase 2 choices, verified facts and state (refs #25)
ADR 0008 collects the market data decisions the spec leaves open: research
history in csv.gz instead of Parquet (no extra package), the immediate second
reading of suspect data, EUR pairs only, Alpaca only with the paper keys,
name-based fund exclusion and the background worker. API-NOTES gets the facts
checked while writing the adapters and the biases that remain in past
universes. ROADMAP and STATE mark the phase done, with the stock history still
waiting for the user's Alpaca keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 15:55:01 +01:00
Alby96andClaude Opus 5.5 aa2766f3ed Add a network proof of the real daily history (refs #18 #21)
A LiveDemo test, outside the normal suite, that downloads the whole OKX EUR
daily history of BTC, ETH and SOL and the Binance BTCUSDT archives since 2017,
then writes the quality report. It is the evidence for the phase criteria that
a fake network cannot give.

The detached database test now retries the file move for a few seconds:
Windows refuses to move the SQLite file while a request of the running server
holds it, which made the test fail now and then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 15:51:37 +01:00
Alby96andClaude Opus 5.5 4a65d4060f Show data sources and the central bank list in Settings (refs #22)
A "Dati" section lists every external source with its state, last answer
and the reason it is off (Alpaca without the paper keys, FRED and Finnhub
without theirs), links the 30-day quality report, and edits the Fed/ECB
meeting list; a list that does not parse is refused with an example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 15:45:06 +01:00
Alby96andClaude Opus 5.5 1e71038209 Run market data inside the server (refs #19 #20 #22 #24)
MarketDataService wires the collector, the reference sources, the
universes and the corporate actions into the scheduler. Short jobs (rates,
calendar, mood, spread samples, corporate actions) run on their schedule;
long downloads (catalogues, the first history, thousands of stocks, the
Binance archives) run on a background worker the jobs only wake, so they
never delay the backup or each other. The live crypto feed follows the
crypto universe, or BTC, ETH and SOL until one exists.

Alpaca uses only the paper account keys; without them the stock part is off
and the sources list says why. Market data is off in demo mode, with
ANUBIS_MARKET_DATA=off, and in tests that bring no fake network.

New: GET /api/data/quality (JSON or Markdown) and /api/data/sources, and
"anubis data report|binance|csv" on the command line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 15:42:10 +01:00
Alby96andClaude Opus 5.5 49ed73e6f6 Add the market data collector, universes and corporate action store (refs #20 #23 #24)
MarketCollector downloads what is missing at startup and after every drop
of the live feed, checks every bar and never corrects in silence: suspect
bars are stored as suspect and read a second time (same values confirm,
different ones replace, impossible ones are dropped), gaps are downloaded
again once and what the source still lacks is recorded as unrecoverable.
A test detaches the feed for ten minutes and finds every minute recovered.

The stock and crypto universes are chosen "as of a date" from the bars up to
that date only: funds, ETFs and ETNs are excluded through nasdaqtraded.txt
and the security name, stocks delisted later still count when they traded.
Corporate actions are kept once per source key and found by ex-date.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 15:33:40 +01:00
Alby96andClaude Opus 5.5 b3fb69e696 Add free reference sources and research importers (refs #21 #22)
Readers for the ECB rates, the Forex Factory calendar, FRED release dates,
Finnhub earnings, the Fed/ECB meeting list, fear and greed, the Cboe VIX
history, nasdaqtraded.txt and the Binance kline archives, each tested on
recorded responses. ReferenceSources downloads them within their limits
and records every attempt in source_state, so limits hold across restarts
and the interface can say which source is off and why (FRED and Finnhub
without a key). The central-bank list is an editable, validated setting.

BinanceImporter checks every monthly zip against its SHA-256 file before
writing it to the research history; CsvImporter reads user files with a
configurable column mapping and reports unreadable lines and gaps instead
of fixing them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 15:20:35 +01:00
Alby96andClaude Opus 5.5 956306a8fa Add OKX Europe and Alpaca market data clients (refs #18 #19)
Both clients are written in house on HttpClient and ClientWebSocket, behind
IMarketDataSource, with a rate limiter on the injected clock and retries
that wait on it too.

OKX: EUR pairs only (trap 14), rules from public/instruments, daily
candles with bar=1Dutc and a refusal of any candle not opening at 00:00
UTC (trap 23), backwards paging of history-candles, errors read from the
code field because OKX answers them with HTTP 200, tickers on the public
stream and candles on the business stream with ping after silence. The
HMAC signer and the demo header are ready for Phase 3.

Alpaca: consolidated daily bars, real and adjusted, dated in New York and
never newer than 15 minutes (trap 8), an explicit feed on every request,
IEX for recent minutes, quotes and the stream (at most 30 symbols),
calendar and clock converted to UTC, corporate actions. Without keys
nothing is sent and the reason is stated. Tests use the documentation's
examples and synthetic answers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 15:07:23 +01:00
Alby96andClaude Opus 5.5 3b13a4258d Add market data schema, catalogue, bar storage and quality checks (refs #18 #20 #21 #23)
Migration 0002 adds the market data tables on both engines: daily and
one-minute bars, corporate actions, FX rates, the exchange calendar,
calendar events, mood indicators, spread samples, data issues, source
state and universe members, plus the trading rules on instruments.

The data layer is format-independent so the venue clients can build on
it: the instrument catalogue (rules from the venue, delisted kept), bar
upserts with ON CONFLICT, the SPEC 6.4 checks with an issue register
that blocks entries until a second reading, freshness by age of the last
datum (trap 10), research history in .csv.gz with exact decimals, and
adjusted series anchored at the decision day with level and quantity
conversion across splits and dividends.

Recorded API responses from OKX, the Alpaca docs and the free sources
are kept byte for byte as test fixtures; the secret scan now ignores
pagination cursors such as Alpaca's next_page_token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 14:58:44 +01:00