Fase 7: raccolta, collegamento e primo giudizio delle notizie, con il classificatore ONNX #97

Merged
Alby96 merged 5 commits from fase-7 into main 2026-11-15 15:34:41 +01:00
5 Commits
Author SHA1 Message Date
Alby96andClaude Opus 5.5 1aede2fc1c Let the server test accept the news feeds and the pinned model host (refs #57)
ci / test (postgres) (push) Successful in 11m31s
ci / test (sqlite) (push) Successful in 5m26s
ci / test (postgres) (pull_request) Successful in 10m7s
ci / test (sqlite) (pull_request) Successful in 5m13s
ci / image (push) Successful in 3m12s
ci / image (pull_request) Successful in 1m46s
The news collector runs alongside the market data in the server test, so its
feeds and the Hugging Face download of the classifier can reach the fake network
before the assertion: they are the hosts the code declares, nothing else.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:44:05 +01:00
Alby96andClaude Opus 5.5 78e77818eb Read the tone of English headlines with the pinned ONNX model of SPEC 4.5 (refs #57)
ci / test (postgres) (push) Failing after 6m20s
ci / test (sqlite) (push) Failing after 5m59s
ci / image (push) Skipped
ci / test (postgres) (pull_request) Failing after 4m51s
ci / test (sqlite) (pull_request) Successful in 5m29s
ci / image (pull_request) Skipped
The classifier is distilroberta-finetuned-financial-news-sentiment-analysis in
Xenova's quantized ONNX export, pinned at revision e6953e30 and checked by size and
SHA-256.

- Model download: the model is not in the repository. The news job downloads it in
  the background into the models folder of the data directory, the first time, and
  retries after 6 hours if that fails. Until then the keywords judge alone, as
  SPEC 10.3 requires.
- Tokenizer: RoBERTa's byte-level BPE travels inside the DLL, read with
  BpeTokenizer and the RoBERTa pre-tokenizer. CodeGenTokenizer.Create adds ids for
  runs of spaces that the model rejects. No special tokens are recognised in
  untrusted text.
- Token check: the test compares the ids on 14 headlines with an independent copy
  of the GPT-2 encoder run on the same files.
- Model test: it reads the tone of clear headlines at 23-24 ms per title on one
  thread. It runs where the model is already downloaded and is skipped in CI.

Microsoft.ML.OnnxRuntime 1.31.1 and Microsoft.ML.Tokenizers 2.0.1 come from the
ADR 0004 list. Google.Protobuf joins the known indirect dependencies.

Anubis and Anubis.Server now declare win-x64 among their runtime identifiers, so
their lock files hold the Windows native packages. Publishing the Windows zip no
longer rewrites the lock files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:00:54 +01:00
Alby96andClaude Opus 5.5 5f0ce6b172 Record the news collection and the first judgement in the project state (refs #56 #57)
ci / test (postgres) (pull_request) Successful in 9m37s
ci / test (sqlite) (pull_request) Successful in 4m20s
ci / image (pull_request) Successful in 2m32s
ci / test (sqlite) (push) Successful in 4m24s
ci / test (postgres) (push) Failing after 9m18s
ci / image (push) Skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:57:49 +01:00
Alby96andClaude Opus 5.5 9ee2367693 Judge every article locally: serious cases by keywords, tone grade and importance (refs #57)
This is the first judgement of SPEC 10.2 point 4, without a language model.

- Serious cases: cyberattack, suspension, fraud, bankruptcy, investigation and
  delisting are found by whole-word phrases in English and Italian, so an
  Italian headline still counts.
- Tone: the five tone grades follow the bounds of the specification.
- Importance: 4 for a serious case, 3 for a negative tone, 2 when the article is
  linked to an instrument, otherwise 1.
- Classifier: it sits behind IToneClassifier and is asked only for English
  titles. The ONNX model plugs in there; without it the tone stays unknown and
  everything else works.

Every article stores its score, case and importance. Each story keeps the mean
score, the case and the highest importance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:57:49 +01:00
Alby96andClaude Opus 5.5 f1ba452aba Collect news from the SPEC 4.4 feeds, group retold stories and link them to instruments (refs #56)
News is the first input of Phase 7. Every two minutes the "notizie" job asks each
enabled feed conditionally and backs off a failing one up to an hour.

- Parsing: RSS 2.0, RSS 1.0 and Atom are read without DTDs or entity expansion, so a
  hostile feed reads nothing from disk.
- Deduplication: articles are deduplicated by canonical URL.
- Grouping: retold titles join one story by word similarity within 48 hours, and
  independent publishers are counted by registrable domain.
- Linking: stories are linked to instruments by source symbols, cashtags, exchange
  tickers, coin tickers and capitalised company names; to the crypto or equity class
  or to the market by theme; to the source topic as a fallback.

ADR 0014 records the thresholds. The tests read a synthetic archive in each source's
real format and run on both engines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:57:49 +01:00