Read the tone of English headlines with the pinned ONNX model of SPEC 4.5 (refs #57)
ci / test (postgres) (push) Failing after 6m20s
ci / test (sqlite) (push) Failing after 5m59s
ci / image (push) Skipped
ci / test (postgres) (pull_request) Failing after 4m51s
ci / test (sqlite) (pull_request) Successful in 5m29s
ci / image (pull_request) Skipped
ci / test (postgres) (push) Failing after 6m20s
ci / test (sqlite) (push) Failing after 5m59s
ci / image (push) Skipped
ci / test (postgres) (pull_request) Failing after 4m51s
ci / test (sqlite) (pull_request) Successful in 5m29s
ci / image (pull_request) Skipped
The classifier is distilroberta-finetuned-financial-news-sentiment-analysis in Xenova's quantized ONNX export, pinned at revision e6953e30 and checked by size and SHA-256. - Model download: the model is not in the repository. The news job downloads it in the background into the models folder of the data directory, the first time, and retries after 6 hours if that fails. Until then the keywords judge alone, as SPEC 10.3 requires. - Tokenizer: RoBERTa's byte-level BPE travels inside the DLL, read with BpeTokenizer and the RoBERTa pre-tokenizer. CodeGenTokenizer.Create adds ids for runs of spaces that the model rejects. No special tokens are recognised in untrusted text. - Token check: the test compares the ids on 14 headlines with an independent copy of the GPT-2 encoder run on the same files. - Model test: it reads the tone of clear headlines at 23-24 ms per title on one thread. It runs where the model is already downloaded and is skipped in CI. Microsoft.ML.OnnxRuntime 1.31.1 and Microsoft.ML.Tokenizers 2.0.1 come from the ADR 0004 list. Google.Protobuf joins the known indirect dependencies. Anubis and Anubis.Server now declare win-x64 among their runtime identifiers, so their lock files hold the Windows native packages. Publishing the Windows zip no longer rewrites the lock files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
5f0ce6b172
commit
78e77818eb
21 files changed
+50715
-26
No files matched your search
@@ -11,3 +11,6 @@ deploy/windows/*.txt text eol=crlf
|
||||
*.ttf binary
|
||||
# Recorded API responses stay byte for byte as the source sent them (CRLF in nasdaqtraded.txt included).
|
||||
tests/Anubis.Tests/Fixtures/** -text
|
||||
# The tokenizer of the news classifier stays byte for byte as published (hashes in ADR 0014).
|
||||
src/Anubis/News/Model/*.json -text
|
||||
src/Anubis/News/Model/*.txt -text
|
||||
@@ -16,4 +16,9 @@
|
||||
<!-- Desktop secret key on Windows (DPAPI, SPEC 15.6). -->
|
||||
<PackageVersion Include="System.Security.Cryptography.ProtectedData" Version="10.0.14" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<!-- Local news classifier (SPEC 4.5, phase 7): the ONNX model and its RoBERTa tokenizer. -->
|
||||
<PackageVersion Include="Microsoft.ML.OnnxRuntime" Version="1.31.1" />
|
||||
<PackageVersion Include="Microsoft.ML.Tokenizers" Version="2.0.1" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
+2
-2
@@ -128,7 +128,7 @@ Prima di scrivere ogni adapter, si rilegge la documentazione corrente e si aggio
|
||||
|---|---|---|---|---|
|
||||
| M1 | `mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis`, licenza Apache-2.0 | confermato | — | https://huggingface.co/mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis |
|
||||
| M2 | Esportazione ONNX pronta nei repository `Xenova/...` | confermato | `Xenova/distilroberta-finetuned-financial-news-sentiment-analysis`: `onnx/model.onnx` (circa 329 MB), `onnx/model_quantized.onnx` (circa 83 MB) e `tokenizer.json`. Revisione da fissare: `e6953e30167c7fc3d1ad62f3df31617dd712fbc1`. Il repository **non dichiara una licenza**: vale quella del modello originale (Apache-2.0), da annotare nell'ADR della Fase 7. | https://huggingface.co/Xenova/distilroberta-finetuned-financial-news-sentiment-analysis |
|
||||
| M3 | 10–20 ms per titolo su CPU | non verificabile | Si misura nella Fase 7, con ONNX Runtime per .NET. | — |
|
||||
| M3 | 10–20 ms per titolo su CPU | misurato | Il 15/11/2026, con ONNX Runtime 1.31.1 per .NET e il modello quantizzato su un solo thread: 23–24 ms per titolo sul PC di sviluppo (test `TheModelReadsTheToneOfFinancialHeadlines`). Con più thread scende, ma Anubis ne usa uno (ADR 0014, punto 8). | — |
|
||||
| M4 | Claude Haiku 5.5: 0,10 $ in ingresso e 0,50 $ in uscita per milione di token (fino a 100.000 token); batch a metà prezzo; non ancora nella panoramica dei modelli | cambiato | Ora **è** nella panoramica, con identificativo `claude-haiku-5-5`. Prezzi confermati. Oltre i 100.000 token costa 0,50 $ e 2,50 $. | https://platform.claude.com/docs/en/about-claude/models/overview · https://platform.claude.com/docs/en/about-claude/pricing |
|
||||
| M5 | Claude Haiku 4.5: 1 $ e 5 $ | confermato | Ora è un modello "legacy", ancora disponibile. | https://platform.claude.com/docs/en/about-claude/pricing |
|
||||
|
||||
@@ -265,7 +265,7 @@ Quando la fase indicata aggiunge il pacchetto, il nome entra nell'elenco del tes
|
||||
| `SQLitePCLRaw.bundle_e_sqlite3`, `SQLitePCLRaw.core`, `SQLitePCLRaw.lib.e_sqlite3`, `SQLitePCLRaw.provider.e_sqlite3` 2.1.13 | Eric Sink, Apache-2.0 | `Microsoft.Data.Sqlite` | 1 (in uso) | Contengono la libreria nativa di SQLite e il suo caricatore. Sono nell'elenco del test dalla Fase 1. |
|
||||
| `Newtonsoft.Json` 13.0.3 | James Newton-King, MIT | `Microsoft.ML`, LightGbm, FastTree | 8 | — |
|
||||
| `LightGBM` 4.6.0 | progetto microsoft/LightGBM, pubblicato a nome di Guolin Ke, MIT | `Microsoft.ML.LightGbm` | 8 | Libreria nativa; in Docker serve `libgomp1`. |
|
||||
| `Google.Protobuf` 3.30.2 | Google, BSD-3-Clause | `Microsoft.ML.Tokenizers` | 7 | — |
|
||||
| `Google.Protobuf` 3.30.2 | Google, BSD-3-Clause | `Microsoft.ML.Tokenizers` | 7 (in uso) | Nell'elenco del test dalla Fase 7. |
|
||||
| Mono.Cecil 0.11.6 | Jb Evain, MIT | estensione di copertura del metapacchetto MSTest | — | Evitato: non si usa il metapacchetto. |
|
||||
|
||||
## Immagini Docker
|
||||
|
||||
+4
-4
@@ -1,6 +1,6 @@
|
||||
# Stato del progetto
|
||||
|
||||
Aggiornato: 2026-11-14
|
||||
Aggiornato: 2026-11-15
|
||||
|
||||
## Fase corrente
|
||||
|
||||
@@ -93,7 +93,8 @@ Aggiornato: 2026-11-14
|
||||
- i feed si leggono senza DTD né entità;
|
||||
- gli articoli doppi si tolgono per indirizzo e le riprese dello stesso fatto si raggruppano in una storia, contando le fonti indipendenti;
|
||||
- ogni storia si collega a strumenti, classi o mercato: simboli, ticker, nomi delle società e delle monete, temi.
|
||||
- Primo giudizio locale (#57, in parte): casi gravi trovati con parole chiave in inglese e in italiano, tono in cinque gradi, importanza da 1 a 4. Ogni articolo e ogni storia lo conservano. Il classificatore ONNX si aggancia dietro un'interfaccia: finché manca, il tono resta sconosciuto e il resto funziona.
|
||||
- Primo giudizio locale (#57): casi gravi trovati con parole chiave in inglese e in italiano, tono in cinque gradi, importanza da 1 a 4. Ogni articolo e ogni storia lo conservano.
|
||||
- Classificatore ONNX (#57, ADR 0014 punto 8): il modello Xenova quantizzato, con revisione e hash fissati, scaricato dal bot nella cartella `models` al primo giro delle notizie. Il tokenizzatore sta nella DLL e dà gli stessi identificativi dell'originale. 23–24 ms per titolo su un thread. Senza il modello giudicano le sole parole chiave.
|
||||
|
||||
## In corso
|
||||
|
||||
@@ -102,8 +103,7 @@ Nulla.
|
||||
## Prossimo passo
|
||||
|
||||
1. L'utente scarica lo zip della release `v0.7.0` da Gitea (Release), lo fa girare e poi incolla il rapporto: doppio clic su `Prepara i dati da incollare.cmd`, oppure il pulsante in Diario e sistema. Il rapporto si analizza e si salva in `docs/diagnostica/`; da lì si chiude #55.
|
||||
2. Intanto la Fase 7 va avanti sul branch `fase-7`. Raccolta (#56) e parole chiave (#57) sono fatte. Restano:
|
||||
- il modello ONNX di #57: pacchetti `Microsoft.ML.OnnxRuntime` e `Microsoft.ML.Tokenizers`, modello Xenova con revisione e hash fissati;
|
||||
2. Intanto la Fase 7 va avanti sul branch `fase-7`. Raccolta (#56) e primo giudizio con il classificatore (#57) sono fatti. Restano:
|
||||
- l'analista facoltativo (#58);
|
||||
- lo stato per strumento (#59);
|
||||
- la schermata (#60).
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# 0014 — Scelte della Fase 7: raccolta e collegamento delle notizie
|
||||
|
||||
Data: 2026-11-14 · Stato: accettata · Precisa SPEC 4.4 e 10.2 (punti 1–3) per l'issue #56, senza aggiungere componenti esterni
|
||||
Data: 2026-11-14 · Stato: accettata · Precisa SPEC 4.4, 4.5 e 10.2 (punti 1–4) per le issue #56 e #57. Usa solo i componenti già ammessi dall'ADR 0004
|
||||
|
||||
## Contesto
|
||||
|
||||
@@ -50,8 +50,38 @@ Inoltre:
|
||||
- L'indirizzo canonico è unico.
|
||||
- Un collegamento è unico per storia, ambito e bersaglio.
|
||||
|
||||
**7. Primo giudizio senza modello linguistico (#57).**
|
||||
- Ogni articolo ha:
|
||||
- un tono da −1 a +1, dato solo per i titoli in inglese;
|
||||
- un eventuale caso grave;
|
||||
- un'importanza da 1 a 4.
|
||||
- Ogni storia tiene il tono medio, il caso grave e l'importanza più alta dei suoi articoli.
|
||||
- I casi gravi si trovano con frasi intere in inglese e in italiano, nell'ordine di 10.2: attacco informatico, sospensione, frode, fallimento, indagine, revoca dalla quotazione. "hackathon" non è "hack".
|
||||
- Le negazioni ("smentisce la frode") contano come il caso. È la scelta prudente: una restrizione in più, mai una in meno.
|
||||
- L'importanza si assegna così: 4 ai casi gravi, 3 a un tono fino a −0,6, 2 agli articoli collegati a uno strumento, 1 al resto. Il 3 per un evento in calendario arriva con #59; il 5 lo dà solo il modello linguistico (#58).
|
||||
|
||||
**8. Classificatore locale (#57).**
|
||||
- Il modello è `onnx/model_quantized.onnx` di `Xenova/distilroberta-finetuned-financial-news-sentiment-analysis`:
|
||||
- revisione `e6953e30167c7fc3d1ad62f3df31617dd712fbc1`;
|
||||
- 82.763.251 byte, SHA-256 `8fcf4873bf5e1679c77205ce621acecdd2cd8dd5baf18e930f10428008ed48ec`, lo stesso pubblicato da Hugging Face;
|
||||
- è la versione quantizzata, quattro volte più piccola di `model.onnx`.
|
||||
- Il repository Xenova non dichiara una licenza. Vale quella del modello originale, `mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis`: Apache-2.0.
|
||||
- Il modello non sta nel repository né nello zip. Al primo giro delle notizie, il bot lo scarica in sottofondo nella cartella `models` dei dati, dall'indirizzo con la revisione fissata, e controlla dimensione e hash. Un file che non corrisponde non si usa mai.
|
||||
- Se il download o l'apertura falliscono, si riprova dopo 6 ore. Intanto le notizie arrivano lo stesso e giudicano le sole parole chiave (SPEC 10.3: tutto funziona senza modello).
|
||||
- Nella modalità dimostrativa e con i dati di mercato spenti non si scarica niente.
|
||||
- Il tokenizzatore sta dentro la DLL (`src/Anubis/News/Model/`, hash nel `README.md` della cartella). È il BPE a livello di byte di RoBERTa, letto con `BpeTokenizer` di `Microsoft.ML.Tokenizers` e il pre-tokenizzatore di RoBERTa.
|
||||
- `CodeGenTokenizer.Create` non va bene: aggiunge identificativi per le sequenze di spazi che RoBERTa non ha, e il modello li rifiuta.
|
||||
- Nessun token speciale viene riconosciuto nel testo: un titolo che contiene `<s>` o `<mask>` resta testo.
|
||||
- Il testo si tronca a 128 token: un titolo ci sta sempre.
|
||||
- La prova che gli identificativi coincidono con l'originale: il test li confronta, su 14 titoli con accenti, simboli e spazi, con quelli di una copia indipendente del codificatore di GPT-2 sugli stessi file. "Hello world" dà il noto `0 31414 232 2`.
|
||||
- Il modello gira su un solo thread, per non togliere il processore al motore di trading: 23–24 ms per titolo su questo PC.
|
||||
- È un po' sopra i 10–20 ms di SPEC 4.5, che non dice su quanti thread.
|
||||
- Con un giro ogni 2 minuti e qualche decina di titoli nuovi l'ora, il costo resta trascurabile.
|
||||
- Il test sul modello vero gira dove il file è già scaricato (`ANUBIS_TEST_MODEL_DIR`). In CI risulta ignorato: i test non scaricano 83 MB a ogni esecuzione.
|
||||
- Lo zip per Windows porta in più la libreria nativa di ONNX Runtime per win-x64 (`onnxruntime.dll`, 17 MB) e passa da 3,4 a 10,7 MB. Per questo `Anubis` e `Anubis.Server` dichiarano `win-x64` tra i loro runtime: il file di blocco contiene anche i pacchetti nativi di Windows, e `tasks release-win` non lo modifica più. Che l'immagine Docker carichi la libreria per linux-x64 (SPEC 3, distribuzione Docker) non è ancora provato: si controlla con il primo avvio del classificatore in Docker.
|
||||
|
||||
## Conseguenze
|
||||
|
||||
- Le notizie si accumulano già nel database. Classificazione, stato per strumento e schermata arrivano con le issue #57–#60.
|
||||
- Le notizie si accumulano già nel database con il loro primo giudizio. Analista, stato per strumento e schermata arrivano con le issue #58–#60.
|
||||
- Le soglie di somiglianza e di età sono costanti nel codice. Se l'uso reale mostra storie unite o divise male, si cambiano con un nuovo ADR e un test che lo mostra.
|
||||
- I test leggono un archivio sintetico nel formato reale di ogni fonte (`tests/Anubis.Tests/Fixtures/news/`). Non chiamano mai internet.
|
||||
@@ -2,6 +2,11 @@
|
||||
"version": 2,
|
||||
"dependencies": {
|
||||
"net10.0-windows7.0": {
|
||||
"Google.Protobuf": {
|
||||
"type": "Transitive",
|
||||
"resolved": "3.30.2",
|
||||
"contentHash": "Y2aOVLIt75yeeEWigg9V9YnjsEm53sADtLGq0gLhwaXpk3iu8tYSoauolyhenagA2sWno2TQ2WujI0HQd6s1Vw=="
|
||||
},
|
||||
"Microsoft.Data.Sqlite.Core": {
|
||||
"type": "Transitive",
|
||||
"resolved": "10.0.14",
|
||||
@@ -23,6 +28,14 @@
|
||||
"Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.0"
|
||||
}
|
||||
},
|
||||
"Microsoft.ML.OnnxRuntime.Managed": {
|
||||
"type": "Transitive",
|
||||
"resolved": "1.31.1",
|
||||
"contentHash": "158JCbNXN9/90LtjF4vHFiJYCF0bqF7CxLjm031jq7dJLEn/CtlHpMJm3UItQSIhQenlx+9Zuqi3zulwnsRVtA==",
|
||||
"dependencies": {
|
||||
"System.Numerics.Tensors": "9.0.0"
|
||||
}
|
||||
},
|
||||
"SQLitePCLRaw.bundle_e_sqlite3": {
|
||||
"type": "Transitive",
|
||||
"resolved": "2.1.13",
|
||||
@@ -50,10 +63,17 @@
|
||||
"SQLitePCLRaw.core": "2.1.13"
|
||||
}
|
||||
},
|
||||
"System.Numerics.Tensors": {
|
||||
"type": "Transitive",
|
||||
"resolved": "9.0.0",
|
||||
"contentHash": "hyJB4UlpAi19Xr9AXzu2NuagKC4lPfHObNMEAA0HmqFz2rX7wKgzeYzO/jM/eBHDhnUGFFEjk5cOoJaxqg5J4A=="
|
||||
},
|
||||
"anubis": {
|
||||
"type": "Project",
|
||||
"dependencies": {
|
||||
"Microsoft.Data.Sqlite": "[10.0.14, )",
|
||||
"Microsoft.ML.OnnxRuntime": "[1.31.1, )",
|
||||
"Microsoft.ML.Tokenizers": "[2.0.1, )",
|
||||
"Npgsql": "[10.0.3, )",
|
||||
"System.Security.Cryptography.ProtectedData": "[10.0.14, )"
|
||||
}
|
||||
@@ -69,6 +89,24 @@
|
||||
"SQLitePCLRaw.core": "2.1.13"
|
||||
}
|
||||
},
|
||||
"Microsoft.ML.OnnxRuntime": {
|
||||
"type": "CentralTransitive",
|
||||
"requested": "[1.31.1, )",
|
||||
"resolved": "1.31.1",
|
||||
"contentHash": "XtsO/1JoKepkWbGk9/DcYtRUTduwJD68kw0eeaH5yy77YBivUZWhC1NBNlVecsBt9EitlQIjCd1hHDNXJ2jbQA==",
|
||||
"dependencies": {
|
||||
"Microsoft.ML.OnnxRuntime.Managed": "1.31.1"
|
||||
}
|
||||
},
|
||||
"Microsoft.ML.Tokenizers": {
|
||||
"type": "CentralTransitive",
|
||||
"requested": "[2.0.1, )",
|
||||
"resolved": "2.0.1",
|
||||
"contentHash": "RZN3zslVWinBCMDAnbZWEY8lQbMkP54Yxm1s5NigPcrvYFzXMrDjNcVM9afq4SZHf1BV1rEmO0p7ZdfBLDbu4g==",
|
||||
"dependencies": {
|
||||
"Google.Protobuf": "3.30.2"
|
||||
}
|
||||
},
|
||||
"Npgsql": {
|
||||
"type": "CentralTransitive",
|
||||
"requested": "[10.0.3, )",
|
||||
|
||||
@@ -2,6 +2,11 @@
|
||||
"version": 2,
|
||||
"dependencies": {
|
||||
"net10.0": {
|
||||
"Google.Protobuf": {
|
||||
"type": "Transitive",
|
||||
"resolved": "3.30.2",
|
||||
"contentHash": "Y2aOVLIt75yeeEWigg9V9YnjsEm53sADtLGq0gLhwaXpk3iu8tYSoauolyhenagA2sWno2TQ2WujI0HQd6s1Vw=="
|
||||
},
|
||||
"Microsoft.Data.Sqlite.Core": {
|
||||
"type": "Transitive",
|
||||
"resolved": "10.0.14",
|
||||
@@ -10,17 +15,12 @@
|
||||
"SQLitePCLRaw.core": "2.1.13"
|
||||
}
|
||||
},
|
||||
"Microsoft.Extensions.DependencyInjection.Abstractions": {
|
||||
"Microsoft.ML.OnnxRuntime.Managed": {
|
||||
"type": "Transitive",
|
||||
"resolved": "10.0.0",
|
||||
"contentHash": "L3AdmZ1WOK4XXT5YFPEwyt0ep6l8lGIPs7F5OOBZc77Zqeo01Of7XXICy47628sdVl0v/owxYJTe86DTgFwKCA=="
|
||||
},
|
||||
"Microsoft.Extensions.Logging.Abstractions": {
|
||||
"type": "Transitive",
|
||||
"resolved": "10.0.0",
|
||||
"contentHash": "FU/IfjDfwaMuKr414SSQNTIti/69bHEMb+QKrskRb26oVqpx3lNFXMjs/RC9ZUuhBhcwDM2BwOgoMw+PZ+beqQ==",
|
||||
"resolved": "1.31.1",
|
||||
"contentHash": "158JCbNXN9/90LtjF4vHFiJYCF0bqF7CxLjm031jq7dJLEn/CtlHpMJm3UItQSIhQenlx+9Zuqi3zulwnsRVtA==",
|
||||
"dependencies": {
|
||||
"Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.0"
|
||||
"System.Numerics.Tensors": "9.0.0"
|
||||
}
|
||||
},
|
||||
"SQLitePCLRaw.bundle_e_sqlite3": {
|
||||
@@ -50,10 +50,17 @@
|
||||
"SQLitePCLRaw.core": "2.1.13"
|
||||
}
|
||||
},
|
||||
"System.Numerics.Tensors": {
|
||||
"type": "Transitive",
|
||||
"resolved": "9.0.0",
|
||||
"contentHash": "hyJB4UlpAi19Xr9AXzu2NuagKC4lPfHObNMEAA0HmqFz2rX7wKgzeYzO/jM/eBHDhnUGFFEjk5cOoJaxqg5J4A=="
|
||||
},
|
||||
"anubis": {
|
||||
"type": "Project",
|
||||
"dependencies": {
|
||||
"Microsoft.Data.Sqlite": "[10.0.14, )",
|
||||
"Microsoft.ML.OnnxRuntime": "[1.31.1, )",
|
||||
"Microsoft.ML.Tokenizers": "[2.0.1, )",
|
||||
"Npgsql": "[10.0.3, )",
|
||||
"System.Security.Cryptography.ProtectedData": "[10.0.14, )"
|
||||
}
|
||||
@@ -69,14 +76,29 @@
|
||||
"SQLitePCLRaw.core": "2.1.13"
|
||||
}
|
||||
},
|
||||
"Microsoft.ML.OnnxRuntime": {
|
||||
"type": "CentralTransitive",
|
||||
"requested": "[1.31.1, )",
|
||||
"resolved": "1.31.1",
|
||||
"contentHash": "XtsO/1JoKepkWbGk9/DcYtRUTduwJD68kw0eeaH5yy77YBivUZWhC1NBNlVecsBt9EitlQIjCd1hHDNXJ2jbQA==",
|
||||
"dependencies": {
|
||||
"Microsoft.ML.OnnxRuntime.Managed": "1.31.1"
|
||||
}
|
||||
},
|
||||
"Microsoft.ML.Tokenizers": {
|
||||
"type": "CentralTransitive",
|
||||
"requested": "[2.0.1, )",
|
||||
"resolved": "2.0.1",
|
||||
"contentHash": "RZN3zslVWinBCMDAnbZWEY8lQbMkP54Yxm1s5NigPcrvYFzXMrDjNcVM9afq4SZHf1BV1rEmO0p7ZdfBLDbu4g==",
|
||||
"dependencies": {
|
||||
"Google.Protobuf": "3.30.2"
|
||||
}
|
||||
},
|
||||
"Npgsql": {
|
||||
"type": "CentralTransitive",
|
||||
"requested": "[10.0.3, )",
|
||||
"resolved": "10.0.3",
|
||||
"contentHash": "7nb5YzXuvWWJxB0J8DiyL3we+X4FOctZrt0fIBnucOIaIevFEEwGQVZKtiu9olXdlNAK1eNgqSral6r/jlhI4w==",
|
||||
"dependencies": {
|
||||
"Microsoft.Extensions.Logging.Abstractions": "10.0.0"
|
||||
}
|
||||
"contentHash": "7nb5YzXuvWWJxB0J8DiyL3we+X4FOctZrt0fIBnucOIaIevFEEwGQVZKtiu9olXdlNAK1eNgqSral6r/jlhI4w=="
|
||||
},
|
||||
"System.Security.Cryptography.ProtectedData": {
|
||||
"type": "CentralTransitive",
|
||||
|
||||
@@ -4,6 +4,9 @@
|
||||
<InvariantGlobalization>false</InvariantGlobalization>
|
||||
<!-- Static rendering only: no blazor.web.js, so no Microsoft.AspNetCore.App.Internal.Assets package (ADR 0007). -->
|
||||
<RequiresAspNetWebAssets>false</RequiresAspNetWebAssets>
|
||||
<!-- The Windows trial zip is published for win-x64 (tasks release-win). Declared here, the lock file also holds the
|
||||
native packages of that runtime (ONNX Runtime), so publishing never changes it. -->
|
||||
<RuntimeIdentifiers>win-x64</RuntimeIdentifiers>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\Anubis\Anubis.csproj" />
|
||||
|
||||
@@ -2,6 +2,11 @@
|
||||
"version": 2,
|
||||
"dependencies": {
|
||||
"net10.0": {
|
||||
"Google.Protobuf": {
|
||||
"type": "Transitive",
|
||||
"resolved": "3.30.2",
|
||||
"contentHash": "Y2aOVLIt75yeeEWigg9V9YnjsEm53sADtLGq0gLhwaXpk3iu8tYSoauolyhenagA2sWno2TQ2WujI0HQd6s1Vw=="
|
||||
},
|
||||
"Microsoft.Data.Sqlite.Core": {
|
||||
"type": "Transitive",
|
||||
"resolved": "10.0.14",
|
||||
@@ -10,6 +15,14 @@
|
||||
"SQLitePCLRaw.core": "2.1.13"
|
||||
}
|
||||
},
|
||||
"Microsoft.ML.OnnxRuntime.Managed": {
|
||||
"type": "Transitive",
|
||||
"resolved": "1.31.1",
|
||||
"contentHash": "158JCbNXN9/90LtjF4vHFiJYCF0bqF7CxLjm031jq7dJLEn/CtlHpMJm3UItQSIhQenlx+9Zuqi3zulwnsRVtA==",
|
||||
"dependencies": {
|
||||
"System.Numerics.Tensors": "9.0.0"
|
||||
}
|
||||
},
|
||||
"SQLitePCLRaw.bundle_e_sqlite3": {
|
||||
"type": "Transitive",
|
||||
"resolved": "2.1.13",
|
||||
@@ -37,10 +50,17 @@
|
||||
"SQLitePCLRaw.core": "2.1.13"
|
||||
}
|
||||
},
|
||||
"System.Numerics.Tensors": {
|
||||
"type": "Transitive",
|
||||
"resolved": "9.0.0",
|
||||
"contentHash": "hyJB4UlpAi19Xr9AXzu2NuagKC4lPfHObNMEAA0HmqFz2rX7wKgzeYzO/jM/eBHDhnUGFFEjk5cOoJaxqg5J4A=="
|
||||
},
|
||||
"anubis": {
|
||||
"type": "Project",
|
||||
"dependencies": {
|
||||
"Microsoft.Data.Sqlite": "[10.0.14, )",
|
||||
"Microsoft.ML.OnnxRuntime": "[1.31.1, )",
|
||||
"Microsoft.ML.Tokenizers": "[2.0.1, )",
|
||||
"Npgsql": "[10.0.3, )",
|
||||
"System.Security.Cryptography.ProtectedData": "[10.0.14, )"
|
||||
}
|
||||
@@ -56,6 +76,24 @@
|
||||
"SQLitePCLRaw.core": "2.1.13"
|
||||
}
|
||||
},
|
||||
"Microsoft.ML.OnnxRuntime": {
|
||||
"type": "CentralTransitive",
|
||||
"requested": "[1.31.1, )",
|
||||
"resolved": "1.31.1",
|
||||
"contentHash": "XtsO/1JoKepkWbGk9/DcYtRUTduwJD68kw0eeaH5yy77YBivUZWhC1NBNlVecsBt9EitlQIjCd1hHDNXJ2jbQA==",
|
||||
"dependencies": {
|
||||
"Microsoft.ML.OnnxRuntime.Managed": "1.31.1"
|
||||
}
|
||||
},
|
||||
"Microsoft.ML.Tokenizers": {
|
||||
"type": "CentralTransitive",
|
||||
"requested": "[2.0.1, )",
|
||||
"resolved": "2.0.1",
|
||||
"contentHash": "RZN3zslVWinBCMDAnbZWEY8lQbMkP54Yxm1s5NigPcrvYFzXMrDjNcVM9afq4SZHf1BV1rEmO0p7ZdfBLDbu4g==",
|
||||
"dependencies": {
|
||||
"Google.Protobuf": "3.30.2"
|
||||
}
|
||||
},
|
||||
"Npgsql": {
|
||||
"type": "CentralTransitive",
|
||||
"requested": "[10.0.3, )",
|
||||
@@ -68,6 +106,22 @@
|
||||
"resolved": "10.0.14",
|
||||
"contentHash": "UuaBUHgmtc9h6CMxaQqFDDJxkS5JVy7XEB+DbtCvD68GFcVM9/AccuNwFpARx8nwCnvjaxzoOBjG3gPz32P1Eg=="
|
||||
}
|
||||
},
|
||||
"net10.0/win-x64": {
|
||||
"SQLitePCLRaw.lib.e_sqlite3": {
|
||||
"type": "Transitive",
|
||||
"resolved": "2.1.13",
|
||||
"contentHash": "D2UAuiGxFYBqeRXjjjGgvJ5UQifzn6wefmSqeKU7XcNGKLbJRP6X7kcPZ6opOBJGdCIlgbSxTvDpBbLvsN594A=="
|
||||
},
|
||||
"Microsoft.ML.OnnxRuntime": {
|
||||
"type": "CentralTransitive",
|
||||
"requested": "[1.31.1, )",
|
||||
"resolved": "1.31.1",
|
||||
"contentHash": "XtsO/1JoKepkWbGk9/DcYtRUTduwJD68kw0eeaH5yy77YBivUZWhC1NBNlVecsBt9EitlQIjCd1hHDNXJ2jbQA==",
|
||||
"dependencies": {
|
||||
"Microsoft.ML.OnnxRuntime.Managed": "1.31.1"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2,6 +2,11 @@
|
||||
"version": 2,
|
||||
"dependencies": {
|
||||
"net10.0-windows7.0": {
|
||||
"Google.Protobuf": {
|
||||
"type": "Transitive",
|
||||
"resolved": "3.30.2",
|
||||
"contentHash": "Y2aOVLIt75yeeEWigg9V9YnjsEm53sADtLGq0gLhwaXpk3iu8tYSoauolyhenagA2sWno2TQ2WujI0HQd6s1Vw=="
|
||||
},
|
||||
"Microsoft.Data.Sqlite.Core": {
|
||||
"type": "Transitive",
|
||||
"resolved": "10.0.14",
|
||||
@@ -23,6 +28,14 @@
|
||||
"Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.0"
|
||||
}
|
||||
},
|
||||
"Microsoft.ML.OnnxRuntime.Managed": {
|
||||
"type": "Transitive",
|
||||
"resolved": "1.31.1",
|
||||
"contentHash": "158JCbNXN9/90LtjF4vHFiJYCF0bqF7CxLjm031jq7dJLEn/CtlHpMJm3UItQSIhQenlx+9Zuqi3zulwnsRVtA==",
|
||||
"dependencies": {
|
||||
"System.Numerics.Tensors": "9.0.0"
|
||||
}
|
||||
},
|
||||
"SQLitePCLRaw.bundle_e_sqlite3": {
|
||||
"type": "Transitive",
|
||||
"resolved": "2.1.13",
|
||||
@@ -50,10 +63,17 @@
|
||||
"SQLitePCLRaw.core": "2.1.13"
|
||||
}
|
||||
},
|
||||
"System.Numerics.Tensors": {
|
||||
"type": "Transitive",
|
||||
"resolved": "9.0.0",
|
||||
"contentHash": "hyJB4UlpAi19Xr9AXzu2NuagKC4lPfHObNMEAA0HmqFz2rX7wKgzeYzO/jM/eBHDhnUGFFEjk5cOoJaxqg5J4A=="
|
||||
},
|
||||
"anubis": {
|
||||
"type": "Project",
|
||||
"dependencies": {
|
||||
"Microsoft.Data.Sqlite": "[10.0.14, )",
|
||||
"Microsoft.ML.OnnxRuntime": "[1.31.1, )",
|
||||
"Microsoft.ML.Tokenizers": "[2.0.1, )",
|
||||
"Npgsql": "[10.0.3, )",
|
||||
"System.Security.Cryptography.ProtectedData": "[10.0.14, )"
|
||||
}
|
||||
@@ -69,6 +89,24 @@
|
||||
"SQLitePCLRaw.core": "2.1.13"
|
||||
}
|
||||
},
|
||||
"Microsoft.ML.OnnxRuntime": {
|
||||
"type": "CentralTransitive",
|
||||
"requested": "[1.31.1, )",
|
||||
"resolved": "1.31.1",
|
||||
"contentHash": "XtsO/1JoKepkWbGk9/DcYtRUTduwJD68kw0eeaH5yy77YBivUZWhC1NBNlVecsBt9EitlQIjCd1hHDNXJ2jbQA==",
|
||||
"dependencies": {
|
||||
"Microsoft.ML.OnnxRuntime.Managed": "1.31.1"
|
||||
}
|
||||
},
|
||||
"Microsoft.ML.Tokenizers": {
|
||||
"type": "CentralTransitive",
|
||||
"requested": "[2.0.1, )",
|
||||
"resolved": "2.0.1",
|
||||
"contentHash": "RZN3zslVWinBCMDAnbZWEY8lQbMkP54Yxm1s5NigPcrvYFzXMrDjNcVM9afq4SZHf1BV1rEmO0p7ZdfBLDbu4g==",
|
||||
"dependencies": {
|
||||
"Google.Protobuf": "3.30.2"
|
||||
}
|
||||
},
|
||||
"Npgsql": {
|
||||
"type": "CentralTransitive",
|
||||
"requested": "[10.0.3, )",
|
||||
|
||||
@@ -1,14 +1,21 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
<PropertyGroup>
|
||||
<RootNamespace>Anubis</RootNamespace>
|
||||
<!-- The Windows trial zip is published for win-x64 (tasks release-win). Declared here, the lock file also holds the
|
||||
native packages of that runtime (ONNX Runtime), so publishing never changes it. -->
|
||||
<RuntimeIdentifiers>win-x64</RuntimeIdentifiers>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Npgsql" />
|
||||
<PackageReference Include="Microsoft.Data.Sqlite" />
|
||||
<PackageReference Include="System.Security.Cryptography.ProtectedData" />
|
||||
<PackageReference Include="Microsoft.ML.OnnxRuntime" />
|
||||
<PackageReference Include="Microsoft.ML.Tokenizers" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<EmbeddedResource Include="Db\Migrations\*.sql" LogicalName="Anubis.Db.Migrations.%(Filename)%(Extension)" />
|
||||
<!-- The tokenizer of the news classifier (ADR 0014); the ONNX model is downloaded at the first start. -->
|
||||
<EmbeddedResource Include="News\Model\*.json;News\Model\*.txt" LogicalName="Anubis.News.Model.%(Filename)%(Extension)" />
|
||||
<EmbeddedResource Include="Demo\Data\*" LogicalName="Anubis.Demo.Data.%(Filename)%(Extension)" />
|
||||
<!-- The backtest outcomes travel with the code (SPEC 12.3): read at the first start. -->
|
||||
<EmbeddedResource Include="..\..\docs\backtests\*.outcome.json" LogicalName="Anubis.Backtests.%(Filename)%(Extension)" />
|
||||
|
||||
@@ -82,7 +82,7 @@ public sealed partial class MarketDataService : IDisposable
|
||||
_universes = new UniverseStore(db, clock);
|
||||
_builder = new UniverseBuilder(db, clock);
|
||||
_states = new SourceStateStore(db, clock);
|
||||
News = new News.NewsCollector(_http, db, clock, _logger);
|
||||
News = new News.NewsCollector(_http, db, clock, _logger) { ModelFolder = data is null ? null : Path.Combine(data.Root, "models") };
|
||||
}
|
||||
|
||||
public OkxClient Okx { get; }
|
||||
@@ -153,6 +153,7 @@ public sealed partial class MarketDataService : IDisposable
|
||||
public void Dispose()
|
||||
{
|
||||
_work.Dispose();
|
||||
News.Dispose();
|
||||
_http.Dispose();
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
# Tokenizer of the local news classifier
|
||||
|
||||
`vocab.json` and `merges.txt` are the byte-level BPE tokenizer of RoBERTa. They are used by the model
|
||||
`mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis` (Apache-2.0), as exported to ONNX in
|
||||
`Xenova/distilroberta-finetuned-financial-news-sentiment-analysis`.
|
||||
|
||||
- Revision: `e6953e30167c7fc3d1ad62f3df31617dd712fbc1`.
|
||||
- Downloaded on 2026-11-15 with the user's permission (SPEC 4.5, ADR 0004, ADR 0014).
|
||||
- SHA-256:
|
||||
- `vocab.json`: `ed19656ea1707df69134c4af35c8ceda2cc9860bf2c3495026153a133670ab5e`
|
||||
- `merges.txt`: `1ce1664773c50f3e0cc8842619a93edc4624525b728b188a9e0be33b7726adc5`
|
||||
|
||||
They travel inside the DLL. The ONNX model itself, `onnx/model_quantized.onnx` (82,763,251 bytes, SHA-256
|
||||
`8fcf4873bf5e1679c77205ce621acecdd2cd8dd5baf18e930f10428008ed48ec`), is too large for the repository. The
|
||||
application downloads it once into the `models` folder of the data directory and checks its hash (`ToneModel`).
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1 @@
|
||||
{"<s>":0,"<pad>":1,"</s>":2,"<unk>":3,".":4,"Ġthe":5,",":6,"Ġto":7,"Ġand":8,"Ġof":9,"Ġa":10,"Ġin":11,"-":12,"Ġfor":13,"Ġthat":14,"Ġon":15,"Ġis":16,"âĢ":17,"'s":18,"Ġwith":19,"ĠThe":20,"Ġwas":21,"Ġ\"":22,"Ġat":23,"Ġit":24,"Ġas":25,"Ġsaid":26,"Ļ":27,"Ġbe":28,"s":29,"Ġby":30,"Ġfrom":31,"Ġare":32,"Ġhave":33,"Ġhas":34,":":35,"Ġ(":36,"Ġhe":37,"ĠI":38,"Ġhis":39,"Ġwill":40,"Ġan":41,"Ġthis":42,")":43,"ĠâĢ":44,"Ġnot":45,"Ŀ":46,"Ġyou":47,"ľ":48,"Ġtheir":49,"Ġor":50,"Ġthey":51,"Ġwe":52,"Ġbut":53,"Ġwho":54,"Ġmore":55,"Ġhad":56,"Ġbeen":57,"Ġwere":58,"Ġabout":59,",\"":60,"Ġwhich":61,"Ġup":62,"Ġits":63,"Ġcan":64,"Ġone":65,"Ġout":66,"Ġalso":67,"Ġ$":68,"Ġher":69,"Ġall":70,"Ġafter":71,".\"":72,"/":73,"Ġwould":74,"'t":75,"Ġyear":76,"Ġwhen":77,"Ġfirst":78,"Ġshe":79,"Ġtwo":80,"Ġover":81,"Ġpeople":82,"ĠA":83,"Ġour":84,"ĠIt":85,"Ġtime":86,"Ġthan":87,"Ġinto":88,"Ġthere":89,"t":90,"ĠHe":91,"Ġnew":92,"ĠâĢĶ":93,"Ġlast":94,"Ġjust":95,"ĠIn":96,"Ġother":97,"Ġso":98,"Ġwhat":99,"I":100,"Ġlike":101,"a":102,"Ġsome":103,"S":104,"ë":105,"Ġthem":106,"Ġyears":107,"'":108,"Ġdo":109,"Ġyour":110,"Ġ-":111,"Ġ1":112,"\"":113,"Ġif":114,"Ġcould":115,"?":116,"Ġno":117,"i":118,"m":119,"Ġget":120,"ĠU":121,"Ġnow":122,"Ġhim":123,"Ġback":124,"ĠBut":125,"ĠâĢĵ":126,"Ġmy":127,"Ġ'":128,"Ġonly":129,"Ġthree":130,";":131,"Ġ2":132,"The":133,"1":134,"Ġpercent":135,"Ġagainst":136,"Ġbefore":137,"Ġcompany":138,"o":139,"ĠTrump":140,"Ġhow":141,"Ġbecause":142,"Ġany":143,"Ġmost":144,"Ġbeing":145,"Ġmake":146,"Ġwhere":147,"Ġduring":148,"Ġthrough":149,"Ġwhile":150,"000":151,"ĠThis":152,"Ġmillion":153,"ing":154,"Ġ3":155,"Ġmade":156,"Ġwell":157,"Ġ10":158,"Ġdown":159,"Ġoff":160,"Ġsays":161,"Ġme":162,"ĠB":163,"Ġgoing":164,"Ġteam":165,"ĠWe":166,"Ġthose":167,"Ġgovernment":168,"Ġway":169,"We":170,"Ġmany":171,"Ġthen":172,"Ġwork":173,"Ġtold":174,"com":175,"2":176,"Ġgame":177,"ĠAnd":178,"in":179,"year":180,"Ġp":181,"Ġvery":182,"Ġday":183,"Ġhome":184,"Ġtake":185,"Ġweek":186,"Ġsince":187,"ĠNew":188,"Ġmay":189,"Ġeven":190,"Ġseason":191,"Ġsee":192,"Ġ2017":193,"Ġstate":194,"Ġ5":195,"ed":196,"Ġshould":197,"Ġaround":198,"Ġ2018":199,"Ġsecond":200,"Ġus":201,"Ġstill":202,"Ġmuch":203,"Ġ4":204,"Ġgood":205,"Ġthink":206,"%":207,"ĠS":208,"Ġthese":209,"Ġmarket":210,"ĠD":211,"th":212,"Ġgo":213,"'re":214,"Ġsuch":215,"Ġknow":216,"Ġincluding":217,"Ġdon":218,"y":219,"Ġnext":220,"ĠP":221,"Ġdid":222,"Ġunder":223,"Ġsay":224,"en":225,"ĠL":226,"Ġbetween":227,"Ġper":228,"ĠK":229,"ĠC":230,"Ġ6":231,"Ġworld":232,"Ġpart":233,"ĠN":234,"Ġright":235,"Ġwant":236,"Ġfour":237,"),":238,"Ġhigh":239,"Ġneed":240,"re":241,"e":242,"It":243,"Ġhelp":244,"5":245,"3":246,"Ġcountry":247,"ĠR":248,"Ġpolice":249,"A":250,"Ġlong":251,"ĠThey":252,"Ġend":253,"er":254,"ĠT":255,"ĠM":256,"u":257,"Ġboth":258,"Ġhere":259,"an":260,"on":261,"Ġ7":262,"Ġde":263,"ĠShe":264,"Ġbusiness":265,"Ġreport":266,"j":267,"ers":268,"Ġreally":269,"ĠPresident":270,"ar":271,"ĠG":272,"ĠFriday":273,"ĠF":274,"Ġbest":275,"Ġsame":276,"Ġanother":277,"Ġset":278,"old":279,"ĠThat":280,"as":281,"n":282,"Ġcome":283,"Ġfamily":284,"Ġpublic":285,"ĠFor":286,"ĠAs":287,"0":288,"ĠH":289,"Ġ8":290,"Ġ20":291,"Ġfive":292,"es":293,"ĠTuesday":294,"Ġn":295,"ĠThursday":296,"Ġquarter":297,"h":298,"Ġtop":299,"Ġgot":300,"Ġlife":301,"ĠMonday":302,"Ġfound":303,"Ġuse":304,"ĠW":305,"4":306,"ĠWednesday":307,"Ġown":308,"Ġaccording":309,"Ġplay":310,"Ġshow":311,"ĠSt":312,"Ġman":313,"Ġleft":314,"ĠUnited":315,"Ġ12":316,"Ġplace":317,"ĠIf":318,"Ġlot":319,"Ġformer":320,"Ġ0":321,").":322,"Ġsupport":323,"ie":324,"Ġbillion":325,"Ġt":326,"Ġshares":327,"!":328,"z":329,"k":330,"ĠState":331,"Ġpoints":332,"Ġgroup":333,"Ġschool":334,"Ġinformation":335,"Ġ2016":336,"al":337,"r":338,"Ġwin":339,"Ġnews":340,"Ġused":341,"Ġput":342,"Ġcity":343,"ĠJ":344,"ĠThere":345,"Ġnumber":346,"C":347,"'ve":348,"Ġeach":349,"Ġtoo":350,"Ġwon":351,"ly":352,"Ġmonth":353,"is":354,"Ġadded":355,"Ġlook":356,"Ġbetter":357,"Ġevery":358,"Ġ&":359,"Ġdays":360,"Ġ9":361,"Ġtook":362,"Ġnight":363,"Ġe":364,"Ġ11":365,"os":366,"Ġfew":367,"or":368,"ĠNorth":369,"ĠYou":370,"Ġthird":371,"Ġgreat":372,"Ġcalled":373,"ĠOn":374,"Ġpast":375,"Ġcame":376,"Ġmonths":377,"ĠSaturday":378,"Ġ15":379,"Ġbig":380,"ĠE":381,"ĠUS":382,"Ġthings":383,"ĠO":384,"Ġd":385,"Ġstart":386,"B":387,"Ġstock":388,"Ġ30":389,"Ġwomen":390,"ĠSouth":391,"ĠMay":392,"Ġnever":393,"Ġpresident":394,"ĠSunday":395,"Ġwithout":396,"man":397,"8":398,"Ġdidn":399,"Ġlocal":400,"6":401,"Ġsomething":402,"Ġcase":403,"ĠAll":404,"it":405,"7":406,"ĠSo":407,"Ġchildren":408,"Ġaway":409,"Ġlittle":410,"Ġsix":411,"ĠCity":412,"ĠCounty":413,"Ġdata":414,"at":415,"Ġalready":416,"d":417,"Ġmoney":418,"Ġearly":419,"Ġacross":420,"Ġexpected":421,"Ġrun":422,"Ġlater":423,"am":424,"ĠpricLine truncated
|
||||
@@ -5,6 +5,7 @@ using Anubis.Data;
|
||||
using Anubis.Db;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.ML.OnnxRuntime;
|
||||
|
||||
namespace Anubis.News;
|
||||
|
||||
@@ -13,7 +14,7 @@ namespace Anubis.News;
|
||||
/// conditional on what the source said the last time (ETag, Last-Modified), and spaced out more and more for a source
|
||||
/// that keeps failing. The text it reads is untrusted data: it is parsed, never followed.
|
||||
/// </summary>
|
||||
public sealed partial class NewsCollector(HttpClient http, Database db, IClock clock, ILogger? logger = null)
|
||||
public sealed partial class NewsCollector(HttpClient http, Database db, IClock clock, ILogger? logger = null) : IDisposable
|
||||
{
|
||||
public const string JobName = "notizie";
|
||||
|
||||
@@ -35,14 +36,69 @@ public sealed partial class NewsCollector(HttpClient http, Database db, IClock c
|
||||
|
||||
public NewsStore Store => _store;
|
||||
|
||||
/// <summary>The local classifier of SPEC 4.5; without it the tone is unknown and only the keywords judge.</summary>
|
||||
public IToneClassifier? Classifier { get; init; }
|
||||
/// <summary>A model that could not be downloaded or opened is tried again after this long.</summary>
|
||||
public static readonly TimeSpan ModelRetry = TimeSpan.FromHours(6);
|
||||
|
||||
private readonly CancellationTokenSource _stopping = new();
|
||||
private IToneClassifier? _classifier;
|
||||
private Task? _loading;
|
||||
private DateTimeOffset _nextModelTry = DateTimeOffset.MinValue;
|
||||
|
||||
/// <summary>
|
||||
/// The local classifier of SPEC 4.5. Without it the tone is unknown and only the keywords judge: the news keep
|
||||
/// coming while the model downloads, and the articles read meanwhile simply have no score.
|
||||
/// </summary>
|
||||
public IToneClassifier? Classifier
|
||||
{
|
||||
get => Volatile.Read(ref _classifier);
|
||||
init => _classifier = value;
|
||||
}
|
||||
|
||||
/// <summary>Where the model is kept (the <c>models</c> folder of the data directory); null: no model, as in the tests.</summary>
|
||||
public string? ModelFolder { get; init; }
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
_stopping.Cancel();
|
||||
_stopping.Dispose();
|
||||
(Volatile.Read(ref _classifier) as IDisposable)?.Dispose();
|
||||
}
|
||||
|
||||
/// <summary>Starts, in the background, the download and the opening of the model, once and then every 6 hours if it fails.</summary>
|
||||
private void StartModel()
|
||||
{
|
||||
if (Classifier is not null || ModelFolder is not { } folder || _loading is { IsCompleted: false } || clock.UtcNow < _nextModelTry)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
_nextModelTry = clock.UtcNow + ModelRetry;
|
||||
var stopping = _stopping.Token;
|
||||
_loading = Task.Run(async () =>
|
||||
{
|
||||
try
|
||||
{
|
||||
var path = await ToneModel.EnsureAsync(http, folder, stopping).ConfigureAwait(false);
|
||||
Volatile.Write(ref _classifier, new OnnxToneClassifier(path));
|
||||
LogModelReady(_logger, ToneModel.Revision[..8]);
|
||||
}
|
||||
catch (OperationCanceledException) when (stopping.IsCancellationRequested)
|
||||
{
|
||||
}
|
||||
catch (Exception e) when (e is HttpRequestException or IOException or InvalidDataException or OperationCanceledException or OnnxRuntimeException
|
||||
or UnauthorizedAccessException)
|
||||
{
|
||||
LogModelFailed(_logger, e.Message);
|
||||
}
|
||||
}, CancellationToken.None);
|
||||
}
|
||||
|
||||
public ScheduledJob Job() => new(JobName, "Lettura delle notizie", Schedule.Every(Interval), RunOnceAsync) { MaxLateness = Interval };
|
||||
|
||||
/// <summary>One round over the sources that are due.</summary>
|
||||
public async Task RunOnceAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
StartModel();
|
||||
await _store.EnsureDefaultsAsync(cancellationToken).ConfigureAwait(false);
|
||||
var dictionary = await DictionaryAsync(cancellationToken).ConfigureAwait(false);
|
||||
var first = true;
|
||||
@@ -167,4 +223,10 @@ public sealed partial class NewsCollector(HttpClient http, Database db, IClock c
|
||||
|
||||
[LoggerMessage(Level = LogLevel.Warning, Message = "News source {Source} failed: {Error}")]
|
||||
private static partial void LogFailed(ILogger logger, string source, string error);
|
||||
|
||||
[LoggerMessage(Level = LogLevel.Information, Message = "News classifier ready (model revision {Revision})")]
|
||||
private static partial void LogModelReady(ILogger logger, string revision);
|
||||
|
||||
[LoggerMessage(Level = LogLevel.Warning, Message = "News classifier not available, keywords only until the next try: {Error}")]
|
||||
private static partial void LogModelFailed(ILogger logger, string error);
|
||||
}
|
||||
@@ -0,0 +1,199 @@
|
||||
using System.Security.Cryptography;
|
||||
using System.Text.Json;
|
||||
using Microsoft.ML.OnnxRuntime;
|
||||
using Microsoft.ML.Tokenizers;
|
||||
|
||||
namespace Anubis.News;
|
||||
|
||||
/// <summary>
|
||||
/// The model of SPEC 4.5: <c>mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis</c> (Apache-2.0) in its
|
||||
/// ONNX export by Xenova, at a fixed revision and with its hash checked (ADR 0004, ADR 0014). The tokenizer travels in
|
||||
/// the DLL; the model, 83 MB, is downloaded once into the <c>models</c> folder of the data directory.
|
||||
/// </summary>
|
||||
public static class ToneModel
|
||||
{
|
||||
public const string Repository = "Xenova/distilroberta-finetuned-financial-news-sentiment-analysis";
|
||||
public const string Revision = "e6953e30167c7fc3d1ad62f3df31617dd712fbc1";
|
||||
public const string RemotePath = "onnx/model_quantized.onnx";
|
||||
public const string FileName = "distilroberta-financial-news-sentiment-e6953e30-quantized.onnx";
|
||||
public const long Size = 82_763_251;
|
||||
public const string Sha256 = "8fcf4873bf5e1679c77205ce621acecdd2cd8dd5baf18e930f10428008ed48ec";
|
||||
|
||||
public static readonly Uri Url = new($"https://huggingface.co/{Repository}/resolve/{Revision}/{RemotePath}");
|
||||
|
||||
/// <summary>The longest download allowed: slow connections get there, a stuck one does not hold the job forever.</summary>
|
||||
public static readonly TimeSpan DownloadTimeout = TimeSpan.FromMinutes(30);
|
||||
|
||||
/// <summary>
|
||||
/// The path of the model in <paramref name="folder"/>, downloaded and checked if it is not there yet. A file of the
|
||||
/// wrong size or hash is never used: it is replaced. Throws when the download fails or does not match.
|
||||
/// </summary>
|
||||
public static async Task<string> EnsureAsync(HttpClient http, string folder, CancellationToken cancellationToken)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(http);
|
||||
Directory.CreateDirectory(folder);
|
||||
var path = Path.Combine(folder, FileName);
|
||||
if (File.Exists(path) && new FileInfo(path).Length == Size && await HashAsync(path, cancellationToken).ConfigureAwait(false) == Sha256)
|
||||
{
|
||||
return path;
|
||||
}
|
||||
|
||||
var part = path + ".part";
|
||||
using (var timeout = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken))
|
||||
{
|
||||
timeout.CancelAfter(DownloadTimeout);
|
||||
using var response = await http.GetAsync(Url, HttpCompletionOption.ResponseHeadersRead, timeout.Token).ConfigureAwait(false);
|
||||
response.EnsureSuccessStatusCode();
|
||||
var target = new FileStream(part, FileMode.Create, FileAccess.Write, FileShare.None, 1 << 16, useAsync: true);
|
||||
await using (target.ConfigureAwait(false))
|
||||
{
|
||||
var source = await response.Content.ReadAsStreamAsync(timeout.Token).ConfigureAwait(false);
|
||||
await using (source.ConfigureAwait(false))
|
||||
{
|
||||
var buffer = new byte[1 << 16];
|
||||
long total = 0;
|
||||
int read;
|
||||
while ((read = await source.ReadAsync(buffer, timeout.Token).ConfigureAwait(false)) > 0)
|
||||
{
|
||||
total += read;
|
||||
if (total > Size)
|
||||
{
|
||||
throw new InvalidDataException($"The model is larger than the {Size} bytes expected.");
|
||||
}
|
||||
|
||||
await target.WriteAsync(buffer.AsMemory(0, read), timeout.Token).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var hash = await HashAsync(part, cancellationToken).ConfigureAwait(false);
|
||||
if (new FileInfo(part).Length != Size || hash != Sha256)
|
||||
{
|
||||
File.Delete(part);
|
||||
throw new InvalidDataException($"The downloaded model does not match: SHA-256 {hash}, expected {Sha256}.");
|
||||
}
|
||||
|
||||
File.Move(part, path, overwrite: true);
|
||||
return path;
|
||||
}
|
||||
|
||||
private static async Task<string> HashAsync(string path, CancellationToken cancellationToken)
|
||||
{
|
||||
var file = File.OpenRead(path);
|
||||
await using (file.ConfigureAwait(false))
|
||||
{
|
||||
return Convert.ToHexStringLower(await SHA256.HashDataAsync(file, cancellationToken).ConfigureAwait(false));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The local classifier of SPEC 4.5 on ONNX Runtime. The tokenizer is RoBERTa's byte-level BPE, the one of GPT-2 with
|
||||
/// <c><s></c> and <c></s></c> around the text; the test compares its ids with the original. One thread, so the
|
||||
/// classifier never takes the processor away from the trading engine; 10–20 ms per title.
|
||||
/// </summary>
|
||||
public sealed class OnnxToneClassifier : IToneClassifier, IDisposable
|
||||
{
|
||||
/// <summary>Titles are short: 128 tokens hold any of them, and the model's limit is 512.</summary>
|
||||
public const int MaxTokens = 128;
|
||||
|
||||
private const int Start = 0;
|
||||
private const int End = 2;
|
||||
|
||||
// Classes of the model (config.json): 0 negative, 1 neutral, 2 positive.
|
||||
private const int NegativeClass = 0;
|
||||
private const int NeutralClass = 1;
|
||||
private const int PositiveClass = 2;
|
||||
|
||||
private static readonly Lazy<BpeTokenizer> Tokenizer = new(LoadTokenizer);
|
||||
|
||||
private readonly InferenceSession _session;
|
||||
private readonly string[] _inputs;
|
||||
private readonly string[] _output;
|
||||
|
||||
public OnnxToneClassifier(string modelPath)
|
||||
{
|
||||
using var options = new SessionOptions
|
||||
{
|
||||
IntraOpNumThreads = 1,
|
||||
InterOpNumThreads = 1,
|
||||
GraphOptimizationLevel = GraphOptimizationLevel.ORT_ENABLE_ALL,
|
||||
};
|
||||
_session = new InferenceSession(modelPath, options);
|
||||
_inputs = [.. _session.InputMetadata.Keys];
|
||||
_output = [_session.OutputMetadata.Keys.First()];
|
||||
if (!_inputs.Contains("input_ids"))
|
||||
{
|
||||
_session.Dispose();
|
||||
throw new InvalidDataException("The model has no input_ids input.");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>The token ids the model reads, as the original RoBERTa tokenizer gives them.</summary>
|
||||
public static IReadOnlyList<int> Ids(string text)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(text);
|
||||
var body = Tokenizer.Value.EncodeToIds(text);
|
||||
var ids = new List<int>(Math.Min(body.Count, MaxTokens - 2) + 2) { Start };
|
||||
ids.AddRange(body.Take(MaxTokens - 2));
|
||||
ids.Add(End);
|
||||
return ids;
|
||||
}
|
||||
|
||||
public ToneScore Classify(string text)
|
||||
{
|
||||
var ids = Ids(text);
|
||||
var shape = new long[] { 1, ids.Count };
|
||||
var values = new List<OrtValue>(_inputs.Length);
|
||||
try
|
||||
{
|
||||
foreach (var name in _inputs)
|
||||
{
|
||||
var data = name switch
|
||||
{
|
||||
"input_ids" => ids.Select(i => (long)i).ToArray(),
|
||||
"attention_mask" => Enumerable.Repeat(1L, ids.Count).ToArray(),
|
||||
_ => new long[ids.Count],
|
||||
};
|
||||
values.Add(OrtValue.CreateTensorValueFromMemory(data, shape));
|
||||
}
|
||||
|
||||
using var run = new RunOptions();
|
||||
using var outputs = _session.Run(run, _inputs, values, _output);
|
||||
var logits = outputs[0].GetTensorDataAsSpan<float>();
|
||||
var max = Math.Max(logits[NegativeClass], Math.Max(logits[NeutralClass], logits[PositiveClass]));
|
||||
double negative = Math.Exp(logits[NegativeClass] - max), neutral = Math.Exp(logits[NeutralClass] - max), positive = Math.Exp(logits[PositiveClass] - max);
|
||||
var sum = negative + neutral + positive;
|
||||
return new ToneScore(positive / sum, negative / sum, neutral / sum);
|
||||
}
|
||||
finally
|
||||
{
|
||||
foreach (var value in values)
|
||||
{
|
||||
value.Dispose();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public void Dispose() => _session.Dispose();
|
||||
|
||||
private static BpeTokenizer LoadTokenizer()
|
||||
{
|
||||
var assembly = typeof(OnnxToneClassifier).Assembly;
|
||||
using var vocab = assembly.GetManifestResourceStream("Anubis.News.Model.vocab.json") ?? throw new InvalidOperationException("vocab.json is missing.");
|
||||
using var merges = assembly.GetManifestResourceStream("Anubis.News.Model.merges.txt") ?? throw new InvalidOperationException("merges.txt is missing.");
|
||||
var vocabulary = JsonSerializer.Deserialize<Dictionary<string, int>>(vocab) ?? throw new InvalidDataException("vocab.json is empty.");
|
||||
using var reader = new StreamReader(merges);
|
||||
var pairs = reader.ReadToEnd().Split('\n').Where(l => l.Length > 0 && !l.StartsWith("#version", StringComparison.Ordinal)).ToList();
|
||||
|
||||
// Not CodeGenTokenizer.Create: it adds CodeGen's own tokens for runs of spaces, ids RoBERTa does not have. No special
|
||||
// tokens either: a headline that contains "<s>" or "<mask>" is read as text, never as a mark for the model.
|
||||
return BpeTokenizer.Create(new BpeOptions(vocabulary)
|
||||
{
|
||||
Merges = pairs,
|
||||
PreTokenizer = RobertaPreTokenizer.Instance,
|
||||
ByteLevel = true,
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -13,6 +13,24 @@
|
||||
"SQLitePCLRaw.core": "2.1.13"
|
||||
}
|
||||
},
|
||||
"Microsoft.ML.OnnxRuntime": {
|
||||
"type": "Direct",
|
||||
"requested": "[1.31.1, )",
|
||||
"resolved": "1.31.1",
|
||||
"contentHash": "XtsO/1JoKepkWbGk9/DcYtRUTduwJD68kw0eeaH5yy77YBivUZWhC1NBNlVecsBt9EitlQIjCd1hHDNXJ2jbQA==",
|
||||
"dependencies": {
|
||||
"Microsoft.ML.OnnxRuntime.Managed": "1.31.1"
|
||||
}
|
||||
},
|
||||
"Microsoft.ML.Tokenizers": {
|
||||
"type": "Direct",
|
||||
"requested": "[2.0.1, )",
|
||||
"resolved": "2.0.1",
|
||||
"contentHash": "RZN3zslVWinBCMDAnbZWEY8lQbMkP54Yxm1s5NigPcrvYFzXMrDjNcVM9afq4SZHf1BV1rEmO0p7ZdfBLDbu4g==",
|
||||
"dependencies": {
|
||||
"Google.Protobuf": "3.30.2"
|
||||
}
|
||||
},
|
||||
"Npgsql": {
|
||||
"type": "Direct",
|
||||
"requested": "[10.0.3, )",
|
||||
@@ -28,6 +46,11 @@
|
||||
"resolved": "10.0.14",
|
||||
"contentHash": "UuaBUHgmtc9h6CMxaQqFDDJxkS5JVy7XEB+DbtCvD68GFcVM9/AccuNwFpARx8nwCnvjaxzoOBjG3gPz32P1Eg=="
|
||||
},
|
||||
"Google.Protobuf": {
|
||||
"type": "Transitive",
|
||||
"resolved": "3.30.2",
|
||||
"contentHash": "Y2aOVLIt75yeeEWigg9V9YnjsEm53sADtLGq0gLhwaXpk3iu8tYSoauolyhenagA2sWno2TQ2WujI0HQd6s1Vw=="
|
||||
},
|
||||
"Microsoft.Data.Sqlite.Core": {
|
||||
"type": "Transitive",
|
||||
"resolved": "10.0.14",
|
||||
@@ -49,6 +72,14 @@
|
||||
"Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.0"
|
||||
}
|
||||
},
|
||||
"Microsoft.ML.OnnxRuntime.Managed": {
|
||||
"type": "Transitive",
|
||||
"resolved": "1.31.1",
|
||||
"contentHash": "158JCbNXN9/90LtjF4vHFiJYCF0bqF7CxLjm031jq7dJLEn/CtlHpMJm3UItQSIhQenlx+9Zuqi3zulwnsRVtA==",
|
||||
"dependencies": {
|
||||
"System.Numerics.Tensors": "9.0.0"
|
||||
}
|
||||
},
|
||||
"SQLitePCLRaw.bundle_e_sqlite3": {
|
||||
"type": "Transitive",
|
||||
"resolved": "2.1.13",
|
||||
@@ -75,6 +106,27 @@
|
||||
"dependencies": {
|
||||
"SQLitePCLRaw.core": "2.1.13"
|
||||
}
|
||||
},
|
||||
"System.Numerics.Tensors": {
|
||||
"type": "Transitive",
|
||||
"resolved": "9.0.0",
|
||||
"contentHash": "hyJB4UlpAi19Xr9AXzu2NuagKC4lPfHObNMEAA0HmqFz2rX7wKgzeYzO/jM/eBHDhnUGFFEjk5cOoJaxqg5J4A=="
|
||||
}
|
||||
},
|
||||
"net10.0/win-x64": {
|
||||
"Microsoft.ML.OnnxRuntime": {
|
||||
"type": "Direct",
|
||||
"requested": "[1.31.1, )",
|
||||
"resolved": "1.31.1",
|
||||
"contentHash": "XtsO/1JoKepkWbGk9/DcYtRUTduwJD68kw0eeaH5yy77YBivUZWhC1NBNlVecsBt9EitlQIjCd1hHDNXJ2jbQA==",
|
||||
"dependencies": {
|
||||
"Microsoft.ML.OnnxRuntime.Managed": "1.31.1"
|
||||
}
|
||||
},
|
||||
"SQLitePCLRaw.lib.e_sqlite3": {
|
||||
"type": "Transitive",
|
||||
"resolved": "2.1.13",
|
||||
"contentHash": "D2UAuiGxFYBqeRXjjjGgvJ5UQifzn6wefmSqeKU7XcNGKLbJRP6X7kcPZ6opOBJGdCIlgbSxTvDpBbLvsN594A=="
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -35,6 +35,9 @@ public sealed class DependencyPolicyTests
|
||||
"SQLitePCLRaw.core",
|
||||
"SQLitePCLRaw.lib.e_sqlite3",
|
||||
"SQLitePCLRaw.provider.e_sqlite3",
|
||||
|
||||
// Protocol buffers, brought by Microsoft.ML.Tokenizers (phase 7).
|
||||
"Google.Protobuf",
|
||||
};
|
||||
|
||||
private static readonly string[] _packageElements = ["PackageReference", "GlobalPackageReference", "PackageDownload"];
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
using System.Diagnostics;
|
||||
using System.Net;
|
||||
using Anubis.News;
|
||||
|
||||
namespace Anubis.Tests;
|
||||
|
||||
/// <summary>The local classifier of SPEC 4.5 (issue #57): the same tokens as the original, the model checked by its hash.</summary>
|
||||
[TestClass]
|
||||
public sealed class ToneModelTests
|
||||
{
|
||||
/// <summary>
|
||||
/// Ids of the original RoBERTa tokenizer on the same vocab.json and merges.txt, computed with an independent copy of
|
||||
/// the GPT-2 byte-level encoder (openai/gpt-2, encoder.py), reading the text as UTF-8. "Hello world" is the well-known
|
||||
/// 0 31414 232 2.
|
||||
/// </summary>
|
||||
private static readonly (string Text, int[] Ids)[] Original =
|
||||
[
|
||||
("Hello world", [0, 31414, 232, 2]),
|
||||
("Apple shares jump after record iPhone sales in China", [0, 20770, 327, 3704, 71, 638, 2733, 647, 11, 436, 2]),
|
||||
("Bitcoin drifts below $90,000 as traders await U.S. data", [0, 42817, 12402, 22833, 874, 68, 3248, 6, 151, 25, 5539, 17396, 121, 4, 104, 4, 414, 2]),
|
||||
("Fire at a chip plant hits Nvidia supply, $NVDA slides", [0, 17260, 23, 10, 6638, 2195, 2323, 18520, 1787, 6, 68, 31668, 3134, 19583, 2]),
|
||||
("Fed officials signal patience on interest rates into December", [0, 38186, 503, 6029, 11383, 15, 773, 1162, 88, 719, 2]),
|
||||
("Exchange halts withdrawals after hack, Solana tokens drained", [0, 9089, 14035, 12193, 1872, 27733, 71, 14157, 6, 4856, 1113, 22121, 23544, 2]),
|
||||
("Tesla recalls vehicles over a software issue (NASDAQ:TSLA)", [0, 39971, 10270, 1734, 81, 10, 2257, 696, 36, 3399, 3223, 35, 2685, 8272, 43, 2]),
|
||||
("Microsoft & OpenAI sign a new cloud deal", [0, 35460, 359, 2117, 15238, 1203, 10, 92, 3613, 432, 2]),
|
||||
("Target cuts its holiday outlook as shoppers pull back", [0, 41858, 2599, 63, 2317, 3839, 25, 9456, 2999, 124, 2]),
|
||||
("It's the company's 3rd quarter: EPS beat by 12%, revenue up 7.5%", [0, 243, 18, 5, 138, 18, 155, 2586, 297, 35, 4721, 1451, 30, 316, 4234, 903, 62, 262, 4, 245, 207, 2]),
|
||||
(" Spaces and\ttabs ", [0, 1437, 39926, 1437, 1437, 8, 50117, 34108, 29, 1437, 1437, 2]),
|
||||
("L'inflazione è vicina all'obiettivo", [0, 574, 108, 179, 4825, 1222, 8525, 952, 11423, 32438, 1243, 70, 108, 2413, 5810, 90, 9697, 2]),
|
||||
("Société Générale cède 3,5% à Paris", [0, 104, 17641, 10221, 1140, 272, 10598, 20765, 1627, 740, 8025, 2794, 155, 6, 245, 207, 6534, 2201, 2]),
|
||||
("Nasdaq ➜ record — €1.2bn deal", [0, 487, 281, 5946, 14333, 17772, 48, 638, 93, 4480, 134, 4, 176, 6316, 432, 2]),
|
||||
];
|
||||
|
||||
[TestMethod]
|
||||
public void TheTokenizerGivesTheIdsOfTheOriginal()
|
||||
{
|
||||
foreach (var (text, ids) in Original)
|
||||
{
|
||||
CollectionAssert.AreEqual(ids, OnnxToneClassifier.Ids(text).ToArray(), text);
|
||||
}
|
||||
|
||||
var marks = OnnxToneClassifier.Ids("<s> buy </s> <mask>");
|
||||
Assert.AreEqual(1, marks.Count(i => i == 0), "Marks written in a headline are text, not marks.");
|
||||
Assert.AreEqual(1, marks.Count(i => i == 2));
|
||||
Assert.IsTrue(OnnxToneClassifier.Ids("\u0000 \u2603 \U0001F680 \u4E2D\u6587 \n\r").All(i => i is >= 0 and < 50265), "Any text gives ids of the vocabulary.");
|
||||
|
||||
var longest = OnnxToneClassifier.Ids(string.Join(' ', Enumerable.Repeat("word", 500)));
|
||||
Assert.HasCount(OnnxToneClassifier.MaxTokens, longest, "A long text is cut, keeping the start and the end marks.");
|
||||
Assert.AreEqual(0, longest[0]);
|
||||
Assert.AreEqual(2, longest[^1]);
|
||||
}
|
||||
|
||||
[TestMethod]
|
||||
public async Task AModelThatDoesNotMatchItsHashIsNeverKept()
|
||||
{
|
||||
var folder = TempFolder.Create("model");
|
||||
try
|
||||
{
|
||||
var http = new FakeHttp().On(u => u.Host == "huggingface.co", _ => new HttpResponseMessage(HttpStatusCode.OK) { Content = new ByteArrayContent(new byte[4096]) });
|
||||
await File.WriteAllBytesAsync(Path.Combine(folder, ToneModel.FileName), new byte[10]).ConfigureAwait(false);
|
||||
await Assert.ThrowsExactlyAsync<InvalidDataException>(() => ToneModel.EnsureAsync(http.Client(), folder, CancellationToken.None)).ConfigureAwait(false);
|
||||
Assert.AreEqual(ToneModel.Url, http.Requests.Single().RequestUri, "The pinned revision, never a moving branch.");
|
||||
Assert.IsFalse(File.Exists(Path.Combine(folder, ToneModel.FileName + ".part")), "A wrong download is deleted.");
|
||||
Assert.AreEqual(10, new FileInfo(Path.Combine(folder, ToneModel.FileName)).Length, "The old file is not taken for a good one: it stays unused.");
|
||||
|
||||
var failing = new FakeHttp().On(u => true, _ => new HttpResponseMessage(HttpStatusCode.NotFound));
|
||||
await Assert.ThrowsExactlyAsync<HttpRequestException>(() => ToneModel.EnsureAsync(failing.Client(), folder, CancellationToken.None)).ConfigureAwait(false);
|
||||
}
|
||||
finally
|
||||
{
|
||||
TempFolder.Delete(folder);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The real model, 83 MB: it runs where it has been downloaded once (ANUBIS_TEST_MODEL_DIR, a folder with the file
|
||||
/// named as in <see cref="ToneModel.FileName"/>), never downloaded by the tests themselves.
|
||||
/// </summary>
|
||||
[TestMethod]
|
||||
public async Task TheModelReadsTheToneOfFinancialHeadlines()
|
||||
{
|
||||
if (Environment.GetEnvironmentVariable("ANUBIS_TEST_MODEL_DIR") is not { Length: > 0 } folder || !File.Exists(Path.Combine(folder, ToneModel.FileName)))
|
||||
{
|
||||
Assert.Inconclusive("Il modello ONNX non è scaricato: impostare ANUBIS_TEST_MODEL_DIR sulla cartella che lo contiene.");
|
||||
return;
|
||||
}
|
||||
|
||||
var offline = new FakeHttp();
|
||||
var path = await ToneModel.EnsureAsync(offline.Client(), folder, CancellationToken.None).ConfigureAwait(false);
|
||||
Assert.IsEmpty(offline.Requests, "A model with the right hash is used as it is.");
|
||||
using var classifier = new OnnxToneClassifier(path);
|
||||
Tone Grade(string title) => NewsJudge.Grade(classifier.Classify(title).Score);
|
||||
|
||||
Assert.IsGreaterThan(0, (int)Grade("Apple shares jump after record iPhone sales in China"));
|
||||
Assert.IsGreaterThan(0, (int)Grade("Microsoft beats estimates and raises its full-year outlook"));
|
||||
Assert.IsLessThan(0, (int)Grade("Exchange halts withdrawals after hack, Solana tokens drained"));
|
||||
Assert.IsLessThan(0, (int)Grade("Target cuts its holiday outlook as shoppers pull back"));
|
||||
Assert.IsLessThan(0, (int)Grade("Tesla shares plunge after a wide miss on deliveries"));
|
||||
Assert.AreEqual(Tone.Neutral, Grade("The company will report its results on Thursday"));
|
||||
var score = classifier.Classify("Apple shares jump");
|
||||
Assert.AreEqual(1, score.Positive + score.Negative + score.Neutral, 1e-6);
|
||||
|
||||
var titles = Original.Select(o => o.Text).ToArray();
|
||||
var watch = Stopwatch.StartNew();
|
||||
for (var round = 0; round < 5; round++)
|
||||
{
|
||||
foreach (var title in titles)
|
||||
{
|
||||
classifier.Classify(title);
|
||||
}
|
||||
}
|
||||
|
||||
var perTitle = watch.Elapsed.TotalMilliseconds / (5 * titles.Length);
|
||||
Console.WriteLine($"Classificatore: {perTitle:F1} ms per titolo su un thread.");
|
||||
Assert.IsLessThan(100, perTitle, "SPEC 4.5 expects 10–20 ms per title on a CPU; a slow test machine gets five times that.");
|
||||
}
|
||||
}
|
||||
@@ -34,6 +34,11 @@
|
||||
"MSTest.Analyzers": "4.5.2"
|
||||
}
|
||||
},
|
||||
"Google.Protobuf": {
|
||||
"type": "Transitive",
|
||||
"resolved": "3.30.2",
|
||||
"contentHash": "Y2aOVLIt75yeeEWigg9V9YnjsEm53sADtLGq0gLhwaXpk3iu8tYSoauolyhenagA2sWno2TQ2WujI0HQd6s1Vw=="
|
||||
},
|
||||
"Microsoft.ApplicationInsights": {
|
||||
"type": "Transitive",
|
||||
"resolved": "2.23.0",
|
||||
@@ -65,6 +70,14 @@
|
||||
"Microsoft.Extensions.DependencyInjection.Abstractions": "10.0.0"
|
||||
}
|
||||
},
|
||||
"Microsoft.ML.OnnxRuntime.Managed": {
|
||||
"type": "Transitive",
|
||||
"resolved": "1.31.1",
|
||||
"contentHash": "158JCbNXN9/90LtjF4vHFiJYCF0bqF7CxLjm031jq7dJLEn/CtlHpMJm3UItQSIhQenlx+9Zuqi3zulwnsRVtA==",
|
||||
"dependencies": {
|
||||
"System.Numerics.Tensors": "9.0.0"
|
||||
}
|
||||
},
|
||||
"Microsoft.Testing.Extensions.Telemetry": {
|
||||
"type": "Transitive",
|
||||
"resolved": "2.5.2",
|
||||
@@ -140,10 +153,17 @@
|
||||
"SQLitePCLRaw.core": "2.1.13"
|
||||
}
|
||||
},
|
||||
"System.Numerics.Tensors": {
|
||||
"type": "Transitive",
|
||||
"resolved": "9.0.0",
|
||||
"contentHash": "hyJB4UlpAi19Xr9AXzu2NuagKC4lPfHObNMEAA0HmqFz2rX7wKgzeYzO/jM/eBHDhnUGFFEjk5cOoJaxqg5J4A=="
|
||||
},
|
||||
"anubis": {
|
||||
"type": "Project",
|
||||
"dependencies": {
|
||||
"Microsoft.Data.Sqlite": "[10.0.14, )",
|
||||
"Microsoft.ML.OnnxRuntime": "[1.31.1, )",
|
||||
"Microsoft.ML.Tokenizers": "[2.0.1, )",
|
||||
"Npgsql": "[10.0.3, )",
|
||||
"System.Security.Cryptography.ProtectedData": "[10.0.14, )"
|
||||
}
|
||||
@@ -151,7 +171,7 @@
|
||||
"anubis.server": {
|
||||
"type": "Project",
|
||||
"dependencies": {
|
||||
"Anubis": "[0.0.1, )"
|
||||
"Anubis": "[0.7.0, )"
|
||||
}
|
||||
},
|
||||
"Microsoft.Data.Sqlite": {
|
||||
@@ -165,6 +185,24 @@
|
||||
"SQLitePCLRaw.core": "2.1.13"
|
||||
}
|
||||
},
|
||||
"Microsoft.ML.OnnxRuntime": {
|
||||
"type": "CentralTransitive",
|
||||
"requested": "[1.31.1, )",
|
||||
"resolved": "1.31.1",
|
||||
"contentHash": "XtsO/1JoKepkWbGk9/DcYtRUTduwJD68kw0eeaH5yy77YBivUZWhC1NBNlVecsBt9EitlQIjCd1hHDNXJ2jbQA==",
|
||||
"dependencies": {
|
||||
"Microsoft.ML.OnnxRuntime.Managed": "1.31.1"
|
||||
}
|
||||
},
|
||||
"Microsoft.ML.Tokenizers": {
|
||||
"type": "CentralTransitive",
|
||||
"requested": "[2.0.1, )",
|
||||
"resolved": "2.0.1",
|
||||
"contentHash": "RZN3zslVWinBCMDAnbZWEY8lQbMkP54Yxm1s5NigPcrvYFzXMrDjNcVM9afq4SZHf1BV1rEmO0p7ZdfBLDbu4g==",
|
||||
"dependencies": {
|
||||
"Google.Protobuf": "3.30.2"
|
||||
}
|
||||
},
|
||||
"Npgsql": {
|
||||
"type": "CentralTransitive",
|
||||
"requested": "[10.0.3, )",
|
||||
|
||||
Reference in new issue
Block a user