Rapporto di diagnostica della raccolta (#39) (#48)
verifica / controlli (push) Successful in 14s
verifica / prove (push) Successful in 12m51s
verifica / installatore (push) Successful in 4m38s
verifica / immagine (push) Successful in 16m39s

This commit was merged in pull request #48.
This commit is contained in:
Alby96 committed 2026-11-13 15:14:11 +01:00
commit e2e64c09e8
55 files changed
+5395 -134

No files matched your search

+3 -1
View File
@@ -535,6 +535,8 @@ Regole: il puntatore non salta mai da un punto all'altro; non si clicca nulla ch
La pagina Attività (mockup 06 e 11) è il cruscotto del motore: che cosa sta facendo adesso, la **striscia della giornata** con sessioni, riposo, rallentamenti e arresti, i quattro tetti principali con i numeri (profili aperti, storie viste, pagine di storico, dati scaricati) e, da «Tutti i tetti», anche gli altri; la coda con orari indicativi, le ultime azioni con l'esito. Gli orari futuri si scrivono sempre come «verso le», perché sono estratti a caso.
**Rapporto di diagnostica.** Dopo una prova dal vivo andata storta, Alberto deve poter mandare un solo file che spieghi perché, senza che nessuno apra i dati di altre persone. «Esporta il rapporto di diagnostica» (pulsante «Diagnostica» nella riga del titolo di Attività) scarica `febe-diagnostica-aaaa-mm-gg.json`: per ogni sessione del motore delle ultime due settimane, e per ogni giorno fuori dalle sessioni, le pagine aperte (il nome della pagina nella mappa del sito e la sua forma, mai l'indirizzo), le risposte ricevute per tipo con numero e misure, che cosa ne hanno fatto i lettori (riconosciute, vuote, ignorate, forma sconosciuta con la sola struttura delle chiavi, in cui resta solo una chiave fatta di parole di un elenco chiuso, preso dai campioni ripuliti e dai vocabolari dei lettori, mai di sole parole che sono anche nomi come «max_mark», e ogni altra diventa `#`; diventano `#` anche le voci di una mappa, cioè le chiavi sorelle con valori della stessa forma, salvo quelle che i lettori conoscono come `user` e `owner`), i bersagli cliccati, le ricette con l'esito (e, per un errore imprevisto, il solo tipo dell'eccezione), i segnali di §7.6 per livello e codice, i rifiuti del Governatore, i file messi in coda e scaricati (per un file fallito o da riprovare, la causa da un elenco chiuso, come `http-404`), e la causa di ogni fine (fine prevista, lavoro finito, tetti esauriti, ricetta che ferma, interruttore aperto, pausa, Alberto al browser, chiusura di Febe, errore). Mai cookie, password, gettoni, corpi delle risposte, nomi utente, didascalie, codici dei post, identificativi, indirizzi interi o testi liberi (messaggi, note, eccezioni): solo valori di elenchi chiusi, conteggi, misure, orari e forme degli indirizzi in cui ogni parte che non è una parola fissa del sito è `{numero}` o `{testo}`; per il CDN solo «cdn» e l'estensione del file. Le tracce finite stanno in `Dati/diagnostica/sessioni-aaaa-mm-gg.jsonl`, una riga per sessione, e si tengono 14 giorni (Febe cancella da sé solo i file più vecchi di quella cartella); le sessioni in corso entrano nel rapporto come «in corso». Una sessione scrive una riga «in corso» quando parte e poi al più ogni 5 minuti alla fine di una ricetta, così una chiusura brusca di Febe non la cancella; leggendo, di una sessione resta la riga finita, altrimenti l'ultima. I segnali entrano anche quando arrivano all'interruttore senza passare dal rilevatore (il cookie di sessione perso), e i file in attesa perché manca la connessione hanno la causa `connessione`. La testa del file dice che cosa sicuramente manca: righe del registro rovinate (`righeRovinate`), errori del registratore, scritture non riuscite comprese (`erroriRegistratore`), file di un giorno che non si sono potuti leggere (`fileIlleggibili`) e sessioni rimaste senza fine per una chiusura brusca (`sessioniSenzaFine`). Anche a zero il rapporto non promette di essere intero: gli ultimi minuti di una chiusura brusca, le ore fuori dalle sessioni del giorno e quello che supera i tetti (contato in `overflow`) possono mancare. Le forme delle pagine della mappa del sito passano per la stessa riduzione degli indirizzi, perché la mappa può venire dalla cartella dei dati. Con un problema di configurazione il motore non parte: l'indirizzo del rapporto risponde lo stesso, con gli elenchi vuoti. Una prova dimostra che nessun valore dei dati arriva nel file.
---
## 8. Dati: database e file
@@ -812,7 +814,7 @@ Il divieto di modalità headless della regola 2 riguarda il browser che visita I
- La sessione di Instagram vive nel profilo del browser dentro la cartella dei dati: Febe non la esporta, non la mostra e non la copia altrove.
- Password e codici non entrano mai nei registri. Nella vista remota i tasti vanno al browser e non si conservano.
- I registri contengono azioni ed esiti, non contenuti: nomi utente sì, didascalie e corpi delle risposte no. Il client CDP registra solo nomi di comandi (§6.1). Per le richieste verso Telegram e verso il CDN non si registrano gli indirizzi: contengono il gettone del bot e le firme dei file.
- I registri contengono azioni ed esiti, non contenuti: nomi utente sì, didascalie e corpi delle risposte no. Il client CDP registra solo nomi di comandi (§6.1). Il rapporto di diagnostica (§7.9) è più severo dei registri: nemmeno i nomi utente, né indirizzi interi, solo forme, conteggi e codici di elenchi chiusi, perché è fatto per uscire dalla macchina di Alberto. Per le richieste verso Telegram e verso il CDN non si registrano gli indirizzi: contengono il gettone del bot e le firme dei file.
- I campioni grezzi della sonda stanno fuori dal repository e si cancellano a fine M0 (§14.5).
### 13.3 Avvisi
+3
View File
@@ -320,3 +320,6 @@ Ogni scelta tecnica presa strada facendo, con data e motivo (specifica §15).
| 2026-11-12 | Vocabolario dei lettori di Instagram: `profile.postCount` = `media_count` e una chiave nuova `highlight.cover` (`cover_media.cropped_image_version`), letta come una versione. Le griglie delle schede Reel e Con tag restano anteprime, non contenuti | Dal secondo HAR: `media_count` compare solo nella scheda del profilo; la copertina delle raccolte è nell'elenco delle raccolte del profilo; le anteprime delle schede non hanno `taken_at` né `video_versions` (regola 9: si legge solo ciò che i campioni mostrano) |
| 2026-11-13 | `febe-sonda da-har` e `richieste` leggono anche un HAR **troncato**: se il file finisce prima del suo JSON, e solo in quel caso, tengono le voci complete di `log.entries`, scartano quella dentro cui il file finisce e lo dicono a schermo e nel resoconto (`Truncated`, `DroppedEntry`). Per riconoscere il caso il lettore JSON lavora in modalità «dati non finiti»: la fine del file non è un errore, un errore di sintassi sì, e un file rotto in qualunque altro modo resta rifiutato. Per `richieste` la voce scartata conta comunque come dati: si chiude dove si ferma (il testo interrotto tenuto fino al taglio) e passa per le stesse regole di una voce intera (base64, pagine con blocchi incorporati, intestazioni, moduli); in più ogni stringa JSON del testo interrotto conta come valore, così un nome che è un dato solo lì viene tolto lo stesso. Resta fuori solo un valore tagliato a metà, di cui c'è solo l'inizio. Lettura comune in `HarFile` | Il 12/11 sera Chrome ha troncato due esportazioni dello stesso giro: la prima a 200 000 byte esatti, la seconda a 57 MB a metà di un corpo di risposta. Prima la sonda diceva solo «non è un file HAR valido (riga …)», senza distinguere un file tagliato da uno sbagliato, e non dava nulla della parte buona. La revisione avversaria ha trovato che la prima versione non contava i valori del testo interrotto né applicava alla voce scartata le regole di base64 e pagine: corretto prima del commit |
| 2026-11-13 | Le foto del profilo hanno un **terzo schema dei nomi**, predefinito `{account}\profilo\{data}_{ora}_{codice}_{n}` (chiave `libreria.schema-profilo`), accanto a quello generale e a quello delle raccolte; Impostazioni ▸ Generali lo mostra come terzo campo, con la cartella `profilo` nel risultato. Niente cartella dell'anno; `{codice}` sono le prime otto cifre dello SHA-256 e `{n}` vale sempre 1, tenuto perché i nomi si leggano come gli altri e l'importazione li riconosca con il riconoscitore di Febe. Senza `{codice}` l'avviso dice che due foto possono avere lo stesso nome; `{raccolta}` è un errore. Una richiesta di salvataggio senza il terzo schema tiene quello salvato | Decisione di Alberto su #38 (12/11): le foto del profilo dentro ogni account, in `<account>\profilo\…`. Con lo schema unico bastava cambiare quello generale perché finissero con i post; uno schema proprio, come per le raccolte, le tiene separate qualunque cosa dica il generale |
| 2026-11-13 | Rapporto di diagnostica della raccolta (#39): un registratore unico (`CollectionDiagnostics`) tiene per ogni sessione del motore, e per ogni giorno fuori dalle sessioni, solo conteggi, misure, orari, valori di elenchi chiusi e forme degli indirizzi; le tracce finite vanno in `Dati/diagnostica/sessioni-aaaa-mm-gg.jsonl` (JSON per riga, 14 giorni, una riga rovinata si salta) e «Esporta il rapporto di diagnostica» in Attività (`GET /api/engine/diagnostica`, ammesso anche con un problema di configurazione) le scarica in un file JSON. La struttura delle chiavi delle risposte sconosciute è quella **severa** (`KeyStructure.DescribeStrict`): resta solo una chiave fatta tutta di parole di un **elenco chiuso** (`StructureWords`: le parole delle chiavi dei campioni ripuliti di `tests/Febe.Tests/campioni`, dei vocabolari dei lettori incorporati e degli indirizzi del sito; alcune parole che sono anche nomi, come «max», solo dentro una chiave composta) e non uguale a un valore stringa dello stesso corpo; ogni altra chiave diventa `#`, anche un nome utente senza punto («marta_92») o un codice di post. Le forme degli indirizzi (`PathPatterns`) e la regola delle chiavi-dato (`KeyStructure.IsDataKey`) passano da `tools/Febe.Sonda` a `Febe.Core`, e la sonda le richiama. Gli eventi «formato non riconosciuto» scrivono ancora la struttura di prima: passarli a quella severa è un seguito possibile | Il file deve poter uscire dalla macchina di Alberto senza ripulitura, quindi niente testo libero per costruzione. Non nel database: nessuna migrazione, dimensione limitata dai tetti delle tracce e dai 14 giorni, cartella che si può cancellare a mano senza toccare nulla. Elenco chiuso e non regole su che aspetto ha un nome: un nome utente fatto di lettere e cifre non si distingue da una chiave, e la revisione l'ha trovato nel rapporto. Ammesso con un problema di configurazione perché l'indirizzo non risponda mai con un errore, ma lì il motore non è partito e il rapporto è vuoto |
| 2026-11-13 | Rapporto di diagnostica (#39), seconda revisione: nella struttura severa una chiave fatta di sole parole «solo composte» (`max_mark`) non passa più, e le **voci di una mappa** (chiavi sorelle i cui valori hanno la stessa forma) diventano `#` anche se fatte di parole dell'elenco, salvo le chiavi intere dei vocabolari dei lettori (`user`, `owner`). Una sessione scrive nel registro una riga «in corso» quando parte e al più ogni 5 minuti a fine ricetta; leggendo resta la riga finita, altrimenti l'ultima, e la testa del rapporto conta `fileIlleggibili` e `sessioniSenzaFine`. Il registratore ascolta anche i cambi dell'interruttore (da `EngineHost`), contando una volta sola lo stesso segnale detto anche dal rilevatore. La prova sulla riservatezza confronta le chiavi della struttura con i campi veri dei campioni e del corpo, non con `StructureWords` | Un nome fatto di parole comuni («the_artist») passava la sola lista; una mappa di nomi si riconosce dalla forma dei valori, a costo di qualche dettaglio (`first_frame` e `igtv_first_frame` diventano `#`). Senza righe intermedie una chiusura brusca cancellava proprio la sessione andata storta e la testa diceva 0; il cookie di sessione perso arriva all'interruttore senza passare dal rilevatore e il rapporto non lo diceva. Una prova che usa la funzione sotto prova come oracolo non trova i suoi errori. Resta il rischio di un solo nome di parole comuni usato come chiave, senza sorelle della stessa forma e mai ripetuto come valore |
| 2026-11-13 | Rapporto di diagnostica (#39), terza revisione: nella struttura severa una chiave con un **valore semplice** (testo, numero, vero o falso, null) resta solo se è un **campo dell'elenco chiuso** (`StructureWords.IsFieldKey`: chiavi intere dei vocabolari dei lettori e delle chiavi dei campioni ripuliti, congelate nel codice perché i campioni non viaggiano con Febe), altrimenti `#`. Le voci di una mappa si riconoscono anche quando i valori non hanno la stessa forma esatta: due valori sorelli entrambi oggetti o entrambi elenchi, di cui uno vuoto, o entrambi elenchi di valori semplici, o con almeno metà delle chiavi in comune, o con un campo d'identità in comune (`pk`, `id`, `username`…); in un elenco di oggetti le chiavi che ha solo il primo diventano `#`; fra le voci restano solo i campi dell'elenco chiuso. Una traccia finita la cui scrittura fallisce si riscrive con la riga successiva e quando si raccoglie il rapporto (al più 16 in attesa); le righe «in corso» non si riscrivono | La regola della stessa forma esatta lasciava passare i nomi in mappe di conteggi o di valori vuoti, in voci con campi facoltativi diversi (il caso normale degli utenti di Instagram) e in elenchi di mappe a una voce. Il prezzo è il dettaglio: un campo nuovo con un valore semplice diventa `#`. Senza la riscrittura, un errore momentaneo proprio sulla riga finale lasciava nel registro solo le righe «in corso» e dopo un riavvio la sessione contava fra quelle senza fine |
+46 -1
View File
@@ -18,7 +18,52 @@
pieni, formati non riconosciuti, e gli avvisi che non sono arrivati (vedi [Avvisi e Telegram](Avvisi-e-Telegram)).
Comandi: **«Metti in pausa»** (Febe non apre nulla finché non premi «Riprendi»), **«Riprendi»**, **«Apri il
browser»**.
browser»**, **«Diagnostica»** (vedi sotto).
## Il rapporto di diagnostica
Se una prova va storta (Febe si ferma, non trova nulla, scarica meno del previsto) e vuoi farla guardare a qualcuno,
manda il **rapporto di diagnostica**: un solo file che racconta che cosa ha fatto il motore, senza i dati di nessuno.
1. Apri **Attività** e premi **«Diagnostica»** nella riga del titolo (al telefono è l'icona del file con la freccia;
il suggerimento dice «Esporta il rapporto di diagnostica»).
2. Il file `febe-diagnostica-aaaa-mm-gg.json` finisce nei download del browser (su Windows nella cartella Download,
come ogni scaricamento della finestra di Febe).
3. Allegalo così com'è al messaggio o alla segnalazione su Gitea, dicendo più o meno a che ora è successo il problema.
Che cosa contiene, per ogni sessione delle ultime due settimane e per le ore fuori dalle sessioni: le pagine aperte
(solo il tipo di pagina e la forma dell'indirizzo, come `/{username}/`), le risposte ricevute per tipo con numero e
misure, che cosa ne hanno fatto i lettori (per le risposte dalla forma sconosciuta, solo la struttura delle chiavi),
le ricette con l'esito, i segnali di arresto, i file messi in coda e scaricati, e **perché ogni sessione è finita**.
Che cosa **non** contiene: nomi utente, didascalie, codici dei post, identificativi, indirizzi, cookie, password,
corpi delle risposte, testi dei messaggi. Lo puoi aprire con un editor di testo e controllare. È più severo dei
registri di Febe, che i nomi utente li scrivono.
Febe tiene le tracce in `Dati/diagnostica` per **14 giorni** e cancella da sé solo i file più vecchi di quella
cartella. Una sessione in corso lascia una riga quando parte e poi al più ogni cinque minuti, alla fine di una
ricetta: se Febe si chiude di colpo (un arresto imprevisto, il computer spento a forza, il contenitore fermato), la
sessione resta nel rapporto fino all'ultima di quelle righe, «in corso» e senza causa della fine.
In cima al file quattro numeri dicono che cosa sicuramente manca:
| Campo | Che cosa conta |
|---|---|
| `righeRovinate` | righe del registro tagliate a metà da uno spegnimento brusco |
| `erroriRegistratore` | errori del registratore da quando Febe è partito, scritture non riuscite comprese |
| `fileIlleggibili` | giorni del registro che non si sono potuti leggere (un altro programma teneva aperto il file, un errore del disco): le loro sessioni non ci sono |
| `sessioniSenzaFine` | sessioni interrotte da una chiusura brusca: di loro manca quello che è successo dopo l'ultima riga |
Anche con tutti e quattro a 0 il rapporto può non essere intero. Di una chiusura brusca si perdono gli ultimi minuti
della sessione e le ore fuori dalle sessioni del giorno (scritte solo a fine giornata o quando Febe si chiude bene).
Gli elenchi di ogni sessione hanno un tetto: quello che lo supera non c'è, e `overflow` della sessione dice quante
voci mancano (pagine, navigazioni, gruppi di risposte, forme sconosciute, segnali, ricette), quante risposte sono
rimaste senza esito perché arrivate in troppe insieme (`requests`) e quante notizie sono arrivate tardi (`late`),
quando la loro sessione era già chiusa.
Con un problema di configurazione il motore non parte e Febe mostra solo la pagina del problema: il rapporto non
serve, perché non c'è nessuna sessione da raccontare. Le tracce delle sessioni precedenti restano in
`Dati/diagnostica` e tornano nel rapporto appena il problema è risolto.
## La striscia rossa
Binary file not shown.

Before

Width:  |  Height:  |  Size: 202 KiB

After

Width:  |  Height:  |  Size: 207 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 189 KiB

After

Width:  |  Height:  |  Size: 170 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 160 KiB

After

Width:  |  Height:  |  Size: 147 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 126 KiB

After

Width:  |  Height:  |  Size: 115 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 147 KiB

After

Width:  |  Height:  |  Size: 136 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 68 KiB

After

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 72 KiB

After

Width:  |  Height:  |  Size: 69 KiB

+4
View File
@@ -83,6 +83,10 @@ schermata del motore fermo si prende per prima, perché «Riprendi» resta spent
collegamento a «Doppioni da decidere», sotto gli elenchi: il mockup non lo prevede (§9.5).
- **Attività, fermo.** La striscia rossa ha «Riprendi» solo nell'intestazione della pagina (non doppio). Nel giorno
la parte dopo l'arresto è tratteggiata in rosso, come nel mockup; lo storico degli incidenti sta in fondo.
- **Attività, «Diagnostica».** Nella riga del titolo c'è un terzo pulsante, «Diagnostica» con l'icona del file con la
freccia, che i mockup 06 e 11 non hanno: scarica il rapporto di diagnostica (#39, §7.9). Nella finestra stretta i
tre pulsanti stanno sulla riga con le scritte; al telefono restano tre icone accanto al titolo, senza andare a capo.
Schermate di Attività (06, 11, 14) rifatte il 13/11 da Windows.
- **Limiti e comportamento.** Gli orari sono campi di testo a 24 ore («07:30») invece del selettore d'ora del
browser, che in alcune lingue mostra AM/PM. I giorni sono cerchi con l'iniziale. La barra di salvataggio resta
sempre visibile: spenta con «Nessuna modifica», si accende alla prima modifica.
+22
View File
@@ -7,6 +7,28 @@ fra le sessioni; ciò che è da fare si legge nelle issue aperte.
## In breve
**13 novembre 2026, pomeriggio: il rapporto di diagnostica della raccolta** (#39).
- Attività ha un terzo pulsante, «Diagnostica», che scarica `febe-diagnostica-AAAA-MM-GG.json`
(`GET /api/engine/diagnostica`, ammesso anche con un problema di configurazione, ma allora vuoto). Per ogni sessione
degli ultimi 14 giorni, più un gruppo per giorno «fuori sessione»: pagine aperte (nome della pagina e forma
dell'indirizzo, mai i nomi), navigazioni, risposte per tipo, forma, stato e dimensione con ciò che i lettori ne
hanno fatto, struttura delle chiavi delle forme sconosciute (stretta: solo parole di un elenco chiuso, il resto `#`),
obiettivi cliccati, giudizi, ricette con esito e segnale, segnali, permessi negati, file in coda e scaricati con la
causa, e il motivo d'arresto (`SessionStopCause`, nuovo). Le intestazioni dicono se il rapporto può essere
incompleto (righe rovinate, file illeggibili, errori del registratore, sessioni senza fine).
- Il registratore (`Febe.Core.Collection.Diagnostics`) scrive in `Dati/diagnostica/sessioni-AAAA-MM-GG.jsonl`, una
riga all'inizio della sessione, ogni 5 minuti e alla fine; non lancia mai eccezioni verso la raccolta. Le forme degli
indirizzi (`PathPatterns`) e la regola delle chiavi-dato sono passate dalla sonda a Core, e la sonda le richiama.
Specifica §7.9 e §13.2, guida (Attività e arresti), righe in `docs/decisioni.md`, `LEGGIMI.md`; schermate di
Attività (06, 11, 14) rifatte e confrontate con i mockup 06 e 11 (`differenze.md`).
- Prove nuove: unità del registratore, del registro e del rapporto, motivi d'arresto sul banco del motore, endpoint e
dimostrazione per scena, e la prova chiesta da #39 che il rapporto non contiene valori (nomi con punto, nomi usati
come chiavi, codici, cursori, indirizzi firmati, cookie), con controlli di mutazione. Tre giri di revisione
avversaria: 17 + 6 + 2 difetti confermati, tutti corretti.
- **Non verificato qui**: le prove con Chromium (fra cui `DiagnosticPrivacyIntegrationTests`, la ricetta vera sul sito
di prova) girano solo nella CI Linux. Il rapporto non è ancora stato provato dal vivo: va guardato alla prova #40.
**13 novembre 2026, giorno: le foto del profilo hanno il loro schema** (#38).
- PR #45 è **unita in `main`** (305791f) con la verifica verde (esecuzione 272). Forzare l'unione non era possibile:
+20
View File
@@ -3,6 +3,7 @@ using Febe.Browser.Recipes;
using Febe.Core.Abstractions;
using Febe.Core.Alerts;
using Febe.Core.Collection;
using Febe.Core.Collection.Diagnostics;
using Febe.Core.Collection.Downloads;
using Febe.Core.Domain;
using Febe.Core.Engine;
@@ -87,6 +88,12 @@ public sealed class FebeApp : IAsyncDisposable
public PerceptualHashIndex? HashIndex { get; private set; }
/// <summary>
/// The recorder of the diagnostic report (issue #39), with its log in <c>Dati/diagnostica</c> (null with a
/// configuration problem). Flushed when Febe stops.
/// </summary>
public CollectionDiagnostics? Diagnostics { get; private set; }
/// <summary>When Alberto last used the browser («Riprendi» after a verification).</summary>
public BrowserUseTracker Tracker { get; } = new();
@@ -247,6 +254,14 @@ public sealed class FebeApp : IAsyncDisposable
userIsDriving: () => browser.UserIsDriving,
isConnected: () => connection.IsConnected == true,
ct: ct).ConfigureAwait(false);
// The diagnostic report (issue #39): one recorder for the engine, the pipeline and the interface. What the
// capture sees goes to the session in progress, if any.
var diagnostics = new CollectionDiagnostics(Clock, () => services.Runtime.CurrentSession?.SessionId, new DiagnosticLog(data.Diagnostics, Clock));
Diagnostics = diagnostics;
services.Diagnostics = diagnostics;
EventHandler<Permit> permitDenied = (_, permit) => diagnostics.PermitDenied(null, permit.Kind, permit.Reason);
services.Governor.Denied += permitDenied;
_unsubscribe.Add(() => services.Governor.Denied -= permitDenied);
var runner = new DeferredRecipeRunner();
var engine = new EngineHost(services, runner);
Engine = engine;
@@ -269,6 +284,7 @@ public sealed class FebeApp : IAsyncDisposable
s.AddSingleton<IEngineSettingsControl>(new EngineHostSettingsControl(engine));
s.AddSingleton(ingest);
s.AddSingleton<IBrowserPreview>(new LiveBrowserPreview(browser, Clock));
s.AddSingleton<IDiagnosticSource>(new RecordedDiagnosticSource(diagnostics));
s.AddSingleton<IConfigurationProblems>(new StartupProblems([]));
if (Import is not null)
{
@@ -370,6 +386,7 @@ public sealed class FebeApp : IAsyncDisposable
// Rule 7: the pages Alberto opens (login, verification) are his, not the engine's, while no session runs.
UserOwnsPage = () => browser.UserIsDriving && services.Runtime.CurrentSession is null,
Log = Log.ForBrowser(),
Diagnostics = Diagnostics,
},
ct).ConfigureAwait(false);
runner.Target = Pipeline.Runner;
@@ -646,6 +663,9 @@ public sealed class FebeApp : IAsyncDisposable
await Quietly(() => pipeline.DisposeAsync().AsTask()).ConfigureAwait(false);
}
// What is still open of the diagnostic report: sessions as in progress, the day outside sessions as it is.
Diagnostics?.Dispose();
if (Connection is { } connection)
{
await connection.DisposeAsync().ConfigureAwait(false);
+43 -15
View File
@@ -198,7 +198,7 @@ public sealed class ResponseTap : IAsyncDisposable
{
if (_inFlight.TryRemove(key, out _))
{
NotifyDropped(key);
NotifyDropped(key, BodyDropReason.Reconnected);
}
}
@@ -273,17 +273,22 @@ public sealed class ResponseTap : IAsyncDisposable
private void OnLoadingFinished(JsonElement p)
{
if (!p.TryGetProperty("requestId", out var idElement) || idElement.GetString() is not { } requestId
|| !_inFlight.TryRemove(requestId, out var pending))
if (!p.TryGetProperty("requestId", out var idElement) || idElement.GetString() is not { } requestId)
{
return;
}
var length = p.TryGetProperty("encodedDataLength", out var len) && len.TryGetDouble(out var d) ? d : 0;
NotifyLoadingFinished(requestId, length);
if (!_inFlight.TryRemove(requestId, out var pending))
{
return;
}
if (length > _options.MaxBodyBytes)
{
Interlocked.Increment(ref _skipped);
NotifyDropped(requestId);
NotifyDropped(requestId, BodyDropReason.TooLarge);
return;
}
@@ -295,7 +300,7 @@ public sealed class ResponseTap : IAsyncDisposable
if (p.TryGetProperty("requestId", out var idElement) && idElement.GetString() is { } requestId
&& _inFlight.TryRemove(requestId, out _))
{
NotifyDropped(requestId);
NotifyDropped(requestId, BodyDropReason.LoadingFailed);
}
}
@@ -318,7 +323,7 @@ public sealed class ResponseTap : IAsyncDisposable
{
if (key != loader && _inFlight.TryRemove(key, out _))
{
NotifyDropped(key);
NotifyDropped(key, BodyDropReason.FrameNavigated);
}
}
Publish(frame.TryGetProperty("url", out var u) ? u.GetString() : null, sameDocument: false);
@@ -373,7 +378,7 @@ public sealed class ResponseTap : IAsyncDisposable
// One unexpected body must not stop the capture of the following ones. Only the type: the message
// of a parsing error may quote the body.
_log.Warn($"Cattura: errore imprevisto nella lettura di una risposta ({ex.GetType().Name}).");
NotifyDropped(pending.Descriptor.RequestId);
NotifyDropped(pending.Descriptor.RequestId, BodyDropReason.Unexpected);
}
}
}
@@ -394,7 +399,7 @@ public sealed class ResponseTap : IAsyncDisposable
if (consumers.Length == 0)
{
NotifyDropped(descriptor.RequestId);
NotifyDropped(descriptor.RequestId, BodyDropReason.NoConsumers);
return;
}
@@ -408,16 +413,22 @@ public sealed class ResponseTap : IAsyncDisposable
// The body may already be evicted from the browser's buffer, or the page navigated away.
Interlocked.Increment(ref _bodyErrors);
_log.Debug($"Cattura: corpo non disponibile ({ex.GetType().Name}).");
NotifyDropped(descriptor.RequestId);
NotifyDropped(descriptor.RequestId, ex is TimeoutException ? BodyDropReason.Timeout : BodyDropReason.Unavailable);
return;
}
var delivered = bytes is null ? 0
: pending.IsPageDocument ? await ConsumeDocumentAsync(descriptor, bytes, consumers, ct).ConfigureAwait(false)
if (bytes is null)
{
NotifyDropped(descriptor.RequestId, BodyDropReason.Unavailable);
return;
}
var delivered = pending.IsPageDocument
? await ConsumeDocumentAsync(descriptor, bytes, consumers, ct).ConfigureAwait(false)
: await ConsumeBodyAsync(descriptor, bytes, consumers, ct).ConfigureAwait(false);
if (delivered == 0)
{
NotifyDropped(descriptor.RequestId);
NotifyDropped(descriptor.RequestId, pending.IsPageDocument ? BodyDropReason.NoJsonBlock : BodyDropReason.NotJson);
}
}
@@ -515,14 +526,31 @@ public sealed class ResponseTap : IAsyncDisposable
return decoded;
}
/// <summary>Tells the observers that a body they may be waiting for will not come.</summary>
private void NotifyDropped(string requestId)
/// <summary>Tells the observers that a body they may be waiting for will not come, and why.</summary>
private void NotifyDropped(string requestId, BodyDropReason reason)
{
foreach (var observer in SnapshotObservers())
{
try
{
observer.OnBodyDropped(requestId);
observer.OnBodyDropped(requestId, reason);
}
catch (Exception ex)
{
_log.Warn("Cattura: errore in un osservatore delle risposte.", ex);
}
}
}
/// <summary>Tells the observers how many bytes a response took on the network (every response, read or not).</summary>
private void NotifyLoadingFinished(string requestId, double encodedBytes)
{
var bytes = encodedBytes is > 0 and < long.MaxValue ? (long)encodedBytes : 0;
foreach (var observer in SnapshotObservers())
{
try
{
observer.OnLoadingFinished(requestId, bytes);
}
catch (Exception ex)
{
+27 -1
View File
@@ -5,6 +5,7 @@ using Febe.Browser.Hosting;
using Febe.Browser.Targets;
using Febe.Core.Abstractions;
using Febe.Core.Collection;
using Febe.Core.Collection.Diagnostics;
using Febe.Core.Collection.Downloads;
using Febe.Core.Collection.Readers;
using Febe.Core.Collection.Recipes;
@@ -84,6 +85,12 @@ public sealed record CollectionPipelineOptions
public Func<bool>? UserOwnsPage { get; init; }
public IBrowserLog? Log { get; init; }
/// <summary>
/// The recorder of the diagnostic report (issue #39): it observes the responses of the page, the readers, the
/// recipes, the stop signals and the download queue. Null: nothing is recorded.
/// </summary>
public CollectionDiagnostics? Diagnostics { get; init; }
}
/// <summary>
@@ -138,7 +145,8 @@ public sealed class CollectionPipeline : IAsyncDisposable
var tap = new ResponseTap(options.Cdp, site.Hosts, options.Clock, options.Log);
var signals = new SignalDetector(rules, options.Breaker, options.Clock) { UserOwnsPage = options.UserOwnsPage };
var capture = new CaptureCollector(new ShapeReader(vocabulary), options.Clock);
var diagnostics = options.Diagnostics;
var capture = new CaptureCollector(new ShapeReader(vocabulary), options.Clock) { Diagnostics = diagnostics };
var driver = new BrowserPageDriver(options.Cdp, targets, site.Hosts, options.Clock, options.Random, new BrowserDriverOptions
{
SiteBaseUrl = site.BaseUrl,
@@ -160,6 +168,7 @@ public sealed class CollectionPipeline : IAsyncDisposable
DataBudget = options.DataBudget,
Breaker = options.Breaker,
Options = options.Recipes,
Diagnostics = diagnostics,
});
var downloads = new DownloadQueue(
options.Storage,
@@ -176,6 +185,18 @@ public sealed class CollectionPipeline : IAsyncDisposable
pipeline._subscriptions.Add(tap.AddConsumer(signals));
pipeline._subscriptions.Add(tap.AddObserver(capture));
pipeline._subscriptions.Add(tap.AddConsumer(capture));
if (diagnostics is not null)
{
diagnostics.SiteMap = siteMap;
pipeline._subscriptions.Add(tap.AddObserver(diagnostics));
EventHandler<Signal> signalRaised = (_, signal) => diagnostics.SignalRaised(signal);
signals.SignalRaised += signalRaised;
pipeline._subscriptions.Add(new Unsubscribe(() => signals.SignalRaised -= signalRaised));
EventHandler<DownloadProgress> progress = (_, step) => diagnostics.DownloadSettled(step);
downloads.Progress += progress;
pipeline._subscriptions.Add(new Unsubscribe(() => downloads.Progress -= progress));
}
await tap.StartAsync(ct).ConfigureAwait(false);
return pipeline;
}
@@ -189,4 +210,9 @@ public sealed class CollectionPipeline : IAsyncDisposable
await Tap.DisposeAsync().ConfigureAwait(false);
}
private sealed class Unsubscribe(Action action) : IDisposable
{
public void Dispose() => action();
}
}
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,212 @@
using System.Globalization;
using System.Text;
using System.Text.Json;
using Febe.Core.Abstractions;
using Febe.Core.Domain;
namespace Febe.Core.Collection.Diagnostics;
/// <summary>
/// What <see cref="DiagnosticLog.ReadAll"/> found: the traces, oldest first, how many lines were damaged, and how many
/// day files could not be opened or read at all (held by another program, a read error of the disk, no permission).
/// </summary>
public sealed record DiagnosticLogContents(IReadOnlyList<SessionTrace> Traces, int DamagedLines, int UnreadableFiles = 0);
/// <summary>
/// The finished traces of the diagnostic report (issue #39), one JSON line each, in
/// <c>Dati/diagnostica/sessioni-AAAA-MM-GG.jsonl</c> by local day of the start of the trace. Kept
/// <see cref="DefaultDays"/> days: older files of its own folder (and only those, by their name) are deleted when a
/// line is written or the log is read. Each line is written whole and flushed; a damaged line (a stop in the middle
/// of a write) is skipped and counted when reading, and so is a file that cannot be read. A session in progress also
/// has lines of its own (its start, then checkpoints): when reading, a trace found more than once (same session and
/// start) is kept once, the finished line over the ones in progress, else the last one. Thread-safe.
/// </summary>
public sealed class DiagnosticLog
{
/// <summary>Days kept, today included.</summary>
public const int DefaultDays = 14;
private const string Prefix = "sessioni-";
private const string Extension = ".jsonl";
private const string DayFormat = "yyyy-MM-dd";
private readonly IClock _clock;
private readonly Lock _gate = new();
private DateOnly? _retentionDay;
public DiagnosticLog(string folder, IClock clock, int days = DefaultDays)
{
ArgumentException.ThrowIfNullOrWhiteSpace(folder);
ArgumentOutOfRangeException.ThrowIfLessThan(days, 1);
Folder = Path.GetFullPath(folder);
_clock = clock ?? throw new ArgumentNullException(nameof(clock));
Days = days;
}
public string Folder { get; }
public int Days { get; }
/// <summary>Name of the file of a local day.</summary>
public static string FileNameFor(DateOnly day) => Prefix + day.ToString(DayFormat, CultureInfo.InvariantCulture) + Extension;
/// <summary>Appends one finished trace to the file of the local day of its start.</summary>
public void Append(SessionTrace trace)
{
ArgumentNullException.ThrowIfNull(trace);
var line = JsonSerializer.Serialize(trace, DiagnosticsJsonContext.Default.SessionTrace);
var bytes = Encoding.UTF8.GetBytes(line + "\n");
var day = LocalCalendar.Today(trace.Start, _clock.LocalZone);
lock (_gate)
{
Directory.CreateDirectory(Folder);
ApplyRetention(force: false);
using var stream = new FileStream(Path.Combine(Folder, FileNameFor(day)), FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.Read);
if (stream.Length > 0)
{
// A stop in the middle of a write leaves a line without its end: start on a new line, so that only
// the damaged line is lost and not this one too.
stream.Seek(-1, SeekOrigin.End);
if (stream.ReadByte() != '\n')
{
stream.WriteByte((byte)'\n');
}
}
stream.Seek(0, SeekOrigin.End);
stream.Write(bytes);
stream.Flush(flushToDisk: true);
}
}
/// <summary>True when a trace that started at <paramref name="start"/> is among the days kept.</summary>
public bool Keeps(DateTimeOffset start) => LocalCalendar.Today(start, _clock.LocalZone) >= FirstDayKept();
/// <summary>
/// Every trace of the days kept, oldest first, once each (see the class); damaged lines and unreadable files are
/// skipped and counted.
/// </summary>
public DiagnosticLogContents ReadAll()
{
var traces = new List<SessionTrace>();
var damaged = 0;
var unreadable = 0;
lock (_gate)
{
if (!Directory.Exists(Folder))
{
return new DiagnosticLogContents([], 0);
}
ApplyRetention(force: true);
var first = FirstDayKept();
foreach (var (day, path) in Files().Where(f => f.Day >= first).OrderBy(f => f.Day))
{
string[] lines;
try
{
using var stream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
using var reader = new StreamReader(stream, Encoding.UTF8);
lines = reader.ReadToEnd().Split('\n');
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
// One file only: the others are still read, and the report says that one is missing.
unreadable++;
continue;
}
foreach (var raw in lines)
{
var line = raw.Trim();
if (line.Length == 0)
{
continue;
}
try
{
if (JsonSerializer.Deserialize(line, DiagnosticsJsonContext.Default.SessionTrace) is { } trace)
{
traces.Add(trace);
}
else
{
damaged++;
}
}
catch (JsonException)
{
damaged++;
}
}
}
}
return new DiagnosticLogContents([.. Latest(traces).OrderBy(t => t.Start)], damaged, unreadable);
}
/// <summary>One trace per session and start: the finished one if there is one, else the last line written.</summary>
internal static IEnumerable<SessionTrace> Latest(IEnumerable<SessionTrace> inWriteOrder)
{
var kept = new Dictionary<(long?, DateTimeOffset), SessionTrace>();
foreach (var trace in inWriteOrder)
{
var key = (trace.Session, trace.Start);
if (!kept.TryGetValue(key, out var earlier) || !(trace.InProgress && !earlier.InProgress))
{
kept[key] = trace;
}
}
return kept.Values;
}
private DateOnly FirstDayKept() => LocalCalendar.Today(_clock.Now, _clock.LocalZone).AddDays(-(Days - 1));
/// <summary>Deletes the files of days before the ones kept (once per day unless forced).</summary>
private void ApplyRetention(bool force)
{
var today = LocalCalendar.Today(_clock.Now, _clock.LocalZone);
if (!force && _retentionDay == today)
{
return;
}
_retentionDay = today;
var first = FirstDayKept();
foreach (var (day, path) in Files())
{
if (day < first)
{
try
{
File.Delete(path);
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
// Tried again at the next write or reading.
}
}
}
}
/// <summary>The files of this log in the folder, by the day in their name (other files are never touched).</summary>
private IEnumerable<(DateOnly Day, string Path)> Files()
{
if (!Directory.Exists(Folder))
{
yield break;
}
foreach (var path in Directory.EnumerateFiles(Folder, Prefix + "*" + Extension))
{
var name = Path.GetFileName(path);
var text = name[Prefix.Length..^Extension.Length];
if (DateOnly.TryParseExact(text, DayFormat, CultureInfo.InvariantCulture, DateTimeStyles.None, out var day))
{
yield return (day, path);
}
}
}
}
@@ -0,0 +1,139 @@
using System.Globalization;
using System.Text;
using System.Text.Json;
using System.Text.RegularExpressions;
using Febe.Core.Domain;
namespace Febe.Core.Collection.Diagnostics;
/// <summary>The head of an exported report.</summary>
/// <param name="Version">Febe's version (three numbers).</param>
/// <param name="Edition">«docker» or «windows».</param>
/// <param name="Created">When the report is written.</param>
/// <param name="Zone">Alberto's time zone: the times of the report are written in it.</param>
/// <param name="Days">Days the log keeps.</param>
/// <param name="DamagedLines">Lines of the log that could not be read («righeRovinate»).</param>
/// <param name="RecorderFaults">Errors the recorder swallowed, failed writes among them («erroriRegistratore»).</param>
/// <param name="UnreadableFiles">Day files of the log that could not be read («fileIlleggibili»).</param>
/// <param name="UnfinishedSessions">Sessions Febe stopped without closing, a crash or a kill («sessioniSenzaFine»).</param>
public sealed record DiagnosticReportHeader(
string Version,
string Edition,
DateTimeOffset Created,
TimeZoneInfo Zone,
int Days = DiagnosticLog.DefaultDays,
int DamagedLines = 0,
int RecorderFaults = 0,
int UnreadableFiles = 0,
int UnfinishedSessions = 0);
/// <summary>
/// The file Alberto sends after a live test that went wrong (issue #39, «Esporta il rapporto di diagnostica»):
/// UTF-8 JSON without BOM, indented, with the traces of the engine sessions and of the days outside sessions. Like
/// the traces, it holds no data value: no address, user name, cookie, body or free text.
/// </summary>
public static partial class DiagnosticReport
{
/// <summary>The value of «rapporto».</summary>
public const string Title = "diagnostica della raccolta";
/// <summary>Name of the exported file for the local day of <paramref name="now"/>.</summary>
public static string FileName(DateTimeOffset now, TimeZoneInfo zone) =>
$"febe-diagnostica-{LocalCalendar.Today(now, zone).ToString("yyyy-MM-dd", CultureInfo.InvariantCulture)}.json";
/// <summary>Writes the report to <paramref name="stream"/> (left open).</summary>
/// <exception cref="InvalidOperationException">Debug builds only: the text holds «://» or «@», which no reduction lets through.</exception>
public static void Write(Stream stream, IReadOnlyList<SessionTrace> traces, DiagnosticReportHeader header)
{
ArgumentNullException.ThrowIfNull(stream);
ArgumentNullException.ThrowIfNull(traces);
ArgumentNullException.ThrowIfNull(header);
var zone = header.Zone;
using var buffer = new MemoryStream();
using (var writer = new Utf8JsonWriter(buffer, new JsonWriterOptions { Indented = true }))
{
writer.WriteStartObject();
writer.WriteString("rapporto", Title);
writer.WriteString("versione", Safe(header.Version));
writer.WriteString("edizione", Safe(header.Edition));
writer.WriteString("creato", Local(header.Created, zone));
writer.WriteString("fuso", Safe(ZoneName(zone)));
writer.WriteNumber("giorni", header.Days);
// Not zero: the report is not the whole story (lines of the log lost, writes or readings that failed, day
// files that could not be read, sessions cut short by a crash).
writer.WriteNumber("righeRovinate", Math.Max(0, header.DamagedLines));
writer.WriteNumber("erroriRegistratore", Math.Max(0, header.RecorderFaults));
writer.WriteNumber("fileIlleggibili", Math.Max(0, header.UnreadableFiles));
writer.WriteNumber("sessioniSenzaFine", Math.Max(0, header.UnfinishedSessions));
WriteTraces(writer, "sessioni", traces.Where(t => t.Session is not null), zone);
WriteTraces(writer, "fuoriSessione", traces.Where(t => t.Session is null), zone);
writer.WriteEndObject();
}
#if DEBUG
var text = Encoding.UTF8.GetString(buffer.GetBuffer(), 0, (int)buffer.Length);
if (text.Contains("://", StringComparison.Ordinal) || text.Contains('@', StringComparison.Ordinal))
{
throw new InvalidOperationException("Il rapporto di diagnostica contiene un indirizzo o un nome: una riduzione non ha funzionato.");
}
#endif
buffer.Position = 0;
buffer.CopyTo(stream);
stream.WriteByte((byte)'\n');
}
private static void WriteTraces(Utf8JsonWriter writer, string name, IEnumerable<SessionTrace> traces, TimeZoneInfo zone)
{
writer.WriteStartArray(name);
foreach (var trace in traces.OrderBy(t => t.Start))
{
JsonSerializer.Serialize(writer, InZone(trace, zone), DiagnosticsJsonContext.Default.SessionTrace);
}
writer.WriteEndArray();
}
/// <summary>The same trace with its times written in Alberto's zone (same instants, local offset).</summary>
private static SessionTrace InZone(SessionTrace trace, TimeZoneInfo zone) => trace with
{
Start = Local(trace.Start, zone),
End = trace.End is { } end ? Local(end, zone) : null,
Pages = [.. trace.Pages.Select(p => p with { At = Local(p.At, zone) })],
Recipes = [.. trace.Recipes.Select(r => r with { At = Local(r.At, zone) })],
Signals = [.. trace.Signals.Select(s => s with { At = Local(s.At, zone) })],
};
private static DateTimeOffset Local(DateTimeOffset instant, TimeZoneInfo zone) => TimeZoneInfo.ConvertTime(instant, zone);
/// <summary>
/// The IANA name of the zone («Europe/Rome»), also on Windows, whose own names have spaces («W. Europe Standard
/// Time»), the one of the computer's region when the Windows name covers several («Europe/Rome» rather than
/// «Europe/Berlin» in Italy); the Windows name with its spaces as underscores when there is no IANA one.
/// </summary>
internal static string ZoneName(TimeZoneInfo zone)
{
string? region = null;
try
{
region = RegionInfo.CurrentRegion.TwoLetterISORegionName;
}
catch (ArgumentException)
{
// No region (invariant culture): the zone's main IANA name.
}
if (region is { Length: 2 } && TimeZoneInfo.TryConvertWindowsIdToIanaId(zone.Id, region, out var regional))
{
return regional;
}
return TimeZoneInfo.TryConvertWindowsIdToIanaId(zone.Id, out var iana) ? iana : zone.Id.Replace(' ', '_');
}
private static string Safe(string? text) => text is not null && SafeText().IsMatch(text) ? text : CollectionDiagnostics.Other;
[GeneratedRegex("^[A-Za-z0-9_./+-]{1,64}$", RegexOptions.CultureInvariant)]
private static partial Regex SafeText();
}
@@ -0,0 +1,186 @@
using System.Text.Json.Serialization;
using Febe.Core.Abstractions;
using Febe.Core.Collection.Downloads;
using Febe.Core.Collection.Recipes;
using Febe.Core.Domain;
using Febe.Core.Engine;
namespace Febe.Core.Collection.Diagnostics;
/// <summary>
/// What one engine session did with the site, for the diagnostic report (issue #39), or what happened outside any
/// session on one local day (<see cref="Session"/> null: Alberto at the browser, downloads between sessions).
/// </summary>
/// <remarks>
/// It holds no data value by construction: only enums, codes from closed lists (signal codes, page and target names
/// of the site map), counts, sizes, times and address patterns where every segment that is not a word of the site
/// is <c>{numero}</c> or <c>{testo}</c>. Never addresses, user names, bodies, cookies or free text.
/// </remarks>
public sealed record SessionTrace
{
/// <summary>The engine session; null for the bucket outside sessions.</summary>
public long? Session { get; init; }
public DateTimeOffset Start { get; init; }
public DateTimeOffset? End { get; init; }
public SessionOutcome? Outcome { get; init; }
public SessionStopCause? StopCause { get; init; }
/// <summary>The recipe status that stopped the session early, when one did.</summary>
public RecipeStatus? StoppedBy { get; init; }
/// <summary>The speed factor of the session (spec §7.7), two decimals.</summary>
public double? Mood { get; init; }
/// <summary>True for a session still running when the trace was taken (or when Febe stopped).</summary>
public bool InProgress { get; init; }
/// <summary>Pages the recipes opened, in order.</summary>
public IReadOnlyList<PageVisit> Pages { get; init; } = [];
/// <summary>Main-frame navigations (also the ones the engine did not ask for), counted by pattern.</summary>
public IReadOnlyList<NavigationGroup> Navigations { get; init; } = [];
/// <summary>Responses the page received, counted by kind, with sizes and what the readers did with them.</summary>
public IReadOnlyList<ResponseGroup> Responses { get; init; } = [];
/// <summary>Distinct key structures of responses the readers did not recognize (strict: no key that could be data).</summary>
public IReadOnlyList<UnknownShape> UnknownShapes { get; init; } = [];
/// <summary>Verdicts of the recipe steps: «riconosciuto», «vuoto», «sconosciuto».</summary>
public IReadOnlyList<VerdictCount> Steps { get; init; } = [];
public IReadOnlyList<TargetCount> Targets { get; init; } = [];
public IReadOnlyList<RecipeLine> Recipes { get; init; } = [];
public IReadOnlyList<SignalLine> Signals { get; init; } = [];
/// <summary>Refusals of the Governor.</summary>
public IReadOnlyList<DenialCount> Denials { get; init; } = [];
public IReadOnlyList<QueuedFiles> FilesQueued { get; init; } = [];
public IReadOnlyList<DownloadGroup> Downloads { get; init; } = [];
/// <summary>Bytes of the files downloaded (what crossed the network), when the queue said.</summary>
public long DownloadedBytes { get; init; }
/// <summary>Items not kept because a list was full: counted, never dropped silently.</summary>
public TraceOverflow Overflow { get; init; } = new();
}
/// <summary>A page a recipe opened: the name in the site map and its shape, never the address.</summary>
public sealed record PageVisit(DateTimeOffset At, string Page, string? Pattern, bool Ok);
/// <summary>Main-frame navigations with the same pattern.</summary>
public sealed record NavigationGroup(HostKind HostKind, string? Page, string Pattern, bool SameDocument, int Count);
/// <summary>Responses with the same host kind, pattern, resource type, MIME type and status class.</summary>
/// <param name="Count">Responses seen.</param>
/// <param name="Bytes">Sum of what crossed the network.</param>
/// <param name="MaxBytes">Largest single response.</param>
/// <param name="Recognized">Bodies (or blocks of a page) the readers recognized.</param>
/// <param name="Empty">Bodies with no contents (an empty page or list).</param>
/// <param name="Unrelated">Bodies that are not about contents (ignored).</param>
/// <param name="Unrecognized">Bodies with a shape the readers do not know.</param>
/// <param name="NotSite">Bodies of a host that is not the site (not read).</param>
/// <param name="NotRead">Responses whose body is not data and is not read at all (images, scripts...).</param>
/// <param name="Contents">Contents the readers found.</param>
/// <param name="Profiles">Profiles the readers found.</param>
/// <param name="Dropped">Bodies that never reached the readers, by reason.</param>
public sealed record ResponseGroup(
HostKind HostKind,
string? Page,
string Pattern,
string? ResourceType,
string Mime,
string Status,
int Count,
long Bytes,
long MaxBytes,
int Recognized,
int Empty,
int Unrelated,
int Unrecognized,
int NotSite,
int NotRead,
int Contents,
int Profiles,
IReadOnlyList<DropCount> Dropped);
public sealed record DropCount(BodyDropReason Reason, int Count);
public sealed record UnknownShape(string Structure, string Pattern, int Count);
public sealed record VerdictCount(string Verdict, int Count);
public sealed record TargetCount(string Target, TargetCheckStatus Status, int Count);
public sealed record RecipeLine(DateTimeOffset At, RecipeKind Kind, RecipeStatus Status, string? Signal, int Actions, int NewEntries)
{
/// <summary>For a recipe that failed on an unexpected error: the type name of the exception, never its message.</summary>
public string? Error { get; init; }
}
public sealed record SignalLine(DateTimeOffset At, SignalLevel Level, string Code);
public sealed record DenialCount(ActionKind Action, DenialReason Reason, int Count);
/// <summary>Saves of contents read by the recipes: how many, and how many files they put in the download queue.</summary>
/// <param name="Outcome">A <see cref="SaveOutcome"/> name, or «Completed» for a «Post singolo» completion.</param>
public sealed record QueuedFiles(EntryKind Kind, string Outcome, int Saves, int Files, int Skipped);
public sealed record DownloadGroup(EntryKind Kind, DownloadStep Step, int Count, long Bytes)
{
/// <summary>Why a file failed or waits for another attempt (<see cref="DownloadCauses"/>), when the queue said.</summary>
public string? Cause { get; init; }
}
/// <summary>Items left out because their list reached its cap (<see cref="CollectionDiagnostics"/>).</summary>
public sealed record TraceOverflow
{
public int Pages { get; init; }
public int Navigations { get; init; }
public int Responses { get; init; }
public int UnknownShapes { get; init; }
public int Signals { get; init; }
public int Recipes { get; init; }
/// <summary>Bodies forgotten while still waiting for their reading (too many responses at once).</summary>
public int Requests { get; init; }
/// <summary>Sizes, readings or drops that arrived after their own trace was closed (counted in the trace open then).</summary>
public int Late { get; init; }
}
/// <summary>What there is to report, with what the report must say about itself.</summary>
/// <param name="Traces">The traces, oldest first.</param>
/// <param name="DamagedLines">Lines of the log that could not be read (a stop in the middle of a write).</param>
/// <param name="RecorderFaults">Errors the recorder swallowed since Febe started (a failed write among them).</param>
public sealed record DiagnosticCollection(IReadOnlyList<SessionTrace> Traces, int DamagedLines = 0, int RecorderFaults = 0)
{
/// <summary>Day files of the log that could not be opened or read: their sessions are missing.</summary>
public int UnreadableFiles { get; init; }
/// <summary>
/// Sessions whose last line in the log is in progress, not running now and not closed by a shutdown: Febe stopped
/// without closing them (a crash, a kill), and what they did after their last line is missing.
/// </summary>
public int UnfinishedSessions { get; init; }
}
[JsonSourceGenerationOptions(
PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase,
UseStringEnumConverter = true,
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull)]
[JsonSerializable(typeof(SessionTrace))]
internal sealed partial class DiagnosticsJsonContext : JsonSerializerContext;
@@ -353,6 +353,7 @@ public sealed class DownloadQueue
{
await _storage.Media.MarkFailedAsync(media.Id, MediaState.Failed, "Indirizzo del file non valido.", _clock.Now, ct).ConfigureAwait(false);
tally.Add(ref tally.Failed);
Report(media, entry, DownloadStep.Failed, cause: DownloadCauses.BadAddress);
return;
}
@@ -382,33 +383,35 @@ public sealed class DownloadQueue
switch (result.Outcome)
{
case FetchOutcome.Completed:
await IngestAsync(account, entry, media, temp, maxBytes, tally, ct).ConfigureAwait(false);
await IngestAsync(account, entry, media, temp, maxBytes, result.BudgetBytes, tally, ct).ConfigureAwait(false);
return;
case FetchOutcome.TooLarge:
var limit = maxBytes is { } m ? ItalianText.Number(Math.Max(1, m / (1024 * 1024))) : "?";
await _storage.Media.MarkFailedAsync(media.Id, MediaState.Skipped, $"Video oltre il limite di {limit} MB impostato per l'account.", _clock.Now, ct).ConfigureAwait(false);
_retryAt.TryRemove(media.Id, out _);
tally.Add(ref tally.Skipped);
Report(media, entry, DownloadStep.Skipped);
Report(media, entry, DownloadStep.Skipped, result.BudgetBytes);
return;
case FetchOutcome.HttpError when result.HttpStatus is { } status && _options.ExpiredStatuses.Contains(status):
await ExpiredAsync(entry, media, tally, singlePostQueued, ct).ConfigureAwait(false);
await ExpiredAsync(entry, media, tally, singlePostQueued, result.BudgetBytes, ct).ConfigureAwait(false);
return;
case FetchOutcome.HostNotAllowed:
await _storage.Media.MarkFailedAsync(media.Id, MediaState.Failed, "L'indirizzo non è del CDN: non si scarica (specifica §6.5).", _clock.Now, ct).ConfigureAwait(false);
_retryAt.TryRemove(media.Id, out _);
tally.Add(ref tally.Failed);
Report(media, entry, DownloadStep.Failed);
Report(media, entry, DownloadStep.Failed, result.BudgetBytes, DownloadCauses.HostNotAllowed);
return;
case FetchOutcome.NetworkError when IsConnectionError(result.Error):
// No connection at all (router restarting, cable out): every file would fail the same way.
// No connection at all (router restarting, cable out): every file would fail the same way. Not the
// file's fault (no attempt counted), but the report says why it is still waiting.
PauseUntil(_clock.Now + _options.ConnectionPause, tally);
Report(media, entry, DownloadStep.Retried, result.BudgetBytes, DownloadCauses.Connection);
return;
default:
var error = result.HttpStatus is { } code
? $"Il CDN ha risposto {code.ToString(CultureInfo.InvariantCulture)}."
: $"Errore di rete durante lo scaricamento ({result.Error ?? "senza dettagli"}).";
await RetryLaterAsync(entry, media, error, tally, ct).ConfigureAwait(false);
await RetryLaterAsync(entry, media, error, tally, result.BudgetBytes, ct, DownloadCauses.Of(result.HttpStatus)).ConfigureAwait(false);
return;
}
}
@@ -421,11 +424,12 @@ public sealed class DownloadQueue
{
await _storage.Media.MarkFailedAsync(media.Id, MediaState.Skipped, "Video oltre il limite impostato per l'account.", _clock.Now, CancellationToken.None).ConfigureAwait(false);
tally.Add(ref tally.Skipped);
Report(media, entry, DownloadStep.Skipped);
}
catch (IOException ex)
{
// Disk full or similar: the file waits for a later pass (nothing was written under the final name).
await RetryLaterAsync(entry, media, $"Scrittura non riuscita: {ex.GetType().Name}.", tally, CancellationToken.None).ConfigureAwait(false);
await RetryLaterAsync(entry, media, $"Scrittura non riuscita: {ex.GetType().Name}.", tally, null, CancellationToken.None, DownloadCauses.Write).ConfigureAwait(false);
}
finally
{
@@ -455,14 +459,14 @@ public sealed class DownloadQueue
/// A failure that may pass: the file waits for a later attempt, a little longer each time; after
/// <see cref="DownloadOptions.MaxAttempts"/> failures it is «fallito».
/// </summary>
private async Task RetryLaterAsync(Entry entry, Media media, string error, Tally tally, CancellationToken ct)
private async Task RetryLaterAsync(Entry entry, Media media, string error, Tally tally, long? bytes, CancellationToken ct, string? cause = null)
{
if (media.Attempts + 1 >= _options.MaxAttempts)
{
await _storage.Media.MarkFailedAsync(media.Id, MediaState.Failed, error, _clock.Now, ct).ConfigureAwait(false);
_retryAt.TryRemove(media.Id, out _);
tally.Add(ref tally.Failed);
Report(media, entry, DownloadStep.Failed);
Report(media, entry, DownloadStep.Failed, bytes, cause);
return;
}
@@ -474,10 +478,12 @@ public sealed class DownloadQueue
}
tally.Add(ref tally.Retried);
Report(media, entry, DownloadStep.Retried, bytes, cause);
}
private async Task IngestAsync(Account account, Entry entry, Media media, string temp, long? maxBytes, Tally tally, CancellationToken ct)
private async Task IngestAsync(Account account, Entry entry, Media media, string temp, long? maxBytes, long fileBytes, Tally tally, CancellationToken ct)
{
var bytes = fileBytes;
MemoryStream? cover = null;
try
{
@@ -489,6 +495,7 @@ public sealed class DownloadQueue
{
await _budget.RecordAsync(fetched.BudgetBytes, account.Id, CancellationToken.None).ConfigureAwait(false);
tally.AddBytes(fetched.BudgetBytes);
bytes += fetched.BudgetBytes;
}
if (fetched.Outcome == FetchOutcome.HttpError && fetched.HttpStatus is { } status)
@@ -526,13 +533,13 @@ public sealed class DownloadQueue
if (saved.Outcome == MediaSaveOutcome.Stored)
{
tally.Add(ref tally.Stored);
Report(media, entry, DownloadStep.Stored);
Report(media, entry, DownloadStep.Stored, bytes);
}
else
{
await IdenticalAsync(entry, media, saved.Identical!, ct).ConfigureAwait(false);
tally.Add(ref tally.Duplicates);
Report(media, entry, DownloadStep.Duplicate);
Report(media, entry, DownloadStep.Duplicate, bytes);
}
}
finally
@@ -582,7 +589,7 @@ public sealed class DownloadQueue
/// A file whose renewed address is refused again and again ends «fallito» after
/// <see cref="DownloadOptions.MaxAttempts"/> failures, instead of sending «Post singolo» to the site for ever.
/// </summary>
private async Task ExpiredAsync(Entry entry, Media media, Tally tally, ConcurrentDictionary<long, bool> singlePostQueued, CancellationToken ct)
private async Task ExpiredAsync(Entry entry, Media media, Tally tally, ConcurrentDictionary<long, bool> singlePostQueued, long? bytes, CancellationToken ct)
{
_retryAt.TryRemove(media.Id, out _);
if (entry.Kind != EntryKind.Story && media.Attempts + 1 >= _options.MaxAttempts)
@@ -590,7 +597,7 @@ public sealed class DownloadQueue
var attempts = ItalianText.Number(media.Attempts + 1);
await _storage.Media.MarkFailedAsync(media.Id, MediaState.Failed, $"Indirizzo rifiutato anche dopo il rinnovo ({attempts} tentativi): il file resta «fallito».", _clock.Now, ct).ConfigureAwait(false);
tally.Add(ref tally.Failed);
Report(media, entry, DownloadStep.Failed);
Report(media, entry, DownloadStep.Failed, bytes, DownloadCauses.RefusedAfterRenewal);
return;
}
@@ -604,7 +611,7 @@ public sealed class DownloadQueue
};
await _storage.Media.MarkFailedAsync(media.Id, entry.Kind == EntryKind.Story ? MediaState.Failed : MediaState.Expired, why, _clock.Now, ct).ConfigureAwait(false);
tally.Add(ref tally.Expired);
Report(media, entry, DownloadStep.Expired);
Report(media, entry, DownloadStep.Expired, bytes);
return;
}
@@ -615,7 +622,7 @@ public sealed class DownloadQueue
}
tally.Add(ref tally.Expired);
Report(media, entry, DownloadStep.Expired);
Report(media, entry, DownloadStep.Expired, bytes);
}
/// <summary>
@@ -632,8 +639,8 @@ public sealed class DownloadQueue
}
}
private void Report(Media media, Entry entry, DownloadStep step) =>
Progress?.Invoke(this, new DownloadProgress(media.Id, entry.Id, entry.AccountId, entry.Kind, step));
private void Report(Media media, Entry entry, DownloadStep step, long? bytes = null, string? cause = null) =>
Progress?.Invoke(this, new DownloadProgress(media.Id, entry.Id, entry.AccountId, entry.Kind, step, bytes) { Cause = cause });
/// <summary>The extension from the content (magic numbers), else from the kind of file.</summary>
private static string Extension(string path, Media media)
@@ -737,7 +744,44 @@ public enum DownloadStep
Skipped,
Expired,
Failed,
/// <summary>Not settled: the file waits for a later attempt (a failure that may pass).</summary>
Retried,
}
/// <summary>Progress of the queue for the interface: identifiers only, never addresses.</summary>
public sealed record DownloadProgress(long MediaId, long EntryId, long AccountId, EntryKind Kind, DownloadStep Step);
/// <param name="Bytes">What crossed the network for this file (its cover included), when known.</param>
public sealed record DownloadProgress(long MediaId, long EntryId, long AccountId, EntryKind Kind, DownloadStep Step, long? Bytes = null)
{
/// <summary>Why the file failed or waits for another attempt (<see cref="DownloadCauses"/>); null when it did not.</summary>
public string? Cause { get; init; }
}
/// <summary>
/// Why a file failed or waits for another attempt, for the diagnostic report (issue #39): a closed list of codes,
/// never an address or the text of an error.
/// </summary>
public static class DownloadCauses
{
/// <summary>A network error that is not a missing connection (the connection pauses the queue instead).</summary>
public const string Network = "rete";
/// <summary>No connection at all: the queue pauses, the file waits without an attempt counted.</summary>
public const string Connection = "connessione";
/// <summary>The file could not be written (disk full or similar).</summary>
public const string Write = "scrittura";
/// <summary>The address of the file is not valid.</summary>
public const string BadAddress = "indirizzo-non-valido";
/// <summary>The address is not of the CDN (spec §6.5).</summary>
public const string HostNotAllowed = "host-non-ammesso";
/// <summary>The renewed address was refused again, too many times.</summary>
public const string RefusedAfterRenewal = "rifiutato-dopo-rinnovo";
/// <summary>«http-404», «http-503»...: the status the CDN answered; <see cref="Network"/> without one.</summary>
public static string Of(int? httpStatus) =>
httpStatus is >= 100 and < 600 ? "http-" + httpStatus.Value.ToString(CultureInfo.InvariantCulture) : Network;
}
@@ -78,4 +78,52 @@ public interface IResponseObserver
void OnBodyDropped(string requestId)
{
}
/// <summary>
/// The same as <see cref="OnBodyDropped(string)"/>, with the reason (for the diagnostic report, issue #39). The tap
/// calls this one; by default it forwards to the overload without a reason.
/// </summary>
void OnBodyDropped(string requestId, BodyDropReason reason) => OnBodyDropped(requestId);
/// <summary>
/// A response announced by <see cref="OnResponse"/> finished loading: <paramref name="encodedBytes"/> is what
/// crossed the network (the browser's encodedDataLength). Called for every response, whether its body is read or not.
/// </summary>
void OnLoadingFinished(string requestId, long encodedBytes)
{
}
}
/// <summary>Why the body of a response did not reach the consumers (closed list: a report may show it).</summary>
public enum BodyDropReason
{
/// <summary>Over the size the tap reads (memory guard).</summary>
TooLarge,
/// <summary>The browser says the request failed.</summary>
LoadingFailed,
/// <summary>The page moved to another document before the body finished.</summary>
FrameNavigated,
/// <summary>The browser was restarted or reconnected.</summary>
Reconnected,
/// <summary>Nobody was listening for bodies.</summary>
NoConsumers,
/// <summary>The browser no longer had the body (evicted from its buffer) or gave none.</summary>
Unavailable,
/// <summary>The body did not arrive in time.</summary>
Timeout,
/// <summary>Declared as data, but not JSON after all.</summary>
NotJson,
/// <summary>A page document without any embedded JSON block.</summary>
NoJsonBlock,
/// <summary>An unexpected error while reading it.</summary>
Unexpected,
}
@@ -0,0 +1,119 @@
namespace Febe.Core.Collection.Privacy;
/// <summary>
/// Turns an address path into a pattern that names nobody (issue #34, issue #39): every segment that is not a known word
/// of the site's addresses becomes <c>{numero}</c> or <c>{testo}</c>. Used by the request shapes of the probe and by the
/// diagnostic report. A missing word only costs detail, never privacy.
/// </summary>
public static class PathPatterns
{
/// <summary>Written in place of a segment made only of digits.</summary>
public const string Number = "{numero}";
/// <summary>Written in place of any other segment that is not a known word.</summary>
public const string Text = "{testo}";
/// <summary>
/// Path segments kept as they are: words of the site's addresses, the same for everyone. Anything else (user names,
/// identifiers, codes) becomes <see cref="Number"/> or <see cref="Text"/>.
/// </summary>
private static readonly HashSet<string> Known = new(StringComparer.Ordinal)
{
"api", "v1", "graphql", "graphql_www", "query", "ajax", "bulk-route-definitions", "navigation", "web", "users",
"user", "web_profile_info", "usernameinfo", "info", "feed", "timeline", "reels_media", "reels_tray", "highlights",
"highlights_tray", "friendships", "following", "followers", "show", "show_many", "search", "topsearch", "fbsearch",
"web_search", "typeahead", "accounts_typeahead", "top_serp", "media", "comments", "usertags", "clips", "stories",
"story", "tags", "sections", "discover", "explore", "chaining", "profile", "posts", "reels", "tagged", "saved",
"qp", "batch_fetch_web", "logging_client_events", "direct_v2", "inbox", "has_new", "get_presence", "wwwgraphql",
"ig", "bz", "web_info", "current_user", "user_info", "locations", "seen", "media_seen", "reel", "p", "accounts",
"edit", "activity", "news", "inbox_v2", "notifications", "badge", "home", "graphql_query",
// Fixed roots of the site's own pages (verification, login, messages...): a stop that lands there must say so,
// not «/{testo}/», and the site map must not take them for a profile (SiteMap.Classify).
"challenge", "checkpoint", "direct", "login", "logout", "emails", "password", "reset", "two_factor", "onetap",
"suspended", "consent", "settings", "help", "legal", "privacy", "terms", "about",
};
/// <summary>
/// File extensions a CDN or other address may show in a report: a closed list, so that the last part of a name
/// with a dot («nome.cognome») never passes as an extension.
/// </summary>
private static readonly HashSet<string> Extensions = new(StringComparer.Ordinal)
{
"jpg", "jpeg", "png", "webp", "heic", "gif", "avif", "mp4", "m4a", "m4v", "webm", "mov", "js", "css", "json",
"woff", "woff2", "ttf", "ico", "svg", "html", "txt",
};
/// <summary>The known words, the same for everyone.</summary>
public static IReadOnlySet<string> KnownSegments => Known;
/// <summary>The extensions <see cref="ExtensionOf"/> may answer.</summary>
public static IReadOnlySet<string> KnownExtensions => Extensions;
/// <summary>The path with each segment that is not a known word replaced by <c>{numero}</c> or <c>{testo}</c>.</summary>
/// <remarks>Anything after «?» or «#» is dropped first: only a path is a path.</remarks>
public static string Of(string path)
{
ArgumentNullException.ThrowIfNull(path);
var cut = path.IndexOfAny(['?', '#']);
if (cut >= 0)
{
path = path[..cut];
}
var segments = path.Split('/');
for (var i = 0; i < segments.Length; i++)
{
var segment = segments[i];
if (segment.Length == 0 || Known.Contains(segment))
{
continue;
}
segments[i] = IsDigits(segment) ? Number : Text;
}
return string.Join('/', segments);
}
/// <summary>True for a word of the site kept by <see cref="Of"/>.</summary>
public static bool IsKnownSegment(string segment) => Known.Contains(segment);
/// <summary>The extension of the last segment of <paramref name="path"/> (lower case, no dot) when it is on the closed list; otherwise null.</summary>
public static string? ExtensionOf(string? path)
{
if (string.IsNullOrEmpty(path))
{
return null;
}
var cut = path.IndexOfAny(['?', '#']);
if (cut >= 0)
{
path = path[..cut];
}
var last = path[(path.LastIndexOf('/') + 1)..];
var dot = last.LastIndexOf('.');
if (dot < 0 || dot == last.Length - 1)
{
return null;
}
var extension = last[(dot + 1)..].ToLowerInvariant();
return Extensions.Contains(extension) ? extension : null;
}
private static bool IsDigits(string segment)
{
foreach (var c in segment)
{
if (!char.IsAsciiDigit(c))
{
return false;
}
}
return segment.Length > 0;
}
}
+295 -12
View File
@@ -13,18 +13,69 @@ public static class KeyStructure
public const int DefaultMaxLength = 600;
private const int MaxDepth = 6;
private const int MaxKeysPerObject = 24;
private const int MaxCollectDepth = 256;
private const int MaxCollectedValues = 100_000;
private const int MaxMapCandidates = 64;
private const int MaxElementsCompared = 32;
public static string Describe(JsonElement root, int maxLength = DefaultMaxLength)
/// <summary>Fields that identify an account or an item: two siblings that both have one are entries of a map.</summary>
private static readonly HashSet<string> IdentityFields = new(StringComparer.OrdinalIgnoreCase)
{
var sb = new StringBuilder();
Append(sb, root, 0);
if (sb.Length > maxLength)
"pk", "pk_id", "id", "fbid", "fbid_v2", "strong_id__", "instagram_pk", "user_id", "userID", "username", "full_name",
"profile_pic_url", "code",
};
public static string Describe(JsonElement root, int maxLength = DefaultMaxLength) =>
Write(root, LooksLikeIdentifier, mapEntries: false, maxLength);
/// <summary>
/// The stricter description of the diagnostic report (issue #39), where not even a name may pass: a key is shown
/// only when it is made of words of a closed list (<see cref="StructureWords.IsStructural"/>), and never when it
/// equals, ignoring case, a string value found anywhere in the same body. Every other key becomes «#»: a user name
/// («marta_92», «jerry»), a post code, a short token, an identifier, a key with a dot or a space.
/// </summary>
/// <remarks>
/// Keys made of words of the list can still be names («the_artist», «street_style»), so two more rules apply, and
/// under both only a field of the closed list (<see cref="StructureWords.IsFieldKey"/>: the keys of the reader
/// vocabularies and of the cleaned samples, such as <c>user</c> and <c>owner</c> of a post) stays:
/// <list type="bullet">
/// <item>a key with a plain value (text, number, true or false, null) is a field or «#»: a map of counts or flags keyed
/// by name never shows;</item>
/// <item>the keys of a map are «#»: two or more keys of one object whose values are alike (both objects or both
/// lists, and one of them empty, or both lists of plain values, or with at least half of their keys in common or a
/// shared identity field such as <c>pk</c>), and in a list of two or more objects the keys of the first one that no
/// other has (a list of maps with one entry each).</item>
/// </list>
/// </remarks>
public static string DescribeStrict(JsonElement root, int maxLength = DefaultMaxLength)
{
var values = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
CollectStrings(root, values, 0);
return Write(root, key => !StructureWords.IsStructural(key) || values.Contains(key), mapEntries: true, maxLength);
}
/// <summary>
/// True when a key is data rather than structure: it looks like an identifier, or it has any character other than
/// ASCII letters, digits, «_» and «-» (a user name with a dot, an address, a phrase). Shared with the sample sanitizer
/// (spec §14.5) and the diagnostic report.
/// </summary>
public static bool IsDataKey(string key)
{
ArgumentNullException.ThrowIfNull(key);
if (LooksLikeIdentifier(key))
{
sb.Length = Math.Max(0, maxLength - 1);
sb.Append('…');
return true;
}
return sb.ToString();
foreach (var c in key)
{
if (!(char.IsAsciiLetterOrDigit(c) || c is '_' or '-'))
{
return true;
}
}
return false;
}
/// <summary>
@@ -70,7 +121,237 @@ public static class KeyStructure
return others > 0 && digits >= 5; // e.g. «id:123456»
}
private static void Append(StringBuilder sb, JsonElement element, int depth)
private static string Write(JsonElement root, Func<string, bool> hidden, bool mapEntries, int maxLength)
{
var sb = new StringBuilder();
Append(sb, root, hidden, mapEntries, 0);
if (sb.Length > maxLength)
{
sb.Length = Math.Max(0, maxLength - 1);
sb.Append('…');
}
return sb.ToString();
}
/// <summary>The string values of the body (keys excluded), for <see cref="DescribeStrict"/>.</summary>
private static void CollectStrings(JsonElement element, HashSet<string> values, int depth)
{
if (depth > MaxCollectDepth || values.Count >= MaxCollectedValues)
{
return;
}
switch (element.ValueKind)
{
case JsonValueKind.Object:
foreach (var property in element.EnumerateObject())
{
CollectStrings(property.Value, values, depth + 1);
}
break;
case JsonValueKind.Array:
foreach (var item in element.EnumerateArray())
{
CollectStrings(item, values, depth + 1);
}
break;
case JsonValueKind.String:
if (element.GetString() is { Length: > 0 and <= 256 } text)
{
values.Add(text);
}
break;
}
}
/// <summary>
/// The keys of <paramref name="element"/> that the strict structure hides besides the ones its predicate already
/// rejects (see <see cref="DescribeStrict"/>): keys with a plain value, entries of a map, and
/// <paramref name="extra"/> (keys that only the first element of a list has), except the fields of the closed list.
/// Null when there is none.
/// </summary>
private static HashSet<string>? MapEntries(JsonElement element, HashSet<string>? extra)
{
HashSet<string>? entries = null;
void Hide(string key)
{
if (!StructureWords.IsFieldKey(key))
{
(entries ??= new HashSet<string>(StringComparer.Ordinal)).Add(key);
}
}
List<(string Key, Shape Shape)>? candidates = null;
var many = false;
foreach (var property in element.EnumerateObject())
{
var shape = ShapeOf(property.Value);
if (shape is null)
{
Hide(property.Name);
}
else if (many || (candidates?.Count ?? 0) >= MaxMapCandidates)
{
// So many objects or lists side by side are a map: no need to compare them.
many = true;
Hide(property.Name);
}
else
{
(candidates ??= []).Add((property.Name, shape));
}
}
if (candidates is { Count: >= 2 })
{
var alike = new bool[candidates.Count];
for (var i = 0; i < candidates.Count; i++)
{
for (var j = i + 1; j < candidates.Count; j++)
{
if (many || Alike(candidates[i].Shape, candidates[j].Shape))
{
alike[i] = alike[j] = true;
}
}
}
for (var i = 0; i < candidates.Count; i++)
{
if (alike[i])
{
Hide(candidates[i].Key);
}
}
}
if (extra is not null)
{
foreach (var key in extra)
{
Hide(key);
}
}
return entries;
}
/// <summary>
/// Two values of one object are alike (entries of a map): both objects or both lists, and one of them empty, or both
/// lists of plain values, or both with keys (of the object, of the first element of the list) half in common or an
/// identity field in common.
/// </summary>
private static bool Alike(Shape a, Shape b)
{
if (a.List != b.List)
{
return false;
}
if (a.Keys is null || b.Keys is null)
{
// A list of plain values is like an empty list or another list of plain values, never like a list of objects.
return a.Keys is not { Count: > 0 } && b.Keys is not { Count: > 0 };
}
if (a.Keys.Count == 0 || b.Keys.Count == 0)
{
return true;
}
var common = 0;
foreach (var key in a.Keys)
{
if (b.Keys.Contains(key))
{
if (IdentityFields.Contains(key))
{
return true;
}
common++;
}
}
var union = a.Keys.Count + b.Keys.Count - common;
return common * 2 >= union;
}
/// <summary>
/// The keys of the first element of a list of two or more objects that no other element has (among the first
/// <see cref="MaxElementsCompared"/>): in a list of maps with one entry each, the entries. Null when there is none.
/// </summary>
private static HashSet<string>? OnlyInFirst(JsonElement array)
{
HashSet<string>? first = null;
var others = 0;
var seen = 0;
foreach (var item in array.EnumerateArray())
{
if (++seen > MaxElementsCompared)
{
break;
}
if (item.ValueKind != JsonValueKind.Object)
{
continue;
}
if (first is null)
{
first = KeysOf(item);
continue;
}
others++;
foreach (var property in item.EnumerateObject())
{
first.Remove(property.Name);
}
}
return others > 0 && first is { Count: > 0 } ? first : null;
}
/// <summary>
/// What a value is, for the map rule: an object or a list (<paramref name="List"/>), with the keys of the object or of
/// the first element of the list (empty: an empty object or list, or a list whose first element is an empty object;
/// null: a list of plain values or of lists).
/// </summary>
private sealed record Shape(bool List, HashSet<string>? Keys);
/// <summary>The shape of a value; null for a plain value (text, number, true or false, null).</summary>
private static Shape? ShapeOf(JsonElement value)
{
switch (value.ValueKind)
{
case JsonValueKind.Object:
return new Shape(false, KeysOf(value));
case JsonValueKind.Array:
using (var items = value.EnumerateArray())
{
if (!items.MoveNext())
{
return new Shape(true, []);
}
return new Shape(true, items.Current.ValueKind == JsonValueKind.Object ? KeysOf(items.Current) : null);
}
default:
return null;
}
}
private static HashSet<string> KeysOf(JsonElement value) =>
value.EnumerateObject().Select(p => p.Name).ToHashSet(StringComparer.Ordinal);
private static void Append(StringBuilder sb, JsonElement element, Func<string, bool> hidden, bool mapEntries, int depth, HashSet<string>? onlyInFirst = null)
{
switch (element.ValueKind)
{
@@ -85,9 +366,10 @@ public static class KeyStructure
var count = 0;
var identifierKeys = 0;
JsonElement? identifierSample = null;
var entries = mapEntries ? MapEntries(element, onlyInFirst) : null;
foreach (var property in element.EnumerateObject())
{
if (LooksLikeIdentifier(property.Name))
if (hidden(property.Name) || entries?.Contains(property.Name) == true)
{
identifierKeys++;
identifierSample ??= property.Value;
@@ -115,7 +397,7 @@ public static class KeyStructure
if (property.Value.ValueKind is JsonValueKind.Object or JsonValueKind.Array)
{
sb.Append(':');
Append(sb, property.Value, depth + 1);
Append(sb, property.Value, hidden, mapEntries, depth + 1);
}
count++;
@@ -132,7 +414,7 @@ public static class KeyStructure
if (identifierSample is { ValueKind: JsonValueKind.Object or JsonValueKind.Array } sample)
{
sb.Append(':');
Append(sb, sample, depth + 1);
Append(sb, sample, hidden, mapEntries, depth + 1);
}
}
@@ -150,7 +432,8 @@ public static class KeyStructure
{
if (item.ValueKind is JsonValueKind.Object or JsonValueKind.Array)
{
Append(sb, item, depth + 1);
var extra = mapEntries && item.ValueKind == JsonValueKind.Object ? OnlyInFirst(element) : null;
Append(sb, item, hidden, mapEntries, depth + 1, extra);
}
break; // the first element stands for the others
@@ -0,0 +1,374 @@
using System.Collections.Frozen;
using System.Text.Json;
using Febe.Core.Collection.Privacy;
namespace Febe.Core.Collection.Readers;
/// <summary>
/// The closed list of words a key may be made of to appear in the strict key structure of the diagnostic report
/// (issue #39, <see cref="KeyStructure.DescribeStrict"/>). Every other key is written «#»: a user name, a post code or a
/// short token used as a key never leaves the machine, even when the body does not repeat it as a value.
/// </summary>
/// <remarks>
/// <para>
/// A key passes when it has only ASCII letters, digits, «_» and «-», does not look like an identifier, and every part
/// between «_» and «-» is a word of the list (case ignored): <c>image_versions2</c>, <c>__typename</c>,
/// <c>has_more_headload_comments</c>. The words come from the keys of the cleaned samples in <c>tests/Febe.Tests/campioni</c>
/// (structure only: the sanitizer already replaced every key that is data), from the shipped reader vocabularies
/// (<c>lettori.instagram.json</c>, <c>lettori.prova.json</c>) and from the words of the site's addresses
/// (<see cref="PathPatterns.KnownSegments"/>). A missing word only costs detail («#»), never privacy.
/// </para>
/// <para>
/// A few words that are also first names, places or opaque codes (<see cref="CompoundOnly"/>) pass only inside a key
/// that also has a word of the list (<c>next_max_id</c>), never alone (a user called «max») nor with one another
/// (<c>max_mark</c>, two first names).
/// </para>
/// <para>
/// A name made only of words of the list («the_artist», «street_style») still passes this check, so the strict structure
/// (<see cref="KeyStructure.DescribeStrict"/>) asks more of two kinds of keys. A key with a plain value (text, number,
/// true or false, null) stays only when it is a field of the closed list (<see cref="IsFieldKey"/>): the entries of a
/// map of counts or flags keyed by name never show. The entries of a map of objects or lists are found by their
/// siblings (similar keys, a shared identity field such as <c>pk</c>, an empty value, lists of plain values, or a key
/// that only the first element of a list has), and among them too only the fields of the closed list stay. A
/// single such name used as the key of an object or a list, with no sibling alike and never repeated as a value in the
/// body, is the residual risk of a closed list of words.
/// </para>
/// </remarks>
public static class StructureWords
{
private const string SampleWords = """
a accessibility account achievements action activation active ad add additional address ads affiliate agent
aggregated ai aigm all allow allowed alpha altezza altro and anonymous api app appearance appreciation archive
are artist as asset associated at attachment attribute attributed attribution attributions audience audio author
authors auto autodub automatically autore autosave available awaiting award background badge badges bandwidth
barra base batch bbox be been behavior benefits best bestie besties bio biography bit blocked blocking bloks
bookmarked boost boosted bottomsheet brand branded breaking broadcast broadcasts business button by cache camera
can candidates canonical caption capture carousel category chaining challenge chat check city client clip clips
club cluster coauthor code codecs collab collaborator collection color comment commenter commenting comments
commerce commission community complete composer composition config configs connected connection consideration
consumption container content contenuti contenuto context contextual control copertina count countdowns counts
cover covered created creation creative creator cropped cropping crosspost crosspostable cta cursor custom cut
cutout dash data debug decision default delayed deleted demotion deny derived details detected detection dettagli
device did disable disabled discover display distribution dopo download downloads downstream durata duration
eager earned edges edit edited editing educational effect effects elements eligibility eligible eligiblity
embeds embodiment emojis enable enabled encoding end enforced enough entities entry entrypoint entrypoints
errors estimated event exclusive expansion expiration expiring explicit explore extensions external face
facepile failure fallback fan favorite fb fbid fbsearch featured feed feedback fetch fetching file filmati
filter filtered filters final first fissato fitting flags floating flush follow followed follower followers
following for formatted foto foundation frame free friction friendship from full gating gen genai geoblocked
gif gifting graphql grid group guardian has hashtag hashtags have hd header headline headload height hidden
hide high highlight highlights host hypercard id identifier ids ig igbio igids igtv image improvements in
incoming indirizzo info inform infos initial initiate inline insert insights instagram integrity interaction
interactive internal interop inventory invited invocation io is item items kb key label labels langs language
largest larghezza lat latest length light like liked likers likes link linked links list live lng load locales
location locations logging long longform main manifest manually marketplace mashed mashup mashups may media
medias member memorialized mention merchandising message messaging meta metadata method mids mime module more
ms music mute muted muting mutual name new news next node nome non nonmimicable note notes notice notices num
number numero nux of on only open options organic original originality outgoing overlap overlay owner page
pagina paid param parent parti partnership parts party path payload per photo pic picture pill pinned pivot
pivots pk places play playback player playlist playlists point pos position post prefetch premium presets preview
previous privacy private privately producer producers product production products professional profile profilo
progressive project promo promotion promotions prompts pronouns pubblicato pubblicazioni public qualities
quando questions quick quiet raccolta raccolte rank ranked ranking reaction reactions reason reattribution
recommendation recs reel reels referral refresh regulated related remix remixes remixing remove rename render
rendered reply report repost reposter representation representations request requesting require reshare reshared
resolution response rest restore restricted restyle result results reusable reuse revamp review ring riservato
risk risorsa risultati ritratto roll root rotation routing row save saved scade scans scroll scrubber search
secondary seconds see seen segments seguiti seguito selector sequence server settings severity share shared
sharing shop shopping short should show sigla size sizes sliders smart snippet social sort sound source spam
spatial speed sponsor sprite spritesheet start starting stato status sticker stickers storia storie story street
string strong style submission submitted submitters subscribe subscriber subscribers subtype successiva successivo
suggested supervision surface tab tag tagged tagging tags taken tappable tappables teen template testo text the
third threading thumbnail thumbnails time timeline timeout timer timestamp tipo tips title titolo to toggle
token tool top topic topics topsearch total totali tracking translated translation translations transparency
tray treatment trend trending trial type typename ufi ui ultima unavailable unicode unified unpublished unseen
unviewable up upcoming upsell uri url urls us use user userid username users usertags utc v1 v2 variant verified
version versione versions versions2 video view viewer viewers views visibile visibility visible visual voci
wearable wearables weight welcome width with x xdt xdtuserdict xpost y you zip
""";
/// <summary>
/// Every key of the cleaned samples of <c>tests/Febe.Tests/campioni</c> (their data keys were already replaced by the
/// sanitizer), frozen here because the samples do not ship with Febe: with the keys of the reader vocabularies, the
/// closed list of fields (<see cref="IsFieldKey"/>).
/// </summary>
private const string SampleKeys = """
__bbox __typename accessibility_caption account_badges account_type achievements_info action_type
active_meta_verified_benefits additional_audio_info additional_audio_username additional_candidates
address_street ai_agent_owner_username ai_agent_type ai_interactive_embodiment_attachment_style_info
ai_label_info ai_topic_filters aigm_account_label_info all_previous_submitters
all_video_dash_prefetch_representations allow_creator_to_rename altezza altro appearance_effect
are_remixes_crosspostable asset_recommendation_info associated_clip_medias
attributed_custom_audio_asset_id attribution_label_text attribution_user audience audio audio_asset_id
audio_asset_start_time_in_ms audio_filter_infos audio_parts audio_parts_by_filter audio_ranking_info
audio_reattribution_info audio_type auto_load_more_enabled autore autosave_to_exclusive_highlight
bandwidth barra base_url basel_encoding_timeout best_audio_cluster_id bio_links biography
biography_with_entities bit_flags blocking bloks_app_url bloks_sticker boost_unavailable_identifier
boost_unavailable_reason boost_unavailable_reason_v2 boosted_status branded_content_tag_info
breaking_content_info breaking_creator_info broadcasts camera_tool can_add_tag can_remix_be_shared_to_fb
can_remix_be_shared_to_fb_expansion can_reply can_reshare can_see_insights_as_brand
can_see_organic_insights can_see_quiet_post_attribution can_toggle_mashups_allowed
can_view_more_preview_comments can_viewer_reshare can_viewer_save candidates caption caption_is_edited
capture_type carousel_media carousel_media_count carousel_parent_id category challenge_info chiave
city_name client_cache_key clips_attribution_info clips_connection clips_creation_entry_point
clips_metadata clips_tab_pinned_user_ids coauthor_producer_can_see_organic_insights coauthor_producers
code codecs collab_follow_button_info collaborator_edit_eligibility color_filters comment comment_count
comment_cover_pos comment_filter_param comment_inform_treatment comment_likes_enabled
commenting_disabled_for_viewer comments comments_disabled commerce_integrity_review_decision
community_notes_info complete connected_member_count consumption_info content_appreciation_info
content_type contenuti contenuto contextual_highlight_info copertina corpo cover_media created_at
created_at_for_fb_app created_at_utc creation_tool_info creative_config
creator_is_in_premium_reactions_alpha cropped_image_version crosspost_metadata cursor custom_list_info
cutout_sticker_info dash_manifest data debug_link default deleted_reason
derived_content_start_time_in_composition_in_ms descrittore detection_method dettagli device_timestamp
did_report_as_spam disable_caption_and_comment disable_use_in_clips_client_cache display_artist
display_labels display_labels_with_translations display_media_id display_uri dopo
downstream_use_xpost_deny_reason durata duration_in_ms duration_selector_seconds edges effect_configs
effect_filter_name effect_preview eligible_for_text_app_activation_badge eligible_insights_entrypoints
eligible_promotions enable_media_notes_production enable_submission_friction enabled end end_cursor
entities entry_point_container errors estimated_scans_sizes expiring_at explore extensions
external_lynx_url external_media_info external_url face_effect_id facepile_top_likers failure_reason
fallback fan_club_id fan_club_info fan_club_name fan_consideration_page_revamp_eligiblity
fb_comment_count fb_comments fb_downstream_use_xpost_metadata fb_like_count fb_play_count fb_user_tags
fbid fbid_v2 featured_label featured_products feed_demotion_control feed_post_reshare_disabled
feed_recs_demotion_control fetch__XDTUserDict file_size_kb filmati filter_options filter_type
first_frame fissato floating_context_item_type floating_context_items follow_hashtag_info followed_by
follower_count following following_count formatted_clips_media_count formatted_mashups_count foto
foundation_improvements_enabled friendship_status full_image_version full_name gating
gen_ai_detection_method gen_ai_tool_info geoblocked_comment_notices group guardian_hypercard_content
has_anonymous_profile_picture has_audio has_been_mashed_up has_besties_media has_chaining
has_created_ssc has_delayed_metadata has_enough_subscribers_for_ssc
has_high_risk_gen_ai_inform_treatment has_liked has_longform_media has_more_comments
has_more_headload_comments has_more_headload_fb_comments has_next_page has_nonmimicable_additional_audio
has_previous_page has_privately_liked has_profile_pic has_restyle_presets has_shared_to_fb
has_story_archive has_tagged_users has_translation has_viewer_saved has_viewer_submitted_note
has_views_fetching has_visible_media_notes hashtags hd_profile_pic_url_info hd_profile_pic_versions
headline height hidden_likes_string_variant hide_creator_marketplace_badge hide_remixing
hide_stickers_for_original_language hide_view_all_comment_entrypoint highlights horizon_attribution_info
host id ig_artist ig_media_sharing_disabled ig_mention ig_play_count ig_quick_promotion_batch_fetch_root
igbio_product igtv_first_frame image_versions2 in incoming_request indirizzo inform_module
initial_representation_ids initiate_at_top inline_audio_label insert_new_comment_to_top instagram_pk
integrity_review_decision interop_messaging_user_fbid inventory_source invited_coauthor_producers io
is_active_on_text_post_app is_active_trial is_affiliate_commission_eligible is_ai_user
is_audio_automatically_attributed is_auto_detected is_awaiting_distribution is_bestie is_bookmarked
is_business is_cannes is_comments_gif_composer_enabled is_coppa_enforced is_covered
is_created_by_media_owner is_creator_requesting_mashup is_cutout_sticker_allowed is_dash_eligible
is_educational is_eligible_content_for_post_roll_ad is_eligible_for_audio_effects
is_eligible_for_autodub is_eligible_for_autodub_upsell is_eligible_for_organic_eager_refresh
is_eligible_for_poe is_eligible_for_poe_premium is_eligible_for_request_a_note is_eligible_for_rta
is_eligible_for_vinyl_sticker is_embeds_disabled is_enabled is_explicit is_fan_club_gifting_eligible
is_fan_club_promo_video is_fan_club_referral_eligible is_favorite is_fb_only is_feed_favorite
is_feed_feedback_aggregated is_feedback_aggregated is_feta_guardian_user is_final is_free_trial_eligible
is_guardian_user is_in_profile_grid is_internal_only is_light_weight_check
is_light_weight_reuse_allowed_check is_longform_player_eligible is_manually_enabled is_memorialized
is_meta_ai_invocation_blocked_by_author is_muting_reel is_open_to_public_submission
is_organic_product_tagging_eligible is_original_audio_download_eligible is_paid_partnership
is_photo_comments_composer_enabled_for_author is_pivot_page_available is_post_live_clips_media
is_private is_professional_account is_public_chat_welcome_video is_quiet_post is_ranked_comment
is_regulated_c18 is_reshare_of_text_post_app_media_in_ig is_restricted is_reuse_allowed
is_reuse_disabled is_ring_creator is_seen is_shared_from_basel is_shared_to_fb is_short_drama
is_social_ufi_disabled is_spatial_image is_story_edited is_third_party_downloads_eligible
is_trending_in_clips is_unpublished is_verified is_visual_reply_commenter_notice_enabled
is_xpost_from_fb items largest_public_bc_id larghezza lat latest_besties_reel_media latest_reel_media
latest_wearables_reel_media_long like_and_view_counts_disabled like_count
liked_by_media_owner_badge_enabled link link_text linked_fb_info linked_media_playlist_data
live_broadcast_id live_broadcast_visibility lng location locations logging_info_token
longform_clip_metadata longform_title magic_cut_end_time magic_cut_start_time
main_feed_carousel_starting_media_id mashup_info mashup_type mashups_allowed
mashups_allowed_for_carousel max_thumbnails_per_sprite may_have_template_info media
media_attributions_data media_count media_cropping_info media_header_display media_id media_note
media_notes media_overlay_info media_repost_count media_reposter_bottomsheet_enabled media_type
media_ui_attributions_data media_ui_attributions_data_v2 merchandising_pill_info message
meta_ai_suggested_prompts meta_verified_benefits_info mezql_token mids mime_type more_available
music_asset_info music_canonical_id music_info music_metadata muted muting mutual_followers_count name
node nome nome_visibile non_privacy_filtered_mashups_media_count note_submission_disabled
num_earned_achievements num_results number_of_qualities numero nux_info oa_owner_is_music_artist
organic_tracking_token original_audio_subtype original_audio_title original_height original_media
original_media_id original_sound_info original_width originality_info outgoing_request
overlap_duration_in_ms owner owner_id page_info pagina parti parti_totali path photo_of_you pk pk_id
places play_count playback_resolution_csvqm playback_resolution_mos position premium_reaction_count
preview preview_comments previous_trend_rank privacy_filtered_mashups_media_count private_reply_status
product_type professional_clips_upsell_type profile_background_color profile_background_secondary_color
profile_context_facepile_users profile_context_links_with_user_ids profile_grid_control_enabled
profile_grid_thumbnail_fitting_style profile_pic_genai_tool_info profile_pic_id profile_pic_url profilo
progressive_download_url progressive_identifier pronouns pubblicato pubblicazioni quando
quick_response_emojis raccolta raccolte rank_token ranked_position reactions reel_media_background
reel_media_seen_timestamp reel_type reels_media reels_on_the_rise_info regulated_news_in_locations
related_ads_pivots_media_info remove_message_entrypoint rendered_width representation_id representations
request_start_time_ms require reshare_count reshared_story_media_author result
reusable_text_attribute_string reusable_text_info ring_creator_metadata riservato risorsa risultati
ritratto rotation saved_collection_ids scade scans_profile scroll_behavior
scrubber_spritesheet_info_candidates search_social_context search_social_context_snippet_type see_more
seen seen_ranked_position segments seguiti seguito sequence_number server_metadata severity
share_count_disabled share_enabled sharing_friction_info sharing_friction_payload shop_routing_user_id
shopping_info short_drama_creator should_allow_audio_editing should_allow_restore
should_have_inform_treatment should_have_sharing_friction should_mute_audio should_mute_audio_reason
should_mute_audio_reason_type should_render_upsell should_request_ads
should_show_author_pog_for_tagged_media_shared_to_profile_grid should_show_category
should_show_playlists_in_profile_tab show_account_transparency_details show_achievements show_ring_award
show_text_post_app_badge show_tips sigla smart_frame social_context social_context_facepile_users
social_context_type social_context_users_count sort_options speed_selector sponsor_tags sprite_height
sprite_urls sprite_width start start_cursor stato status sticker_data storie story_app_attribution
story_bloks_stickers story_bloks_tappables story_countdowns story_cta story_feed_media story_hashtags
story_link_stickers story_locations story_music_stickers story_questions story_sliders strong_id__
subscribe_cta_visible subscriber_count subtype_name_for_REST__ successiva successivo supervision_info
surface_nux_id taken_at tappable_elements teen_project_info template_info testo text
text_post_app_badge_label text_post_app_is_private text_post_share_to_ig_story_stickers
third_party_downloads_enabled threading_enabled thumbnail_duration thumbnail_height thumbnail_width
thumbnails thumbnails_per_row time_at_flush_ms time_created timeline_pinned_user_ids
timer_selector_seconds tipo title titolo top_likers total_clips_count total_thumbnail_num_per_sprite
translated_langs_for_autodub transparency_label transparency_product transparency_product_enabled tray
trend_rank tya_media_related_topics type ultima_storia unicode unified_feedback_enabled unseen_count
unseen_wearables_media_igids unviewable_authors_infos upcoming_event url url_expiration_timestamp_us
user userID user_activation_info user_id user_notes username users usertags valore versione
video_dash_manifest video_duration video_id video_length video_sticker_locales video_versions view_count
viewer viewer_count viewer_has_liked viewer_interaction_settings viewer_premium_reaction_asset_id
viewers visibility visual_comment_reply_sticker_info voci wearable_attribution_info width x
xdt_api__v1__discover__chaining xdt_api__v1__fbsearch__topsearch_connection
xdt_api__v1__feed__reels_media xdt_api__v1__feed__reels_media__connection xdt_api__v1__feed__reels_tray
xdt_api__v1__feed__user_timeline_graphql_connection xdt_api__v1__usertags__user_id__feed_connection
xdt_mark_story_reel_seen xdt_viewer xpost_fb_creator_info y zip
""";
private static readonly FrozenSet<string> CompoundOnlyWords = FrozenSet.Create(
StringComparer.OrdinalIgnoreCase,
"max", "mark", "basel", "cannes", "lynx", "feta", "poe", "drama", "magic", "rise", "vinyl", "horizon", "coppa",
"csvqm", "tya", "rta", "ssc", "oa", "bc", "pog", "mezql", "c18", "mos");
private static readonly Lazy<Lists> Built = new(Build);
/// <summary>Words that pass only as a part of a key that also has a word of the list.</summary>
public static IReadOnlySet<string> CompoundOnly => CompoundOnlyWords;
/// <summary>True for a word of the list (case ignored).</summary>
public static bool IsWord(string word) => Built.Value.Words.Contains(word);
/// <summary>
/// True for a whole key of the shipped reader vocabularies (<c>user</c>, <c>owner</c>, <c>image_versions2</c>), case
/// ignored: the only keys the strict structure keeps among the entries of a map.
/// </summary>
public static bool IsKnownKey(string key) => Built.Value.Keys.Contains(key) && IsStructural(key);
/// <summary>
/// True for a field of the closed list: a whole key of the reader vocabularies (<see cref="IsKnownKey"/>) or of the
/// cleaned samples (<c>is_private</c>, <c>has_next_page</c>, <c>pubblicazioni</c>), case ignored, that is also
/// structural. In the strict structure, a key with a plain value (text, number, true or false, null), and a key
/// among the entries of a map, stays only when it is one of these.
/// </summary>
public static bool IsFieldKey(string key) =>
(Built.Value.Keys.Contains(key) || Built.Value.SampleKeys.Contains(key)) && IsStructural(key);
/// <summary>True when <paramref name="key"/> may appear as it is in the strict key structure.</summary>
public static bool IsStructural(string key)
{
ArgumentNullException.ThrowIfNull(key);
if (key.Length == 0 || key.Length > 120 || KeyStructure.IsDataKey(key))
{
return false;
}
var parts = key.Split(['_', '-'], StringSplitOptions.RemoveEmptyEntries);
if (parts.Length == 0)
{
return false;
}
var words = Built.Value.Words;
var listWord = false;
foreach (var part in parts)
{
if (words.Contains(part))
{
listWord = true;
}
else if (!CompoundOnlyWords.Contains(part))
{
return false;
}
}
// Compound-only words never stand on their own, nor together («max_mark»): at least one part is a word of the list.
return listWord;
}
private static Lists Build()
{
var words = new HashSet<string>(SampleWords.Split((char[]?)null, StringSplitOptions.RemoveEmptyEntries), StringComparer.OrdinalIgnoreCase);
var keys = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
foreach (var segment in PathPatterns.KnownSegments)
{
AddParts(words, segment);
}
foreach (var file in new[] { "lettori.instagram.json", "lettori.prova.json" })
{
try
{
using var document = JsonDocument.Parse(SiteMap.ReadEmbeddedText(file));
AddVocabulary(words, keys, document.RootElement, 0);
}
catch (Exception ex) when (ex is JsonException or IOException)
{
// The sample words are enough: a vocabulary only adds detail.
}
}
words.ExceptWith(CompoundOnlyWords);
var sampleKeys = SampleKeys.Split((char[]?)null, StringSplitOptions.RemoveEmptyEntries).ToFrozenSet(StringComparer.OrdinalIgnoreCase);
return new Lists(words.ToFrozenSet(StringComparer.OrdinalIgnoreCase), keys.ToFrozenSet(StringComparer.OrdinalIgnoreCase), sampleKeys);
}
/// <summary>
/// The key names of a vocabulary file: every string of its lists, split at «.» (whole keys) and then at «_» and «-»
/// (words).
/// </summary>
private static void AddVocabulary(HashSet<string> words, HashSet<string> keys, JsonElement element, int depth)
{
if (depth > 8)
{
return;
}
switch (element.ValueKind)
{
case JsonValueKind.Object:
foreach (var property in element.EnumerateObject())
{
AddVocabulary(words, keys, property.Value, depth + 1);
}
break;
case JsonValueKind.Array:
foreach (var item in element.EnumerateArray())
{
if (item.ValueKind == JsonValueKind.String && item.GetString() is { } path)
{
foreach (var key in path.Split('.', StringSplitOptions.RemoveEmptyEntries))
{
if (!KeyStructure.IsDataKey(key))
{
keys.Add(key);
AddParts(words, key);
}
}
}
else
{
AddVocabulary(words, keys, item, depth + 1);
}
}
break;
}
}
private sealed record Lists(FrozenSet<string> Words, FrozenSet<string> Keys, FrozenSet<string> SampleKeys);
private static void AddParts(HashSet<string> words, string key)
{
foreach (var part in key.Split(['_', '-'], StringSplitOptions.RemoveEmptyEntries))
{
words.Add(part);
}
}
}
@@ -1,5 +1,6 @@
using System.Text.Json;
using Febe.Core.Abstractions;
using Febe.Core.Collection.Diagnostics;
using Febe.Core.Collection.Readers;
namespace Febe.Core.Collection.Recipes;
@@ -39,6 +40,12 @@ public sealed class CaptureCollector : IResponseConsumer, IResponseObserver
public ShapeReader Reader => _reader;
/// <summary>
/// The recorder of the diagnostic report (issue #39), told what the readers did with every body, also the ones
/// not kept (another host, unrelated); null: nothing is recorded.
/// </summary>
public CollectionDiagnostics? Diagnostics { get; init; }
/// <summary>How long a data response may stay «on its way» before it is forgotten (its body may never come).</summary>
public TimeSpan PendingLifetime { get; init; } = TimeSpan.FromSeconds(15);
@@ -165,10 +172,12 @@ public sealed class CaptureCollector : IResponseConsumer, IResponseObserver
if (response.HostKind != HostKind.Site)
{
Observe(response, null, body);
return null;
}
var result = _reader.Read(body);
Observe(response, result, body);
if (result.ViewerId is { } viewerId)
{
lock (_gate)
@@ -197,6 +206,28 @@ public sealed class CaptureCollector : IResponseConsumer, IResponseObserver
}
}
/// <summary>
/// Tells the diagnostics what the readers made of a body; for an unknown shape, its strict key structure, taken
/// now, while the document is alive. Never a reason to fail the reading.
/// </summary>
private void Observe(ResponseDescriptor response, ReadResult? result, JsonElement body)
{
if (Diagnostics is not { } diagnostics)
{
return;
}
try
{
var structure = result?.Shape == ReadShape.Unrecognized ? KeyStructure.DescribeStrict(body) : null;
diagnostics.ReadObserved(response, result, structure);
}
catch (Exception ex) when (ex is not OutOfMemoryException)
{
// Diagnostics are optional: the reading goes on as if they were not there.
}
}
void IResponseObserver.OnResponse(ResponseDescriptor response)
{
if (response.HostKind == HostKind.Site
@@ -1,5 +1,6 @@
using System.Globalization;
using Febe.Core.Abstractions;
using Febe.Core.Collection.Diagnostics;
using Febe.Core.Collection.Readers;
using Febe.Core.Domain;
using Febe.Core.Storage;
@@ -44,6 +45,9 @@ public sealed class CollectionWriter(IFebeStorage storage, IClock clock)
public IFebeStorage Storage => storage;
/// <summary>The recorder of the diagnostic report (issue #39): files put in the download queue. Null: not recorded.</summary>
public CollectionDiagnostics? Diagnostics { get; init; }
/// <summary>
/// A profile seen in the data. The account is identified by its numeric id; when its name changed, a followed
/// profile is renamed (folder kept, notice raised) and a profile that is not followed becomes «Non raggiungibile»
@@ -138,22 +142,29 @@ public sealed class CollectionWriter(IFebeStorage storage, IClock clock)
{
ArgumentNullException.ThrowIfNull(account);
ArgumentNullException.ThrowIfNull(item);
var (result, queued, skipped) = await SaveCoreAsync(account, item, kind, ct).ConfigureAwait(false);
Diagnostics?.FilesQueued(sessionId, kind, queued, result.Outcome.ToString(), skipped);
return result;
}
private async Task<(SaveResult Result, int Queued, int Skipped)> SaveCoreAsync(Account account, ContentItem item, EntryKind kind, CancellationToken ct)
{
if (!Wants(account, kind))
{
return new SaveResult(SaveOutcome.Excluded, null);
return (new SaveResult(SaveOutcome.Excluded, null), 0, 0);
}
var known = await storage.Entries.FindAsync(account.Id, kind, item.Id, ct).ConfigureAwait(false);
if (known is not null)
{
await RefreshFilesAsync(known, item, fromHighlight: false, ct).ConfigureAwait(false);
return new SaveResult(SaveOutcome.Known, known);
return (new SaveResult(SaveOutcome.Known, known), 0, 0);
}
if (kind is EntryKind.Post or EntryKind.Tagged
&& account.Limits.Depth == HistoryDepth.FromDate && account.Limits.FromDate is { } from && item.TakenAt < from)
{
return new SaveResult(SaveOutcome.Excluded, null);
return (new SaveResult(SaveOutcome.Excluded, null), 0, 0);
}
if (kind != EntryKind.ProfilePicture
@@ -166,7 +177,7 @@ public sealed class CollectionWriter(IFebeStorage storage, IClock clock)
Caption = imported.Caption ?? item.Caption,
};
await storage.Entries.UpdateAsync(enriched, ct).ConfigureAwait(false);
return new SaveResult(SaveOutcome.Enriched, enriched);
return (new SaveResult(SaveOutcome.Enriched, enriched), 0, 0);
}
var now = clock.Now;
@@ -175,7 +186,7 @@ public sealed class CollectionWriter(IFebeStorage storage, IClock clock)
var added = await storage.Entries.AddAsync(entry, media, ct).ConfigureAwait(false);
if (!added.Created)
{
return new SaveResult(SaveOutcome.Known, added.Stored.Entry);
return (new SaveResult(SaveOutcome.Known, added.Stored.Entry), 0, 0);
}
if (!item.IsComplete && kind is EntryKind.Post or EntryKind.Tagged)
@@ -183,7 +194,7 @@ public sealed class CollectionWriter(IFebeStorage storage, IClock clock)
await QueueSinglePostAsync(added.Stored.Entry, ct).ConfigureAwait(false);
}
return new SaveResult(SaveOutcome.Created, added.Stored.Entry);
return (new SaveResult(SaveOutcome.Created, added.Stored.Entry), media.Count(m => m.State == MediaState.ToDownload), media.Count(m => m.State == MediaState.Skipped));
}
/// <summary>
@@ -201,6 +212,7 @@ public sealed class CollectionWriter(IFebeStorage storage, IClock clock)
var added = await storage.Entries.AddAsync(picture.Entry, [picture.Media], ct).ConfigureAwait(false);
if (added.Created)
{
Diagnostics?.FilesQueued(null, EntryKind.ProfilePicture, picture.Media.State == MediaState.ToDownload ? 1 : 0, nameof(SaveOutcome.Created));
return new SaveResult(SaveOutcome.Created, added.Stored.Entry);
}
@@ -259,6 +271,7 @@ public sealed class CollectionWriter(IFebeStorage storage, IClock clock)
}
}
Diagnostics?.FilesQueued(null, entry.Kind, changed, CollectionDiagnostics.CompletedOutcome);
if ((entry.Code is null && item.Code is not null) || (entry.Caption is null && item.Caption is not null))
{
var latest = await storage.Entries.GetAsync(entry.Id, ct).ConfigureAwait(false) ?? entry;
+21 -1
View File
@@ -1,4 +1,5 @@
using Febe.Core.Abstractions;
using Febe.Core.Collection.Diagnostics;
using Febe.Core.Collection.Readers;
using Febe.Core.Domain;
using Febe.Core.Engine;
@@ -9,6 +10,9 @@ namespace Febe.Core.Collection.Recipes;
internal sealed class RecipeStopException(RecipeStatus status, string message) : Exception(message)
{
public RecipeStatus Status { get; } = status;
/// <summary>For a failure on an unexpected error: the type name of that exception (diagnostic report, never the message).</summary>
public string? ErrorType { get; init; }
}
/// <summary>
@@ -96,7 +100,19 @@ internal sealed class RecipeRun : IDisposable
public async Task<PageLoad> OpenAsync(string page, IReadOnlyDictionary<string, string>? values = null)
{
var path = Env.SiteMap.BuildPath(page, values);
var load = await Env.Driver.OpenAsync(path, Ct).ConfigureAwait(false);
PageLoad load;
try
{
load = await Env.Driver.OpenAsync(path, Ct).ConfigureAwait(false);
}
catch
{
Env.Diagnostics?.PageOpened(Session.SessionId, page, ok: false);
throw;
}
// The page NAME of the site map, never the path: it holds the user name or the code.
Env.Diagnostics?.PageOpened(Session.SessionId, page, load.Succeeded);
ThrowIfSignal();
if (!load.Succeeded)
{
@@ -114,6 +130,7 @@ internal sealed class RecipeRun : IDisposable
{
ThrowIfSignal();
var result = await Env.Driver.ClickAsync(target, parameters, Ct).ConfigureAwait(false);
Env.Diagnostics?.TargetClicked(Session.SessionId, target, result.Status);
if (!result.Found)
{
ThrowIfSignal();
@@ -167,6 +184,7 @@ internal sealed class RecipeRun : IDisposable
{
if (reads.Any(r => r.Result.Shape == ReadShape.Recognized && expected(r.Result)))
{
Env.Diagnostics?.StepJudged(Session.SessionId, CollectionDiagnostics.VerdictRecognized);
_signals.ObserveNonEmptyPage();
return true;
}
@@ -174,10 +192,12 @@ internal sealed class RecipeRun : IDisposable
var unknown = reads.Where(r => r.Result.Shape == ReadShape.Unrecognized && r.Result.KeyStructure is not null).Select(r => r.Result.KeyStructure!).Distinct().ToList();
if (unknown.Count > 0)
{
Env.Diagnostics?.StepJudged(Session.SessionId, CollectionDiagnostics.VerdictUnknown);
UnknownShapes.AddRange(unknown);
throw Stop(RecipeStatus.UnrecognizedFormat, "La risposta del sito ha una forma che i lettori non riconoscono.");
}
Env.Diagnostics?.StepJudged(Session.SessionId, CollectionDiagnostics.VerdictEmpty);
_signals.ObserveEmptyPage(where);
ThrowIfSignal();
return false;
@@ -297,7 +297,8 @@ public sealed partial class RecipeRunner
{
// Not even an unexpected error leaves the viewer open; what was shown until then is saved all the same.
var stillOpen = await TryCloseViewerAsync(run, viewerPage).ConfigureAwait(false);
return new ViewerResult(shown, Worse(RecipeRun.Stop(RecipeStatus.Failed, $"Errore imprevisto in «{Name(run.Request.Kind)}» ({ex.GetType().Name}): {ex.Message}"), stillOpen));
var failure = new RecipeStopException(RecipeStatus.Failed, $"Errore imprevisto in «{Name(run.Request.Kind)}» ({ex.GetType().Name}): {ex.Message}") { ErrorType = ex.GetType().Name };
return new ViewerResult(shown, Worse(failure, stillOpen));
}
}
catch (OperationCanceledException cancelled)
@@ -1,4 +1,5 @@
using Febe.Core.Abstractions;
using Febe.Core.Collection.Diagnostics;
using Febe.Core.Collection.Readers;
using Febe.Core.Domain;
using Febe.Core.Engine;
@@ -40,6 +41,9 @@ public sealed record RecipeEnvironment
public ICircuitBreaker? Breaker { get; init; }
public RecipeOptions Options { get; init; } = new();
/// <summary>The recorder of the diagnostic report (issue #39). Null: nothing is recorded.</summary>
public CollectionDiagnostics? Diagnostics { get; init; }
}
/// <summary>
@@ -63,7 +67,7 @@ public sealed partial class RecipeRunner : IRecipeRunner
public RecipeRunner(RecipeEnvironment environment)
{
Env = environment ?? throw new ArgumentNullException(nameof(environment));
Writer = new CollectionWriter(environment.Storage, environment.Clock);
Writer = new CollectionWriter(environment.Storage, environment.Clock) { Diagnostics = environment.Diagnostics };
}
public RecipeEnvironment Env { get; }
@@ -134,11 +138,13 @@ public sealed partial class RecipeRunner : IRecipeRunner
await Writer.EventAsync(refusal.Code, refusal.Message + SessionComposer.NotConfiguredTail, EventSeverity.Warning, null, session.SessionId, ct).ConfigureAwait(false);
}
Env.Diagnostics?.RecipeEnded(session.SessionId, request.Kind, RecipeStatus.NotConfigured, null, 0, 0);
return new RecipeOutcome(RecipeStatus.NotConfigured, 0, 0, refusal.Message);
}
if (Env.Breaker is { } breaker && !breaker.State.AllowsSiteActions)
{
Env.Diagnostics?.RecipeEnded(session.SessionId, request.Kind, RecipeStatus.Denied, breaker.State.LastSignal?.Code, 0, 0);
return new RecipeOutcome(RecipeStatus.Denied, 0, 0, "L'interruttore è aperto: nessuna azione verso il sito.");
}
@@ -152,6 +158,7 @@ public sealed partial class RecipeRunner : IRecipeRunner
using var run = new RecipeRun(this, request, session, ct);
RecipeStatus status;
string? text = null;
string? errorType = null;
try
{
await CheckSessionCookieAsync(run).ConfigureAwait(false);
@@ -173,6 +180,7 @@ public sealed partial class RecipeRunner : IRecipeRunner
{
status = stop.Status;
text = stop.Message;
errorType = stop.ErrorType;
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
@@ -189,6 +197,7 @@ public sealed partial class RecipeRunner : IRecipeRunner
{
status = RecipeStatus.Failed;
text = $"Errore imprevisto in «{name}» ({ex.GetType().Name}): {ex.Message}";
errorType = ex.GetType().Name;
}
if (run.Signal is { } signal && status is not (RecipeStatus.StoppedBySignal or RecipeStatus.Cancelled))
@@ -206,6 +215,17 @@ public sealed partial class RecipeRunner : IRecipeRunner
// The timeline is a convenience: failing to write it never changes how the recipe ended.
}
// The report takes the status and the code of the signal, never the message (it may name an account). A recipe
// cancelled because the breaker stopped (a signal reported straight to it, like the session cookie lost) gets
// the breaker's code too.
var breakerState = Env.Breaker?.State;
var signalCode = run.Signal?.Code ?? status switch
{
RecipeStatus.StoppedBySignal => breakerState?.LastSignal?.Code,
RecipeStatus.Cancelled when breakerState is { AllowsSiteActions: false } => breakerState.LastSignal?.Code,
_ => null,
};
Env.Diagnostics?.RecipeEnded(session.SessionId, request.Kind, status, signalCode, run.Actions, run.NewEntries, status == RecipeStatus.Failed ? errorType : null);
_lastKind = request.Kind;
return new RecipeOutcome(status, run.Actions, run.NewEntries, text)
{
+170 -20
View File
@@ -1,6 +1,7 @@
using System.Text;
using System.Text.Json;
using System.Text.RegularExpressions;
using Febe.Core.Collection.Privacy;
namespace Febe.Core.Collection;
@@ -85,6 +86,11 @@ public sealed partial class SiteMap
private readonly Dictionary<string, string> _pages;
private readonly HashSet<string> _configuredTargets;
private readonly HashSet<string> _knownTargets;
private readonly IReadOnlyList<(string Name, Regex Shape)> _classifier;
private readonly HashSet<string> _literals;
/// <summary>The fixed words and the placeholder names of the shipped maps (the only ones a report may show).</summary>
private static readonly Lazy<(HashSet<string> Words, HashSet<string> Placeholders)> Shipped = new(LoadShippedWords);
public SiteMap(string site, IReadOnlyDictionary<string, string?> pages, IEnumerable<string> configuredTargets, IEnumerable<string> knownTargets, string? source = null)
{
@@ -93,6 +99,20 @@ public sealed partial class SiteMap
_pages = pages.Where(p => !string.IsNullOrWhiteSpace(p.Value)).ToDictionary(p => p.Key, p => p.Value!.Trim(), StringComparer.Ordinal);
_configuredTargets = new HashSet<string>(configuredTargets, StringComparer.Ordinal);
_knownTargets = new HashSet<string>(knownTargets, StringComparer.Ordinal);
// More literal segments first (the highlight viewer before the story viewer, whose {username} would take
// «highlights»), shapes without a query before the same shape with one, then the order of SitePages.All: a page
// made only of placeholders (the profile, «/{username}/») comes after every page with a word of its own.
_classifier = [.. _pages
.Select(p => (Name: p.Key, Pattern: p.Value, Shape: p.Value.Split('?')[0]))
.OrderByDescending(p => LiteralSegments(p.Shape))
.ThenBy(p => p.Pattern.Contains('?', StringComparison.Ordinal) ? 1 : 0)
.ThenBy(p => IndexOfPage(p.Name))
.ThenBy(p => p.Name, StringComparer.Ordinal)
.Select(p => (p.Name, new Regex(ShapeRegex(p.Shape, out _), RegexOptions.CultureInvariant | RegexOptions.IgnoreCase, TimeSpan.FromMilliseconds(100))))];
_literals = new HashSet<string>(
_pages.Values.SelectMany(v => v.Split('?')[0].Split('/', StringSplitOptions.RemoveEmptyEntries)).Where(segment => !Placeholder().IsMatch(segment)),
StringComparer.OrdinalIgnoreCase);
}
public string Site { get; }
@@ -144,26 +164,8 @@ public sealed partial class SiteMap
return null;
}
var shape = pattern.Split('?')[0];
var names = new List<string>();
var regex = new StringBuilder("^");
var last = 0;
foreach (Match m in Placeholder().Matches(shape))
{
regex.Append(Regex.Escape(shape[last..m.Index]));
regex.Append("([^/]+)");
names.Add(m.Groups[1].Value);
last = m.Index + m.Length;
}
regex.Append(Regex.Escape(shape[last..]));
if (!shape.EndsWith('/'))
{
regex.Append("/?");
}
regex.Append('$');
var match = Regex.Match(path, regex.ToString(), RegexOptions.CultureInvariant | RegexOptions.IgnoreCase, TimeSpan.FromMilliseconds(100));
var regex = ShapeRegex(pattern.Split('?')[0], out var names);
var match = Regex.Match(path, regex, RegexOptions.CultureInvariant | RegexOptions.IgnoreCase, TimeSpan.FromMilliseconds(100));
if (!match.Success)
{
return null;
@@ -178,6 +180,75 @@ public sealed partial class SiteMap
return result;
}
/// <summary>
/// Which page of the map an address path has the shape of, or null when none (data endpoints, pages the map does not
/// know). Pages with more words of their own are tried first, so «/stories/highlights/1/» is the highlight viewer
/// and not the story viewer of someone called «highlights»; the profile, made only of a placeholder, comes last.
/// A placeholder never takes a fixed word of the site (<see cref="PathPatterns.IsKnownSegment"/>) or of a map:
/// «/challenge/», «/explore/», «/reels/» are not profiles. Only the page NAME comes out: never the values (issue #39).
/// </summary>
public string? Classify(string? path)
{
if (string.IsNullOrEmpty(path))
{
return null;
}
foreach (var (name, shape) in _classifier)
{
try
{
var match = shape.Match(path);
if (match.Success && !TakesAWord(match))
{
return name;
}
}
catch (RegexMatchTimeoutException)
{
return null;
}
}
return null;
}
/// <summary>
/// The shape of the address of page <paramref name="name"/> for a report: without its query, with its
/// {placeholders} and no value (e.g. «/{username}/»); null when the page is not set. A map may come from Alberto's
/// data folder, so the shape is reduced like any path: only the fixed words of the site and of the shipped maps
/// and their placeholders stay, any other segment becomes {numero} or {testo} («/alberto.rossi/following/» written
/// in place of «/{username}/following/» gives «/{testo}/following/»).
/// </summary>
public string? PatternOf(string name) =>
_pages.TryGetValue(name, out var pattern) ? ReportShape(pattern.Split('?')[0]) : null;
/// <summary>A page shape reduced for a report (<see cref="PatternOf"/>).</summary>
public static string ReportShape(string shape)
{
ArgumentNullException.ThrowIfNull(shape);
var (words, placeholders) = Shipped.Value;
var segments = shape.Split('/');
for (var i = 0; i < segments.Length; i++)
{
var segment = segments[i];
if (segment.Length == 0 || PathPatterns.IsKnownSegment(segment) || words.Contains(segment))
{
continue;
}
var whole = Placeholder().Match(segment);
if (whole.Success && whole.Length == segment.Length && placeholders.Contains(whole.Groups[1].Value))
{
continue;
}
segments[i] = PathPatterns.Of("/" + segment)[1..];
}
return string.Join('/', segments);
}
/// <summary>The target map of the local test site (embedded <c>bersagli.prova.json</c>).</summary>
public static SiteMap LoadTestSite() => LoadEmbedded("bersagli.prova.json");
@@ -261,6 +332,85 @@ public sealed partial class SiteMap
}
}
private static string ShapeRegex(string shape, out List<string> names)
{
names = [];
var regex = new StringBuilder("^");
var last = 0;
foreach (Match m in Placeholder().Matches(shape))
{
regex.Append(Regex.Escape(shape[last..m.Index]));
regex.Append("([^/]+)");
names.Add(m.Groups[1].Value);
last = m.Index + m.Length;
}
regex.Append(Regex.Escape(shape[last..]));
if (!shape.EndsWith('/'))
{
regex.Append("/?");
}
regex.Append('$');
return regex.ToString();
}
/// <summary>True when a placeholder of the match took a fixed word of the site or of a map (not a value, so not this page).</summary>
private bool TakesAWord(Match match)
{
var shippedWords = Shipped.Value.Words;
for (var i = 1; i < match.Groups.Count; i++)
{
var value = match.Groups[i].Value;
if (PathPatterns.IsKnownSegment(value) || PathPatterns.IsKnownSegment(value.ToLowerInvariant())
|| _literals.Contains(value) || shippedWords.Contains(value))
{
return true;
}
}
return false;
}
private static (HashSet<string> Words, HashSet<string> Placeholders) LoadShippedWords()
{
var words = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
var placeholders = new HashSet<string>(StringComparer.Ordinal);
foreach (var file in new[] { "bersagli.instagram.json", "bersagli.prova.json" })
{
foreach (var pattern in LoadEmbedded(file)._pages.Values)
{
// The names of the values the recipes fill in, in the path or in the query («{cursor}»).
placeholders.UnionWith(Placeholder().Matches(pattern).Select(m => m.Groups[1].Value));
foreach (var segment in pattern.Split('?')[0].Split('/', StringSplitOptions.RemoveEmptyEntries))
{
if (!Placeholder().IsMatch(segment))
{
words.Add(segment);
}
}
}
}
return (words, placeholders);
}
private static int LiteralSegments(string shape) =>
shape.Split('/', StringSplitOptions.RemoveEmptyEntries).Count(segment => !Placeholder().IsMatch(segment));
private static int IndexOfPage(string name)
{
for (var i = 0; i < SitePages.All.Count; i++)
{
if (SitePages.All[i] == name)
{
return i;
}
}
return int.MaxValue;
}
private static bool IsConfigured(JsonElement target)
{
foreach (var key in new[] { "role", "name", "href" })
+40 -2
View File
@@ -627,12 +627,15 @@ public sealed class EngineHost : IAsyncDisposable
}
s.Runtime.CurrentSession = context;
s.Diagnostics?.SessionStarted(record.Id, start, mood);
s.Runtime.HadSessionSinceStart = true;
Volatile.Write(ref _manualWaiting, false);
s.Runtime.NextSessionAt = null;
s.Events.Publish(EngineNotificationKind.StateChanged);
var outcome = SessionOutcome.Completed;
var cause = SessionStopCause.WorkDone;
SessionSummary? summary = null;
var interrupted = false;
try
{
@@ -642,25 +645,37 @@ public sealed class EngineHost : IAsyncDisposable
sessionCts.Cancel();
}
var summary = await _composer.RunAsync(context, outsideWindowTask, sessionCts.Token).ConfigureAwait(false);
summary = await _composer.RunAsync(context, outsideWindowTask, sessionCts.Token).ConfigureAwait(false);
if (summary.StoppedEarly || !s.Breaker.State.AllowsSiteActions || s.Runtime.IsPaused || s.Runtime.UserIsDriving)
{
outcome = StopOutcome();
cause = ct.IsCancellationRequested ? SessionStopCause.Shutdown : StopCause(summary);
}
else
{
// A shutdown between two recipes leaves the composer as if its work were done: say what it was.
cause = ct.IsCancellationRequested ? SessionStopCause.Shutdown
: s.Clock.Now >= plannedEnd ? SessionStopCause.PlannedEnd
: summary.BudgetsExhausted ? SessionStopCause.BudgetsExhausted
: SessionStopCause.WorkDone;
}
}
catch (OperationCanceledException) when (!ct.IsCancellationRequested)
{
outcome = StopOutcome();
cause = StopCause(null);
}
catch (OperationCanceledException)
{
outcome = SessionOutcome.Interrupted;
cause = SessionStopCause.Shutdown;
interrupted = true;
throw;
}
catch (Exception ex) when (ex is not OutOfMemoryException)
{
outcome = SessionOutcome.Failed;
cause = SessionStopCause.Failed;
await s.Events.WriteAsync(
EventSeverity.Error,
EngineEventCodes.RecipeFailed,
@@ -677,10 +692,29 @@ public sealed class EngineHost : IAsyncDisposable
s.Runtime.CurrentSession = null;
s.Runtime.Activity = null;
await EndSessionAsync(record, context, outcome, outsideWindowTask, interrupted).ConfigureAwait(false);
try
{
await EndSessionAsync(record, context, outcome, outsideWindowTask, interrupted).ConfigureAwait(false);
}
finally
{
s.Diagnostics?.SessionEnded(record.Id, s.Clock.Now, outcome, cause, summary?.StoppedEarlyBy);
}
}
}
/// <summary>
/// Why a session that did not reach its natural end stopped (diagnostic report): an open breaker first, then the
/// pause, then Alberto at the browser, then a recipe outcome; anything else is a cancellation. A shutdown of Febe
/// (the loop's token cancelled) is checked by the caller first: a real recipe returns «Cancelled» instead of throwing.
/// </summary>
private SessionStopCause StopCause(SessionSummary? summary) =>
!_s.Breaker.State.AllowsSiteActions ? SessionStopCause.BreakerOpen
: _s.Runtime.IsPaused ? SessionStopCause.Paused
: _s.Runtime.UserIsDriving ? SessionStopCause.UserDriving
: summary?.StoppedEarly == true ? SessionStopCause.StoppedEarly
: SessionStopCause.Cancelled;
private async Task EndSessionAsync(SessionRecord record, SessionContext context, SessionOutcome outcome, EngineTask? outsideWindowTask, bool interrupted)
{
var s = _s;
@@ -844,6 +878,10 @@ public sealed class EngineHost : IAsyncDisposable
private void OnBreakerChanged(object? sender, BreakerState state)
{
// The diagnostic report records the signal that stopped the breaker, also one the detector never saw (the
// session cookie lost, reported by the connection): a logout must not look like any other stop.
_s.Diagnostics?.BreakerChanged(state);
// Signals arrive on capture threads: the engine is never run inline on them (it may need the browser to answer).
// The composer also checks the breaker before every recipe, so the session stops at once either way.
CancellationTokenSource? session = null;
+7
View File
@@ -1,5 +1,6 @@
using System.Globalization;
using Febe.Core.Abstractions;
using Febe.Core.Collection.Diagnostics;
using Febe.Core.Domain;
using Febe.Core.Storage;
@@ -64,6 +65,12 @@ public sealed class EngineServices
/// <summary>Current engine and caution settings.</summary>
public EngineSettings Settings { get; private set; }
/// <summary>
/// The recorder of the diagnostic report (issue #39): told when sessions start and end, and why. Null: nothing is
/// recorded. Set by the composition before the engine starts.
/// </summary>
public CollectionDiagnostics? Diagnostics { get; set; }
/// <summary>
/// Loads settings, pause flag, gradual-start origin and breaker state from the database. When the gradual start has
/// no origin yet (first run after the first connection) it starts now.
+19 -2
View File
@@ -985,6 +985,7 @@ internal sealed class SessionComposer
else if (recheck.Reason is DenialReason.BreakerOpen or DenialReason.OutsideActiveWindow or DenialReason.Paused or DenialReason.UserIsDriving)
{
state.Stop = true;
state.StopStatus ??= outcome.Status;
}
break;
@@ -1002,6 +1003,7 @@ internal sealed class SessionComposer
}
state.Stop = true;
state.StopStatus ??= outcome.Status;
break;
case RecipeStatus.TargetNotFound:
@@ -1013,6 +1015,7 @@ internal sealed class SessionComposer
await RecordRecipeFailureAsync(kind, account, EngineEventCodes.TargetNotFound,
$"Non trovo nella pagina ciò che serve per {RecipeName(kind)}{Of(account)}: mi fermo e lo segnalo, senza cliccare a caso.", now).ConfigureAwait(false);
state.Stop = true;
state.StopStatus ??= outcome.Status;
break;
case RecipeStatus.UnrecognizedFormat:
@@ -1380,6 +1383,9 @@ internal sealed class SessionComposer
public bool Stop { get; set; }
/// <summary>The recipe status that first set <see cref="Stop"/>, when a recipe did (diagnostic report).</summary>
public RecipeStatus? StopStatus { get; set; }
public int Recipes { get; set; }
/// <summary>Next story check planned by the last visit of a profile not followed.</summary>
@@ -1388,9 +1394,20 @@ internal sealed class SessionComposer
/// <summary>Set while the stories of a visit run: their next check is the visit's.</summary>
public DateTimeOffset? PendingStoriesNext { get; set; }
public SessionSummary Summary() => new(Recipes, Stop);
public SessionSummary Summary() => new(Recipes, Stop)
{
StoppedEarlyBy = StopStatus,
BudgetsExhausted = Exhausted.Count > 0,
};
}
}
/// <summary>What a session did, for its row and its timeline line.</summary>
internal sealed record SessionSummary(int Recipes, bool StoppedEarly);
internal sealed record SessionSummary(int Recipes, bool StoppedEarly)
{
/// <summary>The recipe status that stopped the session early, when a recipe did (diagnostic report, issue #39).</summary>
public RecipeStatus? StoppedEarlyBy { get; init; }
/// <summary>Some budget categories were used up during the session and their work skipped.</summary>
public bool BudgetsExhausted { get; init; }
}
+39
View File
@@ -0,0 +1,39 @@
namespace Febe.Core.Engine;
/// <summary>
/// Why an engine session ended (issue #39, diagnostic report). Finer than <see cref="Domain.SessionOutcome"/>, which
/// puts most of these under «Stopped»; computed by <see cref="EngineHost"/> from what it already looks at, and never
/// stored in the database.
/// </summary>
public enum SessionStopCause
{
/// <summary>The planned length of the session was reached.</summary>
PlannedEnd,
/// <summary>Nothing else to do before the planned end.</summary>
WorkDone,
/// <summary>Ended before the planned end with some budget categories used up (their work was skipped).</summary>
BudgetsExhausted,
/// <summary>A recipe outcome stopped the session (see the recipe status that did).</summary>
StoppedEarly,
/// <summary>The circuit breaker was open: a stop signal of spec §7.6.</summary>
BreakerOpen,
/// <summary>Alberto paused the engine.</summary>
Paused,
/// <summary>Alberto took the browser.</summary>
UserDriving,
/// <summary>Cancelled for another reason.</summary>
Cancelled,
/// <summary>Febe was shutting down.</summary>
Shutdown,
/// <summary>An unexpected error closed the session.</summary>
Failed,
}
+3
View File
@@ -128,6 +128,9 @@ public sealed record DataFolderLayout
/// <summary>Dati/registro: rotating logs.</summary>
public string Logs => Path.Combine(Root, "registro");
/// <summary>Dati/diagnostica: the traces of the diagnostic report, two weeks (issue #39).</summary>
public string Diagnostics => Path.Combine(Root, "diagnostica");
/// <summary>Dati/segnali.json.</summary>
public string SignalsFile => Path.Combine(Root, "segnali.json");
}
+39 -1
View File
@@ -2,9 +2,11 @@ using System.Globalization;
using System.Text.Json;
using Febe.Core.Abstractions;
using Febe.Core.Alerts;
using Febe.Core.Collection.Diagnostics;
using Febe.Core.Domain;
using Febe.Core.Engine;
using Febe.Core.Storage;
using Febe.Web.Demo;
using Febe.Web.Engine;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Http;
@@ -14,13 +16,20 @@ namespace Febe.Web.Api;
/// <summary>
/// What the Attività page needs beside the engine snapshot (spec §7.6, §7.9, mockups 06 and 11): the history of the
/// stops, the still frame of the browser, the probable duplicates waiting for Alberto.
/// stops, the still frame of the browser, the probable duplicates waiting for Alberto, the diagnostic report.
/// </summary>
public static class ActivityApi
{
/// <summary>Days of stops shown by the page.</summary>
public const int HistoryDays = 14;
/// <summary>
/// «Esporta il rapporto di diagnostica» (issue #39, spec §7.9): the traces of the engine sessions of the last two
/// weeks as a JSON file to send, with no data value in it. Also answered with a configuration problem, where the
/// engine never started and the report is empty.
/// </summary>
public const string DiagnosticsPath = "/api/engine/diagnostica";
public static void Map(IEndpointRouteBuilder endpoints)
{
endpoints.MapGet("/api/engine/incidents", async (IIncidentHistory history, IClock clock, CancellationToken ct) =>
@@ -56,8 +65,37 @@ public static class ActivityApi
var import = await storage.Duplicates.CountPendingAsync(DuplicateOrigin.Import, ct).ConfigureAwait(false);
return ApiHttp.Json(new DuplicateCountsDto(download, import), FebeJsonContext.Default.DuplicateCountsDto);
});
endpoints.MapGet(DiagnosticsPath, async (HttpContext context, IDiagnosticSource source, FebeWebOptions options, IClock clock, CancellationToken ct) =>
{
var gathered = await source.GatherAsync(ct).ConfigureAwait(false);
var now = clock.Now;
var zone = clock.LocalZone;
var edition = options.Demo ? DemoRequest.Edition(context) ?? FebeEdition.Desktop : options.Edition;
// Written whole before answering: a failure becomes an error message, never half a file.
using var buffer = new MemoryStream();
DiagnosticReport.Write(
buffer,
gathered.Traces,
new DiagnosticReportHeader(
options.Version,
EditionName(edition),
now,
zone,
DamagedLines: gathered.DamagedLines,
RecorderFaults: gathered.RecorderFaults,
UnreadableFiles: gathered.UnreadableFiles,
UnfinishedSessions: gathered.UnfinishedSessions));
context.Response.Headers.ContentDisposition = $"attachment; filename=\"{DiagnosticReport.FileName(now, zone)}\"";
context.Response.Headers.CacheControl = "no-store";
return Results.Bytes(buffer.ToArray(), "application/json; charset=utf-8");
});
}
/// <summary>The «edizione» of the diagnostic report.</summary>
public static string EditionName(FebeEdition edition) => edition == FebeEdition.Desktop ? "windows" : "docker";
public static async Task<IncidentListDto> IncidentsAsync(IIncidentHistory history, IClock clock, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(history);
+6 -2
View File
@@ -44,9 +44,13 @@ public static class HealthEndpoint
/// </summary>
public sealed class ConfigurationGateMiddleware(RequestDelegate next, IConfigurationProblems problems)
{
/// <summary>API addresses still answered while there is a problem.</summary>
/// <summary>
/// API addresses still answered while there is a problem. The diagnostic report is among them, so that its address
/// never answers with an error: it holds no data value, and with a configuration problem it is empty (the engine
/// never started, nothing was recorded).
/// </summary>
public static readonly IReadOnlyList<string> AllowedApi =
["/api/capabilities", "/api/preferences", "/api/accesso", "/api/uscita", "/api/engine", "/api/eventi"];
["/api/capabilities", "/api/preferences", "/api/accesso", "/api/uscita", "/api/engine", "/api/eventi", ActivityApi.DiagnosticsPath];
public async Task InvokeAsync(HttpContext context)
{
+234
View File
@@ -0,0 +1,234 @@
using Febe.Core.Abstractions;
using Febe.Core.Collection;
using Febe.Core.Collection.Diagnostics;
using Febe.Core.Collection.Downloads;
using Febe.Core.Collection.Privacy;
using Febe.Core.Collection.Recipes;
using Febe.Core.Domain;
using Febe.Core.Engine;
using Febe.Web.Engine;
using Microsoft.AspNetCore.Http;
namespace Febe.Web.Demo;
/// <summary>
/// The diagnostic report of the demo (issue #39): today's sessions of <see cref="DemoWorld"/> told as traces, with
/// counts that follow the session numbers. It follows the scene: «sessione» adds the session in progress, «fermo» ends
/// the last session with the verification (signal <c>challenge-url</c>, breaker open), «scollegato» and «errore» have
/// nothing (the engine never ran). Like the real report, it holds no name: patterns come from the site map and from
/// <see cref="PathPatterns"/>, every other value from a closed list.
/// </summary>
public sealed class DemoDiagnosticSource(IHttpContextAccessor accessor, DemoWorld world, IClock clock) : IDiagnosticSource
{
/// <summary>The verification of the scene «fermo» (spec §7.6, <c>segnali.json</c>).</summary>
public const string ChallengeCode = "challenge-url";
private const string Json = "application/json";
private const string Html = "text/html";
private static readonly SiteMap Map = SiteMap.LoadInstagram();
public Task<IReadOnlyList<SessionTrace>> ListAsync(CancellationToken ct) =>
Task.FromResult(Build(DemoRequest.Scene(accessor.HttpContext), clock.Now));
/// <summary>The traces of a scene at <paramref name="now"/>, oldest first.</summary>
public IReadOnlyList<SessionTrace> Build(DemoScene scene, DateTimeOffset now)
{
if (scene is DemoScene.Disconnected or DemoScene.Error)
{
return [];
}
var sessions = world.Today.Where(s => s.End <= now).ToList();
var traces = new List<SessionTrace>();
for (var i = 0; i < sessions.Count; i++)
{
var stopped = scene == DemoScene.Stopped && i == sessions.Count - 1;
traces.Add(Trace(i + 1, sessions[i], stopped ? Ending.Stopped : Ending.Finished, i));
}
if (scene == DemoScene.Session)
{
traces.Add(Trace(sessions.Count + 1, DemoEngine.CurrentSession(now), Ending.InProgress, sessions.Count));
}
if (sessions.Count > 0 && scene != DemoScene.Stopped)
{
traces.Add(Outside(sessions[0].End + TimeSpan.FromMinutes(3), sessions));
}
return [.. traces.OrderBy(t => t.Start)];
}
private enum Ending
{
Finished,
Stopped,
InProgress,
}
private static SessionTrace Trace(long id, DemoSession s, Ending ending, int index)
{
var profiles = Math.Max(1, s.ProfilesOpened);
var storyAccounts = s.StoriesViewed == 0 ? 0 : Math.Clamp(s.StoriesViewed / 3, 1, profiles + 2);
var stories = Math.Min(s.NewEntries, s.StoriesViewed);
var posts = s.NewEntries - stories;
var storyBytes = s.NewEntries == 0 ? 0 : s.Bytes * stories / s.NewEntries;
var stopped = ending == Ending.Stopped;
// Pages, spread over the session: the home, the profiles, the stories viewers.
var names = new List<string> { SitePages.Home };
names.AddRange(Enumerable.Repeat(SitePages.Profile, profiles));
names.AddRange(Enumerable.Repeat(SitePages.StoryViewer, storyAccounts));
var step = (s.End - s.Start) / (names.Count + 1);
var pages = names.Select((name, i) => new PageVisit(s.Start + (step * (i + 1)), name, Map.PatternOf(name), true)).ToList();
var navigations = new List<NavigationGroup>
{
new(HostKind.Site, SitePages.Home, Pattern(SitePages.Home), false, 1),
new(HostKind.Site, SitePages.Profile, Pattern(SitePages.Profile), false, profiles),
};
if (storyAccounts > 0)
{
navigations.Add(new(HostKind.Site, SitePages.StoryViewer, Pattern(SitePages.StoryViewer), true, s.StoriesViewed));
}
var dataCalls = (profiles * 3) + s.HistoryPages + 4;
var images = s.StoriesViewed + (profiles * 12);
var videos = s.StoriesViewed / 2;
var responses = new List<ResponseGroup>
{
Group(HostKind.Site, SitePages.Home, Pattern(SitePages.Home), "Document", Html, 1, 412_000) with { Recognized = 1, Contents = 6 },
Group(HostKind.Site, SitePages.Profile, Pattern(SitePages.Profile), "Document", Html, profiles, 388_000) with { Recognized = profiles, Profiles = profiles },
Group(HostKind.Site, null, PathPatterns.Of("/graphql/query"), "Fetch", Json, dataCalls, 24_000) with
{
Recognized = profiles + s.HistoryPages,
Empty = profiles / 4,
Unrelated = dataCalls - profiles - s.HistoryPages - (profiles / 4),
Contents = posts,
Profiles = profiles,
},
Group(HostKind.Site, null, PathPatterns.Of("/ajax/bulk-route-definitions/"), "Fetch", Json, profiles + 2, 3_000) with { Unrelated = profiles + 2 },
Group(HostKind.Cdn, null, "cdn *.jpg", "Image", "image/jpeg", images, 140_000) with { NotRead = images },
Group(HostKind.Other, null, CollectionDiagnostics.Other + " *.js", "Script", "application/javascript", 14, 220_000) with { NotRead = 14 },
};
if (storyAccounts > 0)
{
responses.Add(Group(HostKind.Site, null, PathPatterns.Of("/api/v1/feed/reels_media/"), "XHR", Json, storyAccounts, 36_000) with
{
Recognized = storyAccounts,
Contents = s.StoriesViewed,
});
}
if (videos > 0)
{
responses.Add(Group(HostKind.Cdn, null, "cdn *.mp4", "Media", "video/mp4", videos, 2_400_000) with { NotRead = videos });
}
var recipes = new List<RecipeLine> { new(s.Start + step, RecipeKind.FollowingSweep, RecipeStatus.Completed, null, 2 + storyAccounts, Math.Min(2, posts)) };
recipes.AddRange(Enumerable.Range(0, storyAccounts).Select(i => new RecipeLine(
s.Start + (step * (2 + profiles + i)), RecipeKind.AccountStories, RecipeStatus.Completed, null, 3, i == 0 ? stories : 0)));
recipes.AddRange(Enumerable.Range(0, profiles).Select(i => new RecipeLine(
s.Start + (step * (2 + i)), RecipeKind.ProfileCheck, RecipeStatus.Completed, null, 1, i == 0 ? Math.Max(0, posts - 2) : 0)));
var signals = new List<SignalLine>();
if (stopped)
{
// The last profile check met the verification page: stopped there, nothing after it.
var at = s.End - TimeSpan.FromSeconds(40);
var last = recipes.FindLastIndex(r => r.Kind == RecipeKind.ProfileCheck);
recipes[last] = recipes[last] with { At = at, Status = RecipeStatus.StoppedBySignal, Signal = ChallengeCode, NewEntries = 0 };
signals.Add(new SignalLine(at, SignalLevel.Challenge, ChallengeCode));
pages.Add(new PageVisit(at, SitePages.Profile, Map.PatternOf(SitePages.Profile), false));
navigations.Add(new(HostKind.Site, null, PathPatterns.Of("/challenge/"), false, 1));
responses.Add(Group(HostKind.Site, null, PathPatterns.Of("/challenge/"), "Document", Html, 1, 96_000) with { Unrelated = 1 });
}
var steps = new List<VerdictCount>
{
new(CollectionDiagnostics.VerdictRecognized, 1 + profiles + storyAccounts + s.HistoryPages),
};
if (profiles >= 4)
{
steps.Add(new(CollectionDiagnostics.VerdictEmpty, profiles / 4));
}
var targets = new List<TargetCount>();
if (storyAccounts > 0)
{
targets.Add(new(SiteTargets.HomeStory, TargetCheckStatus.Clicked, storyAccounts));
targets.Add(new(SiteTargets.ViewerNext, TargetCheckStatus.Clicked, Math.Max(0, s.StoriesViewed - storyAccounts)));
targets.Add(new(SiteTargets.ViewerClose, TargetCheckStatus.Clicked, storyAccounts));
}
var queued = new List<QueuedFiles>();
var downloads = new List<DownloadGroup>();
if (stories > 0)
{
queued.Add(new(EntryKind.Story, nameof(SaveOutcome.Created), stories, stories, 0));
downloads.Add(new(EntryKind.Story, DownloadStep.Stored, stories, storyBytes));
}
if (posts > 0)
{
queued.Add(new(EntryKind.Post, nameof(SaveOutcome.Created), posts, posts + (posts / 3), 0));
downloads.Add(new(EntryKind.Post, DownloadStep.Stored, posts + (posts / 3), s.Bytes - storyBytes));
}
queued.Add(new(EntryKind.Post, nameof(SaveOutcome.Known), profiles * 2, 0, 0));
return new SessionTrace
{
Session = id,
Start = s.Start,
End = ending == Ending.InProgress ? null : s.End,
Outcome = ending switch
{
Ending.Stopped => SessionOutcome.Stopped,
Ending.InProgress => SessionOutcome.Running,
_ => SessionOutcome.Completed,
},
StopCause = ending switch
{
Ending.Stopped => SessionStopCause.BreakerOpen,
Ending.InProgress => null,
_ => index % 3 == 2 ? SessionStopCause.WorkDone : SessionStopCause.PlannedEnd,
},
Mood = Math.Round(0.85 + (0.05 * (index % 4)), 2),
InProgress = ending == Ending.InProgress,
Pages = pages,
Navigations = navigations,
Responses = responses,
Steps = steps,
Targets = targets,
Recipes = recipes,
Signals = signals,
FilesQueued = queued,
Downloads = downloads,
DownloadedBytes = s.Bytes,
};
}
/// <summary>Today outside the sessions: files of the queue settled between two sessions.</summary>
private static SessionTrace Outside(DateTimeOffset start, IReadOnlyList<DemoSession> sessions)
{
var files = Math.Clamp(sessions.Sum(s => s.NewEntries) / 6, 1, 12);
return new SessionTrace
{
Session = null,
Start = start,
InProgress = true,
Downloads =
[
new(EntryKind.Post, DownloadStep.Stored, files, files * 1_900_000L),
new(EntryKind.Story, DownloadStep.Duplicate, 1, 0),
],
DownloadedBytes = files * 1_900_000L,
};
}
private static string Pattern(string page) => Map.PatternOf(page) ?? PathPatterns.Text;
private static ResponseGroup Group(HostKind host, string? page, string pattern, string resourceType, string mime, int count, long average) =>
new(host, page, pattern, resourceType, mime, "2xx", count, count * average, average + (average / 10), 0, 0, 0, 0, 0, 0, 0, 0, []);
}
+5 -3
View File
@@ -30,6 +30,10 @@ public sealed class DemoEngine(IClock clock, DemoWorld world, IFebeStorage stora
return sessions.Count > 0 ? sessions[^1].End - TimeSpan.FromSeconds(40) : now - TimeSpan.FromMinutes(25);
}
/// <summary>The session in progress of the scene «sessione»: started two and a half minutes ago.</summary>
public static DemoSession CurrentSession(DateTimeOffset now) =>
new(now - TimeSpan.FromMinutes(2.5), now, 2, 6, 0, 18_000_000, 2);
public async Task<EngineSnapshot> GetSnapshotAsync(CancellationToken ct = default)
{
var scene = CurrentScene;
@@ -40,9 +44,7 @@ public sealed class DemoEngine(IClock clock, DemoWorld world, IFebeStorage stora
var inWindow = now >= world.WindowStart && now < world.WindowEnd;
var sessions = scene is DemoScene.Disconnected or DemoScene.Error ? [] : world.Today.Where(s => s.End <= now).ToList();
var stopAt = StopInstant(now);
var current = scene == DemoScene.Session
? new DemoSession(now - TimeSpan.FromMinutes(2.5), now, 2, 6, 0, 18_000_000, 2)
: null;
var current = scene == DemoScene.Session ? CurrentSession(now) : null;
var accounts = await storage.Accounts.ListAsync(ct).ConfigureAwait(false);
var marta = accounts.FirstOrDefault(a => a.Username == "marta.demo")?.Username ?? world.MineUsername;
+1
View File
@@ -99,6 +99,7 @@ public sealed class DemoHost : IAsyncDisposable
services.AddSingleton<IConnectionService, DemoConnectionService>();
services.AddSingleton<IBrowserPreview, DemoBrowserPreview>();
services.AddSingleton<IIncidentHistory, DemoIncidentHistory>();
services.AddSingleton<IDiagnosticSource, DemoDiagnosticSource>();
services.AddSingleton<IDataFolderChange, DemoDataFolderChange>();
services.AddSingleton(demoImport);
});
+40
View File
@@ -0,0 +1,40 @@
using Febe.Core.Collection.Diagnostics;
namespace Febe.Web.Engine;
/// <summary>
/// The traces of the diagnostic report (issue #39) for «Esporta il rapporto di diagnostica» in Attività: the finished
/// ones of the last two weeks and the ones in progress, oldest first. They hold no data value
/// (<see cref="SessionTrace"/>).
/// </summary>
public interface IDiagnosticSource
{
Task<IReadOnlyList<SessionTrace>> ListAsync(CancellationToken ct);
/// <summary>
/// The traces with what the report must say about itself (damaged lines of the log, errors of the recorder). By
/// default the traces of <see cref="ListAsync"/> with nothing lost.
/// </summary>
async Task<DiagnosticCollection> GatherAsync(CancellationToken ct) =>
new(await ListAsync(ct).ConfigureAwait(false));
}
/// <summary>The traces of the recorder of the running Febe (its log in <c>Dati/diagnostica</c>, and what is open).</summary>
public sealed class RecordedDiagnosticSource(CollectionDiagnostics diagnostics) : IDiagnosticSource
{
public Task<IReadOnlyList<SessionTrace>> ListAsync(CancellationToken ct) =>
Task.Run(diagnostics.Collect, ct);
public Task<DiagnosticCollection> GatherAsync(CancellationToken ct) =>
Task.Run(diagnostics.Gather, ct);
}
/// <summary>
/// The default when the composition registers no recorder (tests, a start with a configuration problem, where the
/// engine never started): nothing was recorded, so the report has empty lists.
/// </summary>
public sealed class NoDiagnosticSource : IDiagnosticSource
{
public Task<IReadOnlyList<SessionTrace>> ListAsync(CancellationToken ct) =>
Task.FromResult<IReadOnlyList<SessionTrace>>([]);
}
+1
View File
@@ -67,6 +67,7 @@ public static class FebeWeb
services.TryAddSingleton<IConfigurationProblems, NoConfigurationProblems>();
services.TryAddSingleton<IConnectionService, UnavailableConnectionService>();
services.TryAddSingleton<IBrowserPreview, NoBrowserPreview>();
services.TryAddSingleton<IDiagnosticSource, NoDiagnosticSource>();
services.TryAddSingleton<IIncidentHistory>(sp => new StoredIncidentHistory(sp.GetRequiredService<IFebeStorage>()));
services.TryAddSingleton<IAlertTestSender, NoTelegramSender>();
services.TryAddSingleton<IDataFolderChange, FixedDataFolder>();
+6 -4
View File
@@ -13,7 +13,7 @@ Qui sotto le convenzioni per chi continua il lavoro (composizione di Server e De
| `Hosting/` | `EmbeddedWwwroot`: `IFileProvider` sulle risorse incorporate `wwwroot/…` (o su una cartella, per sviluppo) |
| `Security/` | Controlli di §13.1: Host, Origin, gettone del desktop, password e blocco della veste Docker, intestazioni |
| `Api/` | Endpoint minimi, DTO (`Dtos.cs`), contesto JSON generato (`FebeJsonContext`), conversioni (`ApiMapping`) |
| `Engine/` | `IEngineControl` (comandi al motore), `EngineHostControl`, `OfflineEngine` (nessun motore), `BrowserUseTracker`, `IEngineSettingsControl` (limiti al motore), `IBrowserPreview` (anteprima), `IIncidentHistory` (incidenti) |
| `Engine/` | `IEngineControl` (comandi al motore), `EngineHostControl`, `OfflineEngine` (nessun motore), `BrowserUseTracker`, `IEngineSettingsControl` (limiti al motore), `IBrowserPreview` (anteprima), `IIncidentHistory` (incidenti), `IDiagnosticSource` (rapporto di diagnostica) |
| `Library/` | `FebeLibrary` (archivio + disposizione, `HashIndex`, `DownloadDuplicates`), `LibraryChanges`, esportazione CSV, collegamenti a Instagram, nomi utente, `IDataFolderChange`, `FolderSizes`, `RebuildJob` |
| `Connection/` | Contratto condiviso con il lato server del collegamento: `IConnectionService`, `IConfigurationProblems`, `PROTOCOLLO.md` (vista remota `/api/vista`); `UnavailableConnectionService` è il ripiego |
| `Import/` | `ImportHub`: l'`ImportService` registrato dalla composizione, oppure uno costruito sulla libreria |
@@ -68,6 +68,7 @@ Tutti con un ripiego registrato con `TryAdd`; la composizione registra i propri
| `IEngineSettingsControl` | `StoredEngineSettingsControl`: salva le impostazioni del motore nell'archivio | `new EngineHostSettingsControl(engineHost)`: le applica anche al motore che gira |
| `IBrowserPreview` | `NoBrowserPreview` (nessuna anteprima) | Chi ha il browser: `GetLatestAsync` restituisce l'ultima schermata JPEG (`BrowserFrame`) con l'indirizzo |
| `IIncidentHistory` | `StoredIncidentHistory` (dall'archivio) | Di solito nessuno |
| `IDiagnosticSource` (`Engine/DiagnosticSource.cs`) | `NoDiagnosticSource`: niente registrato, il rapporto ha gli elenchi vuoti (prove, avvio con un problema di configurazione) | `Febe.App`: `RecordedDiagnosticSource` sul registratore `CollectionDiagnostics` (registro in `Dati/diagnostica`, 14 giorni, e sessioni in corso); dimostrazione: `DemoDiagnosticSource`, secondo la scena |
| `IAlertTestSender` (Core) | `NoTelegramSender`: nessun messaggio parte (dimostrazione, prove), la risposta lo dice | La composizione (`Febe.App`, le due vesti): `TelegramAlertChannel` (Core, `Alerts/Telegram.cs`), lo stesso canale che usa `AlertService` |
| `IDataFolderChange` | `FixedDataFolder`: la cartella dei dati non si cambia dall'interfaccia | Desktop: `DesktopDataFolderChange` registra la scelta e `DataFolderMove` sposta la cartella al prossimo avvio, con copia verificata (SHA-256) |
| `ImportService` (Core) | `ImportHub` ne costruisce uno sulla libreria (Docker: solo sotto `FebeWebOptions.ImportRoot`, di serie `/importa`; desktop: qualsiasi cartella) | La composizione, se vuole opzioni diverse (ffmpeg, cartella sorgente) |
@@ -119,6 +120,7 @@ sono anche `day` (`aaaa-mm-gg`) e `clock` calcolati sul fuso di Febe.
| `GET /api/engine/incidents` | Incidenti degli ultimi 14 giorni (rallentamenti, verifiche, arresti), con un consiglio dopo 3 rallentamenti in 7 giorni |
| `POST /api/engine/recipes/{ricetta}/resume` | «Riattiva» una ricetta sospesa dopo tre problemi di fila (`IEngineControl.ResumeRecipeAsync`: metodo con un'implementazione predefinita che risponde «non era sospesa», così le altre implementazioni non cambiano). Attività mostra le ricette sospese da `snapshot.suspendedRecipes` |
| `GET /api/alerts` | Avvisi degli ultimi 14 giorni (§13.3, sezione «Avvisi» di Attività): righe della cronologia con i codici di `AlertCatalog.Codes`, compresi gli avvisi non arrivati |
| `GET /api/engine/diagnostica` | «Esporta il rapporto di diagnostica» (issue #39, §7.9): file JSON `febe-diagnostica-aaaa-mm-gg.json` (giorno locale) in allegato, `no-store`, UTF-8 senza BOM, da `DiagnosticReport.Write` con le tracce di `IDiagnosticSource`. Nessun valore dei dati: solo conteggi, misure, orari, codici da elenchi chiusi e forme degli indirizzi. In testa `righeRovinate`, `erroriRegistratore`, `fileIlleggibili` e `sessioniSenzaFine` (`IDiagnosticSource.GatherAsync`). Risponde anche con un problema di configurazione (`ConfigurationGateMiddleware.AllowedApi`), con gli elenchi vuoti: il motore non è partito |
| `GET /api/browser/preview`, `GET /api/browser/preview/frame` | Anteprima del browser in Attività: indirizzo e immagine JPEG (`no-store`, 404 se non c'è) |
| `GET /api/duplicates/count`, `GET /api/duplicates?after=`, `POST /api/duplicates/{pairId}` (`{ decision, applyToAll }`) | Doppioni dopo uno scaricamento (§9.5): `keepLarger`, `keepExisting`, `keepBoth`; niente si decide da solo |
| `GET/PUT /api/settings/general`, `POST /api/settings/naming-preview`, `/pick-folder`, `/data-folder` | Cartelle (cambio della libreria solo desktop), schema dei nomi con anteprima, tema, avvio, area di notifica |
@@ -197,7 +199,7 @@ Moduli ES, nessuna dipendenza, nessuna build. Durante lo sviluppo si servono da
- `text.js`: `t('chiave.sotto', { n, … })` (forme `singolare|plurale` scelte da `n`), `number` (punti delle migliaia),
`decimal`, `bytes`, `duration`, `around` («verso le 15:30»), `ago` («2 ore fa»), `dayName`.
- `api.js`: `api(percorso, { method, body, signal, raw })` (lancia `ApiError` con il messaggio italiano del server),
`download`, `withCarried`. I parametri `scena`, `veste`, `tema` della dimostrazione restano negli indirizzi interni e
`download(percorso, corpo, nomeDiRiserva)` (POST con il corpo; GET con corpo `null`), `withCarried`. I parametri `scena`, `veste`, `tema` della dimostrazione restano negli indirizzi interni e
viaggiano verso l'API (intestazioni `X-Febe-Scena`, `X-Febe-Veste`).
- `engine.js`: `onEngine`, `onLibrary`, `refreshEngine`. In dimostrazione niente SSE: si chiede il motore ogni 30 s
(così le schermate senza finestra finiscono).
@@ -230,8 +232,8 @@ scrive mai nella libreria vera e non usa la rete.
di configurazione); tema: `?tema=chiaro|scuro`; veste: `?veste=desktop|windows|docker`.
- Indirizzi diretti: `/account?dialogo=aggiungi`, `/account/3/impostazioni`, `/account/2?tipo=storie`,
`/importa?passo=3` (revisione di un'importazione già analizzata), `/doppioni`, `/impostazioni/collegamento?scena=scollegato&veste=docker`.
- Seconda parte (`Demo/DemoPart2.cs`, `Demo/DemoImport.cs`): collegamento, anteprima del browser e incidenti secondo la
scena; tre doppioni dopo uno scaricamento; una sorgente da importare generata in `importa/Backup del vecchio disco`
- Seconda parte (`Demo/DemoPart2.cs`, `Demo/DemoImport.cs`, `Demo/DemoDiagnostics.cs`): collegamento, anteprima del
browser, incidenti e rapporto di diagnostica secondo la scena; tre doppioni dopo uno scaricamento; una sorgente da importare generata in `importa/Backup del vecchio disco`
(copie identiche, varianti, cartelle senza account, file non riconosciuti) e già analizzata all'avvio.
- I dati stanno in `Demo/DemoSeeder.cs` (`Specs`); le prove `tests/Febe.Tests/Web/DemoTests.cs` controllano che i
conti tornino.
+5 -2
View File
@@ -104,9 +104,12 @@ export function eventsUrl() {
return withCarried('/api/eventi');
}
/** Downloads an answer as a file (CSV export). */
/**
* Downloads an answer as a file: a POST with the body (CSV export), or a GET when the body is null (diagnostic report).
* Through api(), so the carried scene and shell travel with it.
*/
export async function download(path, body, fallbackName) {
const response = await api(path, { method: 'POST', body, raw: true });
const response = await api(path, body === null ? { raw: true } : { method: 'POST', body, raw: true });
const blob = await response.blob();
const disposition = response.headers.get('Content-Disposition') || '';
const match = /filename="?([^";]+)"?/.exec(disposition);
+34 -2
View File
@@ -4,7 +4,7 @@
// ones that are not stops, like a backfill completed or a disk almost full). It updates by itself from the event
// stream (the demo asks every 30 seconds). The red strip of a stop is drawn by the page frame.
import { api, withCarried } from '../core/api.js';
import { api, download, withCarried } from '../core/api.js';
import { h, icon, fill } from '../core/dom.js';
import { dotOf, refreshEngine } from '../core/engine.js';
import { openBrowser } from '../core/shell.js';
@@ -97,7 +97,39 @@ function drawActions(page, state) {
: h('button.btn', {
type: 'button', onclick: pause, disabled: ['Off', 'NotConnected'].includes(s.mode), 'aria-label': t('attivita.pausa'),
}, icon('pause'), h('span.testo-btn', t('attivita.pausa')));
page.actions(first, h('button.btn', { type: 'button', onclick: openBrowser, 'aria-label': t('attivita.apriBrowser') }, icon('app-window'), h('span.testo-btn', t('attivita.apriBrowser'))));
page.actions(
first,
h('button.btn', { type: 'button', onclick: openBrowser, 'aria-label': t('attivita.apriBrowser') }, icon('app-window'), h('span.testo-btn', t('attivita.apriBrowser'))),
diagnosticsButton(),
);
}
// True while a report is being exported. The title row is rebuilt at every engine update, so the button of the click
// may be gone by the end: the flag, not the button, says whether an export is running.
let exporting = false;
/** «Diagnostica»: the report of the last two weeks of sessions, with no name in it, to send after a test gone wrong (#39). */
function diagnosticsButton() {
const button = h('button.btn', {
type: 'button', title: t('attivita.esportaDiagnostica'), 'aria-label': t('attivita.esportaDiagnostica'), disabled: exporting,
}, icon('file-down'), h('span.testo-btn', t('attivita.diagnostica')));
button.onclick = async () => {
if (exporting) return;
exporting = true;
button.disabled = true;
try {
await download('/api/engine/diagnostica', null, 'febe-diagnostica.json');
toast(t('attivita.diagnosticaEsportata'));
} catch (error) {
showError(error);
} finally {
exporting = false;
// The button on the page now, which may not be this one.
for (const shown of document.querySelectorAll('[data-diagnostica]')) shown.disabled = false;
}
};
button.dataset.diagnostica = '';
return button;
}
// ------------------------------------------------------------------------------------------------- now
+3
View File
@@ -496,6 +496,9 @@
"pausa": "Metti in pausa",
"riprendi": "Riprendi",
"apriBrowser": "Apri il browser",
"diagnostica": "Diagnostica",
"esportaDiagnostica": "Esporta il rapporto di diagnostica",
"diagnosticaEsportata": "Rapporto di diagnostica esportato: il file è nei download del browser. Contiene solo conteggi, tempi e forme degli indirizzi, nessun nome né contenuto: puoi mandarlo così com'è.",
"anteprima": "Anteprima del browser di Febe",
"anteprimaOra": "Immagine delle {ora}",
"anteprimaVuota": "Il browser è sulla pagina vuota fino alla prossima sessione",
@@ -0,0 +1,576 @@
using System.Text;
using System.Text.Json;
using System.Text.RegularExpressions;
using Febe.Core.Abstractions;
using Febe.Core.Collection;
using Febe.Core.Collection.Diagnostics;
using Febe.Core.Collection.Downloads;
using Febe.Core.Collection.Privacy;
using Febe.Core.Collection.Readers;
using Febe.Core.Collection.Recipes;
using Febe.Core.Domain;
using Febe.Core.Engine;
using Febe.SitoDiProva;
using Febe.Sonda;
using Febe.Tests.Collection.Support;
using Febe.Tests.Support;
namespace Febe.Tests.Collection;
/// <summary>
/// The diagnostic report (issue #39) holds no data value: whatever names, numbers, captions, cursors, signed addresses
/// and cookies reach the recorder, the exported file keeps only closed lists, counts, sizes, times and patterns. The
/// inputs are synthetic, written here with invented values; failure messages never print a value, only its position.
/// </summary>
[TestClass]
public sealed class DiagnosticPrivacyTests
{
/// <summary>Values that stand for personal data: none may reach the report, in any case.</summary>
private static readonly string[] Secrets =
[
"marta", "prova", "bianchi", "jerry", "lucia", "5512345678", "17841401234567", "didascalia", "giornata", "spiaggia",
"QVFIUjBz", "CzX9abQ1kLm", "Zq81abcdEF", "65AABBCC", "sessionid", "segretissimo", "csrftoken", "Tk9QRVNTVA", "esempio",
"ritratto_", "ciaone", "marta_92", "luca", "bruno",
// Names made of words of the closed list (StructureWords): only the map rule and the compound rule hide them.
"the_artist", "street_style", "max_mark",
];
/// <summary>
/// The keys of <see cref="UnknownBody"/> that are fields and not data, written by hand: the structure in the report
/// may hold these (and the fields of the cleaned samples), never any other key. Independent of the word list.
/// </summary>
private static readonly string[] UnknownBodyFields =
["risultati", "voci", "chiave", "valore", "utenti", "seguito", "nome", "a", "b", "menzioni", "post", "fissato", "count", "intestazioni", "versione",
"data", "counts", "node", "tags", "edges", "seen", "items", "pk"];
private const string ProfileBody = """
{
"profilo": {
"numero": "5512345678",
"nome": "marta.prova",
"nome_visibile": "Marta Bianchi",
"riservato": false,
"seguito": true,
"pubblicazioni": 18,
"ritratto": [
{ "indirizzo": "https://cdn.esempio/v/t51/ritratto_marta.jpg?oh=Zq81abcdEF&oe=65AABBCC", "larghezza": 160, "altezza": 160 }
]
},
"raccolte": []
}
""";
private const string UnknownBody = """
{
"risultati": {
"voci": [
{ "chiave": "QVFIUjBzcursore", "valore": "Didascalia: giornata in spiaggia con jerry e lucia, ciaone!" }
]
},
"utenti": {
"marta.prova": { "seguito": true },
"jerry": { "nome": "jerry" },
"17841401234567": { "a": 1 },
"Città di Lucia": { "b": 2 },
"marta_92": { "seguito": true },
"luca": { "seguito": false },
"the_artist": { "seguito": true },
"street_style": { "seguito": false }
},
"max_mark": { "versione": 1 },
"menzioni": "@marta_92 e @luca",
"post": { "CzX9abQ1kLm": { "fissato": true } },
"bruno": [ { "count": 1 } ],
"intestazioni": "sessionid=segretissimo123; csrftoken=Tk9QRVNTVA",
"versione": 3
}
""";
/// <summary>
/// Maps keyed by names made of words of the closed list, in the shapes a same-shape rule misses: counts, empty values
/// and lists of codes, entries with different optional fields, a list of maps with one entry each.
/// </summary>
private const string UnknownMapsBody = """
{
"data": {
"counts": { "the_artist": 3, "street_style": 5 },
"node": {
"tags": { "the_artist": [], "street_style": [ "c1" ] },
"edges": {
"seen": { "the_artist": { "pk": "1", "seen": 2 }, "street_style": { "pk": "2" } }
},
"items": [ { "the_artist": { "pk": "1" } }, { "street_style": { "pk": "2" } } ]
}
}
}
""";
[TestMethod]
public void TheReportHoldsNoNameNumberCaptionCursorAddressOrCookie()
{
var clock = new ManualClock(new DateTimeOffset(2026, 11, 13, 8, 0, 0, TimeSpan.Zero), TestZones.Rome);
long? current = 41;
var diagnostics = new CollectionDiagnostics(clock, () => current) { SiteMap = SiteMap.LoadTestSite() };
var capture = new CaptureCollector(new ShapeReader(ReaderVocabulary.LoadTestSite()), clock) { Diagnostics = diagnostics };
diagnostics.SessionStarted(41, clock.Now, 0.93);
// The page: navigations, responses with sizes, a dropped body, and the bodies through the readers.
diagnostics.OnNavigation(new NavigationDescriptor(HostKind.Site, Site("/profilo/marta.prova/?dopo=QVFIUjBz"), "sito.esempio", "/profilo/marta.prova/", clock.Now));
diagnostics.OnNavigation(new NavigationDescriptor(HostKind.Site, Site("/post/CzX9abQ1kLm/"), "sito.esempio", "/post/CzX9abQ1kLm/", clock.Now, SameDocument: true));
diagnostics.OnNavigation(new NavigationDescriptor(HostKind.Other, "https://altro.esempio/marta.prova", "altro.esempio", "/marta.prova", clock.Now));
var profile = Response("r1", HostKind.Site, "/dati/profilo/marta.prova/", "?dopo=QVFIUjBz");
var unknown = Response("r2", HostKind.Site, "/dati/5512345678/lucia.bianchi/", "?cursore=QVFIUjBz");
var picture = Response("r3", HostKind.Cdn, "/v/t51/ritratto_marta.jpg", "?oh=Zq81abcdEF&oe=65AABBCC", "image/jpeg", "Image");
var other = Response("r4", HostKind.Other, "/collect/marta.prova.gif", "?u=jerry", "image/gif", "Ping");
var dropped = Response("r5", HostKind.Site, "/api/v1/users/5512345678/info/", "", "application/json; charset=utf-8", "XHR");
var page = Response("r6", HostKind.Site, "/post/CzX9abQ1kLm/", "", "text/html", "Document", mainFrame: true);
var odd = Response("r7", HostKind.Site, "/marta.prova/", "", "text/marta.prova", "marta");
foreach (var response in new[] { profile, unknown, picture, other, dropped, page, odd })
{
diagnostics.OnResponse(response);
diagnostics.OnLoadingFinished(response.RequestId, 2048);
}
diagnostics.OnBodyDropped("r5", BodyDropReason.Timeout);
using (var body = JsonDocument.Parse(ProfileBody))
{
Assert.AreEqual(ReadShape.Recognized, capture.Accept(profile, body.RootElement.Clone())?.Result.Shape);
}
using (var body = JsonDocument.Parse(UnknownBody))
{
Assert.AreEqual(ReadShape.Unrecognized, capture.Accept(unknown, body.RootElement.Clone())?.Result.Shape);
Assert.IsNull(capture.Accept(other, body.RootElement.Clone()), "A body of another host is not read.");
}
var maps = Response("r9", HostKind.Site, "/dati/mappe/", "");
diagnostics.OnResponse(maps);
using (var body = JsonDocument.Parse(UnknownMapsBody))
{
Assert.AreEqual(ReadShape.Unrecognized, capture.Accept(maps, body.RootElement.Clone())?.Result.Shape);
}
// The recipes and the engine: names where codes are expected become «altro».
diagnostics.PageOpened(41, SitePages.Profile, ok: true);
diagnostics.PageOpened(41, "marta.prova", ok: false);
diagnostics.TargetClicked(41, SiteTargets.ProfileStoryRing, TargetCheckStatus.Clicked);
diagnostics.TargetClicked(41, "jerry", TargetCheckStatus.NotFound);
diagnostics.StepJudged(41, CollectionDiagnostics.VerdictUnknown);
diagnostics.StepJudged(41, "Didascalia di lucia");
diagnostics.RecipeEnded(41, RecipeKind.ProfileCheck, RecipeStatus.StoppedBySignal, "http-429", 1, 0);
diagnostics.RecipeEnded(41, RecipeKind.ProfileCheck, RecipeStatus.Failed, "marta.prova@esempio", 0, 0, "Errore di marta.prova");
diagnostics.RecipeEnded(41, RecipeKind.ProfileCheck, RecipeStatus.Failed, null, 0, 0, nameof(TimeoutException));
diagnostics.SignalRaised(new Signal(SignalLevel.Throttle, "http-429", Site("/profilo/marta.prova/?dopo=QVFIUjBz"), clock.Now));
diagnostics.SignalRaised(new Signal(SignalLevel.Challenge, "jerry.ciaone", "https://sito.esempio/challenge/?next=/jerry/", clock.Now));
diagnostics.PermitDenied(null, ActionKind.ProfileOpen, DenialReason.HourlyBudgetReached);
diagnostics.FilesQueued(41, EntryKind.Post, 3, nameof(SaveOutcome.Created), skipped: 1);
diagnostics.FilesQueued(41, EntryKind.Post, 1, "segretissimo");
diagnostics.DownloadSettled(new DownloadProgress(17841401234567, 5512345678, 5512345678, EntryKind.Post, DownloadStep.Stored, 2048));
clock.Now = clock.Now.AddMinutes(7);
current = null;
diagnostics.SessionEnded(41, clock.Now, SessionOutcome.Throttled, SessionStopCause.BreakerOpen, RecipeStatus.StoppedBySignal);
// Outside sessions: Alberto at the browser.
diagnostics.OnNavigation(new NavigationDescriptor(HostKind.Site, Site("/profilo/jerry/"), "sito.esempio", "/profilo/jerry/", clock.Now));
diagnostics.OnResponse(Response("r8", HostKind.Site, "/storie/jerry/17841401234567/", ""));
diagnostics.SignalRaised(new Signal(SignalLevel.Disconnect, SignalRules.SessionCookieMissingCode, null, clock.Now));
diagnostics.DownloadSettled(new DownloadProgress(17841401234567, 5512345678, 5512345678, EntryKind.Story, DownloadStep.Failed));
diagnostics.DownloadSettled(new DownloadProgress(1, 2, 3, EntryKind.Story, DownloadStep.Retried) { Cause = DownloadCauses.Of(404) });
diagnostics.DownloadSettled(new DownloadProgress(1, 2, 3, EntryKind.Story, DownloadStep.Failed) { Cause = "https://cdn.esempio/marta.prova.jpg" });
var text = Export(diagnostics.Collect(), clock);
Assert.AreEqual(0, diagnostics.Faults);
using var report = JsonDocument.Parse(text);
Assert.AreEqual(1, report.RootElement.GetProperty("sessioni").GetArrayLength());
Assert.AreEqual(1, report.RootElement.GetProperty("fuoriSessione").GetArrayLength());
StringAssert.Contains(text, "\"structure\"", "The unknown shape is there, by its keys.");
foreach (var hiddenMap in new[] { "{data:{counts:{#},node:{tags:{#:[]},edges:{seen:{#:{pk,seen}}},items:[{#:{pk}}]}}}" })
{
StringAssert.Contains(text, hiddenMap, "The maps are there, without their names.");
}
var tokens = new SensitiveTokens(SanitizerAllowlist.Default);
foreach (var input in new[] { ProfileBody, UnknownBody, UnknownMapsBody })
{
using var document = JsonDocument.Parse(input);
tokens.Add(document.RootElement);
}
ReportLeaks.AssertClean(text, Secrets, tokens.Effective, ReportLeaks.FieldKeys(UnknownBodyFields));
}
[TestMethod]
public void TheLogOnDiskHoldsNoValueEither()
{
using var folder = new TempFolder("diagnostica");
var clock = new ManualClock(new DateTimeOffset(2026, 11, 13, 8, 0, 0, TimeSpan.Zero), TestZones.Rome);
var log = new DiagnosticLog(folder.Combine("diagnostica"), clock);
var diagnostics = new CollectionDiagnostics(clock, () => 7, log) { SiteMap = SiteMap.LoadTestSite() };
diagnostics.SessionStarted(7, clock.Now, 1);
diagnostics.OnResponse(Response("l1", HostKind.Site, "/profilo/marta.prova/", "?dopo=QVFIUjBz"));
diagnostics.ReadObserved(Response("l2", HostKind.Site, "/dati/marta.prova/", ""), new ReadResult { Shape = ReadShape.Unrecognized }, "{utenti:{#:{seguito}}}");
diagnostics.SignalRaised(new Signal(SignalLevel.Throttle, "marta.prova", Site("/marta.prova/"), clock.Now));
diagnostics.SessionEnded(7, clock.Now, SessionOutcome.Completed, SessionStopCause.WorkDone);
diagnostics.Dispose();
var written = string.Concat(Directory.EnumerateFiles(log.Folder).Select(File.ReadAllText));
Assert.IsGreaterThan(0, written.Length);
for (var i = 0; i < Secrets.Length; i++)
{
Assert.IsFalse(written.Contains(Secrets[i], StringComparison.OrdinalIgnoreCase), $"Secret #{i} is in the log.");
}
Assert.IsFalse(written.Contains("://", StringComparison.Ordinal), "An address is in the log.");
}
private static string Export(IReadOnlyList<SessionTrace> traces, IClock clock)
{
using var stream = new MemoryStream();
DiagnosticReport.Write(stream, traces, new DiagnosticReportHeader("0.0.0-prove", "docker", clock.Now, TestZones.Rome));
return Encoding.UTF8.GetString(stream.ToArray());
}
private static string Site(string pathAndQuery) => "https://sito.esempio" + pathAndQuery;
private static ResponseDescriptor Response(string id, HostKind host, string path, string query, string mime = "application/json", string? resourceType = "Fetch", bool mainFrame = false)
{
var origin = host switch
{
HostKind.Site => "sito.esempio",
HostKind.Cdn => "cdn.esempio",
_ => "altro.esempio",
};
return new ResponseDescriptor(id, host, "https://" + origin + path + query, origin, path, 200, mime, resourceType, mainFrame, DateTimeOffset.UnixEpoch);
}
}
/// <summary>
/// The diagnostic report end to end on the local test site, with a real Chromium: a recipe, the export, and none of the
/// world's names, captions, codes or identifiers in the file. Never Instagram.
/// </summary>
[TestClass]
[TestCategory(SyntheticSite.Category)]
[DoNotParallelize]
public sealed class DiagnosticPrivacyIntegrationTests
{
private static SharedChromium? s_chromium;
private static string? s_unavailable;
private static SharedChromium Chromium => s_chromium ?? throw new InvalidOperationException("Browser non avviato.");
[ClassInitialize]
public static async Task StartAsync(TestContext context)
{
s_unavailable = SharedChromium.UnavailableReason();
if (s_unavailable is null)
{
s_chromium = await SharedChromium.StartAsync();
}
}
[ClassCleanup]
public static async Task StopAsync()
{
if (s_chromium is not null)
{
await s_chromium.DisposeAsync();
s_chromium = null;
}
}
[TestInitialize]
public void RequireBrowser()
{
if (s_unavailable is not null)
{
Assert.Inconclusive(s_unavailable);
}
}
[TestMethod]
[Timeout(90_000, CooperativeCancellation = true)]
public async Task ARecipeOnTheTestSiteLeavesNoValueOfTheWorldInTheReport()
{
await using var h = await CollectionHarness.CreateAsync(Chromium, diagnostics: true);
var diagnostics = h.Diagnostics!;
diagnostics.SessionStarted(h.Session.SessionId, h.Clock.Now, 1);
var marta = await h.AddAccountAsync("marta.prova");
var outcome = await h.RunAsync(RecipeKind.ProfileCheck, marta.Id);
Assert.AreEqual(RecipeStatus.Completed, outcome.Status);
await h.RunAsync(RecipeKind.FollowingSweep);
diagnostics.SessionEnded(h.Session.SessionId, h.Clock.Now, SessionOutcome.Completed, SessionStopCause.WorkDone);
using var stream = new MemoryStream();
DiagnosticReport.Write(stream, diagnostics.Collect(), new DiagnosticReportHeader("0.0.0-prove", "docker", h.Clock.Now, TestZones.Rome));
var text = Encoding.UTF8.GetString(stream.ToArray());
using var report = JsonDocument.Parse(text);
var session = report.RootElement.GetProperty("sessioni").EnumerateArray().Single();
Assert.IsGreaterThan(0, session.GetProperty("responses").GetArrayLength(), "The page was seen.");
Assert.IsTrue(session.GetProperty("recipes").EnumerateArray().Any(r => r.GetProperty("kind").GetString() == nameof(RecipeKind.ProfileCheck)));
// Every name, display name, caption, post code, highlight title and identifier of the synthetic world.
var world = new List<string>();
lock (h.Site.World.Gate)
{
foreach (var account in h.Site.World.Accounts)
{
world.Add(account.Username);
world.Add(account.DisplayName);
world.Add(account.Id);
world.AddRange(account.Posts.SelectMany(p => new[] { p.Code, p.Caption, p.Id }));
world.AddRange(account.Stories.Select(s => s.Id));
world.AddRange(account.Highlights.SelectMany(x => new[] { x.Title, x.Id }));
}
}
var vocabulary = ReportLeaks.Vocabulary(SiteMap.LoadTestSite());
var worldTokens = world.SelectMany(SensitiveTokens.Tokenize).Where(t => !vocabulary.Contains(t)).ToHashSet(StringComparer.Ordinal);
Assert.IsGreaterThan(20, worldTokens.Count);
ReportLeaks.AssertClean(text, [], worldTokens, ReportLeaks.FieldKeys([]));
Assert.AreEqual(0, diagnostics.Faults);
}
}
/// <summary>The checks the privacy tests share; their messages name positions and kinds, never values.</summary>
internal static partial class ReportLeaks
{
private static readonly string[] CdpResourceTypes =
[
"Document", "Stylesheet", "Image", "Media", "Font", "Script", "TextTrack", "XHR", "Fetch", "Prefetch", "EventSource",
"WebSocket", "Manifest", "SignedExchange", "Ping", "CSPViolationReport", "Preflight", "FedCM", "Other",
];
/// <summary>
/// No explicit secret and no sensitive token anywhere; no address, mail or query; every pattern made of words of the
/// site and placeholders; every key of a key structure one of <paramref name="fieldKeys"/> (checked against real
/// fields, not against the word list under test); every other text value from a closed list or a time.
/// </summary>
public static void AssertClean(string text, IReadOnlyList<string> secrets, IReadOnlySet<string> sensitive, IReadOnlySet<string> fieldKeys)
{
for (var i = 0; i < secrets.Count; i++)
{
Assert.IsFalse(text.Contains(secrets[i], StringComparison.OrdinalIgnoreCase), $"Secret #{i} is in the report.");
}
var leaked = SensitiveTokens.Tokenize(text).Distinct().Count(sensitive.Contains);
Assert.AreEqual(0, leaked, "Sensitive tokens of the inputs are in the report (count only).");
Assert.IsFalse(text.Contains("://", StringComparison.Ordinal), "An address is in the report.");
Assert.IsFalse(text.Contains('@', StringComparison.Ordinal), "A mail or a handle is in the report.");
Assert.IsFalse(text.Contains('?', StringComparison.Ordinal), "A query is in the report.");
using var report = JsonDocument.Parse(text);
var literals = Literals(SiteMap.LoadTestSite()).Concat(Literals(SiteMap.LoadInstagram())).ToHashSet(StringComparer.Ordinal);
var allowed = ClosedValues();
var position = 0;
Walk(report.RootElement, null, (name, value) =>
{
position++;
switch (name)
{
case "pattern":
Assert.IsTrue(IsPattern(value, literals), $"Pattern #{position} has a segment that is not a word of the site.");
break;
case "structure":
Assert.IsTrue(value == "(nessuna struttura)" || StructureShape().IsMatch(value), $"Structure #{position} has an odd character.");
Assert.IsTrue(
value == "(nessuna struttura)" || StructureKeys().Split(value).Where(k => k.Length > 0).All(k => k == "#" || fieldKeys.Contains(k)),
$"Structure #{position} has a key that is not a field of the samples or of the body.");
Assert.IsTrue(
value == "(nessuna struttura)" || StructureKeys().Split(value).Where(k => k.Length > 0).All(k => k == "#" || StructureWords.IsStructural(k)),
$"Structure #{position} has a key that is not made of words of the closed list.");
break;
case "error":
Assert.IsTrue(value == CollectionDiagnostics.Other || TypeName().IsMatch(value), $"Error #{position} is not a type name.");
break;
case "cause":
Assert.IsTrue(
value is DownloadCauses.Network or DownloadCauses.Connection or DownloadCauses.Write or DownloadCauses.BadAddress or DownloadCauses.HostNotAllowed or DownloadCauses.RefusedAfterRenewal
|| value == CollectionDiagnostics.Other || HttpCause().IsMatch(value),
$"Cause #{position} is not from the closed list.");
break;
case "start" or "end" or "at" or "creato":
Assert.IsTrue(DateTimeOffset.TryParse(value, System.Globalization.CultureInfo.InvariantCulture, out _), $"Time #{position} is not a time.");
break;
case "versione" or "edizione" or "fuso" or "rapporto":
break;
default:
Assert.IsTrue(allowed.Contains(value) || CollectionDiagnostics.MimeOf(value) == value, $"Value #{position} ({name}) is not from a closed list.");
break;
}
});
}
/// <summary>
/// The keys that are fields: every key of the cleaned samples of <c>tests/campioni</c> (their data keys were already
/// replaced by the sanitizer), every whole key of the shipped reader vocabularies, and <paramref name="extra"/>.
/// </summary>
public static HashSet<string> FieldKeys(IEnumerable<string> extra)
{
var keys = new HashSet<string>(extra, StringComparer.Ordinal);
foreach (var file in Directory.EnumerateFiles(Path.Combine(AppContext.BaseDirectory, "campioni"), "*.json", SearchOption.AllDirectories))
{
using var document = JsonDocument.Parse(File.ReadAllText(file));
AddKeys(document.RootElement, keys, vocabulary: false);
}
foreach (var name in new[] { "lettori.instagram.json", "lettori.prova.json" })
{
using var document = JsonDocument.Parse(SiteMap.ReadEmbeddedText(name));
AddKeys(document.RootElement, keys, vocabulary: true);
}
return keys;
}
/// <summary>Words the report is made of by itself (keys, enums, pages, targets, the site's own words).</summary>
public static HashSet<string> Vocabulary(SiteMap map)
{
var words = new HashSet<string>(StringComparer.Ordinal);
foreach (var value in ClosedValues().Concat(Literals(map)).Concat(PropertyNames()).Append(DiagnosticReport.Title).Append(TestZones.Rome.Id))
{
words.UnionWith(SensitiveTokens.Tokenize(value));
}
return words;
}
private static HashSet<string> ClosedValues()
{
var values = new HashSet<string>(StringComparer.Ordinal)
{
CollectionDiagnostics.Other, CollectionDiagnostics.VerdictRecognized, CollectionDiagnostics.VerdictEmpty,
CollectionDiagnostics.VerdictUnknown, CollectionDiagnostics.CompletedOutcome, "http-429", SignalRules.SessionCookieMissingCode,
"0", "1xx", "2xx", "3xx", "4xx", "5xx", string.Empty,
};
values.UnionWith(SitePages.All);
values.UnionWith(SiteTargets.All);
// The signal codes of the rule files shipped with Febe (segnali.json, segnali.prova.json): a closed list too.
foreach (var rules in new[] { SignalRules.LoadDefault(), SignalRules.LoadTestSite() })
{
values.UnionWith(rules.Rules.Select(r => r.Code));
if (rules.EmptyPages is { } empty)
{
values.Add(empty.Code);
}
}
values.UnionWith(CdpResourceTypes);
foreach (var type in new[]
{
typeof(SessionOutcome), typeof(SessionStopCause), typeof(RecipeStatus), typeof(RecipeKind), typeof(SignalLevel), typeof(ActionKind),
typeof(DenialReason), typeof(EntryKind), typeof(DownloadStep), typeof(BodyDropReason), typeof(HostKind), typeof(TargetCheckStatus),
typeof(SaveOutcome),
})
{
values.UnionWith(Enum.GetNames(type));
}
return values;
}
private static IEnumerable<string> PropertyNames() =>
[
"rapporto", "versione", "edizione", "creato", "fuso", "giorni", "sessioni", "fuoriSessione", "session", "start", "end", "outcome",
"stopCause", "stoppedBy", "mood", "inProgress", "pages", "navigations", "responses", "unknownShapes", "steps", "targets", "recipes",
"signals", "denials", "filesQueued", "downloads", "downloadedBytes", "overflow", "page", "pattern", "hostKind", "sameDocument",
"count", "resourceType", "mime", "status", "bytes", "maxBytes", "recognized", "empty", "unrelated", "unrecognized", "notSite",
"notRead", "contents", "profiles", "dropped", "reason", "structure", "verdict", "target", "kind", "signal", "actions",
"newEntries", "level", "code", "action", "saves", "files", "skipped", "step", "requests", "ok", "cdn", "error", "cause", "late",
"righeRovinate", "erroriRegistratore", "fileIlleggibili", "sessioniSenzaFine",
];
private static IEnumerable<string> Literals(SiteMap map) =>
SitePages.All
.Select(map.PatternOf)
.OfType<string>()
.SelectMany(p => p.Split('/', StringSplitOptions.RemoveEmptyEntries))
.Where(s => !s.StartsWith('{'));
private static bool IsPattern(string pattern, HashSet<string> literals)
{
if (!pattern.StartsWith('/'))
{
return OtherHostPattern().IsMatch(pattern);
}
return pattern.Split('/', StringSplitOptions.RemoveEmptyEntries).All(segment =>
segment is PathPatterns.Number or PathPatterns.Text
|| PathPatterns.IsKnownSegment(segment)
|| literals.Contains(segment)
|| Placeholder().IsMatch(segment));
}
/// <summary>The keys of a sample, or the dotted key paths written in a vocabulary, split into whole keys.</summary>
private static void AddKeys(JsonElement element, HashSet<string> keys, bool vocabulary)
{
switch (element.ValueKind)
{
case JsonValueKind.Object:
foreach (var property in element.EnumerateObject())
{
if (!vocabulary)
{
keys.Add(property.Name);
}
AddKeys(property.Value, keys, vocabulary);
}
break;
case JsonValueKind.Array:
foreach (var item in element.EnumerateArray())
{
AddKeys(item, keys, vocabulary);
}
break;
case JsonValueKind.String when vocabulary:
keys.UnionWith(element.GetString()!.Split('.', StringSplitOptions.RemoveEmptyEntries).Where(k => !k.Contains(' ', StringComparison.Ordinal)));
break;
}
}
private static void Walk(JsonElement element, string? name, Action<string, string> visit)
{
switch (element.ValueKind)
{
case JsonValueKind.Object:
foreach (var property in element.EnumerateObject())
{
Walk(property.Value, property.Name, visit);
}
break;
case JsonValueKind.Array:
foreach (var item in element.EnumerateArray())
{
Walk(item, name, visit);
}
break;
case JsonValueKind.String:
visit(name ?? string.Empty, element.GetString() ?? string.Empty);
break;
}
}
[GeneratedRegex(@"^(cdn|altro)( \*\.[a-z0-9]+)?$", RegexOptions.CultureInvariant)]
private static partial Regex OtherHostPattern();
[GeneratedRegex(@"^\{[a-z]+\}$", RegexOptions.CultureInvariant)]
private static partial Regex Placeholder();
[GeneratedRegex(@"^[\p{L}\p{N}_\-#{}:,\[\]…]*$", RegexOptions.CultureInvariant)]
private static partial Regex StructureShape();
[GeneratedRegex(@"[{}:,\[\]…]", RegexOptions.CultureInvariant)]
private static partial Regex StructureKeys();
[GeneratedRegex(@"^[A-Z][A-Za-z0-9_]*$", RegexOptions.CultureInvariant)]
private static partial Regex TypeName();
[GeneratedRegex(@"^http-[1-5][0-9]{2}$", RegexOptions.CultureInvariant)]
private static partial Regex HttpCause();
}
@@ -0,0 +1,846 @@
using System.Text;
using System.Text.Json;
using Febe.Core.Abstractions;
using Febe.Core.Collection;
using Febe.Core.Collection.Diagnostics;
using Febe.Core.Collection.Downloads;
using Febe.Core.Collection.Privacy;
using Febe.Core.Collection.Readers;
using Febe.Core.Collection.Recipes;
using Febe.Core.Domain;
using Febe.Core.Engine;
using Febe.Sonda;
using Febe.Tests.Support;
namespace Febe.Tests.Collection;
/// <summary>
/// The pieces of the diagnostic report (issue #39): path patterns, page names of the site map, the strict key structure,
/// the recorder with its caps and its buckets, the log of two weeks and the exported file. The proof that the report holds
/// no data value is <see cref="DiagnosticPrivacyTests"/>.
/// </summary>
[TestClass]
public sealed class DiagnosticsTests
{
private static readonly DateTimeOffset Start = new(2026, 11, 13, 8, 0, 0, TimeSpan.Zero);
// ---- Path patterns (moved from the probe into Core) ----
[TestMethod]
[DataRow("/mario.rossi/", "/{testo}/")]
[DataRow("/api/v1/feed/user/5512345678/", "/api/v1/feed/user/{numero}/")]
[DataRow("/p/AbC12x/", "/p/{testo}/")]
[DataRow("/stories/highlights/17912345678901234/", "/stories/highlights/{numero}/")]
[DataRow("/graphql/query", "/graphql/query")]
[DataRow("/profilo/marta.prova/?dopo=QVFI#x", "/{testo}/{testo}/")]
[DataRow("", "")]
public void APathPatternKeepsOnlyTheWordsOfTheSite(string path, string expected)
{
Assert.AreEqual(expected, PathPatterns.Of(path));
Assert.AreEqual(PathPatterns.Of(path.Split('?')[0]), RequestShapes.PathPattern(path.Split('?')[0]), "The probe uses the same code.");
}
[TestMethod]
[DataRow("/v/t51.2885-15/foto_n.jpg", "jpg")]
[DataRow("/o1/v/t16/video.MP4", "mp4")]
[DataRow("/profilo/marta.prova", null)]
[DataRow("/senza-estensione/", null)]
[DataRow("/file.", null)]
[DataRow(null, null)]
public void OnlyKnownFileExtensionsAreKept(string? path, string? expected) =>
Assert.AreEqual(expected, PathPatterns.ExtensionOf(path));
// ---- Page names of the site map ----
[TestMethod]
public void AnAddressIsNamedByThePageWithMoreWordsOfItsOwnFirst()
{
var instagram = SiteMap.LoadInstagram();
Assert.AreEqual(SitePages.HighlightViewer, instagram.Classify("/stories/highlights/17912345678901234/"), "Not the stories of «highlights».");
Assert.AreEqual(SitePages.StoryViewer, instagram.Classify("/stories/marta.prova/3456789012345678901/"));
Assert.AreEqual(SitePages.Post, instagram.Classify("/p/CzX9abQ1kLm/"));
Assert.AreEqual(SitePages.Profile, instagram.Classify("/marta.prova/"), "The profile comes last.");
Assert.AreEqual(SitePages.Home, instagram.Classify("/"));
Assert.IsNull(instagram.Classify("/api/v1/users/web_profile_info/"));
Assert.IsNull(instagram.Classify("/graphql/query"));
Assert.IsNull(instagram.Classify(null));
Assert.AreEqual("/{username}/", instagram.PatternOf(SitePages.Profile));
Assert.AreEqual("/stories/highlights/{highlight}/", instagram.PatternOf(SitePages.HighlightViewer));
Assert.IsNull(instagram.PatternOf(SitePages.Following), "Not set on Instagram yet.");
var test = SiteMap.LoadTestSite();
Assert.AreEqual(SitePages.Following, test.Classify("/profilo/marta.prova/seguiti/"));
Assert.AreEqual(SitePages.Profile, test.Classify("/profilo/marta.prova/"), "The shape without a query before the same shape with one.");
Assert.AreEqual(SitePages.ReelsFrom, test.Classify("/profilo/marta.prova/reel/"));
Assert.AreEqual(SitePages.StoryViewer, test.Classify("/storie/marta.prova/12345/"));
Assert.AreEqual("/profilo/{username}/", test.PatternOf(SitePages.ProfileFrom), "Never the query of the shape.");
}
[TestMethod]
[DataRow("/challenge/")]
[DataRow("/explore/")]
[DataRow("/reels/")]
[DataRow("/direct/")]
[DataRow("/accounts/")]
[DataRow("/p/")]
[DataRow("/stories/")]
public void AFixedWordOfTheSiteIsNotAProfile(string path)
{
var instagram = SiteMap.LoadInstagram();
Assert.IsNull(instagram.Classify(path), "A placeholder never takes a word of the site.");
var diagnostics = new CollectionDiagnostics(new ManualClock(Start)) { SiteMap = instagram };
diagnostics.OnNavigation(new NavigationDescriptor(HostKind.Site, "https://indirizzo.non.usato" + path + "?next=/marta.prova/", "indirizzo.non.usato", path, Start));
var navigation = diagnostics.Snapshot(Start).Single().Navigations.Single();
Assert.IsNull(navigation.Page);
Assert.AreEqual(path, navigation.Pattern, "The word itself, not «/{testo}/» and not «/{username}/».");
Assert.AreEqual(SitePages.Profile, instagram.Classify("/marta_92/"), "A real profile is still one.");
Assert.IsNull(SiteMap.LoadTestSite().Classify("/profilo/seguiti/"), "Nor a word of the map itself.");
}
[TestMethod]
public void AShapeWrittenInTheDataFolderIsReducedLikeAnyPath()
{
// Alberto fills «following» with his own address instead of «/{username}/following/».
var map = SiteMap.Parse("""
{
"site": "instagram",
"pages": {
"profile": "/{username}/",
"following": "/alberto.rossi/following/",
"followingFrom": "/alberto.rossi/following/{cursor}/{nome}/1234/",
"post": "/p/{code}/"
}
}
""", "bersagli.instagram.json");
Assert.AreEqual("/{testo}/following/", map.PatternOf(SitePages.Following));
Assert.AreEqual("/{testo}/following/{cursor}/{testo}/{numero}/", map.PatternOf(SitePages.FollowingFrom), "Only the placeholders of the shipped maps.");
Assert.AreEqual("/p/{code}/", map.PatternOf(SitePages.Post));
var clock = new ManualClock(Start);
var diagnostics = new CollectionDiagnostics(clock, () => 1) { SiteMap = map };
diagnostics.SessionStarted(1, clock.Now, 1);
diagnostics.PageOpened(1, SitePages.Following, ok: true);
diagnostics.OnNavigation(new NavigationDescriptor(HostKind.Site, "https://indirizzo.non.usato/alberto.rossi/following/", "indirizzo.non.usato", "/alberto.rossi/following/", clock.Now));
diagnostics.OnResponse(Response("f1", HostKind.Site, "/alberto.rossi/following/", mime: "text/html", resourceType: "Document", mainFrame: true));
var trace = diagnostics.Snapshot(clock.Now).Single();
Assert.AreEqual("/{testo}/following/", trace.Pages.Single().Pattern);
Assert.AreEqual((SitePages.Following, "/{testo}/following/"), (trace.Navigations.Single().Page, trace.Navigations.Single().Pattern));
Assert.AreEqual("/{testo}/following/", trace.Responses.Single().Pattern);
}
// ---- Strict key structure ----
[TestMethod]
public void TheStrictKeyStructureHidesNamesUsedAsKeys()
{
using var body = JsonDocument.Parse("""
{
"utenti": {
"marta.prova": { "seguito": true },
"jerry": { "nome": "Jerry" },
"luca": { "x": 1 }
},
"17841401234567": { "a": 1 },
"Città": { "b": 2 },
"stato": "ok"
}
""");
var strict = KeyStructure.DescribeStrict(body.RootElement);
Assert.AreEqual("{stato,#:{#:{seguito}}}", strict, "Nor «utenti», «luca» and «jerry»: not words of the closed list.");
Assert.AreEqual("{utenti:{marta.prova:{seguito},jerry:{nome},luca:{x}},Città:{b},stato,#:{a}}", KeyStructure.Describe(body.RootElement), "The event of «formato non riconosciuto» is unchanged.");
Assert.IsTrue(KeyStructure.IsDataKey("marta.prova"));
Assert.IsTrue(KeyStructure.IsDataKey("a@b"));
Assert.IsTrue(KeyStructure.IsDataKey("due parole"));
Assert.IsTrue(KeyStructure.IsDataKey("17841401234567"));
Assert.IsFalse(KeyStructure.IsDataKey("image_versions2"));
Assert.IsFalse(KeyStructure.IsDataKey("ultima-storia"));
Assert.AreEqual(KeyStructure.IsDataKey("lucia.rossi"), SensitiveTokens.IsDataKey("lucia.rossi"), "The probe uses the same rule.");
}
[TestMethod]
public void AKeyThatIsNotMadeOfWordsOfTheClosedListIsHiddenEvenWhenTheBodyDoesNotRepeatIt()
{
// A map keyed by user and by post, with no name repeated as a value (only «@marta_92» inside a text).
using var body = JsonDocument.Parse("""
{
"data": {
"users": { "marta_92": { "is_private": false, "follower_count": 3 }, "jerry": { "is_private": true } },
"media": { "CzX9abQ1kLm": { "taken_at": 1, "image_versions2": { "width": 1 } } },
"page_info": { "has_next_page": true, "end_cursor": "QVFIUjBz", "next_max_id": "x" },
"max": { "a": 1 },
"__typename": "XDTUserDict",
"text": "con @marta_92"
}
}
""");
var strict = KeyStructure.DescribeStrict(body.RootElement);
Assert.AreEqual(
"{data:{users:{#:{is_private,follower_count}},media:{#:{taken_at,image_versions2:{width}}},page_info:{has_next_page,end_cursor,#},__typename,text,#:{#}}}",
strict);
foreach (var name in new[] { "marta_92", "jerry", "CzX9abQ1kLm" })
{
Assert.IsFalse(StructureWords.IsStructural(name), "A name or a post code is not structure.");
}
Assert.IsFalse(StructureWords.IsStructural("max"), "A word that is also a first name passes only inside a compound key.");
Assert.IsTrue(StructureWords.IsStructural("next_max_id"));
Assert.IsTrue(StructureWords.IsStructural("pubblicazioni"), "The words of the test site's vocabulary.");
Assert.IsFalse(StructureWords.IsStructural("marta.prova"));
Assert.IsFalse(StructureWords.IsStructural("17841401234567"));
}
[TestMethod]
public void NamesMadeOfWordsOfTheListAreHiddenAsEntriesOfAMapAndCompoundOnlyWordsNeverStandTogether()
{
// Two first names together are a name, not a key: a compound-only word passes only next to a word of the list.
Assert.IsFalse(StructureWords.IsStructural("max_mark"));
Assert.IsFalse(StructureWords.IsStructural("mark-max"));
Assert.IsTrue(StructureWords.IsStructural("max_music"), "«music» is a word of the list: only the map rule can hide it.");
Assert.IsTrue(StructureWords.IsKnownKey("user"));
Assert.IsTrue(StructureWords.IsKnownKey("owner"));
Assert.IsFalse(StructureWords.IsKnownKey("the_artist"));
Assert.IsFalse(StructureWords.IsKnownKey("max"), "A known key is still a structural one.");
// A map keyed by user name, every name made of words of the list, none repeated as a value: the shape of the
// values gives it away. «user» and «owner» of a post have the same shape too, and stay: the readers know them.
using var body = JsonDocument.Parse("""
{
"users": { "the_artist": { "is_private": false }, "max_music": { "is_private": true } },
"playlists": { "street_style": [ { "pk": 1 } ], "live_music": [ { "pk": 2 } ] },
"item": { "user": { "pk": 1, "username": "x" }, "owner": { "pk": 1, "username": "x" }, "caption": { "text": "y" } },
"max_mark": { "count": 1 },
"status": "ok"
}
""");
var strict = KeyStructure.DescribeStrict(body.RootElement);
Assert.AreEqual(
"{users:{#:{is_private}},playlists:{#:[{pk}]},item:{user:{pk,username},owner:{pk,username},caption:{text}},status,#:{#}}",
strict);
foreach (var name in new[] { "the_artist", "max_music", "street_style", "live_music", "max_mark" })
{
Assert.IsFalse(strict.Contains(name, StringComparison.OrdinalIgnoreCase), "A name is in the structure.");
}
}
[TestMethod]
public void NamesMadeOfWordsOfTheListAreHiddenInMapsOfPlainValuesEmptyValuesAndUnlikeEntries()
{
// Maps keyed by user name, every name made of words of the list, none repeated as a value. Known fields among
// the entries stay.
(string Map, string Expected)[] cases =
[
// Counts and flags: a key with a plain value stays only when it is a field.
("""{ "the_artist": 3, "street_style": 5, "like_count": 2 }""", "{like_count,#}"),
("""{ "max_music": true, "live_music": false }""", "{#}"),
("""{ "the_artist": null }""", "{#}"),
// Empty values and lists of codes.
("""{ "the_artist": [], "street_style": [], "carousel_media": [] }""", "{carousel_media:[],#:[]}"),
("""{ "the_artist": {}, "street_style": {} }""", "{#:{}}"),
("""{ "the_artist": [ "c1" ], "street_style": [ "c2", "c3" ] }""", "{#:[]}"),
("""{ "the_artist": [], "street_style": [ "c1" ] }""", "{#:[]}"),
("""{ "the_artist": [], "street_style": [ { "pk": "2" } ] }""", "{#:[]}"),
("""{ "the_artist": {}, "street_style": { "pk": "2" } }""", "{#:{}}"),
// Entries with different optional fields: only «pk» in common, or one field more.
("""{ "the_artist": { "pk": "1", "seen": 2 }, "street_style": { "pk": "2" }, "live_music": { "pk": "3", "is_private": true } }""", "{#:{pk,seen}}"),
("""{ "the_artist": { "seen": 1, "is_private": true }, "street_style": { "seen": 2 } }""", "{#:{seen,is_private}}"),
("""{ "the_artist": { "pk": "1", "seen": 1, "is_private": true }, "street_style": { "pk": "2", "like_count": 1, "follower_count": 3 } }""", "{#:{pk,seen,is_private}}"),
("""{ "the_artist": [ { "pk": "1", "seen": 1 } ], "street_style": [ { "pk": "2" } ] }""", "{#:[{pk,seen}]}"),
// A list of maps with one entry each.
("""[ { "the_artist": { "pk": "1" } }, { "street_style": { "pk": "2" } } ]""", "[{#:{pk}}]"),
// Not a map: the fields of a post stay, and so do the keys every element of a list has.
("""{ "user": { "pk": 1 }, "owner": { "pk": 2 }, "caption": { "text": "y" }, "image_versions2": { "candidates": [] } }""",
"{user:{pk},owner:{pk},caption:{text},image_versions2:{candidates:[]}}"),
("""[ { "pk": 1, "is_private": true }, { "pk": 2, "is_private": false, "seen": 1 } ]""", "[{pk,is_private}]"),
];
foreach (var (map, expected) in cases)
{
using var body = JsonDocument.Parse("{ \"data\": " + map + " }");
var strict = KeyStructure.DescribeStrict(body.RootElement);
Assert.AreEqual("{data:" + expected + "}", strict);
foreach (var name in new[] { "the_artist", "street_style", "max_music", "live_music" })
{
Assert.IsFalse(strict.Contains(name, StringComparison.OrdinalIgnoreCase), "A name is in the structure.");
}
}
Assert.IsTrue(StructureWords.IsFieldKey("is_private"));
Assert.IsTrue(StructureWords.IsFieldKey("user"));
Assert.IsFalse(StructureWords.IsFieldKey("the_artist"));
Assert.IsFalse(StructureWords.IsFieldKey("max"), "A field is still structural.");
}
// ---- The recorder ----
[TestMethod]
public void ListsStopAtTheirCapsAndCountWhatTheyLeaveOut()
{
var clock = new ManualClock(Start);
long? current = 5;
var diagnostics = new CollectionDiagnostics(clock, () => current);
diagnostics.SessionStarted(5, clock.Now, 1.0);
for (var i = 0; i < CollectionDiagnostics.MaxPages + 5; i++)
{
diagnostics.PageOpened(5, SitePages.Profile, ok: true);
}
for (var i = 0; i < CollectionDiagnostics.MaxUnknownShapes + 2; i++)
{
diagnostics.ReadObserved(Response($"u{i}", HostKind.Site, "/api/forma/"), new ReadResult { Shape = ReadShape.Unrecognized }, $"{{chiave{Letters(i)}}}");
}
for (var i = 0; i < CollectionDiagnostics.MaxSignals + 3; i++)
{
diagnostics.SignalRaised(new Signal(SignalLevel.Throttle, "http-429", null, clock.Now));
}
for (var i = 0; i < CollectionDiagnostics.MaxRecipes + 4; i++)
{
diagnostics.RecipeEnded(5, RecipeKind.ProfileCheck, RecipeStatus.Completed, null, 1, 0);
}
current = null;
diagnostics.SessionEnded(5, clock.Now, SessionOutcome.Completed, SessionStopCause.WorkDone);
// Outside sessions: response groups by depth of the address and resource type.
string[] types = ["Document", "Image", "Media", "Font", "Script", "XHR", "Fetch", "Ping", "Other", "Manifest"];
for (var i = 0; i < CollectionDiagnostics.MaxResponseGroups + 7; i++)
{
var path = string.Concat(Enumerable.Repeat("/x", 1 + (i / types.Length))) + "/";
diagnostics.OnResponse(Response($"g{i}", HostKind.Site, path, resourceType: types[i % types.Length]));
}
var session = diagnostics.Recent.Single();
Assert.HasCount(CollectionDiagnostics.MaxPages, session.Pages);
Assert.AreEqual(5, session.Overflow.Pages);
Assert.HasCount(CollectionDiagnostics.MaxUnknownShapes, session.UnknownShapes);
Assert.AreEqual(2, session.Overflow.UnknownShapes);
Assert.HasCount(CollectionDiagnostics.MaxSignals, session.Signals);
Assert.AreEqual(3, session.Overflow.Signals);
Assert.HasCount(CollectionDiagnostics.MaxRecipes, session.Recipes);
Assert.AreEqual(4, session.Overflow.Recipes);
Assert.AreEqual(SessionStopCause.WorkDone, session.StopCause);
// An overflowing body is counted once, not again when it is read (nor once per block of a page).
var late = Response("over", HostKind.Site, "/y/y/y/y/y/y/y/y/y/y/y/y/y/y/y/y/y/y/y/y/y/y/y/y/y/");
diagnostics.OnResponse(late);
diagnostics.OnLoadingFinished("over", 10);
diagnostics.ReadObserved(late, new ReadResult { Shape = ReadShape.Recognized }, null);
diagnostics.ReadObserved(late, ReadResult.EmptyResult, null);
var outside = diagnostics.Snapshot(clock.Now).Single();
Assert.IsNull(outside.Session);
Assert.HasCount(CollectionDiagnostics.MaxResponseGroups, outside.Responses);
Assert.AreEqual(8, outside.Overflow.Responses);
Assert.AreEqual(0, outside.Overflow.Requests);
Assert.AreEqual(0, diagnostics.Faults);
}
[TestMethod]
public void OnlyBodiesStillWaitingCountAsForgottenAndLateNewsIsCountedWhereItArrives()
{
var clock = new ManualClock(Start);
long? current = 6;
var diagnostics = new CollectionDiagnostics(clock, () => current);
diagnostics.SessionStarted(6, clock.Now, 1);
// A long session: thousands of images and data responses, each settled (size, reading or drop).
for (var i = 0; i < 6000; i++)
{
var image = Response($"img{i}", HostKind.Cdn, "/v/t51/foto.jpg", mime: "image/jpeg", resourceType: "Image");
diagnostics.OnResponse(image);
diagnostics.OnLoadingFinished(image.RequestId, 100);
if (i % 3 == 0)
{
var data = Response($"dat{i}", HostKind.Site, "/api/v1/feed/", resourceType: "XHR");
diagnostics.OnResponse(data);
diagnostics.OnLoadingFinished(data.RequestId, 50);
if (i % 2 == 0)
{
diagnostics.ReadObserved(data, ReadResult.EmptyResult, null);
}
else
{
diagnostics.OnBodyDropped(data.RequestId, BodyDropReason.NotJson);
}
}
}
// One body that never got its outcome, then enough new responses to push it out.
var waiting = Response("waiting", HostKind.Site, "/api/v1/feed/", resourceType: "XHR");
diagnostics.OnResponse(waiting);
for (var i = 0; i < 4100; i++)
{
diagnostics.OnResponse(Response($"open{i}", HostKind.Cdn, "/v/t51/video.mp4", mime: "video/mp4", resourceType: "Media"));
}
var trace = diagnostics.Snapshot(clock.Now).Single();
Assert.AreEqual(1, trace.Overflow.Requests, "Only the body still waiting; settled responses and media without a size are not losses.");
var feed = trace.Responses.Single(r => r.Pattern == "/api/v1/feed/");
Assert.AreEqual((2001, 1000, 1000), (feed.Count, feed.Empty, feed.Dropped.Single().Count));
Assert.AreEqual(600_000L, trace.Responses.Single(r => r.Pattern == "cdn *.jpg").Bytes);
// The session ends; what arrives afterwards for its responses goes to the trace open now, as late.
var last = Response("last", HostKind.Site, "/api/v1/feed/", resourceType: "XHR");
diagnostics.OnResponse(last);
current = null;
diagnostics.SessionEnded(6, clock.Now, SessionOutcome.Completed, SessionStopCause.WorkDone);
diagnostics.OnLoadingFinished("last", 70);
diagnostics.ReadObserved(last, ReadResult.EmptyResult, null);
var ended = diagnostics.Recent.Single();
Assert.AreEqual(2002, ended.Responses.Single(r => r.Pattern == "/api/v1/feed/").Count);
Assert.AreEqual(1000, ended.Responses.Single(r => r.Pattern == "/api/v1/feed/").Empty, "A closed trace is not changed.");
var outside = diagnostics.Snapshot(clock.Now).Single();
Assert.IsNull(outside.Session);
Assert.AreEqual(2, outside.Overflow.Late);
Assert.AreEqual(0, outside.Responses.Count);
Assert.AreEqual(0, diagnostics.Faults);
}
[TestMethod]
public void AFailedRecipeSaysTheTypeOfItsErrorAndAFailedDownloadItsCause()
{
var clock = new ManualClock(Start);
var diagnostics = new CollectionDiagnostics(clock, () => 8);
diagnostics.SessionStarted(8, clock.Now, 1);
diagnostics.RecipeEnded(8, RecipeKind.Highlight, RecipeStatus.Failed, null, 2, 0, nameof(TimeoutException));
diagnostics.RecipeEnded(8, RecipeKind.Highlight, RecipeStatus.Failed, null, 2, 0, "System.Collections.Generic.List`1 di marta.prova");
diagnostics.DownloadSettled(new DownloadProgress(1, 2, 3, EntryKind.Post, DownloadStep.Retried) { Cause = DownloadCauses.Of(503) });
diagnostics.DownloadSettled(new DownloadProgress(1, 2, 3, EntryKind.Post, DownloadStep.Retried) { Cause = DownloadCauses.Of(503) });
diagnostics.DownloadSettled(new DownloadProgress(1, 2, 3, EntryKind.Post, DownloadStep.Failed) { Cause = DownloadCauses.Of(null) });
diagnostics.DownloadSettled(new DownloadProgress(1, 2, 3, EntryKind.Post, DownloadStep.Stored, 10));
diagnostics.SessionEnded(8, clock.Now, SessionOutcome.Completed, SessionStopCause.WorkDone);
var trace = diagnostics.Recent.Single();
CollectionAssert.AreEqual(new[] { nameof(TimeoutException), CollectionDiagnostics.Other }, trace.Recipes.Select(r => r.Error).ToArray());
Assert.AreEqual(2, trace.Downloads.Single(d => d.Cause == "http-503").Count);
Assert.AreEqual(DownloadStep.Failed, trace.Downloads.Single(d => d.Cause == DownloadCauses.Network).Step);
Assert.IsNull(trace.Downloads.Single(d => d.Step == DownloadStep.Stored).Cause);
}
[TestMethod]
public void WhatTheCaptureSeesGoesToTheSessionInProgressOrToTheDayOutsideSessions()
{
var clock = new ManualClock(Start);
long? current = null;
var diagnostics = new CollectionDiagnostics(clock, () => current) { SiteMap = SiteMap.LoadTestSite() };
// Alberto at the browser before any session: the day outside sessions.
diagnostics.OnNavigation(new NavigationDescriptor(HostKind.Site, "http://sito/profilo/a/", "sito", "/profilo/a/", clock.Now));
current = 9;
diagnostics.SessionStarted(9, clock.Now, 0.87654);
var document = Response("d1", HostKind.Site, "/profilo/marta.prova/", mime: "text/html", resourceType: "Document", mainFrame: true);
diagnostics.OnResponse(document);
diagnostics.OnLoadingFinished("d1", 50_000);
diagnostics.ReadObserved(document, new ReadResult { Shape = ReadShape.Recognized, Contents = [] }, null);
diagnostics.ReadObserved(document, ReadResult.EmptyResult, null);
var image = Response("i1", HostKind.Cdn, "/v/t51/foto.jpg", mime: "image/jpeg", resourceType: "Image");
diagnostics.OnResponse(image);
diagnostics.OnLoadingFinished("i1", 120_000);
var data = Response("x1", HostKind.Site, "/api/dati/", resourceType: "Fetch");
diagnostics.OnResponse(data);
diagnostics.OnBodyDropped("x1", BodyDropReason.Timeout);
diagnostics.PermitDenied(null, ActionKind.ProfileOpen, DenialReason.HourlyBudgetReached);
diagnostics.FilesQueued(9, EntryKind.Post, 3, nameof(SaveOutcome.Created), skipped: 1);
diagnostics.FilesQueued(9, EntryKind.Post, 0, nameof(SaveOutcome.Known));
diagnostics.StepJudged(9, CollectionDiagnostics.VerdictRecognized);
diagnostics.TargetClicked(9, SiteTargets.ProfileStoryRing, TargetCheckStatus.Clicked);
current = null;
diagnostics.SessionEnded(9, clock.Now, SessionOutcome.Stopped, SessionStopCause.StoppedEarly, RecipeStatus.TargetNotFound);
// After the session: a download, outside sessions; a late body of the session's request goes nowhere new.
diagnostics.DownloadSettled(new DownloadProgress(1, 2, 3, EntryKind.Post, DownloadStep.Stored, 120_000));
diagnostics.DownloadSettled(new DownloadProgress(1, 2, 3, EntryKind.Post, DownloadStep.Retried));
diagnostics.ReadObserved(document, ReadResult.EmptyResult, null);
var session = diagnostics.Recent.Single();
Assert.AreEqual(9L, session.Session);
Assert.AreEqual(0.88, session.Mood);
Assert.AreEqual(SessionStopCause.StoppedEarly, session.StopCause);
Assert.AreEqual(RecipeStatus.TargetNotFound, session.StoppedBy);
Assert.IsFalse(session.InProgress);
var page = session.Responses.Single(r => r.Page == SitePages.Profile);
Assert.AreEqual("/profilo/{username}/", page.Pattern);
Assert.AreEqual((1, 50_000L, 1, 1, 0), (page.Count, page.Bytes, page.Recognized, page.Empty, page.NotRead));
var cdn = session.Responses.Single(r => r.HostKind == HostKind.Cdn);
Assert.AreEqual(("cdn *.jpg", 1, 120_000L), (cdn.Pattern, cdn.NotRead, cdn.MaxBytes));
var dropped = session.Responses.Single(r => r.Pattern == PathPatterns.Of("/api/dati/"));
Assert.AreEqual(BodyDropReason.Timeout, dropped.Dropped.Single().Reason);
Assert.AreEqual(new DenialCount(ActionKind.ProfileOpen, DenialReason.HourlyBudgetReached, 1), session.Denials.Single());
Assert.AreEqual(new QueuedFiles(EntryKind.Post, "Created", 1, 3, 1), session.FilesQueued.Single(q => q.Outcome == "Created"));
Assert.AreEqual(new TargetCount(SiteTargets.ProfileStoryRing, TargetCheckStatus.Clicked, 1), session.Targets.Single());
var outside = diagnostics.Snapshot(clock.Now).Single();
Assert.IsNull(outside.Session);
Assert.IsTrue(outside.InProgress);
Assert.AreEqual(SitePages.Profile, outside.Navigations.Single().Page);
Assert.AreEqual(2, outside.Downloads.Count);
Assert.AreEqual(120_000L, outside.DownloadedBytes);
Assert.AreEqual(0, outside.Responses.Count, "The late body is not a response of the day outside sessions.");
Assert.AreEqual(1, outside.Overflow.Late, "It is counted as late where it arrived.");
}
[TestMethod]
public void TheDayOutsideSessionsIsKeptAtTheEndOfItsLocalDayAndOpenSessionsWhenFebeStops()
{
var clock = new ManualClock(new DateTimeOffset(2026, 11, 13, 21, 0, 0, TimeSpan.Zero), TestZones.Rome);
var diagnostics = new CollectionDiagnostics(clock);
diagnostics.SignalRaised(new Signal(SignalLevel.Throttle, "http-429", "https://sito/profilo/marta.prova/", clock.Now));
clock.Now = clock.Now.AddHours(2); // 00:00 in Rome: a new local day
diagnostics.SignalRaised(new Signal(SignalLevel.Throttle, "http-429", null, clock.Now));
var yesterday = diagnostics.Recent.Single();
Assert.IsNull(yesterday.Session);
Assert.IsFalse(yesterday.InProgress);
Assert.HasCount(1, yesterday.Signals);
diagnostics.SessionStarted(12, clock.Now, 1);
diagnostics.PageOpened(12, SitePages.Home, ok: true);
diagnostics.Dispose();
diagnostics.PageOpened(12, SitePages.Home, ok: true);
var traces = diagnostics.Recent;
Assert.HasCount(3, traces);
var stopped = traces.Single(t => t.Session == 12);
Assert.IsTrue(stopped.InProgress);
Assert.AreEqual(SessionStopCause.Shutdown, stopped.StopCause);
Assert.HasCount(1, stopped.Pages, "Nothing is recorded after the flush.");
Assert.HasCount(1, traces.Single(t => t.Session is null && t.Start >= clock.Now).Signals);
}
[TestMethod]
public void NamesOutsideTheClosedListsBecomeOther()
{
var clock = new ManualClock(Start);
var diagnostics = new CollectionDiagnostics(clock, () => 3);
diagnostics.SessionStarted(3, clock.Now, 1);
diagnostics.PageOpened(3, "marta.prova", ok: false);
diagnostics.TargetClicked(3, "marta.prova", TargetCheckStatus.NotFound);
diagnostics.StepJudged(3, "marta.prova");
diagnostics.RecipeEnded(3, RecipeKind.ProfileCheck, RecipeStatus.StoppedBySignal, "https://sito/marta.prova", 0, 0);
diagnostics.FilesQueued(3, EntryKind.Post, 1, "marta.prova");
diagnostics.OnResponse(Response("m1", HostKind.Other, "/marta.prova", mime: "text/marta prova", resourceType: "marta.prova"));
diagnostics.SessionEnded(3, clock.Now, SessionOutcome.Completed, SessionStopCause.WorkDone);
var trace = diagnostics.Recent.Single();
Assert.AreEqual(CollectionDiagnostics.Other, trace.Pages.Single().Page);
Assert.IsNull(trace.Pages.Single().Pattern);
Assert.AreEqual(CollectionDiagnostics.Other, trace.Targets.Single().Target);
Assert.AreEqual(CollectionDiagnostics.Other, trace.Steps.Single().Verdict);
Assert.AreEqual(CollectionDiagnostics.Other, trace.Recipes.Single().Signal);
Assert.AreEqual(CollectionDiagnostics.Other, trace.FilesQueued.Single().Outcome);
var response = trace.Responses.Single();
Assert.AreEqual((CollectionDiagnostics.Other, "text/altro", CollectionDiagnostics.Other), (response.Pattern, response.Mime, response.ResourceType));
Assert.AreEqual("image/jpeg", CollectionDiagnostics.MimeOf("image/JPEG; charset=binary"));
Assert.AreEqual(CollectionDiagnostics.Other, CollectionDiagnostics.MimeOf("marta/prova"));
}
// ---- The log ----
[TestMethod]
public void TheLogKeepsTwoWeeksSkipsDamagedLinesAndTouchesNoOtherFile()
{
using var folder = new TempFolder("diagnostica");
var clock = new ManualClock(new DateTimeOffset(2026, 11, 13, 9, 0, 0, TimeSpan.Zero), TestZones.Rome);
var log = new DiagnosticLog(folder.Combine("diagnostica"), clock);
Directory.CreateDirectory(log.Folder);
var today = LocalCalendar.Today(clock.Now, clock.LocalZone);
File.WriteAllText(Path.Combine(log.Folder, DiagnosticLog.FileNameFor(today.AddDays(-14))), "{}\n");
File.WriteAllText(Path.Combine(log.Folder, DiagnosticLog.FileNameFor(today.AddDays(-20))), "{}\n");
var kept = Path.Combine(log.Folder, DiagnosticLog.FileNameFor(today.AddDays(-13)));
File.WriteAllText(kept, "{\"session\":1,\"start\":\"2026-10-31T09:00:00+00:00\",\"inProgress\":false}\n{\"session\":tronca");
var other = Path.Combine(log.Folder, "appunti.txt");
File.WriteAllText(other, "di Alberto");
var diagnostics = new CollectionDiagnostics(clock, log: log);
diagnostics.SessionStarted(2, clock.Now, 1.1);
diagnostics.PageOpened(2, SitePages.Home, ok: true);
diagnostics.RecipeEnded(2, RecipeKind.FollowingSweep, RecipeStatus.Completed, null, 1, 4);
clock.Now = clock.Now.AddMinutes(9);
diagnostics.SessionEnded(2, clock.Now, SessionOutcome.Completed, SessionStopCause.PlannedEnd);
var contents = log.ReadAll();
Assert.AreEqual(1, contents.DamagedLines);
CollectionAssert.AreEqual(new long?[] { 1, 2 }, contents.Traces.Select(t => t.Session).ToArray(), "Oldest first.");
var read = contents.Traces[1];
Assert.AreEqual(SessionStopCause.PlannedEnd, read.StopCause);
Assert.AreEqual(SitePages.Home, read.Pages.Single().Page);
Assert.AreEqual(RecipeKind.FollowingSweep, read.Recipes.Single().Kind);
Assert.AreEqual(clock.Now, read.End);
Assert.IsFalse(File.Exists(Path.Combine(log.Folder, DiagnosticLog.FileNameFor(today.AddDays(-14)))));
Assert.IsFalse(File.Exists(Path.Combine(log.Folder, DiagnosticLog.FileNameFor(today.AddDays(-20)))));
Assert.IsTrue(File.Exists(kept));
Assert.IsTrue(File.Exists(other), "Only its own files, by their name.");
Assert.IsTrue(File.Exists(Path.Combine(log.Folder, DiagnosticLog.FileNameFor(today))));
// Febe stopped in the middle of a line of today: the next trace starts on a line of its own.
var todayFile = Path.Combine(log.Folder, DiagnosticLog.FileNameFor(today));
File.AppendAllText(todayFile, "{\"session\":3,\"start\":\"2026-11-13T09:");
diagnostics.SessionStarted(4, clock.Now, 1);
diagnostics.SessionEnded(4, clock.Now, SessionOutcome.Completed, SessionStopCause.WorkDone);
var afterTear = log.ReadAll();
Assert.AreEqual(2, afterTear.DamagedLines, "The torn line only, besides the one of 13 days ago.");
CollectionAssert.AreEqual(new long?[] { 1, 2, 4 }, afterTear.Traces.Select(t => t.Session).ToArray());
Assert.AreEqual(2, diagnostics.Gather().DamagedLines);
// Two weeks later the day of the session is gone too.
clock.Now = clock.Now.AddDays(14);
Assert.HasCount(0, log.ReadAll().Traces);
Assert.IsFalse(diagnostics.Collect().Any(t => t.Session is not null), "Nothing left in the log, nor in memory beyond the days kept.");
}
[TestMethod]
public void ATraceWhoseWriteFailedIsStillInTheReportAndTheReportSaysSo()
{
using var folder = new TempFolder("diagnostica");
var clock = new ManualClock(new DateTimeOffset(2026, 11, 13, 9, 0, 0, TimeSpan.Zero), TestZones.Rome);
// A file where the folder of the log should be: every write fails.
var blocked = folder.Combine("diagnostica");
File.WriteAllText(blocked, "non una cartella");
var diagnostics = new CollectionDiagnostics(clock, log: new DiagnosticLog(blocked, clock));
diagnostics.SessionStarted(5, clock.Now, 1);
diagnostics.PageOpened(5, SitePages.Home, ok: true);
diagnostics.SessionEnded(5, clock.Now, SessionOutcome.Completed, SessionStopCause.WorkDone);
diagnostics.SessionStarted(6, clock.Now, 1);
var gathered = diagnostics.Gather();
CollectionAssert.AreEqual(new long?[] { 5, 6 }, gathered.Traces.Select(t => t.Session).ToArray(), "The finished one from memory, the open one in progress, once each.");
Assert.IsFalse(gathered.Traces[0].InProgress);
Assert.IsTrue(gathered.Traces[1].InProgress);
Assert.IsGreaterThan(0, gathered.RecorderFaults);
using var stream = new MemoryStream();
DiagnosticReport.Write(stream, gathered.Traces, new DiagnosticReportHeader("0.1.2", "docker", clock.Now, TestZones.Rome, DamagedLines: 3, RecorderFaults: gathered.RecorderFaults));
using var json = JsonDocument.Parse(stream.ToArray());
Assert.AreEqual(3, json.RootElement.GetProperty("righeRovinate").GetInt32());
Assert.AreEqual(gathered.RecorderFaults, json.RootElement.GetProperty("erroriRegistratore").GetInt32());
}
[TestMethod]
public void ADayFileThatCannotBeReadIsCountedAndTheOtherDaysAreStillThere()
{
using var folder = new TempFolder("diagnostica");
var clock = new ManualClock(new DateTimeOffset(2026, 11, 13, 9, 0, 0, TimeSpan.Zero), TestZones.Rome);
var log = new DiagnosticLog(folder.Combine("diagnostica"), clock);
var diagnostics = new CollectionDiagnostics(clock, log: log);
diagnostics.SessionStarted(1, clock.Now, 1);
diagnostics.SessionEnded(1, clock.Now.AddMinutes(5), SessionOutcome.Completed, SessionStopCause.WorkDone);
clock.Now = clock.Now.AddDays(1);
diagnostics.SessionStarted(2, clock.Now, 1);
diagnostics.SessionEnded(2, clock.Now.AddMinutes(5), SessionOutcome.Completed, SessionStopCause.WorkDone);
var yesterday = Path.Combine(log.Folder, DiagnosticLog.FileNameFor(LocalCalendar.Today(clock.Now, clock.LocalZone).AddDays(-1)));
// Another program (a backup, a synchronization, an antivirus) holds yesterday's file without sharing it.
DiagnosticLogContents contents;
DiagnosticCollection gathered;
using (new FileStream(yesterday, FileMode.Open, FileAccess.ReadWrite, FileShare.None))
{
contents = log.ReadAll();
gathered = new CollectionDiagnostics(clock, log: log).Gather();
}
Assert.AreEqual(1, contents.UnreadableFiles);
Assert.AreEqual(0, contents.DamagedLines);
CollectionAssert.AreEqual(new long?[] { 2 }, contents.Traces.Select(t => t.Session).ToArray(), "Today is still read.");
Assert.AreEqual(1, gathered.UnreadableFiles, "The report is told that a day is missing.");
Assert.AreEqual(0, gathered.RecorderFaults, "Not the whole log lost: one file.");
Assert.AreEqual(1, gathered.Traces.Count);
using var stream = new MemoryStream();
DiagnosticReport.Write(stream, gathered.Traces, new DiagnosticReportHeader("0.1.2", "docker", clock.Now, TestZones.Rome, UnreadableFiles: gathered.UnreadableFiles));
using var json = JsonDocument.Parse(stream.ToArray());
Assert.AreEqual(1, json.RootElement.GetProperty("fileIlleggibili").GetInt32());
Assert.AreEqual(2, log.ReadAll().Traces.Count, "Released: both days again.");
}
[TestMethod]
public void ASessionCutShortByACrashIsStillInTheReportAsASessionWithoutAnEnd()
{
using var folder = new TempFolder("diagnostica");
var clock = new ManualClock(new DateTimeOffset(2026, 11, 13, 9, 0, 0, TimeSpan.Zero), TestZones.Rome);
var folderOfLog = folder.Combine("diagnostica");
var before = new CollectionDiagnostics(clock, log: new DiagnosticLog(folderOfLog, clock));
before.SessionStarted(7, clock.Now, 1);
Assert.AreEqual(1, new DiagnosticLog(folderOfLog, clock).ReadAll().Traces.Count, "The start is in the log at once.");
before.PageOpened(7, SitePages.Home, ok: true);
clock.Now = clock.Now.AddMinutes(6);
before.RecipeEnded(7, RecipeKind.FollowingSweep, RecipeStatus.Completed, null, 2, 3);
clock.Now = clock.Now.AddMinutes(1);
before.RecipeEnded(7, RecipeKind.ProfileCheck, RecipeStatus.Completed, null, 1, 0);
// While it runs, the report has it once, as it is now (not as its last line in the log).
var running = before.Gather();
Assert.AreEqual(2, running.Traces.Single().Recipes.Count);
Assert.AreEqual(0, running.UnfinishedSessions);
// Febe is killed: no end, no flush. After the restart the session is there up to its last line in progress.
clock.Now = clock.Now.AddMinutes(30);
var after = new CollectionDiagnostics(clock, log: new DiagnosticLog(folderOfLog, clock));
var gathered = after.Gather();
var trace = gathered.Traces.Single();
Assert.AreEqual(7, trace.Session);
Assert.IsTrue(trace.InProgress);
Assert.IsNull(trace.StopCause);
Assert.AreEqual(SitePages.Home, trace.Pages.Single().Page);
Assert.AreEqual(RecipeKind.FollowingSweep, trace.Recipes.Single().Kind, "The checkpoint at the end of the first recipe; the second came too soon after it.");
Assert.AreEqual(1, gathered.UnfinishedSessions);
using var stream = new MemoryStream();
DiagnosticReport.Write(stream, gathered.Traces, new DiagnosticReportHeader("0.1.2", "docker", clock.Now, TestZones.Rome, UnfinishedSessions: gathered.UnfinishedSessions));
using var json = JsonDocument.Parse(stream.ToArray());
Assert.AreEqual(1, json.RootElement.GetProperty("sessioniSenzaFine").GetInt32());
// A session that ends, and one closed by a shutdown, are not without an end; each is in the report once.
after.SessionStarted(8, clock.Now, 1);
after.RecipeEnded(8, RecipeKind.FollowingSweep, RecipeStatus.Completed, null, 1, 0);
after.SessionEnded(8, clock.Now.AddMinutes(2), SessionOutcome.Completed, SessionStopCause.WorkDone);
after.SessionStarted(9, clock.Now.AddMinutes(3), 1);
after.Dispose();
var last = new CollectionDiagnostics(clock, log: new DiagnosticLog(folderOfLog, clock)).Gather();
CollectionAssert.AreEqual(new long?[] { 7, 8, 9 }, last.Traces.Select(t => t.Session).ToArray());
Assert.IsFalse(last.Traces[1].InProgress);
Assert.AreEqual(SessionStopCause.Shutdown, last.Traces[2].StopCause);
Assert.AreEqual(1, last.UnfinishedSessions, "Only the one of the crash.");
}
[TestMethod]
public void AnEndWhoseWriteFailedIsWrittenLaterAndTheSessionDoesNotLookCutShortAfterARestart()
{
using var folder = new TempFolder("diagnostica");
var clock = new ManualClock(new DateTimeOffset(2026, 11, 13, 9, 0, 0, TimeSpan.Zero), TestZones.Rome);
var folderOfLog = folder.Combine("diagnostica");
var before = new CollectionDiagnostics(clock, log: new DiagnosticLog(folderOfLog, clock));
var today = Path.Combine(folderOfLog, DiagnosticLog.FileNameFor(LocalCalendar.Today(clock.Now, clock.LocalZone)));
// The start is written; the end is not (another program holds the file for a moment).
before.SessionStarted(7, clock.Now, 1);
using (new FileStream(today, FileMode.Open, FileAccess.ReadWrite, FileShare.None))
{
before.SessionEnded(7, clock.Now.AddMinutes(5), SessionOutcome.Completed, SessionStopCause.WorkDone);
}
Assert.AreEqual(1, before.Faults, "The failed write is counted.");
Assert.IsTrue(new DiagnosticLog(folderOfLog, clock).ReadAll().Traces.Single().InProgress, "Only the start is in the log so far.");
// The next line written brings the end along.
before.SessionStarted(8, clock.Now.AddMinutes(10), 1);
before.SessionEnded(8, clock.Now.AddMinutes(15), SessionOutcome.Completed, SessionStopCause.WorkDone);
// The same with the report: gathering it writes what is still waiting.
before.SessionStarted(9, clock.Now.AddMinutes(20), 1);
using (new FileStream(today, FileMode.Open, FileAccess.ReadWrite, FileShare.None))
{
before.SessionEnded(9, clock.Now.AddMinutes(25), SessionOutcome.Completed, SessionStopCause.WorkDone);
}
Assert.AreEqual(2, before.Faults);
Assert.AreEqual(0, before.Gather().UnfinishedSessions);
// Febe is killed and restarted: every session ended, none without an end.
var after = new CollectionDiagnostics(clock, log: new DiagnosticLog(folderOfLog, clock)).Gather();
CollectionAssert.AreEqual(new long?[] { 7, 8, 9 }, after.Traces.Select(t => t.Session).ToArray());
Assert.IsTrue(after.Traces.All(t => !t.InProgress && t.StopCause == SessionStopCause.WorkDone));
Assert.AreEqual(0, after.UnfinishedSessions);
Assert.AreEqual(2, before.Faults, "Nothing failed again.");
}
[TestMethod]
public void ASignalReportedStraightToTheBreakerIsInTheTraceOnceEvenWhenTheDetectorSaidItToo()
{
var clock = new ManualClock(Start);
long? current = 3;
var diagnostics = new CollectionDiagnostics(clock, () => current);
diagnostics.SessionStarted(3, clock.Now, 1);
// The session cookie lost: the connection tells the breaker, never the detector.
var lost = new Signal(SignalLevel.Disconnect, SignalRules.SessionCookieMissingCode, "https://sito.esempio/accedi/", clock.Now);
diagnostics.BreakerChanged(new BreakerState(BreakerStatus.Disconnected, null, lost, null));
// A throttle the detector saw: the breaker changes first, then the detector raises it; once in the trace.
clock.Now = clock.Now.AddMinutes(1);
var throttle = new Signal(SignalLevel.Throttle, "http-429", null, clock.Now);
diagnostics.BreakerChanged(new BreakerState(BreakerStatus.Throttled, clock.Now.AddHours(1), throttle, null));
diagnostics.SignalRaised(throttle);
// The breaker reopening is not a signal.
diagnostics.BreakerChanged(new BreakerState(BreakerStatus.Closed, null, throttle, null));
diagnostics.SessionEnded(3, clock.Now, SessionOutcome.Throttled, SessionStopCause.BreakerOpen);
var trace = diagnostics.Recent.Single();
CollectionAssert.AreEqual(
new[] { SignalRules.SessionCookieMissingCode, "http-429" },
trace.Signals.Select(s => s.Code).ToArray());
Assert.AreEqual(SignalLevel.Disconnect, trace.Signals[0].Level);
}
// ---- The exported file ----
[TestMethod]
public void TheReportIsIndentedJsonWithoutBomWithItsHeadAndLocalTimes()
{
var clock = new ManualClock(new DateTimeOffset(2026, 11, 13, 9, 30, 0, TimeSpan.Zero), TestZones.Rome);
var diagnostics = new CollectionDiagnostics(clock);
diagnostics.SessionStarted(4, clock.Now, 1);
diagnostics.SessionEnded(4, clock.Now.AddMinutes(5), SessionOutcome.Throttled, SessionStopCause.BreakerOpen);
diagnostics.SignalRaised(new Signal(SignalLevel.Challenge, "challenge-url", "https://sito/challenge/", clock.Now));
using var stream = new MemoryStream();
DiagnosticReport.Write(stream, diagnostics.Collect(), new DiagnosticReportHeader("0.1.2", "windows", clock.Now, TestZones.Rome));
var bytes = stream.ToArray();
Assert.AreEqual((byte)'{', bytes[0], "No BOM.");
var text = Encoding.UTF8.GetString(bytes);
StringAssert.Contains(text, "\n \"rapporto\": \"diagnostica della raccolta\"", "Indented.");
using var json = JsonDocument.Parse(text);
var root = json.RootElement;
Assert.AreEqual("0.1.2", root.GetProperty("versione").GetString());
Assert.AreEqual("windows", root.GetProperty("edizione").GetString());
Assert.AreEqual("2026-11-13T10:30:00+01:00", root.GetProperty("creato").GetString());
Assert.AreEqual(DiagnosticReport.ZoneName(TestZones.Rome), root.GetProperty("fuso").GetString());
Assert.AreNotEqual(CollectionDiagnostics.Other, root.GetProperty("fuso").GetString());
Assert.AreEqual(14, root.GetProperty("giorni").GetInt32());
Assert.AreEqual(0, root.GetProperty("righeRovinate").GetInt32());
Assert.AreEqual(0, root.GetProperty("erroriRegistratore").GetInt32());
var session = root.GetProperty("sessioni").EnumerateArray().Single();
Assert.AreEqual(4, session.GetProperty("session").GetInt64());
Assert.AreEqual("BreakerOpen", session.GetProperty("stopCause").GetString());
Assert.AreEqual("Throttled", session.GetProperty("outcome").GetString());
Assert.AreEqual("2026-11-13T10:30:00+01:00", session.GetProperty("start").GetString(), "Times in Alberto's zone.");
var outside = root.GetProperty("fuoriSessione").EnumerateArray().Single();
Assert.IsFalse(outside.TryGetProperty("session", out _));
Assert.AreEqual("challenge-url", outside.GetProperty("signals")[0].GetProperty("code").GetString());
Assert.AreEqual("febe-diagnostica-2026-11-13.json", DiagnosticReport.FileName(new DateTimeOffset(2026, 11, 13, 22, 59, 0, TimeSpan.Zero), TestZones.Rome));
Assert.AreEqual("febe-diagnostica-2026-11-14.json", DiagnosticReport.FileName(new DateTimeOffset(2026, 11, 13, 23, 0, 0, TimeSpan.Zero), TestZones.Rome), "The local day.");
}
[TestMethod]
public void TheZoneOfTheWindowsEditionIsWrittenToo()
{
// The clock of Febe for Windows has TimeZoneInfo.Local, whose name there is a Windows one, with spaces.
var windows = TimeZoneInfo.CreateCustomTimeZone("W. Europe Standard Time", TimeSpan.FromHours(1), "W. Europe Standard Time", "W. Europe Standard Time");
using var stream = new MemoryStream();
DiagnosticReport.Write(stream, [], new DiagnosticReportHeader("0.1.2", "windows", Start, windows));
using var json = JsonDocument.Parse(stream.ToArray());
var zone = json.RootElement.GetProperty("fuso").GetString();
Assert.AreNotEqual(CollectionDiagnostics.Other, zone);
Assert.IsTrue(zone is "W._Europe_Standard_Time" || zone!.StartsWith("Europe/", StringComparison.Ordinal), zone);
}
private static ResponseDescriptor Response(string id, HostKind host, string path, string mime = "application/json", string? resourceType = "Fetch", bool mainFrame = false, int status = 200) =>
new(id, host, "https://indirizzo.non.usato" + path, "indirizzo.non.usato", path, status, mime, resourceType, mainFrame, Start);
private static string Letters(int i)
{
var sb = new StringBuilder();
do
{
sb.Append((char)('a' + (i % 26)));
i /= 26;
}
while (i > 0);
return sb.ToString();
}
}
@@ -122,9 +122,14 @@ public sealed class DownloadQueueTests
var fetcher = ScriptedFetcher.Network("net::ERR_INTERNET_DISCONNECTED");
var queue = Queue(library, temp, fetcher, options: new DownloadOptions { StagingFolder = temp.Combine("dati", "temporanei"), MaxConcurrent = 1 });
var steps = new List<DownloadProgress>();
queue.Progress += (_, step) => steps.Add(step);
var report = await queue.RunOnceAsync(CancellationToken.None);
Assert.AreEqual(1, fetcher.Calls, "Every file would fail the same way: the pass ends.");
Assert.IsNotNull(report.PausedUntil);
var waiting = steps.Single();
Assert.AreEqual(DownloadStep.Retried, waiting.Step, "The diagnostic report says why the file is still waiting (issue #39).");
Assert.AreEqual(DownloadCauses.Connection, waiting.Cause);
foreach (var id in entries)
{
Assert.AreEqual(0, (await library.Db.Media.ListByEntryAsync(id)).Single().Attempts, "The router restarting is not the file's fault.");
@@ -8,6 +8,7 @@ using Febe.Browser.Recipes;
using Febe.Browser.Targets;
using Febe.Core.Abstractions;
using Febe.Core.Collection;
using Febe.Core.Collection.Diagnostics;
using Febe.Core.Collection.Downloads;
using Febe.Core.Collection.Readers;
using Febe.Core.Collection.Recipes;
@@ -127,13 +128,21 @@ internal sealed class CollectionHarness : IAsyncDisposable
/// <summary>The engine session the recipes run in (a real row: events refer to it).</summary>
public TestSession Session { get; private set; } = new();
public static async Task<CollectionHarness> CreateAsync(SharedChromium chromium, RecipeOptions? options = null, double factor = 10)
/// <summary>The recorder of the diagnostic report, when asked for: what the page does goes to <see cref="Session"/>.</summary>
public CollectionDiagnostics? Diagnostics { get; private set; }
public static async Task<CollectionHarness> CreateAsync(SharedChromium chromium, RecipeOptions? options = null, double factor = 10, bool diagnostics = false)
{
var site = await SyntheticSite.StartAsync();
var temp = new TempFolder("raccolta");
var library = await TestLibrary.CreateAsync(temp);
var clock = new CompressedClock(factor, TestZones.Rome);
var harness = new CollectionHarness(chromium, site, temp, library, clock);
if (diagnostics)
{
harness.Diagnostics = new CollectionDiagnostics(clock, () => harness.Session.SessionId);
}
harness.Pipeline = await CollectionPipeline.StartAsync(new CollectionPipelineOptions
{
Cdp = chromium.Cdp,
@@ -147,6 +156,7 @@ internal sealed class CollectionHarness : IAsyncDisposable
Random = new SeededRandom(20261013),
StagingFolder = temp.Combine("dati", "temporanei"),
Recipes = options ?? new RecipeOptions(),
Diagnostics = harness.Diagnostics,
});
harness.Session = new TestSession((await library.Db.Sessions.StartAsync(clock.Now)).Id);
await harness.SignInAsync();
@@ -0,0 +1,209 @@
using Febe.Core.Abstractions;
using Febe.Core.Collection;
using Febe.Core.Collection.Diagnostics;
using Febe.Core.Domain;
using Febe.Core.Engine;
using Febe.Tests.Engine.Support;
namespace Febe.Tests.Engine;
/// <summary>Why each session stopped, as the engine tells the diagnostic report (issue #39).</summary>
[TestClass]
public sealed class SessionStopCauseTests
{
private static readonly DateTimeOffset Morning = EngineTestBed.Local(2026, 10, 14, 6, 0);
[TestMethod]
public async Task SessionsThatRunTheirCourseEndByTimeByWorkOrByBudgets()
{
await using var bed = await EngineTestBed.CreateAsync(Morning);
var diagnostics = Record(bed);
await bed.AddAccountAsync("uno.demo", followed: false, Frequency.High);
await bed.AddAccountAsync("due.demo", followed: true);
bed.Start(new ScriptedRunner(bed.Governor, bed.Clock));
await bed.RunUntilAsync(Morning.AddDays(1));
var sessions = await bed.Db.Sessions.ListAsync(Morning, bed.Clock.Now);
var traces = diagnostics.Recent.Where(t => t.Session is not null).ToList();
Assert.IsTrue(traces.Count > 5, $"{traces.Count} traces");
CollectionAssert.AreEquivalent(
sessions.Where(s => s.EndedAt is not null).Select(s => (long?)s.Id).ToList(),
traces.Select(t => t.Session).ToList(),
"One trace for every finished session.");
foreach (var trace in traces)
{
Assert.AreEqual(SessionOutcome.Completed, trace.Outcome);
Assert.IsTrue(trace.StopCause is SessionStopCause.PlannedEnd or SessionStopCause.WorkDone or SessionStopCause.BudgetsExhausted, trace.StopCause.ToString());
Assert.IsNull(trace.StoppedBy);
Assert.IsFalse(trace.InProgress);
Assert.IsTrue(trace.End >= trace.Start);
Assert.IsTrue(trace.Recipes.Count == 0, "The scripted runner bypasses the recipe runner.");
}
Assert.AreEqual(0, diagnostics.Faults);
}
[TestMethod]
public async Task AThrottleEndsTheSessionWithTheBreakerOpen()
{
await using var bed = await EngineTestBed.CreateAsync(Morning);
var diagnostics = Record(bed);
await bed.AddAccountAsync("uno.demo", followed: false, Frequency.High);
var runner = new ScriptedRunner(bed.Governor, bed.Clock);
var throttled = false;
runner.AfterGrant = r =>
{
if (!throttled && r.Kind == RecipeKind.ProfileCheck)
{
throttled = true;
bed.Breaker.Report(new Signal(SignalLevel.Throttle, "http-429", null, bed.Clock.Now));
}
};
bed.Start(runner);
await bed.RunUntilAsync(Morning.AddHours(12));
Assert.IsTrue(throttled);
var trace = diagnostics.Recent.Single(t => t.Outcome == SessionOutcome.Throttled);
Assert.AreEqual(SessionStopCause.BreakerOpen, trace.StopCause);
Assert.AreEqual("http-429", trace.Signals.Single().Code, "The signal that opened the breaker, reported straight to it.");
}
[TestMethod]
public async Task ALogoutDuringASessionSaysItsCodeInTheTrace()
{
// The connection finds the session cookie gone and tells the breaker directly: the detector never sees it.
var session = default(long?);
var trace = await StopHeldSessionTraceAsync(async (bed, _, _) =>
{
session = bed.Runtime.CurrentSession?.SessionId;
bed.Breaker.Report(new Signal(SignalLevel.Disconnect, SignalRules.SessionCookieMissingCode, null, bed.Clock.Now));
// The engine closes the session from a task of its own (the breaker's event may come from a capture thread).
var waited = System.Diagnostics.Stopwatch.StartNew();
while (bed.Runtime.CurrentSession is not null && waited.Elapsed < TimeSpan.FromSeconds(10))
{
await Task.Delay(10);
}
});
Assert.IsNotNull(session);
Assert.AreEqual(SessionStopCause.BreakerOpen, trace.StopCause);
var signal = trace.Signals.Single();
Assert.AreEqual(SignalLevel.Disconnect, signal.Level);
Assert.AreEqual(SignalRules.SessionCookieMissingCode, signal.Code, "A logout does not look like any other stop.");
}
[TestMethod]
public async Task AMissingTargetStopsTheSessionEarlyAndSaysWhichStatus()
{
await using var bed = await EngineTestBed.CreateAsync(Morning);
var diagnostics = Record(bed);
await bed.AddAccountAsync("uno.demo", followed: false, Frequency.High);
var runner = new ScriptedRunner(bed.Governor, bed.Clock);
var missed = false;
runner.Script = r =>
{
if (!missed && r.Kind == RecipeKind.ProfileCheck)
{
missed = true;
return new RecipeOutcome(RecipeStatus.TargetNotFound, 1, 0);
}
return null;
};
bed.Start(runner);
await bed.RunUntilAsync(Morning.AddHours(12));
Assert.IsTrue(missed);
var trace = diagnostics.Recent.Single(t => t.StoppedBy is not null);
Assert.AreEqual(SessionStopCause.StoppedEarly, trace.StopCause);
Assert.AreEqual(RecipeStatus.TargetNotFound, trace.StoppedBy);
}
[TestMethod]
public async Task ThePauseAlbertoAtTheBrowserAndTheShutdownEndTheSessionInProgress()
{
Assert.AreEqual(SessionStopCause.Paused, await StopHeldSessionAsync((bed, host, _) => host.PauseAsync()));
Assert.AreEqual(SessionStopCause.UserDriving, await StopHeldSessionAsync((bed, host, drive) =>
{
drive();
host.NotifyUserDrivingChanged();
return Task.CompletedTask;
}));
Assert.AreEqual(SessionStopCause.Shutdown, await StopHeldSessionAsync((bed, _, _) => bed.StopAsync()));
}
[TestMethod]
public async Task AShutdownDuringARealRecipeIsAShutdownToo()
{
// The real recipe runner does not let the cancellation out: it answers «Cancelled», and the composer ends as
// if a recipe had stopped the session. The pause still says «Paused».
Assert.AreEqual(SessionStopCause.Shutdown, await StopHeldSessionAsync((bed, _, _) => bed.StopAsync(), answersCancelled: true));
Assert.AreEqual(SessionStopCause.Paused, await StopHeldSessionAsync((bed, host, _) => host.PauseAsync(), answersCancelled: true));
}
/// <summary>Starts a session whose first recipe waits an hour, stops it with <paramref name="stop"/>, returns its cause.</summary>
private static async Task<SessionStopCause?> StopHeldSessionAsync(Func<EngineTestBed, EngineHost, Action, Task> stop, bool answersCancelled = false) =>
(await StopHeldSessionTraceAsync(stop, answersCancelled)).StopCause;
/// <summary>The same as <see cref="StopHeldSessionAsync"/>, returning the whole trace.</summary>
private static async Task<SessionTrace> StopHeldSessionTraceAsync(Func<EngineTestBed, EngineHost, Action, Task> stop, bool answersCancelled = false)
{
var driving = false;
await using var bed = await EngineTestBed.CreateAsync(EngineTestBed.Noon, userIsDriving: () => driving);
var diagnostics = Record(bed);
await bed.AddAccountAsync("uno.demo", followed: true);
var runner = new HoldingRunner(new ScriptedRunner(bed.Governor, bed.Clock), bed.Clock, answersCancelled);
var host = bed.Start(runner);
await bed.RunForAsync(TimeSpan.FromMinutes(20));
Assert.IsTrue(runner.Holding, "A session is in progress.");
var session = bed.Runtime.CurrentSession?.SessionId;
Assert.IsNotNull(session);
await stop(bed, host, () => driving = true);
await bed.RunForAsync(TimeSpan.FromMinutes(1));
await bed.StopAsync();
var trace = diagnostics.Recent.Single(t => t.Session == session);
Assert.IsFalse(trace.InProgress);
Assert.IsTrue(trace.End - trace.Start < TimeSpan.FromMinutes(30), "It ended when it was stopped, not after the hour.");
return trace;
}
private static CollectionDiagnostics Record(EngineTestBed bed)
{
var diagnostics = new CollectionDiagnostics(bed.Clock, () => bed.Runtime.CurrentSession?.SessionId);
bed.Services.Diagnostics = diagnostics;
return diagnostics;
}
/// <summary>
/// The first recipe waits an hour of virtual time (until cancelled); the others run as scripted. With
/// <paramref name="answersCancelled"/> a cancellation is answered «Cancelled», as <c>RecipeRunner</c> does.
/// </summary>
private sealed class HoldingRunner(ScriptedRunner inner, IClock clock, bool answersCancelled = false) : IRecipeRunner
{
private int _held;
public bool Holding { get; private set; }
public async Task<RecipeOutcome> RunAsync(RecipeRequest request, ISessionContext session, CancellationToken ct)
{
if (Interlocked.Exchange(ref _held, 1) == 0)
{
Holding = true;
try
{
await clock.Delay(TimeSpan.FromHours(1), ct);
}
catch (OperationCanceledException) when (answersCancelled)
{
return new RecipeOutcome(RecipeStatus.Cancelled, 0, 0, "Ricetta interrotta.");
}
}
return await inner.RunAsync(request, session, ct);
}
public Task RestAsync(CancellationToken ct) => inner.RestAsync(ct);
}
}
+56
View File
@@ -1,7 +1,12 @@
using System.Net;
using System.Text.Json;
using Febe.Core.Collection;
using Febe.Core.Engine;
using Febe.Sonda;
using Febe.Tests.Collection;
using Febe.Tests.Support;
using Febe.Tests.Web.Support;
using Febe.Web.Api;
using Febe.Web.Demo;
namespace Febe.Tests.Web;
@@ -129,6 +134,57 @@ public sealed class DemoPart2Tests
Assert.IsTrue(normal.All(i => !i.Bool("open")));
}
[TestMethod]
public async Task TheDiagnosticReportFollowsTheSceneAndNamesNobody()
{
var finished = s_demo!.World.Today.Count;
Assert.IsGreaterThan(0, finished, "the demo day has sessions");
var names = (await s_demo.Storage.Accounts.ListAsync()).Select(a => a.Username).ToList();
names.AddRange(["marta.demo", "giulia.corre.demo", "tuo.profilo"]);
var vocabulary = ReportLeaks.Vocabulary(SiteMap.LoadInstagram());
var sensitive = names.SelectMany(SensitiveTokens.Tokenize).Where(t => !vocabulary.Contains(t)).ToHashSet(StringComparer.Ordinal);
Assert.IsGreaterThan(20, sensitive.Count);
async Task<JsonElement> ReportAsync(string scene)
{
using var response = await Client.GetAsync($"{ActivityApi.DiagnosticsPath}?scena={scene}");
var text = await response.Content.ReadAsStringAsync();
Assert.AreEqual(HttpStatusCode.OK, response.StatusCode, $"{scene}: {text}");
Assert.AreEqual("attachment", response.Content.Headers.ContentDisposition?.DispositionType);
Assert.AreEqual("no-store", response.Headers.CacheControl?.ToString());
ReportLeaks.AssertClean(text, names, sensitive, ReportLeaks.FieldKeys([]));
return JsonDocument.Parse(text).RootElement.Clone();
}
var normal = await ReportAsync("normale");
Assert.AreEqual("windows", normal.Str("edizione"), "the demo looks like the desktop by default");
var sessions = normal.Items("sessioni").ToList();
Assert.HasCount(finished, sessions);
Assert.IsTrue(sessions.All(s => s.Str("outcome") == nameof(Febe.Core.Domain.SessionOutcome.Completed) && !s.Bool("inProgress")));
Assert.IsTrue(sessions.All(s => s.Items("responses").Any() && s.Items("recipes").Any()));
Assert.AreEqual(0, sessions.Sum(s => s.Items("signals").Count()));
Assert.HasCount(1, normal.Items("fuoriSessione").ToList());
var running = (await ReportAsync("sessione")).Items("sessioni").ToList();
Assert.HasCount(finished + 1, running);
Assert.IsTrue(running[^1].Bool("inProgress"), "the session in progress comes last");
var stopped = (await ReportAsync("fermo")).Items("sessioni").ToList();
Assert.HasCount(finished, stopped);
Assert.AreEqual(nameof(SessionStopCause.BreakerOpen), stopped[^1].Str("stopCause"));
Assert.AreEqual(DemoDiagnosticSource.ChallengeCode, stopped[^1].Items("signals").Single().Str("code"));
Assert.IsTrue(stopped.Take(stopped.Count - 1).All(s => s.Str("stopCause") != nameof(SessionStopCause.BreakerOpen)));
foreach (var scene in new[] { "scollegato", "errore" })
{
var empty = await ReportAsync(scene);
Assert.AreEqual(0, empty.GetProperty("sessioni").GetArrayLength(), scene);
Assert.AreEqual(0, empty.GetProperty("fuoriSessione").GetArrayLength(), scene);
}
Assert.AreEqual("docker", (await ReportAsync("normale&veste=docker")).Str("edizione"));
}
[TestMethod]
public async Task EverySettingsPageHasItsData()
{
@@ -0,0 +1,139 @@
using System.Globalization;
using System.Net;
using System.Text;
using System.Text.Json;
using Febe.Core.Abstractions;
using Febe.Core.Collection;
using Febe.Core.Collection.Diagnostics;
using Febe.Core.Domain;
using Febe.Core.Engine;
using Febe.Tests.Web.Part2Support;
using Febe.Tests.Web.Support;
using Febe.Web;
using Febe.Web.Api;
using Febe.Web.Connection;
using Febe.Web.Engine;
using Microsoft.Extensions.DependencyInjection;
namespace Febe.Tests.Web;
/// <summary>
/// «Esporta il rapporto di diagnostica» in Attività (issue #39, spec §7.9): a JSON attachment named after the local day,
/// never cached, with the traces of the recorder and what is missing from them; empty without one; answered also when
/// a configuration problem keeps the engine off (empty: nothing ran).
/// </summary>
[TestClass]
public sealed class DiagnosticReportApiTests
{
[TestMethod]
public async Task TheReportIsAJsonAttachmentOfTheLocalDayNeverCached()
{
var source = new FixedDiagnostics();
await using var server = await Part2.StartAsync(services: s => s.AddSingleton<IDiagnosticSource>(source));
source.Traces = Traces(server.Clock.Now);
source.DamagedLines = 2;
source.Faults = 1;
using var response = await server.Client.GetAsync(ActivityApi.DiagnosticsPath);
var bytes = await response.Content.ReadAsByteArrayAsync();
Assert.AreEqual(HttpStatusCode.OK, response.StatusCode, Encoding.UTF8.GetString(bytes));
Assert.AreEqual("application/json", response.Content.Headers.ContentType?.MediaType);
Assert.AreEqual("utf-8", response.Content.Headers.ContentType?.CharSet);
Assert.AreEqual("no-store", response.Headers.CacheControl?.ToString());
var disposition = response.Content.Headers.ContentDisposition;
Assert.IsNotNull(disposition);
Assert.AreEqual("attachment", disposition.DispositionType);
var day = LocalCalendar.Today(server.Clock.Now, server.Clock.LocalZone).ToString("yyyy-MM-dd", CultureInfo.InvariantCulture);
Assert.AreEqual($"febe-diagnostica-{day}.json", disposition.FileName?.Trim('"'));
Assert.AreEqual((byte)'{', bytes[0], "UTF-8 without BOM");
using var report = JsonDocument.Parse(bytes);
var root = report.RootElement;
Assert.AreEqual(DiagnosticReport.Title, root.GetProperty("rapporto").GetString());
Assert.AreEqual(server.App.Services.GetRequiredService<FebeWebOptions>().Version, root.GetProperty("versione").GetString());
Assert.AreEqual("docker", root.GetProperty("edizione").GetString());
Assert.AreEqual(DiagnosticReport.ZoneName(server.Clock.LocalZone), root.GetProperty("fuso").GetString());
Assert.AreEqual(14, root.GetProperty("giorni").GetInt32());
Assert.AreEqual(2, root.GetProperty("righeRovinate").GetInt32(), "The report says what it misses.");
Assert.AreEqual(1, root.GetProperty("erroriRegistratore").GetInt32());
var session = root.GetProperty("sessioni").EnumerateArray().Single();
Assert.AreEqual(7, session.GetProperty("session").GetInt64());
Assert.AreEqual(nameof(SessionStopCause.BreakerOpen), session.GetProperty("stopCause").GetString());
Assert.AreEqual(1, root.GetProperty("fuoriSessione").GetArrayLength());
Assert.AreEqual(1, source.Calls);
}
[TestMethod]
public async Task TheWindowsLookSaysWindows()
{
await using var server = await Part2.StartAsync(FebeEdition.Desktop);
using var response = await server.Client.GetAsync(ActivityApi.DiagnosticsPath);
using var report = JsonDocument.Parse(await response.Content.ReadAsStringAsync());
Assert.AreEqual("windows", report.RootElement.GetProperty("edizione").GetString());
}
[TestMethod]
public async Task WithoutARecorderTheReportIsThereAndEmpty()
{
await using var server = await Part2.StartAsync();
using var response = await server.Client.GetAsync(ActivityApi.DiagnosticsPath);
Assert.AreEqual(HttpStatusCode.OK, response.StatusCode);
using var report = JsonDocument.Parse(await response.Content.ReadAsStringAsync());
Assert.AreEqual(0, report.RootElement.GetProperty("sessioni").GetArrayLength());
Assert.AreEqual(0, report.RootElement.GetProperty("fuoriSessione").GetArrayLength());
}
[TestMethod]
public async Task TheReportIsAnsweredEvenWithAConfigurationProblem()
{
CollectionAssert.Contains(ConfigurationGateMiddleware.AllowedApi.ToList(), ActivityApi.DiagnosticsPath);
var problem = new ConfigurationProblem("password-mancante", "Manca la password dell'interfaccia", "Spiegazione.", "Rimedio.");
await using var server = await Part2.StartAsync(services: s => s.AddSingleton<IConfigurationProblems>(new FixedProblems(problem)));
var (blocked, _, _) = await server.GetAsync("/api/duplicates/count");
Assert.AreEqual(HttpStatusCode.ServiceUnavailable, blocked, "the gate is closed for the rest");
var (status, text, _) = await server.GetAsync(ActivityApi.DiagnosticsPath);
Assert.AreEqual(HttpStatusCode.OK, status, text);
using var report = JsonDocument.Parse(text);
Assert.AreEqual(DiagnosticReport.Title, report.RootElement.GetProperty("rapporto").GetString());
Assert.AreEqual(0, report.RootElement.GetProperty("sessioni").GetArrayLength(), "The engine never started.");
}
private static List<SessionTrace> Traces(DateTimeOffset now) =>
[
new SessionTrace
{
Session = 7,
Start = now - TimeSpan.FromMinutes(40),
End = now - TimeSpan.FromMinutes(30),
Outcome = SessionOutcome.Stopped,
StopCause = SessionStopCause.BreakerOpen,
Mood = 0.9,
Pages = [new PageVisit(now - TimeSpan.FromMinutes(39), SitePages.Home, "/", true)],
Signals = [new SignalLine(now - TimeSpan.FromMinutes(30), SignalLevel.Challenge, "challenge-url")],
},
new SessionTrace { Session = null, Start = now - TimeSpan.FromMinutes(20), InProgress = true },
];
private sealed class FixedDiagnostics : IDiagnosticSource
{
public IReadOnlyList<SessionTrace> Traces { get; set; } = [];
public int Calls { get; private set; }
public int DamagedLines { get; set; }
public int Faults { get; set; }
public Task<IReadOnlyList<SessionTrace>> ListAsync(CancellationToken ct)
{
Calls++;
return Task.FromResult(Traces);
}
public Task<DiagnosticCollection> GatherAsync(CancellationToken ct)
{
Calls++;
return Task.FromResult(new DiagnosticCollection(Traces, DamagedLines, Faults));
}
}
}
+4 -34
View File
@@ -4,6 +4,7 @@ using System.Text.Json.Nodes;
using System.Text.RegularExpressions;
using Febe.Core.Abstractions;
using Febe.Core.Collection;
using Febe.Core.Collection.Privacy;
namespace Febe.Sonda;
@@ -49,22 +50,6 @@ public static partial class RequestShapes
/// <summary>The body field that describes a request body that is neither a form nor a JSON object.</summary>
public const string WholeBody = "(corpo)";
/// <summary>
/// Path segments kept as they are: words of the site's addresses, the same for everyone. Anything else (user names,
/// identifiers, codes) becomes <c>{numero}</c> or <c>{testo}</c>. A missing word only costs detail, never privacy.
/// </summary>
private static readonly HashSet<string> KnownSegments = new(StringComparer.Ordinal)
{
"api", "v1", "graphql", "graphql_www", "query", "ajax", "bulk-route-definitions", "navigation", "web", "users",
"user", "web_profile_info", "usernameinfo", "info", "feed", "timeline", "reels_media", "reels_tray", "highlights",
"highlights_tray", "friendships", "following", "followers", "show", "show_many", "search", "topsearch", "fbsearch",
"web_search", "typeahead", "accounts_typeahead", "top_serp", "media", "comments", "usertags", "clips", "stories",
"story", "tags", "sections", "discover", "explore", "chaining", "profile", "posts", "reels", "tagged", "saved",
"qp", "batch_fetch_web", "logging_client_events", "direct_v2", "inbox", "has_new", "get_presence", "wwwgraphql",
"ig", "bz", "web_info", "current_user", "user_info", "locations", "seen", "media_seen", "reel", "p", "accounts",
"edit", "activity", "news", "inbox_v2", "notifications", "badge", "home", "graphql_query",
};
/// <summary>Request headers whose value is the same for everyone and is kept; any other value is removed.</summary>
private static readonly HashSet<string> KnownHeaderValues = new(StringComparer.OrdinalIgnoreCase)
{
@@ -199,24 +184,9 @@ public static partial class RequestShapes
return new RequestShapesReport(count, described, shapes.Count, names.Removed) { Truncated = truncated, DroppedEntry = droppedEntry };
}
/// <summary>The path with each segment that is not a known word replaced by <c>{numero}</c> or <c>{testo}</c>.</summary>
public static string PathPattern(string path)
{
ArgumentNullException.ThrowIfNull(path);
var segments = path.Split('/');
for (var i = 0; i < segments.Length; i++)
{
var segment = segments[i];
if (segment.Length == 0 || KnownSegments.Contains(segment))
{
continue;
}
segments[i] = Digits().IsMatch(segment) ? "{numero}" : "{testo}";
}
return string.Join('/', segments);
}
/// <summary>The path with each segment that is not a known word replaced by <c>{numero}</c> or <c>{testo}</c>
/// (<see cref="PathPatterns.Of"/>, shared with the diagnostic report).</summary>
public static string PathPattern(string path) => PathPatterns.Of(path);
/// <summary>
/// The kind of the value of <paramref name="name"/>, never the value: «numero», «vuoto» or «testo». Kept as they are:
+2 -3
View File
@@ -72,9 +72,8 @@ public sealed class SensitiveTokens
}
}
/// <summary>True when the key is data rather than structure (sanitizer replaces it).</summary>
public static bool IsDataKey(string key) =>
KeyStructure.LooksLikeIdentifier(key) || key.Any(c => !(char.IsAsciiLetterOrDigit(c) || c is '_' or '-'));
/// <summary>True when the key is data rather than structure (sanitizer replaces it): <see cref="KeyStructure.IsDataKey"/>.</summary>
public static bool IsDataKey(string key) => KeyStructure.IsDataKey(key);
private void Walk(JsonElement element, string? key, int depth)
{