Giunto per le richieste dirette, senza cambiare comportamento (#36, S1)
verifica / controlli (push) Successful in 17s
verifica / prove (push) Successful in 18m22s
verifica / installatore (push) Successful in 4m43s
verifica / immagine (push) Successful in 17m20s

Le risposte catturate dal browser e quelle che il client del sito chiederà
devono arrivare ai lettori e ai segnali nello stesso modo: la consegna passa
in ResponseDelivery (Core), che ResponseTap usa con gli stessi conteggi e
messaggi. Arrivano i contratti SiteRequest, SiteAnswer e ISiteRequester con
SiteRequestGuard (un permesso concesso, della categoria giusta, per una sola
richiesta) e RequestCatalog, che legge richieste.<sito>.json e tiene nel
codice i divieti: nomi con Mutation, chat, telemetria, /ajax/, /ws/, percorsi
con segmenti strani. Nessun doc_id nel repository; nessuna operazione di
Instagram è ancora configurata perché le forme della sonda non dicono da dove
vengano alcuni campi (tappa S4). ScriptedRequester risponde dai campioni
ripuliti per le prove. Nulla di nuovo è collegato al motore.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Alby96andClaude Opus 5.5 committed 2026-11-16 12:59:32 +01:00
1 parent d4675a5e21
commit 76dd1b6032
15 files changed
+3488 -68

No files matched your search

+4
View File
@@ -357,3 +357,7 @@ Ogni scelta tecnica presa strada facendo, con data e motivo (specifica §15).
| 2026-11-16 | Sessione salvata in `Dati/sessione/sessione.json`, fuori da SQLite e dalle copie giornaliere: DPAPI dell'utente su Windows (`ProtectedData` del framework desktop, nessun pacchetto nuovo); AES-256-GCM in Docker con la chiave da `FEBE_CHIAVE_SESSIONE` o da `chiave.bin` con permessi del solo proprietario; scrittura atomica; «Scollega» la cancella | Mimante usa AES-GCM con la chiave accanto al file, che su Windows non è protetta; DPAPI lega la sessione all'utente e alla macchina, cioè la regola 4. Le copie giornaliere del database spargerebbero il segreto |
| 2026-11-16 | Il client del sito aggiorna da sé nella sessione salvata i cookie rinnovati dal sito e `x-ig-www-claim`; un nuovo accesso dal browser ricopia tutto e vince; Febe non scrive mai nei cookie del browser; mentre il browser è aperto il motore è fermo | Evita due client sulla stessa sessione e un comando CDP di scrittura (`Network.setCookie`) |
| 2026-11-16 | Su Windows il client del sito usa le impostazioni proxy del sistema, come WebView2; in Docker esce dal contenitore come Chromium. HTTP/2 con ripiego su HTTP/1.1, niente HTTP/3 | Stessa strada di rete del browser che ha creato la sessione; HTTP/3 vorrebbe `libmsquic` nell'immagine |
| 2026-11-16 | La consegna delle risposte ai lettori sta in `ResponseDelivery` (Febe.Core): JSON con profondità 256 o blocchi incorporati nella pagina, uno alla volta, lettori isolati. La usano `ResponseTap` e, dalla tappa S2 di #36, il client del sito (S1, nessun cambiamento di comportamento) | Le risposte catturate e quelle chieste devono arrivare ai lettori e ai segnali nello stesso modo |
| 2026-11-16 | I rifiuti del catalogo delle richieste (`RequestCatalog`) stanno nel codice, non nel JSON: ogni nome che contiene `Mutation`, le famiglie GraphQL di chat, messaggi, presenza, promozioni, pubblicità e telemetria, i percorsi di telemetria e navigazione (`/logging_client_events`, `/ajax/…`, `/ws/`, `/api/v1/web/`, `/api/v1/discover/web/`…), ogni percorso con un host, una query o un valore come primo segmento; GraphQL solo verso `/api/graphql` e `/graphql/query` con POST, gli indirizzi semplici solo con GET; niente intestazione `cookie` nel modello | Una copia di `richieste.<sito>.json` modificata nella cartella dei dati non deve poter togliere un divieto (regole 3, 6) |
| 2026-11-16 | `richieste.<sito>.json` ha chiavi in inglese come gli altri file di configurazione; per ogni intestazione e campo dice la fonte del valore (`constant:`, `cookie:csrftoken`, `session:` user_agent, lsd, fb_dtsg, app_id, claim, account_id, doc_id, `operation`, `variables`, `computed`, `pending` se ancora da stabilire nella tappa S4). Un'operazione con un nome o un valore tolto dalla sonda nelle variabili resta «non configurata» e il client la rifiuta; nessun `doc_id` nel repository. Le categorie del Governatore sono quelle che ci sono già, fino ai tetti per richiesta della tappa S3 | Regola 9: le forme non mostrano tutto, e ciò che manca non si indovina |
| 2026-11-16 | Il contratto del client del sito (S1 di #36) si chiude nel codice: un'operazione con un'intestazione o un campo `pending` (o col nome tolto dalla sonda) non può essere configurata, quindi oggi nessuna operazione di Instagram lo è; un permesso vale per una sola richiesta, contato per riferimento (`SpentPermits`) e speso anche se la richiesta è poi rifiutata; i valori del caso hanno una forma (cifre per gli identificativi, i caratteri di un nome utente, di un codice o di una raccolta) e quelli che finiscono nel percorso non portano `/ ? # %` né punti soli; i percorsi del catalogo non hanno segmenti vuoti, `.` o `..` né caratteri fuori da lettere, cifre e `/_.-{}`; un indirizzo semplice non può puntare a GraphQL; la categoria «ricerca» è solo di un'operazione che porta `{search_text}` e nessun account o contenuto, e viceversa; `SiteAnswer` prende l'operazione del catalogo, non un percorso | Revisione della tappa S1: altrimenti una copia modificata nella cartella dei dati (`/./ajax/bz`, che `System.Uri` riduce a `/ajax/bz`), un valore come `..` o un permesso riusato aggirerebbero i divieti delle regole 4, 8, 9 e 10; il tetto della ricerca è fuori dalle sessioni (§7.4) |
+39 -1
View File
@@ -1,12 +1,50 @@
# Stato
Filo fra le sessioni (specifica §15): che cosa è fatto, che cosa è in corso, che cosa è rimandato e perché.
Aggiornato il 15 novembre 2026, versione 0.1.2 in `main` (il tag `v0.1.2` non è stato creato). **Le attività stanno
Aggiornato il 16 novembre 2026, versione 0.1.2 in `main` (il tag `v0.1.2` non è stato creato). **Le attività stanno
su Gitea**: issue, etichette e traguardi di https://gitea.encke-hake.ts.net/Alby96/Febe. Questo file resta il resoconto
fra le sessioni; ciò che è da fare si legge nelle issue aperte.
## In breve
**16 novembre 2026: richieste dirette con la sessione copiata dal browser** (#36, tappe S0 e S1; #52).
- Il HAR nuovo di Alberto (home, due ricerche, due profili) è stato letto solo in forma aggregata, senza stampare
valori: 487 voci complete, troncato a 55,5 MB, **nessun corpo delle risposte del sito** (va rifatto più corto, #35).
Utile lo stesso: i `doc_id` sono uguali a quelli del 12 novembre per 17 operazioni (H14), e stanno tutti nei moduli
JavaScript pubblici della pagina (`<Operazione>_instagramRelayOperation`, 229 moduli, 67 di scrittura); i gettoni
(`DTSGInitialData`, `LSD`, `CurrentUserInitialData`) si leggono dai moduli; l'elenco dei seguiti ha le sue
operazioni (`usePolarisGetFollowListQuery`). Confronto dei `doc_id` nel commento su #36.
- **Decisioni di Alberto** (7 righe in `docs/decisioni.md`, specifica §1, §2, §3, §6.0 nuova, §6.1, §6.3, §6.5, §6.6
H12–H17, §7.4, §7.6, §13.2, §16 domanda 11, regole 3, 4, 7 e 10 di `CLAUDE.md`): accesso a mano nel browser di
Febe, poi Febe copia cookie, user agent, gettoni e `doc_id` nella sessione salvata cifrata e lavora con richieste
HTTP dirette, senza browser in sottofondo, come Mimante con Bidoo; i rinvii non si seguono e una risposta senza i
campi attesi ferma e avvisa, senza riaccesso automatico; le storie si leggono con una richiesta e non risultano
viste. Le tappe S0–S7 stanno in #36.
- **Il browser che a volte restava vuoto** (#52): nove cause trovate e ordinate (la pagina di riposo `about:blank`
mostrata all'apertura, «Apri il browser» dopo una disconnessione, la finestra nascosta su Windows quando si lascia
la pagina, lo schermo remoto senza controllo dei fotogrammi in Docker, e altre); correzione su un ramo a parte.
- **Tappa S1, il giunto senza cambiare comportamento**: `ResponseDelivery` (in Core) consegna ai lettori le risposte
catturate e, da S2, quelle chieste, nello stesso modo; `ResponseTap` la usa con gli stessi conteggi e messaggi.
Contratti `SiteRequest`, `SiteAnswer` (che non stampa mai corpi, cookie o percorsi concreti) e `ISiteRequester`,
con `SiteRequestGuard`: niente richiesta senza un permesso concesso, della categoria giusta e mai usato prima
(`SpentPermits`); valori del caso con la loro forma (cifre per gli identificativi, nessun «/», «?», «%» nei
segmenti). `RequestCatalog` legge `richieste.instagram.json` e `richieste.prova.json` e rifiuta, nel codice e
quindi anche in una copia modificata nella cartella dei dati: ogni nome con `Mutation`, le famiglie di chat,
posta, presenza, promozioni e telemetria, i percorsi `/ajax/`, `/ws/`, `/logging_client_events`, `/api/v1/web/`,
host, parametri, segmenti «.» o vuoti, GraphQL fuori da `POST /api/graphql` e `/graphql/query`. Nessun `doc_id` nel
repository (si leggono all'accesso, S4). Il catalogo di Instagram ha 10 operazioni, **nessuna ancora configurata**:
le forme della sonda tolgono un nome di campo presente in ogni richiesta GraphQL e non dicono da dove vengano
`__dyn`, `__csr`, `__rev`, `__spin_*`, `__req`, `sec-ch-ua*` e altri (fonti «pending», da chiudere in S4); il
catalogo di prova ne ha 8 configurate su 11. Prove nuove: 87 (`RequestCatalogTests`, `SiteRequesterTests`,
`ResponseDeliveryTests`, con `ScriptedRequester` sui campioni ripuliti). Il campione 000016 viene da
`PolarisProfilePostsQuery`, non da `media/{numero}/info`.
- **Domande per Alberto** (anche su #36): la sonda può tenere i nomi di campo che oggi toglie («(nome tolto)»: sono
nomi di codice, non valori) e il tipo JSON dei valori (numero o testo)? Le corrispondenze `__user`/`av` →
identificativo dell'account e `reel_ids_arr` → account sono dedotte, non confermate.
- **Non verificato**: nessuna richiesta diretta è mai partita verso Instagram (le prove usano solo il sito di prova);
H12, H13, H15, H16 e H17 aspettano le prove dal vivo di Alberto.
**15 novembre 2026: una sola casella per cercare i profili, con suggerimenti e foto** (#37).
- «Aggiungi account» ha una sola casella. Mentre Alberto scrive compaiono subito i suggerimenti locali (profili
+10 -67
View File
@@ -55,13 +55,13 @@ public sealed record ResponseTapStats(long Responses, long JsonBodies, long Body
public sealed class ResponseTap : IAsyncDisposable
{
private const int MaxRequestOrders = 4096;
private static readonly JsonDocumentOptions BodyOptions = new() { MaxDepth = 256 };
private readonly ICdpSession _cdp;
private readonly HostClassifier _hosts;
private readonly IClock _clock;
private readonly IBrowserLog _log;
private readonly ResponseTapOptions _options;
private readonly ResponseDelivery _delivery;
private readonly ConcurrentDictionary<string, PendingBody> _inFlight = new(StringComparer.Ordinal);
private readonly Lock _gate = new();
private readonly List<IResponseConsumer> _consumers = [];
@@ -74,9 +74,8 @@ public sealed class ResponseTap : IAsyncDisposable
private Task? _worker;
private volatile string? _mainFrameId;
private long _responses;
private long _jsonBodies;
private long _bodyErrors;
private long _parseErrors;
private long _decodeErrors;
private long _skipped;
public ResponseTap(ICdpSession cdp, HostClassifier hosts, IClock clock, IBrowserLog? log = null, ResponseTapOptions? options = null)
@@ -86,6 +85,7 @@ public sealed class ResponseTap : IAsyncDisposable
_clock = clock ?? throw new ArgumentNullException(nameof(clock));
_log = log ?? NullBrowserLog.Instance;
_options = options ?? new ResponseTapOptions();
_delivery = new ResponseDelivery("Cattura", (message, exception) => _log.Warn(message, exception));
}
/// <summary>Raised for every main-frame navigation (sanitized address).</summary>
@@ -93,9 +93,9 @@ public sealed class ResponseTap : IAsyncDisposable
public ResponseTapStats Stats => new(
Interlocked.Read(ref _responses),
Interlocked.Read(ref _jsonBodies),
_delivery.JsonBodies,
Interlocked.Read(ref _bodyErrors),
Interlocked.Read(ref _parseErrors),
Interlocked.Read(ref _decodeErrors) + _delivery.ParseErrors,
Interlocked.Read(ref _skipped));
public IDisposable AddConsumer(IResponseConsumer consumer)
@@ -486,74 +486,17 @@ public sealed class ResponseTap : IAsyncDisposable
return;
}
// The same delivery as the answers the site client asks for (issue #36): captured and requested answers reach
// the readers identically.
var delivered = pending.IsPageDocument
? await ConsumeDocumentAsync(descriptor, bytes, consumers, ct).ConfigureAwait(false)
: await ConsumeBodyAsync(descriptor, bytes, consumers, ct).ConfigureAwait(false);
? await _delivery.DeliverDocumentAsync(descriptor, bytes, consumers, ct).ConfigureAwait(false)
: await _delivery.DeliverBodyAsync(descriptor, bytes, consumers, ct).ConfigureAwait(false);
if (delivered == 0)
{
NotifyDropped(descriptor.RequestId, pending.IsPageDocument ? BodyDropReason.NoJsonBlock : BodyDropReason.NotJson);
}
}
/// <summary>The body of a data response, as one answer. Returns how many answers reached the consumers (0 or 1).</summary>
private async Task<int> ConsumeBodyAsync(ResponseDescriptor descriptor, byte[] bytes, IResponseConsumer[] consumers, CancellationToken ct)
{
JsonDocument document;
try
{
document = JsonDocument.Parse(bytes, BodyOptions);
}
catch (JsonException)
{
// Not JSON after all (e.g. Instagram's «for (;;);» answers of /ajax/, which carry no contents): skipped.
Interlocked.Increment(ref _parseErrors);
return 0;
}
using (document)
{
await DeliverAsync(descriptor, document, consumers, ct).ConfigureAwait(false);
return 1;
}
}
/// <summary>
/// The JSON blocks a page document embeds, each handed to the consumers as an answer of its own with the document's
/// descriptor, so that a pagination stays with the block that carries it
/// (<see cref="DataResponses.EmbeddedJsonBlocks"/>). One block at a time: each is disposed before the next is parsed.
/// Returns how many blocks reached the consumers; a document without any is not data, and not a parse error.
/// </summary>
private async Task<int> ConsumeDocumentAsync(ResponseDescriptor descriptor, byte[] bytes, IResponseConsumer[] consumers, CancellationToken ct)
{
var delivered = 0;
foreach (var block in DataResponses.EmbeddedJsonBlocks(Encoding.UTF8.GetString(bytes)))
{
using (block)
{
delivered++;
await DeliverAsync(descriptor, block, consumers, ct).ConfigureAwait(false);
}
}
return delivered;
}
private async Task DeliverAsync(ResponseDescriptor descriptor, JsonDocument body, IResponseConsumer[] consumers, CancellationToken ct)
{
Interlocked.Increment(ref _jsonBodies);
foreach (var consumer in consumers)
{
try
{
await consumer.ConsumeAsync(descriptor, body, ct).ConfigureAwait(false);
}
catch (Exception ex) when (ex is not OperationCanceledException || !ct.IsCancellationRequested)
{
_log.Warn($"Cattura: errore in un lettore ({consumer.GetType().Name}).", ex);
}
}
}
/// <summary>The decoded bytes of a body, or null when the browser gives none or it cannot be decoded.</summary>
private async Task<byte[]?> FetchBodyAsync(string requestId, CancellationToken ct)
{
@@ -582,7 +525,7 @@ public sealed class ResponseTap : IAsyncDisposable
if (!body.TryGetBytesFromBase64(out var decoded))
{
Interlocked.Increment(ref _parseErrors);
Interlocked.Increment(ref _decodeErrors);
return null;
}
@@ -0,0 +1,126 @@
using System.Runtime.CompilerServices;
using Febe.Core.Abstractions;
namespace Febe.Core.Collection.Requests;
/// <summary>
/// Asks the site for one operation of the catalog (issue #36, spec §6.0) and returns its answer. The only way the
/// engine talks to the site without the browser: the HTTP implementation (tappa S2) is the one class of Febe that
/// contacts <c>www.instagram.com</c> (rule 3), the test double answers from the cleaned samples.
/// </summary>
/// <remarks>
/// <list type="bullet">
/// <item>Every call needs a <see cref="Permit"/> the Governor granted for that very request (rule 8, §7.4): a permit per
/// request, never one for several. An implementation refuses, with <see cref="SiteRequestRefusedException"/> and before
/// anything leaves, a permit that is missing, not granted or already presented once (<see cref="SpentPermits"/>), a
/// permit of another category than the operation's (<see cref="SiteOperation.Category"/>), an operation that is not in
/// the catalog, not configured or with a header or field still <see cref="ValueSourceKind.Pending"/>, and values that
/// do not match the operation's placeholders or would change its path. <see cref="SiteRequestGuard"/> does all of
/// these checks.</item>
/// <item>One request at a time: an implementation never sends two together and makes a second call wait (or refuse)
/// until the first has its answer; the minimum spacing of §7.7 is the Governor's.</item>
/// <item>Only reads: the catalog refuses writes when it loads (<see cref="RequestCatalog"/>).</item>
/// <item>Redirects are never followed: a 3xx comes back as a <see cref="SiteAnswer"/> with <see cref="SiteAnswer.IsRedirect"/>.</item>
/// <item>Logs name only the operation or the shape of the path and the status (rule 10).</item>
/// </list>
/// Cancellation throws <see cref="OperationCanceledException"/>. The answer goes to the readers through
/// <see cref="ResponseDelivery"/>, like the captured ones.
/// </remarks>
public interface ISiteRequester
{
Task<SiteAnswer> SendAsync(SiteRequest request, Permit permit, CancellationToken ct);
}
/// <summary>
/// A request that must not leave: no granted permit, a permit already used, another category, an operation outside
/// the catalog or not configured, values that do not match. The message is in Italian and names only the operation and
/// value names.
/// </summary>
public sealed class SiteRequestRefusedException : InvalidOperationException
{
public SiteRequestRefusedException()
{
}
public SiteRequestRefusedException(string message)
: base(message)
{
}
public SiteRequestRefusedException(string message, Exception innerException)
: base(message, innerException)
{
}
}
/// <summary>
/// The permits a site client was already shown (rule 8, §7.4: a permit per request). By reference, not by value: two
/// permits the Governor granted one after the other are two, even when their fields are equal. Holds them weakly, so a
/// long run does not keep them alive.
/// </summary>
public sealed class SpentPermits
{
private static readonly object Marker = new();
private readonly ConditionalWeakTable<Permit, object> _spent = new();
/// <summary>Marks <paramref name="permit"/> as used; false when it was used before.</summary>
public bool TrySpend(Permit permit)
{
ArgumentNullException.ThrowIfNull(permit);
return _spent.TryAdd(permit, Marker);
}
}
/// <summary>The checks every <see cref="ISiteRequester"/> makes before a request leaves.</summary>
public static class SiteRequestGuard
{
/// <summary>
/// The catalog operation <paramref name="request"/> asks for, when it may leave with <paramref name="permit"/>. A
/// granted permit is spent as soon as it is shown, also when the request is then refused: the Governor has already
/// counted it.
/// </summary>
/// <exception cref="SiteRequestRefusedException">It must not leave (see <see cref="ISiteRequester"/>).</exception>
public static SiteOperation Check(RequestCatalog catalog, SiteRequest request, Permit? permit, SpentPermits spent)
{
ArgumentNullException.ThrowIfNull(catalog);
ArgumentNullException.ThrowIfNull(request);
ArgumentNullException.ThrowIfNull(spent);
if (permit is null || !permit.Granted)
{
throw new SiteRequestRefusedException($"Richiesta «{request.Operation}» non inviata: il Governatore non ha dato il permesso.");
}
if (!spent.TrySpend(permit))
{
throw new SiteRequestRefusedException($"Richiesta «{request.Operation}» non inviata: il permesso è già stato usato (un permesso per richiesta).");
}
if (!catalog.TryGet(request.Operation, out var operation))
{
throw new SiteRequestRefusedException($"Richiesta «{request.Operation}» non inviata: l'operazione non è in {catalog.Source}.");
}
if (!operation.IsConfigured)
{
throw new SiteRequestRefusedException($"Richiesta «{request.Operation}» non inviata: l'operazione non è configurata ({operation.NotConfiguredReason}).");
}
// The catalog never loads a configured operation with pending sources; checked again so no path relies on it.
if (operation.PendingValues.Count > 0)
{
throw new SiteRequestRefusedException($"Richiesta «{request.Operation}» non inviata: intestazioni o campi senza una fonte stabilita.");
}
if (permit.Kind != operation.Category)
{
throw new SiteRequestRefusedException($"Richiesta «{request.Operation}» non inviata: il permesso è per {permit.Kind}, l'operazione conta come {operation.Category}.");
}
if (operation.CheckValues(request) is { } problem)
{
throw new SiteRequestRefusedException($"Richiesta «{request.Operation}» non inviata: {problem}");
}
return operation;
}
}
@@ -0,0 +1,869 @@
using System.Collections.Frozen;
using System.Globalization;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Text.RegularExpressions;
using Febe.Core.Abstractions;
namespace Febe.Core.Collection.Requests;
/// <summary>Whether an operation is a GraphQL query of the site or a plain address.</summary>
public enum OperationKind
{
/// <summary>A GraphQL query: POST of a form to one of the GraphQL entry points, named by its friendly name.</summary>
GraphQl,
/// <summary>A plain address read with GET (e.g. <c>/api/v1/media/{numero}/info/</c>).</summary>
Rest,
}
/// <summary>Where the value of a header or of a form field comes from.</summary>
public enum ValueSourceKind
{
/// <summary>A constant, as the request shapes show it (<c>x-asbd-id: 359341</c>).</summary>
Constant,
/// <summary>The value of a cookie of the saved session (<c>x-csrftoken</c> is the cookie <c>csrftoken</c>).</summary>
Cookie,
/// <summary>A value the saved session holds (§6.0): user agent, tokens, account, application, claim, doc_id.</summary>
Session,
/// <summary>The name of the operation (<c>x-fb-friendly-name</c>, <c>fb_api_req_friendly_name</c>).</summary>
Operation,
/// <summary>The variables of the operation, filled with the values of the case (form field <c>variables</c>).</summary>
Variables,
/// <summary>Computed by the HTTP stack (<c>content-length</c>).</summary>
Computed,
/// <summary>
/// Not settled yet: the request shapes show the name but not where its value comes from (a value of the page, of
/// the browser, or a name the probe removed). The site client cannot send it until it is settled (tappa S4).
/// </summary>
Pending,
}
/// <summary>Where one header or form field of an operation takes its value from.</summary>
/// <param name="Kind">The kind of source.</param>
/// <param name="Argument">The constant, the cookie name or the session field; null for the other kinds.</param>
public sealed record ValueSource(ValueSourceKind Kind, string? Argument)
{
/// <summary>The session fields a header or field may take (§6.0, what «Ho finito» copies).</summary>
public static IReadOnlySet<string> SessionFields { get; } = new[]
{
"user_agent", "lsd", "fb_dtsg", "app_id", "claim", "account_id", "doc_id",
}.ToFrozenSet(StringComparer.Ordinal);
/// <summary>The text form of the catalog file: <c>constant:…</c>, <c>cookie:…</c>, <c>session:…</c>, <c>operation</c>, <c>variables</c>, <c>computed</c>, <c>pending</c>.</summary>
public override string ToString() => Kind switch
{
ValueSourceKind.Constant => $"constant:{Argument}",
ValueSourceKind.Cookie => $"cookie:{Argument}",
ValueSourceKind.Session => $"session:{Argument}",
ValueSourceKind.Operation => "operation",
ValueSourceKind.Variables => "variables",
ValueSourceKind.Computed => "computed",
_ => "pending",
};
}
/// <summary>
/// One operation of <c>richieste.&lt;sito&gt;.json</c>: what the site client may ask (issue #36, spec §6.0). Written from
/// the real request shapes of <c>febe-sonda richieste</c> (§14.5) and documented by a cleaned sample of its answer
/// (rule 9). It holds no value of a session: the <c>doc_id</c> of a GraphQL query, the tokens and the account are read
/// from the page at login (tappa S4) and named here only by their source.
/// </summary>
public sealed partial class SiteOperation
{
internal SiteOperation(
string name,
OperationKind kind,
string method,
string path,
ActionKind category,
bool isConfigured,
string? notConfiguredReason,
IReadOnlyDictionary<string, string> parameters,
IReadOnlyDictionary<string, ValueSource> headers,
IReadOnlyDictionary<string, ValueSource> body,
JsonElement? variables,
IReadOnlyList<string> rootFields,
string? sample,
IReadOnlySet<string> required,
IReadOnlySet<string> optional)
{
Name = name;
Kind = kind;
Method = method;
Path = path;
Category = category;
IsConfigured = isConfigured;
NotConfiguredReason = notConfiguredReason;
Parameters = parameters;
Headers = headers;
Body = body;
Variables = variables;
RootFields = rootFields;
Sample = sample;
RequiredValues = required;
OptionalValues = optional;
PathShape = Placeholder().Replace(path, m => CaseValues.ShapeOf(m.Groups[1].Value));
PathValues = Placeholder().Matches(path).Select(m => m.Groups[1].Value).ToFrozenSet(StringComparer.Ordinal);
PendingValues = [.. headers.Where(h => h.Value.Kind == ValueSourceKind.Pending).Select(h => h.Key)
.Concat(body.Where(b => b.Value.Kind == ValueSourceKind.Pending).Select(b => b.Key))];
}
/// <summary>The name a <see cref="SiteRequest"/> asks for: for GraphQL the friendly name of the site's query.</summary>
public string Name { get; }
public OperationKind Kind { get; }
/// <summary><c>GET</c> or <c>POST</c> (POST only for GraphQL queries).</summary>
public string Method { get; }
/// <summary>The path, with the values of the case as <c>{account_id}</c>, <c>{media_id}</c>…</summary>
public string Path { get; }
/// <summary>The path as the log may show it (<c>/api/v1/media/{numero}/info/</c>): never a value.</summary>
public string PathShape { get; }
/// <summary>
/// The category of the Governor the request counts in (§7.4: a permit per request, in the category of what it
/// asks). Mapped on the existing categories, the budgets per request are recomputed in tappa S3: the profile header
/// and the list of a profile's collections count as <see cref="ActionKind.ProfileOpen"/>; a page of posts, reels or
/// tagged contents as <see cref="ActionKind.BackfillPage"/> (the first page too); stories and highlights as
/// <see cref="ActionKind.StoryView"/>; a single post as <see cref="ActionKind.SinglePost"/>; the search box as
/// <see cref="ActionKind.ProfileSearch"/>; the home as <see cref="ActionKind.Home"/>; the followed list as
/// <see cref="ActionKind.FollowingListPage"/>. The file names the category, but the code pins the one budget outside
/// the sessions: <see cref="ActionKind.ProfileSearch"/> only for an operation that carries <c>{search_text}</c> and no
/// account or content, and every operation carrying <c>{search_text}</c> counts there.
/// </summary>
public ActionKind Category { get; }
/// <summary>
/// False when the shapes do not tell everything the request needs (a variable whose name the probe removed, a
/// value the shapes do not show, a header or field whose source is still <see cref="ValueSourceKind.Pending"/>):
/// the site client refuses it (rule 9: what is not known is not guessed). A configured operation has no
/// <see cref="PendingValues"/>: the catalog refuses the file otherwise.
/// </summary>
public bool IsConfigured { get; }
/// <summary>Why it is not configured, in Italian; null when it is.</summary>
public string? NotConfiguredReason { get; }
/// <summary>Query string of a plain address: name and template (<c>{cursor}</c>, <c>{cursor?}</c> when optional, <c>constant:…</c>).</summary>
public IReadOnlyDictionary<string, string> Parameters { get; }
/// <summary>The names of the headers and where each value comes from. Cookies are not a header here: they come from the saved session.</summary>
public IReadOnlyDictionary<string, ValueSource> Headers { get; }
/// <summary>The form fields of a GraphQL query and where each value comes from.</summary>
public IReadOnlyDictionary<string, ValueSource> Body { get; }
/// <summary>The template of the variables of a GraphQL query: constants as the shapes show them, <c>"{name}"</c> for a value of the case.</summary>
public JsonElement? Variables { get; }
/// <summary>
/// Fields every answer must have (dotted paths from the root, e.g. <c>data.user</c>): a JSON without them stops the
/// session and warns (§6.0, §7.6). From the request shapes (<c>campo_radice</c>) or the cleaned sample.
/// </summary>
public IReadOnlyList<string> RootFields { get; }
/// <summary>The cleaned sample of <c>tests/Febe.Tests/campioni/&lt;sito&gt;</c> that documents the answer (file name without extension).</summary>
public string? Sample { get; }
/// <summary>Values of the case the request must carry.</summary>
public IReadOnlySet<string> RequiredValues { get; }
/// <summary>Values of the case the request may carry (optional query parameters).</summary>
public IReadOnlySet<string> OptionalValues { get; }
/// <summary>Headers and form fields whose source is not settled yet (<see cref="ValueSourceKind.Pending"/>).</summary>
public IReadOnlyList<string> PendingValues { get; }
/// <summary>Why <paramref name="request"/> does not fit this operation (in Italian, names only), or null.</summary>
public string? CheckValues(SiteRequest request)
{
ArgumentNullException.ThrowIfNull(request);
var missing = RequiredValues.Where(v => !request.Values.ContainsKey(v)).Order(StringComparer.Ordinal).ToList();
if (missing.Count > 0)
{
return $"manca il valore {string.Join(", ", missing.Select(m => $"«{m}»"))}.";
}
var extra = request.Values.Keys.Where(v => !RequiredValues.Contains(v) && !OptionalValues.Contains(v)).Order(StringComparer.Ordinal).ToList();
if (extra.Count > 0)
{
return $"il valore {string.Join(", ", extra.Select(e => $"«{e}»"))} non serve a questa operazione.";
}
// A value that fills a segment of the path must stay that segment: nothing the HTTP stack would read as another
// segment, a query or a step up («/», «?», «#», «%», «.», «..»).
var unsafeInPath = PathValues.Where(v => request.Values.TryGetValue(v, out var value) && !IsPlainSegment(value)).Order(StringComparer.Ordinal).ToList();
return unsafeInPath.Count > 0
? $"il valore {string.Join(", ", unsafeInPath.Select(v => $"«{v}»"))} non può stare nel percorso (contiene «/», «?», «#», «%», spazi o è fatto solo di punti)."
: null;
}
/// <summary>Values of the case that fill a segment of <see cref="Path"/>.</summary>
public IReadOnlySet<string> PathValues { get; }
private static bool IsPlainSegment(string value) =>
value.Length > 0
&& value.Trim('.').Length > 0
&& !value.Any(c => c is '/' or '\\' or '?' or '#' or '%' or ';' || char.IsWhiteSpace(c) || char.IsControl(c));
/// <summary>True when the answer has every field of <see cref="RootFields"/> (not null).</summary>
public bool HasRootFields(JsonElement root)
{
foreach (var field in RootFields)
{
var node = root;
foreach (var part in field.Split('.'))
{
if (node.ValueKind != JsonValueKind.Object || !node.TryGetProperty(part, out node))
{
return false;
}
}
if (node.ValueKind is JsonValueKind.Null or JsonValueKind.Undefined)
{
return false;
}
}
return true;
}
/// <summary>The name, the shape of the path and the state: what the log may show.</summary>
public override string ToString() => $"{Name} ({Method} {PathShape}{(IsConfigured ? string.Empty : ", non configurata")})";
[GeneratedRegex(@"\{([a-z_]+)\??\}", RegexOptions.CultureInvariant)]
private static partial Regex Placeholder();
}
/// <summary>
/// The operations the site client may ask (issue #36, spec §6.0), loaded from <c>richieste.&lt;sito&gt;.json</c> like the
/// other configuration files: a copy embedded in Febe, an editable copy in the Data folder that wins.
/// </summary>
/// <remarks>
/// The file only narrows: the refusals are in the code, so an edited copy cannot lift them. At load time the catalog
/// refuses, with an Italian message naming the file and the operation:
/// <list type="bullet">
/// <item>any operation whose name contains <c>Mutation</c> (the site's writes; the request shapes show them as
/// <c>…Mutation</c>), and the GraphQL families of chat, inbox, presence, promotions and telemetry
/// (<see cref="ForbiddenNamePrefixes"/>, <see cref="ForbiddenNameParts"/>);</item>
/// <item>the paths of telemetry, navigation and the site's own scripts (<see cref="ForbiddenPathPrefixes"/>), any path
/// with a host or a query string, any segment that is empty, <c>.</c> or <c>..</c> (the HTTP stack would rewrite it),
/// any character outside letters, digits and <c>/_.-{}</c>, any path whose first segment is a value, GraphQL outside
/// its two entry points and a plain address that points at one of them;</item>
/// <item>a plain address that is not read with GET, a GraphQL query without its variables, a header <c>cookie</c>,
/// <c>host</c> or <c>authorization</c>, an unknown key, source, category or value name, the category «ricerca» for
/// anything but a search (<see cref="SiteOperation.Category"/>).</item>
/// </list>
/// An operation is «configured» only when the shapes tell everything it needs, every header and field included (no
/// <see cref="ValueSourceKind.Pending"/>); otherwise the file says why, and the site client refuses it.
/// </remarks>
public sealed partial class RequestCatalog
{
/// <summary>Written by the probe in place of a name that elsewhere was a value: the real name is unknown.</summary>
public const string RemovedName = "(nome tolto)";
/// <summary>Written in a template in place of a value the shapes do not show.</summary>
public const string RemovedValue = "(valore tolto)";
/// <summary>The two GraphQL entry points seen in the request shapes.</summary>
public static IReadOnlySet<string> GraphQlPaths { get; } = new[] { "/api/graphql", "/graphql/query" }.ToFrozenSet(StringComparer.Ordinal);
/// <summary>Paths never asked, also when an edited file lists them: telemetry, navigation and scripts of the page, chat.</summary>
public static IReadOnlyList<string> ForbiddenPathPrefixes { get; } =
[
"/logging_client_events", "/ajax/bz", "/ajax/bulk-route-definitions/", "/ajax/navigation/", "/ajax/", "/ws/",
"/api/v1/web/", "/api/v1/discover/web/", "/api/v1/direct_v2/", "/api/v1/qp/",
];
/// <summary>Beginnings of the GraphQL names of chat, inbox, presence, promotions, telemetry and ads, never asked.</summary>
public static IReadOnlyList<string> ForbiddenNamePrefixes { get; } =
[
"IGD", "IGPresence", "PolarisDirect", "useIGDSystem", "QuickPromotion", "PolarisAPIGetFrCookie", "useFeatureLimits",
"PolarisStoriesV3Seen", "PolarisStoriesV3AdsPool", "usePolarisLike", "usePolarisRegisterInRecentSearches",
"PolarisProfileDirectOrPartnershipInbox",
];
/// <summary>Parts of a name that make it a write or a chat operation wherever they stand.</summary>
public static IReadOnlyList<string> ForbiddenNameParts { get; } = ["Mutation", "Inbox", "Msys", "Presence"];
private static readonly FrozenSet<string> ForbiddenHeaders = new[]
{
"cookie", "host", "authorization", "proxy-authorization", "connection", "transfer-encoding", "set-cookie",
}.ToFrozenSet(StringComparer.Ordinal);
private readonly FrozenDictionary<string, SiteOperation> _byName;
private RequestCatalog(string site, IReadOnlyList<SiteOperation> operations, string? source)
{
Site = site;
Operations = operations;
Source = source ?? $"richieste.{site}.json";
_byName = operations.ToFrozenDictionary(o => o.Name, StringComparer.Ordinal);
}
/// <summary>The site name of the file (<c>instagram</c>, <c>prova</c>).</summary>
public string Site { get; }
/// <summary>Every operation, in the order of the file.</summary>
public IReadOnlyList<SiteOperation> Operations { get; }
/// <summary>Where it came from (file path or embedded name), for messages.</summary>
public string Source { get; }
/// <summary>The configured operations.</summary>
public IEnumerable<SiteOperation> Configured => Operations.Where(o => o.IsConfigured);
public bool TryGet(string name, [System.Diagnostics.CodeAnalysis.NotNullWhen(true)] out SiteOperation? operation) =>
_byName.TryGetValue(name, out operation);
/// <summary>The Instagram catalog (embedded <c>richieste.instagram.json</c>).</summary>
public static RequestCatalog LoadInstagram() => LoadEmbedded("richieste.instagram.json");
/// <summary>The catalog of the local test site (embedded <c>richieste.prova.json</c>).</summary>
public static RequestCatalog LoadTestSite() => LoadEmbedded("richieste.prova.json");
/// <summary>Loads <paramref name="path"/> when it exists, otherwise the embedded file with the same name.</summary>
public static RequestCatalog LoadOrEmbedded(string? path, string embeddedName) =>
!string.IsNullOrEmpty(path) && File.Exists(path) ? Load(path) : LoadEmbedded(embeddedName);
public static RequestCatalog Load(string path) => Parse(File.ReadAllText(path), path);
public static RequestCatalog LoadEmbedded(string fileName) => Parse(SiteMap.ReadEmbeddedText(fileName), fileName);
/// <exception cref="InvalidDataException">The file is not valid or lists something the catalog refuses.</exception>
public static RequestCatalog Parse(string json, string? source = null)
{
var name = source ?? "richieste";
CatalogFile? file;
try
{
file = JsonSerializer.Deserialize(json, RequestCatalogJsonContext.Default.CatalogFile);
}
catch (JsonException ex)
{
throw new InvalidDataException($"{name}: file non valido (riga {(ex.LineNumber ?? 0) + 1}): {ex.Message.Split('.')[0]}.", ex);
}
if (file is null || string.IsNullOrWhiteSpace(file.Site))
{
throw new InvalidDataException($"{name}: manca «site».");
}
if (file.Operations is null)
{
throw new InvalidDataException($"{name}: manca l'elenco «operations».");
}
var operations = new List<SiteOperation>(file.Operations.Count);
var names = new HashSet<string>(StringComparer.Ordinal);
for (var i = 0; i < file.Operations.Count; i++)
{
var dto = file.Operations[i] ?? throw new InvalidDataException($"{name}, operazione {(i + 1).ToString(CultureInfo.InvariantCulture)}: voce vuota.");
var operation = ToOperation(dto, $"{name}, operazione {(i + 1).ToString(CultureInfo.InvariantCulture)}");
if (!names.Add(operation.Name))
{
throw new InvalidDataException($"{name}: l'operazione «{operation.Name}» compare due volte.");
}
operations.Add(operation);
}
return new RequestCatalog(file.Site.Trim(), operations, source);
}
/// <summary>Why the code refuses an operation with this name, whatever the file says; null when it does not.</summary>
public static string? RefusalOfName(string operationName)
{
ArgumentNullException.ThrowIfNull(operationName);
foreach (var part in ForbiddenNameParts)
{
if (operationName.Contains(part, StringComparison.OrdinalIgnoreCase))
{
return part == "Mutation"
? "è una scrittura («Mutation»): Febe chiede solo letture (regola 6)"
: $"è della famiglia di chat, messaggi o presenza («{part}»), che Febe non chiede mai";
}
}
foreach (var prefix in ForbiddenNamePrefixes)
{
if (operationName.StartsWith(prefix, StringComparison.OrdinalIgnoreCase))
{
return $"è della famiglia «{prefix}» (chat, messaggi, presenza, promozioni, pubblicità o telemetria), che Febe non chiede mai";
}
}
return null;
}
/// <summary>Why the code refuses this path, whatever the file says; null when it does not.</summary>
public static string? RefusalOfPath(string path)
{
ArgumentNullException.ThrowIfNull(path);
if (!path.StartsWith('/') || path.StartsWith("//", StringComparison.Ordinal) || path.Contains("://", StringComparison.Ordinal)
|| path.Contains('\\', StringComparison.Ordinal))
{
return "un percorso comincia con una sola «/» e non nomina un host: Febe chiede solo al sito";
}
if (path.IndexOfAny(['?', '#']) >= 0)
{
return "un percorso non porta parametri: vanno in «parameters»";
}
if (path.Contains("..", StringComparison.Ordinal) || !PathCharacters().IsMatch(path))
{
return "il percorso contiene caratteri non ammessi";
}
// The prefixes are compared on the text: no segment the HTTP stack or the server would rewrite («/./», «//»), so
// the path that leaves is the one checked here.
var segments = path.Split('/');
for (var i = 1; i < segments.Length; i++)
{
if ((segments[i].Length == 0 && i < segments.Length - 1) || segments[i] == ".")
{
return "un percorso non ha segmenti vuoti («//») né «.»: il percorso che parte è quello scritto";
}
}
foreach (var prefix in ForbiddenPathPrefixes)
{
if (path.StartsWith(prefix, StringComparison.OrdinalIgnoreCase)
|| path.Equals(prefix.TrimEnd('/'), StringComparison.OrdinalIgnoreCase))
{
return $"«{prefix}» è telemetria, navigazione o chat della pagina, che Febe non chiede mai";
}
}
var first = path.Split('/', StringSplitOptions.RemoveEmptyEntries).FirstOrDefault();
if (first is null || first.StartsWith('{'))
{
return "il primo segmento del percorso dev'essere una parola fissa del sito, non un valore";
}
return null;
}
private static SiteOperation ToOperation(OperationDto dto, string where)
{
if (string.IsNullOrWhiteSpace(dto.Name))
{
throw new InvalidDataException($"{where}: manca «name».");
}
var name = dto.Name.Trim();
where = $"{where} («{name}»)";
if (!OperationName().IsMatch(name))
{
throw new InvalidDataException($"{where}: il nome contiene caratteri non ammessi.");
}
if (RefusalOfName(name) is { } refusedName)
{
throw new InvalidDataException($"{where}: rifiutata, {refusedName}.");
}
var kind = dto.Kind?.Trim().ToLowerInvariant() switch
{
"graphql" => OperationKind.GraphQl,
"rest" => OperationKind.Rest,
_ => throw new InvalidDataException($"{where}: tipo «{dto.Kind}» sconosciuto (ammessi: graphql, rest)."),
};
var method = dto.Method?.Trim().ToUpperInvariant();
var path = dto.Path?.Trim() ?? throw new InvalidDataException($"{where}: manca «path».");
if (RefusalOfPath(path) is { } refusedPath)
{
throw new InvalidDataException($"{where}: rifiutata, {refusedPath}.");
}
var category = ParseCategory(dto.Category, where);
var required = new SortedSet<string>(StringComparer.Ordinal);
var optional = new SortedSet<string>(StringComparer.Ordinal);
foreach (var segment in path.Split('/', StringSplitOptions.RemoveEmptyEntries))
{
if (segment.Contains('{', StringComparison.Ordinal) || segment.Contains('}', StringComparison.Ordinal))
{
required.Add(CaseValue(segment, where, allowOptional: false, out _));
}
}
var parameters = new Dictionary<string, string>(StringComparer.Ordinal);
foreach (var (key, value) in dto.Parameters ?? new Dictionary<string, string?>())
{
if (string.IsNullOrWhiteSpace(key) || key == RemovedName || !FieldName().IsMatch(key))
{
throw new InvalidDataException($"{where}: parametro «{key}» non ammesso.");
}
if (value is null || value.Length == 0)
{
throw new InvalidDataException($"{where}: il parametro «{key}» non dice il suo valore.");
}
if (value.StartsWith('{'))
{
var caseValue = CaseValue(value, where, allowOptional: true, out var isOptional);
(isOptional ? optional : required).Add(caseValue);
}
else if (!value.StartsWith("constant:", StringComparison.Ordinal) || value.Length == "constant:".Length)
{
throw new InvalidDataException($"{where}: il parametro «{key}» vale «{{valore del caso}}» o «constant:…».");
}
parameters[key] = value;
}
var headers = Sources(dto.Headers, where, "intestazione", isBody: false);
foreach (var header in headers.Keys)
{
if (ForbiddenHeaders.Contains(header))
{
throw new InvalidDataException($"{where}: l'intestazione «{header}» non si scrive nel modello (i cookie vengono dalla sessione salvata, l'host è quello del sito).");
}
}
if (headers.TryGetValue("user-agent", out var userAgent) && userAgent != new ValueSource(ValueSourceKind.Session, "user_agent"))
{
throw new InvalidDataException($"{where}: «user-agent» è quello del browser, copiato tale e quale («session:user_agent», regola 4).");
}
var body = Sources(dto.Body, where, "campo", isBody: true);
JsonElement? variables = null;
var removedInVariables = false;
if (kind == OperationKind.GraphQl)
{
if (method != "POST")
{
throw new InvalidDataException($"{where}: una richiesta GraphQL si fa con POST.");
}
if (!GraphQlPaths.Contains(path))
{
throw new InvalidDataException($"{where}: una richiesta GraphQL va a {string.Join(" o ", GraphQlPaths)}.");
}
if (parameters.Count > 0)
{
throw new InvalidDataException($"{where}: una richiesta GraphQL non ha parametri nell'indirizzo.");
}
RequireSource(body, "variables", new ValueSource(ValueSourceKind.Variables, null), where);
RequireSource(body, "doc_id", new ValueSource(ValueSourceKind.Session, "doc_id"), where);
RequireSource(body, "fb_api_req_friendly_name", new ValueSource(ValueSourceKind.Operation, null), where);
if (headers.TryGetValue("x-fb-friendly-name", out var friendly) && friendly.Kind != ValueSourceKind.Operation)
{
throw new InvalidDataException($"{where}: «x-fb-friendly-name» è il nome dell'operazione («operation»).");
}
if (dto.Variables is not { ValueKind: JsonValueKind.Object } template)
{
throw new InvalidDataException($"{where}: manca «variables» (un oggetto, come nelle forme delle richieste).");
}
removedInVariables = Template(template, where, required, depth: 0);
variables = template.Clone();
}
else
{
if (method != "GET")
{
throw new InvalidDataException($"{where}: un indirizzo semplice si legge solo con GET (sola lettura).");
}
// A GET to a GraphQL entry point would skip the checks of a GraphQL query (doc_id from the session, the
// friendly name that is the operation's and that the refusal of «Mutation» looks at).
if (path.Split('/').Any(s => s.Contains("graphql", StringComparison.OrdinalIgnoreCase)))
{
throw new InvalidDataException($"{where}: rifiutata, una richiesta GraphQL si scrive con «kind»: «graphql», non come indirizzo semplice.");
}
if (body.Count > 0 || dto.Variables is { ValueKind: not JsonValueKind.Null and not JsonValueKind.Undefined })
{
throw new InvalidDataException($"{where}: una lettura con GET non ha «body» né «variables».");
}
}
// The search budget belongs to no session and spares the collection caps (§7.4): only a search may count there,
// and a search counts nowhere else, whatever an edited copy says.
var searches = required.Contains(CaseValues.SearchText) || optional.Contains(CaseValues.SearchText);
if (category == ActionKind.ProfileSearch
&& (!required.Contains(CaseValues.SearchText) || required.Concat(optional).Any(v => v is not (CaseValues.SearchText or CaseValues.Cursor))))
{
throw new InvalidDataException($"{where}: rifiutata, la categoria «ricerca» è solo di una ricerca, che porta «{{{CaseValues.SearchText}}}» e nessun account o contenuto.");
}
if (searches && category != ActionKind.ProfileSearch)
{
throw new InvalidDataException($"{where}: rifiutata, un'operazione che porta «{{{CaseValues.SearchText}}}» è una ricerca (categoria «ricerca»).");
}
var rootFields = (dto.RootFields ?? []).Select(f => f?.Trim() ?? string.Empty).ToList();
if (rootFields.Any(f => !RootField().IsMatch(f)))
{
throw new InvalidDataException($"{where}: «rootFields» elenca campi come «data.user», separati da punti.");
}
var sample = string.IsNullOrWhiteSpace(dto.Sample) ? null : dto.Sample.Trim();
if (sample is not null && !SampleName().IsMatch(sample))
{
throw new InvalidDataException($"{where}: «sample» è il nome di un campione ripulito, come «000005-post-video».");
}
var configured = dto.Configured ?? throw new InvalidDataException($"{where}: manca «configured» (true o false).");
var reason = string.IsNullOrWhiteSpace(dto.Reason) ? null : dto.Reason.Trim();
if (configured)
{
if (reason is not null)
{
throw new InvalidDataException($"{where}: «reason» si scrive solo per un'operazione non configurata.");
}
if (removedInVariables)
{
throw new InvalidDataException($"{where}: le variabili hanno un nome o un valore tolto dalla sonda: non può essere configurata (regola 9).");
}
// Every header and field the model lists leaves with its value, or the request is not the browser's (rule 4):
// one whose source is not settled, or whose name the probe removed, keeps the operation not configured.
var pending = headers.Where(h => h.Value.Kind == ValueSourceKind.Pending).Select(h => h.Key)
.Concat(body.Where(b => b.Value.Kind == ValueSourceKind.Pending).Select(b => b.Key)).ToList();
if (pending.Count > 0)
{
throw new InvalidDataException(
$"{where}: {pending.Count.ToString(CultureInfo.InvariantCulture)} intestazioni o campi hanno la fonte «pending» (come {string.Join(", ", pending.Take(3).Select(p => $"«{p}»"))}): non può essere configurata finché non si sa da dove vengono i valori (regola 9).");
}
if (sample is null || rootFields.Count == 0)
{
throw new InvalidDataException($"{where}: un'operazione configurata dice il campione della risposta («sample») e i campi attesi («rootFields»).");
}
}
else if (reason is null)
{
throw new InvalidDataException($"{where}: un'operazione non configurata dice perché («reason»).");
}
return new SiteOperation(
name, kind, method!, path, category, configured, reason, parameters.ToFrozenDictionary(StringComparer.Ordinal),
headers.ToFrozenDictionary(StringComparer.Ordinal), body.ToFrozenDictionary(StringComparer.Ordinal), variables,
rootFields, sample, required.ToFrozenSet(StringComparer.Ordinal), optional.ToFrozenSet(StringComparer.Ordinal));
}
private static Dictionary<string, ValueSource> Sources(Dictionary<string, string?>? entries, string where, string what, bool isBody)
{
var result = new Dictionary<string, ValueSource>(StringComparer.Ordinal);
foreach (var (key, text) in entries ?? new Dictionary<string, string?>())
{
var isRemoved = key == RemovedName;
if (!isRemoved && (isBody ? !FieldName().IsMatch(key) : !HeaderName().IsMatch(key)))
{
throw new InvalidDataException($"{where}: {what} «{key}» non ammesso{(isBody ? string.Empty : " (nomi in minuscolo)")}.");
}
var source = ParseSource(text, $"{where}, {what} «{key}»");
if (isRemoved && source.Kind != ValueSourceKind.Pending)
{
throw new InvalidDataException($"{where}: un {what} con il nome tolto dalla sonda vale solo «pending».");
}
if (source.Kind == ValueSourceKind.Variables && !(isBody && key == "variables"))
{
throw new InvalidDataException($"{where}: «variables» è la fonte del solo campo «variables».");
}
result[key] = source;
}
return result;
}
private static ValueSource ParseSource(string? text, string where)
{
var value = text ?? string.Empty;
var colon = value.IndexOf(':', StringComparison.Ordinal);
var head = colon < 0 ? value : value[..colon];
var argument = colon < 0 ? null : value[(colon + 1)..];
switch (head)
{
case "constant" when !string.IsNullOrEmpty(argument):
return new ValueSource(ValueSourceKind.Constant, argument);
case "cookie" when argument is not null && CookieName().IsMatch(argument):
return new ValueSource(ValueSourceKind.Cookie, argument);
case "session" when argument is not null && ValueSource.SessionFields.Contains(argument):
return new ValueSource(ValueSourceKind.Session, argument);
case "operation" when argument is null:
return new ValueSource(ValueSourceKind.Operation, null);
case "variables" when argument is null:
return new ValueSource(ValueSourceKind.Variables, null);
case "computed" when argument is null:
return new ValueSource(ValueSourceKind.Computed, null);
case "pending" when argument is null:
return new ValueSource(ValueSourceKind.Pending, null);
default:
throw new InvalidDataException(
$"{where}: fonte «{value}» sconosciuta (ammesse: constant:…, cookie:…, session:{string.Join("|", ValueSource.SessionFields.Order(StringComparer.Ordinal))}, operation, variables, computed, pending).");
}
}
private static void RequireSource(Dictionary<string, ValueSource> body, string field, ValueSource expected, string where)
{
if (!body.TryGetValue(field, out var actual) || actual != expected)
{
throw new InvalidDataException($"{where}: una richiesta GraphQL ha il campo «{field}» con la fonte «{expected}».");
}
}
/// <summary>Checks a variables template; collects its values of the case; true when it holds removed names or values.</summary>
private static bool Template(JsonElement element, string where, SortedSet<string> required, int depth)
{
if (depth > 16)
{
throw new InvalidDataException($"{where}: «variables» è troppo profondo.");
}
var removed = false;
switch (element.ValueKind)
{
case JsonValueKind.Object:
foreach (var property in element.EnumerateObject())
{
removed |= property.Name == RemovedName;
removed |= Template(property.Value, where, required, depth + 1);
}
break;
case JsonValueKind.Array:
foreach (var item in element.EnumerateArray())
{
removed |= Template(item, where, required, depth + 1);
}
break;
case JsonValueKind.String:
var text = element.GetString()!;
if (text == RemovedValue)
{
removed = true;
}
else if (text.StartsWith('{') || text.EndsWith('}'))
{
required.Add(CaseValue(text, where, allowOptional: false, out _));
}
break;
}
return removed;
}
private static string CaseValue(string text, string where, bool allowOptional, out bool isOptional)
{
var match = CaseValuePattern().Match(text);
isOptional = match.Success && match.Groups[2].Success;
if (!match.Success || (isOptional && !allowOptional) || !CaseValues.All.Contains(match.Groups[1].Value))
{
throw new InvalidDataException(
$"{where}: «{text}» non è un valore del caso (ammessi: {string.Join(", ", CaseValues.All.Order(StringComparer.Ordinal).Select(v => $"{{{v}}}"))}{(allowOptional ? ", con «?» se facoltativo" : string.Empty)}).");
}
return match.Groups[1].Value;
}
private static ActionKind ParseCategory(string? value, string where) => value?.Trim().ToLowerInvariant() switch
{
"home" => ActionKind.Home,
"profilo" => ActionKind.ProfileOpen,
"storia" => ActionKind.StoryView,
"storico" => ActionKind.BackfillPage,
"post" => ActionKind.SinglePost,
"seguiti" => ActionKind.FollowingListPage,
"ricerca" => ActionKind.ProfileSearch,
_ => throw new InvalidDataException($"{where}: categoria «{value}» sconosciuta (ammesse: home, profilo, storia, storico, post, seguiti, ricerca)."),
};
[GeneratedRegex(@"^[A-Za-z][A-Za-z0-9_.-]{0,127}$", RegexOptions.CultureInvariant)]
private static partial Regex OperationName();
[GeneratedRegex(@"^[A-Za-z0-9/_.{}-]+$", RegexOptions.CultureInvariant)]
private static partial Regex PathCharacters();
[GeneratedRegex(@"^[A-Za-z_][A-Za-z0-9_]{0,63}$", RegexOptions.CultureInvariant)]
private static partial Regex FieldName();
[GeneratedRegex(@"^[a-z][a-z0-9-]{0,63}$", RegexOptions.CultureInvariant)]
private static partial Regex HeaderName();
[GeneratedRegex(@"^[A-Za-z0-9_.-]{1,64}$", RegexOptions.CultureInvariant)]
private static partial Regex CookieName();
[GeneratedRegex(@"^[A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A-Za-z0-9_]*)*$", RegexOptions.CultureInvariant)]
private static partial Regex RootField();
[GeneratedRegex(@"^[0-9]{6}-[a-z0-9-]+$", RegexOptions.CultureInvariant)]
private static partial Regex SampleName();
[GeneratedRegex(@"^\{([a-z_]+)(\?)?\}$", RegexOptions.CultureInvariant)]
private static partial Regex CaseValuePattern();
}
internal sealed class CatalogFile
{
public string? Site { get; set; }
public string? Note { get; set; }
public List<OperationDto?>? Operations { get; set; }
}
internal sealed class OperationDto
{
public string? Name { get; set; }
public string? Kind { get; set; }
public string? Method { get; set; }
public string? Path { get; set; }
public string? Category { get; set; }
public bool? Configured { get; set; }
public string? Reason { get; set; }
public Dictionary<string, string?>? Parameters { get; set; }
public Dictionary<string, string?>? Headers { get; set; }
public Dictionary<string, string?>? Body { get; set; }
public JsonElement? Variables { get; set; }
public List<string?>? RootFields { get; set; }
public string? Sample { get; set; }
public List<string>? Shapes { get; set; }
public string? Note { get; set; }
}
[JsonSourceGenerationOptions(
PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase,
ReadCommentHandling = JsonCommentHandling.Skip,
AllowTrailingCommas = true,
UnmappedMemberHandling = JsonUnmappedMemberHandling.Disallow)]
[JsonSerializable(typeof(CatalogFile))]
internal sealed partial class RequestCatalogJsonContext : JsonSerializerContext;
@@ -0,0 +1,195 @@
using System.Globalization;
using System.Text;
using Febe.Core.Collection.Privacy;
namespace Febe.Core.Collection.Requests;
/// <summary>
/// What the site answered to one <see cref="SiteRequest"/> (issue #36, spec §6.0), with what the stop signals of §7.6
/// need: the status, the declared type, whether it was a redirect and where to (the shape of the path only, never
/// followed), and the names of the cookies the site set or deleted. The body lives in memory only, for the time of the
/// reading (§6.2).
/// </summary>
/// <remarks>
/// It never holds a value of a header or of a cookie: the site client keeps the renewed cookies and the
/// <c>x-ig-www-claim</c> in the saved session itself (decision of 16 November 2026) and reports here only that they
/// came. Its text (<see cref="ToString"/>) shows the operation, the shape of the path, the status, the type and the
/// sizes, never the body, the cookies, a query string or the variables (rule 10).
/// </remarks>
public sealed class SiteAnswer
{
/// <param name="operation">
/// The catalog operation that was asked: its name and its <see cref="SiteOperation.PathShape"/> are what the answer
/// shows, so the path with its values (an identifier, a user name) can never be passed in its place.
/// </param>
/// <param name="status">HTTP status.</param>
/// <param name="mimeType">Declared type (parameters such as the charset are dropped).</param>
/// <param name="body">The body, decoded (memory only).</param>
/// <param name="redirectTo">For a redirect, the <c>Location</c> as the site sent it: only the shape of its path is kept.</param>
/// <param name="cookiesSet">Names of the cookies the site set (<c>Set-Cookie</c> with a value).</param>
/// <param name="cookiesDeleted">Names of the cookies the site deleted (<c>Set-Cookie</c> expired or empty).</param>
/// <param name="setsClaim">True when the site sent a new <c>x-ig-www-claim</c> (<c>x-ig-set-www-claim</c>).</param>
/// <param name="retryAfter">What a <c>Retry-After</c> asked to wait, when the site sent one.</param>
/// <exception cref="ArgumentException">A cookie name that is not a plain name (a value must never slip in).</exception>
public SiteAnswer(
SiteOperation operation,
int status,
string? mimeType,
ReadOnlyMemory<byte> body,
string? redirectTo = null,
IEnumerable<string>? cookiesSet = null,
IEnumerable<string>? cookiesDeleted = null,
bool setsClaim = false,
TimeSpan? retryAfter = null)
{
ArgumentNullException.ThrowIfNull(operation);
Operation = operation.Name;
PathShape = operation.PathShape;
Status = status;
MimeType = (mimeType ?? string.Empty).Split(';')[0].Trim().ToLowerInvariant();
Body = body;
IsRedirect = status is >= 300 and < 400;
RedirectPathShape = IsRedirect || redirectTo is not null ? ShapeOfLocation(redirectTo) : null;
CookiesSet = Names(cookiesSet, nameof(cookiesSet));
CookiesDeleted = Names(cookiesDeleted, nameof(cookiesDeleted));
SetsClaim = setsClaim;
RetryAfter = retryAfter;
}
/// <summary>Name of the catalog operation that was asked.</summary>
public string Operation { get; }
/// <summary>Shape of the path asked (<c>/api/v1/media/{numero}/info/</c>): what the log may show.</summary>
public string PathShape { get; }
public int Status { get; }
/// <summary>Declared type, lower case, without parameters.</summary>
public string MimeType { get; }
/// <summary>The body (memory only, never logged).</summary>
public ReadOnlyMemory<byte> Body { get; }
/// <summary>True for a 3xx: a signal (§7.6), never an address to visit.</summary>
public bool IsRedirect { get; }
/// <summary>
/// For a redirect, the shape of the path it pointed to (<c>/accounts/login/</c>, <c>/challenge/{testo}/</c>):
/// words of the site kept, everything else <c>{numero}</c> or <c>{testo}</c>, no host, no query string. Empty when
/// the site sent no <c>Location</c>.
/// </summary>
public string? RedirectPathShape { get; }
/// <summary>Names of the cookies the site set.</summary>
public IReadOnlyList<string> CookiesSet { get; }
/// <summary>Names of the cookies the site deleted (§7.6: the session cookie deleted is a Disconnessione).</summary>
public IReadOnlyList<string> CookiesDeleted { get; }
/// <summary>True when the site sent a new <c>x-ig-www-claim</c> (kept by the site client in the saved session, never here).</summary>
public bool SetsClaim { get; }
/// <summary>What a <c>Retry-After</c> asked to wait (§7.6, Rallentamento).</summary>
public TimeSpan? RetryAfter { get; }
/// <summary>True when the declared type is JSON, or JavaScript as Instagram serves its GraphQL data (<see cref="DataResponses.IsData"/>).</summary>
public bool IsDeclaredData => DataResponses.IsData(MimeType, "Fetch");
/// <summary>True for an HTML document (§7.6: an HTML where the catalog expects JSON stops the session).</summary>
public bool IsHtml => MimeType is "text/html" or "application/xhtml+xml";
/// <summary>True when the site deleted the cookie <paramref name="name"/>.</summary>
public bool Deletes(string name) => CookiesDeleted.Contains(name, StringComparer.Ordinal);
/// <summary>
/// The shape of the path of a <c>Location</c> (absolute or relative): no scheme, no host, no query string, values
/// replaced by <see cref="PathPatterns"/>. Empty for a missing one.
/// </summary>
public static string ShapeOfLocation(string? location)
{
if (string.IsNullOrWhiteSpace(location))
{
return string.Empty;
}
var text = location.Trim();
var cut = text.IndexOfAny(['?', '#']);
if (cut >= 0)
{
text = text[..cut];
}
var scheme = text.IndexOf("://", StringComparison.Ordinal);
if (scheme >= 0)
{
var slash = text.IndexOf('/', scheme + 3);
text = slash >= 0 ? text[slash..] : "/";
}
else if (text.StartsWith("//", StringComparison.Ordinal))
{
var slash = text.IndexOf('/', 2);
text = slash >= 0 ? text[slash..] : "/";
}
if (!text.StartsWith('/'))
{
text = "/" + text;
}
return PathPatterns.Of(text);
}
/// <summary>Operation, path shape, status, type and sizes: never the body, the cookies or a value.</summary>
public override string ToString()
{
var text = new StringBuilder();
text.Append(CultureInfo.InvariantCulture, $"Risposta a «{Operation}» ({PathShape}): stato {Status}");
text.Append(MimeType.Length == 0 ? ", senza tipo" : $", {MimeType}");
text.Append(CultureInfo.InvariantCulture, $", {Body.Length} byte");
if (IsRedirect)
{
text.Append(RedirectPathShape is { Length: > 0 } shape ? $", rinvio a {shape} (non seguito)" : ", rinvio senza indirizzo (non seguito)");
}
if (CookiesSet.Count > 0 || CookiesDeleted.Count > 0)
{
text.Append(CultureInfo.InvariantCulture, $", cookie impostati {CookiesSet.Count}, cancellati {CookiesDeleted.Count}");
}
if (SetsClaim)
{
text.Append(", nuovo claim");
}
if (RetryAfter is { } wait)
{
text.Append(CultureInfo.InvariantCulture, $", attendere {(long)wait.TotalSeconds} s");
}
return text.Append('.').ToString();
}
private static string[] Names(IEnumerable<string>? names, string parameter)
{
if (names is null)
{
return [];
}
var list = new List<string>();
foreach (var name in names)
{
if (string.IsNullOrEmpty(name) || name.Any(c => c is '=' or ';' or ',' or '"' || char.IsWhiteSpace(c) || char.IsControl(c)))
{
throw new ArgumentException("Un nome di cookie non è un nome semplice: qui vanno solo i nomi, mai i valori.", parameter);
}
if (!list.Contains(name, StringComparer.Ordinal))
{
list.Add(name);
}
}
return [.. list];
}
}
@@ -0,0 +1,120 @@
using System.Collections.Frozen;
using Febe.Core.Collection.Privacy;
namespace Febe.Core.Collection.Requests;
/// <summary>
/// Names of the values of the case that a <see cref="SiteRequest"/> may carry (spec §6.0: «variabili con i soli valori
/// del caso»): who or what is asked for, and where a list goes on. A closed list: everything else a request carries
/// comes from the catalog (<see cref="RequestCatalog"/>) or from the saved session, never from the caller.
/// </summary>
public static class CaseValues
{
/// <summary>Numeric identifier of an account on the site.</summary>
public const string AccountId = "account_id";
/// <summary>User name of an account.</summary>
public const string Username = "username";
/// <summary>Cursor of the next page, as the previous answer gave it.</summary>
public const string Cursor = "cursor";
/// <summary>Text typed in the search box.</summary>
public const string SearchText = "search_text";
/// <summary>Numeric identifier of a content (post, reel, story).</summary>
public const string MediaId = "media_id";
/// <summary>Short code of a post, as in its address.</summary>
public const string Code = "code";
/// <summary>Identifier of a highlight (collection of stories).</summary>
public const string HighlightId = "highlight_id";
/// <summary>Every name of the list.</summary>
public static IReadOnlySet<string> All { get; } = new[] { AccountId, Username, Cursor, SearchText, MediaId, Code, HighlightId }
.ToFrozenSet(StringComparer.Ordinal);
/// <summary>The shape a value takes in a logged path (<see cref="PathPatterns"/>): never the value itself.</summary>
public static string ShapeOf(string name) => name is AccountId or MediaId ? PathPatterns.Number : PathPatterns.Text;
/// <summary>
/// True when <paramref name="value"/> has the form of the value <paramref name="name"/>: digits for identifiers,
/// the characters of a user name, a short code or a highlight; for a cursor or a searched text, no control
/// character. So a value can never carry a path, a query or a step up into the request it fills.
/// </summary>
public static bool HasForm(string name, string value)
{
ArgumentNullException.ThrowIfNull(name);
ArgumentNullException.ThrowIfNull(value);
return name switch
{
AccountId or MediaId => value.Length <= 32 && value.Length > 0 && value.All(char.IsAsciiDigit),
Username => value.Length <= 64 && value.Trim('.').Length > 0 && value.All(c => char.IsAsciiLetterOrDigit(c) || c is '.' or '_'),
Code => value.Length is > 0 and <= 64 && value.All(c => char.IsAsciiLetterOrDigit(c) || c is '_' or '-'),
HighlightId => value.Length is > 0 and <= 64 && value.All(c => char.IsAsciiLetterOrDigit(c) || c is '_' or '-' or ':'),
Cursor or SearchText => value.Length is > 0 and <= 4096 && !value.Any(char.IsControl),
_ => false,
};
}
}
/// <summary>
/// One request to the site (issue #36, spec §6.0): which operation of the catalog, and the values of the case by name.
/// Nothing else: method, path, headers, body and constants come from <c>richieste.&lt;sito&gt;.json</c>, the session
/// values from the saved session. Callers never pass raw headers, bodies or addresses.
/// </summary>
/// <remarks>
/// Its text (<see cref="ToString"/>) names the operation and the names of the values, never the values: user names,
/// identifiers, cursors and searched texts stay out of logs (rule 10).
/// </remarks>
public sealed class SiteRequest
{
/// <param name="operation">Name of the catalog operation (<see cref="SiteOperation.Name"/>).</param>
/// <param name="values">Values of the case, by a name of <see cref="CaseValues"/>.</param>
/// <exception cref="ArgumentException">An unknown value name, an empty value, or one without its form (<see cref="CaseValues.HasForm"/>).</exception>
public SiteRequest(string operation, IReadOnlyDictionary<string, string>? values = null)
{
ArgumentException.ThrowIfNullOrWhiteSpace(operation);
var copy = new Dictionary<string, string>(StringComparer.Ordinal);
foreach (var (name, value) in values ?? new Dictionary<string, string>())
{
if (!CaseValues.All.Contains(name))
{
throw new ArgumentException($"Valore «{name}» sconosciuto: una richiesta porta solo {string.Join(", ", CaseValues.All.Order(StringComparer.Ordinal))}.", nameof(values));
}
if (string.IsNullOrEmpty(value))
{
throw new ArgumentException($"Il valore «{name}» è vuoto.", nameof(values));
}
if (!CaseValues.HasForm(name, value))
{
// Never the value in the message: it may be a user name or a searched text (rule 10).
throw new ArgumentException($"Il valore «{name}» non ha la forma attesa (cifre per gli identificativi; mai un percorso o una query).", nameof(values));
}
copy[name] = value;
}
Operation = operation.Trim();
Values = copy.ToFrozenDictionary(StringComparer.Ordinal);
}
/// <summary>Name of the catalog operation.</summary>
public string Operation { get; }
/// <summary>Values of the case, by name (never logged).</summary>
public IReadOnlyDictionary<string, string> Values { get; }
/// <summary>A request for <paramref name="operation"/> with the given values.</summary>
public static SiteRequest For(string operation, params (string Name, string Value)[] values) =>
new(operation, values.ToDictionary(v => v.Name, v => v.Value, StringComparer.Ordinal));
/// <summary>The operation and the names of the values, never the values.</summary>
public override string ToString() =>
Values.Count == 0
? $"Richiesta «{Operation}»"
: $"Richiesta «{Operation}» (valori: {string.Join(", ", Values.Keys.Order(StringComparer.Ordinal))})";
}
@@ -0,0 +1,110 @@
using System.Text;
using System.Text.Json;
namespace Febe.Core.Collection;
/// <summary>
/// Hands the body of one site answer to the <see cref="IResponseConsumer"/>s (spec §6.2, §6.4), whatever brought it:
/// the response tap of the browser, or the site client of issue #36 (§6.0). Captured and requested answers reach the
/// readers and the stop signals the same way.
/// </summary>
/// <remarks>
/// A data body is parsed as one JSON answer; a page document is split into the JSON blocks it embeds
/// (<see cref="DataResponses.EmbeddedJsonBlocks"/>), each an answer of its own. One answer at a time: each
/// <see cref="JsonDocument"/> is disposed before the next is parsed. A consumer that fails stops neither the others nor
/// the next answer. Bodies stay in memory and are never logged: the warnings name only the consumer's type.
/// Thread-safe (the counters are atomic); the caller decides the order of the answers.
/// </remarks>
public sealed class ResponseDelivery
{
/// <summary>The depth a body may reach: Instagram's answers go deeper than the default 64.</summary>
public const int MaxDepth = 256;
private static readonly JsonDocumentOptions BodyOptions = new() { MaxDepth = MaxDepth };
private readonly string _logPrefix;
private readonly Action<string, Exception?> _warn;
private long _jsonBodies;
private long _parseErrors;
/// <param name="logPrefix">First word of the warnings, in Italian («Cattura», «Client del sito»).</param>
/// <param name="warn">Receives the warnings (message and exception); null discards them.</param>
public ResponseDelivery(string logPrefix, Action<string, Exception?>? warn = null)
{
ArgumentException.ThrowIfNullOrWhiteSpace(logPrefix);
_logPrefix = logPrefix;
_warn = warn ?? ((_, _) => { });
}
/// <summary>JSON answers handed to the consumers (a page document counts once per block).</summary>
public long JsonBodies => Interlocked.Read(ref _jsonBodies);
/// <summary>Data bodies that were not JSON after all (e.g. Instagram's «for (;;);» answers, or deeper than <see cref="MaxDepth"/>).</summary>
public long ParseErrors => Interlocked.Read(ref _parseErrors);
/// <summary>
/// The body of a data answer, as one answer. Returns how many answers reached the consumers: 0 when it is not
/// JSON (the caller tells it as <see cref="BodyDropReason.NotJson"/>), otherwise 1.
/// </summary>
public async Task<int> DeliverBodyAsync(ResponseDescriptor descriptor, ReadOnlyMemory<byte> body, IReadOnlyList<IResponseConsumer> consumers, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(descriptor);
ArgumentNullException.ThrowIfNull(consumers);
JsonDocument document;
try
{
document = JsonDocument.Parse(body, BodyOptions);
}
catch (JsonException)
{
// Not JSON after all (e.g. Instagram's «for (;;);» answers of /ajax/, which carry no contents): skipped.
Interlocked.Increment(ref _parseErrors);
return 0;
}
using (document)
{
await DeliverAsync(descriptor, document, consumers, ct).ConfigureAwait(false);
return 1;
}
}
/// <summary>
/// The JSON blocks a page document embeds, each handed to the consumers as an answer of its own with the document's
/// descriptor, so that a pagination stays with the block that carries it. Returns how many blocks reached the
/// consumers; a document without any (the caller tells it as <see cref="BodyDropReason.NoJsonBlock"/>) is not data,
/// and not a parse error.
/// </summary>
public async Task<int> DeliverDocumentAsync(ResponseDescriptor descriptor, ReadOnlyMemory<byte> document, IReadOnlyList<IResponseConsumer> consumers, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(descriptor);
ArgumentNullException.ThrowIfNull(consumers);
var delivered = 0;
foreach (var block in DataResponses.EmbeddedJsonBlocks(Encoding.UTF8.GetString(document.Span)))
{
using (block)
{
delivered++;
await DeliverAsync(descriptor, block, consumers, ct).ConfigureAwait(false);
}
}
return delivered;
}
private async Task DeliverAsync(ResponseDescriptor descriptor, JsonDocument body, IReadOnlyList<IResponseConsumer> consumers, CancellationToken ct)
{
Interlocked.Increment(ref _jsonBodies);
foreach (var consumer in consumers)
{
try
{
await consumer.ConsumeAsync(descriptor, body, ct).ConfigureAwait(false);
}
catch (Exception ex) when (ex is not OperationCanceledException || !ct.IsCancellationRequested)
{
_warn($"{_logPrefix}: errore in un lettore ({consumer.GetType().Name}).", ex);
}
}
}
}
@@ -0,0 +1,750 @@
{
"site": "instagram",
"note": "Operazioni che il client del sito può chiedere a Instagram (issue #36, specifica §6.0). Scritte il 16 novembre 2026 dalle forme reali delle richieste ricavate con «febe-sonda richieste» dai file HAR del 12 novembre (con e senza scorrimento) e del 16 novembre, fuori dal repository e senza valori; le risposte sono documentate dai campioni ripuliti di tests/Febe.Tests/campioni/instagram (regola 9). Solo letture: il codice rifiuta ogni «Mutation», la telemetria, la navigazione della pagina e le famiglie di chat, messaggi, presenza e promozioni, anche se una copia modificata le elencasse. Nessun doc_id qui: si legge dai moduli della pagina all'accesso (tappa S4). Fonti dei valori: constant (dalle forme), cookie:csrftoken, session:… (copiato a «Ho finito»: user_agent, lsd, fb_dtsg, app_id, claim, account_id, doc_id), operation, variables, computed, pending (nome noto, fonte ancora da stabilire: tappa S4). Le forme scrivono i vero/falso del JSON come true/false, ma non distinguono un numero da un testo di cifre: i numeri delle variabili (count 12, first 3) sono scritti come numeri, da confermare. «(nome tolto)» e «(valore tolto)» sono ciò che la sonda non mostra: un'operazione che li ha nelle variabili resta non configurata. L'elenco dei seguiti manca: nessuna forma con la risposta.",
"operations": [
{
"name": "PolarisSearchBoxRefetchableQuery",
"kind": "graphql",
"method": "POST",
"path": "/api/graphql",
"category": "ricerca",
"configured": false,
"reason": "le forme non mostrano i valori di context, include_reel e search_surface (testi, che la sonda toglie), né da dove vengono rank_token e search_session_id",
"headers": {
"(nome tolto)": "pending",
"accept": "constant:*/*",
"content-length": "computed",
"content-type": "constant:application/x-www-form-urlencoded",
"priority": "pending",
"referer": "pending",
"sec-ch-ua": "pending",
"sec-ch-ua-mobile": "pending",
"sec-ch-ua-platform": "pending",
"sec-fetch-dest": "constant:empty",
"sec-fetch-mode": "constant:cors",
"sec-fetch-site": "constant:same-origin",
"user-agent": "session:user_agent",
"x-asbd-id": "constant:359341",
"x-csrftoken": "cookie:csrftoken",
"x-fb-entry-nonce": "pending",
"x-fb-friendly-name": "operation",
"x-fb-lsd": "session:lsd",
"x-ig-app-id": "session:app_id",
"x-ig-max-touch-points": "pending"
},
"body": {
"(nome tolto)": "pending",
"__a": "constant:1",
"__ccg": "pending",
"__comet_req": "constant:7",
"__crn": "pending",
"__csr": "pending",
"__d": "pending",
"__dyn": "pending",
"__hblp": "pending",
"__hs": "pending",
"__hsdp": "pending",
"__hsi": "pending",
"__req": "pending",
"__rev": "pending",
"__s": "pending",
"__sjsp": "pending",
"__spin_b": "pending",
"__spin_r": "pending",
"__spin_t": "pending",
"__user": "session:account_id",
"av": "session:account_id",
"doc_id": "session:doc_id",
"dpr": "constant:1",
"fb_api_caller_class": "constant:RelayModern",
"fb_api_req_friendly_name": "operation",
"fb_dtsg": "session:fb_dtsg",
"server_timestamps": "constant:true",
"variables": "variables"
},
"variables": {
"data": {
"context": "(valore tolto)",
"include_reel": "(valore tolto)",
"query": "{search_text}",
"rank_token": "(valore tolto)",
"search_session_id": "(valore tolto)",
"search_surface": "(valore tolto)"
},
"hasQuery": true
},
"rootFields": [
"data.xdt_api__v1__fbsearch__topsearch_connection"
],
"sample": "000002-ricerca-profili",
"shapes": [
"2026-11-12",
"2026-11-16"
],
"note": "include_reel è un testo, non un vero/falso (la sonda scrive i vero/falso del JSON come true/false). La x-ig-max-touch-points c'è solo nelle forme del 12/11."
},
{
"name": "PolarisProfilePageContentQuery",
"kind": "graphql",
"method": "POST",
"path": "/api/graphql",
"category": "profilo",
"configured": false,
"reason": "una variabile della forma ha il nome tolto dalla sonda («(nome tolto)»): senza il nome vero la richiesta non si scrive (regola 9); la variabile senza nome è un vero/falso di valore non noto",
"headers": {
"(nome tolto)": "pending",
"accept": "constant:*/*",
"content-length": "computed",
"content-type": "constant:application/x-www-form-urlencoded",
"priority": "pending",
"referer": "pending",
"sec-ch-ua": "pending",
"sec-ch-ua-mobile": "pending",
"sec-ch-ua-platform": "pending",
"sec-fetch-dest": "constant:empty",
"sec-fetch-mode": "constant:cors",
"sec-fetch-site": "constant:same-origin",
"user-agent": "session:user_agent",
"x-asbd-id": "constant:359341",
"x-csrftoken": "cookie:csrftoken",
"x-fb-entry-nonce": "pending",
"x-fb-friendly-name": "operation",
"x-fb-lsd": "session:lsd",
"x-ig-app-id": "session:app_id",
"x-ig-max-touch-points": "pending"
},
"body": {
"(nome tolto)": "pending",
"__a": "constant:1",
"__ccg": "pending",
"__comet_req": "constant:7",
"__crn": "pending",
"__csr": "pending",
"__d": "pending",
"__dyn": "pending",
"__hblp": "pending",
"__hs": "pending",
"__hsdp": "pending",
"__hsi": "pending",
"__req": "pending",
"__rev": "pending",
"__s": "pending",
"__sjsp": "pending",
"__spin_b": "pending",
"__spin_r": "pending",
"__spin_t": "pending",
"__user": "session:account_id",
"av": "session:account_id",
"doc_id": "session:doc_id",
"dpr": "constant:1",
"fb_api_caller_class": "constant:RelayModern",
"fb_api_req_friendly_name": "operation",
"fb_dtsg": "session:fb_dtsg",
"server_timestamps": "constant:true",
"variables": "variables"
},
"variables": {
"(nome tolto)": "(valore tolto)",
"__relay_internal__pv__PolarisShortDramaEnabledrelayprovider": true,
"__relay_internal__pv__PolarisWebSchoolsEnabledrelayprovider": false,
"enable_integrity_filters": true,
"id": "{account_id}"
},
"rootFields": [
"data.user"
],
"sample": "000009-scheda-profilo",
"shapes": [
"2026-11-12",
"2026-11-16"
]
},
{
"name": "PolarisProfilePostsQuery",
"kind": "graphql",
"method": "POST",
"path": "/api/graphql",
"category": "storico",
"configured": false,
"reason": "una variabile della forma ha il nome tolto dalla sonda («(nome tolto)»): senza il nome vero la richiesta non si scrive (regola 9)",
"headers": {
"(nome tolto)": "pending",
"accept": "constant:*/*",
"content-length": "computed",
"content-type": "constant:application/x-www-form-urlencoded",
"priority": "pending",
"referer": "pending",
"sec-ch-ua": "pending",
"sec-ch-ua-mobile": "pending",
"sec-ch-ua-platform": "pending",
"sec-fetch-dest": "constant:empty",
"sec-fetch-mode": "constant:cors",
"sec-fetch-site": "constant:same-origin",
"user-agent": "session:user_agent",
"x-asbd-id": "constant:359341",
"x-csrftoken": "cookie:csrftoken",
"x-fb-entry-nonce": "pending",
"x-fb-friendly-name": "operation",
"x-fb-lsd": "session:lsd",
"x-ig-app-id": "session:app_id",
"x-ig-max-touch-points": "pending"
},
"body": {
"(nome tolto)": "pending",
"__a": "constant:1",
"__ccg": "pending",
"__comet_req": "constant:7",
"__crn": "pending",
"__csr": "pending",
"__d": "pending",
"__dyn": "pending",
"__hblp": "pending",
"__hs": "pending",
"__hsdp": "pending",
"__hsi": "pending",
"__req": "pending",
"__rev": "pending",
"__s": "pending",
"__sjsp": "pending",
"__spin_b": "pending",
"__spin_r": "pending",
"__spin_t": "pending",
"__user": "session:account_id",
"av": "session:account_id",
"doc_id": "session:doc_id",
"dpr": "constant:1",
"fb_api_caller_class": "constant:RelayModern",
"fb_api_req_friendly_name": "operation",
"fb_dtsg": "session:fb_dtsg",
"server_timestamps": "constant:true",
"variables": "variables"
},
"variables": {
"(nome tolto)": false,
"__relay_internal__pv__PolarisSEOIsCrawlerrelayprovider": false,
"__relay_internal__pv__PolarisShortDramaEnabledrelayprovider": true,
"data": {
"count": 12,
"include_reel_media_seen_timestamp": true,
"include_relationship_info": true,
"latest_besties_reel_media": true,
"latest_reel_media": true
},
"username": "{username}"
},
"rootFields": [
"data.xdt_api__v1__feed__user_timeline_graphql_connection"
],
"sample": "000010-post-del-profilo",
"shapes": [
"2026-11-12",
"2026-11-16"
],
"note": "Solo la prima pagina: le forme non hanno ancora una pagina successiva (nessun cursore nelle variabili). Anche 000016-serie-con-video viene da questa operazione (stesso indirizzo di 000010 nel descrittore)."
},
{
"name": "PolarisProfileReelsTabContentQuery",
"kind": "graphql",
"method": "POST",
"path": "/api/graphql",
"category": "storico",
"configured": false,
"reason": "le variabili sono note, ma intestazioni e campi della pagina e del browser (come __dyn, __csr, referer, sec-ch-ua e quello con il nome tolto dalla sonda) hanno ancora la fonte «pending»: finché non si sa da dove vengono i valori la richiesta non si scrive (regola 9, tappa S4)",
"headers": {
"(nome tolto)": "pending",
"accept": "constant:*/*",
"content-length": "computed",
"content-type": "constant:application/x-www-form-urlencoded",
"priority": "pending",
"referer": "pending",
"sec-ch-ua": "pending",
"sec-ch-ua-mobile": "pending",
"sec-ch-ua-platform": "pending",
"sec-fetch-dest": "constant:empty",
"sec-fetch-mode": "constant:cors",
"sec-fetch-site": "constant:same-origin",
"user-agent": "session:user_agent",
"x-asbd-id": "constant:359341",
"x-csrftoken": "cookie:csrftoken",
"x-fb-entry-nonce": "pending",
"x-fb-friendly-name": "operation",
"x-fb-lsd": "session:lsd",
"x-ig-app-id": "session:app_id",
"x-ig-max-touch-points": "pending"
},
"body": {
"(nome tolto)": "pending",
"__a": "constant:1",
"__ccg": "pending",
"__comet_req": "constant:7",
"__crn": "pending",
"__csr": "pending",
"__d": "pending",
"__dyn": "pending",
"__hblp": "pending",
"__hs": "pending",
"__hsdp": "pending",
"__hsi": "pending",
"__req": "pending",
"__rev": "pending",
"__s": "pending",
"__sjsp": "pending",
"__spin_b": "pending",
"__spin_r": "pending",
"__spin_t": "pending",
"__user": "session:account_id",
"av": "session:account_id",
"doc_id": "session:doc_id",
"dpr": "constant:1",
"fb_api_caller_class": "constant:RelayModern",
"fb_api_req_friendly_name": "operation",
"fb_dtsg": "session:fb_dtsg",
"server_timestamps": "constant:true",
"variables": "variables"
},
"variables": {
"__relay_internal__pv__PolarisShortDramaEnabledrelayprovider": true,
"data": {
"include_feed_video": true,
"page_size": 12,
"target_user_id": "{account_id}"
},
"user_id": "{account_id}"
},
"rootFields": [
"data.fetch__XDTUserDict.clips_connection"
],
"sample": "000011-reel-del-profilo",
"shapes": [
"2026-11-12"
],
"note": "Solo la prima pagina. Vista solo nelle forme del 12/11."
},
{
"name": "PolarisProfileTaggedTabContentQuery",
"kind": "graphql",
"method": "POST",
"path": "/api/graphql",
"category": "storico",
"configured": false,
"reason": "le variabili sono note, ma intestazioni e campi della pagina e del browser (come __dyn, __csr, referer, sec-ch-ua e quello con il nome tolto dalla sonda) hanno ancora la fonte «pending»: finché non si sa da dove vengono i valori la richiesta non si scrive (regola 9, tappa S4)",
"headers": {
"(nome tolto)": "pending",
"accept": "constant:*/*",
"content-length": "computed",
"content-type": "constant:application/x-www-form-urlencoded",
"priority": "pending",
"referer": "pending",
"sec-ch-ua": "pending",
"sec-ch-ua-mobile": "pending",
"sec-ch-ua-platform": "pending",
"sec-fetch-dest": "constant:empty",
"sec-fetch-mode": "constant:cors",
"sec-fetch-site": "constant:same-origin",
"user-agent": "session:user_agent",
"x-asbd-id": "constant:359341",
"x-csrftoken": "cookie:csrftoken",
"x-fb-entry-nonce": "pending",
"x-fb-friendly-name": "operation",
"x-fb-lsd": "session:lsd",
"x-ig-app-id": "session:app_id",
"x-ig-max-touch-points": "pending"
},
"body": {
"(nome tolto)": "pending",
"__a": "constant:1",
"__ccg": "pending",
"__comet_req": "constant:7",
"__crn": "pending",
"__csr": "pending",
"__d": "pending",
"__dyn": "pending",
"__hblp": "pending",
"__hs": "pending",
"__hsdp": "pending",
"__hsi": "pending",
"__req": "pending",
"__rev": "pending",
"__s": "pending",
"__sjsp": "pending",
"__spin_b": "pending",
"__spin_r": "pending",
"__spin_t": "pending",
"__user": "session:account_id",
"av": "session:account_id",
"doc_id": "session:doc_id",
"dpr": "constant:1",
"fb_api_caller_class": "constant:RelayModern",
"fb_api_req_friendly_name": "operation",
"fb_dtsg": "session:fb_dtsg",
"server_timestamps": "constant:true",
"variables": "variables"
},
"variables": {
"__relay_internal__pv__PolarisSEOIsCrawlerrelayprovider": false,
"__relay_internal__pv__PolarisShortDramaEnabledrelayprovider": true,
"count": 12,
"user_id": "{account_id}"
},
"rootFields": [
"data.xdt_api__v1__usertags__user_id__feed_connection"
],
"sample": "000012-con-tag",
"shapes": [
"2026-11-12"
],
"note": "Solo la prima pagina. Vista solo nelle forme del 12/11."
},
{
"name": "PolarisProfileStoryHighlightsTrayContentQuery",
"kind": "graphql",
"method": "POST",
"path": "/api/graphql",
"category": "profilo",
"configured": false,
"reason": "le variabili sono note, ma intestazioni e campi della pagina e del browser (come __dyn, __csr, referer, sec-ch-ua e quello con il nome tolto dalla sonda) hanno ancora la fonte «pending»: finché non si sa da dove vengono i valori la richiesta non si scrive (regola 9, tappa S4)",
"headers": {
"(nome tolto)": "pending",
"accept": "constant:*/*",
"content-length": "computed",
"content-type": "constant:application/x-www-form-urlencoded",
"priority": "pending",
"referer": "pending",
"sec-ch-ua": "pending",
"sec-ch-ua-mobile": "pending",
"sec-ch-ua-platform": "pending",
"sec-fetch-dest": "constant:empty",
"sec-fetch-mode": "constant:cors",
"sec-fetch-site": "constant:same-origin",
"user-agent": "session:user_agent",
"x-asbd-id": "constant:359341",
"x-csrftoken": "cookie:csrftoken",
"x-fb-entry-nonce": "pending",
"x-fb-friendly-name": "operation",
"x-fb-lsd": "session:lsd",
"x-ig-app-id": "session:app_id",
"x-ig-max-touch-points": "pending"
},
"body": {
"(nome tolto)": "pending",
"__a": "constant:1",
"__ccg": "pending",
"__comet_req": "constant:7",
"__crn": "pending",
"__csr": "pending",
"__d": "pending",
"__dyn": "pending",
"__hblp": "pending",
"__hs": "pending",
"__hsdp": "pending",
"__hsi": "pending",
"__req": "pending",
"__rev": "pending",
"__s": "pending",
"__sjsp": "pending",
"__spin_b": "pending",
"__spin_r": "pending",
"__spin_t": "pending",
"__user": "session:account_id",
"av": "session:account_id",
"doc_id": "session:doc_id",
"dpr": "constant:1",
"fb_api_caller_class": "constant:RelayModern",
"fb_api_req_friendly_name": "operation",
"fb_dtsg": "session:fb_dtsg",
"server_timestamps": "constant:true",
"variables": "variables"
},
"variables": {
"user_id": "{account_id}"
},
"rootFields": [
"data.highlights"
],
"sample": "000013-raccolte-del-profilo",
"shapes": [
"2026-11-12",
"2026-11-16"
]
},
{
"name": "PolarisStoriesV3ReelPageStandaloneQuery",
"kind": "graphql",
"method": "POST",
"path": "/api/graphql",
"category": "storia",
"configured": false,
"reason": "una variabile della forma ha il nome tolto dalla sonda («(nome tolto)»): senza il nome vero la richiesta non si scrive (regola 9)",
"headers": {
"(nome tolto)": "pending",
"accept": "constant:*/*",
"content-length": "computed",
"content-type": "constant:application/x-www-form-urlencoded",
"priority": "pending",
"referer": "pending",
"sec-ch-ua": "pending",
"sec-ch-ua-mobile": "pending",
"sec-ch-ua-platform": "pending",
"sec-fetch-dest": "constant:empty",
"sec-fetch-mode": "constant:cors",
"sec-fetch-site": "constant:same-origin",
"user-agent": "session:user_agent",
"x-asbd-id": "constant:359341",
"x-csrftoken": "cookie:csrftoken",
"x-fb-entry-nonce": "pending",
"x-fb-friendly-name": "operation",
"x-fb-lsd": "session:lsd",
"x-ig-app-id": "session:app_id",
"x-ig-max-touch-points": "pending"
},
"body": {
"(nome tolto)": "pending",
"__a": "constant:1",
"__ccg": "pending",
"__comet_req": "constant:7",
"__crn": "pending",
"__csr": "pending",
"__d": "pending",
"__dyn": "pending",
"__hblp": "pending",
"__hs": "pending",
"__hsdp": "pending",
"__hsi": "pending",
"__req": "pending",
"__rev": "pending",
"__s": "pending",
"__sjsp": "pending",
"__spin_b": "pending",
"__spin_r": "pending",
"__spin_t": "pending",
"__user": "session:account_id",
"av": "session:account_id",
"doc_id": "session:doc_id",
"dpr": "constant:1",
"fb_api_analytics_tags": "pending",
"fb_api_caller_class": "constant:RelayModern",
"fb_api_req_friendly_name": "operation",
"fb_dtsg": "session:fb_dtsg",
"qpl_active_flow_ids": "pending",
"server_timestamps": "constant:true",
"variables": "variables"
},
"variables": {
"(nome tolto)": true,
"reel_ids_arr": [
"{account_id}"
]
},
"rootFields": [
"data.xdt_api__v1__feed__reels_media"
],
"sample": "000014-storie-di-un-profilo",
"shapes": [
"2026-11-12"
],
"note": "reel_ids_arr è un elenco di numeri: che sia l'identificativo dell'account è da confermare. Vista solo nelle forme del 12/11."
},
{
"name": "PolarisStoriesV3HighlightsPageQuery",
"kind": "graphql",
"method": "POST",
"path": "/graphql/query",
"category": "storia",
"configured": false,
"reason": "una variabile della forma ha il nome tolto dalla sonda («(nome tolto)»): senza il nome vero la richiesta non si scrive (regola 9)",
"headers": {
"(nome tolto)": "pending",
"accept": "constant:*/*",
"content-length": "computed",
"content-type": "constant:application/x-www-form-urlencoded",
"priority": "pending",
"referer": "pending",
"sec-ch-ua": "pending",
"sec-ch-ua-mobile": "pending",
"sec-ch-ua-platform": "pending",
"sec-fetch-dest": "constant:empty",
"sec-fetch-mode": "constant:cors",
"sec-fetch-site": "constant:same-origin",
"user-agent": "session:user_agent",
"x-asbd-id": "constant:359341",
"x-bloks-version-id": "constant:e9f7142825abe6e3e169776ae34e2fc4aeb2fa7ccdf139d43730b2f13505e200",
"x-csrftoken": "cookie:csrftoken",
"x-fb-entry-nonce": "pending",
"x-fb-friendly-name": "operation",
"x-fb-lsd": "session:lsd",
"x-ig-app-id": "session:app_id",
"x-ig-max-touch-points": "pending",
"x-root-field-name": "constant:xdt_api__v1__feed__reels_media__connection"
},
"body": {
"(nome tolto)": "pending",
"__a": "constant:1",
"__ccg": "pending",
"__comet_req": "constant:7",
"__crn": "pending",
"__csr": "pending",
"__d": "pending",
"__dyn": "pending",
"__hblp": "pending",
"__hs": "pending",
"__hsdp": "pending",
"__hsi": "pending",
"__req": "pending",
"__rev": "pending",
"__s": "pending",
"__sjsp": "pending",
"__spin_b": "pending",
"__spin_r": "pending",
"__spin_t": "pending",
"__user": "session:account_id",
"av": "session:account_id",
"doc_id": "session:doc_id",
"dpr": "constant:1",
"fb_api_caller_class": "constant:RelayModern",
"fb_api_req_friendly_name": "operation",
"fb_dtsg": "session:fb_dtsg",
"server_timestamps": "constant:true",
"variables": "variables"
},
"variables": {
"(nome tolto)": true,
"first": 3,
"initial_reel_id": "{highlight_id}",
"last": 2,
"reel_ids": [
"{highlight_id}"
]
},
"rootFields": [
"data.xdt_api__v1__feed__reels_media__connection"
],
"sample": "000006-raccolta-in-evidenza",
"shapes": [
"2026-11-12"
],
"note": "reel_ids è l'elenco delle raccolte che la pagina chiede insieme: la forma ne mostra la struttura, non quante. rootFields dal campo_radice delle forme. Vista solo nelle forme del 12/11."
},
{
"name": "PolarisStoriesV3HighlightsPagePaginationQuery",
"kind": "graphql",
"method": "POST",
"path": "/graphql/query",
"category": "storia",
"configured": false,
"reason": "una variabile della forma ha il nome tolto dalla sonda («(nome tolto)»): senza il nome vero la richiesta non si scrive (regola 9); anche il valore di first non è noto (un numero che cambia)",
"headers": {
"(nome tolto)": "pending",
"accept": "constant:*/*",
"content-length": "computed",
"content-type": "constant:application/x-www-form-urlencoded",
"priority": "pending",
"referer": "pending",
"sec-ch-ua": "pending",
"sec-ch-ua-mobile": "pending",
"sec-ch-ua-platform": "pending",
"sec-fetch-dest": "constant:empty",
"sec-fetch-mode": "constant:cors",
"sec-fetch-site": "constant:same-origin",
"user-agent": "session:user_agent",
"x-asbd-id": "constant:359341",
"x-bloks-version-id": "constant:e9f7142825abe6e3e169776ae34e2fc4aeb2fa7ccdf139d43730b2f13505e200",
"x-csrftoken": "cookie:csrftoken",
"x-fb-entry-nonce": "pending",
"x-fb-friendly-name": "operation",
"x-fb-lsd": "session:lsd",
"x-ig-app-id": "session:app_id",
"x-ig-max-touch-points": "pending",
"x-root-field-name": "constant:xdt_api__v1__feed__reels_media__connection"
},
"body": {
"(nome tolto)": "pending",
"__a": "constant:1",
"__ccg": "pending",
"__comet_req": "constant:7",
"__crn": "pending",
"__csr": "pending",
"__d": "pending",
"__dyn": "pending",
"__hblp": "pending",
"__hs": "pending",
"__hsdp": "pending",
"__hsi": "pending",
"__req": "pending",
"__rev": "pending",
"__s": "pending",
"__sjsp": "pending",
"__spin_b": "pending",
"__spin_r": "pending",
"__spin_t": "pending",
"__user": "session:account_id",
"av": "session:account_id",
"doc_id": "session:doc_id",
"dpr": "constant:1",
"fb_api_analytics_tags": "pending",
"fb_api_caller_class": "constant:RelayModern",
"fb_api_req_friendly_name": "operation",
"fb_dtsg": "session:fb_dtsg",
"qpl_active_flow_ids": "pending",
"server_timestamps": "constant:true",
"variables": "variables"
},
"variables": {
"(nome tolto)": true,
"after": "{cursor}",
"before": null,
"first": "(valore tolto)",
"initial_reel_id": "{highlight_id}",
"is_highlight": true,
"last": null,
"reel_ids": [
"{highlight_id}"
]
},
"rootFields": [
"data.xdt_api__v1__feed__reels_media__connection"
],
"sample": "000007-raccolta-pagina-successiva",
"shapes": [
"2026-11-12"
],
"note": "rootFields dal campo_radice delle forme. Vista solo nelle forme del 12/11."
},
{
"name": "media-info",
"kind": "rest",
"method": "GET",
"path": "/api/v1/media/{media_id}/info/",
"category": "post",
"configured": false,
"reason": "il percorso è noto, ma intestazioni della pagina e del browser (come referer, sec-ch-ua e quella con il nome tolto dalla sonda) hanno ancora la fonte «pending»: finché non si sa da dove vengono i valori la richiesta non si scrive (regola 9, tappa S4)",
"headers": {
"(nome tolto)": "pending",
"accept": "constant:*/*",
"priority": "pending",
"referer": "pending",
"sec-ch-ua": "pending",
"sec-ch-ua-mobile": "pending",
"sec-ch-ua-platform": "pending",
"sec-fetch-dest": "constant:empty",
"sec-fetch-mode": "constant:cors",
"sec-fetch-site": "constant:same-origin",
"user-agent": "session:user_agent",
"x-asbd-id": "constant:359341",
"x-csrftoken": "cookie:csrftoken",
"x-ig-app-id": "session:app_id",
"x-ig-max-touch-points": "pending",
"x-ig-www-claim": "session:claim",
"x-requested-with": "constant:XMLHttpRequest",
"x-web-session-id": "pending"
},
"rootFields": [
"items"
],
"sample": "000005-post-video",
"shapes": [
"2026-11-12"
],
"note": "Un post singolo per identificativo (GET, nessun parametro). Vista solo nelle forme del 12/11, 2 volte."
}
]
}
+199
View File
@@ -0,0 +1,199 @@
{
"site": "prova",
"note": "Operazioni che il client del sito chiederà al sito di prova locale (tools/Febe.SitoDiProva, specifica §14.2) dalla tappa S2 di #36: le stesse risposte JSON che la pagina chiede a /dati/, documentate dai campioni ripuliti di tests/Febe.Tests/campioni/prova. Il sito di prova non controlla le intestazioni; le regole di rifiuto sono le stesse di Instagram (nel codice). La scrittura /dati/ricerche-recenti (POST) non c'è e non può esserci: un indirizzo semplice si legge solo con GET.",
"operations": [
{
"name": "inizio",
"kind": "rest",
"method": "GET",
"path": "/dati/inizio",
"category": "home",
"configured": true,
"headers": {
"accept": "constant:application/json",
"user-agent": "session:user_agent"
},
"rootFields": [
"contenuti"
],
"sample": "000001-inizio"
},
{
"name": "inizio-altri",
"kind": "rest",
"method": "GET",
"path": "/dati/inizio/altri",
"category": "home",
"configured": false,
"reason": "manca il campione ripulito della risposta in tests/Febe.Tests/campioni/prova",
"parameters": {
"dopo": "{cursor}"
},
"headers": {
"accept": "constant:application/json",
"user-agent": "session:user_agent"
}
},
{
"name": "profilo",
"kind": "rest",
"method": "GET",
"path": "/dati/profilo/{username}",
"category": "profilo",
"configured": true,
"parameters": {
"scheda": "constant:post"
},
"headers": {
"accept": "constant:application/json",
"user-agent": "session:user_agent"
},
"rootFields": [
"profilo"
],
"sample": "000002-profilo-pubblico",
"note": "Anche 000004-profilo-privato-non-seguito: lo stato del profilo, senza contenuti."
},
{
"name": "profilo-altri",
"kind": "rest",
"method": "GET",
"path": "/dati/profilo/{username}/altri",
"category": "storico",
"configured": true,
"parameters": {
"dopo": "{cursor}",
"scheda": "constant:post"
},
"headers": {
"accept": "constant:application/json",
"user-agent": "session:user_agent"
},
"rootFields": [
"contenuti",
"pagina"
],
"sample": "000003-profilo-pagina-successiva",
"note": "Anche 000010-pagina-vuota."
},
{
"name": "taggati",
"kind": "rest",
"method": "GET",
"path": "/dati/profilo/{username}",
"category": "storico",
"configured": true,
"parameters": {
"scheda": "constant:taggati"
},
"headers": {
"accept": "constant:application/json",
"user-agent": "session:user_agent"
},
"rootFields": [
"contenuti"
],
"sample": "000009-taggati"
},
{
"name": "reel",
"kind": "rest",
"method": "GET",
"path": "/dati/profilo/{username}",
"category": "storico",
"configured": false,
"reason": "manca il campione ripulito della risposta in tests/Febe.Tests/campioni/prova",
"parameters": {
"scheda": "constant:reel"
},
"headers": {
"accept": "constant:application/json",
"user-agent": "session:user_agent"
}
},
{
"name": "post",
"kind": "rest",
"method": "GET",
"path": "/dati/post/{code}",
"category": "post",
"configured": true,
"headers": {
"accept": "constant:application/json",
"user-agent": "session:user_agent"
},
"rootFields": [
"contenuto"
],
"sample": "000005-post-serie"
},
{
"name": "storie",
"kind": "rest",
"method": "GET",
"path": "/dati/storie/{username}",
"category": "storia",
"configured": true,
"headers": {
"accept": "constant:application/json",
"user-agent": "session:user_agent"
},
"rootFields": [
"storie"
],
"sample": "000006-storie"
},
{
"name": "evidenza",
"kind": "rest",
"method": "GET",
"path": "/dati/evidenza/{highlight_id}",
"category": "storia",
"configured": true,
"headers": {
"accept": "constant:application/json",
"user-agent": "session:user_agent"
},
"rootFields": [
"raccolta"
],
"sample": "000007-raccolta"
},
{
"name": "seguiti",
"kind": "rest",
"method": "GET",
"path": "/dati/seguiti",
"category": "seguiti",
"configured": true,
"parameters": {
"dopo": "{cursor?}"
},
"headers": {
"accept": "constant:application/json",
"user-agent": "session:user_agent"
},
"rootFields": [
"seguiti"
],
"sample": "000008-seguiti",
"note": "La prima pagina senza «dopo»."
},
{
"name": "cerca",
"kind": "rest",
"method": "GET",
"path": "/dati/cerca",
"category": "ricerca",
"configured": false,
"reason": "manca il campione ripulito della risposta in tests/Febe.Tests/campioni/prova",
"parameters": {
"testo": "{search_text}"
},
"headers": {
"accept": "constant:application/json",
"user-agent": "session:user_agent"
}
}
]
}
@@ -0,0 +1,374 @@
using System.Text.Json;
using System.Text.Json.Nodes;
using System.Text.RegularExpressions;
using Febe.Core.Abstractions;
using Febe.Core.Collection;
using Febe.Core.Collection.Requests;
namespace Febe.Tests.Collection;
/// <summary>
/// The catalog of the site client (issue #36, S1): both shipped files load, every operation is documented by a real
/// cleaned sample, and the refusals live in the code, so an edited copy in the Data folder cannot lift them.
/// </summary>
[TestClass]
public sealed partial class RequestCatalogTests
{
[TestMethod]
public void BothShippedCatalogsLoadAndAreEditableWithoutRecompiling()
{
var instagram = RequestCatalog.LoadInstagram();
var prova = RequestCatalog.LoadTestSite();
Assert.AreEqual("instagram", instagram.Site);
Assert.AreEqual("prova", prova.Site);
Assert.IsTrue(File.Exists(Path.Combine(AppContext.BaseDirectory, "config", "richieste.instagram.json")));
Assert.IsTrue(File.Exists(Path.Combine(AppContext.BaseDirectory, "config", "richieste.prova.json")));
CollectionAssert.AreEquivalent(
new[]
{
"PolarisSearchBoxRefetchableQuery", "PolarisProfilePageContentQuery", "PolarisProfilePostsQuery",
"PolarisProfileReelsTabContentQuery", "PolarisProfileTaggedTabContentQuery",
"PolarisProfileStoryHighlightsTrayContentQuery", "PolarisStoriesV3ReelPageStandaloneQuery",
"PolarisStoriesV3HighlightsPageQuery", "PolarisStoriesV3HighlightsPagePaginationQuery", "media-info",
},
instagram.Operations.Select(o => o.Name).ToArray());
// Only what the shapes tell whole is configured: a variable whose name the probe removed, a value it does not
// show, or a header or field whose source is still pending leaves the operation not configured (rule 9). Every
// Instagram operation has such headers and fields until tappa S4; for four of them only those are missing.
Assert.AreEqual(0, instagram.Configured.Count());
CollectionAssert.AreEquivalent(
new[] { "PolarisProfileReelsTabContentQuery", "PolarisProfileTaggedTabContentQuery", "PolarisProfileStoryHighlightsTrayContentQuery", "media-info" },
instagram.Operations.Where(o => o.NotConfiguredReason!.Contains("pending", StringComparison.Ordinal)).Select(o => o.Name).ToArray());
Assert.IsTrue(instagram.Operations.All(o => o.PendingValues.Count > 0));
Assert.IsTrue(instagram.Operations.Where(o => !o.IsConfigured).All(o => !string.IsNullOrWhiteSpace(o.NotConfiguredReason)));
Assert.IsFalse(instagram.Operations.Any(o => o.Name.Contains("Following", StringComparison.OrdinalIgnoreCase)), "No shapes with answers for the followed list yet.");
Assert.IsTrue(prova.Configured.Count() >= 8);
}
[TestMethod]
public void TheInstagramCatalogHoldsNoDocIdNorAnyLongNumber()
{
// doc_id values are read from the page modules at login (S4): the file names the source only.
var text = SiteMap.ReadEmbeddedText("richieste.instagram.json");
Assert.IsFalse(LongNumber().IsMatch(text), "No identifier, doc_id or application id in the repository.");
foreach (var operation in RequestCatalog.LoadInstagram().Operations.Where(o => o.Kind == OperationKind.GraphQl))
{
Assert.AreEqual(new ValueSource(ValueSourceKind.Session, "doc_id"), operation.Body["doc_id"]);
Assert.AreEqual(new ValueSource(ValueSourceKind.Cookie, "csrftoken"), operation.Headers["x-csrftoken"], "Decision of 16 November.");
Assert.AreEqual(new ValueSource(ValueSourceKind.Session, "user_agent"), operation.Headers["user-agent"]);
Assert.IsFalse(operation.Headers.ContainsKey("cookie"));
}
}
[TestMethod]
[DataRow("instagram")]
[DataRow("prova")]
public void EveryOperationIsDocumentedByACleanedSampleThatHasItsRootFields(string site)
{
var catalog = site == "instagram" ? RequestCatalog.LoadInstagram() : RequestCatalog.LoadTestSite();
var folder = Path.Combine(AppContext.BaseDirectory, "campioni", site);
foreach (var operation in catalog.Operations.Where(o => o.Sample is not null))
{
var file = Path.Combine(folder, operation.Sample + ".json");
Assert.IsTrue(File.Exists(file), $"{operation.Name}: manca {operation.Sample}.");
using var sample = JsonDocument.Parse(File.ReadAllText(file));
Assert.IsTrue(operation.HasRootFields(sample.RootElement.GetProperty("corpo")), $"{operation.Name}: {operation.Sample} non ha {string.Join(", ", operation.RootFields)}.");
Assert.IsTrue(operation.RootFields.Count > 0);
}
foreach (var operation in catalog.Configured)
{
Assert.IsNotNull(operation.Sample, $"{operation.Name} is configured without a sample.");
}
}
[TestMethod]
public void PathShapesNameNoValueAndValuesMustMatch()
{
var media = Get(RequestCatalog.LoadInstagram(), "media-info");
Assert.AreEqual("/api/v1/media/{numero}/info/", media.PathShape);
Assert.AreEqual(ActionKind.SinglePost, media.Category);
Assert.IsNull(media.CheckValues(SiteRequest.For("media-info", (CaseValues.MediaId, "3141592653"))));
StringAssert.Contains(media.CheckValues(new SiteRequest("media-info")), "media_id");
StringAssert.Contains(media.CheckValues(SiteRequest.For("media-info", (CaseValues.MediaId, "1"), (CaseValues.Cursor, "c"))), "cursor");
var following = Get(RequestCatalog.LoadTestSite(), "seguiti");
Assert.IsNull(following.CheckValues(new SiteRequest("seguiti")), "The cursor is optional on the first page.");
Assert.IsNull(following.CheckValues(SiteRequest.For("seguiti", (CaseValues.Cursor, "c12"))));
var reels = Get(RequestCatalog.LoadInstagram(), "PolarisProfileReelsTabContentQuery");
CollectionAssert.AreEquivalent(new[] { CaseValues.AccountId }, reels.RequiredValues.ToArray(), "From the variables template.");
Assert.AreEqual(ActionKind.BackfillPage, reels.Category);
Assert.IsTrue(reels.PendingValues.Count > 0, "Page values and the name removed by the probe wait for S4.");
}
[TestMethod]
[DataRow("PolarisStoriesV3SeenMutation")]
[DataRow("usePolarisLikeMediaXIGLikeMutation")]
[DataRow("usePolarisRegisterInRecentSearchesMutation")]
[DataRow("PolarisFollowMUTATION")]
[DataRow("SomethingMutation_subscribe")]
public void WritesAreRefusedEvenInAnEditedCopy(string name)
{
var refused = Refuse(ShippedWith(GraphQl(name)));
StringAssert.Contains(refused, name);
StringAssert.Contains(refused, "Mutation");
}
[TestMethod]
[DataRow("IGDInboxTrayQuery")]
[DataRow("IGDBadgeCountOffMsysQuery")]
[DataRow("PolarisDirectInboxQuery")]
[DataRow("IGPresenceUnifiedSetupQuery")]
[DataRow("useIGDSystemFolderUnreadThreadCountQuery")]
[DataRow("QuickPromotionSupportIGSchemaBatchFetchQuery")]
[DataRow("PolarisAPIGetFrCookieQuery")]
[DataRow("PolarisStoriesV3AdsPoolQuery")]
[DataRow("PolarisProfileDirectOrPartnershipInboxMessageEligibilityQuery")]
public void ChatInboxPresenceAndTelemetryFamiliesAreRefused(string name) =>
StringAssert.Contains(Refuse(ShippedWith(GraphQl(name))), "rifiutata");
[TestMethod]
[DataRow("/logging_client_events")]
[DataRow("/ajax/bz")]
[DataRow("/ajax/bulk-route-definitions/")]
[DataRow("/ajax/navigation/")]
[DataRow("/ajax/qualcosa/")]
[DataRow("/ws/chat/")]
[DataRow("/api/v1/web/likes/{media_id}/like/")]
[DataRow("/api/v1/discover/web/explore_grid/")]
[DataRow("https://www.facebook.com/ajax/sync/")]
[DataRow("//edge-chat.facebook.com/pull")]
[DataRow("/api/v1/media/{media_id}/info/?x=1")]
[DataRow("/{username}/{cursor}/")]
[DataRow("/AJAX/BZ")]
[DataRow("/ws")]
[DataRow("/./ajax/bz")]
[DataRow("/api/v1/./web/search/topsearch/")]
[DataRow("/api/v1//web/fxcal/")]
[DataRow("/api/v1/media/{media_id}/./info/")]
[DataRow("/api/v1/media/{media_id}/../../web/likes/")]
[DataRow("/ajax;x/bz")]
[DataRow("/api/v1/%77eb/likes/")]
[DataRow("/api/graphql")]
[DataRow("/graphql/query")]
[DataRow("/api/graphql/")]
[DataRow("/GraphQL/query")]
[DataRow("/api/v1/graphql_www")]
public void EachForbiddenPathFamilyIsRefusedEvenInAnEditedCopy(string path) =>
StringAssert.Contains(Refuse(ShippedWith(Rest("lettura-vietata", path))), "rifiutata");
[TestMethod]
public void GraphQlOnlyThroughItsTwoEntryPointsAndPlainAddressesOnlyWithGet()
{
var elsewhere = GraphQl("PolarisProfilePageContentQuery2");
elsewhere["path"] = "/api/v1/graphql";
StringAssert.Contains(Refuse(ShippedWith(elsewhere)), "/api/graphql");
var post = Rest("scrittura", "/dati/ricerche-recenti");
post["method"] = "POST";
StringAssert.Contains(Refuse(Alone(post)), "GET");
var get = GraphQl("PolarisProfilePageContentQuery2");
get["method"] = "GET";
StringAssert.Contains(Refuse(ShippedWith(get)), "POST");
}
[TestMethod]
public void UnknownKeysSourcesAndNamesAreErrors()
{
var file = JsonNode.Parse(SiteMap.ReadEmbeddedText("richieste.prova.json"))!.AsObject();
file["sconosciuta"] = 1;
StringAssert.Contains(Refuse(file.ToJsonString()), "richieste.prova.json");
var unknownKey = Rest("post2", "/dati/post/{code}");
unknownKey["doc_id"] = "123";
Refuse(Alone(unknownKey));
var unknownSource = Rest("post2", "/dati/post/{code}");
unknownSource["headers"] = new JsonObject { ["accept"] = "browser:accept" };
StringAssert.Contains(Refuse(Alone(unknownSource)), "fonte");
var unknownSession = Rest("post2", "/dati/post/{code}");
unknownSession["headers"] = new JsonObject { ["x-prova"] = "session:password" };
StringAssert.Contains(Refuse(Alone(unknownSession)), "fonte");
var cookieHeader = Rest("post2", "/dati/post/{code}");
cookieHeader["headers"] = new JsonObject { ["cookie"] = "cookie:sessionid" };
StringAssert.Contains(Refuse(Alone(cookieHeader)), "cookie");
var ownAgent = Rest("post2", "/dati/post/{code}");
ownAgent["headers"] = new JsonObject { ["user-agent"] = "constant:Febe/1.0" };
StringAssert.Contains(Refuse(Alone(ownAgent)), "user-agent");
var unknownValue = Rest("post2", "/dati/post/{password}");
StringAssert.Contains(Refuse(Alone(unknownValue)), "valore del caso");
var unknownCategory = Rest("post2", "/dati/post/{code}");
unknownCategory["category"] = "scaricati";
StringAssert.Contains(Refuse(Alone(unknownCategory)), "categoria");
var noReason = Rest("post2", "/dati/post/{code}");
noReason["configured"] = false;
StringAssert.Contains(Refuse(Alone(noReason)), "reason");
StringAssert.Contains(Refuse(ShippedWith(Get(RequestCatalog.LoadTestSite(), "post"), "prova")), "due volte");
StringAssert.Contains(Refuse("{ non è json"), "non valido");
}
[TestMethod]
[DataRow("PolarisProfileStoryHighlightsTrayContentQuery")]
[DataRow("media-info")]
public void AnOperationWithPendingHeadersOrFieldsCannotBeConfigured(string name)
{
// The variables are known, but not where every header and field takes its value: sending without them, or
// with guessed values, would not be the browser's request (rules 4 and 9).
var file = JsonNode.Parse(SiteMap.ReadEmbeddedText("richieste.instagram.json"))!.AsObject();
var operation = file["operations"]!.AsArray().Single(o => o!["name"]!.GetValue<string>() == name)!.AsObject();
operation["configured"] = true;
operation.Remove("reason");
var refused = Refuse(file.ToJsonString());
StringAssert.Contains(refused, "pending");
StringAssert.Contains(refused, "regola 9");
// The removed name alone is enough.
var removed = Rest("post2", "/dati/post/{code}");
removed["headers"] = new JsonObject { [RequestCatalog.RemovedName] = "pending" };
StringAssert.Contains(Refuse(Alone(removed)), "pending");
}
[TestMethod]
public void TheSearchBudgetIsOnlyForASearch()
{
// The search budget belongs to no session (§7.4): an edited copy cannot move stories or a post there.
var stories = Rest("storie2", "/dati/storie/{username}");
stories["category"] = "ricerca";
StringAssert.Contains(Refuse(Alone(stories)), "ricerca");
var searchWithAccount = Rest("cerca2", "/dati/cerca/{username}");
searchWithAccount["category"] = "ricerca";
searchWithAccount["parameters"] = new JsonObject { ["testo"] = "{search_text}" };
StringAssert.Contains(Refuse(Alone(searchWithAccount)), "ricerca");
// And a search counts nowhere else.
var searchAsPost = Rest("cerca3", "/dati/cerca");
searchAsPost["parameters"] = new JsonObject { ["testo"] = "{search_text}" };
StringAssert.Contains(Refuse(Alone(searchAsPost)), "ricerca");
Assert.AreEqual(ActionKind.ProfileSearch, Get(RequestCatalog.LoadInstagram(), "PolarisSearchBoxRefetchableQuery").Category);
Assert.AreEqual(ActionKind.ProfileSearch, Get(RequestCatalog.LoadTestSite(), "cerca").Category);
}
[TestMethod]
public void AValueThatWouldChangeThePathIsRefusedBeforeItLeaves()
{
// A cursor may hold «/» (it is opaque), but not in a segment of the path, where it would become two.
var catalog = RequestCatalog.Parse(Alone(Rest("pagina", "/dati/pagine/{cursor}")), "richieste.prova.json");
var page = Get(catalog, "pagina");
CollectionAssert.AreEquivalent(new[] { CaseValues.Cursor }, page.PathValues.ToArray());
Assert.IsNull(page.CheckValues(SiteRequest.For("pagina", (CaseValues.Cursor, "QVFCabc"))));
foreach (var value in new[] { "a/b", "a?b", "a#b", "a%2Fb", "...", "a b" })
{
var problem = page.CheckValues(SiteRequest.For("pagina", (CaseValues.Cursor, value)));
Assert.IsNotNull(problem, value);
StringAssert.Contains(problem, "percorso");
Assert.IsFalse(problem.Contains(value, StringComparison.Ordinal), "Never the value (rule 10).");
}
}
[TestMethod]
public void AnOperationWithRemovedNamesInItsVariablesCannotBeConfigured()
{
var text = SiteMap.ReadEmbeddedText("richieste.instagram.json");
var file = JsonNode.Parse(text)!.AsObject();
var page = file["operations"]!.AsArray().Single(o => o!["name"]!.GetValue<string>() == "PolarisProfilePageContentQuery")!.AsObject();
page["configured"] = true;
page.Remove("reason");
StringAssert.Contains(Refuse(file.ToJsonString()), "regola 9");
}
[TestMethod]
public void AnEditedCopyInTheDataFolderWinsButKeepsTheRefusals()
{
var folder = Path.Combine(Path.GetTempPath(), "febe-richieste-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(folder);
try
{
var path = Path.Combine(folder, "richieste.prova.json");
var file = JsonNode.Parse(SiteMap.ReadEmbeddedText("richieste.prova.json"))!.AsObject();
file["operations"]!.AsArray().RemoveAt(0);
File.WriteAllText(path, file.ToJsonString());
var copy = RequestCatalog.LoadOrEmbedded(path, "richieste.prova.json");
Assert.AreEqual(path, copy.Source);
Assert.IsFalse(copy.TryGet("inizio", out _));
Assert.IsTrue(RequestCatalog.LoadOrEmbedded(Path.Combine(folder, "manca.json"), "richieste.prova.json").TryGet("inizio", out _));
file["operations"]!.AsArray().Add(Rest("telemetria", "/logging_client_events"));
File.WriteAllText(path, file.ToJsonString());
var refused = Assert.ThrowsExactly<InvalidDataException>(() => RequestCatalog.LoadOrEmbedded(path, "richieste.prova.json"));
StringAssert.Contains(refused.Message, path);
}
finally
{
Directory.Delete(folder, recursive: true);
}
}
private static SiteOperation Get(RequestCatalog catalog, string name) =>
catalog.TryGet(name, out var operation) ? operation : throw new AssertFailedException($"«{name}» non è nel catalogo.");
private static string Refuse(string json) =>
Assert.ThrowsExactly<InvalidDataException>(() => RequestCatalog.Parse(json, "richieste.prova.json")).Message;
/// <summary>The shipped Instagram (or test site) catalog with one more operation, as an edited copy would be.</summary>
private static string ShippedWith(JsonObject operation, string site = "instagram")
{
var file = JsonNode.Parse(SiteMap.ReadEmbeddedText($"richieste.{site}.json"))!.AsObject();
file["operations"]!.AsArray().Add(operation);
return file.ToJsonString();
}
/// <summary>The shipped catalog with the operation as JSON again (used to add it twice).</summary>
private static string ShippedWith(SiteOperation existing, string site)
{
var file = JsonNode.Parse(SiteMap.ReadEmbeddedText($"richieste.{site}.json"))!.AsObject();
var copy = file["operations"]!.AsArray().Single(o => o!["name"]!.GetValue<string>() == existing.Name)!.DeepClone();
file["operations"]!.AsArray().Add(copy);
return file.ToJsonString();
}
private static string Alone(JsonObject operation) =>
new JsonObject { ["site"] = "prova", ["operations"] = new JsonArray(operation) }.ToJsonString();
private static JsonObject Rest(string name, string path) => new()
{
["name"] = name,
["kind"] = "rest",
["method"] = "GET",
["path"] = path,
["category"] = "post",
["configured"] = true,
["rootFields"] = new JsonArray("contenuto"),
["sample"] = "000005-post-serie",
};
private static JsonObject GraphQl(string name) => new()
{
["name"] = name,
["kind"] = "graphql",
["method"] = "POST",
["path"] = "/api/graphql",
["category"] = "profilo",
["configured"] = true,
["headers"] = new JsonObject { ["x-fb-friendly-name"] = "operation" },
["body"] = new JsonObject { ["variables"] = "variables", ["doc_id"] = "session:doc_id", ["fb_api_req_friendly_name"] = "operation" },
["variables"] = new JsonObject { ["id"] = "{account_id}" },
["rootFields"] = new JsonArray("data.user"),
["sample"] = "000009-scheda-profilo",
};
[GeneratedRegex("[0-9]{10,}")]
private static partial Regex LongNumber();
}
@@ -0,0 +1,248 @@
using System.Collections.Concurrent;
using System.Text;
using System.Text.Json;
using System.Text.Json.Nodes;
using Febe.Browser.Capture;
using Febe.Browser.Diagnostics;
using Febe.Core.Abstractions;
using Febe.Core.Collection;
using Febe.Core.Collection.Readers;
using Febe.Core.Collection.Recipes;
using Febe.Core.Collection.Requests;
using Febe.Tests.Browser.TestSupport;
using Febe.Tests.Collection.Support;
using Febe.Tests.Support;
namespace Febe.Tests.Collection;
/// <summary>
/// The delivery of answers to the readers, shared by the response tap and the site client (issue #36, S1): a cleaned
/// real sample reaches <see cref="CaptureCollector"/> the same whether the browser captured it or the site client asked
/// for it.
/// </summary>
[TestClass]
public sealed class ResponseDeliveryTests
{
private static readonly HostClassifier Hosts = new(["127.0.0.1:5000"], ["127.0.0.1:5001"]);
[TestMethod]
[DataRow("000002-ricerca-profili")]
[DataRow("000005-post-video")]
[DataRow("000006-raccolta-in-evidenza")]
[DataRow("000007-raccolta-pagina-successiva")]
[DataRow("000009-scheda-profilo")]
[DataRow("000010-post-del-profilo")]
[DataRow("000011-reel-del-profilo")]
[DataRow("000012-con-tag")]
[DataRow("000013-raccolte-del-profilo")]
[DataRow("000014-storie-di-un-profilo")]
[DataRow("000015-vassoio-storie")]
[DataRow("000016-serie-con-video")]
public async Task ASampleReachesTheReadersAsThroughTheCapture(string sample)
{
var (mime, resource, body) = Sample(sample);
var isDocument = resource == "Document";
var text = isDocument
? $"""<!DOCTYPE html><html><head><script type="application/json">{body}</script></head><body></body></html>"""
: body;
// Through the browser: the tap fetches the body and hands it on.
var clock = new VirtualClock();
var cdp = new FakeCdpSession(clock);
cdp.Responder = (method, p) => method switch
{
"Page.getFrameTree" => new JsonObject { ["frameTree"] = new JsonObject { ["frame"] = new JsonObject { ["id"] = "F1", ["url"] = "about:blank" } } },
"Network.getResponseBody" when p!["requestId"]!.GetValue<string>() == "r1" => new JsonObject { ["body"] = text, ["base64Encoded"] = false },
"Network.getResponseBody" => new JsonObject { ["body"] = """{"fine":1}""", ["base64Encoded"] = false },
_ => null,
};
var captured = new CaptureCollector(new ShapeReader(ReaderVocabulary.LoadInstagram()), clock);
var seen = new RecordingConsumer();
await using (var tap = new ResponseTap(cdp, Hosts, clock))
{
tap.AddConsumer(captured);
tap.AddObserver(captured);
tap.AddConsumer(seen);
await tap.StartAsync();
cdp.Raise("Network.responseReceived", isDocument ? Document("r1") : Received("r1", mime));
cdp.Raise("Network.loadingFinished", new JsonObject { ["requestId"] = "r1", ["encodedDataLength"] = text.Length });
cdp.Raise("Network.responseReceived", Received("fine", "application/json"));
cdp.Raise("Network.loadingFinished", new JsonObject { ["requestId"] = "fine", ["encodedDataLength"] = 10 });
await TestWait.UntilAsync(() => seen.Items.Any(i => i.RequestId == "fine"), "risposta di chiusura");
}
// Through the delivery alone, as the site client hands its answers on, with the same descriptor.
var descriptor = seen.Items.First(i => i.RequestId == "r1");
var requested = new CaptureCollector(new ShapeReader(ReaderVocabulary.LoadInstagram()), clock);
var delivery = new ResponseDelivery("Client del sito");
var bytes = Encoding.UTF8.GetBytes(text);
var delivered = isDocument
? await delivery.DeliverDocumentAsync(descriptor, bytes, [requested], CancellationToken.None)
: await delivery.DeliverBodyAsync(descriptor, bytes, [requested], CancellationToken.None);
Assert.AreEqual(1, delivered);
Assert.AreEqual(1, delivery.JsonBodies);
Assert.AreEqual(0, delivery.ParseErrors);
Assert.IsTrue(Read(captured) > 0, $"{sample}: the readers keep nothing of it, so the comparison would prove nothing.");
Assert.AreEqual(Read(captured), Read(requested), sample);
Assert.AreEqual(Describe(captured), Describe(requested), sample);
}
[TestMethod]
public async Task AnAnswerAskedByTheSiteClientReachesTheReadersLikeTheSample()
{
var catalog = SettledCatalog.Instagram();
var requester = new ScriptedRequester(catalog);
var answer = await requester.SendAsync(
SiteRequest.For("PolarisProfileStoryHighlightsTrayContentQuery", (CaseValues.AccountId, "4815162342")),
Permit.Grant(ActionKind.ProfileOpen, 7),
CancellationToken.None);
var clock = new VirtualClock();
var collector = new CaptureCollector(new ShapeReader(ReaderVocabulary.LoadInstagram()), clock);
var descriptor = new ResponseDescriptor("richiesta-1", HostKind.Site, "https://www.instagram.com" + answer.PathShape, "www.instagram.com", answer.PathShape, answer.Status, answer.MimeType, "Fetch", false, clock.Now);
var delivered = await new ResponseDelivery("Client del sito").DeliverBodyAsync(descriptor, answer.Body, [collector], CancellationToken.None);
Assert.AreEqual(1, delivered);
var read = collector.Since(0).Single();
Assert.AreEqual(3, read.Result.Highlights.Count, "The three collections of the sample.");
using var document = JsonDocument.Parse(answer.Body);
Assert.IsTrue(catalog.Configured.Single(o => o.Name == answer.Operation).HasRootFields(document.RootElement));
}
[TestMethod]
public async Task NotJsonAndDocumentsWithoutBlocksDeliverNothing()
{
var consumer = new RecordingConsumer();
var delivery = new ResponseDelivery("Client del sito");
var descriptor = Descriptor("text/javascript");
Assert.AreEqual(0, await delivery.DeliverBodyAsync(descriptor, Encoding.UTF8.GetBytes("for (;;);{\"a\":1}"), [consumer], CancellationToken.None));
Assert.AreEqual(0, await delivery.DeliverBodyAsync(descriptor, Encoding.UTF8.GetBytes(new string('[', 300) + new string(']', 300)), [consumer], CancellationToken.None), "Deeper than 256.");
Assert.AreEqual(0, await delivery.DeliverDocumentAsync(Descriptor("text/html"), Encoding.UTF8.GetBytes("<html><body>niente</body></html>"), [consumer], CancellationToken.None));
Assert.AreEqual(2, delivery.ParseErrors, "A document without blocks is not a parse error.");
Assert.AreEqual(0, delivery.JsonBodies);
Assert.IsTrue(consumer.Items.IsEmpty);
}
[TestMethod]
public async Task AFailingReaderStopsNeitherTheOthersNorTheNextBlockAndIsWarnedWithoutContents()
{
var warnings = new ConcurrentQueue<(string Message, Exception? Error)>();
var delivery = new ResponseDelivery("Client del sito", (message, error) => warnings.Enqueue((message, error)));
var failing = new FailingConsumer();
var consumer = new RecordingConsumer();
var html = """<html><script type="application/json">{"segreto":"a"}</script><script type="application/json">{"segreto":"b"}</script></html>""";
var delivered = await delivery.DeliverDocumentAsync(Descriptor("text/html"), Encoding.UTF8.GetBytes(html), [failing, consumer], CancellationToken.None);
Assert.AreEqual(2, delivered);
Assert.AreEqual(2, consumer.Items.Count);
Assert.AreEqual(2, failing.Calls);
Assert.AreEqual(2, warnings.Count);
Assert.IsTrue(warnings.All(w => w.Message == "Client del sito: errore in un lettore (FailingConsumer)."));
Assert.IsFalse(warnings.Any(w => w.Message.Contains("segreto", StringComparison.Ordinal)));
}
[TestMethod]
public async Task TheTapKeepsItsStatsAndItsMessagesWithTheSharedDelivery()
{
var clock = new VirtualClock();
var cdp = new FakeCdpSession(clock);
cdp.Responder = (method, p) => method switch
{
"Page.getFrameTree" => new JsonObject { ["frameTree"] = new JsonObject { ["frame"] = new JsonObject { ["id"] = "F1", ["url"] = "about:blank" } } },
"Network.getResponseBody" when p!["requestId"]!.GetValue<string>() == "bad64" => new JsonObject { ["body"] = "%%%", ["base64Encoded"] = true },
"Network.getResponseBody" when p!["requestId"]!.GetValue<string>() == "nojson" => new JsonObject { ["body"] = "for (;;);", ["base64Encoded"] = false },
"Network.getResponseBody" => new JsonObject { ["body"] = """{"a":1}""", ["base64Encoded"] = false },
_ => null,
};
var messages = new ConcurrentQueue<string>();
var log = new DelegateBrowserLog((_, message, _) => messages.Enqueue(message));
var seen = new RecordingConsumer();
await using var tap = new ResponseTap(cdp, Hosts, clock, log);
tap.AddConsumer(new FailingConsumer());
tap.AddConsumer(seen);
await tap.StartAsync();
foreach (var id in new[] { "bad64", "nojson", "ok" })
{
cdp.Raise("Network.responseReceived", Received(id, "application/json"));
cdp.Raise("Network.loadingFinished", new JsonObject { ["requestId"] = id, ["encodedDataLength"] = 10 });
}
await TestWait.UntilAsync(() => seen.Items.Any(i => i.RequestId == "ok"), "risposta valida");
var stats = tap.Stats;
Assert.AreEqual(3, stats.Responses);
Assert.AreEqual(1, stats.JsonBodies);
Assert.AreEqual(2, stats.ParseErrors, "A base64 body that does not decode and a body that is not JSON, as before.");
Assert.AreEqual(0, stats.BodyErrors);
Assert.IsTrue(messages.Contains("Cattura: errore in un lettore (FailingConsumer)."), "The tap's message is unchanged.");
}
/// <summary>Reads of the sample's answer the collector kept (unrelated answers are not kept), search ones included.</summary>
private static int Read(CaptureCollector collector) =>
collector.Since(0).Count(r => r.Response.RequestId == "r1") + collector.SearchSince(default).Count(r => r.Response.RequestId == "r1");
private static string Describe(CaptureCollector collector)
{
var options = new JsonSerializerOptions { WriteIndented = false };
// Only the sample's answer: the closing answer of the capture run is not part of the comparison.
var reads = JsonSerializer.Serialize(collector.Since(0).Where(r => r.Response.RequestId == "r1").Select(r => new { r.Response, r.Result }), options);
var search = JsonSerializer.Serialize(collector.SearchSince(default).Where(r => r.Response.RequestId == "r1").Select(r => new { r.Response, r.Result }), options);
return reads + search;
}
private static (string Mime, string Resource, string Body) Sample(string name)
{
var file = Path.Combine(AppContext.BaseDirectory, "campioni", "instagram", name + ".json");
using var sample = JsonDocument.Parse(File.ReadAllText(file));
var descriptor = sample.RootElement.GetProperty("descrittore");
return (descriptor.GetProperty("tipo").GetString()!, descriptor.GetProperty("risorsa").GetString()!, sample.RootElement.GetProperty("corpo").GetRawText());
}
private static ResponseDescriptor Descriptor(string mime) =>
new("r1", HostKind.Site, "http://127.0.0.1:5000/api/graphql", "127.0.0.1:5000", "/api/graphql", 200, mime, "Fetch", false, DateTimeOffset.UnixEpoch);
private static JsonObject Received(string id, string mime) => new()
{
["requestId"] = id,
["loaderId"] = "L1",
["frameId"] = "F1",
["type"] = "XHR",
["response"] = new JsonObject { ["url"] = "http://127.0.0.1:5000/api/graphql", ["status"] = 200, ["mimeType"] = mime },
};
private static JsonObject Document(string id) => new()
{
["requestId"] = id,
["loaderId"] = id,
["frameId"] = "F1",
["type"] = "Document",
["response"] = new JsonObject { ["url"] = "http://127.0.0.1:5000/", ["status"] = 200, ["mimeType"] = "text/html; charset=utf-8" },
};
private sealed class RecordingConsumer : IResponseConsumer
{
public ConcurrentQueue<ResponseDescriptor> Items { get; } = new();
public ValueTask ConsumeAsync(ResponseDescriptor response, JsonDocument body, CancellationToken ct)
{
Items.Enqueue(response);
return ValueTask.CompletedTask;
}
}
private sealed class FailingConsumer : IResponseConsumer
{
private int _calls;
public int Calls => Volatile.Read(ref _calls);
public ValueTask ConsumeAsync(ResponseDescriptor response, JsonDocument body, CancellationToken ct)
{
Interlocked.Increment(ref _calls);
throw new InvalidOperationException("Lettore di prova che fallisce sempre.");
}
}
}
@@ -0,0 +1,228 @@
using System.Text;
using System.Text.Json;
using Febe.Core.Abstractions;
using Febe.Core.Collection.Requests;
using Febe.Tests.Collection.Support;
namespace Febe.Tests.Collection;
/// <summary>
/// The contract of the site client (issue #36, S1) on its test double: no request without a permit the Governor
/// granted for it (rule 8), only operations of the catalog, and answers that carry the stop cases of §7.6 without ever
/// showing a body, a cookie or a value in their text (rule 10).
/// </summary>
[TestClass]
public sealed class SiteRequesterTests
{
private static readonly SiteRequest Tray = SiteRequest.For("PolarisProfileStoryHighlightsTrayContentQuery", (CaseValues.AccountId, "4815162342"));
[TestMethod]
public async Task NothingLeavesWithoutAGrantedPermit()
{
var requester = new ScriptedRequester(SettledCatalog.Instagram());
var denied = Permit.Deny(ActionKind.ProfileOpen, 7, DenialReason.HourlyBudgetReached, "Tetto orario raggiunto.");
var refused = await Assert.ThrowsExactlyAsync<SiteRequestRefusedException>(() => requester.SendAsync(Tray, denied, CancellationToken.None));
StringAssert.Contains(refused.Message, "permesso");
await Assert.ThrowsExactlyAsync<SiteRequestRefusedException>(() => requester.SendAsync(Tray, null!, CancellationToken.None));
Assert.AreEqual(2, requester.Refused);
Assert.AreEqual(0, requester.Sent.Count);
}
[TestMethod]
public async Task APermitOfAnotherCategoryOrARequestOutsideTheCatalogIsRefused()
{
var requester = new ScriptedRequester(SettledCatalog.Instagram());
// A permit for one category never covers a request of another (§7.4: a permit per request, in its category).
var other = await Assert.ThrowsExactlyAsync<SiteRequestRefusedException>(() => requester.SendAsync(Tray, Permit.Grant(ActionKind.StoryView, 7), CancellationToken.None));
StringAssert.Contains(other.Message, "ProfileOpen");
await Assert.ThrowsExactlyAsync<SiteRequestRefusedException>(() =>
requester.SendAsync(new SiteRequest("PolarisStoriesV3SeenMutation"), Permit.Grant(ActionKind.StoryView, 7), CancellationToken.None));
var notConfigured = await Assert.ThrowsExactlyAsync<SiteRequestRefusedException>(() =>
requester.SendAsync(SiteRequest.For("PolarisProfilePageContentQuery", (CaseValues.AccountId, "1")), Permit.Grant(ActionKind.ProfileOpen, 7), CancellationToken.None));
StringAssert.Contains(notConfigured.Message, "non è configurata");
await Assert.ThrowsExactlyAsync<SiteRequestRefusedException>(() =>
requester.SendAsync(SiteRequest.For("PolarisProfileStoryHighlightsTrayContentQuery", (CaseValues.Username, "marta")), Permit.Grant(ActionKind.ProfileOpen, 7), CancellationToken.None));
Assert.AreEqual(4, requester.Refused);
Assert.AreEqual(0, requester.Sent.Count);
Assert.ThrowsExactly<ArgumentException>(() => SiteRequest.For("media-info", ("x-csrftoken", "valore")), "A request carries no header.");
}
[TestMethod]
public async Task AnOperationWithHeadersOrFieldsStillPendingNeverLeaves()
{
// The shipped file: the variables of the tray are known, but not where __dyn, referer, sec-ch-ua... come from.
var shipped = RequestCatalog.LoadInstagram();
Assert.IsTrue(shipped.TryGet(Tray.Operation, out var tray));
Assert.IsFalse(tray.IsConfigured);
Assert.IsTrue(tray.PendingValues.Count > 0);
var requester = new ScriptedRequester(shipped);
var refused = await Assert.ThrowsExactlyAsync<SiteRequestRefusedException>(() => requester.SendAsync(Tray, Permit.Grant(ActionKind.ProfileOpen, 7), CancellationToken.None));
StringAssert.Contains(refused.Message, "pending");
Assert.AreEqual(0, requester.Sent.Count);
}
[TestMethod]
public async Task APermitServesOneRequestOnly()
{
var requester = new ScriptedRequester(SettledCatalog.Instagram());
var permit = Permit.Grant(ActionKind.ProfileOpen, 7);
await requester.SendAsync(Tray, permit, CancellationToken.None);
var again = await Assert.ThrowsExactlyAsync<SiteRequestRefusedException>(() => requester.SendAsync(Tray, permit, CancellationToken.None));
StringAssert.Contains(again.Message, "già stato usato");
// Equal fields, another grant: a new permit (by reference, not by value).
await requester.SendAsync(Tray, Permit.Grant(ActionKind.ProfileOpen, 7), CancellationToken.None);
// A permit shown with a refused request is spent too: the Governor has already counted it.
var spent = Permit.Grant(ActionKind.ProfileOpen, 7);
await Assert.ThrowsExactlyAsync<SiteRequestRefusedException>(() =>
requester.SendAsync(SiteRequest.For(Tray.Operation, (CaseValues.Username, "marta")), spent, CancellationToken.None));
await Assert.ThrowsExactlyAsync<SiteRequestRefusedException>(() => requester.SendAsync(Tray, spent, CancellationToken.None));
Assert.AreEqual(2, requester.Sent.Count);
Assert.AreEqual(3, requester.Refused);
}
[TestMethod]
[DataRow(CaseValues.MediaId, "1/../../web/likes/2/like")]
[DataRow(CaseValues.MediaId, "12a")]
[DataRow(CaseValues.AccountId, "..")]
[DataRow(CaseValues.Username, "..")]
[DataRow(CaseValues.Username, "marta/../ajax")]
[DataRow(CaseValues.Username, "marta?x=1")]
[DataRow(CaseValues.Code, "Ab#c")]
[DataRow(CaseValues.HighlightId, "highlight:1/2")]
[DataRow(CaseValues.Cursor, "riga\nnuova")]
public void AValueWithoutItsFormCannotBeAsked(string name, string value)
{
var refused = Assert.ThrowsExactly<ArgumentException>(() => SiteRequest.For("media-info", (name, value)));
StringAssert.Contains(refused.Message, name);
Assert.IsFalse(refused.Message.Contains(value, StringComparison.Ordinal), "Never the value in the message (rule 10).");
}
[TestMethod]
public void TheFormsOfTheSitesValuesAreAccepted()
{
Assert.IsTrue(CaseValues.HasForm(CaseValues.AccountId, "4815162342"));
Assert.IsTrue(CaseValues.HasForm(CaseValues.Username, "marta.prova"));
Assert.IsTrue(CaseValues.HasForm(CaseValues.Username, "_marta_"));
Assert.IsTrue(CaseValues.HasForm(CaseValues.Code, "DAbc-12_x"));
Assert.IsTrue(CaseValues.HasForm(CaseValues.HighlightId, "highlight:17890123"));
Assert.IsTrue(CaseValues.HasForm(CaseValues.Cursor, "QVFC/cursore+=="));
Assert.IsTrue(CaseValues.HasForm(CaseValues.SearchText, "marta prova #estate"));
}
[TestMethod]
[DataRow("instagram")]
[DataRow("prova")]
public async Task EveryConfiguredOperationIsAnsweredFromItsSample(string site)
{
// Instagram as tappa S4 will leave it: the shipped file has no configured operation yet.
var catalog = site == "instagram" ? SettledCatalog.Instagram() : RequestCatalog.LoadTestSite();
var requester = new ScriptedRequester(catalog);
Assert.IsTrue(catalog.Configured.Any());
foreach (var operation in catalog.Configured)
{
var request = new SiteRequest(operation.Name, operation.RequiredValues.ToDictionary(v => v, v => v == CaseValues.AccountId || v == CaseValues.MediaId ? "31415" : "valore", StringComparer.Ordinal));
var answer = await requester.SendAsync(request, Permit.Grant(operation.Category, null), CancellationToken.None);
Assert.AreEqual(200, answer.Status, operation.Name);
Assert.IsTrue(answer.IsDeclaredData, operation.Name);
Assert.AreEqual(operation.PathShape, answer.PathShape);
using var body = JsonDocument.Parse(answer.Body);
Assert.IsTrue(operation.HasRootFields(body.RootElement), operation.Name);
}
Assert.AreEqual(catalog.Configured.Count(), requester.Sent.Count);
}
[TestMethod]
public async Task TheStopCasesOfSection76CanBeScripted()
{
var catalog = SettledCatalog.Instagram();
var tray = catalog.Configured.Single(o => o.Name == Tray.Operation);
var requester = new ScriptedRequester(catalog).Then(
ScriptedReply.TooManyRequests,
ScriptedReply.RedirectToLogin,
ScriptedReply.RedirectToChallenge,
ScriptedReply.HtmlInsteadOfJson,
ScriptedReply.MissingRootField,
ScriptedReply.SessionCookieDeleted);
// A new permit for every request (§7.4).
static Permit Granted() => Permit.Grant(ActionKind.ProfileOpen, 7);
var slow = await requester.SendAsync(Tray, Granted(), CancellationToken.None);
Assert.AreEqual(429, slow.Status);
var login = await requester.SendAsync(Tray, Granted(), CancellationToken.None);
Assert.IsTrue(login.IsRedirect);
Assert.AreEqual("/accounts/login/", login.RedirectPathShape, "Only the shape of the path: no host, no query string.");
var challenge = await requester.SendAsync(Tray, Granted(), CancellationToken.None);
Assert.AreEqual("/challenge/{testo}/", challenge.RedirectPathShape);
var html = await requester.SendAsync(Tray, Granted(), CancellationToken.None);
Assert.IsTrue(html.IsHtml);
Assert.IsFalse(html.IsDeclaredData);
Assert.AreEqual("text/html", html.MimeType);
var missing = await requester.SendAsync(Tray, Granted(), CancellationToken.None);
using (var body = JsonDocument.Parse(missing.Body))
{
Assert.IsFalse(tray.HasRootFields(body.RootElement), "JSON without the expected field: stop and warn (§6.0).");
}
var deleted = await requester.SendAsync(Tray, Granted(), CancellationToken.None);
Assert.IsTrue(deleted.Deletes("sessionid"));
var again = await requester.SendAsync(Tray, Granted(), CancellationToken.None);
Assert.AreEqual(200, again.Status, "After the script, the sample again.");
Assert.AreEqual(7, requester.Sent.Count);
}
[TestMethod]
public void TheTextOfAnswersAndRequestsShowsNoBodyCookieOrValue()
{
Assert.IsTrue(RequestCatalog.LoadInstagram().TryGet("media-info", out var media));
var answer = new SiteAnswer(
media,
302,
"application/json; charset=utf-8",
Encoding.UTF8.GetBytes("""{"segreto":"corpo-riservato"}"""),
redirectTo: "https://www.instagram.com/challenge/Ab12Cd/?next=%2Fmarta.prova%2F&token=xyz",
cookiesSet: ["csrftoken", "rur"],
cookiesDeleted: ["sessionid"],
setsClaim: true,
retryAfter: TimeSpan.FromMinutes(5));
var text = answer.ToString();
foreach (var leak in new[] { "segreto", "corpo-riservato", "csrftoken", "rur", "sessionid", "Ab12Cd", "marta", "token", "xyz", "next", "instagram.com" })
{
Assert.IsFalse(text.Contains(leak, StringComparison.OrdinalIgnoreCase), $"«{leak}» in «{text}».");
}
StringAssert.Contains(text, "media-info");
StringAssert.Contains(text, "/api/v1/media/{numero}/info/");
StringAssert.Contains(text, "302");
StringAssert.Contains(text, "/challenge/{testo}/");
// The path shown is always the catalog's shape: an answer cannot be given the path with its values.
var plain = new SiteAnswer(media, 200, "application/json", ReadOnlyMemory<byte>.Empty);
Assert.AreEqual("/api/v1/media/{numero}/info/", plain.PathShape);
Assert.AreEqual("media-info", plain.Operation);
var request = SiteRequest.For("PolarisProfilePostsQuery", (CaseValues.Username, "marta.prova"), (CaseValues.Cursor, "QVFCcursore"));
Assert.IsFalse(request.ToString().Contains("marta", StringComparison.Ordinal));
Assert.IsFalse(request.ToString().Contains("QVFC", StringComparison.Ordinal));
StringAssert.Contains(request.ToString(), "username");
Assert.ThrowsExactly<ArgumentException>(() => new SiteAnswer(media, 200, "text/html", ReadOnlyMemory<byte>.Empty, cookiesSet: ["sessionid=abc"]));
}
}
@@ -0,0 +1,166 @@
using System.Collections.Concurrent;
using System.Text;
using System.Text.Json.Nodes;
using Febe.Core.Abstractions;
using Febe.Core.Collection.Requests;
namespace Febe.Tests.Collection.Support;
/// <summary>What <see cref="ScriptedRequester"/> answers to the next request, instead of the sample.</summary>
public enum ScriptedReply
{
/// <summary>The cleaned sample of the operation (<see cref="SiteOperation.Sample"/>).</summary>
Sample,
/// <summary>A 429 without body (§7.6, Rallentamento).</summary>
TooManyRequests,
/// <summary>A 302 to the login page of the site, never followed.</summary>
RedirectToLogin,
/// <summary>A 302 to the challenge page of the site, never followed.</summary>
RedirectToChallenge,
/// <summary>A 200 with an HTML page where the catalog expects JSON.</summary>
HtmlInsteadOfJson,
/// <summary>The sample without its expected root field (the last one of <see cref="SiteOperation.RootFields"/>).</summary>
MissingRootField,
/// <summary>The sample, with the session cookie deleted by the site.</summary>
SessionCookieDeleted,
}
/// <summary>
/// The site client of the tests (issue #36): answers the requests of the catalog from the cleaned samples of
/// <c>tests/Febe.Tests/campioni/&lt;sito&gt;</c>, by operation name, and can be scripted to answer the stop cases of
/// §7.6. It honours the <see cref="ISiteRequester"/> contract: <see cref="SiteRequestGuard"/> before anything (a permit
/// used once is spent, <see cref="SpentPermits"/>), and one request at a time. Nothing leaves the process.
/// </summary>
public sealed class ScriptedRequester : ISiteRequester
{
private readonly RequestCatalog _catalog;
private readonly string _samples;
private readonly ConcurrentQueue<ScriptedReply> _script = new();
private readonly ConcurrentQueue<SiteRequest> _sent = new();
private readonly SpentPermits _spent = new();
private int _busy;
private int _refused;
/// <param name="catalog">The catalog the requests are checked against.</param>
/// <param name="sampleFolder">Folder of the cleaned samples; by default <c>campioni/&lt;site of the catalog&gt;</c> next to the tests.</param>
public ScriptedRequester(RequestCatalog catalog, string? sampleFolder = null)
{
_catalog = catalog ?? throw new ArgumentNullException(nameof(catalog));
_samples = sampleFolder ?? Path.Combine(AppContext.BaseDirectory, "campioni", catalog.Site);
}
/// <summary>Path of the login page a redirect points to.</summary>
public string LoginLocation { get; init; } = "https://www.instagram.com/accounts/login/?next=%2F";
/// <summary>Path of the challenge page a redirect points to.</summary>
public string ChallengeLocation { get; init; } = "/challenge/AbCdEf123/?next=%2F";
/// <summary>Name of the session cookie (<see cref="ScriptedReply.SessionCookieDeleted"/>).</summary>
public string SessionCookie { get; init; } = "sessionid";
/// <summary>Requests that passed the guard, in order.</summary>
public IReadOnlyList<SiteRequest> Sent => [.. _sent];
/// <summary>Requests refused before anything was answered.</summary>
public int Refused => Volatile.Read(ref _refused);
/// <summary>The replies of the next requests, in order; after them, the samples again.</summary>
public ScriptedRequester Then(params ScriptedReply[] replies)
{
foreach (var reply in replies)
{
_script.Enqueue(reply);
}
return this;
}
public async Task<SiteAnswer> SendAsync(SiteRequest request, Permit permit, CancellationToken ct)
{
SiteOperation operation;
try
{
operation = SiteRequestGuard.Check(_catalog, request, permit, _spent);
}
catch (SiteRequestRefusedException)
{
Interlocked.Increment(ref _refused);
throw;
}
if (Interlocked.Exchange(ref _busy, 1) == 1)
{
Interlocked.Increment(ref _refused);
throw new SiteRequestRefusedException($"Richiesta «{request.Operation}» non inviata: un'altra richiesta è ancora in corso (una per volta).");
}
try
{
ct.ThrowIfCancellationRequested();
await Task.Yield();
_sent.Enqueue(request);
var reply = _script.TryDequeue(out var next) ? next : ScriptedReply.Sample;
return Answer(operation, reply);
}
finally
{
Volatile.Write(ref _busy, 0);
}
}
/// <summary>The descriptor and the body of the sample of <paramref name="operation"/>.</summary>
public (string MimeType, JsonNode Body) Sample(SiteOperation operation)
{
ArgumentNullException.ThrowIfNull(operation);
if (operation.Sample is not { } name)
{
throw new InvalidOperationException($"«{operation.Name}» non ha un campione.");
}
var file = Directory.EnumerateFiles(_samples, $"{name}.json").Single();
var sample = JsonNode.Parse(File.ReadAllText(file))!;
return (sample["descrittore"]!["tipo"]!.GetValue<string>(), sample["corpo"]!.DeepClone());
}
private SiteAnswer Answer(SiteOperation operation, ScriptedReply reply)
{
switch (reply)
{
case ScriptedReply.TooManyRequests:
return new SiteAnswer(operation, 429, "text/plain", ReadOnlyMemory<byte>.Empty);
case ScriptedReply.RedirectToLogin:
return new SiteAnswer(operation, 302, "text/html", ReadOnlyMemory<byte>.Empty, redirectTo: LoginLocation);
case ScriptedReply.RedirectToChallenge:
return new SiteAnswer(operation, 302, "text/html", ReadOnlyMemory<byte>.Empty, redirectTo: ChallengeLocation);
case ScriptedReply.HtmlInsteadOfJson:
return new SiteAnswer(operation, 200, "text/html; charset=utf-8", Encoding.UTF8.GetBytes("<!DOCTYPE html><html><head><title>Pagina</title></head><body></body></html>"));
}
var (mime, body) = Sample(operation);
if (reply == ScriptedReply.MissingRootField)
{
var parts = operation.RootFields[^1].Split('.');
var parent = body;
foreach (var part in parts[..^1])
{
parent = parent[part]!;
}
parent.AsObject().Remove(parts[^1]);
}
var bytes = Encoding.UTF8.GetBytes(body.ToJsonString());
return new SiteAnswer(
operation,
200,
mime,
bytes,
cookiesDeleted: reply == ScriptedReply.SessionCookieDeleted ? [SessionCookie] : null);
}
}
@@ -0,0 +1,50 @@
using System.Text.Json.Nodes;
using Febe.Core.Collection;
using Febe.Core.Collection.Requests;
namespace Febe.Tests.Collection.Support;
/// <summary>
/// The shipped Instagram catalog as tappa S4 will leave it for some operations: their headers and fields with a
/// «pending» source dropped and the operation configured. Only for the tests of the site client contract, which need
/// an Instagram operation the guard lets through; the shipped file keeps them not configured (rule 9).
/// </summary>
public static class SettledCatalog
{
/// <summary>The operations whose variables the shapes show whole (only the headers and fields wait for S4).</summary>
public static IReadOnlyList<string> KnownVariables { get; } =
[
"PolarisProfileReelsTabContentQuery", "PolarisProfileTaggedTabContentQuery",
"PolarisProfileStoryHighlightsTrayContentQuery", "media-info",
];
/// <summary>The Instagram catalog with <paramref name="names"/> (by default <see cref="KnownVariables"/>) settled.</summary>
public static RequestCatalog Instagram(params string[] names)
{
var settle = names.Length == 0 ? KnownVariables : names;
var file = JsonNode.Parse(SiteMap.ReadEmbeddedText("richieste.instagram.json"))!.AsObject();
foreach (var operation in file["operations"]!.AsArray().Select(o => o!.AsObject()))
{
if (!settle.Contains(operation["name"]!.GetValue<string>()))
{
continue;
}
foreach (var part in new[] { "headers", "body" })
{
if (operation[part] is JsonObject sources)
{
foreach (var key in sources.Where(s => s.Value?.GetValue<string>() == "pending").Select(s => s.Key).ToList())
{
sources.Remove(key);
}
}
}
operation["configured"] = true;
operation.Remove("reason");
}
return RequestCatalog.Parse(file.ToJsonString(), "richieste.instagram.json (sistemato per le prove)");
}
}