fix(server): keep convert dry runs like Reorganize's, say what a new sequence lacks

The convert endpoint planned the whole selection inside the HTTP request, again for
every mode, quality keystroke and confirmation, with no limit: behind a proxy an
all-sequence conversion could never get past its dry run or its apply.

- ReorganizePlanCache becomes PlanCache, shared by Reorganize and Convert: one plan
  per key made apart from the request, two at once across both kinds, eight kept,
  15-minute timeout, 5-minute lifetime, 503 plan_dropped. Keys are typed records
  (ReorganizePlanKey, ConvertPlanKey) that also say which exceptions are refusals.
- A conversion plan is keyed by archive, a convert stamp of the index
  (SequenceRepository.GetConvertStampAsync: the Reorganize stamp plus, per planned
  sequence, the photos' count, sizes, formats, pixel sizes and missing, corrupt,
  preview-only and undecodable flags), the sequences and the target with the
  quality that counts. Not the mode: the request's mode and options are set on the
  kept plan, and the free space is measured again at every request through the
  engine. A plan made while the volume was away is not kept (503 archive_offline).
- New sequence holds the converted copies alone: the dry run says the skipped
  photos are not copied into it (Archive.Convert.NotCopied.*, Server.Convert.OnPcNew),
  the panel's totals, column and confirmation give the count, and so does the
  desktop's summary.
- Panel: the preview is asked in Replace's neutral words; a dry run cut off by a
  proxy is asked again while the totals show how long it takes, plan_dropped is
  retried once; an apply the server let go plans again, a changed one shows
  Admin.Convert.PlanChanged, and one with no answer is not resent (its job may be
  queued).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit f0b1e379ce7c08b6d3459ab25ca6214c8f366f6c)
This commit is contained in:
Alby96 committed 2026-10-27 22:15:51 +01:00
1 parent d0bbd77c92
commit 8174852a2c
22 files changed
+927 -342

No files matched your search

+1 -1
View File
@@ -86,7 +86,7 @@ Dependencies point one way: `Core` ← `Core.Windows`, `Data`, `Protocol` ← `S
| `Ui/Controls` | `ChronosWindow` (chrome, language direction), `NavRail`, `JobBar`, `ToastHost` (`Toast.Closed` once a toast leaves the screen), `Field` (placeholder, icon, error), `Chip`, `AutoSlider`, `StatusGlyph`, `EmptyState`, `Icon`, `Bidi` (right-to-left rules), `Motion` (animations off for screenshots) |
| `Ui/Interop` | DWM calls of the window chrome; `TrayIcon` (`Shell_NotifyIconW`: the Uploader's tray, the client's Windows notifications) |
| `Ui/Visuals` | `DrawingElement` (hover readout, item keyboard and activation) and its charts (`BarChart`, `SeriesChart`, `StackedBar`, `HeatmapStrip`, `TimelineBands`), `Filmstrip`, `PreviewViewport`, `CurveEditor`, `SequenceCard`, `VirtualizingWrapPanel`, `SegmentedControl`, `SplitButton` |
| `Server/Api`, `Server/Events` | Endpoint groups (`XxxEndpoints.Map`; `ScheduleEndpoints`: `/api/v1/schedules`; `LibraryEndpoints`: the Library filters' facets, sequence thumbnail, photo delete, bulk delete/retag/jobs; `TrashEndpoints`: list, restore, empty; `ArchiveEndpoints`: settings, statistics, scan jobs, Reorganize's dry run and `move` job, with `ReorganizePlanCache`: a dry run's plan, made apart from its request, kept a few minutes while what it reads from the index stays, for the same dry run and its confirmation; Convert's dry run and `convert` job), API errors, the WebSocket `EventHub` |
| `Server/Api`, `Server/Events` | Endpoint groups (`XxxEndpoints.Map`; `ScheduleEndpoints`: `/api/v1/schedules`; `LibraryEndpoints`: the Library filters' facets, sequence thumbnail, photo delete, bulk delete/retag/jobs; `TrashEndpoints`: list, restore, empty; `ArchiveEndpoints`: settings, statistics, scan jobs, Reorganize's dry run and `move` job, Convert's dry run and `convert` job, both through `PlanCache`: a dry run's plan, made apart from its request, a few at once, kept a few minutes while what it reads from the index stays (its `PlanKey`), for the same dry run and its confirmation), API errors, the WebSocket `EventHub` |
| `Server/Security` | Accounts, password hashing, tokens, session cookies, login guard, access rules, audit log |
| `Server/Storage`, `Server/Jobs` | Archive and processing modules (the archive module's `SequenceChanged` publishes `sequenceChanged` `reorganized`), uploads (`UploadMaintenance`: recovery at start, and the sweep that is the `uploads` clean-up step; `UploadService.ReceiveAgainAsync`: a verified upload whose filing was refused for now goes back to receiving, 503), render store (files a render under the sequence's lease, where the index has the folder then; `FolderMovedException`, 503 `folder_moved`, when the sequence is not in it); `JobEngineHost`, `SchedulerHost` (the `Scheduler` on the engine, started after it; archive.json read every 10 minutes whatever the scan schedule does; every schedule change and run published as a `scheduleChanged` event) and `ServerScheduleHost` (time zone, archive refreshed from archive.json, system schedules), render node service, monitor and availability |
| `Server/Hosting`, `Server/Admin/web` | `ServerHost`, `ServerLogging` (console and file levels the logging filters read at every line, the file sink's limits, the engine's level; applied again when Settings › Logs is saved), certificates, discovery beacon, embedded static files (`EmbeddedStaticFiles`: `no-cache` and a content ETag, so an upgrade never mixes old and new modules), Windows service (start and stop failures in the Application event log); the admin panel (ES modules, one per page; `schedules.js` is the Jobs page's Scheduled tab; the Library's views in `pages/library/`: `sequence.js` a sequence and its photo viewer, `trash.js` the archive's trash, `shared.js` cards, selection, grid keyboard and the retry of thumbnails a finished job may have built; `upload.js` the upload dialog the Library opens on demand, hashing in the module worker `hash-worker.js` with the in-house incremental SHA-256 of `sha256.js`, self-tested before its first use, and sending the capture time `capture-time.js` reads: the Exif DateTimeOriginal of a JPEG or TIFF-based file, else the file's time; `reorganize.js` the Reorganize dialog of the Library's selection and of the Archive page's save toast: template switches and fields, a live dry run of `POST /archive/reorganize` one request at a time, Apply that sends the shown plan's fingerprint back; `convert.js` the Convert dialog of the Library's selection: format, JPEG quality and a mode chosen at every run (Replace or New sequence, never preselected), a debounced dry run of `POST /archive/convert` per sequence (photos converted, left as they are and why, blocked, size before and after, the space line), Apply that confirms Replace by a typed word and sends the shown plan's fingerprint back) |
+8 -1
View File
@@ -92,6 +92,7 @@
<data name="Admin.Convert.ChooseMode" xml:space="preserve"><value>Choose Replace or New sequence to convert.</value><comment>long</comment></data>
<data name="Admin.Convert.ChooseMode.Tip" xml:space="preserve"><value>Choose Replace or New sequence first</value></data>
<data name="Admin.Convert.Column.Converted" xml:space="preserve"><value>Converted</value></data>
<data name="Admin.Convert.Column.NotCopied" xml:space="preserve"><value>Not copied</value></data>
<data name="Admin.Convert.Column.Sequence" xml:space="preserve"><value>Sequence</value></data>
<data name="Admin.Convert.Column.Size" xml:space="preserve"><value>Size</value></data>
<data name="Admin.Convert.Column.Unchanged" xml:space="preserve"><value>Unchanged</value></data>
@@ -101,6 +102,8 @@
<data name="Admin.Convert.Confirm.Replace.Tip" xml:space="preserve"><value>Queue the job that converts the photos, verifies each conversion and moves the originals to the trash</value></data>
<data name="Admin.Convert.ConfirmNew.one" xml:space="preserve"><value>{0} photo is converted into a copy in a new sequence; the original stays as it is.</value><comment>long</comment></data>
<data name="Admin.Convert.ConfirmNew.other" xml:space="preserve"><value>{0} photos are converted into copies in a new sequence beside each one; the originals stay as they are.</value><comment>long</comment></data>
<data name="Admin.Convert.ConfirmNewNotCopied.one" xml:space="preserve"><value>{0} photo is not copied into the new sequence, for the reason the preview gives.</value><comment>long</comment></data>
<data name="Admin.Convert.ConfirmNewNotCopied.other" xml:space="preserve"><value>{0} photos are not copied into the new sequences, for the reasons the preview gives.</value><comment>long</comment></data>
<data name="Admin.Convert.ConfirmNewTitle" xml:space="preserve"><value>Convert to {0} in new sequences?</value><comment>long</comment></data>
<data name="Admin.Convert.ConfirmReplace.one" xml:space="preserve"><value>{0} photo in {1} is replaced by its conversion; the original goes to the trash, restorable until it is emptied.</value><comment>long</comment></data>
<data name="Admin.Convert.ConfirmReplace.other" xml:space="preserve"><value>{0} photos in {1} are replaced by their conversions; the originals go to the trash, restorable until it is emptied.</value><comment>long</comment></data>
@@ -118,14 +121,17 @@
<data name="Admin.Convert.Mode.NewSequence.Tip" xml:space="preserve"><value>The converted copies go into a new sequence beside each one; the originals and their sequence stay exactly as they are</value></data>
<data name="Admin.Convert.Mode.Replace" xml:space="preserve"><value>Replace</value></data>
<data name="Admin.Convert.Mode.Replace.Tip" xml:space="preserve"><value>The converted files become the sequence's photos; the originals go to the archive's trash, restorable until it is emptied</value></data>
<data name="Admin.Convert.NoAnswer" xml:space="preserve"><value>The server did not answer: check on the Jobs page whether the conversion was queued before converting again.</value><comment>long</comment></data>
<data name="Admin.Convert.NoSpace.Tip" xml:space="preserve"><value>Free some space on the archive's volume first</value></data>
<data name="Admin.Convert.Nothing.Tip" xml:space="preserve"><value>Nothing in the preview converts</value></data>
<data name="Admin.Convert.NothingToConvert" xml:space="preserve"><value>Nothing to convert: every photo stays as it is, for the reasons shown.</value><comment>long</comment></data>
<data name="Admin.Convert.OpenJob" xml:space="preserve"><value>Open job</value></data>
<data name="Admin.Convert.OpenJob.Tip" xml:space="preserve"><value>Show the job and its log on the Jobs page</value></data>
<data name="Admin.Convert.PlanItem" xml:space="preserve"><value>{0} · {1} · {2}</value><comment>invariant; a sequence's title, the photos it converts, and their size before and after</comment></data>
<data name="Admin.Convert.PlanChanged" xml:space="preserve"><value>The archive changed during the confirmation; check the new preview and apply again.</value><comment>long</comment></data>
<data name="Admin.Convert.PlanChanged" xml:space="preserve"><value>The archive changed since the preview; check the new one and apply again.</value><comment>long</comment></data>
<data name="Admin.Convert.Planning" xml:space="preserve"><value>Planning…</value></data>
<data name="Admin.Convert.Planning.Tip" xml:space="preserve"><value>The server reads every photo folder of the selection: a large archive takes minutes, and the preview appears once it is done</value></data>
<data name="Admin.Convert.PlanningFor" xml:space="preserve"><value>Planning… {0}</value><comment>{0} how long the dry run has taken so far, e.g. "2 min"</comment></data>
<data name="Admin.Convert.Preview" xml:space="preserve"><value>Preview</value></data>
<data name="Admin.Convert.Preview.Tip" xml:space="preserve"><value>What each sequence converts and leaves as it is, with the size before and after (an estimate); nothing has changed yet</value></data>
<data name="Admin.Convert.Quality" xml:space="preserve"><value>Quality</value></data>
@@ -141,6 +147,7 @@
<data name="Admin.Convert.TitleAll" xml:space="preserve"><value>Convert all sequences</value></data>
<data name="Admin.Convert.TooMany" xml:space="preserve"><value>Convert takes at most {0} sequences at once, or every sequence: clear the filters and select all.</value><comment>long</comment></data>
<data name="Admin.Convert.Totals" xml:space="preserve"><value>Sequences converted: {0} · photos: {1} · unchanged: {2} · size: {3} → {4}</value><comment>long</comment></data>
<data name="Admin.Convert.TotalsNew" xml:space="preserve"><value>Sequences converted: {0} · photos: {1} · not copied: {2} · size: {3} → {4}</value><comment>long: the totals with New sequence chosen; {2} the photos the new sequences lack</comment></data>
<data name="Admin.Dashboard.Activity" xml:space="preserve"><value>Recent activity</value></data>
<data name="Admin.Dashboard.AllLogs" xml:space="preserve"><value>All logs</value></data>
<data name="Admin.Dashboard.AllLogs.Tip" xml:space="preserve"><value>Open the audit log</value></data>
+8
View File
@@ -59,6 +59,14 @@
<data name="Archive.Convert.HeadingAll" xml:space="preserve"><value>Convert {0}</value><comment>long: dialog heading; {0} archive name</comment></data>
<data name="Archive.Convert.Mode" xml:space="preserve"><value>Mode</value></data>
<data name="Archive.Convert.NewSequence" xml:space="preserve"><value>New sequence</value></data>
<data name="Archive.Convert.NotCopied.Missing" xml:space="preserve"><value>Missing photos are not copied into the new sequence.</value><comment>long: why photos are not in the new sequence a conversion makes, in its dry run</comment></data>
<data name="Archive.Convert.NotCopied.NoNativeDecoder" xml:space="preserve"><value>Photos of a format this computer cannot read at full resolution are not copied into the new sequence.</value><comment>long: why photos are not in the new sequence a conversion makes, in its dry run</comment></data>
<data name="Archive.Convert.NotCopied.PreviewOnly" xml:space="preserve"><value>Photos read only from their embedded preview here are not copied into the new sequence.</value><comment>long: why photos are not in the new sequence a conversion makes, in its dry run</comment></data>
<data name="Archive.Convert.NotCopied.SameFormat" xml:space="preserve"><value>Photos already in this format are not copied into the new sequence.</value><comment>long: why photos are not in the new sequence a conversion makes, in its dry run</comment></data>
<data name="Archive.Convert.NotCopied.Undecodable" xml:space="preserve"><value>Unreadable photos are not copied into the new sequence.</value><comment>long: why photos are not in the new sequence a conversion makes, in its dry run</comment></data>
<data name="Archive.Convert.NotCopied.Unhashed" xml:space="preserve"><value>Photos never verified are not copied into the new sequence; verify the sequence first.</value><comment>long: why photos are not in the new sequence a conversion makes, in its dry run</comment></data>
<data name="Archive.Convert.NotCopiedPhotos.one" xml:space="preserve"><value>{0:N0} not copied</value><comment>photos a conversion into a new sequence leaves out of it</comment></data>
<data name="Archive.Convert.NotCopiedPhotos.other" xml:space="preserve"><value>{0:N0} not copied</value><comment>photos a conversion into a new sequence leaves out of it</comment></data>
<data name="Archive.Convert.NotPlaced" xml:space="preserve"><value>The converted photos could not take their place. {0}</value><comment>long: job log; {0} why (one sentence)</comment></data>
<data name="Archive.Convert.PhotoFailed" xml:space="preserve"><value>{0} could not be converted. {1}</value><comment>long: job log; {0} the photo's file, {1} why (one sentence)</comment></data>
<data name="Archive.Convert.Quality" xml:space="preserve"><value>Quality</value></data>
+1
View File
@@ -24,6 +24,7 @@
<data name="Server.Conflict.TakeServer.Title" xml:space="preserve"><value>Take the server's settings?</value></data>
<data name="Server.Connect" xml:space="preserve"><value>Connect</value></data>
<data name="Server.Convert.OnPc" xml:space="preserve"><value>Convert these photos on a PC: the server cannot read them at full resolution.</value><comment>long: why photos are not converted in a conversion dry run on the server</comment></data>
<data name="Server.Convert.OnPcNew" xml:space="preserve"><value>Not copied into the new sequence: the server cannot read these photos at full resolution; convert them on a PC.</value><comment>long: why photos are not in the new sequence a conversion on the server makes, in its dry run</comment></data>
<data name="Server.Disconnect" xml:space="preserve"><value>Disconnect</value></data>
<data name="Server.Empty" xml:space="preserve"><value>Not connected to a server</value></data>
<data name="Server.Error.AccountLocked" xml:space="preserve"><value>Too many failed sign-ins; the account is locked for a while.</value><comment>long</comment></data>
+8 -1
View File
@@ -92,6 +92,7 @@
<data name="Admin.Convert.ChooseMode" xml:space="preserve"><value>Scegli Sostituisci o Nuova sequenza per convertire.</value></data>
<data name="Admin.Convert.ChooseMode.Tip" xml:space="preserve"><value>Prima scegli Sostituisci o Nuova sequenza</value></data>
<data name="Admin.Convert.Column.Converted" xml:space="preserve"><value>Convertite</value></data>
<data name="Admin.Convert.Column.NotCopied" xml:space="preserve"><value>Non copiate</value></data>
<data name="Admin.Convert.Column.Sequence" xml:space="preserve"><value>Sequenza</value></data>
<data name="Admin.Convert.Column.Size" xml:space="preserve"><value>Dimensione</value></data>
<data name="Admin.Convert.Column.Unchanged" xml:space="preserve"><value>Invariate</value></data>
@@ -101,6 +102,8 @@
<data name="Admin.Convert.Confirm.Replace.Tip" xml:space="preserve"><value>Metti in coda il job che converte le foto, verifica ogni conversione e sposta gli originali nel cestino</value></data>
<data name="Admin.Convert.ConfirmNew.one" xml:space="preserve"><value>{0} foto viene convertita in una copia in una nuova sequenza; l'originale resta com'è.</value></data>
<data name="Admin.Convert.ConfirmNew.other" xml:space="preserve"><value>{0} foto vengono convertite in copie, in una nuova sequenza accanto a ognuna; gli originali restano come sono.</value></data>
<data name="Admin.Convert.ConfirmNewNotCopied.one" xml:space="preserve"><value>{0} foto non viene copiata nella nuova sequenza, per il motivo indicato nell'anteprima.</value></data>
<data name="Admin.Convert.ConfirmNewNotCopied.other" xml:space="preserve"><value>{0} foto non vengono copiate nelle nuove sequenze, per i motivi indicati nell'anteprima.</value></data>
<data name="Admin.Convert.ConfirmNewTitle" xml:space="preserve"><value>Convertire in {0} in nuove sequenze?</value></data>
<data name="Admin.Convert.ConfirmReplace.one" xml:space="preserve"><value>{0} foto in {1} viene sostituita dalla sua conversione; l'originale va nel cestino, ripristinabile finché non viene svuotato.</value></data>
<data name="Admin.Convert.ConfirmReplace.other" xml:space="preserve"><value>{0} foto in {1} vengono sostituite dalle loro conversioni; gli originali vanno nel cestino, ripristinabili finché non viene svuotato.</value></data>
@@ -116,13 +119,16 @@
<data name="Admin.Convert.Mode.NewSequence.Tip" xml:space="preserve"><value>Le copie convertite vanno in una nuova sequenza accanto a ognuna; gli originali e la loro sequenza restano esattamente come sono</value></data>
<data name="Admin.Convert.Mode.Replace" xml:space="preserve"><value>Sostituisci</value></data>
<data name="Admin.Convert.Mode.Replace.Tip" xml:space="preserve"><value>I file convertiti diventano le foto della sequenza; gli originali vanno nel cestino dell'archivio, ripristinabili finché non viene svuotato</value></data>
<data name="Admin.Convert.NoAnswer" xml:space="preserve"><value>Il server non ha risposto: controlla nella pagina Attività se la conversione è stata accodata prima di convertire di nuovo.</value></data>
<data name="Admin.Convert.NoSpace.Tip" xml:space="preserve"><value>Prima libera spazio sul volume dell'archivio</value></data>
<data name="Admin.Convert.Nothing.Tip" xml:space="preserve"><value>Nell'anteprima non c'è niente da convertire</value></data>
<data name="Admin.Convert.NothingToConvert" xml:space="preserve"><value>Niente da convertire: ogni foto resta com'è, per i motivi indicati.</value></data>
<data name="Admin.Convert.OpenJob" xml:space="preserve"><value>Apri job</value></data>
<data name="Admin.Convert.OpenJob.Tip" xml:space="preserve"><value>Mostra il job e il suo log nella pagina Job</value></data>
<data name="Admin.Convert.PlanChanged" xml:space="preserve"><value>L'archivio è cambiato durante la conferma; controlla la nuova anteprima e applica di nuovo.</value></data>
<data name="Admin.Convert.PlanChanged" xml:space="preserve"><value>L'archivio è cambiato dopo l'anteprima; controlla quella nuova e applica di nuovo.</value></data>
<data name="Admin.Convert.Planning" xml:space="preserve"><value>Pianificazione…</value></data>
<data name="Admin.Convert.Planning.Tip" xml:space="preserve"><value>Il server legge ogni cartella di foto della selezione: un archivio grande richiede minuti, e l'anteprima compare quando ha finito</value></data>
<data name="Admin.Convert.PlanningFor" xml:space="preserve"><value>Pianificazione… {0}</value></data>
<data name="Admin.Convert.Preview" xml:space="preserve"><value>Anteprima</value></data>
<data name="Admin.Convert.Preview.Tip" xml:space="preserve"><value>Cosa converte e cosa lascia com'è ogni sequenza, con la dimensione prima e dopo (una stima); niente è ancora cambiato</value></data>
<data name="Admin.Convert.Quality" xml:space="preserve"><value>Qualità</value></data>
@@ -137,6 +143,7 @@
<data name="Admin.Convert.TitleAll" xml:space="preserve"><value>Converti tutte le sequenze</value></data>
<data name="Admin.Convert.TooMany" xml:space="preserve"><value>Converti al massimo {0} sequenze alla volta, o tutte: togli i filtri e seleziona tutto.</value></data>
<data name="Admin.Convert.Totals" xml:space="preserve"><value>Sequenze convertite: {0} · foto: {1} · invariate: {2} · dimensione: {3} → {4}</value></data>
<data name="Admin.Convert.TotalsNew" xml:space="preserve"><value>Sequenze convertite: {0} · foto: {1} · non copiate: {2} · dimensione: {3} → {4}</value></data>
<data name="Admin.Dashboard.Activity" xml:space="preserve"><value>Attività recenti</value></data>
<data name="Admin.Dashboard.AllLogs" xml:space="preserve"><value>Tutti i log</value></data>
<data name="Admin.Dashboard.AllLogs.Tip" xml:space="preserve"><value>Apri il registro di controllo</value></data>
+8
View File
@@ -56,6 +56,14 @@
<data name="Archive.Convert.HeadingAll" xml:space="preserve"><value>Converti {0}</value></data>
<data name="Archive.Convert.Mode" xml:space="preserve"><value>Modalità</value></data>
<data name="Archive.Convert.NewSequence" xml:space="preserve"><value>Nuova sequenza</value></data>
<data name="Archive.Convert.NotCopied.Missing" xml:space="preserve"><value>Le foto mancanti non vengono copiate nella nuova sequenza.</value></data>
<data name="Archive.Convert.NotCopied.NoNativeDecoder" xml:space="preserve"><value>Le foto di un formato che questo computer non legge a piena risoluzione non vengono copiate nella nuova sequenza.</value></data>
<data name="Archive.Convert.NotCopied.PreviewOnly" xml:space="preserve"><value>Le foto lette qui solo dalla loro anteprima incorporata non vengono copiate nella nuova sequenza.</value></data>
<data name="Archive.Convert.NotCopied.SameFormat" xml:space="preserve"><value>Le foto già in questo formato non vengono copiate nella nuova sequenza.</value></data>
<data name="Archive.Convert.NotCopied.Undecodable" xml:space="preserve"><value>Le foto illeggibili non vengono copiate nella nuova sequenza.</value></data>
<data name="Archive.Convert.NotCopied.Unhashed" xml:space="preserve"><value>Le foto mai verificate non vengono copiate nella nuova sequenza; verifica prima la sequenza.</value></data>
<data name="Archive.Convert.NotCopiedPhotos.one" xml:space="preserve"><value>{0:N0} non copiata</value></data>
<data name="Archive.Convert.NotCopiedPhotos.other" xml:space="preserve"><value>{0:N0} non copiate</value></data>
<data name="Archive.Convert.NotPlaced" xml:space="preserve"><value>Le foto convertite non hanno potuto prendere il loro posto. {0}</value></data>
<data name="Archive.Convert.PhotoFailed" xml:space="preserve"><value>Impossibile convertire {0}. {1}</value></data>
<data name="Archive.Convert.Quality" xml:space="preserve"><value>Qualità</value></data>
+1
View File
@@ -22,6 +22,7 @@
<data name="Server.Conflict.TakeServer.Title" xml:space="preserve"><value>Usare le impostazioni del server?</value></data>
<data name="Server.Connect" xml:space="preserve"><value>Connetti</value></data>
<data name="Server.Convert.OnPc" xml:space="preserve"><value>Converti queste foto su un PC: il server non le legge a piena risoluzione.</value></data>
<data name="Server.Convert.OnPcNew" xml:space="preserve"><value>Non copiate nella nuova sequenza: il server non legge queste foto a piena risoluzione; convertile su un PC.</value></data>
<data name="Server.Disconnect" xml:space="preserve"><value>Disconnetti</value></data>
<data name="Server.Empty" xml:space="preserve"><value>Non connesso a un server</value></data>
<data name="Server.Error.AccountLocked" xml:space="preserve"><value>Troppi accessi falliti; l'account è bloccato per un po'.</value></data>
@@ -124,6 +124,11 @@ public sealed class ConvertDialogViewModel : ArchiveDialogViewModel
{
if (value is { } mode && !Enum.IsDefined(mode)) return;
if (!Set(ref _mode, value)) return;
// The rows say what becomes of the photos skipped in the mode's words: not copied into a new sequence.
lock (_gate)
{
if (_run.Plan is { } plan) _run = _run with { Rows = RowsOf(plan) };
}
OnPropertyChanged(nameof(IsReplace));
OnPropertyChanged(nameof(IsNewSequence));
OnPlanChanged();
@@ -218,7 +223,7 @@ public sealed class ConvertDialogViewModel : ArchiveDialogViewModel
{
_planning?.Cancel();
_planning = null;
_run = new DryRun(Current(), plan, ConvertPlans.Rows(plan));
_run = new DryRun(Current(), plan, RowsOf(plan));
}
BusyText = null;
ClearError();
@@ -238,11 +243,14 @@ public sealed class ConvertDialogViewModel : ArchiveDialogViewModel
Quality.Refresh();
lock (_gate)
{
if (_run.Plan is { } plan) _run = _run with { Rows = ConvertPlans.Rows(plan) };
if (_run.Plan is { } plan) _run = _run with { Rows = RowsOf(plan) };
}
base.OnLanguageChanged();
}
/// <summary>The rows of <paramref name="plan"/> in the mode chosen (as planned while none is: the skipped photos stay as they are).</summary>
private IReadOnlyList<ConvertRow> RowsOf(ConvertPlan plan) => ConvertPlans.Rows(_mode is { } mode ? plan with { Mode = mode } : plan);
/// <summary>Typing makes the dry run on screen old at once (Apply waits) and plans again once the typing pauses.</summary>
private void OnTyped()
{
@@ -290,7 +298,8 @@ public sealed class ConvertDialogViewModel : ArchiveDialogViewModel
try
{
var plan = await Task.Run(() => _archives.PlanConvertAsync(_archive.Id, set, options!, mode, cts.Token), cts.Token);
result = new DryRun(options, plan, ConvertPlans.Rows(plan));
// In the mode chosen now: it may have been chosen while the plan was made.
result = new DryRun(options, plan, RowsOf(plan));
}
catch (OperationCanceledException) when (cts.IsCancellationRequested)
{
+17 -6
View File
@@ -42,14 +42,22 @@ public static class ConvertPlans
/// <summary>A valid quality as typed (see <see cref="ValidateQuality"/>).</summary>
public static int ParseQuality(string text) => int.Parse(text.Trim(), NumberStyles.Integer, Loc.Culture);
/// <summary>The rows of a plan in its order: every sequence that converts, or whose photos stay as they are, with why.</summary>
/// <summary>
/// The rows of a plan in its order: every sequence that converts, or whose photos stay as they are, with why in the
/// words of the plan's mode (<see cref="Row"/>).
/// </summary>
public static IReadOnlyList<ConvertRow> Rows(ConvertPlan plan)
{
ArgumentNullException.ThrowIfNull(plan);
return [.. plan.Sequences.Where(s => s.Frames.Count > 0 || s.Skipped.Count > 0 || s.Blocked.Count > 0).Select(Row)];
return [.. plan.Sequences.Where(s => s.Frames.Count > 0 || s.Skipped.Count > 0 || s.Blocked.Count > 0).Select(s => Row(s, plan.Mode))];
}
public static ConvertRow Row(ConvertSequencePlan step)
/// <summary>
/// One sequence of a plan: what converts, then what blocks it, then the photos skipped with why. With
/// <see cref="ConvertMode.NewSequence"/> those are not copied into the new sequence, which holds the converted copies
/// alone; with <see cref="ConvertMode.Replace"/> they stay as they are.
/// </summary>
public static ConvertRow Row(ConvertSequencePlan step, ConvertMode mode = ConvertMode.Replace)
{
ArgumentNullException.ThrowIfNull(step);
var named = !string.IsNullOrWhiteSpace(step.Title);
@@ -59,7 +67,8 @@ public static class ConvertPlans
: string.Empty;
// What blocks the whole sequence comes first: the photos it would skip anyway do not matter until it can convert.
List<string> kept = [.. step.Blocked.Distinct().Select(b => Loc.L("Archive.Convert.Blocked." + b))];
kept.AddRange(step.Skipped.Select(s => Loc.L("Archive.Convert.Skipped." + s.Reason) + Loc.Separator + Loc.Plural("Shell.Status.Photos", s.Frames)));
var why = mode == ConvertMode.NewSequence ? "Archive.Convert.NotCopied." : "Archive.Convert.Skipped.";
kept.AddRange(step.Skipped.Select(s => Loc.L(why + s.Reason) + Loc.Separator + Loc.Plural("Shell.Status.Photos", s.Frames)));
return new ConvertRow(step.SequenceId, name, !named, note, kept, step.Blocked.Count > 0);
}
@@ -68,7 +77,8 @@ public static class ConvertPlans
/// <summary>
/// "12 of 36 change · 3,600 photos · 41 GB → 32 GB · 20 skipped": the sequences converted of those planned, then the
/// photos and their sizes when something converts, the photos skipped when there are any.
/// photos and their sizes when something converts, the photos skipped when there are any ("20 not copied" with
/// <see cref="ConvertMode.NewSequence"/>: the new sequences lack them).
/// </summary>
public static string Summary(ConvertPlan plan)
{
@@ -79,7 +89,8 @@ public static class ConvertPlans
parts.Add(Loc.Plural("Shell.Status.Photos", plan.Frames));
parts.Add(Sizes(plan.BytesBefore, plan.BytesAfter));
}
if (plan.Skipped > 0) parts.Add(Loc.Plural("Archive.Convert.SkippedPhotos", plan.Skipped));
if (plan.Skipped > 0)
parts.Add(Loc.Plural(plan.Mode == ConvertMode.NewSequence ? "Archive.Convert.NotCopiedPhotos" : "Archive.Convert.SkippedPhotos", plan.Skipped));
return string.Join(Loc.Separator, parts);
}
@@ -469,6 +469,50 @@ public sealed class SequenceRepository
}
}
/// <summary>
/// A digest of what a conversion of <paramref name="sequenceIds"/> (null: every sequence) plans from in the index of
/// <paramref name="archiveId"/> (owner review 2, R7): what a Reorganize's does (<see cref="GetStampAsync"/>: every sequence
/// and its folder, which decide what "every sequence" is and its order; of the sequences planned, their title, photo
/// count, frames version and how many of their photos are hashed), and of the photos of the sequences planned, what
/// decides whether each converts and the size it is estimated at: their format, size and pixel size, and the flags a
/// scan, a verify or a decode records without another frames version (missing, corrupt, read only from the preview,
/// undecodable). Another value once one of these changes. A change made on disk alone, which the index does not have
/// (a file gone before a scan finds it), leaves it as it was.
/// </summary>
public async Task<string> GetConvertStampAsync(Guid archiveId, IReadOnlyCollection<Guid>? sequenceIds, CancellationToken cancellationToken = default)
{
var stamp = await GetStampAsync(archiveId, sequenceIds, cancellationToken).ConfigureAwait(false);
// Per sequence planned: its photos, their bytes, formats and pixel sizes, and how many carry each flag that decides.
var columns = "count(*), COALESCE(SUM(f.size), 0), COALESCE(SUM(f.format), 0), COALESCE(SUM(f.width), 0), COALESCE(SUM(f.height), 0), " +
string.Join(", ", new[] { FrameFlags.Missing, FrameFlags.Corrupt, FrameFlags.PreviewOnly, FrameFlags.Undecodable }
.Select(flag => $"COALESCE(SUM(CASE WHEN (f.flags & {(int)flag}) <> 0 THEN 1 ELSE 0 END), 0)"));
static (Guid Id, string Photos) Read(DbRow r) =>
(r.GetGuid(0), string.Join('\u001f', Enumerable.Range(1, 9).Select(i => r.GetInt64(i).ToString(CultureInfo.InvariantCulture))));
var sums = new List<(Guid Id, string Photos)>();
if (sequenceIds is null)
{
sums.AddRange(await _db.ListAsync(
$"SELECT f.sequence_id, {columns} FROM frames f JOIN sequences s ON s.id = f.sequence_id WHERE s.archive_id = $1 GROUP BY f.sequence_id",
[archiveId], Read, cancellationToken).ConfigureAwait(false));
}
else
{
// 500 ids per statement stays below SQLite's historical 999-variable limit.
foreach (var chunk in sequenceIds.Distinct().Chunk(500))
{
var p = new SqlParameters();
var ids = string.Join(", ", chunk.Select(id => p.Add(id)));
sums.AddRange(await _db.ListAsync($"SELECT f.sequence_id, {columns} FROM frames f WHERE f.sequence_id IN ({ids}) GROUP BY f.sequence_id",
p.ToArray(), Read, cancellationToken).ConfigureAwait(false));
}
}
using var hash = IncrementalHash.CreateHash(HashAlgorithmName.SHA256);
hash.AppendData(Encoding.UTF8.GetBytes(stamp));
foreach (var (id, photos) in sums.OrderBy(s => s.Id))
hash.AppendData(Encoding.UTF8.GetBytes(string.Create(CultureInfo.InvariantCulture, $"\n{id:N}\u001f{photos}")));
return Convert.ToHexStringLower(hash.GetHashAndReset());
}
/// <summary>
/// How many photos of each of <paramref name="sequenceIds"/> (null: every sequence of <paramref name="archiveId"/>) have
/// their SHA-256 in the index; a sequence without one is not listed.
+9 -4
View File
@@ -131,9 +131,13 @@ public sealed record ReorganizePlanDto(IReadOnlyList<ReorganizeStepDto> Steps, i
/// <see cref="Target"/> (<c>jpeg</c>, <c>tiff16</c> or <c>linear-dng</c>, the archival choice) with <see cref="Quality"/> 1–100
/// for a JPEG (default 95, ignored otherwise). <see cref="Mode"/>, chosen at every run: <c>replace</c> (the converted files
/// become the frames, the originals go to the archive's trash) or <c>new-sequence</c> (copies in a new sequence, the source
/// untouched). <see cref="DryRun"/> answers the plan (<see cref="ConvertPlanDto"/>) and changes nothing; the change itself
/// sends that plan's <see cref="Fingerprint"/> back (409 <c>plan_changed</c> when the archive changed since) and runs as a
/// <c>convert</c> job (202).
/// untouched, the photos it leaves as they are not copied into it). <see cref="DryRun"/> answers the plan
/// (<see cref="ConvertPlanDto"/>) and changes nothing; the server keeps it a few minutes, a refusal too, so the same dry run
/// again (also after one cut off, in either mode, at any quality of a lossless target) is answered from it with the free
/// space measured again; 503 <c>plan_dropped</c> when the server stopped making it (it took too long, or too many were
/// asked at once). The change itself sends that plan's <see cref="Fingerprint"/> back (409 <c>plan_changed</c> when the
/// index changed since in what the plan reads: the sequences' rows, and the formats, sizes, hashes and flags of their
/// photos) and runs as a <c>convert</c> job (202), which plans again and leaves as it is a sequence that no longer fits.
/// </summary>
public sealed record ConvertRequest(IReadOnlyList<Guid>? SequenceIds, bool All, string? Target, int? Quality, string? Mode, bool DryRun,
string? Fingerprint = null);
@@ -143,7 +147,8 @@ public sealed record ConvertRequest(IReadOnlyList<Guid>? SequenceIds, bool All,
/// server cannot read them at full resolution: a CR3, a HEIC, a body its RAW decoder does not support), <c>undecodable</c>,
/// <c>missing</c>, <c>unhashed</c> (never verified), <c>eight_bit</c> (a RAW the host develops at 8 bits only, for a 16-bit
/// target), <c>color_profile</c> (a colour profile the target can neither keep nor convert to) or <c>too_large</c>, how
/// many, and the reason as one localised sentence.
/// many, and the reason as one localised sentence: with <c>new-sequence</c>, that the new sequence lacks them (it holds the
/// converted copies alone).
/// </summary>
public sealed record ConvertSkippedDto(string Reason, int Photos, string Message);
+84 -19
View File
@@ -1,12 +1,14 @@
// Convert dialog of the admin panel (owner review 2, R7; BRIEF §12.5, §14.8): the photos of the Library's selection, or of
// every sequence, converted to JPEG, TIFF-16 or linear DNG. What happens to the originals is chosen at every run, never
// preselected: Replace (the converted files become the sequence's photos, the originals go to the archive's trash) or New
// sequence (converted copies in a new sequence, the source untouched). Each change asks the server for a dry run (nothing
// is touched) and the table shows, per sequence, what converts, what stays as it is and why, and the size before and
// after. Apply confirms that very plan by its fingerprint (Replace by typing a word), and the server runs it as a job
// (kind "convert") that verifies every converted file before anything changes.
// sequence (converted copies in a new sequence, the source untouched; the photos left as they are are not in it). Each
// change asks the server for a dry run (nothing is touched) and the table shows, per sequence, what converts, what stays
// as it is and why, and the size before and after. The server makes a dry run apart from the request and keeps it a few
// minutes, so one that takes longer than a proxy lets a request last is asked for again until it is made. Apply confirms
// that very plan by its fingerprint (Replace by typing a word), and the server runs it as a job (kind "convert") that
// verifies every converted file before anything changes.
import { t, plural, applyStrings } from "./i18n.js";
import { escapeHtml, attr, formatBytes, formatNumber, toast, errorText, confirmDialog, markInvalid, clearInvalid } from "./ui.js";
import { escapeHtml, attr, formatBytes, formatNumber, formatDuration, toast, errorText, confirmDialog, markInvalid, clearInvalid } from "./ui.js";
import { listed } from "./pages/library/shared.js";
const ENDPOINT = "/api/v1/archive/convert";
@@ -14,6 +16,16 @@ const ENDPOINT = "/api/v1/archive/convert";
const PLAN_DELAY = 400;
const PAGE = 200;
const DEFAULT_QUALITY = 95;
/**
* A dry run the server is still making: a request cut off after CUT_AFTER ms or by a proxy's timeout (502, 504, 524, or
* no answer) is asked again after RETRY_DELAY ms, for up to PLAN_PATIENCE ms (the server stops a plan after 15 minutes);
* one the server stopped (503 plan_dropped) DROPPED_RETRIES times more. Past SLOW_PLAN seconds the totals tell how long.
*/
const CUT_AFTER = 20000;
const RETRY_DELAY = 2000;
const PLAN_PATIENCE = 16 * 60 * 1000;
const DROPPED_RETRIES = 1;
const SLOW_PLAN = 10;
/** The formats, as the API names them, with their strings; linear DNG, the archival choice, is the one chosen first. */
const TARGETS = [
{ value: "jpeg", label: "Admin.Convert.Format.Jpeg", tip: "Admin.Convert.Format.Jpeg.Tip" },
@@ -27,10 +39,12 @@ const MODES = [
{ value: "new-sequence", label: "Admin.Convert.Mode.NewSequence", tip: "Admin.Convert.Mode.NewSequence.Tip" },
];
/**
* The mode the dry run is asked with while none is chosen: what converts does not depend on the mode, so the table shows
* the plan at once; Apply waits for a choice, and a choice other than this one plans again (the fingerprint names it).
* The mode the dry run is asked with while none is chosen, whose words say only that photos stay as they are: what
* converts does not depend on the mode, so the table shows the plan at once; Apply waits for a choice, and New sequence
* asks again (the fingerprint names the mode, and the rows say what the new sequence lacks), answered from the plan the
* server keeps.
*/
const PREVIEW_MODE = "new-sequence";
const PREVIEW_MODE = "replace";
let dialogs = 0;
@@ -80,7 +94,7 @@ export function openConvert({ api, target, names = new Map() } = {}) {
<div class="table convert" role="table" aria-label="${attr(t("Admin.Convert.Preview"))}">
<div class="thead" role="row"><span role="columnheader" data-i18n="Admin.Convert.Column.Sequence"></span>
<span role="columnheader" class="end" data-i18n="Admin.Convert.Column.Converted"></span>
<span role="columnheader" data-i18n="Admin.Convert.Column.Unchanged"></span>
<span role="columnheader" class="convert-left" data-i18n="Admin.Convert.Column.Unchanged"></span>
<span role="columnheader" class="end" data-i18n="Admin.Convert.Column.Size"></span></div>
<div role="rowgroup" class="convert-rows"></div>
</div>
@@ -105,6 +119,7 @@ export function openConvert({ api, target, names = new Map() } = {}) {
const next = dialog.querySelector(".convert-next");
const preview = dialog.querySelector(".plan-preview");
const rows = dialog.querySelector(".convert-rows");
const leftHeader = dialog.querySelector(".convert-left");
const foot = dialog.querySelector(".table-foot");
const shownText = dialog.querySelector(".convert-shown");
const errorLine = dialog.querySelector(".error");
@@ -212,10 +227,18 @@ export function openConvert({ api, target, names = new Map() } = {}) {
const body = request();
const key = JSON.stringify(body);
planning = true;
totals.textContent = t("Admin.Convert.Planning");
// Waiting: how long it has taken, once that is long, with why in the tooltip.
const started = Date.now();
const showWait = () => {
const seconds = Math.floor((Date.now() - started) / 1000);
totals.textContent = seconds < SLOW_PLAN ? t("Admin.Convert.Planning") : t("Admin.Convert.PlanningFor", formatDuration(seconds));
};
const ticking = setInterval(showWait, 1000);
showWait();
totals.title = t("Admin.Convert.Planning.Tip");
drawButtons();
try {
const answer = await api.post(ENDPOINT, { ...body, dryRun: true });
const answer = await dryRun(body, key);
if (!dialog.open) return;
plan = { ...answer, key };
shown = PAGE;
@@ -228,6 +251,8 @@ export function openConvert({ api, target, names = new Map() } = {}) {
if (key === keyNow()) showProblem(err);
} finally {
planning = false;
clearInterval(ticking);
totals.title = "";
}
if (again) {
again = false;
@@ -238,6 +263,28 @@ export function openConvert({ api, target, names = new Map() } = {}) {
drawPlan();
}
/**
* The dry run of body. The server makes it apart from the request and keeps it a few minutes, so a request cut off on
* the way (a proxy's timeout) is asked again, and answered once the plan is made; a plan the server stopped (let go for
* newer ones, past its time) is asked for once more. Gives up past PLAN_PATIENCE, or once the fields changed or the
* dialog closed: the last failure is thrown.
*/
async function dryRun(body, key) {
const started = Date.now();
let dropped = 0;
for (;;) {
const sent = Date.now();
try {
return await api.post(ENDPOINT, { ...body, dryRun: true });
} catch (err) {
const retry = cutOff(err, Date.now() - sent) || (err?.code === "plan_dropped" && dropped++ < DROPPED_RETRIES);
if (!retry || Date.now() - started > PLAN_PATIENCE || !dialog.open || key !== keyNow()) throw err;
await new Promise((done) => setTimeout(done, RETRY_DELAY));
if (!dialog.open || key !== keyNow()) throw err;
}
}
}
function drawPlan() {
const current = plan !== null && plan.key === keyNow();
preview.classList.toggle("stale", !current);
@@ -247,8 +294,11 @@ export function openConvert({ api, target, names = new Map() } = {}) {
next.hidden = true;
preview.hidden = true;
} else {
// New sequence's words: the photos left as they are are not copied into the new sequence.
const copies = plan.mode === "new-sequence";
leftHeader.textContent = t(copies ? "Admin.Convert.Column.NotCopied" : "Admin.Convert.Column.Unchanged");
totals.textContent = plan.photos === 0 ? t("Admin.Convert.NothingToConvert")
: t("Admin.Convert.Totals", formatNumber(plan.sequences), formatNumber(plan.photos), formatNumber(plan.skipped),
: t(copies ? "Admin.Convert.TotalsNew" : "Admin.Convert.Totals", formatNumber(plan.sequences), formatNumber(plan.photos), formatNumber(plan.skipped),
formatBytes(plan.bytesBefore), formatBytes(plan.bytesAfter));
space.hidden = plan.photos === 0;
space.textContent = plan.space.message;
@@ -309,9 +359,9 @@ export function openConvert({ api, target, names = new Map() } = {}) {
/**
* Replace asks for a word to be typed (the originals leave the sequence for the trash); New sequence for a plain
* confirmation. The plan sent back is the one the table showed. The fields cannot change meanwhile, but a dry run
* scheduled before may answer during it: the same plan again (same request and fingerprint) is applied all the same; a
* different one is shown instead, and Apply waits for the user.
* confirmation that says how many photos the new sequences lack. The plan sent back is the one the table showed. The
* fields cannot change meanwhile, but a dry run scheduled before may answer during it: the same plan again (same
* request and fingerprint) is applied all the same; a different one is shown instead, and Apply waits for the user.
*/
async function apply() {
if (applyButton.disabled) return;
@@ -321,6 +371,8 @@ export function openConvert({ api, target, names = new Map() } = {}) {
const going = shownPlan.steps.filter((s) => (s.blocked ?? []).length === 0 && s.photos > 0);
const list = listed(going.map((s) => t("Admin.Convert.PlanItem", names.get(s.sequenceId) || s.title || folderName(s.folder),
plural("Admin.Dashboard.PhotoCount", s.photos), t("Admin.Convert.SizeChange", formatBytes(s.bytesBefore), formatBytes(s.bytesAfter)))));
// The photos of the sequences converted that stay as they are: their new sequences hold the converted copies alone.
const notCopied = going.reduce((sum, s) => sum + (s.skipped ?? []).reduce((photos, k) => photos + k.photos, 0), 0);
const replace = body.mode === "replace";
const confirmed = await confirmDialog(replace ? {
title: t("Admin.Convert.ConfirmReplaceTitle", format),
@@ -334,7 +386,8 @@ export function openConvert({ api, target, names = new Map() } = {}) {
typedKind: "word",
} : {
title: t("Admin.Convert.ConfirmNewTitle", format),
body: plural("Admin.Convert.ConfirmNew", shownPlan.photos),
body: [plural("Admin.Convert.ConfirmNew", shownPlan.photos), notCopied > 0 ? plural("Admin.Convert.ConfirmNewNotCopied", notCopied) : ""]
.filter(Boolean).join(" "),
list,
confirm: t("Admin.Convert.Confirm.New"),
confirmTip: t("Admin.Convert.Confirm.New.Tip"),
@@ -360,11 +413,15 @@ export function openConvert({ api, target, names = new Map() } = {}) {
} catch (err) {
busy = false;
if (!dialog.open) return;
if (err?.code === "plan_changed" || err?.code === "not_enough_space" || err?.code === "nothing_to_convert") {
// The archive changed since the dry run (or its volume filled up): the new plan is shown, and Apply waits for it.
notice = err.code === "plan_changed" ? err.message || t("Admin.Error.Unreachable") : errorText(err);
if (["plan_changed", "plan_dropped", "not_enough_space", "nothing_to_convert"].includes(err?.code)) {
// The archive changed since the dry run (or its volume filled up, or the server let the plan go before it was
// confirmed): nothing was queued; the plan is made again and shown, and Apply waits for it.
notice = err.code === "plan_changed" ? t("Admin.Convert.PlanChanged") : errorText(err);
plan = null;
schedule(0);
} else if (cutOff(err, Infinity)) {
// No answer: the job may have been queued all the same, and Apply again would queue another one.
showError(t("Admin.Convert.NoAnswer"));
} else {
showProblem(err);
}
@@ -394,6 +451,14 @@ export function openConvert({ api, target, names = new Map() } = {}) {
});
}
/**
* Whether a request that failed after ms milliseconds was cut off on the way rather than answered: a proxy's timeout (504,
* or Cloudflare's 524), or a closed connection or a 502 once it had lasted CUT_AFTER (sooner, the server is not there).
*/
function cutOff(err, ms) {
return err?.status === 504 || err?.status === 524 || ((err?.status === 0 || err?.status === 502) && ms >= CUT_AFTER);
}
/** A format's name in the panel's language ("jpeg" → "JPEG", "linear-dng" → "Linear DNG"). */
function targetLabel(value) {
const known = TARGETS.find((o) => o.value === value);
+81 -24
View File
@@ -161,14 +161,14 @@ public static class ArchiveEndpoints
/// completes as it starts, no one: no client is connected yet). Photos a sequence receives between the dry run and the
/// confirmation under the names the plan leaves them keep the fingerprint as it was. A dry run's plan is made apart from
/// the request, a few at once and each for a limited time, and kept a few minutes, a refusal too, while what it reads
/// from the index stays as it was (<see cref="ReorganizePlanCache"/>; the sequences' rows, and how many photos of those
/// planned are hashed): the same dry run again (also after the first was cut off), and its confirmation, take it instead
/// of planning again. So <c>plan_changed</c> answers a change the index has, photos a verify hashed among them; one made
/// from the index stays as it was (<see cref="PlanCache"/>; the sequences' rows, and how many photos of those planned
/// are hashed): the same dry run again (also after the first was cut off), and its confirmation, take it instead of
/// planning again. So <c>plan_changed</c> answers a change the index has, photos a verify hashed among them; one made
/// on disk alone since the dry run (a folder made by hand, a desktop writing to a shared archive before a scan) is found
/// by the job, which then fails and changes nothing.
/// </summary>
private static async Task<IResult> ReorganizeAsync(HttpContext context, Caller caller, ReorganizeRequest request, ChronosDatabase db, ServerArchive archive,
[FromServices] ArchiveService? archives, JobEngineHost host, AuditLog audit, ReorganizePlanCache plans)
[FromServices] ArchiveService? archives, JobEngineHost host, AuditLog audit, PlanCache plans)
{
var ct = context.RequestAborted;
if (request.All == (request.SequenceIds is not null) || request.SequenceIds is { Count: 0 or > LibraryEndpoints.MaxBulkIds })
@@ -181,7 +181,7 @@ public static class ArchiveEndpoints
return ApiErrors.Result(context, StatusCodes.Status503ServiceUnavailable, "archive_offline");
var sequences = request.All ? null : request.SequenceIds;
// Stamped before planning: a change committed while the plan is made gives the next request another key.
var key = ReorganizePlanCache.Key.Of(archive.Id, await db.Sequences.GetStampAsync(archive.Id, sequences, ct), sequences, templates);
var key = ReorganizePlanKey.Of(archive.Id, await db.Sequences.GetStampAsync(archive.Id, sequences, ct), sequences, templates);
RenamePlan plan;
try
{
@@ -258,19 +258,28 @@ public static class ArchiveEndpoints
/// Convert: the photos of <c>sequenceIds</c> or, with <c>all</c>, of every sequence of the archive, converted to
/// <c>target</c> (<c>jpeg</c>, <c>tiff16</c>, <c>linear-dng</c>) at <c>quality</c> (JPEG, 1–100, default 95) in <c>mode</c>
/// (<c>replace</c> or <c>new-sequence</c>; BRIEF §12.5). With <c>dryRun</c> the plan, nothing touched: per sequence the
/// photos converted, their size and the estimate after, the photos left as they are with why (<c>convert_on_pc</c> for a
/// format the server cannot read at full resolution), what blocks it; the totals; the space needed and free; the
/// fingerprint. Otherwise the request carries that fingerprint and the server queues a <c>convert</c> job (202) that
/// applies exactly that plan, leaving as it is (and reporting) a sequence that changed before it ran. 400
/// <c>bad_request</c> (details <c>sequenceIds</c>: neither or both of them, none, or more than
/// <see cref="LibraryEndpoints.MaxBulkIds"/>; <c>target</c>, <c>quality</c>, <c>mode</c>, <c>fingerprint</c>); 404
/// <c>not_found</c> (<c>sequence</c>: not in the archive); 409 <c>plan_changed</c> (details: the fingerprint now),
/// <c>nothing_to_convert</c>, <c>not_enough_space</c> (details: the bytes needed), <c>job_unavailable</c>; 503
/// <c>archive_offline</c>. Audited (<c>archive_converted</c>: sequences · photos · target · mode); the job tells each sequence
/// it replaces the frames of as <c>sequenceChanged</c> <c>converted</c>, each new one as <c>created</c>.
/// photos converted, their size and the estimate after, the photos left as they are with why in the mode's words (with
/// <c>new-sequence</c>, that the new sequence lacks them; <c>convert_on_pc</c> for a format the server cannot read at
/// full resolution), what blocks it; the totals; the space needed and free; the fingerprint. Otherwise the request
/// carries that fingerprint and the server queues a <c>convert</c> job (202) that applies exactly that plan, leaving as it
/// is (and reporting) a sequence that changed before it ran. 400 <c>bad_request</c> (details <c>sequenceIds</c>: neither
/// or both of them, none, or more than <see cref="LibraryEndpoints.MaxBulkIds"/>; <c>target</c>, <c>quality</c>,
/// <c>mode</c>, <c>fingerprint</c>); 404 <c>not_found</c> (<c>sequence</c>: not in the archive); 409 <c>plan_changed</c>
/// (details: the fingerprint now), <c>nothing_to_convert</c>, <c>not_enough_space</c> (details: the bytes needed),
/// <c>job_unavailable</c>; 503 <c>archive_offline</c>, <c>plan_dropped</c> (the plan was stopped: past its time, let go
/// for newer ones, or the server stopping). Audited (<c>archive_converted</c>: sequences · photos · target · mode); the job
/// tells each sequence it replaces the frames of as <c>sequenceChanged</c> <c>converted</c>, each new one as
/// <c>created</c>. A dry run's plan is made apart from the request, a few at once (Reorganize's among them) and each for
/// a limited time, and kept a few minutes, a refusal too, while what it reads from the index stays as it was
/// (<see cref="PlanCache"/>, <see cref="ConvertPlanKey"/>: the sequences' rows, and of the photos of those planned their
/// formats, sizes, hashes and the flags that decide): the same dry run again (also after the first was cut off), in
/// either mode and at any quality of a lossless target, and its confirmation, take it instead of planning again; only
/// the free space is measured again at every request. So <c>plan_changed</c> answers a change the index has. One made on
/// disk alone since the plan, or a job that started or ended on its sequences (which a kept plan shows as it found
/// them, busy or not), is found by the job, which plans again and leaves as it is a sequence that no longer fits.
/// </summary>
private static async Task<IResult> ConvertAsync(HttpContext context, Caller caller, ConvertRequest request, ChronosDatabase db, ServerArchive archive,
[FromServices] ArchiveService? archives, JobEngineHost host, AuditLog audit)
[FromServices] ArchiveService? archives, JobEngineHost host, AuditLog audit, PlanCache plans)
{
var ct = context.RequestAborted;
if (request.All == (request.SequenceIds is not null) || request.SequenceIds is { Count: 0 or > LibraryEndpoints.MaxBulkIds })
@@ -282,16 +291,33 @@ public static class ArchiveEndpoints
if (!request.DryRun && string.IsNullOrWhiteSpace(request.Fingerprint)) return ApiErrors.BadRequest(context, "bad_request", "fingerprint");
if (ArchiveService.GetAvailability(archive.Current) != ArchiveAvailability.Online)
return ApiErrors.Result(context, StatusCodes.Status503ServiceUnavailable, "archive_offline");
var options = new ConvertOptions(target, quality);
var sequences = request.All ? null : request.SequenceIds;
var engine = LibraryEndpoints.Engine(archives, db);
// Stamped before planning: a change committed while the plan is made gives the next request another key.
var key = ConvertPlanKey.Of(archive.Id, await db.Sequences.GetConvertStampAsync(archive.Id, sequences, ct), sequences, options);
ConvertPlan plan;
try
{
plan = await LibraryEndpoints.Engine(archives, db).PlanConvertAsync(archive.Id, request.All ? null : request.SequenceIds,
new ConvertOptions(target, quality), mode, ct);
// Made whatever becomes of this request, until the cache stops it: the same request again takes it, in either mode.
var kept = await plans.GetOrPlanAsync(key, stop => PlanConvertAsync(engine, archive.Current, sequences, options, stop), ct);
// What converts is the same in either mode and at any quality of a lossless target: the request's own are set on
// it, and the free space is the volume's now.
plan = kept with { Options = options, Mode = mode, Space = kept.Space with { Free = await FreeSpaceAsync(engine, archive.Id, options, ct) } };
}
catch (ArgumentException)
{
return ApiErrors.NotFound(context, "sequence");
}
catch (ArchiveOfflineException)
{
return ApiErrors.Result(context, StatusCodes.Status503ServiceUnavailable, "archive_offline");
}
catch (OperationCanceledException) when (!ct.IsCancellationRequested)
{
// Stopped: past its time (a read that hangs), let go for newer plans, or the server stopping.
return ApiErrors.Result(context, StatusCodes.Status503ServiceUnavailable, "plan_dropped");
}
var fingerprint = plan.Fingerprint();
if (request.DryRun) return Results.Json(Plan(context, plan, fingerprint), ProtocolJson.Default.ConvertPlanDto);
if (!string.Equals(fingerprint, request.Fingerprint, StringComparison.Ordinal)) return ApiErrors.Conflict(context, "plan_changed", fingerprint);
@@ -299,12 +325,33 @@ public static class ArchiveEndpoints
if (!plan.Space.Enough) return ApiErrors.Conflict(context, "not_enough_space", plan.Space.Required.ToString(CultureInfo.InvariantCulture));
if (!host.CanRun(JobKind.Convert)) return ApiErrors.Conflict(context, "job_unavailable", JobWire.Kind(JobKind.Convert));
var job = await host.Engine.EnqueueAsync(ArchiveJobs.Convert(archive.Current, plan), caller.Id, ct);
plans.Forget(key);
await audit.WriteAsync(caller, context, "archive_converted", "job", job.Id.ToString(),
$"{plan.Converted.Count()} · {plan.Frames} · {Wire(target)} · {Wire(mode)}");
return Results.Json(Dto.Job(job), ProtocolJson.Default.JobDto, statusCode: StatusCodes.Status202Accepted);
}
/// <summary>The conversion dry run as the API answers it, each reason in the request's language.</summary>
/// <summary>
/// The conversion plan <see cref="PlanCache"/> keeps for a <see cref="ConvertPlanKey"/>, made in one mode for both (the
/// request's is set on it). One made while the archive's volume was away read nothing of it: it is not kept, and the
/// request is answered <c>archive_offline</c>.
/// </summary>
private static async Task<ConvertPlan> PlanConvertAsync(ArchiveService engine, Archive archive, IReadOnlyList<Guid>? sequences, ConvertOptions options,
CancellationToken cancellationToken)
{
var plan = await engine.PlanConvertAsync(archive.Id, sequences, options, ConvertMode.Replace, cancellationToken);
if (plan.Sequences.Any(s => s.Blocked.Contains(ConvertBlock.Offline))) throw new ArchiveOfflineException(archive.RootPath, ArchiveAvailability.Offline);
return plan;
}
/// <summary>
/// The free space of the archive's volume now, as the engine measures it for a conversion's plan and its job
/// (<see cref="ArchiveServiceOptions.FreeSpace"/>): the plan of no sequence reads no sidecar and no photo.
/// </summary>
private static async Task<long?> FreeSpaceAsync(ArchiveService engine, Guid archiveId, ConvertOptions options, CancellationToken cancellationToken) =>
(await engine.PlanConvertAsync(archiveId, [], options, ConvertMode.Replace, cancellationToken)).Space.Free;
/// <summary>The conversion dry run as the API answers it, each reason in the request's language and in its mode's words.</summary>
private static ConvertPlanDto Plan(HttpContext context, ConvertPlan plan, string fingerprint)
{
var culture = RequestCulture.Of(context);
@@ -314,7 +361,8 @@ public static class ArchiveEndpoints
: Loc.Text(space.Enough ? "Archive.Convert.Space" : "Archive.Convert.SpaceShort", [Size(space.Required, culture), Size(free, culture)], culture);
return new ConvertPlanDto(Wire(plan.Options.Target), plan.Options.Quality, Wire(plan.Mode),
[.. plan.Sequences.Select(s => new ConvertSequenceDto(s.SequenceId, s.Title, s.Folder, s.FramesVersion, s.Frames.Count, s.BytesBefore, s.BytesAfter,
[.. s.Skipped.GroupBy(k => Wire(k.Reason)).Select(g => new ConvertSkippedDto(g.Key, g.Sum(k => k.Frames), SkipMessage(g.First().Reason, culture)))],
[.. s.Skipped.GroupBy(k => Wire(k.Reason))
.Select(g => new ConvertSkippedDto(g.Key, g.Sum(k => k.Frames), SkipMessage(g.First().Reason, plan.Mode, culture)))],
[.. s.Blocked.Select(b => new ConvertBlockedDto(Wire(b), Loc.Get("Archive.Convert.Blocked." + b, culture) ?? Wire(b)))]))],
plan.Converted.Count(),
plan.Frames,
@@ -370,10 +418,19 @@ public static class ArchiveEndpoints
private static ConvertMode? ModeOf(string? text) =>
Enum.GetValues<ConvertMode>().Select(m => (ConvertMode?)m).FirstOrDefault(m => string.Equals(Wire(m!.Value), text?.Trim(), StringComparison.OrdinalIgnoreCase));
/// <summary>Why photos are not converted, in <paramref name="culture"/>: on the server, a format it cannot read says to convert on a PC.</summary>
private static string SkipMessage(ConvertSkip reason, CultureInfo culture) => reason is ConvertSkip.NoNativeDecoder or ConvertSkip.PreviewOnly
? Loc.Get("Server.Convert.OnPc", culture) ?? Wire(reason)
: Loc.Get("Archive.Convert.Skipped." + reason, culture) ?? Wire(reason);
/// <summary>
/// Why photos are not converted, in <paramref name="culture"/>: on the server, a format it cannot read says to convert on
/// a PC. With <see cref="ConvertMode.NewSequence"/> it says that the new sequence lacks them (it holds the converted
/// copies alone); otherwise that they stay as they are.
/// </summary>
private static string SkipMessage(ConvertSkip reason, ConvertMode mode, CultureInfo culture)
{
var onPc = reason is ConvertSkip.NoNativeDecoder or ConvertSkip.PreviewOnly;
var key = mode == ConvertMode.NewSequence
? onPc ? "Server.Convert.OnPcNew" : "Archive.Convert.NotCopied." + reason
: onPc ? "Server.Convert.OnPc" : "Archive.Convert.Skipped." + reason;
return Loc.Get(key, culture) ?? Wire(reason);
}
/// <summary>A size as the dry run's space line shows it: GB from one gigabyte, MB below.</summary>
private static string Size(long bytes, CultureInfo culture) => bytes >= 1_000_000_000
+267
View File
@@ -0,0 +1,267 @@
using Chronos.Core.Archiving;
using Chronos.Core.Conversion;
using Chronos.Data.Repositories;
namespace Chronos.Server.Api;
/// <summary>
/// The dry runs of Reorganize and Convert kept a short while (owner review 2, R7): the same dry run asked again, and the
/// confirmation that follows it, take the plan made then instead of reading every sidecar and photo folder of the selection
/// again (with <c>all</c>, of the whole archive), while what the plan reads from the index is as it was then (the stamp its
/// key holds: <see cref="SequenceRepository.GetStampAsync"/>, <see cref="SequenceRepository.GetConvertStampAsync"/>) and for
/// at most <see cref="Lifetime"/> once made. A refusal (a folder inside a sequence's, a sequence not in the archive) follows
/// from the same reads, so it is kept as a plan is (<see cref="PlanKey.Refuses"/>); a failure is not. A plan is made apart
/// from the request that asked for it, once for every request of the same key: a request cut off meanwhile (a proxy's
/// timeout) leaves it to be made, and the same request again waits for it or takes it. At most <see cref="Concurrency"/>
/// plans are made at once, of either kind; one still being made past <see cref="PlanTimeout"/>, let go for newer ones
/// (<see cref="Capacity"/>), or when the server stops, is stopped, and whoever waits for it is told it was dropped
/// (<see cref="OperationCanceledException"/>). A change made on disk alone, which the index does not have, is not seen by a
/// kept plan, and neither is one of the jobs running: the job that applies a plan plans again and leaves as it is what no
/// longer matches it, so nothing changes that the dry run did not show.
/// </summary>
public sealed class PlanCache : IDisposable
{
/// <summary>How long a plan is kept once made; then its timer lets it go.</summary>
public static readonly TimeSpan Lifetime = TimeSpan.FromMinutes(5);
/// <summary>
/// Plans kept or being made at most, of either kind (each holds a step for every sequence it plans). One more lets go
/// the kept plan asked for first; with none kept, the plan being made asked for first, which stops.
/// </summary>
public const int Capacity = 8;
/// <summary>Plans made at once at most, of either kind; the others wait for their turn.</summary>
public const int Concurrency = 2;
/// <summary>
/// How long a plan may take, from when it was asked (its wait for a turn included): then it stops. A read that hangs
/// still holds its turn until it returns.
/// </summary>
public static readonly TimeSpan PlanTimeout = TimeSpan.FromMinutes(15);
private readonly TimeProvider _time;
private readonly Lock _gate = new();
private readonly Dictionary<PlanKey, Entry> _plans = [];
private readonly SemaphoreSlim _turns = new(Concurrency);
private readonly CancellationTokenRegistration _stopping;
private long _order;
/// <param name="lifetime">The host, whose stopping stops the plans being made.</param>
public PlanCache(TimeProvider time, IHostApplicationLifetime? lifetime = null)
{
_time = time;
if (lifetime is not null) _stopping = lifetime.ApplicationStopping.Register(Clear);
}
/// <summary>Plans kept or being made now.</summary>
internal int Count
{
get
{
lock (_gate) return _plans.Count;
}
}
/// <summary>
/// The plan kept for <paramref name="key"/>, the one being made for it, or the one <paramref name="plan"/> starts making
/// now, apart from the caller, then kept; <paramref name="plan"/> stops when its token is cancelled. A refusal
/// (<see cref="PlanKey.Refuses"/>) is kept as a plan is; a failure is passed to every caller waiting for it and not kept,
/// and so is a plan stopped (<see cref="OperationCanceledException"/>). <paramref name="cancellationToken"/> stops the
/// wait only: the plan is still made, and kept.
/// </summary>
public async Task<TPlan> GetOrPlanAsync<TPlan>(PlanKey<TPlan> key, Func<CancellationToken, Task<TPlan>> plan, CancellationToken cancellationToken = default)
where TPlan : class
{
ArgumentNullException.ThrowIfNull(key);
ArgumentNullException.ThrowIfNull(plan);
Entry? entry;
bool start;
lock (_gate)
{
// Never past its lifetime, whatever its timer.
if (_plans.TryGetValue(key, out entry) && entry.Made is { } made && _time.GetUtcNow() - made >= Lifetime) Remove(key);
start = !_plans.TryGetValue(key, out entry);
if (start)
{
if (_plans.Count >= Capacity) Remove(_plans.MinBy(p => (p.Value.Made is null, p.Value.Order)).Key);
var asked = new Entry(++_order);
asked.Deadline = _time.CreateTimer(_ => TimeOut(key, asked), null, PlanTimeout, Timeout.InfiniteTimeSpan);
_plans[key] = entry = asked;
}
}
if (start) _ = MakeAsync(key, entry!, async stop => await plan(stop).ConfigureAwait(false));
// The key's kind names the plan's: what was made for it is a TPlan.
return (TPlan)await entry!.Plan.Task.WaitAsync(cancellationToken).ConfigureAwait(false);
}
/// <summary>Forgets the plan kept for <paramref name="key"/>: its job is queued, and what the job changes gives the index another stamp.</summary>
public void Forget(PlanKey key)
{
lock (_gate) Remove(key);
}
public void Dispose()
{
_stopping.Dispose();
Clear();
}
/// <summary>Lets every plan go; those being made stop.</summary>
private void Clear()
{
lock (_gate)
{
foreach (var key in _plans.Keys.ToList()) Remove(key);
}
}
private async Task MakeAsync(PlanKey key, Entry entry, Func<CancellationToken, Task<object>> plan)
{
var stop = entry.Stop.Token;
object? made = null;
Exception? refused = null;
try
{
await _turns.WaitAsync(stop).ConfigureAwait(false);
var making = Task.Run(() => plan(stop), CancellationToken.None);
// The turn ends with the work, also when no one waits for it any more (a read that hangs).
_ = making.ContinueWith(t =>
{
_ = t.Exception;
_turns.Release();
}, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
made = await making.WaitAsync(stop).ConfigureAwait(false);
}
catch (Exception ex) when (ex is not OperationCanceledException && key.Refuses(ex))
{
// A folder inside a sequence's, a sequence not in the archive: what the key holds decides it, as it does a plan.
refused = ex;
}
catch (Exception ex)
{
// A failure, or the plan stopped (past its time, let go, the server stopping): the next request plans again.
lock (_gate)
{
if (_plans.GetValueOrDefault(key) == entry) Remove(key);
}
if (ex is OperationCanceledException canceled) entry.Plan.SetCanceled(canceled.CancellationToken);
else entry.Plan.SetException(ex);
// Observed here: a request cut off before it ended no longer waits for it.
_ = entry.Plan.Task.Exception;
return;
}
lock (_gate)
{
if (_plans.GetValueOrDefault(key) == entry)
{
entry.Made = _time.GetUtcNow();
entry.Deadline?.Dispose();
entry.Expiry = _time.CreateTimer(_ => Expire(key, entry), null, Lifetime, Timeout.InfiniteTimeSpan);
}
}
if (refused is null)
{
entry.Plan.SetResult(made!);
return;
}
entry.Plan.SetException(refused);
_ = entry.Plan.Task.Exception;
}
/// <summary>A plan still being made at <see cref="PlanTimeout"/> stops.</summary>
private void TimeOut(PlanKey key, Entry entry)
{
lock (_gate)
{
// Made as its deadline fired: kept for its lifetime.
if (_plans.GetValueOrDefault(key) == entry && entry.Made is null) Remove(key);
}
}
private void Expire(PlanKey key, Entry entry)
{
lock (_gate)
{
if (_plans.GetValueOrDefault(key) == entry) Remove(key);
}
}
private void Remove(PlanKey key)
{
if (!_plans.Remove(key, out var entry)) return;
entry.Deadline?.Dispose();
entry.Expiry?.Dispose();
// Stopped outside this lock: the wait for it ends on the thread pool.
if (entry.Made is null) _ = entry.Stop.CancelAsync();
}
/// <summary>A plan being made (<see cref="Made"/> null) or kept, and the order it was asked in.</summary>
private sealed class Entry(long order)
{
public long Order { get; } = order;
public TaskCompletionSource<object> Plan { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously);
/// <summary>Stops the plan while it is made; a source with no timer and no parent, so nothing to release.</summary>
public CancellationTokenSource Stop { get; } = new();
public DateTimeOffset? Made { get; set; }
/// <summary>Stops the plan at <see cref="PlanTimeout"/> while it is made.</summary>
public ITimer? Deadline { get; set; }
/// <summary>Lets the plan go at <see cref="Lifetime"/> once made.</summary>
public ITimer? Expiry { get; set; }
}
}
/// <summary>What a kept plan is kept by (<see cref="PlanCache"/>): everything it is made from, by value, its kind included.</summary>
public abstract record PlanKey
{
/// <summary>
/// Whether <paramref name="exception"/>, thrown while the plan is made, refuses the request for what the key holds (kept
/// as a plan is: the same reads refuse it again) rather than failing it (not kept).
/// </summary>
public abstract bool Refuses(Exception exception);
/// <summary>The sequences of a request as a key holds them: <c>*</c> for every sequence, else the ids in the order given.</summary>
protected static string Selection(IReadOnlyList<Guid>? sequenceIds) =>
sequenceIds is null ? "*" : string.Join(',', sequenceIds.Select(id => id.ToString("N")));
}
/// <summary>The key of a plan of <typeparamref name="TPlan"/>: what <see cref="PlanCache.GetOrPlanAsync"/> answers for it.</summary>
public abstract record PlanKey<TPlan> : PlanKey where TPlan : class;
/// <summary>
/// What a Reorganize plan is kept by: the archive, the stamp of what it reads from the index
/// (<see cref="SequenceRepository.GetStampAsync"/>), the sequences (all of them, or the ids in the order given, which decides
/// the folder that gets " (2)") and the templates. A folder inside a sequence's (<see cref="NestedRenameException"/>) or a
/// sequence not in the archive (<see cref="ArgumentException"/>) refuses it.
/// </summary>
public sealed record ReorganizePlanKey(Guid ArchiveId, string Stamp, string Sequences, RenameTemplates Templates) : PlanKey<RenamePlan>
{
/// <param name="sequenceIds">The sequences of the request; null for every sequence of the archive.</param>
public static ReorganizePlanKey Of(Guid archiveId, string stamp, IReadOnlyList<Guid>? sequenceIds, RenameTemplates templates) =>
new(archiveId, stamp, Selection(sequenceIds), templates);
public override bool Refuses(Exception exception) => exception is NestedRenameException || exception.GetType() == typeof(ArgumentException);
}
/// <summary>
/// What a conversion plan is kept by: the archive, the stamp of what it reads from the index
/// (<see cref="SequenceRepository.GetConvertStampAsync"/>), the sequences (all of them, or the ids in the order given) and
/// the target with the quality that counts (<see cref="ConvertOptions.Key"/>: a lossless target's ignores it). Not the
/// mode: what converts is the same in either, so the plan made with one serves the other (the endpoint sets the request's).
/// A sequence not in the archive (<see cref="ArgumentException"/>) refuses it.
/// </summary>
public sealed record ConvertPlanKey(Guid ArchiveId, string Stamp, string Sequences, string Options) : PlanKey<ConvertPlan>
{
/// <param name="sequenceIds">The sequences of the request; null for every sequence of the archive.</param>
public static ConvertPlanKey Of(Guid archiveId, string stamp, IReadOnlyList<Guid>? sequenceIds, ConvertOptions options)
{
ArgumentNullException.ThrowIfNull(options);
return new(archiveId, stamp, Selection(sequenceIds), options.Key);
}
public override bool Refuses(Exception exception) => exception.GetType() == typeof(ArgumentException);
}
@@ -1,223 +0,0 @@
using Chronos.Core.Archiving;
using Chronos.Data.Repositories;
namespace Chronos.Server.Api;
/// <summary>
/// Reorganize's dry runs kept a short while (owner review 2, R7): the same dry run asked again, and the confirmation that
/// follows it, take the plan made then instead of reading every sidecar and photo folder of the selection again (with
/// <c>all</c>, of the whole archive), while what the plan reads from the index is as it was then
/// (<see cref="SequenceRepository.GetStampAsync"/>) and for at most <see cref="Lifetime"/> once made. A refusal (a folder
/// inside a sequence's, a sequence not in the archive) follows from the same reads, so it is kept as a plan is; a failure
/// is not. A plan is made apart from the request that asked for it, once for every request of the same key: a request cut
/// off meanwhile (a proxy's timeout) leaves it to be made, and the same request again waits for it or takes it. At most
/// <see cref="Concurrency"/> plans are made at once; one still being made past <see cref="PlanTimeout"/>, let go for newer
/// ones (<see cref="Capacity"/>), or when the server stops, is stopped, and whoever waits for it is told it was dropped
/// (<see cref="OperationCanceledException"/>). A change made on disk alone, which the index does not have, is not seen by a
/// kept plan: the <c>move</c> job plans again and refuses a plan that no longer matches the archive, so nothing changes
/// that the dry run did not show.
/// </summary>
public sealed class ReorganizePlanCache : IDisposable
{
/// <summary>How long a plan is kept once made; then its timer lets it go.</summary>
public static readonly TimeSpan Lifetime = TimeSpan.FromMinutes(5);
/// <summary>
/// Plans kept or being made at most (each holds a step for every sequence it changes or carries, or whose photos keep
/// their names). One more lets go the kept plan asked for first; with none kept, the plan being made asked for first,
/// which stops.
/// </summary>
public const int Capacity = 8;
/// <summary>Plans made at once at most; the others wait for their turn.</summary>
public const int Concurrency = 2;
/// <summary>
/// How long a plan may take, from when it was asked (its wait for a turn included): then it stops. A read that hangs
/// still holds its turn until it returns.
/// </summary>
public static readonly TimeSpan PlanTimeout = TimeSpan.FromMinutes(15);
private readonly TimeProvider _time;
private readonly Lock _gate = new();
private readonly Dictionary<Key, Entry> _plans = [];
private readonly SemaphoreSlim _turns = new(Concurrency);
private readonly CancellationTokenRegistration _stopping;
private long _order;
/// <param name="lifetime">The host, whose stopping stops the plans being made.</param>
public ReorganizePlanCache(TimeProvider time, IHostApplicationLifetime? lifetime = null)
{
_time = time;
if (lifetime is not null) _stopping = lifetime.ApplicationStopping.Register(Clear);
}
/// <summary>
/// What a plan is kept by: the archive, the stamp of what it reads from the index, the sequences (all of them, or the ids
/// in the order given, which decides the folder that gets " (2)") and the templates.
/// </summary>
public readonly record struct Key(Guid ArchiveId, string Stamp, string Sequences, RenameTemplates Templates)
{
/// <param name="sequenceIds">The sequences of the request; null for every sequence of the archive.</param>
public static Key Of(Guid archiveId, string stamp, IReadOnlyList<Guid>? sequenceIds, RenameTemplates templates) =>
new(archiveId, stamp, sequenceIds is null ? "*" : string.Join(',', sequenceIds.Select(id => id.ToString("N"))), templates);
}
/// <summary>Plans kept or being made now.</summary>
internal int Count
{
get
{
lock (_gate) return _plans.Count;
}
}
/// <summary>
/// The plan kept for <paramref name="key"/>, the one being made for it, or the one <paramref name="plan"/> starts making
/// now, apart from the caller, then kept; <paramref name="plan"/> stops when its token is cancelled. A refusal
/// (<see cref="NestedRenameException"/>, <see cref="ArgumentException"/>) is kept as a plan is; a failure is passed to
/// every caller waiting for it and not kept, and so is a plan stopped (<see cref="OperationCanceledException"/>).
/// <paramref name="cancellationToken"/> stops the wait only: the plan is still made, and kept.
/// </summary>
public Task<RenamePlan> GetOrPlanAsync(Key key, Func<CancellationToken, Task<RenamePlan>> plan, CancellationToken cancellationToken = default)
{
ArgumentNullException.ThrowIfNull(plan);
Entry? entry;
bool start;
lock (_gate)
{
// Never past its lifetime, whatever its timer.
if (_plans.TryGetValue(key, out entry) && entry.Made is { } made && _time.GetUtcNow() - made >= Lifetime) Remove(key);
start = !_plans.TryGetValue(key, out entry);
if (start)
{
if (_plans.Count >= Capacity) Remove(_plans.MinBy(p => (p.Value.Made is null, p.Value.Order)).Key);
var asked = new Entry(++_order);
asked.Deadline = _time.CreateTimer(_ => TimeOut(key, asked), null, PlanTimeout, Timeout.InfiniteTimeSpan);
_plans[key] = entry = asked;
}
}
if (start) _ = MakeAsync(key, entry!, plan);
return entry!.Plan.Task.WaitAsync(cancellationToken);
}
/// <summary>Forgets the plan kept for <paramref name="key"/>: its job is queued, and what the job changes gives the index another stamp.</summary>
public void Forget(Key key)
{
lock (_gate) Remove(key);
}
public void Dispose()
{
_stopping.Dispose();
Clear();
}
/// <summary>Lets every plan go; those being made stop.</summary>
private void Clear()
{
lock (_gate)
{
foreach (var key in _plans.Keys.ToList()) Remove(key);
}
}
private async Task MakeAsync(Key key, Entry entry, Func<CancellationToken, Task<RenamePlan>> plan)
{
var stop = entry.Stop.Token;
RenamePlan? made = null;
Exception? refused = null;
try
{
await _turns.WaitAsync(stop).ConfigureAwait(false);
var making = Task.Run(() => plan(stop), CancellationToken.None);
// The turn ends with the work, also when no one waits for it any more (a read that hangs).
_ = making.ContinueWith(t =>
{
_ = t.Exception;
_turns.Release();
}, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
made = await making.WaitAsync(stop).ConfigureAwait(false);
}
catch (Exception ex) when (ex is NestedRenameException || ex.GetType() == typeof(ArgumentException))
{
// A folder inside a sequence's, a sequence not in the archive: what the key holds decides it, as it does a plan.
refused = ex;
}
catch (Exception ex)
{
// A failure, or the plan stopped (past its time, let go, the server stopping): the next request plans again.
lock (_gate)
{
if (_plans.GetValueOrDefault(key) == entry) Remove(key);
}
if (ex is OperationCanceledException canceled) entry.Plan.SetCanceled(canceled.CancellationToken);
else entry.Plan.SetException(ex);
// Observed here: a request cut off before it ended no longer waits for it.
_ = entry.Plan.Task.Exception;
return;
}
lock (_gate)
{
if (_plans.GetValueOrDefault(key) == entry)
{
entry.Made = _time.GetUtcNow();
entry.Deadline?.Dispose();
entry.Expiry = _time.CreateTimer(_ => Expire(key, entry), null, Lifetime, Timeout.InfiniteTimeSpan);
}
}
if (refused is null)
{
entry.Plan.SetResult(made!);
return;
}
entry.Plan.SetException(refused);
_ = entry.Plan.Task.Exception;
}
/// <summary>A plan still being made at <see cref="PlanTimeout"/> stops.</summary>
private void TimeOut(Key key, Entry entry)
{
lock (_gate)
{
// Made as its deadline fired: kept for its lifetime.
if (_plans.GetValueOrDefault(key) == entry && entry.Made is null) Remove(key);
}
}
private void Expire(Key key, Entry entry)
{
lock (_gate)
{
if (_plans.GetValueOrDefault(key) == entry) Remove(key);
}
}
private void Remove(Key key)
{
if (!_plans.Remove(key, out var entry)) return;
entry.Deadline?.Dispose();
entry.Expiry?.Dispose();
// Stopped outside this lock: the wait for it ends on the thread pool.
if (entry.Made is null) _ = entry.Stop.CancelAsync();
}
/// <summary>A plan being made (<see cref="Made"/> null) or kept, and the order it was asked in.</summary>
private sealed class Entry(long order)
{
public long Order { get; } = order;
public TaskCompletionSource<RenamePlan> Plan { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously);
/// <summary>Stops the plan while it is made; a source with no timer and no parent, so nothing to release.</summary>
public CancellationTokenSource Stop { get; } = new();
public DateTimeOffset? Made { get; set; }
/// <summary>Stops the plan at <see cref="PlanTimeout"/> while it is made.</summary>
public ITimer? Deadline { get; set; }
/// <summary>Lets the plan go at <see cref="Lifetime"/> once made.</summary>
public ITimer? Expiry { get; set; }
}
}
+1 -1
View File
@@ -142,7 +142,7 @@ public static class ServerHost
services.AddSingleton<RenderStore>();
services.AddSingleton<RenderUploads>();
services.AddSingleton<SequenceJobs>();
services.AddSingleton<ReorganizePlanCache>();
services.AddSingleton<PlanCache>();
services.AddSingleton<RenderNodeService>();
services.AddSingleton<JobEngineHost>();
services.AddHostedService(sp => sp.GetRequiredService<JobEngineHost>());
@@ -70,6 +70,18 @@ public sealed class ConvertDialogTests(TestContext context)
Loc.Plural("Archive.Convert.SkippedPhotos", 3)), ConvertPlans.Summary(plan));
Assert.Equal(Loc.Plural("Archive.Rename.Summary", 0, 1), ConvertPlans.Summary(plan with { Sequences = [comet] }), "nothing converts, nothing more is said");
// Into a new sequence, which holds the converted copies alone: the photos skipped are not copied into it.
var copies = plan with { Mode = ConvertMode.NewSequence };
Assert.SequenceEqual(
[
Loc.L("Archive.Convert.NotCopied.SameFormat") + Loc.Separator + Loc.Plural("Shell.Status.Photos", 2),
Loc.L("Archive.Convert.NotCopied.Unhashed") + Loc.Separator + Loc.Plural("Shell.Status.Photos", 1),
],
ConvertPlans.Rows(copies)[0].Kept);
Assert.SequenceEqual(busy.Kept, ConvertPlans.Rows(copies)[1].Kept, "what blocks a sequence reads the same");
Assert.Equal(string.Join(Loc.Separator, Loc.Plural("Archive.Rename.Summary", 2, 4), Loc.Plural("Shell.Status.Photos", 4), ConvertPlans.Sizes(3_500_000, 6_700_000),
Loc.Plural("Archive.Convert.NotCopiedPhotos", 3)), ConvertPlans.Summary(copies));
// The space line: enough, short, or not measurable.
var required = Formatting.Bytes(plan.Space.Required);
Assert.Equal(new ConvertSpaceLine(Loc.L("Archive.Convert.Space", required, Formatting.Bytes(40_000_000_000)), false), ConvertPlans.Space(plan.Space));
@@ -145,11 +157,17 @@ public sealed class ConvertDialogTests(TestContext context)
Assert.Equal(Loc.L("Tooltips.Archive.ConvertApplyNew"), dialog.ApplyToolTip);
Assert.True(dialog.ApplyCommand.CanExecute(null));
// JPEG to JPEG converts nothing; a quality outside 1-100 blocks a JPEG only.
// JPEG to JPEG converts nothing: into a new sequence those photos are not copied, which the rows and the summary
// say, and Replace's words are that they stay as they are. A quality outside 1-100 blocks a JPEG only.
dialog.IsJpeg = true;
Assert.Equal(Loc.L("Archive.Blocked.Planning"), dialog.ApplyToolTip, "the dry run on screen is not the one of the format chosen");
await dialog.PlanAsync();
Assert.SequenceEqual([Loc.L("Archive.Convert.NotCopied.SameFormat") + Loc.Separator + Loc.Plural("Shell.Status.Photos", 2)], dialog.Rows[0].Kept);
Assert.Equal(Loc.Plural("Archive.Convert.NotCopiedPhotos", 4), dialog.SummaryText.Split(Loc.Separator)[^1]);
dialog.IsReplace = true;
Assert.SequenceEqual([Loc.L("Archive.Convert.Skipped.SameFormat") + Loc.Separator + Loc.Plural("Shell.Status.Photos", 2)], dialog.Rows[0].Kept);
Assert.Equal(Loc.Plural("Archive.Convert.SkippedPhotos", 4), dialog.SummaryText.Split(Loc.Separator)[^1]);
dialog.IsNewSequence = true;
Assert.Equal(Loc.L("Archive.Blocked.NothingToConvert"), dialog.ApplyToolTip);
// A click on Apply takes the focus from the quality first, which commits it: the dry run on screen is of that
// quality already, so no other one starts under the click.
@@ -1,3 +1,4 @@
using Chronos.Core.Imaging;
using Chronos.Core.Model;
using Chronos.Data.Repositories;
using Chronos.Data.Tests.Infrastructure;
@@ -6,8 +7,9 @@ using Chronos.Testing;
namespace Chronos.Data.Tests.Repositories;
/// <summary>
/// The stamp of what a Reorganize plans from in an archive's index (<see cref="SequenceRepository.GetStampAsync"/>) on both
/// dialects: the server makes a kept Reorganize plan again once it is another one, and only then (owner review 2, R7).
/// The stamps of what a Reorganize and a conversion plan from in an archive's index (<see cref="SequenceRepository.GetStampAsync"/>,
/// <see cref="SequenceRepository.GetConvertStampAsync"/>) on both dialects: the server makes a kept plan again once its
/// stamp is another one, and only then (owner review 2, R7).
/// </summary>
public abstract class SequenceStampTests(TestContext context) : RepositoryTest(context)
{
@@ -77,6 +79,53 @@ public abstract class SequenceStampTests(TestContext context) : RepositoryTest(c
Assert.False(verified == all || verified == uploaded, "a photo hashed");
}
[Test]
public async Task TheConvertStampFollowsThePhotosAConversionPlansFrom()
{
var db = await OpenChronosAsync();
var archive = TestData.Archive();
await db.Archives.UpsertAsync(archive, Ct);
var roof = TestData.Sequence(archive.Id, new DateTime(2026, 9, 28, 18, 0, 0), "Roof");
var ridge = TestData.Sequence(archive.Id, new DateTime(2026, 9, 29, 7, 0, 0), "Ridge");
foreach (var sequence in new[] { roof, ridge }) await db.Sequences.UpsertAsync(sequence, Ct);
var photo = await PhotoAsync(db, roof, 0);
var elsewhere = await PhotoAsync(db, ridge, 0);
IReadOnlyList<Guid> planned = [roof.Id];
var stamp = await db.Sequences.GetConvertStampAsync(archive.Id, planned, Ct);
Assert.Equal(stamp, await db.Sequences.GetConvertStampAsync(archive.Id, planned, Ct));
Assert.Equal(stamp, await db.Sequences.GetConvertStampAsync(archive.Id, [roof.Id, roof.Id], Ct));
Assert.NotEqual(stamp, await db.Sequences.GetConvertStampAsync(archive.Id, null, Ct));
// What a conversion does not read: the photos of a sequence not planned, light statistics, tags, a verify time.
Assert.Equal(1, await db.Frames.UpdateFlagsAsync([elsewhere.Id], FrameFlags.Missing, FrameFlags.None, Ct));
Assert.Equal(1, await db.Frames.UpdateLightStatsAsync([new FrameLightStats(photo.Id, -2.5, 0.01, 0.02)], Ct));
Assert.True(await db.Sequences.SetTagsAsync(roof.Id, ["sky"], Ct));
Assert.True(await db.Sequences.RecordVerifiedAsync(roof.Id, SequenceStatus.Verified, SequenceStatus.None, TestData.Now, Ct));
Assert.Equal(stamp, await db.Sequences.GetConvertStampAsync(archive.Id, planned, Ct));
// What it reads, without another frames version: each flag that keeps a photo as it is (a scan's, a verify's, a
// decode's), the format a decode records, the hash a verify records. Cleared again, the stamp is the one before.
var seen = new HashSet<string>(StringComparer.Ordinal) { stamp };
foreach (var flag in new[] { FrameFlags.Missing, FrameFlags.Corrupt, FrameFlags.PreviewOnly, FrameFlags.Undecodable })
{
Assert.Equal(1, await db.Frames.UpdateFlagsAsync([photo.Id], flag, FrameFlags.None, Ct));
Assert.True(seen.Add(await db.Sequences.GetConvertStampAsync(archive.Id, planned, Ct)), flag.ToString());
Assert.Equal(1, await db.Frames.UpdateFlagsAsync([photo.Id], FrameFlags.None, flag, Ct));
Assert.Equal(stamp, await db.Sequences.GetConvertStampAsync(archive.Id, planned, Ct));
}
Assert.Equal(1, await db.Frames.UpdateTiersAsync([new FrameTierFields(photo.Id, FrameFlags.None, 0, 0, ImageFileFormat.Dng)], Ct));
Assert.True(seen.Add(await db.Sequences.GetConvertStampAsync(archive.Id, planned, Ct)), "format");
var unhashed = await PhotoAsync(db, roof, 1, hashed: false);
Assert.True(seen.Add(await db.Sequences.GetConvertStampAsync(archive.Id, planned, Ct)), "a photo added");
await VerifyAsync(db, unhashed);
Assert.True(seen.Add(await db.Sequences.GetConvertStampAsync(archive.Id, planned, Ct)), "hashed by a verify");
// With every sequence planned, a flag on any photo changes it.
var all = await db.Sequences.GetConvertStampAsync(archive.Id, null, Ct);
Assert.Equal(1, await db.Frames.UpdateFlagsAsync([elsewhere.Id], FrameFlags.None, FrameFlags.Missing, Ct));
Assert.NotEqual(all, await db.Sequences.GetConvertStampAsync(archive.Id, null, Ct));
}
/// <summary>
/// A photo indexed into a sequence, as an upload does, or without its hash, as a scan does: the frame, then the row's
/// counts and frames version.
+123 -5
View File
@@ -1,10 +1,13 @@
using System.Globalization;
using System.Net;
using System.Net.Http.Json;
using Microsoft.Extensions.DependencyInjection;
using Chronos.Core.Codecs.Jpeg;
using Chronos.Core.Imaging;
using Chronos.Core.Localization;
using Chronos.Core.Model;
using Chronos.Protocol;
using Chronos.Server.Api;
using Chronos.Server.Events;
using Chronos.Server.Storage;
using Chronos.Testing;
@@ -12,8 +15,9 @@ using Chronos.Testing;
namespace Chronos.Server.Tests.Api;
/// <summary>
/// Mass conversion on the server (owner review 2, R7): <c>POST /api/v1/archive/convert</c>, its dry run, the <c>convert</c> job
/// that applies the confirmed plan, what the request may not do, and the formats the server leaves to a PC.
/// Mass conversion on the server (owner review 2, R7): <c>POST /api/v1/archive/convert</c>, its dry run and the plan kept from
/// it in either mode (<see cref="PlanCache"/>) with the free space measured at every request, the <c>convert</c> job that
/// applies the confirmed plan, what the request may not do, and the formats the server leaves to a PC.
/// </summary>
[Category("server")]
public sealed class ConvertTests(TestContext context)
@@ -174,12 +178,126 @@ public sealed class ConvertTests(TestContext context)
await UntilIdleAsync(server);
admin.DefaultRequestHeaders.AcceptLanguage.ParseAdd("it");
using var dry = await admin.PostAsJsonAsync(Url, new ConvertRequest([id], false, "linear-dng", null, "new-sequence", DryRun: true), ProtocolJson.Default.ConvertRequest);
var plan = await dry.ReadAsync(ProtocolJson.Default.ConvertPlanDto);
var italian = CultureInfo.GetCultureInfo("it");
var plan = await DryRunAsync(admin, new ConvertRequest([id], false, "linear-dng", null, "replace", DryRun: true));
Assert.Equal((1, 2), (plan.Photos, plan.Skipped));
var skipped = plan.Steps.Single().Skipped.Single();
Assert.Equal(("convert_on_pc", 2), (skipped.Reason, skipped.Photos));
Assert.Equal(Chronos.Core.Localization.Loc.Get("Server.Convert.OnPc", System.Globalization.CultureInfo.GetCultureInfo("it")), skipped.Message);
Assert.Equal(Loc.Get("Server.Convert.OnPc", italian), skipped.Message);
// Into a new sequence, which holds the converted copies alone: the dry run says the copy lacks them.
var copies = await DryRunAsync(admin, new ConvertRequest([id], false, "linear-dng", null, "new-sequence", DryRun: true));
Assert.Equal((1, 2), (copies.Photos, copies.Skipped));
var left = copies.Steps.Single().Skipped.Single();
Assert.Equal(("convert_on_pc", 2, Loc.Get("Server.Convert.OnPcNew", italian)), (left.Reason, left.Photos, left.Message));
}
[Test]
public async Task ADryRunIsKeptInEitherModeWhileTheIndexStaysAsItWas()
{
await using var server = await TestServer.StartAsync(context, services: ArchiveModule.Register);
var (editor, _, _) = await server.UserAsync("alberto", "editor");
var admin = await server.AdminAsync();
admin.DefaultRequestHeaders.AcceptLanguage.ParseAdd("it");
var italian = CultureInfo.GetCultureInfo("it");
var plans = server.Services.GetRequiredService<PlanCache>();
Guid id = default;
for (var i = 0; i < 3; i++) id = (await Uploads.UploadAsync(editor, Photo(i), $"IMG_{i + 1:0000}.JPG", sessionKey: "roof")).SequenceId;
await UntilIdleAsync(server);
var request = new ConvertRequest([id], false, "tiff16", null, "replace", DryRun: true);
var shown = await DryRunAsync(admin, request);
Assert.Equal(("replace", 3, 0), (shown.Mode, shown.Photos, shown.Skipped));
Assert.Equal(1, plans.Count);
// A photo gone on disk alone is not in the index: the same dry run again, in the other mode or at another quality of
// a lossless target, is the plan made before, in the request's mode and quality.
var first = (await server.Database.Frames.ListBySequenceAsync(id))[0];
var file = Path.Combine(server.Options.ArchivePath, first.RelativePath);
var bytes = await File.ReadAllBytesAsync(file);
File.Delete(file);
var copies = await DryRunAsync(admin, request with { Mode = "new-sequence" });
Assert.Equal(("new-sequence", 3, 0), (copies.Mode, copies.Photos, copies.Skipped));
Assert.NotEqual(shown.Fingerprint, copies.Fingerprint);
var quality = await DryRunAsync(admin, request with { Quality = 50 });
Assert.Equal((50, 3, shown.Fingerprint), (quality.Quality, quality.Photos, quality.Fingerprint));
Assert.Equal(1, plans.Count);
// A change of what it reads plans again at once: the photo is found missing, and each mode says what becomes of it.
await Uploads.UploadAsync(editor, Photo(3), "IMG_0004.JPG", sessionKey: "roof");
await UntilIdleAsync(server);
var replan = await DryRunAsync(admin, request);
Assert.Equal((3, 1), (replan.Photos, replan.Skipped));
var stays = replan.Steps.Single().Skipped.Single();
Assert.Equal(("missing", 1, Loc.Get("Archive.Convert.Skipped.Missing", italian)), (stays.Reason, stays.Photos, stays.Message));
var left = (await DryRunAsync(admin, request with { Mode = "new-sequence" })).Steps.Single().Skipped.Single();
Assert.Equal(("missing", 1, Loc.Get("Archive.Convert.NotCopied.Missing", italian)), (left.Reason, left.Photos, left.Message));
// The photo back on disk alone: still the plan kept. Another upload plans again, and the confirmation takes that
// plan, then lets it go once its job is queued.
await File.WriteAllBytesAsync(file, bytes);
Assert.Equal(replan.Fingerprint, (await DryRunAsync(admin, request)).Fingerprint);
await Uploads.UploadAsync(editor, Photo(4), "IMG_0005.JPG", sessionKey: "roof");
await UntilIdleAsync(server);
var whole = await DryRunAsync(admin, request);
Assert.Equal((5, 0), (whole.Photos, whole.Skipped));
var kept = plans.Count;
JobDto job;
using (var applied = await admin.PostAsJsonAsync(Url, request with { DryRun = false, Fingerprint = whole.Fingerprint }, ProtocolJson.Default.ConvertRequest))
{
await applied.ExpectAsync(HttpStatusCode.Accepted);
job = await applied.ReadAsync(ProtocolJson.Default.JobDto);
}
Assert.Equal(kept - 1, plans.Count);
await UntilJobAsync(admin, job.Id, "done");
Assert.True((await server.Database.Frames.ListBySequenceAsync(id)).All(f => f.Format == ImageFileFormat.Tiff), "converted");
// Converted: the next dry run is another plan, which says why nothing converts, in New sequence's words too.
var done = await DryRunAsync(admin, request with { Mode = "new-sequence" });
Assert.Equal((0, 5), (done.Photos, done.Skipped));
Assert.Equal(Loc.Get("Archive.Convert.NotCopied.SameFormat", italian), done.Steps.Single().Skipped.Single().Message);
}
[Test]
public async Task TheFreeSpaceIsMeasuredAgainAtEveryRequest()
{
await using var server = await TestServer.StartAsync(context, services: s =>
{
ArchiveModule.Register(s);
s.AddSingleton<FakeVolumeSpace>();
s.AddSingleton<VolumeSpace>(sp => sp.GetRequiredService<FakeVolumeSpace>());
});
var volumes = server.Services.GetRequiredService<FakeVolumeSpace>();
var (editor, _, _) = await server.UserAsync("alberto", "editor");
var admin = await server.AdminAsync();
Guid id = default;
for (var i = 0; i < 2; i++) id = (await Uploads.UploadAsync(editor, Photo(i), $"IMG_{i + 1:0000}.JPG", sessionKey: "roof")).SequenceId;
await UntilIdleAsync(server);
var request = new ConvertRequest([id], false, "linear-dng", null, "replace", DryRun: true);
const long Full = 1_000_000, Free = 100_000_000_000;
volumes.Set(server.Options.ArchivePath, new VolumeFigures(Full, 4 * Free));
var tight = await DryRunAsync(admin, request);
Assert.Equal((false, (long?)Full), (tight.Space.Enough, tight.Space.Free));
using (var refused = await admin.PostAsJsonAsync(Url, request with { DryRun = false, Fingerprint = tight.Fingerprint }, ProtocolJson.Default.ConvertRequest))
Assert.Equal(tight.Space.Required.ToString(CultureInfo.InvariantCulture), (await refused.ExpectErrorAsync(HttpStatusCode.Conflict, "not_enough_space")).Details);
// Space freed since: the plan kept, with the free space now; and the confirmation measures it again too.
volumes.Set(server.Options.ArchivePath, new VolumeFigures(Free, 4 * Free));
var freed = await DryRunAsync(admin, request);
Assert.Equal((true, (long?)Free, tight.Fingerprint, tight.Space.Required), (freed.Space.Enough, freed.Space.Free, freed.Fingerprint, freed.Space.Required));
Assert.Equal(1, server.Services.GetRequiredService<PlanCache>().Count);
volumes.Set(server.Options.ArchivePath, new VolumeFigures(Full, 4 * Free));
using (var filled = await admin.PostAsJsonAsync(Url, request with { DryRun = false, Fingerprint = freed.Fingerprint }, ProtocolJson.Default.ConvertRequest))
await filled.ExpectErrorAsync(HttpStatusCode.Conflict, "not_enough_space");
volumes.Set(server.Options.ArchivePath, new VolumeFigures(Free, 4 * Free));
using (var applied = await admin.PostAsJsonAsync(Url, request with { DryRun = false, Fingerprint = freed.Fingerprint }, ProtocolJson.Default.ConvertRequest))
await UntilJobAsync(admin, (await applied.ReadAsync(ProtocolJson.Default.JobDto)).Id, "done");
Assert.True((await server.Database.Frames.ListBySequenceAsync(id)).All(f => f.Format == ImageFileFormat.Dng), "converted");
}
private static async Task<ConvertPlanDto> DryRunAsync(HttpClient client, ConvertRequest request)
{
using var response = await client.PostAsJsonAsync(Url, request with { DryRun = true }, ProtocolJson.Default.ConvertRequest);
return await response.ReadAsync(ProtocolJson.Default.ConvertPlanDto);
}
/// <summary>Every reason a photo stays as it is has its wire name and its sentence in the catalog (en and it).</summary>
@@ -0,0 +1,113 @@
using System.Collections.Concurrent;
using Chronos.Core.Archiving;
using Chronos.Core.Conversion;
using Chronos.Server.Api;
using Chronos.Testing;
namespace Chronos.Server.Tests.Api;
/// <summary>
/// The dry runs the server keeps (<see cref="PlanCache"/>, owner review 2, R7) as Convert's endpoint uses them: what a
/// conversion plan is kept by (<see cref="ConvertPlanKey"/>), what refuses it, and the turns and room it shares with
/// Reorganize's plans. The rest of what the cache does is in <see cref="ReorganizeTests"/>.
/// </summary>
[Category("server")]
public sealed class PlanCacheTests
{
private static readonly TimeSpan Wait = TimeSpan.FromSeconds(TestTiming.Budget(30));
[Test]
public void AConversionPlanIsKeptByWhatConvertsNotByTheMode()
{
var archive = Guid.NewGuid();
var (x, y) = (Guid.NewGuid(), Guid.NewGuid());
var tiff = ConvertPlanKey.Of(archive, "stamp", [x, y], new ConvertOptions(ConvertTarget.Tiff16, 80));
// The mode is not part of it, and neither is the quality of a lossless target; a JPEG's is.
Assert.Equal(tiff, ConvertPlanKey.Of(archive, "stamp", [x, y], new ConvertOptions(ConvertTarget.Tiff16)));
Assert.Equal(ConvertPlanKey.Of(archive, "stamp", [x], new ConvertOptions(ConvertTarget.Jpeg, 80)),
ConvertPlanKey.Of(archive, "stamp", [x], new ConvertOptions(ConvertTarget.Jpeg, 80)));
Assert.NotEqual(ConvertPlanKey.Of(archive, "stamp", [x], new ConvertOptions(ConvertTarget.Jpeg, 80)),
ConvertPlanKey.Of(archive, "stamp", [x], new ConvertOptions(ConvertTarget.Jpeg, 95)));
// Another target, order of the sequences, every sequence, stamp or archive is another plan.
Assert.NotEqual(tiff, ConvertPlanKey.Of(archive, "stamp", [x, y], new ConvertOptions(ConvertTarget.LinearDng)));
Assert.NotEqual(tiff, ConvertPlanKey.Of(archive, "stamp", [y, x], new ConvertOptions(ConvertTarget.Tiff16)));
Assert.NotEqual(tiff, ConvertPlanKey.Of(archive, "stamp", null, new ConvertOptions(ConvertTarget.Tiff16)));
Assert.NotEqual(tiff, ConvertPlanKey.Of(archive, "other", [x, y], new ConvertOptions(ConvertTarget.Tiff16)));
Assert.NotEqual(tiff, ConvertPlanKey.Of(Guid.NewGuid(), "stamp", [x, y], new ConvertOptions(ConvertTarget.Tiff16)));
// A Reorganize of the same sequences is never the same plan.
Assert.NotEqual<PlanKey>(ConvertPlanKey.Of(archive, "stamp", null, new ConvertOptions(ConvertTarget.Tiff16)),
ReorganizePlanKey.Of(archive, "stamp", null, new RenameTemplates(Folder: "moved/{date}")));
// A sequence not in the archive refuses it; a failure (a sidecar unreadable, a bug) does not.
Assert.True(tiff.Refuses(new ArgumentException("Not in the archive.")));
foreach (var failure in new Exception[] { new ArgumentNullException("plan"), new IOException("The disk went away."), new NestedRenameException("a/b", "a") })
Assert.False(tiff.Refuses(failure), failure.GetType().Name);
}
[Test]
public async Task ConversionAndReorganizePlansShareTheTurnsAndTheRoom()
{
var clock = new SteppedTimeProvider();
using var plans = new PlanCache(clock);
var archive = Guid.NewGuid();
var templates = new RenameTemplates(Folder: "moved/{date}");
var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
var started = new ConcurrentDictionary<string, bool>();
Func<CancellationToken, Task<RenamePlan>> Rename(int i) => async stop =>
{
started["reorganize" + i] = true;
await release.Task.WaitAsync(stop);
return new RenamePlan(Guid.NewGuid(), archive, []);
};
var made = 0;
Task<ConvertPlan> Convert(CancellationToken stop)
{
started["convert"] = true;
Interlocked.Increment(ref made);
return Task.FromResult(new ConvertPlan(archive, new ConvertOptions(ConvertTarget.LinearDng), ConvertMode.Replace, [], new ConvertSpace(0, null)));
}
// Every turn taken by a Reorganize: a conversion waits for one.
var renames = Enumerable.Range(0, PlanCache.Concurrency)
.Select(i => plans.GetOrPlanAsync(ReorganizePlanKey.Of(archive, "stamp", [new Guid(i, 0, 0, new byte[8])], templates), Rename(i))).ToList();
await EventuallyAsync(() => started.Count == PlanCache.Concurrency);
var key = ConvertPlanKey.Of(archive, "stamp", null, new ConvertOptions(ConvertTarget.LinearDng));
var converting = plans.GetOrPlanAsync(key, Convert);
await Task.Delay(200);
Assert.False(started.ContainsKey("convert"), "no turn while Reorganize holds them");
release.SetResult();
var plan = await converting;
foreach (var rename in renames) await rename;
Assert.True(ReferenceEquals(plan, await plans.GetOrPlanAsync(key, Convert)), "kept");
Assert.Equal(1, made);
Assert.Equal(PlanCache.Concurrency + 1, plans.Count);
// A refusal is kept as a plan is, a failure is not; the room is shared: the oldest of either kind goes first.
var unknown = ConvertPlanKey.Of(archive, "stamp", [Guid.NewGuid()], new ConvertOptions(ConvertTarget.LinearDng));
await Assert.ThrowsAsync<ArgumentException>(() => plans.GetOrPlanAsync(unknown, _ => throw new ArgumentException("Not in the archive.")));
await Assert.ThrowsAsync<ArgumentException>(() => plans.GetOrPlanAsync(unknown, Convert));
var offline = ConvertPlanKey.Of(archive, "stamp", [Guid.NewGuid()], new ConvertOptions(ConvertTarget.Tiff16));
await Assert.ThrowsAsync<ArchiveOfflineException>(() => plans.GetOrPlanAsync(offline, _ => throw new ArchiveOfflineException("/archive", ArchiveAvailability.Offline)));
await plans.GetOrPlanAsync(offline, Convert);
Assert.Equal(2, made);
for (var i = 0; i < PlanCache.Capacity; i++)
{
clock.Advance(TimeSpan.FromSeconds(1));
await plans.GetOrPlanAsync(ConvertPlanKey.Of(archive, "stamp", [new Guid(i, 0, 0, new byte[8])], new ConvertOptions(ConvertTarget.Jpeg)), Convert);
}
Assert.Equal(PlanCache.Capacity, plans.Count);
await plans.GetOrPlanAsync(key, Convert);
Assert.Equal(3 + PlanCache.Capacity, made);
}
/// <summary>Waits until <paramref name="condition"/> holds.</summary>
private static async Task EventuallyAsync(Func<bool> condition, [System.Runtime.CompilerServices.CallerArgumentExpression(nameof(condition))] string? expression = null)
{
var until = DateTime.UtcNow + Wait;
while (!condition())
{
Assert.True(DateTime.UtcNow < until, expression);
await Task.Delay(10);
}
}
}
@@ -16,7 +16,7 @@ namespace Chronos.Server.Tests.Api;
/// <summary>
/// Reorganize on the server (owner review 2, R7): <c>POST /api/v1/archive/reorganize</c>, its dry run and the plan kept from it
/// for the confirmation (<see cref="ReorganizePlanCache"/>), the <c>move</c> job that applies the confirmed plan (paused and
/// for the confirmation (<see cref="PlanCache"/>), the <c>move</c> job that applies the confirmed plan (paused and
/// resumed too), and what the request may not do, against the real archive engine.
/// </summary>
[Category("server")]
@@ -562,7 +562,7 @@ public sealed class ReorganizeTests(TestContext context)
Directory.CreateDirectory(target);
Assert.Equal(free.Fingerprint, (await DryRunAsync(admin, request)).Fingerprint);
await UntilIdleAsync(server);
clock.Advance(ReorganizePlanCache.Lifetime);
clock.Advance(PlanCache.Lifetime);
var later = await DryRunAsync(admin, request);
Assert.Equal("moved/2026-09-28 (2)", later.Steps.Single().ToFolder);
using (var changed = await admin.PostAsJsonAsync(Url, request with { DryRun = false, Fingerprint = free.Fingerprint }, ProtocolJson.Default.ReorganizeRequest))
@@ -604,7 +604,7 @@ public sealed class ReorganizeTests(TestContext context)
public async Task KeptPlansAreTheNewestFewWithinTheirLifetime()
{
var clock = new ManualTimeProvider();
using var plans = new ReorganizePlanCache(clock);
using var plans = new PlanCache(clock);
var archive = Guid.NewGuid();
var templates = new RenameTemplates(Folder: "moved/{date}");
var made = 0;
@@ -613,38 +613,38 @@ public sealed class ReorganizeTests(TestContext context)
made++;
return Task.FromResult(new RenamePlan(Guid.NewGuid(), archive, []));
}
ReorganizePlanCache.Key Key(int i) => ReorganizePlanCache.Key.Of(archive, "stamp", [new Guid(i, 0, 0, new byte[8])], templates);
ReorganizePlanKey Key(int i) => ReorganizePlanKey.Of(archive, "stamp", [new Guid(i, 0, 0, new byte[8])], templates);
var kept = await plans.GetOrPlanAsync(Key(0), Plan);
Assert.True(ReferenceEquals(kept, await plans.GetOrPlanAsync(Key(0), Plan)), "kept");
Assert.Equal(1, made);
// Another order of the sequences, every sequence, another stamp or other templates is another plan.
var (x, y) = (Guid.NewGuid(), Guid.NewGuid());
Assert.NotEqual(ReorganizePlanCache.Key.Of(archive, "stamp", [x, y], templates), ReorganizePlanCache.Key.Of(archive, "stamp", [y, x], templates));
Assert.Equal(ReorganizePlanCache.Key.Of(archive, "stamp", [x, y], templates), ReorganizePlanCache.Key.Of(archive, "stamp", [x, y], templates with { }));
Assert.NotEqual(ReorganizePlanCache.Key.Of(archive, "stamp", null, templates), ReorganizePlanCache.Key.Of(archive, "stamp", [x], templates));
Assert.NotEqual(Key(0), ReorganizePlanCache.Key.Of(archive, "other", [new Guid(0, 0, 0, new byte[8])], templates));
Assert.NotEqual(Key(0), ReorganizePlanCache.Key.Of(archive, "stamp", [new Guid(0, 0, 0, new byte[8])], templates with { File = "{n}" }));
Assert.NotEqual(ReorganizePlanKey.Of(archive, "stamp", [x, y], templates), ReorganizePlanKey.Of(archive, "stamp", [y, x], templates));
Assert.Equal(ReorganizePlanKey.Of(archive, "stamp", [x, y], templates), ReorganizePlanKey.Of(archive, "stamp", [x, y], templates with { }));
Assert.NotEqual(ReorganizePlanKey.Of(archive, "stamp", null, templates), ReorganizePlanKey.Of(archive, "stamp", [x], templates));
Assert.NotEqual(Key(0), ReorganizePlanKey.Of(archive, "other", [new Guid(0, 0, 0, new byte[8])], templates));
Assert.NotEqual(Key(0), ReorganizePlanKey.Of(archive, "stamp", [new Guid(0, 0, 0, new byte[8])], templates with { File = "{n}" }));
// At most Capacity plans: the oldest goes first.
for (var i = 1; i <= ReorganizePlanCache.Capacity; i++)
for (var i = 1; i <= PlanCache.Capacity; i++)
{
clock.Advance(TimeSpan.FromSeconds(1));
await plans.GetOrPlanAsync(Key(i), Plan);
}
Assert.Equal(1 + ReorganizePlanCache.Capacity, made);
await plans.GetOrPlanAsync(Key(ReorganizePlanCache.Capacity), Plan);
Assert.Equal(1 + ReorganizePlanCache.Capacity, made);
Assert.Equal(1 + PlanCache.Capacity, made);
await plans.GetOrPlanAsync(Key(PlanCache.Capacity), Plan);
Assert.Equal(1 + PlanCache.Capacity, made);
await plans.GetOrPlanAsync(Key(0), Plan);
Assert.Equal(2 + ReorganizePlanCache.Capacity, made);
Assert.Equal(2 + PlanCache.Capacity, made);
// A plan is kept for its lifetime, from when it was made, even before its timer lets it go.
clock.Advance(ReorganizePlanCache.Lifetime - TimeSpan.FromSeconds(1));
clock.Advance(PlanCache.Lifetime - TimeSpan.FromSeconds(1));
await plans.GetOrPlanAsync(Key(0), Plan);
Assert.Equal(2 + ReorganizePlanCache.Capacity, made);
Assert.Equal(2 + PlanCache.Capacity, made);
clock.Advance(TimeSpan.FromSeconds(1));
await plans.GetOrPlanAsync(Key(0), Plan);
Assert.Equal(3 + ReorganizePlanCache.Capacity, made);
Assert.Equal(3 + PlanCache.Capacity, made);
// A refusal follows from the same reads, and is kept as a plan is: a folder inside a sequence's, a sequence not in the
// archive. A failure is not kept, and neither is a plan or refusal forgotten.
@@ -652,22 +652,22 @@ public sealed class ReorganizeTests(TestContext context)
await Assert.ThrowsAsync<NestedRenameException>(() => plans.GetOrPlanAsync(Key(99), Plan));
await Assert.ThrowsAsync<ArgumentException>(() => plans.GetOrPlanAsync(Key(98), _ => throw new ArgumentException("Not in the archive.")));
await Assert.ThrowsAsync<ArgumentException>(() => plans.GetOrPlanAsync(Key(98), Plan));
Assert.Equal(3 + ReorganizePlanCache.Capacity, made);
Assert.Equal(3 + PlanCache.Capacity, made);
await Assert.ThrowsAsync<IOException>(() => plans.GetOrPlanAsync(Key(97), _ => throw new IOException("The sidecar cannot be read.")));
await plans.GetOrPlanAsync(Key(97), Plan);
Assert.Equal(4 + ReorganizePlanCache.Capacity, made);
Assert.Equal(4 + PlanCache.Capacity, made);
plans.Forget(Key(99));
await plans.GetOrPlanAsync(Key(99), Plan);
Assert.Equal(5 + ReorganizePlanCache.Capacity, made);
Assert.Equal(5 + PlanCache.Capacity, made);
}
[Test]
public async Task APlanIsMadeOnceForEveryRequestAndOutlivesOneCutOff()
{
var clock = new SteppedTimeProvider();
using var plans = new ReorganizePlanCache(clock);
using var plans = new PlanCache(clock);
var templates = new RenameTemplates(Folder: "moved/{date}");
var key = ReorganizePlanCache.Key.Of(Guid.NewGuid(), "stamp", null, templates);
var key = ReorganizePlanKey.Of(Guid.NewGuid(), "stamp", null, templates);
var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
var made = 0;
async Task<RenamePlan> SlowPlan(CancellationToken stop)
@@ -687,22 +687,22 @@ public sealed class ReorganizeTests(TestContext context)
var again = plans.GetOrPlanAsync(key, SlowPlan);
var meanwhile = plans.GetOrPlanAsync(key, SlowPlan);
// Being made, it is not the one let go for other plans.
for (var i = 0; i < ReorganizePlanCache.Capacity; i++)
await plans.GetOrPlanAsync(ReorganizePlanCache.Key.Of(key.ArchiveId, "stamp", [new Guid(i, 0, 0, new byte[8])], templates), PlanAtOnce);
for (var i = 0; i < PlanCache.Capacity; i++)
await plans.GetOrPlanAsync(ReorganizePlanKey.Of(key.ArchiveId, "stamp", [new Guid(i, 0, 0, new byte[8])], templates), PlanAtOnce);
release.SetResult();
var plan = await again;
Assert.True(ReferenceEquals(plan, await meanwhile), "one plan for both");
Assert.True(ReferenceEquals(plan, await plans.GetOrPlanAsync(key, SlowPlan)), "kept");
Assert.Equal(1 + ReorganizePlanCache.Capacity, made);
Assert.Equal(ReorganizePlanCache.Capacity, plans.Count);
Assert.Equal(1 + PlanCache.Capacity, made);
Assert.Equal(PlanCache.Capacity, plans.Count);
// Its timer lets it go at the end of its lifetime, with no request asking.
clock.Advance(ReorganizePlanCache.Lifetime - TimeSpan.FromSeconds(1));
Assert.Equal(ReorganizePlanCache.Capacity, plans.Count);
clock.Advance(PlanCache.Lifetime - TimeSpan.FromSeconds(1));
Assert.Equal(PlanCache.Capacity, plans.Count);
clock.Advance(TimeSpan.FromSeconds(1));
Assert.Equal(0, plans.Count);
await plans.GetOrPlanAsync(key, SlowPlan);
Assert.Equal(2 + ReorganizePlanCache.Capacity, made);
Assert.Equal(2 + PlanCache.Capacity, made);
Task<RenamePlan> PlanAtOnce(CancellationToken stop)
{
@@ -715,10 +715,10 @@ public sealed class ReorganizeTests(TestContext context)
public async Task PlansAreMadeAFewAtOnceAndStopWhenLetGoOrPastTheirTime()
{
var clock = new SteppedTimeProvider();
using var plans = new ReorganizePlanCache(clock);
using var plans = new PlanCache(clock);
var archive = Guid.NewGuid();
var templates = new RenameTemplates(Folder: "moved/{date}");
ReorganizePlanCache.Key Key(int i) => ReorganizePlanCache.Key.Of(archive, "stamp", [new Guid(i, 0, 0, new byte[8])], templates);
ReorganizePlanKey Key(int i) => ReorganizePlanKey.Of(archive, "stamp", [new Guid(i, 0, 0, new byte[8])], templates);
var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
var started = new ConcurrentDictionary<int, bool>();
var stopped = new ConcurrentDictionary<int, bool>();
@@ -739,16 +739,16 @@ public sealed class ReorganizeTests(TestContext context)
};
// A few are made at once; the others wait for their turn.
var asked = Enumerable.Range(0, ReorganizePlanCache.Capacity).Select(i => plans.GetOrPlanAsync(Key(i), Plan(i))).ToList();
await EventuallyAsync(() => started.Count == ReorganizePlanCache.Concurrency);
var asked = Enumerable.Range(0, PlanCache.Capacity).Select(i => plans.GetOrPlanAsync(Key(i), Plan(i))).ToList();
await EventuallyAsync(() => started.Count == PlanCache.Concurrency);
await Task.Delay(200);
Assert.SequenceEqual(Enumerable.Range(0, ReorganizePlanCache.Concurrency), started.Keys.Order());
Assert.SequenceEqual(Enumerable.Range(0, PlanCache.Concurrency), started.Keys.Order());
// One more lets go the plan asked first, being made with none kept: it stops, its waiter is told, the next one starts.
var newest = plans.GetOrPlanAsync(Key(ReorganizePlanCache.Capacity), Plan(ReorganizePlanCache.Capacity));
var newest = plans.GetOrPlanAsync(Key(PlanCache.Capacity), Plan(PlanCache.Capacity));
await Assert.ThrowsAsync<OperationCanceledException>(() => asked[0]);
await EventuallyAsync(() => stopped.ContainsKey(0) && started.ContainsKey(ReorganizePlanCache.Concurrency));
Assert.Equal(ReorganizePlanCache.Capacity, plans.Count);
await EventuallyAsync(() => stopped.ContainsKey(0) && started.ContainsKey(PlanCache.Concurrency));
Assert.Equal(PlanCache.Capacity, plans.Count);
release.SetResult();
foreach (var plan in asked.Skip(1).Append(newest)) await plan;
Assert.SequenceEqual([0], stopped.Keys);
@@ -765,7 +765,7 @@ public sealed class ReorganizeTests(TestContext context)
var stuck = plans.GetOrPlanAsync(Key(21), Hanging(21));
await EventuallyAsync(() => started.ContainsKey(20) && started.ContainsKey(21));
var waiting = plans.GetOrPlanAsync(Key(22), Plan(22));
clock.Advance(ReorganizePlanCache.PlanTimeout - TimeSpan.FromSeconds(1));
clock.Advance(PlanCache.PlanTimeout - TimeSpan.FromSeconds(1));
Assert.False(hung.IsCompleted, "still in time");
clock.Advance(TimeSpan.FromSeconds(1));
foreach (var plan in new[] { hung, stuck, waiting }) await Assert.ThrowsAsync<OperationCanceledException>(() => plan);
@@ -66,6 +66,19 @@ public sealed partial class AdminConvertTests(TestContext context)
Assert.Contains("if (plan !== shownPlan && (plan?.key !== shownPlan.key || plan.fingerprint !== shownPlan.fingerprint)) {", dialog);
Assert.Contains(@"showError(t(""Admin.Convert.PlanChanged""))", dialog);
Assert.True(!dialog.Contains("plan !== shownPlan) return;", StringComparison.Ordinal), "a confirmed apply is dropped when the same plan is answered again");
// Until a mode is chosen the preview is asked in Replace's words (the photos stay as they are); New sequence asks
// again and says what the new sequences lack, in the rows, the totals and the confirmation.
Assert.Contains(@"const PREVIEW_MODE = ""replace"";", dialog);
Assert.Contains(@"t(copies ? ""Admin.Convert.TotalsNew"" : ""Admin.Convert.Totals""", dialog);
Assert.Contains(@"plural(""Admin.Convert.ConfirmNewNotCopied"", notCopied)", dialog);
// A dry run the server is still making is asked again (a proxy cut the request off, or the server let the plan go)
// while the totals say how long it takes; a confirmation the server let go plans again, one that changed says so,
// and one with no answer is not sent again (its job may be queued).
Assert.Contains(@"cutOff(err, Date.now() - sent) || (err?.code === ""plan_dropped"" && dropped++ < DROPPED_RETRIES)", dialog);
Assert.Contains(@"t(""Admin.Convert.PlanningFor"", formatDuration(seconds))", dialog);
Assert.Contains(@"[""plan_changed"", ""plan_dropped"", ""not_enough_space"", ""nothing_to_convert""].includes(err?.code)", dialog);
Assert.Contains(@"notice = err.code === ""plan_changed"" ? t(""Admin.Convert.PlanChanged"") : errorText(err);", dialog);
Assert.Contains(@"showError(t(""Admin.Convert.NoAnswer""));", dialog);
// The panel runs under a strict CSP: no inline handlers or styles.
Assert.True(!Regex.IsMatch(dialog, @"\son[a-z]+="""), "inline handler");
Assert.True(!dialog.Contains(" style=", StringComparison.Ordinal), "inline style");
@@ -105,24 +118,24 @@ public sealed partial class AdminConvertTests(TestContext context)
var browser = await Browser.SignInAsync(server);
var italian = CultureInfo.GetCultureInfo("it");
// The body the dialog writes: the selection's ids, the format, the quality for a JPEG only, the mode (the preview's
// while none is chosen), then dryRun.
// The body the dialog writes: the selection's ids, the format, the quality for a JPEG only, the mode (the preview's,
// Replace, while none is chosen), then dryRun.
StringContent Body(string target, string quality, string mode, string tail) =>
Json($$"""{"all":false,"sequenceIds":["{{id}}"],"target":"{{target}}","quality":{{quality}},"mode":"{{mode}}",{{tail}}}""");
string preview;
using (var dry = await browser.SendAsync(HttpMethod.Post, Url, Body("linear-dng", "null", "new-sequence", "\"dryRun\":true")))
string fingerprint;
using (var dry = await browser.SendAsync(HttpMethod.Post, Url, Body("linear-dng", "null", "replace", "\"dryRun\":true")))
{
await dry.ExpectAsync(HttpStatusCode.OK);
using var plan = JsonDocument.Parse(await dry.Content.ReadAsStringAsync());
var root = plan.RootElement;
Assert.Equal(("linear-dng", "new-sequence"), (root.GetProperty("target").GetString(), root.GetProperty("mode").GetString()));
Assert.Equal(("linear-dng", "replace"), (root.GetProperty("target").GetString(), root.GetProperty("mode").GetString()));
Assert.Equal((1, 2, 0), (root.GetProperty("sequences").GetInt32(), root.GetProperty("photos").GetInt32(), root.GetProperty("skipped").GetInt32()));
Assert.True(root.GetProperty("bytesBefore").GetInt64() > 0 && root.GetProperty("bytesAfter").GetInt64() > 0, "sizes");
var space = root.GetProperty("space");
Assert.True(space.GetProperty("enough").GetBoolean(), "enough");
Assert.False(string.IsNullOrWhiteSpace(space.GetProperty("message").GetString()));
preview = root.GetProperty("fingerprint").GetString()!;
Assert.Equal(64, preview.Length);
fingerprint = root.GetProperty("fingerprint").GetString()!;
Assert.Equal(64, fingerprint.Length);
var step = root.GetProperty("steps").EnumerateArray().Single();
Assert.Equal(id, step.GetProperty("sequenceId").GetGuid());
Assert.True(step.GetProperty("folder").GetString()!.Length > 0, "folder");
@@ -132,14 +145,13 @@ public sealed partial class AdminConvertTests(TestContext context)
Assert.Equal((0, 0), (step.GetProperty("skipped").GetArrayLength(), step.GetProperty("blocked").GetArrayLength()));
}
// Replace chosen: the same photos, another fingerprint, so the dialog plans again before Apply.
string fingerprint;
using (var dry = await browser.SendAsync(HttpMethod.Post, Url, Body("linear-dng", "null", "replace", "\"dryRun\":true")))
// New sequence chosen: the same photos, another fingerprint, so the dialog asks again before Apply (the server
// answers from the plan it kept).
using (var dry = await browser.SendAsync(HttpMethod.Post, Url, Body("linear-dng", "null", "new-sequence", "\"dryRun\":true")))
{
using var plan = JsonDocument.Parse(await dry.Content.ReadAsStringAsync());
Assert.Equal(2, plan.RootElement.GetProperty("photos").GetInt32());
fingerprint = plan.RootElement.GetProperty("fingerprint").GetString()!;
Assert.NotEqual(preview, fingerprint);
Assert.Equal(("new-sequence", 2), (plan.RootElement.GetProperty("mode").GetString(), plan.RootElement.GetProperty("photos").GetInt32()));
Assert.NotEqual(fingerprint, plan.RootElement.GetProperty("fingerprint").GetString());
}
// A quality the server refuses names its field: the dialog marks Quality.
@@ -154,6 +166,14 @@ public sealed partial class AdminConvertTests(TestContext context)
Assert.Equal(("same_format", 2), (skipped.GetProperty("reason").GetString(), skipped.GetProperty("photos").GetInt32()));
Assert.Equal(Loc.Get("Archive.Convert.Skipped.SameFormat", italian), skipped.GetProperty("message").GetString());
}
// Into a new sequence, the same photos are not copied into it, and the table says so.
using (var same = await browser.SendAsync(HttpMethod.Post, Url, Body("jpeg", "90", "new-sequence", "\"dryRun\":true")))
{
using var plan = JsonDocument.Parse(await same.Content.ReadAsStringAsync());
var skipped = plan.RootElement.GetProperty("steps")[0].GetProperty("skipped").EnumerateArray().Single();
Assert.Equal(("same_format", 2), (skipped.GetProperty("reason").GetString(), skipped.GetProperty("photos").GetInt32()));
Assert.Equal(Loc.Get("Archive.Convert.NotCopied.SameFormat", italian), skipped.GetProperty("message").GetString());
}
// A plan that no longer matches is refused with the one there is now: the dialog plans again.
using (var stale = await browser.SendAsync(HttpMethod.Post, Url, Body("linear-dng", "null", "replace", $"\"dryRun\":false,\"fingerprint\":\"{new string('0', 64)}\"")))
Assert.Equal(fingerprint, (await stale.ExpectErrorAsync(HttpStatusCode.Conflict, "plan_changed")).Details);