fix(archive): settle trash journals under their leases and say why restored photos stayed

Second fix round of owner review 2, phase 3 (engine, server, desktop).

- A photo never hashed whose original is missing can be deleted again: it goes
  into its trash record without a file and without a tombstone (nothing is left
  to name), stays listed until Empty erases it, and a restore keeps it with the
  reason "no file". FrameDeleteBlock.Unhashable now blocks only a present
  original that cannot be read; the API answers it as 409 photo_unreadable
  naming the photo, and every block has its own code and message.
- Deleting a whole sequence first settles an interrupted photo deletion,
  restore or ingest of it (ReadLocked under its lease), so no journal or half
  entry travels into the sequence's trash entry; it refuses while a restore
  into it waits for its busy entry.
- FrameTrash.Settle takes the photo entry's lease and leaves the journal for
  later while the entry is busy (an emptying), so it never races EraseAsync;
  meanwhile no other restore writes its journal over it and scans do not adopt
  its files.
- The hashes computed for frames that stay (to find a twin of a deleted photo)
  go into the sidecar and index, and only a frame whose first and last 64 KiB
  match a deleted photo's is read whole.
- A restore into a new sequence rewrites its record after each photo filed, so
  a crash or a database failure never leaves filed photos listed.
- An upload completion no longer has a window before the commit: the ingest
  checks the sidecar's tombstones under the sequence lease a deletion takes
  (IngestRequest.RefuseRemoved), unless the upload asked to bring it back.
- A restore reports each kept photo with its reason (no longer matches its
  record, could not be moved back, already in the sequence, had no file); the
  desktop line and the API message say so in en and it, and the tooltip holds
  the whole sentence. The desktop tells TrashMismatchException by its type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 1858f1f450a35fb8383ede8d80ce5f8de9b7ceac)
This commit is contained in:
Alby96 committed 2026-10-27 13:53:41 +01:00
1 parent a133306fb0
commit 4518ffe655
29 files changed
+856 -162

No files matched your search

+1 -1
View File
@@ -58,7 +58,7 @@ Dependencies point one way: `Core` ← `Core.Windows`, `Data`, `Protocol` ← `S
| `Jobs` | `JobEngine` (one job at a time; reviews the priority queue and makes a lower running job yield, requeued with its checkpoint; no history purge, the housekeeping schedule's clean-up does it), `JobPriorityQueue` (a level per job kind, the claim order, settings round trip), `IJobHandler`, `IJobHandlerProvider`, `JobContext` (progress, checkpoints, follow-up jobs that never interrupt their parent), `JobThroughput`/`JobEta` (rate and bounded time left) |
| `Scheduling` | `Scheduler` (one per host: seeds the host's system schedules, claims due runs by compare-and-set, catch-up, clock changes, create/edit/reset/delete/run now as API v1 DTOs), `ScheduleRecurrence` (the in-house `every:`/`daily:`/`weekly:`/`monthly:` form, wall-clock times across DST), `ScheduleActions` (action names, parameters, availability, the archive jobs a run queues), `IScheduleHost` and `SystemSchedule` (what the server and the desktop supply), `ScheduleOutcome`, `ScheduleMapping` (model ↔ DTO) |
| `Housekeeping` | `BackupJobHandler`, `CleanupJobHandler`, `ICleanupStep` with the shared steps (`CleanupSteps`: job history, audit log, trash report that never deletes); hosts add their own steps through `ArchiveServiceOptions.CleanupSteps` |
| `Archiving` | `ArchiveService` (create/adopt/open, archive settings written to `archive.json` then the index), `ArchiveIndexer` (scan, rebuild), `ArchiveIngest` (uploads and imports), `ImportService`, `ArchiveVerifier`, `ArchiveImages` (thumbnail and proxy passes, several photos at once through `OrderedWork`, which hands results back in frame order), `FrameTiers` (decode tier, `PreviewOnly`/`Undecodable` flags and preview size of frames, written to sidecar and index by the light analysis, deep analysis, renders and image passes), `GprCompanions` (the JPEG of each GoPro GPR, found in the index when an import kept it in another sequence), `ArchiveMaintenance`, `MassRenamer`, `SequenceTrash` (delete into the archive's hidden trash with a dry run, restore; the folder moves back when the index step fails; lists sequence and photo entries; `SequenceTrash.Empty`: the emptying's dry run, fingerprint and erase, each entry checked against its fingerprint again under its lease), `FrameTrash` (photos of a sequence into `trash/photos` with a dry run, a journal that undoes an interrupted deletion whose entry stays unlisted until its sidecar is written, tombstones (a photo never hashed is hashed as it goes), positions parked and placed back, proxies and thumbnails moved to follow their frames; `FrameTrash.Restore`: restore re-hashes each file, under a journal of its own into the sequence, into a new sequence through the ingest when theirs is gone, each original staged inside its own entry), job handlers |
| `Archiving` | `ArchiveService` (create/adopt/open, archive settings written to `archive.json` then the index), `ArchiveIndexer` (scan, rebuild), `ArchiveIngest` (uploads and imports), `ImportService`, `ArchiveVerifier`, `ArchiveImages` (thumbnail and proxy passes, several photos at once through `OrderedWork`, which hands results back in frame order), `FrameTiers` (decode tier, `PreviewOnly`/`Undecodable` flags and preview size of frames, written to sidecar and index by the light analysis, deep analysis, renders and image passes), `GprCompanions` (the JPEG of each GoPro GPR, found in the index when an import kept it in another sequence), `ArchiveMaintenance`, `MassRenamer`, `SequenceTrash` (delete into the archive's hidden trash with a dry run, restore; the folder moves back when the index step fails; lists sequence and photo entries; `SequenceTrash.Empty`: the emptying's dry run, fingerprint and erase, each entry checked against its fingerprint again under its lease), `FrameTrash` (photos of a sequence into `trash/photos` with a dry run, a journal that undoes an interrupted deletion whose entry stays unlisted until its sidecar is written, tombstones (a photo never hashed is hashed as it goes; one whose original is missing goes without a file or tombstone; a frame that stays and may be a twin is hashed, its hash kept), positions parked and placed back, proxies and thumbnails moved to follow their frames; `FrameTrash.Restore`: restore re-hashes each file and says why each kept photo stayed (`KeptReason`), under a journal of its own into the sequence that another writer settles only under the entry's lease, into a new sequence through the ingest when theirs is gone, each original staged inside its own entry and the record rewritten after each photo filed), job handlers |
| `Processing` | `ProcessingService`, `SequenceAnalyser`, `DeepReports` (a saved deep report while it still describes the frames), `SequenceRender`, `ProjectStore`, `ProjectBundles` (a `.chronos` file for a sequence, and the sequence a file names), `RenderNodeClient`, job handlers |
| `Sync` | `SyncService` (sync table of one server: compare, link by content, settle conflicts, enqueue transfers), `SyncRules` (status, settings merge and differences, no I/O), `SyncTombstones` (photos either side removed are not offered back, a replaced photo counts as its replacement), `SyncPositions` (the settings' frame positions translated by photo to the frames of the copy they go to), `SyncLinkStore` (`sync_links`, `sync_state`), `SyncTransfers` and `SyncJobHandler` (resumable, hash-verified uploads and downloads as jobs; downloads commit through the archive ingest), `SyncAgent` (background comparison, server events, backoff, offline proxies), `ProxyCache` and `RemoteSequence` (server sequences opened with proxies), `ISyncRemote` (the server calls; `ChronosSyncRemote` over `ChronosClient`) |
-2
View File
@@ -759,8 +759,6 @@
<data name="Admin.Trash.EntryCount.one" xml:space="preserve"><value>{0} entry</value></data>
<data name="Admin.Trash.EntryCount.other" xml:space="preserve"><value>{0} entries</value></data>
<data name="Admin.Trash.IsEmpty" xml:space="preserve"><value>The trash is empty</value></data>
<data name="Admin.Trash.Kept.one" xml:space="preserve"><value>{0} file stays in the trash: it no longer matches its record.</value><comment>long</comment></data>
<data name="Admin.Trash.Kept.other" xml:space="preserve"><value>{0} files stay in the trash: they no longer match their record.</value><comment>long</comment></data>
<data name="Admin.Trash.Kind" xml:space="preserve"><value>Kind</value></data>
<data name="Admin.Trash.Kind.Photos" xml:space="preserve"><value>Photos</value></data>
<data name="Admin.Trash.Kind.Replaced" xml:space="preserve"><value>Replaced</value></data>
+12 -2
View File
@@ -266,8 +266,18 @@
<data name="Archive.Trash.Erase" xml:space="preserve"><value>Erase for good</value></data>
<data name="Archive.Trash.Failed" xml:space="preserve"><value>The sequence could not come back from the trash</value><comment>long</comment></data>
<data name="Archive.Trash.Heading" xml:space="preserve"><value>Trash · {0}</value><comment>long</comment></data>
<data name="Archive.Trash.KeptFiles.one" xml:space="preserve"><value>{0:N0} file stayed in the trash: it no longer matches its record.</value><comment>long</comment></data>
<data name="Archive.Trash.KeptFiles.other" xml:space="preserve"><value>{0:N0} files stayed in the trash: they no longer match their record.</value><comment>long</comment></data>
<data name="Archive.Trash.Kept.InSequence.one" xml:space="preserve"><value>{0:N0} photo stayed in the trash: the sequence holds it already.</value><comment>long</comment></data>
<data name="Archive.Trash.Kept.InSequence.other" xml:space="preserve"><value>{0:N0} photos stayed in the trash: the sequence holds them already.</value><comment>long</comment></data>
<data name="Archive.Trash.Kept.Mismatch.one" xml:space="preserve"><value>{0:N0} photo stayed in the trash: it no longer matches its record.</value><comment>long</comment></data>
<data name="Archive.Trash.Kept.Mismatch.other" xml:space="preserve"><value>{0:N0} photos stayed in the trash: they no longer match their record.</value><comment>long</comment></data>
<data name="Archive.Trash.Kept.NoFile.one" xml:space="preserve"><value>{0:N0} photo stayed in the trash: its file was already missing when it was deleted.</value><comment>long</comment></data>
<data name="Archive.Trash.Kept.NoFile.other" xml:space="preserve"><value>{0:N0} photos stayed in the trash: their files were already missing when they were deleted.</value><comment>long</comment></data>
<data name="Archive.Trash.Kept.NotMoved.one" xml:space="preserve"><value>{0:N0} photo stayed in the trash: it could not be moved back.</value><comment>long</comment></data>
<data name="Archive.Trash.Kept.NotMoved.other" xml:space="preserve"><value>{0:N0} photos stayed in the trash: they could not be moved back.</value><comment>long</comment></data>
<data name="Archive.Trash.KeptMore.one" xml:space="preserve"><value>+ {0:N0} more</value><comment>long</comment></data>
<data name="Archive.Trash.KeptMore.other" xml:space="preserve"><value>+ {0:N0} more</value><comment>long</comment></data>
<data name="Archive.Trash.KeptPhotos.one" xml:space="preserve"><value>{0:N0} photo stayed in the trash.</value><comment>long</comment></data>
<data name="Archive.Trash.KeptPhotos.other" xml:space="preserve"><value>{0:N0} photos stayed in the trash.</value><comment>long</comment></data>
<data name="Archive.Trash.Kind.Photos" xml:space="preserve"><value>Photos</value></data>
<data name="Archive.Trash.Kind.Replaced" xml:space="preserve"><value>Replaced originals</value></data>
<data name="Archive.Trash.Kind.Sequence" xml:space="preserve"><value>Sequence</value></data>
+1
View File
@@ -74,6 +74,7 @@
<data name="Server.Error.NotChronos" xml:space="preserve"><value>That address is not a Chronos Server.</value><comment>long</comment></data>
<data name="Server.Error.PasswordChangeRequired" xml:space="preserve"><value>Change your password first.</value><comment>long</comment></data>
<data name="Server.Error.PhotoRemoved" xml:space="preserve"><value>The photo was deleted from the sequence while it was being uploaded, so it was not added.</value><comment>long</comment></data>
<data name="Server.Error.PhotoUnreadable" xml:space="preserve"><value>{0} cannot be read, and without its hash other copies would not learn it was deleted; close the program using it or check its permissions, then try again.</value><comment>long</comment></data>
<data name="Server.Error.ProxyUntrusted" xml:space="preserve"><value>This page did not come through a trusted proxy over HTTPS. Open it through your proxy, or add the proxy's address to CHRONOS_TRUSTED_PROXIES.</value><comment>long</comment></data>
<data name="Server.Error.ProxiesDisabled" xml:space="preserve"><value>This archive makes no proxies.</value><comment>long</comment></data>
<data name="Server.Error.ProxyMismatch" xml:space="preserve"><value>The proxy was made from another photo.</value><comment>long</comment></data>
-2
View File
@@ -757,8 +757,6 @@
<data name="Admin.Trash.EntryCount.one" xml:space="preserve"><value>{0} voce</value></data>
<data name="Admin.Trash.EntryCount.other" xml:space="preserve"><value>{0} voci</value></data>
<data name="Admin.Trash.IsEmpty" xml:space="preserve"><value>Il cestino è vuoto</value></data>
<data name="Admin.Trash.Kept.one" xml:space="preserve"><value>{0} file resta nel cestino: non corrisponde più al suo record.</value><comment>long</comment></data>
<data name="Admin.Trash.Kept.other" xml:space="preserve"><value>{0} file restano nel cestino: non corrispondono più al loro record.</value><comment>long</comment></data>
<data name="Admin.Trash.Kind" xml:space="preserve"><value>Tipo</value></data>
<data name="Admin.Trash.Kind.Photos" xml:space="preserve"><value>Foto</value></data>
<data name="Admin.Trash.Kind.Replaced" xml:space="preserve"><value>Sostituite</value></data>
+12 -2
View File
@@ -259,8 +259,18 @@
<data name="Archive.Trash.Erase" xml:space="preserve"><value>Cancella per sempre</value></data>
<data name="Archive.Trash.Failed" xml:space="preserve"><value>Impossibile ripristinare la sequenza dal cestino</value></data>
<data name="Archive.Trash.Heading" xml:space="preserve"><value>Cestino · {0}</value></data>
<data name="Archive.Trash.KeptFiles.one" xml:space="preserve"><value>{0:N0} file è rimasto nel cestino: non corrisponde più al suo registro.</value></data>
<data name="Archive.Trash.KeptFiles.other" xml:space="preserve"><value>{0:N0} file sono rimasti nel cestino: non corrispondono più al loro registro.</value></data>
<data name="Archive.Trash.Kept.InSequence.one" xml:space="preserve"><value>{0:N0} foto è rimasta nel cestino: la sequenza la contiene già.</value></data>
<data name="Archive.Trash.Kept.InSequence.other" xml:space="preserve"><value>{0:N0} foto sono rimaste nel cestino: la sequenza le contiene già.</value></data>
<data name="Archive.Trash.Kept.Mismatch.one" xml:space="preserve"><value>{0:N0} foto è rimasta nel cestino: non corrisponde più al suo registro.</value></data>
<data name="Archive.Trash.Kept.Mismatch.other" xml:space="preserve"><value>{0:N0} foto sono rimaste nel cestino: non corrispondono più al loro registro.</value></data>
<data name="Archive.Trash.Kept.NoFile.one" xml:space="preserve"><value>{0:N0} foto è rimasta nel cestino: quando è stata eliminata il suo file mancava già.</value></data>
<data name="Archive.Trash.Kept.NoFile.other" xml:space="preserve"><value>{0:N0} foto sono rimaste nel cestino: quando sono state eliminate i loro file mancavano già.</value></data>
<data name="Archive.Trash.Kept.NotMoved.one" xml:space="preserve"><value>{0:N0} foto è rimasta nel cestino: non è stato possibile riportarla nella sequenza.</value></data>
<data name="Archive.Trash.Kept.NotMoved.other" xml:space="preserve"><value>{0:N0} foto sono rimaste nel cestino: non è stato possibile riportarle nella sequenza.</value></data>
<data name="Archive.Trash.KeptMore.one" xml:space="preserve"><value>+ un'altra</value></data>
<data name="Archive.Trash.KeptMore.other" xml:space="preserve"><value>+ altre {0:N0}</value></data>
<data name="Archive.Trash.KeptPhotos.one" xml:space="preserve"><value>{0:N0} foto è rimasta nel cestino.</value></data>
<data name="Archive.Trash.KeptPhotos.other" xml:space="preserve"><value>{0:N0} foto sono rimaste nel cestino.</value></data>
<data name="Archive.Trash.Kind.Photos" xml:space="preserve"><value>Foto</value></data>
<data name="Archive.Trash.Kind.Replaced" xml:space="preserve"><value>Originali sostituiti</value></data>
<data name="Archive.Trash.Kind.Sequence" xml:space="preserve"><value>Sequenza</value></data>
+1
View File
@@ -72,6 +72,7 @@
<data name="Server.Error.NotChronos" xml:space="preserve"><value>Questo indirizzo non è un Chronos Server.</value></data>
<data name="Server.Error.PasswordChangeRequired" xml:space="preserve"><value>Cambia prima la password.</value></data>
<data name="Server.Error.PhotoRemoved" xml:space="preserve"><value>La foto è stata eliminata dalla sequenza mentre veniva caricata, quindi non è stata aggiunta.</value><comment>long</comment></data>
<data name="Server.Error.PhotoUnreadable" xml:space="preserve"><value>Impossibile leggere {0}: senza il suo hash le altre copie non saprebbero che è stato eliminato. Chiudi il programma che lo usa o controlla i permessi, poi riprova.</value></data>
<data name="Server.Error.ProxyUntrusted" xml:space="preserve"><value>Questa pagina non è arrivata da un proxy attendibile in HTTPS. Aprila tramite il proxy o aggiungi l'indirizzo del proxy a CHRONOS_TRUSTED_PROXIES.</value></data>
<data name="Server.Error.ProxiesDisabled" xml:space="preserve"><value>Questo archivio non crea proxy.</value></data>
<data name="Server.Error.ProxyMismatch" xml:space="preserve"><value>Il proxy è stato creato da un'altra foto.</value></data>
@@ -66,7 +66,10 @@ public abstract class ArchiveDialogViewModel : LocalizedViewModel
/// <summary>Why the last dry run or apply stopped (<see cref="ArchiveFailures.Cause"/>), or what it left undone; null when nothing failed.</summary>
public string? ErrorText { get => _errorText; private set => Set(ref _errorText, value); }
/// <summary>The failure as the engine reported it (or what a <see cref="ShowMessage"/> names), for the tooltip.</summary>
/// <summary>
/// The tooltip of <see cref="ErrorText"/>: the whole sentence (the line may cut it), then the failure as the engine
/// reported it, or what a <see cref="ShowMessage"/> names.
/// </summary>
public string? ErrorToolTip { get => _errorToolTip; private set => Set(ref _errorToolTip, value); }
protected void Close() => _close(this);
@@ -74,14 +77,17 @@ public abstract class ArchiveDialogViewModel : LocalizedViewModel
protected void ShowError(Exception ex)
{
ErrorText = ArchiveFailures.Cause(ex);
ErrorToolTip = ex.Message;
ErrorToolTip = ex.Message == ErrorText ? ErrorText : ErrorText + Environment.NewLine + ex.Message;
}
/// <summary>Shows a sentence localized already, such as what an action that half succeeded left undone; <paramref name="tip"/> details it.</summary>
/// <summary>
/// Shows a sentence localized already, such as what an action that half succeeded left undone; <paramref name="tip"/>
/// details it, and repeats the sentence first (the line may cut it).
/// </summary>
protected void ShowMessage(string text, string? tip = null)
{
ErrorText = text;
ErrorToolTip = tip ?? text;
ErrorToolTip = tip is null || tip.StartsWith(text, StringComparison.Ordinal) ? tip ?? text : text + Environment.NewLine + tip;
}
protected void ClearError()
+4 -6
View File
@@ -1,4 +1,3 @@
using System.IO;
using Chronos.App.Infrastructure;
using Chronos.Core.Archiving;
using Chronos.Core.Localization;
@@ -17,12 +16,11 @@ public static class ArchiveFailures
{
ArchiveOfflineException offline => Loc.L(offline.Availability == ArchiveAvailability.ReadOnly ? "Archive.Error.ReadOnly" : "Archive.Error.Offline",
Formatting.LeftToRight(offline.Root)),
// Already one localized sentence each.
// Already one localized sentence each; TrashMismatchException is a trash entry none of whose files matches its record
// any more (FrameTrash.RestoreAsync). Other IOExceptions and InvalidDataExceptions (a sidecar of another sequence)
// carry technical text.
LeaseUnavailableException or NewerFormatException or StaleRenamePlanException or NestedRenameException or StaleDeletePlanException
or SequenceInLibraryException => ex.Message,
// A trash entry none of whose files matches its record any more (FrameTrash.RestoreAsync), told by its sentence: other
// InvalidDataExceptions, such as a sidecar of another sequence, carry technical text.
InvalidDataException when ex.Message == Loc.L("Archive.Error.TrashMismatch") => ex.Message,
or SequenceInLibraryException or TrashMismatchException => ex.Message,
UnauthorizedAccessException => Loc.L("Archive.Failed.Access"),
DatabaseException => Loc.L("Archive.Failed.Index"),
ArgumentException => Loc.L("Archive.Failed.Gone"),
@@ -171,10 +171,11 @@ public sealed class TrashDialogViewModel : ArchiveDialogViewModel
BusyText = null;
if (result.Kept.Count > 0)
{
// Files that no longer match their record stay in the trash: the entry stays, with what is left, and the line
// says how many (the tooltip names them).
// Photos that stay in the trash keep their entry, with what is left; the line says how many stayed and why,
// and the tooltip repeats it with their names.
await LoadAsync();
ShowMessage(Loc.Plural("Archive.Trash.KeptFiles", result.Kept.Count), KeptList(result.Kept));
var (text, tip) = KeptText(result.Kept);
ShowMessage(text, tip);
}
else
{
@@ -195,12 +196,28 @@ public sealed class TrashDialogViewModel : ArchiveDialogViewModel
}
}
/// <summary>The files a restore kept in the trash, one per line; past <see cref="DeletePlans.ListedLines"/> the rest are counted.</summary>
private static string KeptList(IReadOnlyList<string> kept)
/// <summary>
/// The line of what a restore kept in the trash, and its tooltip: one reason's sentence ("2 photos stayed in the trash:
/// they could not be moved back."), or how many stayed when they stayed for different reasons. The tooltip holds that
/// line, then each reason's sentence (when there are several) with its photos' names, one per line, the names counted
/// past <see cref="DeletePlans.ListedLines"/> in all.
/// </summary>
internal static (string Text, string ToolTip) KeptText(IReadOnlyList<KeptPhoto> kept)
{
var lines = kept.Take(DeletePlans.ListedLines).Select(Formatting.LeftToRight).ToList();
if (kept.Count > DeletePlans.ListedLines) lines.Add(Loc.Plural("Archive.Delete.More", kept.Count - DeletePlans.ListedLines));
return string.Join(Environment.NewLine, lines);
var groups = kept.GroupBy(k => k.Reason).OrderBy(g => g.Key).ToList();
var text = groups.Count == 1 ? KeptPhoto.Sentence(groups[0].Key, groups[0].Count(), Loc.Culture) : Loc.Plural("Archive.Trash.KeptPhotos", kept.Count);
var lines = new List<string> { text };
var listed = 0;
foreach (var group in groups)
{
if (groups.Count > 1) lines.Add(KeptPhoto.Sentence(group.Key, group.Count(), Loc.Culture));
foreach (var photo in group)
{
if (listed++ < DeletePlans.ListedLines) lines.Add(Formatting.LeftToRight(photo.Name));
}
}
if (kept.Count > DeletePlans.ListedLines) lines.Add(Loc.Plural("Archive.Trash.KeptMore", kept.Count - DeletePlans.ListedLines));
return (text, string.Join(Environment.NewLine, lines));
}
/// <summary>
+3
View File
@@ -88,6 +88,9 @@ public sealed class LeaseFile : IDisposable
}
}
/// <summary>The current holder of the lease <paramref name="name"/> in <paramref name="locksDirectory"/>, or null when free or unreadable.</summary>
public static LeaseRecord? Holder(string locksDirectory, string name) => Read(PathFor(locksDirectory, name));
/// <summary>The current holder of a lease, or null when free or unreadable.</summary>
/// <remarks>
/// Readers share read, write and delete so they never block a renewal, a release or a takeover. A renewal rewrites
+11 -2
View File
@@ -257,10 +257,19 @@ public sealed class ArchiveIndexer
/// sidecar then tells every host, so the follow-ups make them again (the images on a server,
/// <see cref="ArchiveServiceOptions.ImagesAfterScan"/>).
/// </summary>
private static SequenceManifest? Reconcile(SequencePaths sequence, SequenceManifest manifest, CancellationToken cancellationToken) =>
SequenceAdopter.Reconcile(sequence, manifest, cancellationToken) is { } reconciled
private static SequenceManifest? Reconcile(SequencePaths sequence, SequenceManifest manifest, CancellationToken cancellationToken)
{
// A restore from the trash that could not be settled yet (its entry busy) may have photos in the folder its sidecar
// does not list: they are its own, not photos to adopt, until a later writer settles it.
if (File.Exists(sequence.RestoreJournalPath))
{
Log.Info("archive", $"{sequence.RelativeFolder} is not reconciled while a restore from the trash into it waits");
return null;
}
return SequenceAdopter.Reconcile(sequence, manifest, cancellationToken) is { } reconciled
? reconciled with { Status = reconciled.Status & ~CoveringStatus }
: null;
}
/// <summary>
/// Applies <paramref name="edit"/> to the sidecar (null writes nothing) and indexes the result, both under the sequence
+11 -2
View File
@@ -12,8 +12,9 @@ namespace Chronos.Data.Archiving;
/// <summary>A verified staged file and how to name it.</summary>
/// <param name="PlannedName">Name chosen by an import plan; null = the archive's file template.</param>
/// <param name="RefuseRemoved">See <see cref="IngestRequest.RefuseRemoved"/>.</param>
internal sealed record StagedFrame(string StagedPath, string OriginalFileName, long Size, string Sha256, FrameMetadata Metadata,
DateTime? ClientCapturedAt, string? PlannedName);
DateTime? ClientCapturedAt, string? PlannedName, bool RefuseRemoved = false);
/// <summary>A sequence found or created by key (upload session, card session, or a new title).</summary>
/// <param name="PreferredFolder">Folder of an import plan; null = the archive's folder template.</param>
@@ -54,7 +55,7 @@ public sealed class ArchiveIngest : IArchiveIngest
if (!staged.Exists || staged.Length != request.Size)
throw new FileNotFoundException("The staged file is missing or has another size.", request.StagedPath);
var frame = new StagedFrame(request.StagedPath, request.OriginalFileName, request.Size, request.Sha256,
MetadataReader.Read(request.StagedPath, cancellationToken), request.ClientCapturedAt, null);
MetadataReader.Read(request.StagedPath, cancellationToken), request.ClientCapturedAt, null, request.RefuseRemoved);
Guid sequenceId;
if (request.SequenceId is { } id)
@@ -297,6 +298,7 @@ public sealed class ArchiveIngest : IArchiveIngest
var photosChanged = !inStep && await _indexer.PhotosChangedAsync(archive, sequence, cancellationToken).ConfigureAwait(false);
var before = manifest.Frames.Count;
var held = manifest.Frames.Where(f => f.Sha256 is not null).ToLookup(f => f.Sha256!, StringComparer.OrdinalIgnoreCase);
var tombstoned = (manifest.Removed ?? []).Select(r => r.Sha256).ToHashSet(StringComparer.OrdinalIgnoreCase);
// Content this batch files (new frames and repairs): another copy of it in the batch is a duplicate of that frame.
var filed = new Dictionary<string, Guid>(StringComparer.OrdinalIgnoreCase);
@@ -305,6 +307,13 @@ public sealed class ArchiveIngest : IArchiveIngest
var duplicates = new List<(Pending Pending, Guid FrameId)>();
foreach (var p in batch)
{
// Deleted from the sequence (its tombstone, and no frame holds the content) and not asked back: checked under the
// lease a deletion takes, so none lands between this check and the commit (owner review 2, R6).
if (p.Frame.RefuseRemoved && tombstoned.Contains(p.Frame.Sha256) && !held[p.Frame.Sha256].Any())
{
p.Completion.TrySetException(new PhotoRemovedException(sequenceId, p.Frame.Sha256));
continue;
}
if (filed.TryGetValue(p.Frame.Sha256, out var same))
{
duplicates.Add((p, same));
+6 -3
View File
@@ -364,7 +364,7 @@ public sealed class ArchiveService : IJobHandlerProvider
CancellationToken cancellationToken = default) =>
Trash.DeleteAsync(plan, deletedBy, progress, cancellationToken);
/// <summary>The sequences in an archive's trash, newest first.</summary>
/// <summary>The entries of an archive's trash, newest first: deleted sequences and photos deleted from sequences (<see cref="SequenceTrash.ListAsync"/>).</summary>
public async Task<IReadOnlyList<TrashEntry>> ListTrashAsync(Guid archiveId, CancellationToken cancellationToken = default) =>
await Trash.ListAsync(await LoadArchiveAsync(archiveId, cancellationToken).ConfigureAwait(false), cancellationToken).ConfigureAwait(false);
@@ -379,6 +379,9 @@ public sealed class ArchiveService : IJobHandlerProvider
/// </summary>
/// <exception cref="DirectoryNotFoundException">The entry is not in the trash.</exception>
/// <exception cref="SequenceInLibraryException">A sequence entry whose sequence is in the index already.</exception>
/// <exception cref="TrashMismatchException">No file of a photo entry matches its record; nothing moved.</exception>
/// <exception cref="InvalidDataException">A sidecar is damaged or another sequence's (the trashed folder's own, or that of
/// the sequence the photos go back into); nothing moved.</exception>
public async Task<TrashRestoreResult> RestoreEntryAsync(Guid archiveId, string entryName, JobEngine? followUps = null,
CancellationToken cancellationToken = default)
{
@@ -469,9 +472,9 @@ public sealed class ArchiveService : IJobHandlerProvider
/// <summary>
/// What a restore from the trash did: the kind of entry, the sequence restored (or the one the photos went back into),
/// and for photos how many came back, the files kept in the trash and whether they went into a new sequence.
/// and for photos how many came back, the photos kept in the trash (each with why) and whether they went into a new sequence.
/// </summary>
public sealed record TrashRestoreResult(TrashKind Kind, Sequence Sequence, int Photos, IReadOnlyList<string> Kept, bool NewSequence);
public sealed record TrashRestoreResult(TrashKind Kind, Sequence Sequence, int Photos, IReadOnlyList<KeptPhoto> Kept, bool NewSequence);
/// <summary>The fields of a sequence the user describes it with (sidecar and index).</summary>
public sealed record SequenceInfo(string? Title, IReadOnlyList<string> Tags, int Rating, string? Notes);
+137 -46
View File
@@ -20,20 +20,26 @@ public sealed partial class FrameTrash
/// </summary>
private const string RestoringFolder = ".restoring";
/// <summary>Who takes an entry's lease while settling a restore another writer of its sequence finds (<see cref="Settle"/>).</summary>
private static readonly string SettleOwner = "chronos@" + Environment.MachineName;
/// <summary>
/// Brings the photos of a trash entry back. Each file is re-hashed against its record: a file that no longer matches
/// (or is missing) stays in the trash with its photo, which is reported, as is a photo whose content an intact frame of
/// the sequence holds again. The others move back to their places (a name taken since, or one of its companions',
/// gets <c>_2</c>, <c>_3</c>…) with the same frame records, inserted by capture time, and their tombstones go; a
/// journal (<see cref="FrameRestoreJournal"/>) settles a restore a crash interrupted, and once the first file moves
/// the restore runs to its end whatever the caller cancels. When the sequence is gone (deleted, or never indexed in
/// this archive), they go into a new sequence through the archive ingest, titled and placed as theirs was; a cancel
/// there stops before the next photo, with the record kept true. The entry goes once nothing is left in it.
/// the sequence holds again, and one whose files could not move. Each kept photo says why (<see cref="KeptReason"/>).
/// The others move back to their places (a name taken since, or one of its companions', gets <c>_2</c>, <c>_3</c>…)
/// with the same frame records, inserted by capture time, and their tombstones go; a journal
/// (<see cref="FrameRestoreJournal"/>) settles a restore a crash interrupted, and once the first file moves the restore
/// runs to its end whatever the caller cancels. When the sequence is gone (deleted, or never indexed in this archive),
/// they go into a new sequence through the archive ingest, titled and placed as theirs was, the record rewritten after
/// each photo filed; a cancel there stops before the next photo. The entry goes once nothing is left in it.
/// </summary>
/// <param name="name">The entry's <see cref="TrashEntry.Name"/> (<c>photos-…</c>).</param>
/// <exception cref="DirectoryNotFoundException">The entry is not in the trash.</exception>
/// <exception cref="TrashMismatchException">No file of the entry matches its record; nothing moved.</exception>
/// <exception cref="InvalidDataException">The sequence's sidecar is damaged or belongs to another sequence; nothing moved.</exception>
/// <exception cref="LeaseUnavailableException">The entry or the sequence is busy, or the sequence holds a restore of another
/// entry that is busy and not settled yet; nothing moved.</exception>
public async Task<PhotoRestoreResult> RestoreAsync(Archive archive, string name, JobEngine? followUps = null, CancellationToken cancellationToken = default)
{
ArgumentNullException.ThrowIfNull(archive);
@@ -42,8 +48,10 @@ public sealed partial class FrameTrash
ArchiveVolume.EnsureWritable(paths);
var entryFolder = Path.Combine(Directory(paths), DiskName(name));
var recordPath = entryFolder + RecordExtension;
using (await LeaseFile.AcquireAsync(paths.LocksDirectory, SequenceTrash.EntryLease(name), _writer.Options.Owner, _writer.Options.LeaseWait,
_writer.Options.LeaseDuration, cancellationToken).ConfigureAwait(false))
// The entry's lease by the name the trash lists it under, the one an emptying and a settle take.
var lease = SequenceTrash.EntryLease(EntryPrefix + DiskName(name));
using (await LeaseFile.AcquireAsync(paths.LocksDirectory, lease, _writer.Options.Owner, _writer.Options.LeaseWait, _writer.Options.LeaseDuration,
cancellationToken).ConfigureAwait(false))
{
if (!System.IO.Directory.Exists(entryFolder) || ReadRecord(recordPath) is not { } listed)
throw new DirectoryNotFoundException($"{name} is not in the trash of {archive.Name}.");
@@ -52,7 +60,10 @@ public sealed partial class FrameTrash
// What an earlier restore a crash interrupted left half-way is settled first, so the record read next is true.
Unstage(entryFolder);
if (into is not null && File.Exists(paths.Sequence(into.Folder).RestoreJournalPath))
await _writer.UpdateAsync(paths, into.Id, into.Folder, _ => null, cancellationToken).ConfigureAwait(false);
{
await using (await _writer.LockAsync(paths, into.Id, cancellationToken).ConfigureAwait(false))
SequenceWriter.ReadLocked(paths.Sequence(into.Folder), into.Id, lease);
}
if (!System.IO.Directory.Exists(entryFolder) || ReadRecord(recordPath) is not { } record)
throw new DirectoryNotFoundException($"{name} is not in the trash of {archive.Name}.");
var (intact, kept) = await CheckAsync(entryFolder, record, cancellationToken).ConfigureAwait(false);
@@ -60,29 +71,30 @@ public sealed partial class FrameTrash
if (intact.Count == 0) throw new TrashMismatchException();
var had = into?.Status ?? SequenceStatus.None;
PhotoRestoreResult result;
List<TrashedPhoto> stays;
List<Stay> stays;
var stopped = false;
if (into is not null)
{
// The record is settled under the sequence's lease, with the sidecar.
(result, var left) = await IntoSequenceAsync(archive, paths, into, recordPath, entryFolder, intact, cancellationToken).ConfigureAwait(false);
(result, var left) = await IntoSequenceAsync(archive, paths, into, recordPath, entryFolder, lease, intact, cancellationToken).ConfigureAwait(false);
stays = [.. kept, .. left];
}
else
{
var restore = await IntoNewSequenceAsync(archive, paths, record, entryFolder, intact, cancellationToken).ConfigureAwait(false);
var restore = await IntoNewSequenceAsync(archive, paths, record, recordPath, entryFolder, intact, cancellationToken).ConfigureAwait(false);
stays = [.. kept, .. restore.Left];
Rewrite(recordPath, entryFolder, record with { Photos = stays, Leftovers = Merge(record.Leftovers, restore.Leftovers) });
Rewrite(recordPath, entryFolder, record with { Photos = [.. stays.Select(s => s.Photo)], Leftovers = Merge(record.Leftovers, restore.Leftovers) });
restore.Failure?.Throw();
stopped = restore.Stopped;
if (stopped && restore.Restored == 0) throw new OperationCanceledException(cancellationToken);
if (restore.SequenceId is not { } id) throw new IOException(Core.Localization.Loc.L("Archive.Error.TrashMismatch"));
if (restore.SequenceId is not { } id)
throw new IOException($"No photo of trash entry {Path.GetFileName(entryFolder)} could be filed into a new sequence; they stay in the trash.");
var row = await _db.Sequences.GetAsync(id, CancellationToken.None).ConfigureAwait(false)
?? throw new InvalidOperationException($"The photos of {record.SequenceTitle} were restored but their sequence is not indexed.");
result = new PhotoRestoreResult(row, restore.Restored, [], true);
}
Log.Info("archive", $"{record.SequenceTitle}: {result.Restored} photo(s) restored from the trash, {stays.Count} kept there");
var all = result with { Kept = [.. stays.Select(p => p.Frame.Original ?? Path.GetFileName(p.Frame.File))] };
var all = result with { Kept = [.. stays.Select(s => new KeptPhoto(s.Photo.Frame.Original ?? Path.GetFileName(s.Photo.Frame.File), s.Reason))] };
if (result.Restored > 0) await QueueAsync(followUps, archive, all.Sequence, had, CancellationToken.None).ConfigureAwait(false);
// A cancel stopped a restore into a new sequence before its last photo: those not done stay in the trash, listed.
if (stopped) throw new OperationCanceledException(cancellationToken);
@@ -90,15 +102,26 @@ public sealed partial class FrameTrash
}
}
/// <summary>Each photo whose files all match their record (size and SHA-256), and those that stay in the trash.</summary>
private static async Task<(List<TrashedPhoto> Intact, List<TrashedPhoto> Kept)> CheckAsync(string entryFolder, PhotoTrashRecord record,
/// <summary>A photo a restore leaves in its entry, and why.</summary>
private sealed record Stay(TrashedPhoto Photo, KeptReason Reason);
/// <summary>
/// Each photo whose files all match their record (size and SHA-256), and those that stay in the trash: a file that does
/// not match, or no file at all (its original was missing when it was deleted).
/// </summary>
private static async Task<(List<TrashedPhoto> Intact, List<Stay> Kept)> CheckAsync(string entryFolder, PhotoTrashRecord record,
CancellationToken cancellationToken)
{
var intact = new List<TrashedPhoto>();
var kept = new List<TrashedPhoto>();
var kept = new List<Stay>();
foreach (var photo in record.Photos)
{
var ok = photo.Files.Count > 0;
if (photo.Files.Count == 0)
{
kept.Add(new Stay(photo, KeptReason.NoFile));
continue;
}
var ok = true;
foreach (var file in photo.Files)
{
var path = Path.Combine(entryFolder, file.Path);
@@ -112,7 +135,8 @@ public sealed partial class FrameTrash
break;
}
}
(ok ? intact : kept).Add(photo);
if (ok) intact.Add(photo);
else kept.Add(new Stay(photo, KeptReason.Mismatch));
}
return (intact, kept);
}
@@ -122,17 +146,20 @@ public sealed partial class FrameTrash
/// move, then every move to its end, the sidecar, and the record settled (<see cref="Settle"/>). Returns what came
/// back and the photos left in the trash.
/// </summary>
private async Task<(PhotoRestoreResult, List<TrashedPhoto>)> IntoSequenceAsync(Archive archive, ArchivePaths paths, Sequence sequence, string recordPath,
string entryFolder, List<TrashedPhoto> intact, CancellationToken cancellationToken)
/// <param name="lease">The entry's lease, which the caller holds.</param>
private async Task<(PhotoRestoreResult, List<Stay>)> IntoSequenceAsync(Archive archive, ArchivePaths paths, Sequence sequence, string recordPath,
string entryFolder, string lease, List<TrashedPhoto> intact, CancellationToken cancellationToken)
{
var folder = paths.Sequence(sequence.Folder);
var left = new List<TrashedPhoto>();
var left = new List<Stay>();
int restored;
SequenceManifest updated;
var indexed = true;
await using (await _writer.LockAsync(paths, sequence.Id, cancellationToken).ConfigureAwait(false))
{
var manifest = SequenceWriter.ReadLocked(folder, sequence.Id);
var manifest = SequenceWriter.ReadLocked(folder, sequence.Id, lease);
// A restore of another entry, busy now, is not settled yet: its journal is not written over.
EnsureSettled(folder);
var inStep = await _indexer.InStepAsync(archive, folder, manifest, cancellationToken).ConfigureAwait(false);
var photosChanged = !inStep && await _indexer.PhotosChangedAsync(archive, folder, cancellationToken).ConfigureAwait(false);
var ids = manifest.Frames.Select(f => f.Id).ToHashSet();
@@ -145,7 +172,7 @@ public sealed partial class FrameTrash
// still holds the content counts; a missing or damaged one leaves the trash's copy to come back.
if (ids.Contains(photo.Frame.Id) || (photo.Frame.Sha256 is { } sha && holders[sha].Any(f => ArchiveIngest.IsIntact(folder, f))))
{
left.Add(photo);
left.Add(new Stay(photo, KeptReason.InSequence));
continue;
}
var original = photo.Files[0];
@@ -171,7 +198,7 @@ public sealed partial class FrameTrash
foreach (var (photo, frame, moves) in planned)
{
if (MoveIn(paths, folder, moves.Moves)) back.Add(frame);
else left.Add(photo);
else left.Add(new Stay(photo, KeptReason.NotMoved));
}
updated = manifest;
if (back.Count > 0)
@@ -191,11 +218,11 @@ public sealed partial class FrameTrash
{
// The sidecar lists none of them: their files go back into the entry, whose record is as it was.
Log.Warning("archive", $"{sequence.DisplayTitle}: the sidecar could not take the photos restored; they go back into the trash", ex);
Settle(folder, manifest, journal);
Settle(folder, manifest, journal, lease);
throw;
}
}
Settle(folder, updated, journal);
Settle(folder, updated, journal, lease);
restored = back.Count;
if (restored > 0)
{
@@ -246,15 +273,29 @@ public sealed partial class FrameTrash
/// read or written under the sequence's lease, has it: each photo the sidecar lists again leaves the entry's record (a
/// companion of it still in the entry stays there as one of the record's leftovers), and each file of the others that
/// reached the sequence goes back into the entry, whose record still lists it (unless the entry was emptied since: the
/// file then stays where it is, logged, for a scan to adopt). Then the journal goes. A failure (logged) keeps the
/// journal, and the next writer of the sequence settles it again.
/// file then stays where it is, logged, for a scan to adopt). Then the journal goes. All of it under the entry's lease
/// as well, which an emptying holds: an entry busy now (or a failure, logged) keeps the journal, and the next writer of
/// the sequence settles it (<see cref="EnsureSettled"/> keeps writers that would add files from going on meanwhile).
/// </summary>
internal static void Settle(SequencePaths sequence, SequenceManifest manifest, FrameRestoreJournal journal)
/// <param name="heldLease">The entry lease the caller holds already (its own restore), if any.</param>
internal static void Settle(SequencePaths sequence, SequenceManifest manifest, FrameRestoreJournal journal, string? heldLease = null)
{
var archive = sequence.Archive;
var recordPath = archive.FromRelative(journal.Record);
var entryFolder = archive.FromRelative(journal.Folder);
var listed = manifest.Frames.Select(f => f.Id).ToHashSet();
var lease = EntryLease(entryFolder);
LeaseFile? taken = null;
if (lease != heldLease)
{
taken = LeaseFile.TryAcquire(archive.LocksDirectory, lease, SettleOwner);
if (taken is null)
{
Log.Info("archive", $"A restore from the trash into {sequence.RelativeFolder} waits: its entry {Path.GetFileName(entryFolder)} is busy");
return;
}
}
using var held = taken;
try
{
var record = ReadRecord(recordPath);
@@ -299,14 +340,17 @@ public sealed partial class FrameTrash
}
}
/// <summary>Settles the restore a crash interrupted (<see cref="Settle"/>), if any; the caller holds the sequence's lease.</summary>
internal static void RecoverRestore(SequencePaths sequence, SequenceManifest manifest)
/// <summary>
/// Settles the restore a crash interrupted (<see cref="Settle"/>), if any; the caller holds the sequence's lease, and
/// <paramref name="heldLease"/> when it holds an entry's lease too.
/// </summary>
internal static void RecoverRestore(SequencePaths sequence, SequenceManifest manifest, string? heldLease = null)
{
if (!File.Exists(sequence.RestoreJournalPath)) return;
if (FrameRestoreJournal.Read(sequence) is { } journal)
{
Log.Warning("archive", $"An interrupted restore from the trash into {sequence.RelativeFolder} is settled");
Settle(sequence, manifest, journal);
Log.Warning("archive", $"An interrupted restore from the trash into {sequence.RelativeFolder} is settled unless its entry is busy");
Settle(sequence, manifest, journal, heldLease);
}
else
{
@@ -314,11 +358,28 @@ public sealed partial class FrameTrash
}
}
/// <summary>
/// Refuses to go on while a restore into <paramref name="sequence"/> is still not settled after
/// <see cref="SequenceWriter.ReadLocked"/> (its entry was busy, or a file could not move): a writer that would write
/// another journal over it, or move the sequence's folder away with it, waits until it is. The caller holds the
/// sequence's lease.
/// </summary>
/// <exception cref="LeaseUnavailableException">The restore is not settled; nothing was changed.</exception>
internal static void EnsureSettled(SequencePaths sequence)
{
if (!File.Exists(sequence.RestoreJournalPath)) return;
var lease = FrameRestoreJournal.Read(sequence) is { } journal ? EntryLease(sequence.Archive.FromRelative(journal.Folder)) : "restore";
throw new LeaseUnavailableException(lease, LeaseFile.Holder(sequence.Archive.LocksDirectory, lease));
}
/// <summary>The lease of the photo entry in <paramref name="entryFolder"/>, as a restore and an emptying of it take it.</summary>
private static string EntryLease(string entryFolder) => SequenceTrash.EntryLease(EntryPrefix + Path.GetFileName(entryFolder));
/// <summary>
/// What a restore into a new sequence did: the photos left in the trash, the companions that did not follow theirs, and
/// what stopped it before its last photo (a failure other than a file's, or a cancel).
/// </summary>
private sealed record NewSequenceRestore(Guid? SequenceId, int Restored, List<TrashedPhoto> Left, List<TrashedFile> Leftovers, ExceptionDispatchInfo? Failure,
private sealed record NewSequenceRestore(Guid? SequenceId, int Restored, List<Stay> Left, List<TrashedFile> Leftovers, ExceptionDispatchInfo? Failure,
bool Stopped);
/// <summary>
@@ -327,13 +388,15 @@ public sealed partial class FrameTrash
/// that is free, and keyed by its id, so every entry of that sequence comes back into the same new one. Each original
/// is staged inside its own entry (<see cref="RestoringFolder"/>); once it leaves the entry its commit runs to the end,
/// and when it is not filed, whatever the failure, it goes back. Companions follow their photo under a name chosen so
/// that theirs are free. A cancel, or a failure other than a file's, stops before the next photo.
/// that theirs are free. The record is rewritten after each photo filed, so a crash or a failure later never leaves it
/// listing one that is back. A cancel, or a failure other than a file's, stops before the next photo.
/// </summary>
private async Task<NewSequenceRestore> IntoNewSequenceAsync(Archive archive, ArchivePaths paths, PhotoTrashRecord record, string entryFolder,
List<TrashedPhoto> intact, CancellationToken cancellationToken)
private async Task<NewSequenceRestore> IntoNewSequenceAsync(Archive archive, ArchivePaths paths, PhotoTrashRecord record, string recordPath,
string entryFolder, List<TrashedPhoto> intact, CancellationToken cancellationToken)
{
var left = new List<TrashedPhoto>();
var left = new List<Stay>();
var leftovers = new List<TrashedFile>();
var filed = new HashSet<Guid>();
var staging = Path.Combine(entryFolder, RestoringFolder);
var key = $"trash:{record.SequenceId:N}";
var target = new SequenceTarget(key, record.SequenceTitle, record.SequenceFolder.Length > 0 ? record.SequenceFolder : null, null);
@@ -348,7 +411,7 @@ public sealed partial class FrameTrash
stopped |= failure is null && cancellationToken.IsCancellationRequested;
if (failure is not null || stopped)
{
left.Add(photo);
left.Add(new Stay(photo, KeptReason.NotMoved));
continue;
}
if (folder is null && sequenceId is { } known && await _db.Sequences.GetAsync(known, CancellationToken.None).ConfigureAwait(false) is { } row)
@@ -365,7 +428,7 @@ public sealed partial class FrameTrash
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
Log.Warning("archive", $"{original.Path} could not be staged for a restore; it stays in the trash", ex);
left.Add(photo);
left.Add(new Stay(photo, KeptReason.NotMoved));
continue;
}
IngestResult? result = null;
@@ -393,15 +456,17 @@ public sealed partial class FrameTrash
{
Log.Error("archive", $"{original.Path} stays staged in its trash entry; the next restore puts it back", ex);
}
left.Add(photo);
left.Add(new Stay(photo, result is { Duplicate: true } ? KeptReason.InSequence : KeptReason.NotMoved));
continue;
}
restored++;
filed.Add(photo.Frame.Id);
if (result is null)
{
// Filed although its commit failed afterwards (the ingest's journal completes it): where is not known, so its
// companions stay in the trash.
leftovers.AddRange(photo.Files.Skip(1));
Follow();
continue;
}
sequenceId ??= result.SequenceId;
@@ -423,11 +488,30 @@ public sealed partial class FrameTrash
leftovers.Add(companion);
}
}
Follow();
}
// A photo filed by a commit that failed afterwards may be the first in the new sequence.
if (sequenceId is null && restored > 0)
sequenceId = await _db.Sequences.FindBySessionKeyAsync(archive.Id, key, CancellationToken.None).ConfigureAwait(false);
return new NewSequenceRestore(sequenceId, restored, left, leftovers, failure, stopped);
// The record without the photos filed so far, and with the companions that did not follow; a record that cannot be
// written now (logged) is written once more when the restore ends.
void Follow()
{
try
{
Rewrite(recordPath, entryFolder, record with
{
Photos = [.. record.Photos.Where(p => !filed.Contains(p.Frame.Id))],
Leftovers = Merge(record.Leftovers, leftovers),
});
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
Log.Warning("archive", $"Trash record {recordPath} does not follow the photos restored yet; it is written again at the end", ex);
}
}
}
/// <summary>Puts back into an entry what a restore into a new sequence left staged in it (a crash, or a file that could not move back).</summary>
@@ -449,12 +533,19 @@ public sealed partial class FrameTrash
FrameRemovalJournal.RemoveEmpty(staging);
}
/// <summary>Rewrites an entry's record with what it still holds; when nothing is left in its folder, the record goes too.</summary>
/// <summary>
/// Rewrites an entry's record with what it still holds; when it lists nothing and nothing is left in its folder, the
/// record goes too. While it lists a photo, its folder stays (empty when only photos without a file are left), so the
/// entry stays listed and Empty can erase it.
/// </summary>
private static void Rewrite(string recordPath, string entryFolder, PhotoTrashRecord record)
{
FrameRemovalJournal.RemoveEmpty(entryFolder);
var done = record.Photos.Count == 0 && record.Leftovers is not { Count: > 0 };
if (done) FrameRemovalJournal.RemoveEmpty(entryFolder);
else if (System.IO.Directory.Exists(entryFolder))
foreach (var child in System.IO.Directory.EnumerateDirectories(entryFolder).ToList()) FrameRemovalJournal.RemoveEmpty(child);
// Only an entry left without a file goes; anything still in its folder keeps it listed, never erased by a restore.
if (record.Photos.Count == 0 && record.Leftovers is not { Count: > 0 } && !System.IO.Directory.Exists(entryFolder))
if (done && !System.IO.Directory.Exists(entryFolder))
{
File.Delete(recordPath);
return;
+122 -41
View File
@@ -28,8 +28,10 @@ public enum FrameDeleteBlock
WouldEmpty,
/// <summary>
/// A photo named was never hashed and its original is missing (<see cref="FrameDeletePlan.Unhashable"/>): no tombstone
/// could tell sync it was deleted, so a copy elsewhere would come back. A verify or a scan settles it first.
/// A photo named was never hashed and its original is on disk but cannot be read (another program holds it, or no
/// access; <see cref="FrameDeletePlan.Unhashable"/>): it is hashed as it goes, for the tombstone that tells sync it was
/// deleted, so it waits until it can be read. One whose original is missing is not blocked: it goes without a file and
/// without a tombstone, as there is no content left to name.
/// </summary>
Unhashable,
}
@@ -42,18 +44,52 @@ public sealed record FrameDeletePlan(Sequence Sequence, IReadOnlyList<Frame> Fra
{
public bool CanDelete => Blocked.Count == 0 && Frames.Count > 0;
/// <summary>The frames named that block the plan as <see cref="FrameDeleteBlock.Unhashable"/>.</summary>
/// <summary>The frames named that block the plan as <see cref="FrameDeleteBlock.Unhashable"/>, in index order.</summary>
public IReadOnlyList<Frame> Unhashable { get; init; } = [];
}
/// <summary>What a photo deletion did: the trash entry made and the sequence as it is now.</summary>
public sealed record FrameDeleteResult(TrashEntry Entry, Sequence Sequence);
/// <summary>Why a photo stayed in the trash when its entry was restored.</summary>
public enum KeptReason
{
/// <summary>A file of it no longer matches its record (changed, or gone from the entry).</summary>
Mismatch,
/// <summary>Its files match, but they could not be moved back (open elsewhere, no access, or the archive refused them).</summary>
NotMoved,
/// <summary>The sequence holds its content again (restored before, or filed again since): the trash keeps its copy.</summary>
InSequence,
/// <summary>Its original was already missing when it was deleted: the entry holds its record without a file.</summary>
NoFile,
}
/// <summary>A photo a restore left in the trash: its name (as on the card or upload source) and why.</summary>
public sealed record KeptPhoto(string Name, KeptReason Reason)
{
/// <summary>"2 photos stayed in the trash: they no longer match their record." in <paramref name="culture"/>.</summary>
public static string Sentence(KeptReason reason, int count, System.Globalization.CultureInfo culture) =>
Core.Localization.Loc.Text(reason switch
{
KeptReason.Mismatch => "Archive.Trash.Kept.Mismatch",
KeptReason.NotMoved => "Archive.Trash.Kept.NotMoved",
KeptReason.InSequence => "Archive.Trash.Kept.InSequence",
_ => "Archive.Trash.Kept.NoFile",
}, [count], culture);
/// <summary>One sentence per reason, in the order of <see cref="KeptReason"/>, each counting its photos.</summary>
public static IReadOnlyList<string> Sentences(IEnumerable<KeptPhoto> kept, System.Globalization.CultureInfo culture) =>
[.. kept.GroupBy(k => k.Reason).OrderBy(g => g.Key).Select(g => Sentence(g.Key, g.Count(), culture))];
}
/// <summary>
/// What a restore of photos did: the sequence they went back into (a new one when theirs is gone), how many came back, and
/// the files kept in the trash because they no longer match their record (or their photo is in the sequence again).
/// the photos kept in the trash, each with why (<see cref="KeptReason"/>).
/// </summary>
public sealed record PhotoRestoreResult(Sequence Sequence, int Restored, IReadOnlyList<string> Kept, bool NewSequence);
public sealed record PhotoRestoreResult(Sequence Sequence, int Restored, IReadOnlyList<KeptPhoto> Kept, bool NewSequence);
/// <summary>
/// Deleting photos of a sequence without erasing anything (owner review 2, R6; BRIEF §0.2 rule 8): <see cref="PlanDeleteAsync"/>
@@ -126,32 +162,40 @@ public sealed partial class FrameTrash
if (busy.Contains(sequence.Id) || busy.Contains(sequence.ArchiveId)) blocked.Add(FrameDeleteBlock.Busy);
if (frames.Count != wanted.Count || (expectedFramesVersion is { } expected && expected != sequence.FramesVersion)) blocked.Add(FrameDeleteBlock.FramesChanged);
if (frames.Count > 0 && frames.Count == all.Count) blocked.Add(FrameDeleteBlock.WouldEmpty);
// A photo without a hash is hashed as it goes; one whose original is gone as well could leave no tombstone.
var unhashable = online ? Unhashable(paths, frames) : [];
// A photo without a hash is hashed as it goes, for its tombstone: one whose original cannot be read waits.
var unhashable = online ? Unreadable(paths, frames) : [];
if (unhashable.Count > 0) blocked.Add(FrameDeleteBlock.Unhashable);
return new FrameDeletePlan(sequence, frames, frames.Sum(f => f.Size), blocked) { Unhashable = unhashable };
}
/// <summary>
/// The frames never hashed whose original is not on disk, in index order; a directory holding many of them is listed
/// once rather than asked about each (a share answers one listing faster than thousands of lookups).
/// The frames never hashed whose original is on disk but cannot be opened for reading, in index order: the first
/// <c>256</c> of them are opened (nothing is read), so a dry run of thousands stays quick on a share; one past those that
/// cannot be read refuses the deletion itself, which hashes them (an <see cref="IOException"/>, nothing moved). A missing
/// original is no reason: nothing is left to hash.
/// </summary>
private static List<Frame> Unhashable(ArchivePaths paths, IReadOnlyList<Frame> frames)
private static List<Frame> Unreadable(ArchivePaths paths, IReadOnlyList<Frame> frames)
{
const int Lookups = 64;
var names = OperatingSystem.IsWindows() || OperatingSystem.IsMacOS() ? StringComparer.OrdinalIgnoreCase : StringComparer.Ordinal;
var missing = new List<Frame>();
foreach (var directory in frames.Where(f => f.Sha256 is null).GroupBy(f => Path.GetDirectoryName(paths.FromRelative(f.RelativePath)) ?? string.Empty))
const int Probes = 256;
return [.. frames.Where(f => f.Sha256 is null).OrderBy(f => f.Index).Take(Probes).Where(f => !CanRead(paths.FromRelative(f.RelativePath)))];
}
/// <summary>Whether a file opens for reading; one that is not there counts as readable (there is nothing to read).</summary>
private static bool CanRead(string path)
{
try
{
if (directory.Count() <= Lookups || !System.IO.Directory.Exists(directory.Key))
{
missing.AddRange(directory.Where(f => !File.Exists(paths.FromRelative(f.RelativePath))));
continue;
}
var present = System.IO.Directory.EnumerateFiles(directory.Key).Select(p => Path.GetFileName(p)).ToHashSet(names);
missing.AddRange(directory.Where(f => !present.Contains(Path.GetFileName(paths.FromRelative(f.RelativePath)))));
using var handle = File.OpenHandle(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete);
return true;
}
catch (Exception ex) when (ex is FileNotFoundException or DirectoryNotFoundException)
{
return true;
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
return false;
}
return [.. missing.OrderBy(f => f.Index)];
}
/// <summary>
@@ -202,8 +246,10 @@ public sealed partial class FrameTrash
var name = EntryName(trash, id);
var entryFolder = Path.Combine(trash, name);
var recordPath = entryFolder + RecordExtension;
var photos = await TrashedAsync(folder, id, removed, kept, cancellationToken).ConfigureAwait(false);
var held = await HeldAsync(folder, kept, photos, cancellationToken).ConfigureAwait(false);
var photos = await TrashedAsync(folder, removed, kept, cancellationToken).ConfigureAwait(false);
var (held, hashed) = await HeldAsync(folder, kept, photos, cancellationToken).ConfigureAwait(false);
// The hashes computed for frames that stay go into the sidecar with them: a later deletion does not read them again.
if (hashed.Count > 0) kept = [.. kept.Select(f => hashed.TryGetValue(f.Id, out var sha) ? f with { Sha256 = sha } : f)];
var now = DateTimeOffset.UtcNow;
var record = new PhotoTrashRecord(TrashKind.Photos, id, revised.Sequence.DisplayTitle, revised.Sequence.Folder, photos,
photos.Sum(p => p.Files.Sum(f => f.Size)), now, deletedBy);
@@ -290,11 +336,12 @@ public sealed partial class FrameTrash
/// <summary>
/// The files of each frame that goes: its original (with the hash it was archived with, or one computed now for a frame
/// never hashed) under the entry's <c>originals/</c>, and an XMP companion (<c>IMG_0001.xmp</c>, <c>IMG_0001.CR2.xmp</c>)
/// no frame that stays shares. A frame whose original is missing goes without a file; one never hashed as well refuses
/// the plan (<see cref="FrameDeleteBlock.Unhashable"/>).
/// no frame that stays shares. A frame whose original is missing goes without a file: its record stays in the entry,
/// which cannot bring it back, and one never hashed leaves no tombstone either, so a copy of it elsewhere (another PC,
/// a card) is not known to be deleted.
/// </summary>
/// <exception cref="StaleDeletePlanException">A frame never hashed has lost its original since the plan.</exception>
private static async Task<List<TrashedPhoto>> TrashedAsync(SequencePaths folder, Guid sequenceId, List<SidecarFrame> removed, List<SidecarFrame> kept,
/// <exception cref="IOException">An original never hashed cannot be read; nothing moved.</exception>
private static async Task<List<TrashedPhoto>> TrashedAsync(SequencePaths folder, List<SidecarFrame> removed, List<SidecarFrame> kept,
CancellationToken cancellationToken)
{
var used = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
@@ -322,32 +369,66 @@ public sealed partial class FrameTrash
}
}
}
else if (sha is null)
{
throw new StaleDeletePlanException(sequenceId);
}
photos.Add(new TrashedPhoto(frame with { Sha256 = sha }, files, null));
}
return photos;
}
/// <summary>
/// The SHA-256 the frames that stay hold: those of their sidecar, and that of each one never hashed whose size is a
/// deleted photo's (hashed now), so a duplicate that stays keeps its content out of the tombstones.
/// The SHA-256 the frames that stay hold, so a duplicate that stays keeps its content out of the tombstones: those of
/// their sidecar, and that of each one never hashed that may hold a deleted photo's content, hashed now (returned by
/// frame, for the sidecar). The work is bounded: only a frame with a deleted photo's size is looked at, and only one
/// whose first and last 64 KiB match that photo's (<see cref="EdgesAsync"/>) is read whole, so a sequence of raw files of
/// one size reads 128 KiB a frame, not every file.
/// </summary>
private static async Task<HashSet<string>> HeldAsync(SequencePaths folder, List<SidecarFrame> kept, List<TrashedPhoto> photos,
CancellationToken cancellationToken)
private static async Task<(HashSet<string> Held, Dictionary<Guid, string> Hashed)> HeldAsync(SequencePaths folder, List<SidecarFrame> kept,
List<TrashedPhoto> photos, CancellationToken cancellationToken)
{
var held = kept.Where(f => f.Sha256 is not null).Select(f => f.Sha256!).ToHashSet(StringComparer.OrdinalIgnoreCase);
var sizes = photos.Where(p => p.Files.Count > 0).Select(p => p.Files[0].Size).ToHashSet();
foreach (var frame in kept.Where(f => f.Sha256 is null && sizes.Contains(f.Size)))
var hashed = new Dictionary<Guid, string>();
var deleted = photos.Where(p => p.Files.Count > 0).GroupBy(p => p.Files[0].Size).ToDictionary(g => g.Key, g => g.Select(p => p.Files[0].From).ToList());
var edges = new Dictionary<long, HashSet<string>>();
foreach (var frame in kept.Where(f => f.Sha256 is null && deleted.ContainsKey(f.Size)))
{
cancellationToken.ThrowIfCancellationRequested();
var file = new FileInfo(folder.File(frame.File));
if (file.Exists && sizes.Contains(file.Length))
held.Add(await ContentHash.Sha256Async(file.FullName, null, cancellationToken).ConfigureAwait(false));
if (!file.Exists || file.Length != frame.Size) continue;
if (!edges.TryGetValue(frame.Size, out var known))
{
// The deleted photos of that size are still in place: the moves come later.
known = new HashSet<string>(StringComparer.Ordinal);
foreach (var from in deleted[frame.Size]) known.Add(await EdgesAsync(folder.File(from), cancellationToken).ConfigureAwait(false));
edges[frame.Size] = known;
}
if (!known.Contains(await EdgesAsync(file.FullName, cancellationToken).ConfigureAwait(false))) continue;
var sha = await ContentHash.Sha256Async(file.FullName, null, cancellationToken).ConfigureAwait(false);
hashed[frame.Id] = sha;
held.Add(sha);
}
return held;
return (held, hashed);
}
/// <summary>
/// SHA-256 of a file's first and last 64 KiB (the whole file when smaller): two files of one size whose edges differ
/// differ in content, so only those whose edges match need a full hash.
/// </summary>
private static async Task<string> EdgesAsync(string path, CancellationToken cancellationToken)
{
const int Edge = 64 * 1024;
await using var stream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read | FileShare.Delete, 1,
FileOptions.Asynchronous | FileOptions.RandomAccess);
using var hash = System.Security.Cryptography.IncrementalHash.CreateHash(System.Security.Cryptography.HashAlgorithmName.SHA256);
var buffer = new byte[Edge];
var length = stream.Length;
var read = await stream.ReadAtLeastAsync(buffer, (int)System.Math.Min(Edge, length), throwOnEndOfStream: false, cancellationToken).ConfigureAwait(false);
hash.AppendData(buffer, 0, read);
if (length > Edge)
{
stream.Position = System.Math.Max(Edge, length - Edge);
read = await stream.ReadAtLeastAsync(buffer, (int)(length - stream.Position), throwOnEndOfStream: false, cancellationToken).ConfigureAwait(false);
hash.AppendData(buffer, 0, read);
}
return Convert.ToHexStringLower(hash.GetHashAndReset());
}
/// <summary>
+16 -1
View File
@@ -5,6 +5,10 @@ namespace Chronos.Data.Archiving;
/// Exactly one of <see cref="SequenceId"/>, <see cref="NewSequenceTitle"/> or <see cref="SessionKey"/> selects the target:
/// an existing sequence, a new one with that title, or the sequence grouped automatically by session.
/// </summary>
/// <param name="RefuseRemoved">Content the target sequence tombstoned (a photo deleted from it, none of its frames holding
/// that content) is refused with <see cref="PhotoRemovedException"/>, checked under the sequence's lease with the commit, so
/// no deletion lands in between: an upload that does not ask to bring the photo back. Otherwise filing it drops the
/// tombstone (a card imported again, a sync download, an explicit upload again).</param>
public sealed record IngestRequest(
Guid ArchiveId,
Guid? SequenceId,
@@ -15,7 +19,18 @@ public sealed record IngestRequest(
long Size,
string Sha256,
DateTime? ClientCapturedAt,
Guid? UserId);
Guid? UserId,
bool RefuseRemoved = false);
/// <summary>
/// The target sequence tombstoned the content of an <see cref="IngestRequest"/> that refuses it
/// (<see cref="IngestRequest.RefuseRemoved"/>): the photo was deleted from it. Nothing was filed; the staged file stays.
/// </summary>
public sealed class PhotoRemovedException(Guid sequenceId, string sha256)
: InvalidOperationException($"Sequence {sequenceId} tombstoned {sha256}: the photo was deleted from it and is not filed again.")
{
public Guid SequenceId { get; } = sequenceId;
}
/// <summary>
/// The committed frame; <see cref="Duplicate"/> when an intact frame of the sequence already held that SHA-256 (the file
+19 -3
View File
@@ -175,6 +175,10 @@ public sealed partial class SequenceTrash
/// <param name="progress">Sequences moved so far.</param>
/// <exception cref="StaleDeletePlanException">A sequence moved, vanished or got busy since the plan; nothing was deleted.</exception>
/// <exception cref="ArchiveOfflineException">An archive is offline; sequences before it are in the trash.</exception>
/// <exception cref="LeaseUnavailableException">A sequence, or the trash entry of a restore into it a crash interrupted, is
/// busy; sequences before it are in the trash.</exception>
/// <exception cref="InvalidDataException">The sidecar of a sequence with an interrupted journal is damaged; sequences
/// before it are in the trash.</exception>
public async Task<IReadOnlyList<TrashEntry>> DeleteAsync(DeletePlan plan, string? deletedBy = null, IProgress<int>? progress = null,
CancellationToken cancellationToken = default)
{
@@ -369,13 +373,25 @@ public sealed partial class SequenceTrash
var folder = paths.Sequence(sequence.Folder);
if (!Directory.Exists(folder.Folder) || (File.Exists(folder.ManifestPath) && SidecarId(folder.ManifestPath) != sequence.Id))
throw new StaleDeletePlanException(sequence.Id);
// A photo deletion, a restore from the trash or an ingest a crash interrupted is settled first, as any writer does
// (SequenceWriter.ReadLocked): neither its journal nor the files it moved travel into this entry, where emptying it
// would erase them while their own entry still lists them. A restore whose entry is busy keeps the folder here.
var (files, bytes) = (step.Files, step.Bytes);
if (File.Exists(folder.ManifestPath) && (File.Exists(folder.RemovalJournalPath) || File.Exists(folder.RestoreJournalPath)
|| File.Exists(folder.IngestJournalPath)))
{
SequenceWriter.ReadLocked(folder, sequence.Id);
FrameTrash.EnsureSettled(folder);
// Files came back into the folder, or left it: the record counts what moves.
(files, bytes) = Measure(folder.Folder, cancellationToken);
}
var trash = TrashDirectory(paths);
Directory.CreateDirectory(trash);
var name = EntryName(paths, sequence.Id, new HashSet<string>());
var target = Path.Combine(trash, name);
var tombstones = await _db.FrameRemovals.ListBySequenceAsync(sequence.Id, cancellationToken).ConfigureAwait(false);
var record = new TrashRecord(row, await _db.Renders.ListBySequenceAsync(sequence.Id, cancellationToken).ConfigureAwait(false), step.Files,
step.Bytes, DateTimeOffset.UtcNow, deletedBy, tombstones.Count > 0 ? SequenceMapping.ToTombstones(tombstones) : null);
var record = new TrashRecord(row, await _db.Renders.ListBySequenceAsync(sequence.Id, cancellationToken).ConfigureAwait(false), files,
bytes, DateTimeOffset.UtcNow, deletedBy, tombstones.Count > 0 ? SequenceMapping.ToTombstones(tombstones) : null);
cancellationToken.ThrowIfCancellationRequested();
// The record first: a crash before the move leaves a record without a folder, which the listing ignores.
var recordPath = target + RecordExtension;
@@ -401,7 +417,7 @@ public sealed partial class SequenceTrash
if (MoveBack(target, folder.Folder)) File.Delete(recordPath);
throw;
}
Log.Info("archive", $"{row.DisplayTitle} moved to the trash: {row.Folder} → {name} ({step.Files} files, {step.Bytes} bytes)");
Log.Info("archive", $"{row.DisplayTitle} moved to the trash: {row.Folder} → {name} ({files} files, {bytes} bytes)");
return Entry(archive.Id, name, record);
}
}
+5 -4
View File
@@ -89,17 +89,18 @@ internal sealed class SequenceWriter(ArchiveServiceOptions options)
/// <summary>
/// Reads the sidecar of a locked sequence, undoing an interrupted photo deletion (<see cref="FrameRemovalJournal"/>),
/// settling an interrupted restore from the trash (<see cref="FrameRestoreJournal"/>) and completing an interrupted
/// ingest batch first.
/// settling an interrupted restore from the trash (<see cref="FrameRestoreJournal"/>; left for later while its trash
/// entry is busy) and completing an interrupted ingest batch first.
/// </summary>
/// <param name="heldEntryLease">The trash entry lease the caller holds already (a restore of that entry), if any.</param>
/// <exception cref="InvalidDataException">The sidecar in the folder belongs to another sequence.</exception>
public static SequenceManifest ReadLocked(SequencePaths sequence, Guid sequenceId)
public static SequenceManifest ReadLocked(SequencePaths sequence, Guid sequenceId, string? heldEntryLease = null)
{
var manifest = ManifestStore.ReadSequence(sequence.ManifestPath);
if (manifest.Id != sequenceId)
throw new InvalidDataException($"{sequence.ManifestPath} belongs to sequence {manifest.Id}, not {sequenceId}.");
FrameRemovalJournal.Recover(sequence, manifest);
FrameTrash.RecoverRestore(sequence, manifest);
FrameTrash.RecoverRestore(sequence, manifest, heldEntryLease);
if (IngestJournal.Recover(sequence, manifest) is { } recovered)
{
ManifestStore.WriteSequence(sequence.ManifestPath, recovered);
+7 -3
View File
@@ -26,10 +26,14 @@ public sealed record TrashListDto(IReadOnlyList<TrashEntryDto> Entries, long Byt
/// <summary>
/// <c>POST /api/v1/archive/trash/{name}/restore</c>: the sequence the entry went back into (a new one when the photos'
/// sequence is gone: <see cref="NewSequence"/>), how many photos came back, and the files kept in the trash because they
/// no longer match their record.
/// sequence is gone: <see cref="NewSequence"/>), how many photos came back, and the photos kept in the trash: their names
/// (<see cref="Kept"/>), why each stayed (<see cref="KeptReasons"/>, in the same order: <c>mismatch</c>, its files no
/// longer match their record; <c>not_moved</c>, they could not be moved back; <c>in_sequence</c>, the sequence holds it
/// already; <c>no_file</c>, its original was missing when it was deleted) and one sentence per reason in the request's
/// language (<see cref="Message"/>; null when none stayed).
/// </summary>
public sealed record TrashRestoreDto(string Name, string Kind, Guid SequenceId, int Photos, IReadOnlyList<string> Kept, bool NewSequence);
public sealed record TrashRestoreDto(string Name, string Kind, Guid SequenceId, int Photos, IReadOnlyList<string> Kept, bool NewSequence,
IReadOnlyList<string>? KeptReasons = null, string? Message = null);
/// <summary>
/// <c>POST /api/v1/archive/trash/empty</c>: erase the entries <see cref="Names"/> (null: every entry). A dry run answers the
@@ -100,7 +100,8 @@ export async function renderTrash(root, { api, setMeta, actions, onEvent }, { ba
}
const lines = [result.kind === "sequence" ? t("Admin.Trash.RestoredSequence", entry.title)
: result.newSequence ? plural("Admin.Trash.RestoredNew", result.photos) : plural("Admin.Trash.RestoredPhotos", result.photos, entry.title)];
if (result.kept.length > 0) lines.push(plural("Admin.Trash.Kept", result.kept.length));
// The server says, in the page's language, how many photos stayed in the trash and why (one sentence per reason).
if (result.kept.length > 0 && result.message) lines.push(result.message);
toast(lines.join(" "), result.kept.length > 0);
await load();
}
+24 -8
View File
@@ -100,9 +100,11 @@ public static class LibraryEndpoints
/// (<c>frameIds</c>: none, or more than <see cref="MaxFrameIds"/>); 404 <c>not_found</c> (<c>sequence</c>, also when it
/// left the index meanwhile); 409 <c>frames_conflict</c> (the frames changed since <c>expectedFramesVersion</c>, or a
/// photo named is not the sequence's; details: the frames version now), <c>sequence_busy</c>, <c>would_empty</c> (every
/// photo would go: delete the sequence instead), <c>sidecar_mismatch</c> (the sidecar in its folder is damaged or another
/// sequence's), <c>folder_in_use</c>; 413 <c>too_large</c> (a body above <see cref="FrameDeleteBodyLimit"/>); 503
/// <c>archive_offline</c>. Publishes <c>sequenceChanged</c> <c>frames</c>; the thumbnails, proxies and analyses follow as jobs.
/// photo would go: delete the sequence instead), <c>photo_unreadable</c> (a photo never hashed cannot be read, so its
/// deletion could leave no tombstone; one whose original is missing goes without one), <c>sidecar_mismatch</c> (the
/// sidecar in its folder is damaged or another sequence's), <c>folder_in_use</c>; 413 <c>too_large</c> (a body above
/// <see cref="FrameDeleteBodyLimit"/>); 503 <c>archive_offline</c>. Publishes <c>sequenceChanged</c> <c>frames</c>; the
/// thumbnails, proxies and analyses follow as jobs.
/// </summary>
private static async Task<IResult> DeleteFramesAsync(HttpContext context, Caller caller, Guid id, FrameDeleteRequest request, ChronosDatabase db,
[FromServices] ArchiveService? archives, JobEngineHost host, AuditLog audit, EventHub events, IHostApplicationLifetime lifetime)
@@ -121,7 +123,7 @@ public static class LibraryEndpoints
{
return ApiErrors.NotFound(context, "sequence");
}
if (plan.Blocked.Count > 0) return Refused(context, plan.Blocked[0], plan.Sequence);
if (plan.Blocked.Count > 0) return Refused(context, plan);
if (request.DryRun)
return Results.Json(new FrameDeleteResultDto(id, plan.Frames.Count, plan.Bytes, plan.Sequence.FramesVersion, plan.Sequence.FrameCount, null, true),
ProtocolJson.Default.FrameDeleteResultDto);
@@ -134,7 +136,7 @@ public static class LibraryEndpoints
}
catch (ArchiveOfflineException)
{
return Refused(context, FrameDeleteBlock.Offline, plan.Sequence);
return Refused(context, plan with { Blocked = [FrameDeleteBlock.Offline] });
}
catch (ArgumentException)
{
@@ -162,12 +164,20 @@ public static class LibraryEndpoints
result.Entry.Name, false), ProtocolJson.Default.FrameDeleteResultDto);
}
private static IResult Refused(HttpContext context, FrameDeleteBlock reason, Sequence sequence) => reason switch
/// <summary>
/// A plan's first block, each with its own code: 503 <c>archive_offline</c>; 409 <c>sequence_busy</c>,
/// <c>frames_conflict</c> (details: the frames version now), <c>would_empty</c>, <c>photo_unreadable</c> (a photo never
/// hashed cannot be read, so it could leave no tombstone; the message names the first, details: how many).
/// </summary>
private static IResult Refused(HttpContext context, FrameDeletePlan plan) => plan.Blocked[0] switch
{
FrameDeleteBlock.Offline => ApiErrors.Result(context, StatusCodes.Status503ServiceUnavailable, "archive_offline"),
FrameDeleteBlock.Busy => ApiErrors.Conflict(context, "sequence_busy"),
FrameDeleteBlock.FramesChanged => ApiErrors.Conflict(context, "frames_conflict", plan.Sequence.FramesVersion.ToString(CultureInfo.InvariantCulture)),
FrameDeleteBlock.WouldEmpty => ApiErrors.Conflict(context, "would_empty"),
_ => ApiErrors.Conflict(context, "frames_conflict", sequence.FramesVersion.ToString(CultureInfo.InvariantCulture)),
FrameDeleteBlock.Unhashable => ApiErrors.Result(context, StatusCodes.Status409Conflict, "photo_unreadable",
plan.Unhashable.Count.ToString(CultureInfo.InvariantCulture), plan.Unhashable.Count > 0 ? Path.GetFileName(plan.Unhashable[0].RelativePath) : "?"),
var other => throw new ArgumentOutOfRangeException(nameof(plan), other, "A block without an error code."),
};
/// <summary>A deletion refused under the lease, said as the plan is now (<paramref name="otherwise"/> when nothing blocks it); 404 when the sequence is gone.</summary>
@@ -183,7 +193,7 @@ public static class LibraryEndpoints
{
return ApiErrors.NotFound(context, "sequence");
}
return Refused(context, again.Blocked.Count > 0 ? again.Blocked[0] : otherwise, again.Sequence);
return Refused(context, again.Blocked.Count > 0 ? again : again with { Blocked = [otherwise] });
}
/// <summary>
@@ -234,6 +244,12 @@ public static class LibraryEndpoints
items[sequence.Id] = Skip(context, sequence.Id, "sequence_busy");
continue;
}
catch (InvalidDataException)
{
// An interrupted journal could not be settled first: its sidecar is damaged (a scan rebuilds it).
items[sequence.Id] = Skip(context, sequence.Id, "sidecar_mismatch");
continue;
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
items[sequence.Id] = Skip(context, sequence.Id, "folder_in_use");
@@ -204,6 +204,11 @@ public static class SequenceEndpoints
{
return Refused(context, DeleteBlock.Busy);
}
catch (InvalidDataException)
{
// An interrupted journal could not be settled first: the sidecar is damaged (a scan rebuilds it). Nothing moved.
return Refused(context, DeleteBlock.ForeignSidecar);
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
// A file of the folder is open elsewhere: nothing moved.
+15 -2
View File
@@ -50,7 +50,9 @@ public static partial class TrashEndpoints
/// <c>sequence_in_library</c> (that sequence is in the library already), <c>sequence_busy</c>, <c>sidecar_mismatch</c>
/// (the sidecar of the sequence the photos go back into is damaged or another sequence's), <c>folder_in_use</c>; 422
/// <c>trash_mismatch</c> (no file matches its record any more, or a deleted sequence's own sidecar no longer names it);
/// 503 <c>archive_offline</c>. Publishes <c>sequenceChanged</c>: <c>created</c> for a sequence, <c>frames</c> for photos.
/// 503 <c>archive_offline</c>. The answer names the photos that stayed in the trash, why each did and one sentence per
/// reason in the request's language (<see cref="TrashRestoreDto"/>). Publishes <c>sequenceChanged</c>: <c>created</c> for
/// a sequence, <c>frames</c> for photos.
/// </summary>
private static async Task<IResult> RestoreAsync(HttpContext context, Caller caller, string name, ChronosDatabase db, ServerArchive archive,
[FromServices] ArchiveService? archives, JobEngineHost host, AuditLog audit, EventHub events, IHostApplicationLifetime lifetime)
@@ -99,10 +101,21 @@ public static partial class TrashEndpoints
await audit.WriteAsync(caller, context, "trash_restored", "sequence", sequence.Id.ToString(), $"{name} · {sequence.DisplayTitle}");
events.Publish(new SequenceChangedEvent(sequence.Id, result.Kind == TrashKind.Sequence || result.NewSequence ? "created" : "frames",
sequence.FramesVersion, sequence.SettingsVersion));
var culture = RequestCulture.Of(context);
return Results.Json(new TrashRestoreDto(name, result.Kind.ToString().ToLowerInvariant(), sequence.Id, result.Kind == TrashKind.Sequence ? sequence.FrameCount : result.Photos,
result.Kept, result.NewSequence), ProtocolJson.Default.TrashRestoreDto);
[.. result.Kept.Select(k => k.Name)], result.NewSequence, [.. result.Kept.Select(k => ReasonCode(k.Reason))],
result.Kept.Count == 0 ? null : string.Join(" ", KeptPhoto.Sentences(result.Kept, culture))), ProtocolJson.Default.TrashRestoreDto);
}
/// <summary>The code of why a photo stayed in the trash (<see cref="TrashRestoreDto.KeptReasons"/>).</summary>
private static string ReasonCode(KeptReason reason) => reason switch
{
KeptReason.Mismatch => "mismatch",
KeptReason.NotMoved => "not_moved",
KeptReason.InSequence => "in_sequence",
_ => "no_file",
};
/// <summary>
/// Empties the trash, or the entries <c>names</c> names. With <c>dryRun</c> the plan: the entries measured on disk, their
/// totals and a fingerprint, nothing touched. Otherwise the request carries that fingerprint, and the server queues an
+19 -9
View File
@@ -181,14 +181,9 @@ public sealed class UploadService(
if (!await db.Uploads.TryTransitionAsync(id, UploadState.Receiving, UploadState.Committing, cancellationToken: stopping))
return Fail<CompleteUploadResponse>(409, "upload_state");
// Deleted from the target while its bytes were on the way (the start checked before): the upload does not bring the
// photo back (owner review 2, R6).
if (!restore && await RemovedFromTargetAsync(upload, stopping))
{
logger.LogInformation("Upload {Upload} ({File}) not filed: the photo was deleted from its sequence meanwhile", id, upload.FileName);
await db.Uploads.TryTransitionAsync(id, UploadState.Committing, UploadState.Cancelled, cancellationToken: CancellationToken.None);
store.Delete(id);
return Fail<CompleteUploadResponse>(409, UploadErrors.PhotoRemoved);
}
// photo back (owner review 2, R6). Asked here to answer early; the ingest asks again under the sequence's lease, which
// a deletion takes, so one that lands after this check is caught there.
if (!restore && await RemovedFromTargetAsync(upload, stopping)) return await RemovedAsync(upload);
var user = await db.Users.GetAsync(caller.Id, stopping);
if (user is { QuotaBytes: > 0 } && user.UsedBytes + upload.Size > user.QuotaBytes && !await HeldByTargetAsync(upload, stopping))
@@ -213,7 +208,13 @@ public sealed class UploadService(
upload.Size,
upload.Sha256,
upload.CapturedAt,
caller.Id), stopping);
caller.Id,
RefuseRemoved: !restore), stopping);
}
catch (PhotoRemovedException)
{
UploadStore.TryDelete(staged);
return await RemovedAsync(upload);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
@@ -320,6 +321,15 @@ public sealed class UploadService(
return copies.Any(f => f.SequenceId == target);
}
/// <summary>An upload whose photo was deleted from its sequence meanwhile: cancelled, its part file gone, 409 <see cref="UploadErrors.PhotoRemoved"/>.</summary>
private async Task<Outcome<CompleteUploadResponse>> RemovedAsync(Upload upload)
{
logger.LogInformation("Upload {Upload} ({File}) not filed: the photo was deleted from its sequence meanwhile", upload.Id, upload.FileName);
await db.Uploads.TryTransitionAsync(upload.Id, UploadState.Committing, UploadState.Cancelled, cancellationToken: CancellationToken.None);
store.Delete(upload.Id);
return Fail<CompleteUploadResponse>(409, UploadErrors.PhotoRemoved);
}
/// <summary>The sequence the upload files into tombstoned its content (<see cref="UploadRules.RemovedFromTargetAsync"/>).</summary>
private async Task<bool> RemovedFromTargetAsync(Upload upload, CancellationToken cancellationToken)
{
@@ -171,7 +171,9 @@ public sealed class TrashDialogTests(TestContext context)
await RestoreAsync(Only(dialog.Rows));
Assert.Equal(Loc.Plural("Archive.Trash.KeptFiles", 1), dialog.ErrorText, "the sentence itself, not a generic file error");
Assert.Equal(Loc.Plural("Archive.Trash.Kept.Mismatch", 1), dialog.ErrorText, "the sentence itself, not a generic file error");
// The line may cut the sentence: the tooltip holds it whole, then the photo's name.
Assert.True(dialog.ErrorToolTip!.StartsWith(dialog.ErrorText + Environment.NewLine, StringComparison.Ordinal), dialog.ErrorToolTip);
Assert.Contains(frames[2].OriginalFileName ?? Path.GetFileName(frames[2].RelativePath), dialog.ErrorToolTip);
Assert.Equal(1, restored, "the other photo came back");
var left = Only(dialog.Rows);
@@ -200,13 +202,47 @@ public sealed class TrashDialogTests(TestContext context)
await RestoreAsync(Only(dialog.Rows));
Assert.Equal(Loc.L("Archive.Error.TrashMismatch"), dialog.ErrorText, "the sentence itself, not a generic file error");
Assert.Equal(dialog.ErrorText, dialog.ErrorToolTip, "the whole sentence, once");
Assert.Equal(0, restored);
Assert.Equal(2, Only(dialog.Rows).Entry.Frames);
Assert.Equal(2, (await fx.Db.Sequences.GetAsync(sequence.Id))!.FrameCount);
// Any other invalid data (a sidecar of another sequence, say) still reads as a file error.
// Told by its type: an IOException that merely carries the same words, or any other invalid data (a sidecar of
// another sequence, say), still reads as a file error.
Assert.Equal(Loc.L("Archive.Error.TrashMismatch"), ArchiveFailures.Cause(new TrashMismatchException()));
Assert.Equal(Loc.L("Archive.Failed.Files"), ArchiveFailures.Cause(new IOException(Loc.L("Archive.Error.TrashMismatch"))));
Assert.Equal(Loc.L("Archive.Failed.Files"), ArchiveFailures.Cause(new InvalidDataException("sequence.json belongs to sequence 42, not 43.")));
}
[Test]
public void WhatARestoreKeptSaysWhyForEachReasonAndNamesThePhotos()
{
Loc.SetCulture(CultureInfo.GetCultureInfo("en"));
// One reason: its sentence on the line, repeated whole in the tooltip above the names.
var (one, oneTip) = TrashDialogViewModel.KeptText([new KeptPhoto("IMG_0001.CR3", KeptReason.NotMoved), new KeptPhoto("IMG_0002.CR3", KeptReason.NotMoved)]);
Assert.Equal(Loc.Plural("Archive.Trash.Kept.NotMoved", 2), one);
Assert.Equal(string.Join(Environment.NewLine, one, Formatting.LeftToRight("IMG_0001.CR3"), Formatting.LeftToRight("IMG_0002.CR3")), oneTip);
// Several reasons: how many stayed on the line; each reason's sentence in the tooltip, names counted past the list.
var kept = Enumerable.Range(0, DeletePlans.ListedLines + 3)
.Select(i => new KeptPhoto($"IMG_{i:0000}.JPG", i % 3 == 0 ? KeptReason.InSequence : i % 3 == 1 ? KeptReason.Mismatch : KeptReason.NoFile)).ToList();
var (several, tip) = TrashDialogViewModel.KeptText(kept);
Assert.Equal(Loc.Plural("Archive.Trash.KeptPhotos", kept.Count), several);
var lines = tip.Split(Environment.NewLine);
Assert.Equal(several, lines[0]);
foreach (var reason in new[] { KeptReason.Mismatch, KeptReason.InSequence, KeptReason.NoFile })
Assert.Contains(KeptPhoto.Sentence(reason, kept.Count(k => k.Reason == reason), Loc.Culture), lines);
Assert.DoesNotContain(Loc.Plural("Archive.Trash.Kept.NotMoved", 1), lines);
Assert.Equal(Loc.Plural("Archive.Trash.KeptMore", 3), lines[^1]);
Assert.Equal(DeletePlans.ListedLines, lines.Count(l => l.Contains("IMG_", StringComparison.Ordinal)));
// Italian counts photos, which are feminine.
var italian = CultureInfo.GetCultureInfo("it");
Assert.Equal("1 foto è rimasta nel cestino: non corrisponde più al suo registro.", KeptPhoto.Sentence(KeptReason.Mismatch, 1, italian));
Assert.Equal("3 foto sono rimaste nel cestino: la sequenza le contiene già.", KeptPhoto.Sentence(KeptReason.InSequence, 3, italian));
Assert.Equal("+ altre 3", Loc.Text("Archive.Trash.KeptMore", [3], italian));
Assert.Equal("+ un'altra", Loc.Text("Archive.Trash.KeptMore", [1], italian));
}
private static TrashRowViewModel Only(IReadOnlyList<TrashRowViewModel> rows)
{
Assert.Count(1, rows);
@@ -258,7 +258,7 @@ public abstract class FrameTrashTests(TestContext context) : DatabaseTest(contex
var restored = await fx.Service.RestoreEntryAsync(fx.Archive.Id, deleted.Entry.Name, null, fx.Ct);
Assert.Equal(1, restored.Photos);
Assert.SequenceEqual([frames[2].OriginalFileName!], restored.Kept);
Assert.SequenceEqual([new KeptPhoto(frames[2].OriginalFileName!, KeptReason.Mismatch)], restored.Kept);
Assert.SequenceEqual([frames[0].Id, frames[1].Id, frames[3].Id], (await fx.Db.Frames.ListBySequenceAsync(sequence.Id)).Select(f => f.Id));
Assert.SequenceEqual([frames[2].Sha256], (await fx.Db.FrameRemovals.ListBySequenceAsync(sequence.Id)).Select(r => r.Sha256));
var left = Assert.Contains(await fx.Service.ListTrashAsync(fx.Archive.Id, fx.Ct), e => e.Name == deleted.Entry.Name);
@@ -705,7 +705,7 @@ public abstract class FrameTrashTests(TestContext context) : DatabaseTest(contex
var restored = await fx.Service.RestoreEntryAsync(fx.Archive.Id, deleted.Entry.Name, null, fx.Ct);
Assert.Equal(0, restored.Photos);
Assert.SequenceEqual([frames[1].OriginalFileName!], restored.Kept);
Assert.SequenceEqual([new KeptPhoto(frames[1].OriginalFileName!, KeptReason.NotMoved)], restored.Kept);
Assert.FileMissing(photo);
Assert.FileMissing(folder.RestoreJournalPath);
var entryFolder = EntryFolder(fx, deleted.Entry);
@@ -806,13 +806,92 @@ public abstract class FrameTrashTests(TestContext context) : DatabaseTest(contex
Assert.True(await fx.Db.FrameRemovals.IsRemovedAsync(sequence.Id, sha));
Assert.Equal(sha, Sha(Path.Combine(EntryFolder(fx, deleted.Entry), "originals", Path.GetFileName(frames[1].RelativePath))));
// One whose original is gone as well could leave none: the plan refuses it, and says which.
// One whose original is gone as well has no content left to name: it goes without a file and without a tombstone,
// and its record stays in the trash, listed until Empty erases it (nothing can bring it back).
var third = Sha(fx.Paths.FromRelative(frames[2].RelativePath));
File.Delete(fx.Paths.FromRelative(frames[3].RelativePath));
var row = (await fx.Db.Sequences.GetAsync(sequence.Id))!;
var refused = await fx.Service.FrameTrash.PlanDeleteAsync(sequence.Id, [frames[2].Id, frames[3].Id], row.FramesVersion, fx.Ct);
Assert.SequenceEqual([FrameDeleteBlock.Unhashable], refused.Blocked);
Assert.SequenceEqual([frames[3].Id], refused.Unhashable.Select(f => f.Id));
await Assert.ThrowsAsync<StaleDeletePlanException>(() => fx.Service.FrameTrash.DeleteAsync(refused, null, null, fx.Ct));
var gone = await DeleteAsync(fx, (await fx.Db.Sequences.GetAsync(sequence.Id))!, frames[2], frames[3]);
Assert.Equal((2, 1), (gone.Entry.Frames, gone.Entry.Files));
Assert.SequenceEqual(new[] { sha, third }.Order(), Assert.NotNull(ManifestStore.ReadSequence(folder.ManifestPath).Removed).Select(r => r.Sha256).Order());
Assert.SequenceEqual([frames[0].Id], ManifestStore.ReadSequence(folder.ManifestPath).Frames.Select(f => f.Id));
var restored = await fx.Service.RestoreEntryAsync(fx.Archive.Id, gone.Entry.Name, null, fx.Ct);
Assert.Equal(1, restored.Photos);
Assert.SequenceEqual([KeptReason.NoFile], restored.Kept.Select(k => k.Reason));
var left = Assert.Contains(await fx.Service.ListTrashAsync(fx.Archive.Id, fx.Ct), e => e.Name == gone.Entry.Name);
Assert.Equal((1, 0), (left.Frames, left.Files));
await Assert.ThrowsAsync<TrashMismatchException>(() => fx.Service.RestoreEntryAsync(fx.Archive.Id, gone.Entry.Name, null, fx.Ct));
var plan = await fx.Service.PlanEmptyTrashAsync(fx.Archive.Id, [gone.Entry.Name], fx.Ct);
var items = plan.Entries.Select(e => new EmptyTrashItem(e.Name, SequenceTrash.EntryFingerprint(e))).ToList();
var emptied = await fx.Service.Trash.EmptyAsync(fx.Archive, new EmptyTrashParameters(items, plan.Fingerprint), 0, (_, _) => ValueTask.CompletedTask, fx.Ct);
Assert.Equal(1, emptied.Erased);
Assert.DoesNotContain(gone.Entry.Name, (await fx.Service.ListTrashAsync(fx.Archive.Id, fx.Ct)).Select(e => e.Name));
}
[Test]
[WindowsOnly]
public async Task APhotoNeverHashedThatCannotBeReadWaitsUntilItCan()
{
var fx = await FixtureAsync();
var (sequence, frames) = await AdoptedAsync(fx, "Shoot", 3);
var original = fx.Paths.FromRelative(frames[1].RelativePath);
var sha = Sha(original);
// Another program holds it: it could not be hashed for its tombstone, so the plan refuses it and says which.
using (new FileStream(original, FileMode.Open, FileAccess.Read, FileShare.None))
{
var refused = await fx.Service.FrameTrash.PlanDeleteAsync(sequence.Id, [frames[1].Id], sequence.FramesVersion, fx.Ct);
Assert.SequenceEqual([FrameDeleteBlock.Unhashable], refused.Blocked);
Assert.SequenceEqual([frames[1].Id], refused.Unhashable.Select(f => f.Id));
await Assert.ThrowsAsync<StaleDeletePlanException>(() => fx.Service.FrameTrash.DeleteAsync(refused, null, null, fx.Ct));
}
Assert.Equal(sha, Sha(original));
await DeleteAsync(fx, sequence, frames[1]);
Assert.True(await fx.Db.FrameRemovals.IsRemovedAsync(sequence.Id, sha));
}
[Test]
public async Task AFrameThatStaysHashedToFindATwinKeepsItsHashAndOnlyLikelyTwinsAreReadWhole()
{
var fx = await FixtureAsync();
var shoot = Path.Combine(fx.Archive.RootPath, "Raw");
Directory.CreateDirectory(shoot);
// Originals of one size, as uncompressed raw files are: only one whose first and last 64 KiB match a deleted photo's
// is read whole.
const int Size = 300_000;
static byte[] Padded(int seed)
{
var bytes = new byte[Size];
Photos.Jpeg(Start, seed).CopyTo(bytes, 0);
return bytes;
}
var deleted = Padded(1);
var near = (byte[])deleted.Clone();
near[Size / 2] ^= 0xFF;
File.WriteAllBytes(Path.Combine(shoot, "IMG_0001.JPG"), Padded(2));
File.WriteAllBytes(Path.Combine(shoot, "IMG_0002.JPG"), deleted);
File.WriteAllBytes(Path.Combine(shoot, "IMG_0003.JPG"), deleted);
File.WriteAllBytes(Path.Combine(shoot, "IMG_0004.JPG"), near);
await fx.ScanAsync();
var id = (await fx.Db.Sequences.ListLocationsAsync(fx.Archive.Id)).Single(l => l.Folder == "Raw").Id;
var frames = await fx.Db.Frames.ListBySequenceAsync(id);
Assert.True(frames.All(f => f.Sha256 is null), "adopted, not verified");
await DeleteAsync(fx, (await fx.Db.Sequences.GetAsync(id))!, frames.Single(f => f.RelativePath.EndsWith("IMG_0002.JPG", StringComparison.Ordinal)));
var sidecar = ManifestStore.ReadSequence(fx.Paths.Sequence("Raw").ManifestPath);
Assert.Null(sidecar.Removed, "its twin stays");
string? InSidecar(string name) => sidecar.Frames.Single(f => f.File.EndsWith(name, StringComparison.Ordinal)).Sha256;
Assert.Equal(Convert.ToHexStringLower(SHA256.HashData(deleted)), InSidecar("IMG_0003.JPG"));
Assert.Equal(Convert.ToHexStringLower(SHA256.HashData(near)), InSidecar("IMG_0004.JPG"));
Assert.Null(InSidecar("IMG_0001.JPG"), "its first 64 KiB differ: never read whole");
// The index holds the hashes too, so a later deletion does not read those files again.
var now = await fx.Db.Frames.ListBySequenceAsync(id);
Assert.Equal(InSidecar("IMG_0003.JPG"), now.Single(f => f.RelativePath.EndsWith("IMG_0003.JPG", StringComparison.Ordinal)).Sha256);
Assert.Null(now.Single(f => f.RelativePath.EndsWith("IMG_0001.JPG", StringComparison.Ordinal)).Sha256);
}
[Test]
@@ -974,6 +1053,148 @@ public abstract class FrameTrashTests(TestContext context) : DatabaseTest(contex
Assert.FileExists(Path.Combine(SequenceTrash.TrashDirectory(fx.Paths), names[1], "added.bin"));
}
[Test]
public async Task ASequenceDeletedAfterAnInterruptedPhotoDeletionTakesNoHalfEntryAlong()
{
var fx = await FixtureAsync();
var (sequence, frames) = await UploadedAsync(fx, 3);
var folder = fx.Paths.Sequence(sequence.Folder);
// As a photo deletion leaves it when the process dies after its first move: journal, pending record, one file moved.
var name = $"{sequence.Id:N}-20261027120000";
var entry = Path.Combine(FrameTrash.Directory(fx.Paths), name);
var record = entry + ".json";
var sidecar = ManifestStore.ReadSequence(folder.ManifestPath).Frames.Take(2).ToList();
var files = sidecar.Select(f => new TrashedFile(f.File, "originals/" + Path.GetFileName(f.File), f.Size, f.Sha256)).ToList();
Directory.CreateDirectory(Path.Combine(entry, "originals"));
File.WriteAllText(FrameRemovalJournal.PendingRecord(record), JsonSerializer.Serialize(new PhotoTrashRecord(TrashKind.Photos, sequence.Id, sequence.DisplayTitle,
sequence.Folder, [.. sidecar.Zip(files, (f, t) => new TrashedPhoto(f, [t], null))], files.Sum(f => f.Size), DateTimeOffset.UtcNow, "alberto"),
TrashJson.Default.PhotoTrashRecord));
var moves = files.Select(f => new FrameRemovalMove(f.From, fx.Paths.ToRelative(Path.Combine(entry, f.Path)))).ToList();
FrameRemovalJournal.Write(folder, new FrameRemovalJournal(fx.Paths.ToRelative(record), fx.Paths.ToRelative(entry), [.. sidecar.Select(f => f.Id)], moves));
File.Move(folder.File(moves[0].From), fx.Paths.FromRelative(moves[0].To));
await DeleteSequenceAsync(fx, sequence.Id);
// The deletion was undone first: every photo went with its sequence, and neither its journal nor half an entry is left.
var deleted = Assert.Contains(await fx.Service.ListTrashAsync(fx.Archive.Id, fx.Ct), e => e.Kind == TrashKind.Sequence);
var trashed = fx.Paths.Sequence(".chronos/trash/" + deleted.Name);
Assert.FileMissing(trashed.RemovalJournalPath);
foreach (var frame in frames) Assert.Equal(frame.Sha256, Sha(trashed.File(frame.RelativePath[(sequence.Folder.Length + 1)..])));
Assert.False(Directory.Exists(entry));
Assert.FileMissing(FrameRemovalJournal.PendingRecord(record));
Assert.DoesNotContain(TrashKind.Photos, (await fx.Service.ListTrashAsync(fx.Archive.Id, fx.Ct)).Select(e => e.Kind));
}
[Test]
public async Task ARestoreWhoseEntryIsBusyWaitsAndNothingGoesOnOverIt()
{
var fx = await FixtureAsync();
var (sequence, frames) = await UploadedAsync(fx, 5);
var folder = fx.Paths.Sequence(sequence.Folder);
var deleted = await DeleteAsync(fx, sequence, frames[1], frames[2]);
var other = await DeleteAsync(fx, (await fx.Db.Sequences.GetAsync(sequence.Id))!, frames[3]);
// A restore died after moving one original back, before the sidecar; the entry is being emptied meanwhile.
var journal = RestoreJournal(fx, deleted.Entry);
journal.Write(folder);
var moved = journal.Photos[0].Moves[0];
File.Move(fx.Paths.FromRelative(moved.To), folder.File(moved.From));
using (Assert.NotNull(LeaseFile.TryAcquire(fx.Paths.LocksDirectory, SequenceTrash.EntryLease(deleted.Entry.Name), "other")))
{
// Another writer leaves the journal for later: it never moves a file into an entry being erased.
await fx.Service.SetSequenceInfoAsync(sequence.Id, "Harbour", [], 0, null, fx.Ct);
Assert.FileExists(folder.RestoreJournalPath);
Assert.FileExists(folder.File(moved.From));
Assert.FileMissing(fx.Paths.FromRelative(moved.To));
// A scan does not adopt the photo that restore moved in.
File.SetLastWriteTimeUtc(folder.File(moved.From), DateTime.UtcNow);
await fx.ScanAsync();
Assert.Count(2, await fx.Db.Frames.ListBySequenceAsync(sequence.Id));
// Neither a restore of another entry into the sequence nor the sequence's deletion goes on over it.
await Assert.ThrowsAsync<LeaseUnavailableException>(() => fx.Service.RestoreEntryAsync(fx.Archive.Id, other.Entry.Name, null, fx.Ct));
await Assert.ThrowsAsync<LeaseUnavailableException>(() => DeleteSequenceAsync(fx, sequence.Id));
Assert.FileExists(folder.RestoreJournalPath);
Assert.FileExists(folder.ManifestPath);
}
// Once the entry is free, the next writer settles it: the original goes back into its entry, and both entries come back.
await fx.Service.SetSequenceInfoAsync(sequence.Id, "Harbour", [], 0, null, fx.Ct);
Assert.FileMissing(folder.RestoreJournalPath);
Assert.Equal(frames[1].Sha256, Sha(fx.Paths.FromRelative(moved.To)));
Assert.FileMissing(folder.File(moved.From));
Assert.Equal(2, (await fx.Service.RestoreEntryAsync(fx.Archive.Id, deleted.Entry.Name, null, fx.Ct)).Photos);
Assert.Equal(1, (await fx.Service.RestoreEntryAsync(fx.Archive.Id, other.Entry.Name, null, fx.Ct)).Photos);
Assert.SequenceEqual(frames.Select(f => f.Id), (await fx.Db.Frames.ListBySequenceAsync(sequence.Id)).Select(f => f.Id));
}
[Test]
public async Task ARestoreIntoANewSequenceStoppedBetweenPhotosKeepsItsRecordTrue()
{
var fx = await FixtureAsync();
var (sequence, frames) = await UploadedAsync(fx, 4);
var first = await DeleteAsync(fx, sequence, frames[1]);
var second = await DeleteAsync(fx, (await fx.Db.Sequences.GetAsync(sequence.Id))!, frames[2], frames[3]);
await DeleteSequenceAsync(fx, sequence.Id);
var made = await fx.Service.RestoreEntryAsync(fx.Archive.Id, first.Entry.Name, null, fx.Ct);
Assert.True(made.NewSequence);
var entryFolder = EntryFolder(fx, second.Entry);
var staged = Path.Combine(entryFolder, ".restoring", "originals", Path.GetFileName(frames[2].RelativePath));
using var cancel = CancellationTokenSource.CreateLinkedTokenSource(fx.Ct);
// The new sequence is held while the first photo waits to be filed into it, and the restore is cancelled meanwhile.
Task restore;
using (Assert.NotNull(LeaseFile.TryAcquire(fx.Paths.LocksDirectory, $"seq-{made.Sequence.Id:N}", "other")))
{
restore = fx.Service.RestoreEntryAsync(fx.Archive.Id, second.Entry.Name, null, cancel.Token);
var until = DateTime.UtcNow + TimeSpan.FromSeconds(TestTiming.Budget(20));
while (!File.Exists(staged))
{
Assert.True(DateTime.UtcNow < until, "the first photo was never staged");
await Task.Delay(10, fx.Ct);
}
cancel.Cancel();
}
await Assert.ThrowsAsync<OperationCanceledException>(() => restore);
// The first photo is filed, the second stays, and the record lists exactly what is left.
Assert.SequenceEqual([frames[1].Sha256, frames[2].Sha256], (await fx.Db.Frames.ListBySequenceAsync(made.Sequence.Id)).Select(f => f.Sha256));
var record = Assert.NotNull(FrameTrash.ReadEntry(fx.Paths, second.Entry.Name));
Assert.SequenceEqual([frames[3].Id], record.Photos.Select(p => p.Frame.Id));
Assert.Equal(frames[3].Size, record.Bytes);
Assert.Equal(frames[3].Sha256, Sha(Path.Combine(entryFolder, "originals", Path.GetFileName(frames[3].RelativePath))));
Assert.False(Directory.Exists(Path.Combine(entryFolder, ".restoring")));
var rest = await fx.Service.RestoreEntryAsync(fx.Archive.Id, second.Entry.Name, null, fx.Ct);
Assert.Equal((1, made.Sequence.Id), (rest.Photos, rest.Sequence.Id));
Assert.Empty(rest.Kept);
Assert.False(Directory.Exists(entryFolder));
}
[Test]
public async Task AnIngestThatRefusesRemovedContentChecksTheTombstonesUnderTheLease()
{
var fx = await FixtureAsync();
var (sequence, frames) = await UploadedAsync(fx, 3);
await DeleteAsync(fx, sequence, frames[1]);
// The same photo staged again (an upload whose completion checked the index before the deletion landed).
var (path, size, sha) = fx.Stage(Start.AddSeconds(10), 101);
Assert.Equal(frames[1].Sha256, sha);
var request = new IngestRequest(fx.Archive.Id, sequence.Id, null, null, path, frames[1].OriginalFileName!, size, sha, null, null, RefuseRemoved: true);
var refused = await Assert.ThrowsAsync<PhotoRemovedException>(() => fx.Service.Ingest.CommitAsync(request, fx.Ct));
Assert.Equal(sequence.Id, refused.SequenceId);
Assert.FileExists(path);
Assert.Count(2, ManifestStore.ReadSequence(fx.Paths.Sequence(sequence.Folder).ManifestPath).Frames);
Assert.True(await fx.Db.FrameRemovals.IsRemovedAsync(sequence.Id, sha));
// Asked back, it is filed and its tombstone goes.
var back = await fx.Service.Ingest.CommitAsync(request with { RefuseRemoved = false }, fx.Ct);
Assert.False(back.Duplicate);
Assert.False(await fx.Db.FrameRemovals.IsRemovedAsync(sequence.Id, sha));
Assert.Count(3, await fx.Db.Frames.ListBySequenceAsync(sequence.Id));
}
/// <summary>An entry of originals a conversion replaced (a later phase), its one photo replaced by <paramref name="replacement"/>.</summary>
private static string WriteReplaced(ArchiveFixture fx, Sequence sequence, string stamp, string replacement)
{
@@ -4,6 +4,7 @@ using System.Net.Http.Json;
using Microsoft.Extensions.DependencyInjection;
using Chronos.Core.Codecs.Jpeg;
using Chronos.Core.Model;
using Chronos.Data.Archiving;
using Chronos.Protocol;
using Chronos.Server.Storage;
using Chronos.Testing;
@@ -274,6 +275,73 @@ public sealed class LibraryTests(TestContext context)
Assert.Null(await server.Database.Sequences.GetAsync(t));
}
[Test]
public async Task ARestoreSaysWhichPhotosStayedAndWhy()
{
await using var server = await TestServer.StartAsync(context, services: ArchiveModule.Register);
var (editor, _, _) = await server.UserAsync("alberto", "editor");
var admin = await server.AdminAsync();
Guid id = default;
for (var i = 0; i < 4; i++) id = (await Uploads.UploadAsync(editor, Photo(120 + i), $"IMG_{i + 1:0000}.JPG", sessionKey: "kept")).SequenceId;
await UntilIdleAsync(server);
var frames = await server.Database.Frames.ListBySequenceAsync(id);
string name;
using (var deleted = await editor.PostAsJsonAsync($"/api/v1/sequences/{id}/frames/delete", new FrameDeleteRequest([frames[1].Id, frames[2].Id], null, false),
ProtocolJson.Default.FrameDeleteRequest))
name = Assert.NotNull((await deleted.ReadAsync(ProtocolJson.Default.FrameDeleteResultDto)).TrashEntry);
await UntilIdleAsync(server);
var trashed = Path.Combine(PhotoEntryFolder(server, name), "originals", Path.GetFileName(frames[2].RelativePath));
var bytes = await File.ReadAllBytesAsync(trashed, context.Cancellation);
bytes[^10] ^= 0x01;
await File.WriteAllBytesAsync(trashed, bytes, context.Cancellation);
using var restored = await admin.PostAsync($"/api/v1/archive/trash/{name}/restore", null);
var result = await restored.ReadAsync(ProtocolJson.Default.TrashRestoreDto);
Assert.Equal(1, result.Photos);
Assert.SequenceEqual([frames[2].OriginalFileName!], result.Kept);
Assert.SequenceEqual(["mismatch"], Assert.NotNull(result.KeptReasons));
Assert.Equal(KeptPhoto.Sentence(KeptReason.Mismatch, 1, System.Globalization.CultureInfo.GetCultureInfo("en")), result.Message);
}
[Test]
[WindowsOnly]
public async Task APhotoNeverHashedIsDeletedUnlessItCannotBeRead()
{
await using var server = await TestServer.StartAsync(context, services: ArchiveModule.Register);
var admin = await server.AdminAsync();
// Photos copied into the archive by hand and scanned: adopted, never hashed.
var shoot = Path.Combine(server.Options.ArchivePath, "Shoot");
Directory.CreateDirectory(shoot);
for (var i = 0; i < 3; i++) await File.WriteAllBytesAsync(Path.Combine(shoot, $"IMG_{i + 1:0000}.JPG"), Photo(130 + i), context.Cancellation);
var archives = server.Services.GetRequiredService<ArchiveService>();
await archives.Indexer.ScanAsync(server.Services.GetRequiredService<ServerArchive>().Current, ScanMode.Incremental, null, _ => ValueTask.CompletedTask,
context.Cancellation);
await UntilIdleAsync(server);
var id = (await server.Database.Sequences.ListLocationsAsync(server.Services.GetRequiredService<ServerArchive>().Id)).Single(l => l.Folder == "Shoot").Id;
var frames = await server.Database.Frames.ListBySequenceAsync(id);
Assert.True(frames.All(f => f.Sha256 is null), "adopted, not verified");
var url = $"/api/v1/sequences/{id}/frames/delete";
// Another program holds one: it cannot be hashed for its tombstone, and the answer names it (no frames_conflict loop).
var held = Path.Combine(server.Options.ArchivePath, frames[1].RelativePath);
using (new FileStream(held, FileMode.Open, FileAccess.Read, FileShare.None))
{
using var refused = await admin.PostAsJsonAsync(url, new FrameDeleteRequest([frames[1].Id], null, false), ProtocolJson.Default.FrameDeleteRequest);
var error = await refused.ExpectErrorAsync(HttpStatusCode.Conflict, "photo_unreadable");
Assert.Equal("1", error.Details);
Assert.Contains("IMG_0002.JPG", error.Message);
}
Assert.FileExists(held);
// One whose original is gone has nothing left to hash: it goes, without a tombstone.
File.Delete(Path.Combine(server.Options.ArchivePath, frames[2].RelativePath));
using (var gone = await admin.PostAsJsonAsync(url, new FrameDeleteRequest([frames[2].Id], null, false), ProtocolJson.Default.FrameDeleteRequest))
Assert.NotNull((await gone.ReadAsync(ProtocolJson.Default.FrameDeleteResultDto)).TrashEntry);
Assert.Empty(await server.Database.FrameRemovals.ListBySequenceAsync(id));
Assert.SequenceEqual([frames[0].Id, frames[1].Id], (await server.Database.Frames.ListBySequenceAsync(id)).Select(f => f.Id));
}
[Test]
public async Task APhotoDeleteTakesAsManyIdsAsItsLimit()
{
@@ -8,6 +8,7 @@ using Microsoft.Extensions.DependencyInjection;
using Chronos.Core.Codecs.Jpeg;
using Chronos.Core.Localization;
using Chronos.Core.Model;
using Chronos.Data.Archiving;
using Chronos.Protocol;
using Chronos.Server.Storage;
using Chronos.Testing;
@@ -228,6 +229,42 @@ public sealed class BrowserUploadTests(TestContext context)
Assert.SequenceEqual([keeper.FrameId], (await server.Database.Frames.ListBySequenceAsync(sequence.Id)).Select(f => f.Id));
}
[Test]
public async Task APhotoDeletedBetweenTheCompletionsCheckAndItsCommitIsNotFiled()
{
// The completion asks the index first; a deletion that lands after that and before the commit is caught by the
// ingest itself, which asks the sidecar's tombstones under the lease the deletion takes.
GatedIngest? gate = null;
await using var server = await TestServer.StartAsync(context, o => o.ChunkSizeBytes = Chunk, s =>
{
ArchiveModule.Register(s);
s.AddSingleton<IArchiveIngest>(sp => gate = new GatedIngest(sp.GetRequiredService<ArchiveService>().Ingest));
});
var browser = await Browser.SignInAsync(server);
var sequence = await CreateAsync(browser, "Roof");
var photo = Photo(17);
var sha = Uploads.Sha(photo);
var keeper = await UploadAsync(browser, sequence.Id, "IMG_0000.JPG", Photo(18));
var first = await UploadAsync(browser, sequence.Id, "IMG_0001.JPG", photo);
await UntilIdleAsync(server);
var frame = Assert.NotNull(await server.Database.Frames.GetAsync(first.FrameId));
File.Delete(Path.Combine(server.Options.ArchivePath, frame.RelativePath));
var repair = await StartAsync(browser, sequence.Id, "IMG_0001.JPG", photo, HttpStatusCode.Created);
var id = Assert.NotNull(repair.UploadId);
await SendChunksAsync(browser, id, photo);
Assert.NotNull(gate).BeforeCommit = () => DeleteAsync(server, browser, sequence.Id, first.FrameId);
using (var complete = await CompleteAsync(browser, id))
await complete.ExpectErrorAsync(HttpStatusCode.Conflict, UploadErrors.PhotoRemoved);
Assert.SequenceEqual([keeper.FrameId], (await server.Database.Frames.ListBySequenceAsync(sequence.Id)).Select(f => f.Id));
Assert.True(await server.Database.FrameRemovals.IsRemovedAsync(sequence.Id, sha), "the tombstone stays");
Assert.False(server.Services.GetRequiredService<UploadStore>().Exists(id), "the partial file is gone");
Assert.FileMissing(Path.Combine(server.Options.StagingPath, id.ToString("N") + ".jpg"));
using (var late = await CompleteAsync(browser, id))
await late.ExpectErrorAsync(HttpStatusCode.Conflict, "upload_state");
}
[Test]
public async Task AStaleTombstoneBesideAFrameOfTheSameContentRefusesNoRepair()
{
@@ -426,6 +463,22 @@ public sealed class BrowserUploadTests(TestContext context)
}
}
/// <summary>The archive's ingest with a step run once before the next commit (what lands between the completion's checks and it).</summary>
private sealed class GatedIngest(IArchiveIngest inner) : IArchiveIngest
{
public Func<Task>? BeforeCommit { get; set; }
public async Task<IngestResult> CommitAsync(IngestRequest request, CancellationToken cancellationToken = default)
{
if (BeforeCommit is { } step)
{
BeforeCommit = null;
await step();
}
return await inner.CommitAsync(request, cancellationToken);
}
}
/// <summary>A JPEG of a few chunks: noise does not compress below the chunk size.</summary>
private static byte[] Photo(int seed)
{