diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 408531e4..d0ce5f6b 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -215,6 +215,14 @@ match. A 1.1 server works as before, without Integrate; a 1.2 server adds the ph again: Integrate is offered and new photos move again. - Upload of a sequence kept apart from (or unlinked from) a server sequence makes a new server sequence instead of adding the photos to the one kept apart and linking it again. + - A merge resumed after a failed copy of a render no longer replaces a render of the kept sequence made meanwhile + under the same name: the other's render is copied beside it under a free name. + - A photo a merge keeps in two versions because the two only matched by size and time is listed under Two versions, + also at a whole second, and Main renders only the kept version. + - An Integrate no longer trashes a losing version that became the only copy while it ran: it stops and plans again. + - Download originals of a server sequence unlinked or kept apart from its earlier download makes a new local + sequence instead of filling the old one again, re-linking it and overwriting its details. + - An upload retried after its pair was kept apart makes a new server sequence instead of linking the old one again. ## [1.1.1] — 2026-11-16 diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md index 6d7d268b..51cc9ddd 100644 --- a/docs/DECISIONS.md +++ b/docs/DECISIONS.md @@ -1014,3 +1014,8 @@ One line per decision: date · decision · alternative · reason. | 2026-11-17 | The Integrate's Projects stage counts a project as kept for good only for a refusal a retry would meet again (`TransplantFormatException`, `InvalidDataException`, `ProjectException`, a non-transient server refusal, no project home on either side); any other error (a home out of reach, a network error) fails the job with the journal and the copy's step kept, so a retry finishes that copy (#73) | Count every error as a kept project | One transient error deleted the journal, and the next Integrate copied the half-made copy back as a new project on each side | | 2026-11-17 | A pair's "deleted on server" mark is cleared whenever the server lists that sequence again (whatever the hit count) and whenever a link to that very sequence is made (upload, download, Integrate, Link, auto-link) (#73) | Clear only a single hit | A pair at two hits stayed "deleted on server" for good, even linked again, and its new photos never moved | | 2026-11-17 | An upload without a target of a local sequence kept apart from a sequence of that server goes under a session key of the job (`chronos-sync::`, the same on a retry) instead of the sequence's own; a link made over a Kept apart pair moves it to Linked (#73) | Refuse the upload | The server files the sequence's own key into the sequence the user kept separate, and re-linked it without asking; the store and the link disagreed | +| 2026-11-18 | A merge's Renders stage never deletes or overwrites what sits at a journaled render target: a file it keeps only when it is the render itself (its recorded SHA-256, else the source's content), a folder only when every file in it is the source's file of that name; anything else, or a path another render of the kept sequence names, makes the copy take a new free name, journaled before the copy (#73) | Refuse while a render row names the path | After a failed run the name was free again, and a new render of the kept sequence (same date, title and preset) went for good, bypassing the trash; a free name keeps both | +| 2026-11-18 | A merge's Verify stage writes the hash it read for each kept frame the index had none for (sidecar and index, under the lease, only while the file keeps its stamp); the Details stage leaves the versions the verify filed out of Main as the plan leaves its own (on the plan's payload, else on the kept sequence's settings as they are then, at their version); Two versions also lists a whole-second instant holding two hashed frames of one size and other content (#73) | Hash the other's unhashed photos at plan time | The verify reads every file anyway; the plan stays a dry run that writes nothing; at a whole second the size is what the comparison matched them by | +| 2026-11-18 | The Integrate journal keeps the losers the plan listed as last copies (`LastHere`, `LastThere`); before each trash step a loser not listed must still be held by another usable copy (the local copy's other frames, another sequence of this index, the server sequence's fresh manifest; for a server loser its other frames or any sequence of this index), else the journal goes and the job fails as a stale plan (#73) | Keep the loser and end Linked | As the merge's Replace stage: nothing goes to a trash as a last copy without being shown | +| 2026-11-18 | Download originals of a server sequence without a local copy goes under a key of the job (`chronos-sync::`, the same on a retry) when the sequence holding the server sequence's own key has a pair or a link for that server (kept apart, unlinked, asked again, linked elsewhere): a new local sequence, and its details are taken only by it; a holder with neither (a run that stopped before linking) is joined as before (#73) | Refuse the download and ask for Link or Ask again | The download twin of the round-7 upload rule: the shared key filed the photos back into the sequence the user kept separate, re-linked it and overwrote its details | +| 2026-11-18 | An upload never links again from its checkpoint a server sequence whose pair with the local sequence is Kept apart (unlinked or kept apart since the attempt that made it): the upload goes on as one without a target, under the job's key (#73) | Make Unlink and Keep apart cancel a pending upload | The checkpoint branch never reached the session-key rule, and a retry or a restart moved the Kept apart pair back to Linked | diff --git a/src/Chronos.Data/Archiving/ArchiveVerifier.cs b/src/Chronos.Data/Archiving/ArchiveVerifier.cs index 82a726c6..0e45341d 100644 --- a/src/Chronos.Data/Archiving/ArchiveVerifier.cs +++ b/src/Chronos.Data/Archiving/ArchiveVerifier.cs @@ -18,6 +18,9 @@ public readonly record struct VerifyPosition(int Sequence, int Frame); /// public readonly record struct VerifyProgress(long BytesDone, long BytesTotal, int Sequence, int Sequences, string Title, int Frame, int Frames); +/// A frame's file hashed by another job: its SHA-256 and the stamp the file had before it was read. +internal readonly record struct HashedFile(Guid FrameId, string Sha256, FileStamp Stamp); + /// Outcome of one verify run (frames before a resume point are counted by the run that checked them). public sealed record VerifyReport(int Sequences, long Checked, long Ok, long Hashed, IReadOnlyList Problems); @@ -195,6 +198,43 @@ public sealed class ArchiveVerifier } } + /// + /// Records first hashes another job read (a merge's verify stage, owner review 6): each frame of + /// the index has no hash for, whose sidecar has none (or this one) and whose file still has the stamp it was hashed at, + /// gets that hash in the sidecar and the index, under one lease, as a verify gives an adopted frame its first hash; flags + /// stay as they are. Returns how many. + /// + internal async Task RecordHashesAsync(Archive archive, ArchivePaths paths, Sequence target, IReadOnlyList hashes, + CancellationToken cancellationToken) + { + if (hashes.Count == 0) return 0; + await using (await _writer.LockAsync(paths, target.Id, cancellationToken).ConfigureAwait(false)) + { + var sequence = await SequenceWriter.FolderAsync(_db, paths, target, cancellationToken).ConfigureAwait(false); + var manifest = SequenceWriter.ReadLocked(sequence, target.Id); + var current = manifest.Frames.ToDictionary(f => f.Id); + // The sidecar may hold the hash already (another index wrote it): only that one is recorded, never another. + var found = hashes.Where(h => current.TryGetValue(h.FrameId, out var now) + && (string.IsNullOrEmpty(now.Sha256) || string.Equals(now.Sha256, h.Sha256, StringComparison.OrdinalIgnoreCase)) + && FileStamp.Of(sequence.File(now.File)) == h.Stamp).DistinctBy(h => h.FrameId).ToList(); + if (found.Count == 0) return 0; + var inStep = await _indexer.InStepAsync(archive, sequence, manifest, cancellationToken).ConfigureAwait(false); + var changes = found.ToDictionary(h => h.FrameId, h => (Func)(f => f with { Sha256 = h.Sha256.ToLowerInvariant() })); + ManifestStore.WriteSequence(sequence.ManifestPath, SequenceFrames.Update(manifest, changes)); + await using (var tx = await _db.BeginAsync(cancellationToken).ConfigureAwait(false)) + { + if (await tx.Sequences.GetForUpdateAsync(target.Id, cancellationToken).ConfigureAwait(false) is not null) + { + await tx.Frames.UpdateIntegrityAsync(found.ConvertAll(h => new FrameIntegrity(h.FrameId, current[h.FrameId].Flags, h.Sha256)), cancellationToken) + .ConfigureAwait(false); + } + await tx.CommitAsync(cancellationToken).ConfigureAwait(false); + } + if (inStep) await _indexer.RecordStampsAsync(archive, sequence, target.Id, cancellationToken).ConfigureAwait(false); + return found.Count; + } + } + /// A frame's check with the stamp its file had before it was read (null: no file). private readonly record struct PendingCheck(FrameCheck Check, FileStamp? File); } diff --git a/src/Chronos.Data/Archiving/Jobs/MergeJobHandler.cs b/src/Chronos.Data/Archiving/Jobs/MergeJobHandler.cs index ab284845..13be4f94 100644 --- a/src/Chronos.Data/Archiving/Jobs/MergeJobHandler.cs +++ b/src/Chronos.Data/Archiving/Jobs/MergeJobHandler.cs @@ -450,16 +450,7 @@ internal sealed class MergeJobHandler(ChronosDatabase db, SequenceMerger merger, if (inside && (File.Exists(source) || Directory.Exists(source))) { // Inside the folder that goes to the trash: copied (verified) into the kept sequence's exports first. - if (!journal.RenderTargets.TryGetValue(step.RenderId, out var target)) - { - target = keepPaths.ToRelative(Free(keepPaths.Sequence(keep.Folder).Exports, Path.GetFileName(source))); - journal.RenderTargets[step.RenderId] = target; - Save(); - } - var destination = keepPaths.FromRelative(target); - if (Directory.Exists(source)) await CopyFolderAsync(source, destination).ConfigureAwait(false); - else await CopyRenderAsync(source, destination, render.Sha256, step.FileName).ConfigureAwait(false); - output = target; + output = await CopyInAsync(step, render, source, keep).ConfigureAwait(false); } else { @@ -482,43 +473,94 @@ internal sealed class MergeJobHandler(ChronosDatabase db, SequenceMerger merger, } } + /// + /// Copies the render (a file or an image-sequence folder) into the kept sequence's exports + /// under the name journaled for it, and returns that archive-relative path. Nothing this job did not copy is ever + /// overwritten or deleted: when the journaled name holds something else by now (a render of the kept sequence made + /// after a failed run freed the name) or another render of the kept sequence names it, the copy takes a new free name, + /// journaled before it is made. + /// + private async Task CopyInAsync(RenderStep step, Render render, string source, Sequence keep) + { + var exports = keepPaths.Sequence(keep.Folder).Exports; + var taken = (await _db.Renders.ListBySequenceAsync(journal.KeepId, _ct).ConfigureAwait(false)) + .Where(r => r.Id != step.RenderId).Select(r => Path.GetFullPath(SyncService.RenderFile(keepArchive, r))) + .ToHashSet(StringComparer.OrdinalIgnoreCase); + var folder = Directory.Exists(source); + if (!journal.RenderTargets.TryGetValue(step.RenderId, out var target) + || taken.Contains(Path.GetFullPath(keepPaths.FromRelative(target))) + || !await OursAsync(source, keepPaths.FromRelative(target), render.Sha256, folder).ConfigureAwait(false)) + { + target = keepPaths.ToRelative(Free(exports, Path.GetFileName(source), taken)); + journal.RenderTargets[step.RenderId] = target; + Save(); + } + var destination = keepPaths.FromRelative(target); + if (folder) await CopyFolderAsync(source, destination).ConfigureAwait(false); + else await CopyRenderAsync(source, destination, render.Sha256, step.FileName).ConfigureAwait(false); + return target; + } + + /// + /// Whether holds nothing, or only what this job copies there: the render itself (by its + /// recorded SHA-256, else by the source's content) or, for a folder, files that each are the source's file of that name. + /// VerifiedCopy never leaves a partial file, so a run that stopped midway left whole files or none. + /// + private async Task OursAsync(string source, string destination, string? sha, bool folder) + { + if (!File.Exists(destination) && !Directory.Exists(destination)) return true; + if (!folder) + { + if (!File.Exists(destination)) return false; + var held = await ContentHash.Sha256Async(destination, null, _ct).ConfigureAwait(false); + var expected = sha is { Length: > 0 } ? sha : await ContentHash.Sha256Async(source, null, _ct).ConfigureAwait(false); + return string.Equals(held, expected, StringComparison.OrdinalIgnoreCase); + } + if (!Directory.Exists(destination)) return false; + foreach (var file in Directory.EnumerateFiles(destination, "*", SearchOption.AllDirectories)) + { + var original = Path.Combine(source, Path.GetRelativePath(destination, file)); + if (!File.Exists(original) || new FileInfo(original).Length != new FileInfo(file).Length + || !string.Equals(await ContentHash.Sha256Async(file, null, _ct).ConfigureAwait(false), + await ContentHash.Sha256Async(original, null, _ct).ConfigureAwait(false), StringComparison.Ordinal)) + return false; + } + return true; + } + + /// Copies a render file to ( said what is there is this copy). private async Task CopyRenderAsync(string source, string destination, string? sha, string name) { - if (File.Exists(destination)) - { - // A run that copied it before its row moved: kept when it is the same file. - if (sha is { Length: > 0 } && string.Equals(await ContentHash.Sha256Async(destination, null, _ct).ConfigureAwait(false), sha, StringComparison.OrdinalIgnoreCase)) - return; - File.Delete(destination); - } + // A run that copied it before its row moved: the same file, kept. + if (File.Exists(destination)) return; var copied = await VerifiedCopy.CopyAsync(source, destination, verify: true, null, _ct).ConfigureAwait(false); if (sha is { Length: > 0 } && !string.Equals(copied, sha, StringComparison.OrdinalIgnoreCase)) { + // This job's own copy, of a render that changed since it was recorded. File.Delete(destination); throw new InvalidOperationException(Loc.L("Errors.Copies.ChangedFile", name)); } } + /// Copies the files of a render folder that lacks ( checked the others). private async Task CopyFolderAsync(string source, string destination) { foreach (var file in Directory.EnumerateFiles(source, "*", SearchOption.AllDirectories)) { var target = Path.Combine(destination, Path.GetRelativePath(source, file)); - if (File.Exists(target) && new FileInfo(target).Length == new FileInfo(file).Length - && string.Equals(await ContentHash.Sha256Async(target, null, _ct).ConfigureAwait(false), - await ContentHash.Sha256Async(file, null, _ct).ConfigureAwait(false), StringComparison.Ordinal)) - continue; - if (File.Exists(target)) File.Delete(target); + if (File.Exists(target)) continue; await VerifiedCopy.CopyAsync(file, target, verify: true, null, _ct).ConfigureAwait(false); } } - private static string Free(string folder, string name) + /// A name in that nothing on disk and no render of uses. + private static string Free(string folder, string name, IReadOnlySet taken) { var stem = Path.GetFileNameWithoutExtension(name); var extension = Path.GetExtension(name); var candidate = Path.Combine(folder, name); - for (var n = 2; File.Exists(candidate) || Directory.Exists(candidate); n++) candidate = Path.Combine(folder, $"{stem} ({n}){extension}"); + for (var n = 2; File.Exists(candidate) || Directory.Exists(candidate) || taken.Contains(Path.GetFullPath(candidate)); n++) + candidate = Path.Combine(folder, $"{stem} ({n}){extension}"); return candidate; } @@ -537,14 +579,22 @@ internal sealed class MergeJobHandler(ChronosDatabase db, SequenceMerger merger, var total = keepFrames.Count + dropFrames.Count + flagged.Count; var done = 0; - // What the kept sequence holds now, hashed again. + // What the kept sequence holds now, hashed again. A frame the index had no hash for gets this one (owner Q3): a + // photo the merge keeps in two versions is listed under Two versions only once both are hashed. var held = new Dictionary(StringComparer.Ordinal); + var first = new List(); foreach (var frame in keepFrames.Where(f => (f.Flags & CopySides.UnusableFlags) == 0)) { _ct.ThrowIfCancellationRequested(); await context.ReportAsync(Done(), Total, new JobDetail(JobStep.Verify, frame.FileName, ++done, total)).ConfigureAwait(false); - if (await HashAsync(keepPaths.FromRelative(frame.RelativePath)).ConfigureAwait(false) is { } sha) held.TryAdd(sha, frame.Id); + var path = keepPaths.FromRelative(frame.RelativePath); + var stamp = frame.Sha256 is { Length: > 0 } ? null : FileStamp.Of(path); + if (await HashAsync(path).ConfigureAwait(false) is not { } sha) continue; + held.TryAdd(sha, frame.Id); + if (stamp is { } read) first.Add(new HashedFile(frame.Id, sha, read)); } + if (first.Count > 0 && await _db.Sequences.GetAsync(journal.KeepId, _ct).ConfigureAwait(false) is { } keepRow) + await owner.Archives.Verifier.RecordHashesAsync(keepArchive, keepPaths, keepRow, first, _ct).ConfigureAwait(false); var keepRemoved = (await _db.FrameRemovals.ListBySequenceAsync(journal.KeepId, _ct).ConfigureAwait(false)) .Select(r => r.Sha256.ToLowerInvariant()).ToHashSet(StringComparer.Ordinal); @@ -786,11 +836,13 @@ internal sealed class MergeJobHandler(ChronosDatabase db, SequenceMerger merger, private async Task DetailsAsync() { - if (journal.Payload is null || journal.DetailsDone) return; + if (journal.DetailsDone) return; + var payload = await PayloadAsync().ConfigureAwait(false); + if (payload is not { } write) return; await context.ReportAsync(Done(), Total, new JobDetail(JobStep.Details)).ConfigureAwait(false); try { - await new SequencePayloadWriter(_db, owner.Archives).ApplyAsync(journal.KeepId, journal.Payload, journal.KeepSettingsVersion, _ct).ConfigureAwait(false); + await new SequencePayloadWriter(_db, owner.Archives).ApplyAsync(journal.KeepId, write.Payload, write.SettingsVersion, _ct).ConfigureAwait(false); } catch (SyncConflictChangedException) { @@ -803,6 +855,43 @@ internal sealed class MergeJobHandler(ChronosDatabase db, SequenceMerger merger, Save(); } + /// + /// What the Details stage writes, and at which settings version of the kept sequence: the plan's payload (at the + /// version the plan read), with the second versions the verify filed (photos the plan compared by size and time only, + /// another photo after all) left out of Main as the plan leaves its own out (owner Q3). Without a planned payload, + /// the kept sequence's settings as they are now with those versions left out, at the version read with them. Null + /// when nothing is written. + /// + private async Task<(SyncPayload Payload, int SettingsVersion)?> PayloadAsync() + { + var versions = journal.SecondVersions.Select(id => journal.Verified.GetValueOrDefault(id)?.Sha256).OfType() + .Select(s => s.ToLowerInvariant()).Distinct(StringComparer.Ordinal).ToList(); + var planned = journal.Payload is { } p ? (p, journal.KeepSettingsVersion) : ((SyncPayload, int)?)null; + if (versions.Count == 0) return planned; + SyncPayload basis; + int version; + if (planned is { } known) + { + (basis, version) = known; + } + else + { + var keep = await _db.Sequences.GetAsync(journal.KeepId, _ct).ConfigureAwait(false) ?? throw new StaleCopyPlanException(); + basis = CopyPayload.Portable(SyncPayload.Of(keep), await SequencePayloadWriter.PhotosAsync(_db, journal.KeepId, null, _ct).ConfigureAwait(false)); + version = keep.SettingsVersion; + } + try + { + var excluded = CopyPayload.ExcludeVersions(basis, versions); + return planned is null && excluded.SameAs(basis) ? null : (excluded, version); + } + catch (Exception ex) when (ex is System.Text.Json.JsonException or InvalidDataException) + { + // Settings that are not a project of this Chronos: the versions stay included, as the plan leaves its own. + return planned; + } + } + // ---- projects ---- private async Task ProjectsAsync() diff --git a/src/Chronos.Data/Copies/CopySides.cs b/src/Chronos.Data/Copies/CopySides.cs index 4f2e0e2d..a5e979da 100644 --- a/src/Chronos.Data/Copies/CopySides.cs +++ b/src/Chronos.Data/Copies/CopySides.cs @@ -67,7 +67,8 @@ public sealed class CopySides(ChronosDatabase db) /// /// The photos one sequence keeps in two versions (owner Q3, design §4.1 Versions): a sub-second capture instant holding - /// exactly two usable, hashed frames of different content, as a Union merge or Integrate leaves them. Read from the + /// exactly two usable, hashed frames of different content, as a Union merge or Integrate leaves them, or a whole second + /// holding two such frames of the same size (matched by size and time, another photo after all). Read from the /// frames themselves, so they stay listed after the pair that made them is gone; each pair once, in index order. /// public async Task> VersionsAsync(Guid sequenceId, CancellationToken ct) @@ -78,9 +79,11 @@ public sealed class CopySides(ChronosDatabase db) foreach (var group in frames.Where(f => (f.Flags & UnusableFlags) == 0 && f.Sha256 is { Length: > 0 } && f.CapturedAt is not null) .GroupBy(f => CopyComparer.Millisecond(f.CapturedAt!.Value))) { - if (group.Key % 1000 == 0) continue; var versions = group.DistinctBy(f => f.Sha256!.ToLowerInvariant(), StringComparer.Ordinal).OrderBy(f => f.Index).ToList(); if (versions.Count != 2 || group.Count() != 2) continue; + // A camera that records whole seconds takes several photos in one: two at a whole second are two versions only + // when they also share the size, as the comparison matched them (an uncompressed RAW series, say). + if (group.Key % 1000 == 0 && versions[0].Size != versions[1].Size) continue; result.Add(new PhotoVersions(versions[0].Id, versions[0].FileName, versions[0].Sha256!.ToLowerInvariant(), versions[1].Id, versions[1].FileName, versions[1].Sha256!.ToLowerInvariant()) { diff --git a/src/Chronos.Data/Copies/IntegrateJobHandler.cs b/src/Chronos.Data/Copies/IntegrateJobHandler.cs index a414115c..4a423902 100644 --- a/src/Chronos.Data/Copies/IntegrateJobHandler.cs +++ b/src/Chronos.Data/Copies/IntegrateJobHandler.cs @@ -192,6 +192,8 @@ internal sealed class IntegrateJobHandler(SyncService sync, IntegratePlanner pla : p.Action == PhotoAction.TrashThere && sent.GetValueOrDefault(p.Key) is { Length: > 0 })) .Select(p => new IntegrateTrash(p.Key, p.Action == PhotoAction.TrashHere ? CopySide.ThisPc : CopySide.Other, p.FrameId!.Value, p.FileName, (p.Action == PhotoAction.TrashHere ? fetched[p.Key] : sent[p.Key])!))], + LastHere = [.. plan.LastCopies.Where(l => l.Side == CopySide.ThisPc).Select(l => l.Key.ToLowerInvariant())], + LastThere = [.. plan.LastCopies.Where(l => l.Side == CopySide.Other).Select(l => l.Key.ToLowerInvariant())], Payload = plan.Payload, LocalPayload = plan.LocalPayload, RemotePayload = plan.RemotePayload, @@ -347,6 +349,9 @@ internal sealed class IntegrateJobHandler(SyncService sync, IntegratePlanner pla Keep(step, Loc.L("Archive.Log.KeptReason.NotInPlace")); return; } + // Held by another copy when planned, by nothing now (that sequence deleted or flagged while the photos moved): a + // last copy the plan did not list. It stays; the journal goes and a retry plans again against both sides as they are. + if (!await LoserHeldHereAsync(loser).ConfigureAwait(false)) Stale(); try { var local = await _db.Sequences.GetAsync(journal.LocalId, _ct).ConfigureAwait(false) ?? throw new StaleCopyPlanException(); @@ -391,6 +396,8 @@ internal sealed class IntegrateJobHandler(SyncService sync, IntegratePlanner pla Keep(step, Loc.L("Archive.Log.KeptReason.NotInPlaceServer")); return; } + // As on this PC: a loser the plan counted as held elsewhere that nothing holds now is never trashed unlisted. + if (!await LoserHeldThereAsync(manifest, step.FrameId).ConfigureAwait(false)) Stale(); try { // At the frames version of the manifest that showed the winner in place: a server sequence changed since (the @@ -419,6 +426,47 @@ internal sealed class IntegrateJobHandler(SyncService sync, IntegratePlanner pla context.Log(Loc.L("Archive.Log.CopyKept", step.FileName, why)); } + /// + /// Whether this PC's losing version may go to the trash, asked as the plan asked it but now: a + /// last copy the plan listed, or held by another usable frame of the local copy, by another sequence of this index, or + /// by the server sequence (a fresh manifest). An unhashed loser was never compared by content and is not asked. + /// + private async Task LoserHeldHereAsync(Frame loser) + { + if (loser.Sha256 is not { Length: > 0 } recorded) return true; + var sha = recorded.ToLowerInvariant(); + if (journal.LastHere.Contains(sha)) return true; + foreach (var frame in await _db.Frames.FindBySha256Async(sha, _ct).ConfigureAwait(false)) + if (frame.SequenceId == journal.LocalId && frame.Id != loser.Id && (frame.Flags & CopySides.UnusableFlags) == 0) return true; + if ((await new CopySides(_db).HeldElsewhereAsync([sha], [journal.LocalId], _ct).ConfigureAwait(false)).Contains(sha)) return true; + SequenceManifest manifest; + try + { + manifest = await remote.GetManifestAsync(journal.RemoteId, _ct).ConfigureAwait(false); + } + catch (ChronosApiException ex) when (ex.Status == HttpStatusCode.NotFound) + { + throw await GoneAsync(ex).ConfigureAwait(false); + } + return manifest.Frames.Any(f => (f.Flags & (long)CopySides.UnusableFlags) == 0 && string.Equals(f.Sha256, sha, StringComparison.OrdinalIgnoreCase)); + } + + /// + /// Whether the server's losing version (frame of ) may go to its + /// trash: a last copy the plan listed, or held by another usable frame of the server sequence or by a sequence of this + /// index now. An unhashed loser is not asked. + /// + private async Task LoserHeldThereAsync(SequenceManifest manifest, Guid loserId) + { + if (manifest.Frames.FirstOrDefault(f => f.Id == loserId) is not { Sha256.Length: > 0 } loser) return true; + var sha = loser.Sha256.ToLowerInvariant(); + if (journal.LastThere.Contains(sha)) return true; + if (manifest.Frames.Any(f => f.Id != loserId && (f.Flags & (long)CopySides.UnusableFlags) == 0 + && string.Equals(f.Sha256, sha, StringComparison.OrdinalIgnoreCase))) + return true; + return (await new CopySides(_db).HeldElsewhereAsync([sha], [], _ct).ConfigureAwait(false)).Contains(sha); + } + /// Whether a usable frame of the local copy holds , its file hashed again now. private async Task WinnerHereAsync(string sha) { diff --git a/src/Chronos.Data/Copies/IntegrateJournal.cs b/src/Chronos.Data/Copies/IntegrateJournal.cs index 99d9984a..c4581d36 100644 --- a/src/Chronos.Data/Copies/IntegrateJournal.cs +++ b/src/Chronos.Data/Copies/IntegrateJournal.cs @@ -61,6 +61,15 @@ internal sealed class IntegrateJournal public List Trash { get; set; } = []; + /// + /// Raw SHA-256 of this PC's losing versions the plan listed as last copies (the user confirmed them): any other goes to + /// the trash only while another copy still holds it (asked again at the trash, which the plan may not wait for). + /// + public List LastHere { get; set; } = []; + + /// Raw SHA-256 of the server's losing versions the plan listed as last copies; as . + public List LastThere { get; set; } = []; + /// The one portable payload both sides get; null when the details stay as they are. public SyncPayload? Payload { get; set; } diff --git a/src/Chronos.Data/Sync/SyncTransfers.cs b/src/Chronos.Data/Sync/SyncTransfers.cs index 943fd703..174d0ae4 100644 --- a/src/Chronos.Data/Sync/SyncTransfers.cs +++ b/src/Chronos.Data/Sync/SyncTransfers.cs @@ -538,8 +538,9 @@ internal sealed class SyncTransfers(SyncService sync) if (local is null && wanted.Count > 0) { // The first original creates the local sequence (grouped by the server id); the others join it. + var key = await DownloadKeyAsync(context, remote, remoteId, archive.Id).ConfigureAwait(false); var (staged, sha) = await FetchOriginalAsync(remote, wanted[0], staging, ct).ConfigureAwait(false); - var first = await CommitAsync(remoteId, archive.Id, null, wanted[0], staged, sha, captured, ct).ConfigureAwait(false); + var first = await CommitAsync(key, archive.Id, null, wanted[0], staged, sha, captured, ct).ConfigureAwait(false); local = await db.Sequences.GetAsync(first.SequenceId, ct).ConfigureAwait(false) ?? throw new InvalidOperationException(Loc.L("Errors.Sync.LocalMissing")); await sync.LinkAsync(remote.Server, remoteId, local, ct).ConfigureAwait(false); @@ -574,7 +575,7 @@ internal sealed class SyncTransfers(SyncService sync) } else { - await CommitAsync(remoteId, archive.Id, sequenceId, frame, staged, sha, captured, token).ConfigureAwait(false); + await CommitAsync(null, archive.Id, sequenceId, frame, staged, sha, captured, token).ConfigureAwait(false); } await progress.StepAsync().ConfigureAwait(false); }).ConfigureAwait(false); @@ -942,7 +943,7 @@ internal sealed class SyncTransfers(SyncService sync) /// /// The server sequence an upload fills and whether this job creates it: the linked one, the one an interrupted - /// attempt created, or none yet (the first original creates it). + /// attempt created (unless the user kept it apart or unlinked it since), or none yet (the first original creates it). /// private async Task<(Guid? RemoteId, bool Created)> TargetAsync(JobContext context, ISyncRemote remote, Sequence local) { @@ -952,6 +953,10 @@ internal sealed class SyncTransfers(SyncService sync) foreach (var candidate in new[] { link?.RemoteId, started }) { if (candidate is not { } id) continue; + // The sequence an interrupted attempt created, unlinked or kept apart by the user since: never linked again from + // the checkpoint. The upload makes a new sequence (SessionKeyAsync), as one of a kept-apart sequence does. + if (id != link?.RemoteId && await sync.Copies.FindRemoteAsync(local.Id, remote.Server, id, ct).ConfigureAwait(false) is { State: CopyState.KeptApart }) + continue; if (await ExistsAsync(remote, id, ct).ConfigureAwait(false)) { if (link?.RemoteId != id) await sync.LinkAsync(remote.Server, id, local, ct).ConfigureAwait(false); @@ -978,6 +983,24 @@ internal sealed class SyncTransfers(SyncService sync) return apart ? string.Create(CultureInfo.InvariantCulture, $"{own}:{context.Job.Id:N}") : own; } + /// + /// The session key the first original of a download without a local copy goes under: the server sequence's own + /// (), which groups the download in the local sequence an earlier download of it made, + /// unless that sequence has been unlinked or kept apart from it, or linked to another sequence of this server since (it + /// has a pair or a link for this server). The user chose to keep such a sequence separate: the download makes a new + /// local sequence instead, under a key of this job (the same on a retry), and never fills or overwrites the other. + /// + private async Task DownloadKeyAsync(JobContext context, ISyncRemote remote, Guid remoteId, Guid archiveId) + { + var ct = context.Cancellation; + var own = SyncService.SessionKey(remoteId); + if (await sync.Database.Sequences.FindBySessionKeyAsync(archiveId, own, ct).ConfigureAwait(false) is not { } holder) return own; + var apart = (await sync.Copies.ListByLocalAsync(holder, ct).ConfigureAwait(false)) + .Any(c => c.Kind == CopyKind.LocalServer && string.Equals(c.Server, remote.Server, StringComparison.Ordinal) && c.LocalId == holder) + || await sync.Links.FindByLocalAsync(remote.Server, holder, ct).ConfigureAwait(false) is not null; + return apart ? string.Create(CultureInfo.InvariantCulture, $"{own}:{context.Job.Id:N}") : own; + } + private static async Task ExistsAsync(ISyncRemote remote, Guid id, CancellationToken cancellationToken) { try @@ -1060,11 +1083,14 @@ internal sealed class SyncTransfers(SyncService sync) public bool Refuse() => Interlocked.Exchange(ref _refused, 1) == 0; } - /// Commits one original from staging into the local sequence (created from the server id when null). - private async Task CommitAsync(Guid remoteId, Guid archiveId, Guid? sequenceId, ManifestFrame frame, string staged, string sha, + /// + /// Commits one original from staging into the local sequence , or (when null) the one of + /// , created by the first original (). + /// + private async Task CommitAsync(string? sessionKey, Guid archiveId, Guid? sequenceId, ManifestFrame frame, string staged, string sha, IReadOnlyDictionary captured, CancellationToken cancellationToken) { - var request = new IngestRequest(archiveId, sequenceId, null, sequenceId is null ? SyncService.SessionKey(remoteId) : null, staged, frame.FileName, + var request = new IngestRequest(archiveId, sequenceId, null, sequenceId is null ? sessionKey : null, staged, frame.FileName, frame.Size, sha, captured.GetValueOrDefault(frame.Id), null); var result = await sync.Archives.Ingest.CommitAsync(request, cancellationToken).ConfigureAwait(false); // A duplicate stays in staging (the sequence already holds that content). diff --git a/tests/Chronos.Data.Tests/Archiving/SequenceMergerTests.cs b/tests/Chronos.Data.Tests/Archiving/SequenceMergerTests.cs index 25a0cd09..e3c7cc71 100644 --- a/tests/Chronos.Data.Tests/Archiving/SequenceMergerTests.cs +++ b/tests/Chronos.Data.Tests/Archiving/SequenceMergerTests.cs @@ -404,6 +404,79 @@ public abstract class SequenceMergerTests(TestContext context) : DatabaseTest(co Assert.Equal(stages.Count, projects.Copied.Count); } + [Test] + public async Task AResumedRendersStageNeverOverwritesARenderMadeAtItsTargetMeanwhile() + { + var fx = await FixtureAsync(); + var s = await ScenarioAsync(fx, 0); + // The other also holds an image-sequence export inside its folder. + var frames = Path.Combine(fx.Paths.Sequence(s.Drop.Folder).Exports, "frames"); + Directory.CreateDirectory(frames); + File.WriteAllBytes(Path.Combine(frames, "0001.png"), [1, 2, 3]); + var folder = new Render + { + Id = Guid.CreateVersion7(), + SequenceId = s.Drop.Id, + PresetName = "PNG", + OutputPath = fx.Paths.ToRelative(frames), + Bytes = 3, + Created = DateTimeOffset.UtcNow, + }; + await fx.Db.Renders.InsertAsync(folder, fx.Ct); + var merger = Merger(fx); + var (engine, handler) = Engine(fx, merger, new FakeProjects(), new FakeWorkspace()); + await engine.StartAsync(Ct); + var failedOnce = false; + handler.BeforeStage = stage => + { + if (stage != MergeStage.Renders || failedOnce) return Task.CompletedTask; + failedOnce = true; + throw new IOException("the other's drive failed a read"); + }; + var plan = await merger.PlanAsync(s.Copy.Id, s.Keep.Id, s.Drop.Id, s.Choices, NoContext, Ct); + var job = await merger.EnqueueAsync(engine, new FoundCopyStore(fx.Db), plan, s.Keep.DisplayTitle, Ct); + await fx.WaitForAsync(job.Id, JobState.Failed); + + // The failed run had journaled where the copies go and copied nothing; meanwhile the user renders the kept sequence + // with the same names (same date, title and preset). + var keep = (await fx.Db.Sequences.GetAsync(s.Keep.Id))!; + var exports = fx.Paths.Sequence(keep.Folder).Exports; + var clip = Path.Combine(exports, "clip.mp4"); + var images = Path.Combine(exports, "frames"); + Directory.CreateDirectory(images); + var fresh = RandomNumberGenerator.GetBytes(2048); + File.WriteAllBytes(clip, fresh); + File.WriteAllBytes(Path.Combine(images, "0001.png"), [9, 9, 9]); + await fx.Db.Renders.InsertAsync(new Render + { + Id = Guid.CreateVersion7(), + SequenceId = keep.Id, + PresetName = "1080p", + OutputPath = fx.Paths.ToRelative(clip), + Bytes = fresh.Length, + Sha256 = Sha(fresh), + Created = DateTimeOffset.UtcNow, + }, fx.Ct); + var journal = MergeJournal.Read(fx.Paths, job.Id)!; + journal.RenderTargets[s.Render.Id] = fx.Paths.ToRelative(clip); + journal.RenderTargets[folder.Id] = fx.Paths.ToRelative(images); + journal.Save(fx.Paths); + + await engine.ResumeAsync(job.Id, Ct); + await fx.WaitForAsync(job.Id, JobState.Done); + + // The new renders are as the user made them; the other's go beside them under free names. + Assert.SequenceEqual(fresh, File.ReadAllBytes(clip)); + Assert.SequenceEqual(new byte[] { 9, 9, 9 }, File.ReadAllBytes(Path.Combine(images, "0001.png"))); + Assert.Count(1, Directory.GetFiles(images)); + var movedClip = (await fx.Db.Renders.GetAsync(s.Render.Id))!; + Assert.NotEqual(fx.Paths.ToRelative(clip), movedClip.OutputPath); + Assert.Equal(s.Render.Sha256, Sha(File.ReadAllBytes(fx.Paths.FromRelative(movedClip.OutputPath)))); + var movedFolder = (await fx.Db.Renders.GetAsync(folder.Id))!; + Assert.NotEqual(fx.Paths.ToRelative(images), movedFolder.OutputPath); + Assert.SequenceEqual(new byte[] { 1, 2, 3 }, File.ReadAllBytes(Path.Combine(fx.Paths.FromRelative(movedFolder.OutputPath), "0001.png"))); + } + [Test] public async Task APhotoOfTheOtherThatChangedOnDiskFailsTheVerifyAndNothingGoesToTheTrash() { @@ -477,6 +550,77 @@ public abstract class SequenceMergerTests(TestContext context) : DatabaseTest(co Assert.Null(await fx.Db.Sequences.GetAsync(drop.Id)); } + [Test] + public async Task ASecondVersionOfAnUnhashedKeptPhotoIsListedAndLeftOutOfMain() + { + var fx = await FixtureAsync(); + // The kept one adopted and not verified, the other hashed: one photo of the same size and sub-second time, other bytes. + var keep = await SequenceAsync(fx, "keep", [(1, 250, "IMG_0001.JPG")]); + var drop = await SequenceAsync(fx, "drop", [(2, 250, "IMG_0001.JPG")]); + var access = new DbAccess(fx.Db.Database); + await access.ExecuteAsync("UPDATE frames SET size = 1000 WHERE sequence_id = $1 OR sequence_id = $2", [keep.Id, drop.Id], Ct); + await access.ExecuteAsync("UPDATE frames SET sha256 = NULL WHERE sequence_id = $1", [keep.Id], Ct); + var copy = await PairAsync(fx, keep.Id, drop.Id); + var merger = Merger(fx); + var (engine, _) = Engine(fx, merger, new FakeProjects(), new FakeWorkspace()); + await engine.StartAsync(Ct); + var plan = await merger.PlanAsync(copy.Id, keep.Id, drop.Id, await ChoicesAsync(fx, keep.Id, drop.Id, CopyPhotoPolicy.Union), NoContext, Ct); + + var job = await merger.EnqueueAsync(engine, new FoundCopyStore(fx.Db), plan, keep.DisplayTitle, Ct); + await fx.WaitForAsync(job.Id, JobState.Done); + + // The kept photo has the hash the merge read, so Two versions lists the pair, the other's version left out of Main. + Assert.Contains(Sha(1, 250), (await fx.Db.Frames.ListBySequenceAsync(keep.Id)).Select(f => f.Sha256)); + AssertVersions(await new CopySides(fx.Db).VersionsAsync(keep.Id, Ct), Sha(1, 250), Sha(2, 250)); + } + + [Test] + public async Task APhotoTheVerifyFindsIsAnotherVersionIsListedAtAWholeSecondAndLeftOutOfMain() + { + var fx = await FixtureAsync(); + // A camera that records whole seconds, photos of one size (an uncompressed RAW series): neither side hashed, compared + // as the same photo, another photo after all. + var (mine, theirs) = EqualSizeSeeds(0); + var keep = await SequenceAsync(fx, "keep", [(mine, 0, "IMG_0001.JPG")]); + var drop = await SequenceAsync(fx, "drop", [(theirs, 0, "IMG_0001.JPG")]); + await new DbAccess(fx.Db.Database).ExecuteAsync("UPDATE frames SET sha256 = NULL WHERE sequence_id = $1 OR sequence_id = $2", [keep.Id, drop.Id], Ct); + var copy = await PairAsync(fx, keep.Id, drop.Id); + var merger = Merger(fx); + var (engine, _) = Engine(fx, merger, new FakeProjects(), new FakeWorkspace()); + await engine.StartAsync(Ct); + var plan = await merger.PlanAsync(copy.Id, keep.Id, drop.Id, await ChoicesAsync(fx, keep.Id, drop.Id, CopyPhotoPolicy.Union), NoContext, Ct); + Assert.Empty(plan.Photos); + + var job = await merger.EnqueueAsync(engine, new FoundCopyStore(fx.Db), plan, keep.DisplayTitle, Ct); + await fx.WaitForAsync(job.Id, JobState.Done); + + // Filed at the verify as a second version: listed, and Main renders one photo at that instant. + Assert.True((await HashesAsync(fx, keep.Id)).SetEquals([Sha(mine, 0), Sha(theirs, 0)])); + AssertVersions(await new CopySides(fx.Db).VersionsAsync(keep.Id, Ct), Sha(mine, 0), Sha(theirs, 0)); + } + + /// One photo listed in two versions: in Main, left out. + private static void AssertVersions(IReadOnlyList listed, string kept, string incoming) + { + Assert.Count(1, listed); + var versions = listed[0]; + var excluded = new Dictionary { [versions.FirstSha256] = versions.FirstExcluded, [versions.SecondSha256] = versions.SecondExcluded }; + Assert.False(excluded[kept]); + Assert.True(excluded[incoming]); + } + + /// Two seeds whose photos at have the same size and other bytes. + private static (int, int) EqualSizeSeeds(int ms) + { + var bySize = new Dictionary(); + for (var seed = 1; ; seed++) + { + var length = Photos.Jpeg(Start.AddMilliseconds(ms), seed).Length; + if (bySize.TryGetValue(length, out var other)) return (other, seed); + bySize[length] = seed; + } + } + [Test] public async Task AnUnhashedPhotoOfTheKeptOneWithTheSameContentIsNoLastCopy() { diff --git a/tests/Chronos.Data.Tests/Copies/IntegrateTests.cs b/tests/Chronos.Data.Tests/Copies/IntegrateTests.cs index 1dd8097d..7a91a8e7 100644 --- a/tests/Chronos.Data.Tests/Copies/IntegrateTests.cs +++ b/tests/Chronos.Data.Tests/Copies/IntegrateTests.cs @@ -527,6 +527,43 @@ public sealed class IntegrateTests(TestContext context) Assert.Equal(CopyState.Linked, (await f.Sync.Copies.GetAsync(pair.Id, f.Ct))!.State); } + [Test] + public async Task ALoserWhoseOtherHolderGoesBeforeTheTrashIsNeverTrashedUnlisted() + { + await using var f = await FixtureAsync(context); + var (engine, handler) = Engine(f); + await using var _ = engine; + await engine.StartAsync(f.Ct); + + // This PC's losing version (the server wins) and the server's (this PC wins), each held by a third local sequence + // when planned: neither is a last copy. That sequence's copy is flagged corrupt while the photos move. + foreach (var (n, policy, loser) in new[] { (1, CopyPhotoPolicy.OtherWins, 104), (2, CopyPhotoPolicy.ThisPcWins, 205) }) + { + var (local, remote, pair) = await ScenarioAsync(f, n); + var third = await LocalAsync(f, $"Third {n}", [(loser, 30_250, "IMG_0004.JPG")]); + var plan = await Planner(f).PlanAsync(f.Remote, pair, await ChoicesAsync(f, local.Id, remote.Id, policy), PlanContext(), f.Ct); + Assert.True(plan.Destructive, policy.ToString()); + Assert.Empty(plan.LastCopies); + handler.BeforeStage = async stage => + { + if (stage != IntegrateStage.Trash) return; + var held = (await f.Db.Frames.ListBySequenceAsync(third.Id, cancellationToken: f.Ct)).Single(); + await f.Db.Frames.UpdateFlagsAsync([held.Id], FrameFlags.Corrupt, FrameFlags.None, f.Ct); + }; + var job = await CopyJobs.EnqueueIntegrateAsync(engine, f.Sync.Copies, plan, local.DisplayTitle, f.Ct); + var failed = await f.Archive.WaitForAsync(job.Id, JobState.Failed); + + // Its only copy now: it stays where it was (nothing in either trash), and the journal goes so a retry plans again. + Assert.Equal(Loc.L("Errors.Copies.StalePlan"), failed.Error); + if (policy == CopyPhotoPolicy.OtherWins) Assert.Contains(Sha(loser, 30_250), await LocalHashesAsync(f, local.Id)); + else Assert.Contains(Sha(loser, 30_250), RemoteHashes(remote)); + Assert.Empty(f.Remote.Trashed); + Assert.False(File.Exists(IntegrateJournal.PathFor(f.Archive.Paths, job.Id))); + handler.BeforeStage = null; + await WaitIdleAsync(f, local.Id); + } + } + [Test] public async Task TheServersLoserGoesToItsTrashAtTheVersionItsDryRunRead() { diff --git a/tests/Chronos.Data.Tests/Sync/SyncTransferCopyTests.cs b/tests/Chronos.Data.Tests/Sync/SyncTransferCopyTests.cs index c4646f25..8e88613a 100644 --- a/tests/Chronos.Data.Tests/Sync/SyncTransferCopyTests.cs +++ b/tests/Chronos.Data.Tests/Sync/SyncTransferCopyTests.cs @@ -2,6 +2,7 @@ using System.Security.Cryptography; using Chronos.Core.Model; using Chronos.Data.Copies; using Chronos.Data.Jobs; +using Chronos.Data.Repositories; using Chronos.Data.Sync; using Chronos.Protocol; using Chronos.Testing; @@ -35,6 +36,19 @@ public sealed class SyncTransferCopyTests(TestContext context) return f; } + /// Waits until no pass a download queued works on . + private static async Task WaitIdleAsync(SyncFixture f, Guid sequenceId) + { + var until = DateTime.UtcNow.AddSeconds(30); + while (DateTime.UtcNow < until) + { + var held = await f.Db.Jobs.ListPendingTargetsAsync(JobKind.Thumbs, f.Ct); + held.UnionWith(await f.Db.Jobs.ListPendingTargetsAsync(JobKind.LightAnalyse, f.Ct)); + if (!held.Contains(sequenceId)) return; + await Task.Delay(20, f.Ct); + } + } + [Test] public async Task OnlyThePhotosAskedForGoEachWay() { @@ -172,6 +186,83 @@ public sealed class SyncTransferCopyTests(TestContext context) Assert.True(f.Remote.Sequences[link.RemoteId].Frames.Select(fr => fr.Sha256).ToHashSet().SetEquals(photos.Select(Sha))); } + [Test] + public async Task ARetriedUploadNeverLinksAgainTheServerSequenceItsFirstAttemptMadeAndTheUserKeptApart() + { + await using var f = Capable(await SyncFixture.CreateAsync(context)); + var photos = f.NewPhotos(3); + var local = await f.LocalAsync("Comet", [photos[0], photos[1]]); + var engine = await f.StartEngineAsync(); + var job = await f.Sync.EnqueueUploadAsync(engine, f.Server, local); + await f.Archive.WaitForAsync(job.Id, JobState.Done); + var started = Assert.NotNull(await f.Sync.Links.FindByLocalAsync(f.Server, local.Id, f.Ct)).RemoteId; + // The user keeps the pair apart while a run of the upload that made it is still to be retried. + await f.Sync.UnlinkAsync(f.Server, started, local.Id, f.Ct); + var pair = (await f.Sync.Copies.ListByServerAsync(f.Server, f.Ct)).Single(c => c.RemoteId == started); + await f.AddLocalAsync(local.Id, photos[2]); + + // The retry resumes from the checkpoint naming the sequence that run made. + var row = new Job + { + Id = Guid.CreateVersion7(), + Kind = JobKind.UploadCommit, + TargetIds = [local.Id], + State = JobState.Running, + NodeId = "t#0", + Created = DateTimeOffset.UtcNow, + Started = DateTimeOffset.UtcNow, + Checkpoint = started.ToString(), + }; + await f.Db.Jobs.EnqueueAsync(row, f.Ct); + var retry = new JobContext(new JobEngine(f.Db, [], new JobEngineOptions { WorkerId = "t" }), row, f.Ct); + await new SyncTransfers(f.Sync).UploadAsync(retry, f.Remote, local.Id); + + // The sequence kept apart stays apart, unlinked and as it was; a new one holds every photo. + Assert.Equal(CopyState.KeptApart, (await f.Sync.Copies.GetAsync(pair.Id, f.Ct))!.State); + Assert.Null((await f.Sync.Links.GetAsync(f.Server, started, f.Ct))!.LocalId); + Assert.True(f.Remote.Sequences[started].Frames.Select(fr => fr.Sha256).ToHashSet().SetEquals([Sha(photos[0]), Sha(photos[1])])); + var link = Assert.NotNull(await f.Sync.Links.FindByLocalAsync(f.Server, local.Id, f.Ct)); + Assert.NotEqual(started, link.RemoteId); + Assert.True(f.Remote.Sequences[link.RemoteId].Frames.Select(fr => fr.Sha256).ToHashSet().SetEquals(photos.Select(Sha))); + } + + [Test] + public async Task DownloadingAgainAServerSequenceUnlinkedFromItsDownloadMakesANewLocalSequence() + { + await using var f = Capable(await SyncFixture.CreateAsync(context)); + var photos = f.NewPhotos(3); + var remote = f.Remote.Add("Comet", photos); + var engine = await f.StartEngineAsync(); + var job = await f.Sync.EnqueueDownloadAsync(engine, f.Server, remote.Id, "Comet", SyncDownloadMode.Originals, f.Archive.Archive.Id); + await f.Archive.WaitForAsync(job.Id, JobState.Done); + var local = Assert.NotNull(Assert.NotNull(await f.Sync.Links.GetAsync(f.Server, remote.Id, f.Ct)).LocalId); + await WaitIdleAsync(f, local); + + // Here: one photo deleted, the title edited; then the pair is unlinked (kept apart). + var sequence = (await f.Db.Sequences.GetAsync(local, f.Ct))!; + var gone = (await f.Db.Frames.ListBySequenceAsync(local, cancellationToken: f.Ct))[0]; + var delete = await f.Archive.Service.FrameTrash.PlanDeleteAsync(local, [gone.Id], sequence.FramesVersion, f.Ct); + await f.Archive.Service.FrameTrash.DeleteAsync(delete, cancellationToken: f.Ct); + await f.Db.Sequences.EditAsync(local, new SequenceEdit { Title = "Mine" }, null, f.Ct); + await f.Sync.UnlinkAsync(f.Server, remote.Id, local, f.Ct); + var pair = (await f.Sync.Copies.ListByServerAsync(f.Server, f.Ct)).Single(c => c.RemoteId == remote.Id && c.LocalId == local); + Assert.Equal(CopyState.KeptApart, pair.State); + + // Download originals again: a separate local copy. + job = await f.Sync.EnqueueDownloadAsync(engine, f.Server, remote.Id, "Comet", SyncDownloadMode.Originals, f.Archive.Archive.Id); + await f.Archive.WaitForAsync(job.Id, JobState.Done); + + // The sequence kept apart is as the user left it: its photos, its deletion, its title, its pair. + var after = (await f.Db.Sequences.GetAsync(local, f.Ct))!; + Assert.Equal("Mine", after.Title); + Assert.Count(2, await f.Db.Frames.ListBySequenceAsync(local, cancellationToken: f.Ct)); + Assert.Contains(gone.Sha256, (await f.Db.FrameRemovals.ListBySequenceAsync(local, f.Ct)).Select(r => r.Sha256)); + Assert.Equal(CopyState.KeptApart, (await f.Sync.Copies.GetAsync(pair.Id, f.Ct))!.State); + var copy = Assert.NotNull(Assert.NotNull(await f.Sync.Links.GetAsync(f.Server, remote.Id, f.Ct)).LocalId); + Assert.NotEqual(local, copy); + Assert.True((await LocalHashesAsync(f, copy)).SetEquals(photos.Select(Sha))); + } + [Test] public async Task AnAutomaticUploadAfterTheServerCopyIsGoneFailsAndCreatesNothing() {