Files
Chronos/src/Chronos.Data/Archiving/SequenceWriter.cs
T
Alby96andClaude Opus 5.5 e2ecc409a3 feat(server): the server keeps its sequences' projects, pushed and pulled like a PC's archive
Owner review 2, phase 5b (R8): a server is a project home. A workspace can
list a sequence's projects, make, rename and delete them, read their
versions and push new ones through /api/v1/sequences/{sid}/projects. The
endpoints run on the archive's ProjectService exactly as a desktop's archive
does (sidecars first, then the index, under the sequence's lease), so no
version is stored apart and a desktop sharing the archive sees the same
projects.

- Protocol: project, version, create, rename, version page, push answer
  (outcome, stored version, head, plus code and message on 409/422) and
  home frames DTOs, the projectChanged event, the "projects" capability and
  the ChronosClient calls; a push returns its outcome without throwing on
  409 or 422. ProjectWire maps them to and from the model for both sides.
- Server: Read lists and reads, Edit makes, renames and pushes, Delete (the
  sequence's owner, the project's maker or an admin) deletes; refusals with
  stable codes, 503 while the sequence is busy, offline or moved; the
  pushing account becomes the version's author; creations, renames and
  deletions audited, pushes logged.
- ProjectService tells the host what each call changed, compared under the
  lease, so projectChanged also follows Main's external versions; a Main
  version mirrored into the settings sends sequenceChanged too, for older
  clients. ProjectCatchUp takes settings a PATCH, an analysis or a
  conversion wrote into Main at once, for sequences that have a Main.
- Renders of a project version: POST /jobs checks the project and version
  against the sequence and pins a project alone to its head; render nodes
  get the version's settings placed on the frames they claim; the render
  records the project and version.
- Admin panel: the sequence page lists its projects (Main marked, head,
  last change, versions on demand) with Rename and Delete for the others,
  live on projectChanged. It edits no settings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 11:15:19 +01:00

204 lines
11 KiB
C#
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
using System.Collections.Concurrent;
using Chronos.Core.Archiving;
using Chronos.Core.Imaging;
using Chronos.Core.Model;
namespace Chronos.Data.Archiving;
/// <summary>Settings shared by the archive services.</summary>
public sealed record ArchiveServiceOptions
{
/// <summary>Decoders for thumbnails and proxies beyond embedded previews and JPEG (WIC on Windows).</summary>
public ImageDecoder Decoder { get; init; } = new([]);
/// <summary>Name written into lease files so a waiting writer can say who holds a sequence.</summary>
public string Owner { get; init; } = "chronos@" + Environment.MachineName;
/// <summary>How long a writer waits for a sequence held by another process before giving up.</summary>
public TimeSpan LeaseWait { get; init; } = TimeSpan.FromMinutes(2);
public TimeSpan LeaseDuration { get; init; } = LeaseFile.DefaultDuration;
/// <summary>
/// Scans queue thumbnails and proxies of the sequences they index without them, as an upload does (the server, whose
/// clients load both from it: BRIEF §4.3, §14.4). The desktop decodes its thumbnails on demand and makes proxies when
/// asked.
/// </summary>
public bool ImagesAfterScan { get; init; }
/// <summary>
/// Photos a pass works on at once, asked at the start of each sequence (owner review 1, R2: one job runs at a time and
/// the host's parallelism decides how many photos it processes at once; Settings › Rendering › Parallelism on the
/// desktop, <c>CHRONOS_PARALLELISM</c> on the server). Default: half the logical cores.
/// </summary>
public Func<int> Parallelism { get; init; } = () => DefaultParallelism;
/// <summary>
/// Memory the photos a proxy pass decodes at once may take, in bytes, asked with <see cref="Parallelism"/>: fewer at once
/// for large originals. Default: a quarter of the memory available, at least 1 GiB.
/// </summary>
public Func<long> MemoryBudget { get; init; } = () => DefaultMemoryBudget;
/// <summary>
/// The host's own clean-up steps (the server's uploads sweep, the desktop's cache trim), run by
/// <see cref="Core.Model.JobKind.Cleanup"/> jobs after the shared ones of <see cref="Housekeeping.CleanupSteps"/> when a
/// schedule names them (owner review 2, R10). A name the shared steps use already is ignored.
/// </summary>
public IReadOnlyList<Housekeeping.ICleanupStep> CleanupSteps { get; init; } = [];
/// <summary>
/// Folder of the copies <see cref="Core.Model.JobKind.Backup"/> jobs make; null: the <c>backups</c> folder next to the
/// SQLite index (<see cref="DatabaseBackups.DefaultDirectory"/>), where the desktop and the server keep them.
/// </summary>
public string? BackupDirectory { get; init; }
/// <summary>
/// Told after a Reorganize changed a sequence's folder, title or file names (change <c>reorganized</c>), once its step is
/// journaled, or once its folder moved and its index row followed when the step then failed: by its job, or by an archive
/// open that completed a plan an interruption left (<see cref="ArchiveService.OpenAsync"/>). Told too after a conversion
/// replaced a sequence's frames (<c>converted</c>) or made a new sequence of them (<c>created</c>).
/// A server tells its connected clients (BRIEF §14.3); the service it opens its archive with at start sets none, as no
/// client is connected yet. The desktop reloads an editor open on the sequence. A failure is logged and never fails the job.
/// </summary>
public Func<Guid, string, CancellationToken, Task>? SequenceChanged { get; init; }
/// <summary>
/// Told after a call of <see cref="Projects.ProjectService"/> changed a sequence's projects (owner review 2, R8): a project
/// made, renamed or deleted, a new head (a push, settings another path wrote taken as Main's external version, a merge, the
/// versions another program sharing the archive wrote found in the sidecars), and whether the call wrote Main's head into
/// the sequence's settings. Told once per call, after the sequence's lease is let go. A server tells its connected clients
/// (BRIEF §14.3); a failure is logged and never fails the call.
/// </summary>
public Func<Projects.ProjectChangeSet, CancellationToken, Task>? ProjectsChanged { get; init; }
/// <summary>
/// The free bytes a write under a folder of an archive (its root, its staging folder) may count on, as the host counts
/// them: a server takes the tighter of its space volume (owner review 2, R2) and the folder's own volume, the desktop asks
/// Windows, which measures a network share too; null: the folder's drive (<see cref="DriveInfo"/>). A conversion checks its
/// estimate against it before it starts, before each sequence and before each photo (R7). The function returns null when
/// the space cannot be measured.
/// </summary>
public Func<string, long?>? FreeSpace { get; init; }
/// <summary>Half the logical cores, at least one: the parallelism when the host sets none.</summary>
public static int DefaultParallelism => Math.Max(1, Environment.ProcessorCount / 2);
/// <summary>A quarter of the memory available to this process, at least 1 GiB.</summary>
public static long DefaultMemoryBudget => Math.Max(1L << 30, GC.GetGCMemoryInfo().TotalAvailableMemoryBytes / 4);
}
/// <summary>
/// Read-modify-write of sidecars, serialised per sequence: an in-process gate for this process's writers plus a lease
/// file for writers in other processes on the same share (server and client). Every edit sees the latest sidecar,
/// with an interrupted ingest batch completed first.
/// </summary>
internal sealed class SequenceWriter(ArchiveServiceOptions options)
{
private readonly ConcurrentDictionary<Guid, SemaphoreSlim> _gates = new();
public ArchiveServiceOptions Options => options;
/// <summary>Holds the sequence exclusively until disposed.</summary>
public Task<IAsyncDisposable> LockAsync(ArchivePaths archive, Guid sequenceId, CancellationToken cancellationToken) =>
LockAsync(archive, sequenceId, null, cancellationToken);
/// <summary>
/// Holds the sequence exclusively until disposed, waiting at most <paramref name="wait"/> for this process's other writers
/// and for the lease (the options' lease wait when null).
/// </summary>
/// <exception cref="LeaseUnavailableException">Another writer kept the sequence for longer than the wait.</exception>
public async Task<IAsyncDisposable> LockAsync(ArchivePaths archive, Guid sequenceId, TimeSpan? wait, CancellationToken cancellationToken)
{
var name = $"seq-{sequenceId:N}";
var gate = _gates.GetOrAdd(sequenceId, _ => new SemaphoreSlim(1, 1));
if (wait is { } limit)
{
if (!await gate.WaitAsync(limit, cancellationToken).ConfigureAwait(false)) throw new LeaseUnavailableException(name, null);
}
else
{
await gate.WaitAsync(cancellationToken).ConfigureAwait(false);
}
try
{
var lease = await LeaseFile.AcquireAsync(archive.LocksDirectory, name, options.Owner, wait ?? options.LeaseWait, options.LeaseDuration,
cancellationToken).ConfigureAwait(false);
return new Held(gate, lease);
}
catch
{
gate.Release();
throw;
}
}
/// <summary>
/// Reads the sidecar of a locked sequence, completing an interrupted Reorganize of its files (<see cref="FileRenameJournal"/>),
/// undoing an interrupted photo deletion (<see cref="FrameRemovalJournal"/>), undoing or completing an interrupted frame
/// replacement (<see cref="FrameReplaceJournal"/>, a conversion or its undo), settling an interrupted restore from the
/// trash (<see cref="FrameRestoreJournal"/>; left for later while its trash entry is busy) and completing an interrupted
/// ingest batch first. A sidecar a rename recovery rewrites leaves the index behind it until the rename resumes or the
/// next scan reads it.
/// </summary>
/// <param name="heldEntryLease">The trash entry lease the caller holds already (a restore of that entry), if any.</param>
/// <exception cref="InvalidDataException">The sidecar in the folder belongs to another sequence.</exception>
public static SequenceManifest ReadLocked(SequencePaths sequence, Guid sequenceId, string? heldEntryLease = null)
{
var manifest = ManifestStore.ReadSequence(sequence.ManifestPath);
if (manifest.Id != sequenceId)
throw new InvalidDataException($"{sequence.ManifestPath} belongs to sequence {manifest.Id}, not {sequenceId}.");
if (File.Exists(sequence.FileRenameJournalPath))
{
if (FileRenameJournal.Recover(sequence, manifest) is { } renamed)
{
ManifestStore.WriteSequence(sequence.ManifestPath, renamed);
manifest = renamed;
}
FileRenameJournal.Delete(sequence);
}
FrameRemovalJournal.Recover(sequence, manifest);
FrameTrash.RecoverReplace(sequence, manifest, heldEntryLease);
FrameTrash.RecoverRestore(sequence, manifest, heldEntryLease);
if (IngestJournal.Recover(sequence, manifest) is { } recovered)
{
ManifestStore.WriteSequence(sequence.ManifestPath, recovered);
manifest = recovered;
}
IngestJournal.Delete(sequence);
return manifest;
}
/// <summary>
/// The folder of a sequence whose lock the caller holds, as the index has it now. A Reorganize moves a folder and
/// updates its row under the sequence's lease (and its thumbnail lease), so a writer that read the row before taking
/// the lease writes where the sequence is, never into the folder it left, which the write would create again. The
/// folder of <paramref name="sequence"/> when the row is gone (reading its sidecar then fails as it did).
/// </summary>
public static async Task<SequencePaths> FolderAsync(ChronosDatabase db, ArchivePaths archive, Sequence sequence, CancellationToken cancellationToken) =>
archive.Sequence((await db.Sequences.GetAsync(sequence.Id, cancellationToken).ConfigureAwait(false))?.Folder ?? sequence.Folder);
/// <summary>Applies <paramref name="edit"/> to the sidecar under the lock; a null result writes nothing.</summary>
public async Task<SequenceManifest> UpdateAsync(ArchivePaths archive, Guid sequenceId, string folder,
Func<SequenceManifest, SequenceManifest?> edit, CancellationToken cancellationToken)
{
var sequence = archive.Sequence(folder);
await using (await LockAsync(archive, sequenceId, cancellationToken).ConfigureAwait(false))
{
var manifest = ReadLocked(sequence, sequenceId);
if (edit(manifest) is not { } updated) return manifest;
ManifestStore.WriteSequence(sequence.ManifestPath, updated);
return updated;
}
}
private sealed class Held(SemaphoreSlim gate, LeaseFile lease) : IAsyncDisposable
{
public ValueTask DisposeAsync()
{
lease.Dispose();
gate.Release();
return ValueTask.CompletedTask;
}
}
}