fix(convert): reserve the photos still writing, keep photo ids out of the job row, never copy a source twice

Stage A's space guard reserved, before each photo, only that photo and the
ones after it, so at the end of a sequence it counted a single photo while
up to twice the parallelism were still converting: a volume filling
meanwhile could go into the reserve. It now reserves the photo and every
photo started whose converted file has not landed yet (a file lands whole,
by a rename, before it is verified), so nothing in flight is missed and
nothing landed is counted twice, at the end of a sequence as anywhere.

The confirmed photo ids (about 21 MB for a million photos) lived in the
job's parameters, read by every dry run's busy check, by each sequence turn
of the conversion and by the job lists. They are now written whole, under a
job id chosen before the job is queued, into convert-<job>.photos beside
the journal, and go with it; the row keeps each sequence's digest, so the
job still converts exactly the photos confirmed (without the file, the
whole plan made again must match). The busy check reads only ids, kinds and
targets (and a verify's parameters), never the asking job's row; a
conversion's turn reads only its creation time; the server's job list and
dashboard and the engine's queue peeks read jobs without parameters.

A retried New sequence conversion could make a second copy. The journal now
names the copy as soon as its photos are filed; a retry finishes that copy
whatever changed in the source since, never remakes one deleted since, and
needs no space again for what it holds; and a source another conversion
copied in the same format after this one was confirmed is done with, with
no copy made (new outcome Superseded, with its job log line).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Alby96andClaude Opus 5.5 committed 2026-11-03 09:36:00 +01:00
1 parent b471a33bfc
commit 8ea25228eb
14 files changed
+966 -201

No files matched your search

@@ -17,7 +17,9 @@ namespace Chronos.Data.Archiving.Jobs;
/// one when the source had it, thumbnails and proxies) are queued as follow-ups, as soon as it is done. Sequences left as
/// they were (changed after the plan, busy, a photo another program holds, a photo that failed) are logged and fail the
/// job at the end, after the others are converted; a retry tries again those busy, held or failed, with what they staged,
/// and leaves one that changed after the plan as it is (its log line says to plan again).
/// and leaves one that changed after the plan as it is (its log line says to plan again). A retry of a New sequence
/// conversion finishes the copies it made and makes none of a source another conversion copied in the same format since
/// this job was queued (its creation time: when the plan was confirmed).
/// </summary>
internal sealed class ConvertJobHandler(ChronosDatabase db, MassConverter converter, Func<Guid, string, CancellationToken, Task>? sequenceChanged) : IJobHandler
{
@@ -39,7 +41,8 @@ internal sealed class ConvertJobHandler(ChronosDatabase db, MassConverter conver
var archive = await db.Archives.GetAsync(parameters.ArchiveId, context.Cancellation).ConfigureAwait(false)
?? throw new InvalidOperationException($"Convert job {context.Job.Id} does not name a known archive.");
var name = context.Job.CreatedBy is { } user ? (await db.Users.GetAsync(user, context.Cancellation).ConfigureAwait(false))?.Username : null;
var result = await converter.RunAsync(archive, parameters, context.Job.Id, new ConvertRequester(name, context.Job.CreatedBy), new ConvertCallbacks(
var by = new ConvertRequester(name, context.Job.CreatedBy, context.Job.Created);
var result = await converter.RunAsync(archive, parameters, context.Job.Id, by, new ConvertCallbacks(
(done, total) => context.ReportAsync(done, total),
(done, total) => context.CheckpointAsync(Checkpoint, done, total),
context.Log, TellAsync, requests => QueueAsync(context, requests)), context.Cancellation).ConfigureAwait(false);