fix(server): a verify that hashes photos plans Reorganize again; keep refusals, bound planning

The kept-plan stamp left out whether a planned sequence's photos are hashed.
A verify writes frames.sha256 (and only status and verified_at on the row)
without another frames version, so after verifying never-hashed photos the
same dry run kept answering "verify the sequence first" for up to five
minutes, and its confirmation queued a job certain to fail.

- SequenceRepository.GetStampAsync adds, for every planned sequence, how many
  of its photos have their hash in the index (one grouped query over the
  archive with all, the selected ids in batches of 500 otherwise).
- ReorganizePlanCache keeps a refusal (nested_target, not_found) as it keeps
  a plan: for the same stamp and templates it is the same, and a long plan
  that ends refused after a cut-off is no longer planned again on every retry.
  Failures are still not kept.
- Planning is bounded: at most two plans are made at once, the others wait for
  their turn; a plan still being made 15 minutes after it was asked, let go at
  capacity, or when the server stops is cancelled, and whoever waits for it
  gets 503 plan_dropped. A read that hangs keeps its turn until it returns but
  no longer keeps its waiters.
- The Capacity summary now says what eviction does (the kept plan asked for
  first, else the plan being made asked for first); the stamp, protocol and
  endpoint docs say that a verify hashing photos is a change the plan reads.
- Tests: a scan's photos, verified, are planned again and renamed by the job;
  the stamp follows hashes of planned photos only; refusals kept, failures
  not; turns, eviction and deadline of plans being made, a hung read included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 56582df45453fc555e525f19de7de1803204d12c)
This commit is contained in:
Alby96 committed 2026-10-27 19:16:42 +01:00
1 parent c3756d03c9
commit 1fb040afe6
8 files changed
+320 -58

No files matched your search

@@ -431,9 +431,11 @@ public sealed class SequenceRepository
/// A digest of what a Reorganize of <paramref name="sequenceIds"/> (null: every sequence) plans from in the index of
/// <paramref name="archiveId"/> (owner review 2, R7): every sequence of the archive and its folder, which decide the
/// folders that get " (2)" and what a moving folder carries; of the sequences planned, their title, first capture,
/// camera, lens, photo count and frames version; of the others inside their folders, the title. Another value once one
/// of these changes. Any other write (an upload into a sequence not planned, a verify, a status, tag or note change)
/// leaves it as it was, and so does a change made on disk alone, which the index does not have.
/// camera, lens, photo count, frames version and how many of their photos are hashed (only those get a new name, and a
/// verify hashes photos without another frames version); of the others inside their folders, the title. Another value
/// once one of these changes. Any other write (an upload into a sequence not planned, a verify that hashes no photo of
/// a planned sequence, a status, tag or note change) leaves it as it was, and so does a change made on disk alone, which
/// the index does not have.
/// </summary>
public async Task<string> GetStampAsync(Guid archiveId, IReadOnlyCollection<Guid>? sequenceIds, CancellationToken cancellationToken = default)
{
@@ -443,12 +445,15 @@ public sealed class SequenceRepository
$"{r.GetStringOrNull(2)}\u001f{r.GetDateTimeOrNull(3):O}\u001f{r.GetStringOrNull(4)}\u001f{r.GetStringOrNull(5)}\u001f{r.GetStringOrNull(6)}\u001f" +
$"{r.GetInt32(7)}\u001f{r.GetInt32(8)}")), cancellationToken).ConfigureAwait(false);
var selected = sequenceIds?.ToHashSet();
var hashed = await CountHashedAsync(archiveId, selected, cancellationToken).ConfigureAwait(false);
var planned = new HashSet<string>(rows.Where(r => selected?.Contains(r.Id) != false && r.Folder.Length > 0).Select(r => r.Folder),
StringComparer.OrdinalIgnoreCase);
using var hash = IncrementalHash.CreateHash(HashAlgorithmName.SHA256);
foreach (var row in rows)
{
var read = selected?.Contains(row.Id) != false ? row.Read : Inside(row.Folder) ? row.Title : null;
var read = selected?.Contains(row.Id) != false
? string.Create(CultureInfo.InvariantCulture, $"{row.Read}\u001f{hashed.GetValueOrDefault(row.Id)}")
: Inside(row.Folder) ? row.Title : null;
hash.AppendData(Encoding.UTF8.GetBytes(string.Create(CultureInfo.InvariantCulture, $"{row.Id:N}\u001f{row.Folder}\u001e{read}\n")));
}
return Convert.ToHexStringLower(hash.GetHashAndReset());
@@ -464,6 +469,33 @@ public sealed class SequenceRepository
}
}
/// <summary>
/// How many photos of each of <paramref name="sequenceIds"/> (null: every sequence of <paramref name="archiveId"/>) have
/// their SHA-256 in the index; a sequence without one is not listed.
/// </summary>
private async Task<Dictionary<Guid, long>> CountHashedAsync(Guid archiveId, IReadOnlyCollection<Guid>? sequenceIds, CancellationToken cancellationToken)
{
var counts = new Dictionary<Guid, long>();
if (sequenceIds is null)
{
foreach (var (id, count) in await _db.ListAsync(
"SELECT f.sequence_id, count(*) FROM frames f JOIN sequences s ON s.id = f.sequence_id " +
"WHERE s.archive_id = $1 AND f.sha256 IS NOT NULL GROUP BY f.sequence_id",
[archiveId], r => (r.GetGuid(0), r.GetInt64(1)), cancellationToken).ConfigureAwait(false)) counts[id] = count;
return counts;
}
// 500 ids per statement stays below SQLite's historical 999-variable limit.
foreach (var chunk in sequenceIds.Chunk(500))
{
var p = new SqlParameters();
var ids = string.Join(", ", chunk.Select(id => p.Add(id)));
foreach (var (id, count) in await _db.ListAsync(
$"SELECT sequence_id, count(*) FROM frames WHERE sequence_id IN ({ids}) AND sha256 IS NOT NULL GROUP BY sequence_id",
p.ToArray(), r => (r.GetGuid(0), r.GetInt64(1)), cancellationToken).ConfigureAwait(false)) counts[id] = count;
}
return counts;
}
/// <summary>
/// The sequences of <paramref name="archiveId"/> a rolling verification checks next (owner review 2, R10): those never
/// verified first, then the longest unverified, at most <paramref name="limit"/>. With <paramref name="verifiedBefore"/>