fix(sync,upload): keep tombstones honest and let an explicit upload bring a photo back
Sync: - A server frame that has not been hashed is known by content only once it is downloaded: one this PC deleted (or holds as its replacement) is now deleted from staging instead of filed again, which also dropped its tombstone. - An unhashed local original that turns out to be a photo the server deleted is not uploaded, and the archive verify then writes the hashes into the index, so the row settles instead of offering Upload for good. - A tombstone leaves a photo out of the comparison only when its side no longer holds that content, so a tombstone left beside a held copy does not keep the other side's copy missing forever. Uploads: - StartUploadRequest.Restore (optional, never sent by sync or older clients) lets an explicit user upload file a photo deleted from the sequence again, also after the trash was emptied; the completion repeats it with ?restore=true and the ingest drops the tombstone. - The completion checks the tombstone again: a photo deleted while its bytes were on the way is not filed (409 photo_removed, upload cancelled); the protocol client reports that as not sent. - A tombstone of content a frame of the target still holds refuses no repair. - Hidden dot names (AppleDouble "._" files) are refused, as the archive scanner leaves them out. Web upload dialog: - Upload again on rows marked Deleted before; the tooltip no longer promises a trash restore that may be impossible. - Sends a capture time: the Exif DateTimeOriginal of a JPEG or TIFF-based file (minimal in-house reader, capture-time.js), else the file's time, so photos without Exif keep their order. - Esc is stopped at keydown during a run (Chromium allows one prevented cancel per activation); hidden dot files are left out; a completion refused while another one files the upload waits for it; the same photo picked under two names waits for its first copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit d051bcfc351f5a08500ca864a6508576a95492a4)
This commit is contained in:
1 parent
eb607d47aa
commit
316c182b19
18 files changed
+753
-63
No files matched your search
@@ -88,7 +88,7 @@ Dependencies point one way: `Core` ← `Core.Windows`, `Data`, `Protocol` ← `S
|
||||
| `Server/Api`, `Server/Events` | Endpoint groups (`XxxEndpoints.Map`; `ScheduleEndpoints`: `/api/v1/schedules`; `LibraryEndpoints`: the Library filters' facets, sequence thumbnail, photo delete, bulk delete/retag/jobs; `TrashEndpoints`: list, restore, empty), API errors, the WebSocket `EventHub` |
|
||||
| `Server/Security` | Accounts, password hashing, tokens, session cookies, login guard, access rules, audit log |
|
||||
| `Server/Storage`, `Server/Jobs` | Archive and processing modules, uploads (`UploadMaintenance`: recovery at start, and the sweep that is the `uploads` clean-up step), render store; `JobEngineHost`, `SchedulerHost` (the `Scheduler` on the engine, started after it; archive.json read every 10 minutes whatever the scan schedule does; every schedule change and run published as a `scheduleChanged` event) and `ServerScheduleHost` (time zone, archive refreshed from archive.json, system schedules), render node service, monitor and availability |
|
||||
| `Server/Hosting`, `Server/Admin/web` | `ServerHost`, `ServerLogging` (console and file levels the logging filters read at every line, the file sink's limits, the engine's level; applied again when Settings › Logs is saved), certificates, discovery beacon, embedded static files (`EmbeddedStaticFiles`: `no-cache` and a content ETag, so an upgrade never mixes old and new modules), Windows service (start and stop failures in the Application event log); the admin panel (ES modules, one per page; `schedules.js` is the Jobs page's Scheduled tab; the Library's views in `pages/library/`: `sequence.js` a sequence and its photo viewer, `trash.js` the archive's trash, `shared.js` cards, selection and grid keyboard; `upload.js` the upload dialog the Library opens on demand, hashing in the module worker `hash-worker.js` with the in-house incremental SHA-256 of `sha256.js`, self-tested before its first use) |
|
||||
| `Server/Hosting`, `Server/Admin/web` | `ServerHost`, `ServerLogging` (console and file levels the logging filters read at every line, the file sink's limits, the engine's level; applied again when Settings › Logs is saved), certificates, discovery beacon, embedded static files (`EmbeddedStaticFiles`: `no-cache` and a content ETag, so an upgrade never mixes old and new modules), Windows service (start and stop failures in the Application event log); the admin panel (ES modules, one per page; `schedules.js` is the Jobs page's Scheduled tab; the Library's views in `pages/library/`: `sequence.js` a sequence and its photo viewer, `trash.js` the archive's trash, `shared.js` cards, selection and grid keyboard; `upload.js` the upload dialog the Library opens on demand, hashing in the module worker `hash-worker.js` with the in-house incremental SHA-256 of `sha256.js`, self-tested before its first use, and sending the capture time `capture-time.js` reads: the Exif DateTimeOriginal of a JPEG or TIFF-based file, else the file's time) |
|
||||
|
||||
### 2.5 Hosts
|
||||
|
||||
|
||||
+4
-2
@@ -781,6 +781,8 @@
|
||||
<data name="Admin.Trash.Title" xml:space="preserve"><value>Trash</value></data>
|
||||
<data name="Admin.Upload.Add" xml:space="preserve"><value>Add photos</value></data>
|
||||
<data name="Admin.Upload.Add.Tip" xml:space="preserve"><value>Choose photos on this computer; you can also drop them on this dialog</value></data>
|
||||
<data name="Admin.Upload.Again" xml:space="preserve"><value>Upload {0} again</value></data>
|
||||
<data name="Admin.Upload.Again.Tip" xml:space="preserve"><value>Upload again: add this photo to the sequence again, although it was deleted from it; Upload sends it</value></data>
|
||||
<data name="Admin.Upload.Bytes" xml:space="preserve"><value>{0} of {1}</value></data>
|
||||
<data name="Admin.Upload.Cancel" xml:space="preserve"><value>Cancel</value></data>
|
||||
<data name="Admin.Upload.Cancel.Tip" xml:space="preserve"><value>Stop now: the photos not finished yet are discarded on the server</value></data>
|
||||
@@ -812,14 +814,14 @@
|
||||
<data name="Admin.Upload.Skipped.Folders" xml:space="preserve"><value>Folders skipped: {0}</value></data>
|
||||
<data name="Admin.Upload.Skipped.Folders.Tip" xml:space="preserve"><value>Folders are imported with the desktop app; here only photos are uploaded</value></data>
|
||||
<data name="Admin.Upload.Skipped.Formats" xml:space="preserve"><value>Not photos: {0}</value></data>
|
||||
<data name="Admin.Upload.Skipped.Formats.Tip" xml:space="preserve"><value>Files in a format the server does not read as photos were left out</value></data>
|
||||
<data name="Admin.Upload.Skipped.Formats.Tip" xml:space="preserve"><value>Hidden files and files in a format the server does not read as photos were left out</value></data>
|
||||
<data name="Admin.Upload.Skipped.Size" xml:space="preserve"><value>Too large: {0}</value></data>
|
||||
<data name="Admin.Upload.Skipped.Size.Tip" xml:space="preserve"><value>Files larger than the server's upload limit ({0}) were left out</value></data>
|
||||
<data name="Admin.Upload.Start" xml:space="preserve"><value>Upload</value></data>
|
||||
<data name="Admin.Upload.Start.Tip" xml:space="preserve"><value>Send the photos not on the server yet; the server checks each one against its SHA-256</value></data>
|
||||
<data name="Admin.Upload.State.Cancelled" xml:space="preserve"><value>Cancelled</value></data>
|
||||
<data name="Admin.Upload.State.Deleted" xml:space="preserve"><value>Deleted before</value></data>
|
||||
<data name="Admin.Upload.State.Deleted.Tip" xml:space="preserve"><value>Deleted from this sequence earlier: restore it from the trash to bring it back</value></data>
|
||||
<data name="Admin.Upload.State.Deleted.Tip" xml:space="preserve"><value>Deleted from this sequence earlier: restore it from the trash while it is there, or upload it again</value></data>
|
||||
<data name="Admin.Upload.State.Done" xml:space="preserve"><value>Uploaded</value></data>
|
||||
<data name="Admin.Upload.State.Done.Tip" xml:space="preserve"><value>On the server and verified against its SHA-256</value></data>
|
||||
<data name="Admin.Upload.State.Exists" xml:space="preserve"><value>Already there</value></data>
|
||||
|
||||
@@ -73,6 +73,7 @@
|
||||
<data name="Server.Error.NotFound" xml:space="preserve"><value>Not found.</value><comment>long</comment></data>
|
||||
<data name="Server.Error.NotChronos" xml:space="preserve"><value>That address is not a Chronos Server.</value><comment>long</comment></data>
|
||||
<data name="Server.Error.PasswordChangeRequired" xml:space="preserve"><value>Change your password first.</value><comment>long</comment></data>
|
||||
<data name="Server.Error.PhotoRemoved" xml:space="preserve"><value>The photo was deleted from the sequence while it was being uploaded, so it was not added.</value><comment>long</comment></data>
|
||||
<data name="Server.Error.ProxyUntrusted" xml:space="preserve"><value>This page did not come through a trusted proxy over HTTPS. Open it through your proxy, or add the proxy's address to CHRONOS_TRUSTED_PROXIES.</value><comment>long</comment></data>
|
||||
<data name="Server.Error.ProxiesDisabled" xml:space="preserve"><value>This archive makes no proxies.</value><comment>long</comment></data>
|
||||
<data name="Server.Error.ProxyMismatch" xml:space="preserve"><value>The proxy was made from another photo.</value><comment>long</comment></data>
|
||||
|
||||
@@ -778,6 +778,8 @@
|
||||
<data name="Admin.Trash.Title" xml:space="preserve"><value>Cestino</value></data>
|
||||
<data name="Admin.Upload.Add" xml:space="preserve"><value>Aggiungi foto</value></data>
|
||||
<data name="Admin.Upload.Add.Tip" xml:space="preserve"><value>Scegli le foto su questo computer; puoi anche trascinarle in questa finestra</value></data>
|
||||
<data name="Admin.Upload.Again" xml:space="preserve"><value>Carica di nuovo {0}</value></data>
|
||||
<data name="Admin.Upload.Again.Tip" xml:space="preserve"><value>Carica di nuovo: aggiungi questa foto alla sequenza anche se ne era stata eliminata; Carica la invia</value></data>
|
||||
<data name="Admin.Upload.Bytes" xml:space="preserve"><value>{0} di {1}</value></data>
|
||||
<data name="Admin.Upload.Cancel" xml:space="preserve"><value>Annulla</value></data>
|
||||
<data name="Admin.Upload.Cancel.Tip" xml:space="preserve"><value>Ferma ora: le foto non ancora finite vengono scartate sul server</value></data>
|
||||
@@ -809,14 +811,14 @@
|
||||
<data name="Admin.Upload.Skipped.Folders" xml:space="preserve"><value>Cartelle saltate: {0}</value></data>
|
||||
<data name="Admin.Upload.Skipped.Folders.Tip" xml:space="preserve"><value>Le cartelle si importano con l'app desktop; qui si caricano solo foto</value></data>
|
||||
<data name="Admin.Upload.Skipped.Formats" xml:space="preserve"><value>Non foto: {0}</value></data>
|
||||
<data name="Admin.Upload.Skipped.Formats.Tip" xml:space="preserve"><value>Esclusi i file in un formato che il server non legge come foto</value></data>
|
||||
<data name="Admin.Upload.Skipped.Formats.Tip" xml:space="preserve"><value>Esclusi i file nascosti e quelli in un formato che il server non legge come foto</value></data>
|
||||
<data name="Admin.Upload.Skipped.Size" xml:space="preserve"><value>Troppo grandi: {0}</value></data>
|
||||
<data name="Admin.Upload.Skipped.Size.Tip" xml:space="preserve"><value>Esclusi i file oltre il limite di caricamento del server ({0})</value></data>
|
||||
<data name="Admin.Upload.Start" xml:space="preserve"><value>Carica</value></data>
|
||||
<data name="Admin.Upload.Start.Tip" xml:space="preserve"><value>Invia le foto non ancora sul server; il server controlla ognuna con il suo SHA-256</value></data>
|
||||
<data name="Admin.Upload.State.Cancelled" xml:space="preserve"><value>Annullata</value></data>
|
||||
<data name="Admin.Upload.State.Deleted" xml:space="preserve"><value>Eliminata prima</value></data>
|
||||
<data name="Admin.Upload.State.Deleted.Tip" xml:space="preserve"><value>Eliminata in precedenza da questa sequenza: ripristinala dal cestino per riaverla</value></data>
|
||||
<data name="Admin.Upload.State.Deleted.Tip" xml:space="preserve"><value>Eliminata in precedenza da questa sequenza: ripristinala dal cestino finché c'è, o caricala di nuovo</value></data>
|
||||
<data name="Admin.Upload.State.Done" xml:space="preserve"><value>Caricata</value></data>
|
||||
<data name="Admin.Upload.State.Done.Tip" xml:space="preserve"><value>Sul server e verificata con il suo SHA-256</value></data>
|
||||
<data name="Admin.Upload.State.Exists" xml:space="preserve"><value>Già presente</value></data>
|
||||
|
||||
@@ -71,6 +71,7 @@
|
||||
<data name="Server.Error.NotFound" xml:space="preserve"><value>Non trovato.</value></data>
|
||||
<data name="Server.Error.NotChronos" xml:space="preserve"><value>Questo indirizzo non è un Chronos Server.</value></data>
|
||||
<data name="Server.Error.PasswordChangeRequired" xml:space="preserve"><value>Cambia prima la password.</value></data>
|
||||
<data name="Server.Error.PhotoRemoved" xml:space="preserve"><value>La foto è stata eliminata dalla sequenza mentre veniva caricata, quindi non è stata aggiunta.</value><comment>long</comment></data>
|
||||
<data name="Server.Error.ProxyUntrusted" xml:space="preserve"><value>Questa pagina non è arrivata da un proxy attendibile in HTTPS. Aprila tramite il proxy o aggiungi l'indirizzo del proxy a CHRONOS_TRUSTED_PROXIES.</value></data>
|
||||
<data name="Server.Error.ProxiesDisabled" xml:space="preserve"><value>Questo archivio non crea proxy.</value></data>
|
||||
<data name="Server.Error.ProxyMismatch" xml:space="preserve"><value>Il proxy è stato creato da un'altra foto.</value></data>
|
||||
|
||||
@@ -167,7 +167,9 @@ public static class SyncRules
|
||||
/// <summary>
|
||||
/// <see cref="CompareOriginals(IReadOnlyCollection{FramePhoto}, IReadOnlyCollection{FramePhoto})"/> honouring the photos
|
||||
/// either side removed (owner review 2, R6): a photo the server deleted is not missing there, one this PC deleted is not
|
||||
/// missing here, and a photo replaced by a converted file on either side is compared as its replacement.
|
||||
/// missing here, and a photo replaced by a converted file on either side is compared as its replacement. A tombstone
|
||||
/// leaves out only a photo its side no longer holds: one left beside a frame of the same content (the other copy of a
|
||||
/// duplicate, a file put back by hand) would otherwise keep the other side's copy missing for good.
|
||||
/// </summary>
|
||||
public static (int MissingRemote, int MissingLocal) CompareOriginals(IReadOnlyCollection<FramePhoto> local, IReadOnlyCollection<FramePhoto> remote,
|
||||
SyncTombstones tombstones)
|
||||
@@ -176,8 +178,12 @@ public static class SyncRules
|
||||
ArgumentNullException.ThrowIfNull(remote);
|
||||
ArgumentNullException.ThrowIfNull(tombstones);
|
||||
if (tombstones.IsEmpty) return CompareOriginals(local, remote);
|
||||
return CompareOriginals([.. local.Where(p => !tombstones.DeletedRemotely(p.Sha256)).Select(tombstones.Canonical)],
|
||||
[.. remote.Where(p => !tombstones.DeletedLocally(p.Sha256)).Select(tombstones.Canonical)]);
|
||||
var mine = local.Select(p => (p.Sha256, Photo: tombstones.Canonical(p))).ToList();
|
||||
var theirs = remote.Select(p => (p.Sha256, Photo: tombstones.Canonical(p))).ToList();
|
||||
var minePhotos = new FramePhotos(mine.Select((p, i) => (i, p.Photo)));
|
||||
var theirPhotos = new FramePhotos(theirs.Select((p, i) => (i, p.Photo)));
|
||||
return CompareOriginals([.. mine.Where(p => !tombstones.DeletedRemotely(p.Sha256) || theirPhotos.IndexOf(p.Photo) is not null).Select(p => p.Photo)],
|
||||
[.. theirs.Where(p => !tombstones.DeletedLocally(p.Sha256) || minePhotos.IndexOf(p.Photo) is not null).Select(p => p.Photo)]);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -70,8 +70,8 @@ internal sealed class SyncTransfers(SyncService sync)
|
||||
// server has filed one, which creates the sequence (an original it already holds elsewhere creates nothing).
|
||||
while (remoteId is null && pending.TryDequeue(out var first))
|
||||
{
|
||||
var sent = await UploadFrameAsync(context, remote, null, SyncService.SessionKey(local.Id), paths.FromRelative(first.RelativePath), first, hashes, ct)
|
||||
.ConfigureAwait(false);
|
||||
var sent = await UploadFrameAsync(context, remote, null, SyncService.SessionKey(local.Id), paths.FromRelative(first.RelativePath), first, tombstones,
|
||||
hashes, ct).ConfigureAwait(false);
|
||||
if (sent is null) skipped++;
|
||||
else if (sent.SequenceId != Guid.Empty)
|
||||
{
|
||||
@@ -85,11 +85,15 @@ internal sealed class SyncTransfers(SyncService sync)
|
||||
|
||||
await Parallel.ForEachAsync(pending, Workers(sync.Options.Uploads, ct), async (frame, token) =>
|
||||
{
|
||||
if (await UploadFrameAsync(context, remote, target, null, paths.FromRelative(frame.RelativePath), frame, hashes, token).ConfigureAwait(false) is null)
|
||||
Interlocked.Increment(ref skipped);
|
||||
var sent = await UploadFrameAsync(context, remote, target, null, paths.FromRelative(frame.RelativePath), frame, tombstones, hashes, token)
|
||||
.ConfigureAwait(false);
|
||||
if (sent is null) Interlocked.Increment(ref skipped);
|
||||
await progress.StepAsync().ConfigureAwait(false);
|
||||
}).ConfigureAwait(false);
|
||||
await SendProxiesAsync(context, remote, target, local, paths, frames, hashes).ConfigureAwait(false);
|
||||
// An original the index has no hash for turned out to be a photo the server deleted: the verify gives the index
|
||||
// the hashes (as before a download), so the comparison leaves it out instead of offering its upload again and again.
|
||||
if (hashes.Values.Any(tombstones.DeletedRemotely)) await VerifyAsync(archive, local.Id, ct).ConfigureAwait(false);
|
||||
|
||||
if (created) await sync.PushAllAsync(remote, target, ct).ConfigureAwait(false);
|
||||
await sync.RecompareAsync(remote, target, ct).ConfigureAwait(false);
|
||||
@@ -240,7 +244,7 @@ internal sealed class SyncTransfers(SyncService sync)
|
||||
await progress.StepAsync().ConfigureAwait(false);
|
||||
}
|
||||
var sequenceId = local?.Id;
|
||||
var restored = 0;
|
||||
int restored = 0, refused = 0;
|
||||
await Parallel.ForEachAsync(rest, Workers(sync.Options.Downloads, ct), async (frame, token) =>
|
||||
{
|
||||
var (staged, sha) = await FetchOriginalAsync(remote, frame, staging, token).ConfigureAwait(false);
|
||||
@@ -249,6 +253,14 @@ internal sealed class SyncTransfers(SyncService sync)
|
||||
Restore(staged, place);
|
||||
Interlocked.Increment(ref restored);
|
||||
}
|
||||
else if (frame.Sha256.Length == 0 && (tombstones.DeletedLocally(sha) || have.Contains(tombstones.Canonical(sha)!)))
|
||||
{
|
||||
// An original the server has not hashed is known by content only now: one this PC deleted (or holds as
|
||||
// the converted file that replaced it) is not filed again, which would also drop its tombstone.
|
||||
File.Delete(staged);
|
||||
Interlocked.Increment(ref refused);
|
||||
if (tombstones.DeletedLocally(sha)) context.Log($"{frame.FileName}: deleted on this PC, not downloaded");
|
||||
}
|
||||
else
|
||||
{
|
||||
await CommitAsync(remoteId, archive.Id, sequenceId, frame, staged, sha, captured, token).ConfigureAwait(false);
|
||||
@@ -261,7 +273,7 @@ internal sealed class SyncTransfers(SyncService sync)
|
||||
|
||||
if (created) await sync.TakeAllAsync(remote, remoteId, ct).ConfigureAwait(false);
|
||||
await sync.RecompareAsync(remote, remoteId, ct).ConfigureAwait(false);
|
||||
if (sequenceId is { } downloaded && wanted.Count > restored)
|
||||
if (sequenceId is { } downloaded && wanted.Count > restored + refused)
|
||||
{
|
||||
var label = local!.DisplayTitle;
|
||||
await EnqueueFollowUpAsync(context, ArchiveJobs.Thumbnails([downloaded], label)).ConfigureAwait(false);
|
||||
@@ -497,10 +509,11 @@ internal sealed class SyncTransfers(SyncService sync)
|
||||
/// <summary>
|
||||
/// Hashes the original again and sends it into <paramref name="sequenceId"/> (or the sequence of
|
||||
/// <paramref name="sessionKey"/>); null (logged) when it is gone or no longer the archived file. The hash of an
|
||||
/// original the index has none for goes into <paramref name="hashes"/>.
|
||||
/// original the index has none for goes into <paramref name="hashes"/>; known by content only now, such an original
|
||||
/// the server deleted (<paramref name="tombstones"/>) is not sent back.
|
||||
/// </summary>
|
||||
private static async Task<CompleteUploadResponse?> UploadFrameAsync(JobContext context, ISyncRemote remote, Guid? sequenceId, string? sessionKey, string path,
|
||||
Frame frame, ConcurrentDictionary<Guid, string> hashes, CancellationToken cancellationToken)
|
||||
Frame frame, SyncTombstones tombstones, ConcurrentDictionary<Guid, string> hashes, CancellationToken cancellationToken)
|
||||
{
|
||||
if (!File.Exists(path))
|
||||
{
|
||||
@@ -508,14 +521,22 @@ internal sealed class SyncTransfers(SyncService sync)
|
||||
return null;
|
||||
}
|
||||
var sha = await ChronosClient.HashFileAsync(path, cancellationToken).ConfigureAwait(false);
|
||||
if (frame.Sha256 is null) hashes[frame.Id] = sha;
|
||||
var notSent = new CompleteUploadResponse(Guid.Empty, sequenceId ?? Guid.Empty, -1);
|
||||
if (frame.Sha256 is null)
|
||||
{
|
||||
hashes[frame.Id] = sha;
|
||||
if (tombstones.DeletedRemotely(sha))
|
||||
{
|
||||
context.Log($"{frame.FileName}: deleted on the server, not uploaded");
|
||||
return notSent;
|
||||
}
|
||||
}
|
||||
if (frame.Sha256 is { } archived && !archived.Equals(sha, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
context.Log($"{frame.FileName}: changed since it was archived, not uploaded");
|
||||
return null;
|
||||
}
|
||||
return await remote.UploadFileAsync(path, sequenceId, sessionKey, sha, frame.CapturedAt, cancellationToken).ConfigureAwait(false)
|
||||
?? new CompleteUploadResponse(Guid.Empty, sequenceId ?? Guid.Empty, -1);
|
||||
return await remote.UploadFileAsync(path, sequenceId, sessionKey, sha, frame.CapturedAt, cancellationToken).ConfigureAwait(false) ?? notSent;
|
||||
}
|
||||
|
||||
/// <summary>Commits one original from staging into the local sequence (created from the server id when null).</summary>
|
||||
|
||||
@@ -165,7 +165,9 @@ public sealed class ChronosClient : IDisposable
|
||||
|
||||
/// <summary>
|
||||
/// Uploads one file with resumable, idempotent chunks; the server verifies the SHA-256 before committing it
|
||||
/// (BRIEF §14.4). Returns the committed frame id. Transient failures retry with exponential backoff.
|
||||
/// (BRIEF §14.4). Returns the committed frame id; null when the server holds the photo without a frame to name: it was
|
||||
/// deleted from the target sequence, before the upload or while it was on its way (owner review 2, R6). Transient
|
||||
/// failures retry with exponential backoff.
|
||||
/// </summary>
|
||||
public async Task<CompleteUploadResponse?> UploadFileAsync(
|
||||
string path,
|
||||
@@ -189,7 +191,15 @@ public sealed class ChronosClient : IDisposable
|
||||
|
||||
var uploadId = start.UploadId ?? throw new InvalidOperationException("Server returned no upload id.");
|
||||
await SendChunksAsync(path, info.Length, uploadId, start.ChunkSize, start.Received, progress, ct).ConfigureAwait(false);
|
||||
return await RetryAsync(() => SendAsync<CompleteUploadResponse>(HttpMethod.Post, $"api/v1/uploads/{uploadId}/complete", ProtocolJson.Default.CompleteUploadResponse, ct), ct).ConfigureAwait(false);
|
||||
try
|
||||
{
|
||||
return await RetryAsync(() => SendAsync<CompleteUploadResponse>(HttpMethod.Post, $"api/v1/uploads/{uploadId}/complete", ProtocolJson.Default.CompleteUploadResponse, ct), ct)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
catch (ChronosApiException ex) when (ex is { Status: HttpStatusCode.Conflict, Error.Code: UploadErrors.PhotoRemoved })
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -2,12 +2,26 @@ namespace Chronos.Protocol;
|
||||
|
||||
/// <summary>
|
||||
/// <c>POST /api/v1/uploads</c> (BRIEF §14.4). The client hashes the file first; when the server already holds that
|
||||
/// SHA-256 it answers 200 with <see cref="StartUploadResponse.AlreadyExists"/> and no upload is needed.
|
||||
/// SHA-256 it answers 200 with <see cref="StartUploadResponse.AlreadyExists"/> and no upload is needed. A photo deleted
|
||||
/// from the target sequence is answered the same way, without a frame (owner review 2, R6), unless <see cref="Restore"/>
|
||||
/// asks to file it there again: only an explicit user action sends it (the admin panel's "Upload again"), never sync or a
|
||||
/// watch folder, and the completion repeats it (<c>complete?restore=true</c>). Filing it drops the photo's tombstone.
|
||||
/// </summary>
|
||||
public sealed record StartUploadRequest(Guid? SequenceId, string FileName, long Size, string Sha256, DateTime? CapturedAt, string? SessionKey);
|
||||
public sealed record StartUploadRequest(Guid? SequenceId, string FileName, long Size, string Sha256, DateTime? CapturedAt, string? SessionKey,
|
||||
bool Restore = false);
|
||||
|
||||
public sealed record StartUploadResponse(Guid? UploadId, int ChunkSize, IReadOnlyList<int> Received, bool AlreadyExists, Guid? FrameId);
|
||||
|
||||
/// <summary>Error codes of the frame upload calls that clients act on.</summary>
|
||||
public static class UploadErrors
|
||||
{
|
||||
/// <summary>
|
||||
/// 409 from <c>POST /api/v1/uploads/{id}/complete</c>: the photo was deleted from the target sequence while it was on its
|
||||
/// way and the completion did not ask to restore it, so it was not filed and the upload is cancelled (owner review 2, R6).
|
||||
/// </summary>
|
||||
public const string PhotoRemoved = "photo_removed";
|
||||
}
|
||||
|
||||
/// <summary><c>PUT /api/v1/uploads/{id}/chunks/{n}</c> response.</summary>
|
||||
public sealed record ChunkReceived(int Index, IReadOnlyList<int> Received);
|
||||
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
// The capture time the upload dialog sends with a photo (owner review 2, R3), as local wall time without an offset
|
||||
// ("2025-05-01T06:30:00", the way the desktop clients send it): the Exif DateTimeOriginal of a JPEG or of a TIFF-based file
|
||||
// (TIFF, DNG and most RAW formats), else the file's modification time. The server keeps the time a photo records itself
|
||||
// and uses this one only for a photo that records none (a PNG, an export stripped of its Exif data), so such photos keep
|
||||
// their order instead of taking the moment they arrived. The reader is minimal: it follows one IFD pointer, checks every
|
||||
// offset against the bytes it read and gives up (null) on anything it does not expect.
|
||||
|
||||
/** Bytes read from the start of a file: every camera writes the Exif block within them. */
|
||||
const READ = 256 * 1024;
|
||||
|
||||
/** The capture time of a picked File; null when neither the photo nor the file gives a usable one. */
|
||||
export async function captureTime(file) {
|
||||
try {
|
||||
const taken = exifCaptureTime(new Uint8Array(await file.slice(0, READ).arrayBuffer()));
|
||||
if (taken) return taken;
|
||||
} catch {
|
||||
// Unreadable here: the file's own time stands in, and the server reads the photo again anyway.
|
||||
}
|
||||
const modified = file.lastModified;
|
||||
return Number.isFinite(modified) && modified > 0 ? localIso(new Date(modified)) : null;
|
||||
}
|
||||
|
||||
/** The Exif DateTimeOriginal of the JPEG or TIFF whose first bytes are given, as "yyyy-MM-ddTHH:mm:ss"; null when there is none. */
|
||||
export function exifCaptureTime(bytes) {
|
||||
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
|
||||
const size = view.byteLength;
|
||||
let tiff = -1;
|
||||
if (size >= 4 && view.getUint16(0) === 0xffd8) {
|
||||
// JPEG: the segments before the image data, up to the APP1 that starts with "Exif\0\0".
|
||||
for (let at = 2; at + 4 <= size;) {
|
||||
const marker = view.getUint16(at);
|
||||
if ((marker & 0xff00) !== 0xff00 || marker === 0xffda) break;
|
||||
const length = view.getUint16(at + 2);
|
||||
if (marker === 0xffe1 && length >= 16 && at + 10 <= size && view.getUint32(at + 4) === 0x45786966 && view.getUint16(at + 8) === 0) {
|
||||
tiff = at + 10;
|
||||
break;
|
||||
}
|
||||
if (length < 2) break;
|
||||
at += 2 + length;
|
||||
}
|
||||
} else {
|
||||
tiff = 0;
|
||||
}
|
||||
if (tiff < 0 || tiff + 8 > size) return null;
|
||||
const order = view.getUint16(tiff);
|
||||
if (order !== 0x4949 && order !== 0x4d4d) return null;
|
||||
const little = order === 0x4949;
|
||||
const u16 = (at) => view.getUint16(at, little);
|
||||
const u32 = (at) => view.getUint32(at, little);
|
||||
if (u16(tiff + 2) !== 42) return null;
|
||||
|
||||
// The 12-byte entry of a tag in the IFD at an offset from the TIFF header; -1 when it is not within reach.
|
||||
const entry = (ifd, tag) => {
|
||||
const at = tiff + ifd;
|
||||
if (ifd < 8 || at + 2 > size) return -1;
|
||||
const count = u16(at);
|
||||
for (let i = 0; i < count; i++) {
|
||||
const e = at + 2 + i * 12;
|
||||
if (e + 12 > size) return -1;
|
||||
if (u16(e) === tag) return e;
|
||||
}
|
||||
return -1;
|
||||
};
|
||||
const exif = entry(u32(tiff + 4), 0x8769);
|
||||
if (exif < 0) return null;
|
||||
const original = entry(u32(exif + 8), 0x9003);
|
||||
// ASCII "YYYY:MM:DD HH:MM:SS" and its terminator, stored after the IFD.
|
||||
if (original < 0 || u16(original + 2) !== 2 || u32(original + 4) < 19) return null;
|
||||
const text = tiff + u32(original + 8);
|
||||
if (text + 19 > size) return null;
|
||||
let value = "";
|
||||
for (let i = 0; i < 19; i++) value += String.fromCharCode(view.getUint8(text + i));
|
||||
const m = /^(\d{4}):(\d{2}):(\d{2}) (\d{2}):(\d{2}):(\d{2})$/.exec(value);
|
||||
if (!m) return null;
|
||||
const [year, month, day, hour, minute, second] = m.slice(1).map(Number);
|
||||
// A real date and time only ("0000:00:00 00:00:00" is what cameras write without a clock): checked in UTC, where every
|
||||
// wall time exists.
|
||||
const check = new Date(Date.UTC(year, month - 1, day, hour, minute, second));
|
||||
if (year < 1900 || check.getUTCFullYear() !== year || check.getUTCMonth() !== month - 1 || check.getUTCDate() !== day ||
|
||||
check.getUTCHours() !== hour || check.getUTCMinutes() !== minute || check.getUTCSeconds() !== second) return null;
|
||||
return `${m[1]}-${m[2]}-${m[3]}T${m[4]}:${m[5]}:${m[6]}`;
|
||||
}
|
||||
|
||||
/** Local wall time with milliseconds and no offset; null outside the years a server date takes. */
|
||||
export function localIso(date) {
|
||||
const year = date.getFullYear();
|
||||
if (!(year >= 1 && year <= 9999)) return null;
|
||||
const p = (n, width = 2) => String(n).padStart(width, "0");
|
||||
return `${p(year, 4)}-${p(date.getMonth() + 1)}-${p(date.getDate())}T${p(date.getHours())}:${p(date.getMinutes())}:${p(date.getSeconds())}` +
|
||||
`.${p(date.getMilliseconds(), 3)}`;
|
||||
}
|
||||
@@ -1,18 +1,22 @@
|
||||
// Upload dialog of the admin panel (owner review 2, R3; BRIEF §14.4, §14.8): photos from this computer into an existing
|
||||
// or a new sequence, chosen with the file picker or dropped on the dialog. Folders are not taken: importing a whole folder
|
||||
// stays a feature of the desktop app. Each photo is hashed in a module worker (hash-worker.js, which self-tests its
|
||||
// SHA-256 first and refuses to hash when a check fails), started, sent in the chunks the server does not have yet and
|
||||
// completed; the server verifies the hash again before it files the photo.
|
||||
// SHA-256 first and refuses to hash when a check fails), started with its capture time (capture-time.js), sent in the
|
||||
// chunks the server does not have yet and completed; the server verifies the hash again before it files the photo. A
|
||||
// photo deleted from the sequence before is not filed again unless the user asks for it on its row (Upload again).
|
||||
import { ApiFailure } from "./api.js";
|
||||
import { t, plural, applyStrings, culture } from "./i18n.js";
|
||||
import { escapeHtml, attr, bar, formatBytes, formatNumber, errorText } from "./ui.js";
|
||||
import { icons } from "./icons.js";
|
||||
import { captureTime } from "./capture-time.js";
|
||||
|
||||
/** Photos sent at once, and chunks in flight for each: four requests, half of what the server takes from one account. */
|
||||
const FILES_AT_ONCE = 2;
|
||||
const CHUNKS_AT_ONCE = 2;
|
||||
/** Retries of a call that failed for a passing reason (no answer, 429, 503), after 1, 2 and 4 seconds. */
|
||||
const RETRIES = 3;
|
||||
/** Checks of an upload another completion is filing, after 1, 2, 4 and then every 8 seconds: about a minute in all. */
|
||||
const SETTLE_CHECKS = 10;
|
||||
const SEARCH_DELAY = 250;
|
||||
const LIST_LIMIT = 50;
|
||||
/** States of a photo already on the server: a later run leaves it alone. */
|
||||
@@ -48,6 +52,8 @@ export function openUpload({ api, sequence, onDone } = {}) {
|
||||
let searchTimer = null;
|
||||
let searchToken = 0;
|
||||
let frame = 0;
|
||||
// Per run: by SHA-256, the send of the last photo of that content, which the next copy waits for.
|
||||
let sameContent = new Map();
|
||||
const items = [];
|
||||
const skipped = { folders: 0, formats: 0, size: 0 };
|
||||
const titles = new Map();
|
||||
@@ -103,6 +109,17 @@ export function openUpload({ api, sequence, onDone } = {}) {
|
||||
else dialog.close();
|
||||
});
|
||||
list.addEventListener("click", (e) => {
|
||||
// Upload again: a photo deleted from the sequence before goes back into it with the next run, on the user's word.
|
||||
const again = e.target.closest('[data-act="again"]');
|
||||
if (again && !running) {
|
||||
const item = items.find((i) => i.key === again.closest("[data-key]").dataset.key);
|
||||
if (item?.state !== "deleted") return;
|
||||
item.restore = true;
|
||||
setState(item, "waiting");
|
||||
drawButtons();
|
||||
startButton.focus();
|
||||
return;
|
||||
}
|
||||
const button = e.target.closest('[data-act="remove"]');
|
||||
if (!button || running) return;
|
||||
const index = items.findIndex((i) => i.key === button.closest("[data-key]").dataset.key);
|
||||
@@ -134,7 +151,12 @@ export function openUpload({ api, sequence, onDone } = {}) {
|
||||
take(files, folders);
|
||||
});
|
||||
|
||||
// Esc closes only while nothing is being sent; Cancel stops a run.
|
||||
// Esc closes only while nothing is being sent; Cancel stops a run. Chromium lets a dialog's cancel be prevented only
|
||||
// once per user activation, so during a run Esc is stopped at its keydown; the cancel handler stays for close requests
|
||||
// that do not come from the keyboard (a back gesture).
|
||||
dialog.addEventListener("keydown", (e) => {
|
||||
if (running && e.key === "Escape") e.preventDefault();
|
||||
});
|
||||
dialog.addEventListener("cancel", (e) => {
|
||||
if (running) e.preventDefault();
|
||||
});
|
||||
@@ -234,13 +256,17 @@ export function openUpload({ api, sequence, onDone } = {}) {
|
||||
`<option disabled>${escapeHtml(t("Admin.Upload.NoMatch"))}</option>`;
|
||||
}
|
||||
|
||||
/** Adds picked or dropped files: photos once each; folders, other files and files above the limit are counted. */
|
||||
/**
|
||||
* Adds picked or dropped files: photos once each; folders, other files and files above the limit are counted. Hidden
|
||||
* dot files (the AppleDouble "._" companions a Mac leaves on cards among them) are left out like the archive scanner
|
||||
* leaves them out, and the server refuses them too.
|
||||
*/
|
||||
function take(files, folders) {
|
||||
skipped.folders += folders;
|
||||
for (const file of files) {
|
||||
const dot = file.name.lastIndexOf(".");
|
||||
const extension = dot < 0 ? "" : file.name.slice(dot + 1).toLowerCase();
|
||||
if (file.size === 0 || !extensions?.has(extension)) {
|
||||
if (file.size === 0 || file.name.startsWith(".") || !extensions?.has(extension)) {
|
||||
skipped.formats++;
|
||||
continue;
|
||||
}
|
||||
@@ -250,14 +276,16 @@ export function openUpload({ api, sequence, onDone } = {}) {
|
||||
}
|
||||
const key = `${file.name}\u0000${file.size}\u0000${file.lastModified}`;
|
||||
if (items.some((i) => i.key === key)) continue;
|
||||
const item = { key, file, state: "waiting", hashed: 0, sent: 0, uploadId: null, message: "", row: null };
|
||||
const item = { key, file, state: "waiting", hashed: 0, sent: 0, uploadId: null, restore: false, message: "", row: null };
|
||||
item.row = document.createElement("li");
|
||||
item.row.className = "upload-file";
|
||||
item.row.dataset.key = key;
|
||||
item.row.innerHTML = `<span class="name ellipsis" dir="auto" title="${attr(file.name)}">${escapeHtml(file.name)}</span>
|
||||
<span class="size num muted">${escapeHtml(formatBytes(file.size))}</span><span class="status"></span>
|
||||
<button class="btn icon small subtle" type="button" data-act="remove" title="${attr(t("Admin.Upload.Remove.Tip"))}"
|
||||
aria-label="${attr(t("Admin.Upload.Remove", file.name))}">${icons.close}</button>`;
|
||||
aria-label="${attr(t("Admin.Upload.Remove", file.name))}">${icons.close}</button>
|
||||
<button class="btn icon small subtle" type="button" data-act="again" hidden title="${attr(t("Admin.Upload.Again.Tip"))}"
|
||||
aria-label="${attr(t("Admin.Upload.Again", file.name))}">${icons.upload}</button>`;
|
||||
list.appendChild(item.row);
|
||||
items.push(item);
|
||||
drawItem(item);
|
||||
@@ -341,7 +369,7 @@ export function openUpload({ api, sequence, onDone } = {}) {
|
||||
}
|
||||
status.innerHTML = (fraction === null ? "" : bar(fraction)) +
|
||||
`<span class="state ${tone} ellipsis"${tip ? ` title="${attr(tip)}"` : ""}>${escapeHtml(text)}</span>`;
|
||||
item.row.querySelector('[data-act="remove"]').hidden = !removable(item);
|
||||
drawRowButtons(item);
|
||||
}
|
||||
|
||||
/** A photo comes off the list only between runs, and only while it is not on the server. */
|
||||
@@ -349,6 +377,12 @@ export function openUpload({ api, sequence, onDone } = {}) {
|
||||
return !running && !FINISHED.has(item.state) && !ACTIVE.has(item.state);
|
||||
}
|
||||
|
||||
/** Remove, or for a photo deleted from the sequence before, Upload again; both only between runs. */
|
||||
function drawRowButtons(item) {
|
||||
item.row.querySelector('[data-act="remove"]').hidden = !removable(item);
|
||||
item.row.querySelector('[data-act="again"]').hidden = running || item.state !== "deleted";
|
||||
}
|
||||
|
||||
/** Progress redraws at most once a frame: the worker reports every 8 MiB, chunks finish four at a time. */
|
||||
function update(item) {
|
||||
item.dirty = true;
|
||||
@@ -400,7 +434,7 @@ export function openUpload({ api, sequence, onDone } = {}) {
|
||||
addButton.disabled = running || extensions === null;
|
||||
closeButton.textContent = t(running ? "Admin.Upload.Cancel" : "Admin.Upload.Close");
|
||||
closeButton.title = t(running ? "Admin.Upload.Cancel.Tip" : "Admin.Upload.Close.Tip");
|
||||
for (const item of items) item.row.querySelector('[data-act="remove"]').hidden = !removable(item);
|
||||
for (const item of items) drawRowButtons(item);
|
||||
targetHost.querySelectorAll("input, select").forEach((field) => {
|
||||
field.disabled = running;
|
||||
});
|
||||
@@ -419,6 +453,7 @@ export function openUpload({ api, sequence, onDone } = {}) {
|
||||
showError("");
|
||||
running = true;
|
||||
controller = new AbortController();
|
||||
sameContent = new Map();
|
||||
const signal = controller.signal;
|
||||
window.addEventListener("beforeunload", guard);
|
||||
drawButtons();
|
||||
@@ -520,7 +555,9 @@ export function openUpload({ api, sequence, onDone } = {}) {
|
||||
|
||||
/**
|
||||
* One photo: hashed, started (the server may already have it, or have had it before it was deleted), the missing
|
||||
* chunks sent two at a time, completed. True when the library changed.
|
||||
* chunks sent two at a time, completed. True when the library changed. The same photo picked under another name
|
||||
* waits for the first copy, which settles it for both: two lanes would otherwise complete the one upload the server
|
||||
* gives them and refuse each other.
|
||||
*/
|
||||
async function send(item, sequenceId, signal) {
|
||||
const file = item.file;
|
||||
@@ -533,12 +570,32 @@ export function openUpload({ api, sequence, onDone } = {}) {
|
||||
});
|
||||
if (signal.aborted) throw signal.reason;
|
||||
setState(item, "starting");
|
||||
const earlier = sameContent.get(sha256);
|
||||
let settle;
|
||||
sameContent.set(sha256, new Promise((resolve) => {
|
||||
settle = resolve;
|
||||
}));
|
||||
try {
|
||||
await earlier;
|
||||
if (signal.aborted) throw signal.reason;
|
||||
return await sendHashed(item, sha256, sequenceId, signal);
|
||||
} finally {
|
||||
settle();
|
||||
}
|
||||
}
|
||||
|
||||
/** A hashed photo: started (asking to bring it back when the user chose Upload again), its chunks sent, completed. */
|
||||
async function sendHashed(item, sha256, sequenceId, signal) {
|
||||
const file = item.file;
|
||||
const restore = item.restore;
|
||||
const capturedAt = await captureTime(file);
|
||||
const start = await retry(() => api.post("/api/v1/uploads", {
|
||||
sequenceId, fileName: file.name, size: file.size, sha256, capturedAt: null, sessionKey: null,
|
||||
sequenceId, fileName: file.name, size: file.size, sha256, capturedAt, sessionKey: null, restore,
|
||||
}), signal);
|
||||
if (start.alreadyExists) {
|
||||
// No frame: the photo was deleted from this sequence; a restore from the trash brings it back (R6).
|
||||
setState(item, start.frameId ? "exists" : "deleted");
|
||||
// No frame: the photo was deleted from this sequence; a restore from the trash brings it back while the trash
|
||||
// holds it, or Upload again on its row (R6).
|
||||
finish(item, start.frameId ? "exists" : "deleted");
|
||||
return false;
|
||||
}
|
||||
item.uploadId = start.uploadId;
|
||||
@@ -567,16 +624,57 @@ export function openUpload({ api, sequence, onDone } = {}) {
|
||||
}
|
||||
};
|
||||
await Promise.all(Array.from({ length: Math.min(CHUNKS_AT_ONCE, missing.length) }, lane));
|
||||
if (failure !== null) throw failure;
|
||||
// A chunk refused because the upload is no longer receiving: another completion of it is filing it (or it ended),
|
||||
// which the completion below waits for and reports.
|
||||
if (failure !== null && !filing(failure)) throw failure;
|
||||
setState(item, "verifying");
|
||||
await retry(() => api.post(`/api/v1/uploads/${item.uploadId}/complete`), signal);
|
||||
try {
|
||||
await complete(item.uploadId, restore, signal);
|
||||
} catch (err) {
|
||||
if (!(err instanceof ApiFailure && err.status === 409 && err.code === "photo_removed")) throw err;
|
||||
// Deleted from the sequence while it was on its way: not filed, and the server dropped the upload (R6).
|
||||
item.uploadId = null;
|
||||
finish(item, "deleted");
|
||||
return false;
|
||||
}
|
||||
item.uploadId = null;
|
||||
setState(item, "done");
|
||||
finish(item, "done");
|
||||
return true;
|
||||
}
|
||||
|
||||
/** A photo is settled on the server: a later run leaves it alone, until Upload again asks to bring it back. */
|
||||
function finish(item, state) {
|
||||
item.restore = false;
|
||||
setState(item, state);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/** 409 upload_state: the upload is not receiving now, because a completion is filing it or it ended. */
|
||||
function filing(err) {
|
||||
return err instanceof ApiFailure && err.status === 409 && err.code === "upload_state";
|
||||
}
|
||||
|
||||
/**
|
||||
* Completes an upload (with restore, a photo deleted from the sequence meanwhile is filed anyway). A completion refused
|
||||
* while another one is filing the upload (the same photo started twice, a retry after a lost answer) waits for it: once
|
||||
* the upload is committed, completing again answers its frame. Only an upload that failed or was cancelled, or one that
|
||||
* stays busy for about a minute, fails the photo.
|
||||
*/
|
||||
async function complete(uploadId, restore, signal) {
|
||||
const url = `/api/v1/uploads/${uploadId}/complete${restore ? "?restore=true" : ""}`;
|
||||
for (let check = 0; ; check++) {
|
||||
try {
|
||||
return await retry(() => api.post(url), signal);
|
||||
} catch (err) {
|
||||
if (signal.aborted || !filing(err) || check >= SETTLE_CHECKS) throw err;
|
||||
await sleep(1000 * Math.min(2 ** check, 8), signal);
|
||||
const upload = await retry(() => api.get(`/api/v1/uploads/${uploadId}`), signal);
|
||||
if (upload?.state === "failed" || upload?.state === "cancelled") throw err;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** What dropped: the files, and how many folders were left out (the entries API tells them apart where it exists). */
|
||||
function dropped(transfer) {
|
||||
const files = [];
|
||||
|
||||
@@ -27,8 +27,9 @@ public static class UploadEndpoints
|
||||
(await service.GetStateAsync(caller, id, context.RequestAborted)).ToResult(context, s => Results.Json(s, ProtocolJson.Default.UploadState)));
|
||||
uploads.MapDelete("/{id:guid}", async (HttpContext context, Caller caller, Guid id, UploadService service) =>
|
||||
(await service.CancelAsync(caller, id, context.RequestAborted)).ToResult(context, _ => Results.NoContent()));
|
||||
uploads.MapPost("/{id:guid}/complete", async (HttpContext context, Caller caller, Guid id, UploadService service) =>
|
||||
(await service.CompleteAsync(caller, id)).ToResult(context, r => Results.Json(r, ProtocolJson.Default.CompleteUploadResponse)))
|
||||
// ?restore=true repeats the start's StartUploadRequest.Restore: a photo deleted from the sequence meanwhile is filed anyway.
|
||||
uploads.MapPost("/{id:guid}/complete", async (HttpContext context, Caller caller, Guid id, bool? restore, UploadService service) =>
|
||||
(await service.CompleteAsync(caller, id, restore == true)).ToResult(context, r => Results.Json(r, ProtocolJson.Default.CompleteUploadResponse)))
|
||||
.Requires(Permissions.Upload);
|
||||
uploads.MapPost("/cards", ReportCard).Requires(Permissions.Upload);
|
||||
}
|
||||
|
||||
@@ -56,13 +56,24 @@ public static class UploadRules
|
||||
return intact.FirstOrDefault(f => f.SequenceId == target) ?? intact.FirstOrDefault();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A name the archive scanner would never take as a photo (<c>ArchiveScanner</c>: hidden dot files, among them the
|
||||
/// AppleDouble <c>._</c> companions a Mac leaves on exFAT cards): a frame upload refuses it, so no client files what an
|
||||
/// import of the same folder would leave out.
|
||||
/// </summary>
|
||||
public static bool IsHiddenName(string fileName) => fileName.StartsWith('.');
|
||||
|
||||
/// <summary>
|
||||
/// Whether <paramref name="target"/> tombstoned <paramref name="sha256"/> (owner review 2, R6): a photo deleted from that
|
||||
/// sequence (or replaced by a converted file) that an upload must not bring back. The upload is answered as
|
||||
/// <c>alreadyExists</c> without a frame, so a client neither sends the file nor treats the answer as a failure.
|
||||
/// <c>alreadyExists</c> without a frame, so a client neither sends the file nor treats the answer as a failure. A
|
||||
/// tombstone of content a frame of the target still holds (<paramref name="copies"/>: another copy of a duplicate, a
|
||||
/// file put back by hand) is stale and refuses nothing, so such a frame can still be repaired.
|
||||
/// </summary>
|
||||
public static async Task<bool> RemovedFromTargetAsync(ChronosDatabase db, string sha256, Guid? target, CancellationToken cancellationToken) =>
|
||||
target is { } sequence && await db.FrameRemovals.IsRemovedAsync(sequence, sha256, cancellationToken).ConfigureAwait(false);
|
||||
public static async Task<bool> RemovedFromTargetAsync(ChronosDatabase db, string sha256, Guid? target, IReadOnlyCollection<Frame> copies,
|
||||
CancellationToken cancellationToken) =>
|
||||
target is { } sequence && !copies.Any(f => f.SequenceId == sequence)
|
||||
&& await db.FrameRemovals.IsRemovedAsync(sequence, sha256, cancellationToken).ConfigureAwait(false);
|
||||
|
||||
/// <summary>
|
||||
/// The sequence an upload files into: its own, or the one its session key names (the user's own key without one);
|
||||
|
||||
@@ -49,7 +49,8 @@ public sealed class UploadService(
|
||||
|
||||
public async Task<Outcome<StartUploadResponse>> StartAsync(Caller caller, StartUploadRequest request, CancellationToken cancellationToken)
|
||||
{
|
||||
if (UploadRules.SafeFileName(request.FileName) is not { } fileName) return Fail<StartUploadResponse>(400, "invalid_file_name");
|
||||
if (UploadRules.SafeFileName(request.FileName) is not { } fileName || UploadRules.IsHiddenName(fileName))
|
||||
return Fail<StartUploadResponse>(400, "invalid_file_name");
|
||||
if (!UploadRules.IsSha256(request.Sha256)) return Fail<StartUploadResponse>(400, "invalid_hash");
|
||||
if (request.Size <= 0) return Fail<StartUploadResponse>(400, "bad_request", "size");
|
||||
if (request.Size > options.MaxUploadBytes) return Fail<StartUploadResponse>(413, "too_large", options.MaxUploadBytes.ToString(CultureInfo.InvariantCulture));
|
||||
@@ -62,11 +63,14 @@ public sealed class UploadService(
|
||||
var copies = await db.Frames.FindBySha256Async(sha, cancellationToken);
|
||||
var target = await UploadRules.TargetAsync(db, archive.Id, request.SequenceId, sessionKey, caller.Id, cancellationToken);
|
||||
// A photo deleted from the target sequence is not filed into it again: an old client syncing a copy that still holds
|
||||
// it, or a watch folder sending it again, hears that the server has it (owner review 2, R6). A restore brings it back.
|
||||
if (await UploadRules.RemovedFromTargetAsync(db, sha, target, cancellationToken))
|
||||
return new StartUploadResponse(null, options.ChunkSizeBytes, [], AlreadyExists: true, null);
|
||||
// Only a copy that is still whole stands for the upload: a client may delete its file on this answer.
|
||||
if (UploadRules.StandIn(archive.Root, copies, target) is { } known)
|
||||
// it, or a watch folder sending it again, hears that the server has it (owner review 2, R6). A restore from the trash
|
||||
// brings it back while the trash holds it; a user who uploads it again on purpose asks with Restore, and filing it
|
||||
// drops its tombstone.
|
||||
var removed = await UploadRules.RemovedFromTargetAsync(db, sha, target, copies, cancellationToken);
|
||||
if (removed && !request.Restore) return new StartUploadResponse(null, options.ChunkSizeBytes, [], AlreadyExists: true, null);
|
||||
// Only a copy that is still whole stands for the upload: a client may delete its file on this answer. A photo
|
||||
// brought back is filed into its sequence, whatever other sequences hold it.
|
||||
if (UploadRules.StandIn(archive.Root, removed ? [] : copies, target) is { } known)
|
||||
return new StartUploadResponse(null, options.ChunkSizeBytes, [], AlreadyExists: true, known.Id);
|
||||
|
||||
var resumable = await db.Uploads.FindReceivingAsync(caller.Id, sha, cancellationToken);
|
||||
@@ -158,9 +162,11 @@ public sealed class UploadService(
|
||||
|
||||
/// <summary>
|
||||
/// Verifies and commits a frame upload. Completing an already committed upload answers its frame again, so a client
|
||||
/// may retry a completion whose response it lost. Runs to the end even if the client disconnects.
|
||||
/// may retry a completion whose response it lost. Runs to the end even if the client disconnects. A photo deleted from
|
||||
/// the target sequence since the upload started is not filed (409 <see cref="UploadErrors.PhotoRemoved"/>, the upload
|
||||
/// cancelled) unless <paramref name="restore"/> repeats the start's <see cref="StartUploadRequest.Restore"/>.
|
||||
/// </summary>
|
||||
public async Task<Outcome<CompleteUploadResponse>> CompleteAsync(Caller caller, Guid id)
|
||||
public async Task<Outcome<CompleteUploadResponse>> CompleteAsync(Caller caller, Guid id, bool restore = false)
|
||||
{
|
||||
var stopping = lifetime.ApplicationStopping;
|
||||
var upload = await db.Uploads.GetAsync(id, stopping);
|
||||
@@ -174,6 +180,15 @@ public sealed class UploadService(
|
||||
if (!ArchiveHasRoom(upload)) return Fail<CompleteUploadResponse>(507, "insufficient_storage");
|
||||
if (!await db.Uploads.TryTransitionAsync(id, UploadState.Receiving, UploadState.Committing, cancellationToken: stopping))
|
||||
return Fail<CompleteUploadResponse>(409, "upload_state");
|
||||
// Deleted from the target while its bytes were on the way (the start checked before): the upload does not bring the
|
||||
// photo back (owner review 2, R6).
|
||||
if (!restore && await RemovedFromTargetAsync(upload, stopping))
|
||||
{
|
||||
logger.LogInformation("Upload {Upload} ({File}) not filed: the photo was deleted from its sequence meanwhile", id, upload.FileName);
|
||||
await db.Uploads.TryTransitionAsync(id, UploadState.Committing, UploadState.Cancelled, cancellationToken: CancellationToken.None);
|
||||
store.Delete(id);
|
||||
return Fail<CompleteUploadResponse>(409, UploadErrors.PhotoRemoved);
|
||||
}
|
||||
|
||||
var user = await db.Users.GetAsync(caller.Id, stopping);
|
||||
if (user is { QuotaBytes: > 0 } && user.UsedBytes + upload.Size > user.QuotaBytes && !await HeldByTargetAsync(upload, stopping))
|
||||
@@ -305,6 +320,14 @@ public sealed class UploadService(
|
||||
return copies.Any(f => f.SequenceId == target);
|
||||
}
|
||||
|
||||
/// <summary>The sequence the upload files into tombstoned its content (<see cref="UploadRules.RemovedFromTargetAsync"/>).</summary>
|
||||
private async Task<bool> RemovedFromTargetAsync(Upload upload, CancellationToken cancellationToken)
|
||||
{
|
||||
var target = await UploadRules.TargetAsync(db, archive.Id, upload.SequenceId, upload.SessionKey, upload.UserId, cancellationToken);
|
||||
return target is not null && await UploadRules.RemovedFromTargetAsync(db, upload.Sha256, target,
|
||||
await db.Frames.FindBySha256Async(upload.Sha256, cancellationToken), cancellationToken);
|
||||
}
|
||||
|
||||
private StartUploadResponse Started(Upload upload) =>
|
||||
new(upload.Id, upload.ChunkSize, UploadRules.ReceivedChunks(upload), AlreadyExists: false, FrameId: null);
|
||||
|
||||
|
||||
@@ -97,6 +97,27 @@ public sealed class SyncRulesTests
|
||||
new FrameRemoval(localId, "bb", "b2", RemovalReason.Replaced, at)]).Select(r => r.Reason));
|
||||
}
|
||||
|
||||
[Test]
|
||||
public void ATombstoneBesideAPhotoItsSideStillHoldsHidesNothing()
|
||||
{
|
||||
var time = new DateTime(2026, 9, 28, 18, 0, 0);
|
||||
var at = new DateTimeOffset(2026, 10, 27, 10, 0, 0, TimeSpan.Zero);
|
||||
FramePhoto Photo(string sha, int second = 0) => new(sha, 100 + second, time.AddSeconds(second));
|
||||
// The server tombstoned "xx" and still holds it (it deleted one of two copies, or the file came back by hand): this
|
||||
// PC's copy is not missing there, and the server's is not missing here.
|
||||
var server = new SyncTombstones([], [new RemovedPhotoDto("xx", null, "deleted", at)]);
|
||||
Assert.Equal((0, 0), SyncRules.CompareOriginals([Photo("aa"), Photo("xx", 1)], [Photo("aa"), Photo("XX", 1)], server));
|
||||
// Also when the server has not hashed its copy: it holds it by size and capture time.
|
||||
Assert.Equal((0, 0), SyncRules.CompareOriginals([Photo("aa"), Photo("xx", 1)], [Photo("aa"), Photo("", 1)], server));
|
||||
// The same on this side.
|
||||
var here = new SyncTombstones([new FrameRemoval(Guid.CreateVersion7(), "xx", null, RemovalReason.Deleted, at)], null);
|
||||
Assert.Equal((0, 0), SyncRules.CompareOriginals([Photo("aa"), Photo("xx", 1)], [Photo("aa"), Photo("xx", 1)], here));
|
||||
// A tombstone of a photo its side no longer holds still leaves the other side's copy out.
|
||||
Assert.Equal((0, 0), SyncRules.CompareOriginals([Photo("aa"), Photo("xx", 1)], [Photo("aa")], server));
|
||||
Assert.Equal((0, 0), SyncRules.CompareOriginals([Photo("aa")], [Photo("aa"), Photo("xx", 1)], here));
|
||||
Assert.Equal((1, 0), SyncRules.CompareOriginals([Photo("aa"), Photo("xx", 1)], [Photo("aa")], SyncTombstones.None));
|
||||
}
|
||||
|
||||
[Test]
|
||||
public void FramesEitherSideHasNotHashedAreMatchedBySizeAndCaptureTime()
|
||||
{
|
||||
|
||||
@@ -151,6 +151,90 @@ public sealed class SyncTransferTests(TestContext context)
|
||||
Assert.SequenceEqual(photos[1..].Select(Sha), kept.Select(k => k.Sha256!));
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task APhotoDeletedHereIsNotFetchedAgainFromAServerThatHasNotHashedIt()
|
||||
{
|
||||
await using var f = await SyncFixture.CreateAsync(context);
|
||||
var photos = f.NewPhotos(3);
|
||||
// A folder the server adopted and never verified (its manifest lists no SHA-256), downloaded here.
|
||||
var remote = f.Remote.Add("Adopted on the server", photos);
|
||||
remote.Unhashed = true;
|
||||
await f.RefreshAsync();
|
||||
var engine = await f.StartEngineAsync();
|
||||
var first = await f.Sync.EnqueueDownloadAsync(engine, f.Server, remote.Id, "Adopted on the server", SyncDownloadMode.Originals, f.Archive.Archive.Id);
|
||||
await f.Archive.WaitForAsync(first.Id, JobState.Done);
|
||||
var local = Assert.NotNull((await f.EntryAsync("Adopted on the server")).Local);
|
||||
|
||||
// This PC deletes a photo the server still lists, unhashed: only its content tells it is the one deleted here.
|
||||
await UntilIdleAsync(f);
|
||||
var gone = (await f.Db.Frames.ListBySequenceAsync(local.Id, cancellationToken: f.Ct)).Single(fr => fr.Sha256 == Sha(photos[0]));
|
||||
var sequence = (await f.Db.Sequences.GetAsync(local.Id, f.Ct))!;
|
||||
var plan = await f.Archive.Service.FrameTrash.PlanDeleteAsync(local.Id, [gone.Id], sequence.FramesVersion, f.Ct);
|
||||
await f.Archive.Service.FrameTrash.DeleteAsync(plan, null, null, f.Ct);
|
||||
await f.RefreshAsync();
|
||||
|
||||
var again = await f.Sync.EnqueueDownloadAsync(engine, f.Server, remote.Id, "Adopted on the server", SyncDownloadMode.Originals, null);
|
||||
var done = await f.Archive.WaitForAsync(again.Id, JobState.Done);
|
||||
|
||||
var kept = await f.Db.Frames.ListBySequenceAsync(local.Id, cancellationToken: f.Ct);
|
||||
Assert.True(kept.Select(k => k.Sha256).ToHashSet().SetEquals(photos[1..].Select(Sha)), "the photo deleted here stays deleted");
|
||||
Assert.True(await f.Db.FrameRemovals.IsRemovedAsync(local.Id, Sha(photos[0])), "and keeps its tombstone");
|
||||
Assert.Contains("deleted on this PC, not downloaded", done.LogTail);
|
||||
Assert.False(Directory.Exists(Path.Combine(f.Archive.Paths.StagingDirectory, "sync-" + remote.Id.ToString("N"))), "staging cleaned up");
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task AnUnverifiedCopyDoesNotSendBackAPhotoTheServerDeleted()
|
||||
{
|
||||
await using var f = await SyncFixture.CreateAsync(context);
|
||||
var photos = f.NewPhotos(4);
|
||||
// An adopted folder (never verified, so no hashes here) that still holds a photo the server has deleted since.
|
||||
var folder = Path.Combine(f.Archive.Paths.Root, "Adopted");
|
||||
Directory.CreateDirectory(folder);
|
||||
for (var i = 0; i < 4; i++) await File.WriteAllBytesAsync(Path.Combine(folder, $"DSC_{i + 1:0000}.JPG"), photos[i], f.Ct);
|
||||
await f.Archive.ScanAsync();
|
||||
var local = Assert.Contains((await f.Db.Sequences.ListAsync(new Chronos.Data.Repositories.SequenceQuery(), f.Ct)).Items, s => s.Folder == "Adopted");
|
||||
var remote = f.Remote.Add("Adopted", photos[..3]);
|
||||
remote.Removed.Add(new RemovedPhotoDto(Sha(photos[3]), null, "deleted", DateTimeOffset.UtcNow));
|
||||
await f.Sync.LinkAsync(f.Server, remote.Id, local.Id, f.Ct);
|
||||
await f.RefreshAsync();
|
||||
// Without its hash the photo cannot be told from one the server lacks: the upload finds out.
|
||||
Assert.Equal(1, (await f.EntryAsync(local.DisplayTitle)).Link!.MissingRemote);
|
||||
var engine = await f.StartEngineAsync();
|
||||
|
||||
var job = await f.Sync.EnqueueUploadAsync(engine, f.Server, local);
|
||||
var done = await f.Archive.WaitForAsync(job.Id, JobState.Done);
|
||||
|
||||
Assert.Equal(3, remote.Frames.Count);
|
||||
Assert.Contains("deleted on the server, not uploaded", done.LogTail);
|
||||
// Its hash is in the index now: the comparison leaves it out, so the row settles instead of offering Upload for good.
|
||||
Assert.True((await f.Db.Frames.ListBySequenceAsync(local.Id, cancellationToken: f.Ct)).All(fr => fr.Sha256 is not null), "hashed by the verify");
|
||||
var entry = await f.EntryAsync(local.DisplayTitle);
|
||||
Assert.Equal((SyncStatus.InSync, 0, 0), (entry.Status, entry.Link!.MissingRemote, entry.Link.MissingLocal));
|
||||
var calls = remote.Frames.Count;
|
||||
var next = await f.Sync.EnqueueUploadAsync(engine, f.Server, local);
|
||||
var settled = await f.Archive.WaitForAsync(next.Id, JobState.Done);
|
||||
Assert.Equal((4L, 4L, calls), (settled.ProgressDone, settled.ProgressTotal, remote.Frames.Count));
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task ATombstoneBesideAPhotoTheServerStillHoldsLeavesTheCopiesInSync()
|
||||
{
|
||||
await using var f = await SyncFixture.CreateAsync(context);
|
||||
var photos = f.NewPhotos(2);
|
||||
var local = await f.LocalAsync("Comet", photos);
|
||||
// The server holds both photos and a tombstone of the second: it deleted one of two copies of it, or the file came
|
||||
// back by hand. Neither side lacks anything.
|
||||
var remote = f.Remote.Add("Comet", photos);
|
||||
remote.Removed.Add(new RemovedPhotoDto(Sha(photos[1]), null, "deleted", DateTimeOffset.UtcNow));
|
||||
await f.Sync.LinkAsync(f.Server, remote.Id, local.Id, f.Ct);
|
||||
await f.RefreshAsync();
|
||||
|
||||
var entry = await f.EntryAsync("Comet");
|
||||
Assert.Equal((0, 0), (entry.Link!.MissingRemote, entry.Link.MissingLocal));
|
||||
Assert.Equal(SyncStatus.InSync, entry.Status);
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task AResumedUploadSendsOnlyWhatTheServerLacks()
|
||||
{
|
||||
@@ -634,6 +718,20 @@ public sealed class SyncTransferTests(TestContext context)
|
||||
|
||||
private static string Sha(byte[] bytes) => Convert.ToHexStringLower(System.Security.Cryptography.SHA256.HashData(bytes));
|
||||
|
||||
/// <summary>Waits until no job is waiting or running (the thumbnails and analyses a download queues after it).</summary>
|
||||
private static async Task UntilIdleAsync(SyncFixture f)
|
||||
{
|
||||
var until = DateTime.UtcNow + TimeSpan.FromSeconds(TestTiming.Budget(30));
|
||||
foreach (var state in new[] { JobState.Queued, JobState.Running, JobState.Queued, JobState.Running })
|
||||
{
|
||||
while ((await f.Db.Jobs.ListAsync(state, cancellationToken: f.Ct)).Items.Count > 0)
|
||||
{
|
||||
Assert.True(DateTime.UtcNow < until, $"jobs still {state}");
|
||||
await Task.Delay(50, f.Ct);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>The photos (SHA-256) the local copy's project excludes, shows as hero, keys the camera on, and waits for.</summary>
|
||||
private static async Task<(string[] Excluded, string? Hero, string[] Keyframes, string[] Pending)> NamedAsync(SyncFixture f, Guid localId)
|
||||
{
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using System.Net;
|
||||
using System.Net.Http.Json;
|
||||
using System.Security.Cryptography;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Chronos.Core.Archiving;
|
||||
@@ -419,6 +420,59 @@ public sealed class ArchiveModuleTests(TestContext context)
|
||||
Assert.Equal(photo.Length, (await me.ReadAsync(ProtocolJson.Default.UserDto)).UsedBytes);
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task TheProtocolClientTakesAPhotoDeletedWhileOnItsWayAsNotSent()
|
||||
{
|
||||
// Owner review 2, R6: a repair upload whose photo the Library deletes meanwhile is refused at its completion
|
||||
// (photo_removed, nothing filed); the protocol client reports it like a photo deleted before: null, not a failure.
|
||||
await using var server = await TestServer.StartAsync(context, services: ArchiveModule.Register);
|
||||
var (_, _, token) = await server.UserAsync("alberto", "editor");
|
||||
var admin = await server.AdminAsync();
|
||||
var path = Path.Combine(context.CreateTempDirectory("source"), "IMG_0001.JPG");
|
||||
var photo = Photo(5);
|
||||
await File.WriteAllBytesAsync(path, photo);
|
||||
using var client = new ChronosClient(server.HttpBase, new CertificatePinning([]), "Chronos Uploader (tests)") { Token = token };
|
||||
var sequence = await client.CreateSequenceAsync(new CreateSequenceRequest("Roof", null), context.Cancellation);
|
||||
// A sequence keeps one photo at least: the other is the one deleted.
|
||||
var keeper = Path.Combine(Path.GetDirectoryName(path)!, "IMG_0000.JPG");
|
||||
await File.WriteAllBytesAsync(keeper, Photo(6));
|
||||
var kept = Assert.NotNull(await client.UploadFileAsync(keeper, sequence.Id, null, ct: context.Cancellation));
|
||||
var first = Assert.NotNull(await client.UploadFileAsync(path, sequence.Id, null, ct: context.Cancellation));
|
||||
await UntilIdleAsync(server);
|
||||
File.Delete(Path.Combine(server.Options.ArchivePath, Assert.NotNull(await server.Database.Frames.GetAsync(first.FrameId)).RelativePath));
|
||||
|
||||
using var deleting = new BeforeCompletion(async () =>
|
||||
{
|
||||
await UntilIdleAsync(server);
|
||||
var current = Assert.NotNull(await server.Database.Sequences.GetAsync(sequence.Id));
|
||||
using var delete = await admin.PostAsJsonAsync($"/api/v1/sequences/{sequence.Id}/frames/delete",
|
||||
new FrameDeleteRequest([first.FrameId], current.FramesVersion, false), ProtocolJson.Default.FrameDeleteRequest);
|
||||
await delete.ExpectAsync(HttpStatusCode.OK);
|
||||
});
|
||||
using var racing = new ChronosClient(server.HttpBase, new CertificatePinning([]), "Chronos Uploader (tests)", deleting) { Token = token };
|
||||
Assert.Null(await racing.UploadFileAsync(path, sequence.Id, null, ct: context.Cancellation));
|
||||
|
||||
Assert.True(deleting.Ran, "the completion was reached");
|
||||
Assert.SequenceEqual([kept.FrameId], (await server.Database.Frames.ListBySequenceAsync(sequence.Id)).Select(f => f.Id));
|
||||
Assert.True(await server.Database.FrameRemovals.IsRemovedAsync(sequence.Id, Convert.ToHexStringLower(SHA256.HashData(photo))), "the tombstone stays");
|
||||
}
|
||||
|
||||
/// <summary>Runs a step on the server before the first upload completion goes through, then sends it on.</summary>
|
||||
private sealed class BeforeCompletion(Func<Task> step) : DelegatingHandler(new SocketsHttpHandler())
|
||||
{
|
||||
public bool Ran { get; private set; }
|
||||
|
||||
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
if (!Ran && request.Method == HttpMethod.Post && request.RequestUri?.AbsolutePath.EndsWith("/complete", StringComparison.Ordinal) == true)
|
||||
{
|
||||
Ran = true;
|
||||
await step();
|
||||
}
|
||||
return await base.SendAsync(request, cancellationToken);
|
||||
}
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task AnIntactCopyElsewhereDoesNotStandForTheDamagedOneOfTheTargetSequence()
|
||||
{
|
||||
|
||||
@@ -3,6 +3,7 @@ using System.Net;
|
||||
using System.Net.Http.Headers;
|
||||
using System.Net.Http.Json;
|
||||
using System.Text;
|
||||
using System.Text.RegularExpressions;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Chronos.Core.Codecs.Jpeg;
|
||||
using Chronos.Core.Localization;
|
||||
@@ -89,12 +90,9 @@ public sealed class BrowserUploadTests(TestContext context)
|
||||
Assert.True(again.AlreadyExists);
|
||||
Assert.Equal(frames[0].FrameId, again.FrameId);
|
||||
|
||||
// Deleted from the sequence: also answered as there, without a frame ("deleted before; restore it from the trash").
|
||||
await UntilIdleAsync(server);
|
||||
var current = Assert.NotNull(await server.Database.Sequences.GetAsync(sequence.Id));
|
||||
using (var delete = await browser.SendAsync(HttpMethod.Post, $"/api/v1/sequences/{sequence.Id}/frames/delete",
|
||||
Json($$"""{"frameIds":["{{frames[1].FrameId}}"],"expectedFramesVersion":{{current.FramesVersion}},"dryRun":false}""")))
|
||||
Assert.NotNull((await delete.ReadAsync(ProtocolJson.Default.FrameDeleteResultDto)).TrashEntry);
|
||||
// Deleted from the sequence: also answered as there, without a frame ("deleted before": restore it from the trash, or
|
||||
// upload it again).
|
||||
await DeleteAsync(server, browser, sequence.Id, frames[1].FrameId);
|
||||
var deleted = await StartAsync(browser, sequence.Id, "IMG_0002.JPG", photos[1], HttpStatusCode.OK);
|
||||
Assert.True(deleted.AlreadyExists);
|
||||
Assert.Null(deleted.UploadId);
|
||||
@@ -130,6 +128,181 @@ public sealed class BrowserUploadTests(TestContext context)
|
||||
Assert.Equal(Loc.Get("Server.Error.QuotaExceeded", CultureInfo.GetCultureInfo("it")), error.Message);
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task UploadAgainBringsBackAPhotoDeletedBeforeEvenAfterTheTrashIsEmptied()
|
||||
{
|
||||
// Owner review 2, R6 and R3: a tombstone outlives the trash and keeps sync and old clients from filing the photo
|
||||
// again, but the user who picks it in the dialog and chooses Upload again gets it back into the sequence.
|
||||
await using var server = await TestServer.StartAsync(context, o => o.ChunkSizeBytes = Chunk, ArchiveModule.Register);
|
||||
var browser = await Browser.SignInAsync(server);
|
||||
var sequence = await CreateAsync(browser, "Roof");
|
||||
var photo = Photo(11);
|
||||
var sha = Uploads.Sha(photo);
|
||||
// A sequence keeps one photo at least: the other is the one deleted.
|
||||
var keeper = await UploadAsync(browser, sequence.Id, "IMG_0000.JPG", Photo(10));
|
||||
var first = await UploadAsync(browser, sequence.Id, "IMG_0001.JPG", photo);
|
||||
await DeleteAsync(server, browser, sequence.Id, first.FrameId);
|
||||
await EmptyTrashAsync(server, browser);
|
||||
Assert.True(await server.Database.FrameRemovals.IsRemovedAsync(sequence.Id, sha), "the tombstone outlives the trash");
|
||||
|
||||
// Picked again: deleted before, nothing started.
|
||||
var refused = await StartAsync(browser, sequence.Id, "IMG_0001.JPG", photo, HttpStatusCode.OK);
|
||||
Assert.True(refused is { AlreadyExists: true, UploadId: null, FrameId: null }, "answered as deleted before");
|
||||
|
||||
// Upload again whose completion does not repeat the restore (an older page): refused at the end, the upload dropped.
|
||||
var started = await StartAsync(browser, sequence.Id, "IMG_0001.JPG", photo, HttpStatusCode.Created, restore: true);
|
||||
var dropped = Assert.NotNull(started.UploadId);
|
||||
await SendChunksAsync(browser, dropped, photo);
|
||||
using (var complete = await CompleteAsync(browser, dropped))
|
||||
{
|
||||
var error = await complete.ExpectErrorAsync(HttpStatusCode.Conflict, UploadErrors.PhotoRemoved);
|
||||
Assert.Equal(Loc.Get("Server.Error.PhotoRemoved", CultureInfo.GetCultureInfo("it")), error.Message);
|
||||
}
|
||||
using (var state = await browser.SendAsync(HttpMethod.Get, $"/api/v1/uploads/{dropped}"))
|
||||
Assert.Equal("cancelled", (await state.ReadAsync(ProtocolJson.Default.UploadState)).State);
|
||||
Assert.False(server.Services.GetRequiredService<UploadStore>().Exists(dropped), "the partial file is gone");
|
||||
Assert.True(await server.Database.FrameRemovals.IsRemovedAsync(sequence.Id, sha), "still deleted");
|
||||
|
||||
// Upload again, start and completion: filed into the sequence, and its tombstone is gone.
|
||||
var back = await UploadAsync(browser, sequence.Id, "IMG_0001.JPG", photo, restore: true);
|
||||
Assert.Equal(sequence.Id, back.SequenceId);
|
||||
var frames = await server.Database.Frames.ListBySequenceAsync(sequence.Id);
|
||||
Assert.SequenceEqual([keeper.FrameId, back.FrameId], frames.Select(f => f.Id).Order());
|
||||
Assert.Equal(sha, frames.Single(f => f.Id == back.FrameId).Sha256);
|
||||
Assert.False(await server.Database.FrameRemovals.IsRemovedAsync(sequence.Id, sha), "the tombstone went with the upload");
|
||||
var again = await StartAsync(browser, sequence.Id, "IMG_0001.JPG", photo, HttpStatusCode.OK);
|
||||
Assert.Equal(back.FrameId, again.FrameId);
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task UploadAgainFilesThePhotoIntoItsSequenceWhateverOtherSequencesHold()
|
||||
{
|
||||
// A copy in another sequence stands for an ordinary upload; one brought back on purpose goes where the user put it.
|
||||
await using var server = await TestServer.StartAsync(context, o => o.ChunkSizeBytes = Chunk, ArchiveModule.Register);
|
||||
var browser = await Browser.SignInAsync(server);
|
||||
var roof = await CreateAsync(browser, "Roof");
|
||||
var yard = await CreateAsync(browser, "Yard");
|
||||
var photo = Photo(12);
|
||||
await UploadAsync(browser, roof.Id, "IMG_0000.JPG", Photo(10));
|
||||
var first = await UploadAsync(browser, roof.Id, "IMG_0001.JPG", photo);
|
||||
await DeleteAsync(server, browser, roof.Id, first.FrameId);
|
||||
var elsewhere = await UploadAsync(browser, yard.Id, "IMG_0001.JPG", photo);
|
||||
|
||||
var deleted = await StartAsync(browser, roof.Id, "IMG_0001.JPG", photo, HttpStatusCode.OK);
|
||||
Assert.Null(deleted.FrameId);
|
||||
var back = await UploadAsync(browser, roof.Id, "IMG_0001.JPG", photo, restore: true);
|
||||
Assert.Equal(roof.Id, back.SequenceId);
|
||||
Assert.NotEqual(elsewhere.FrameId, back.FrameId);
|
||||
Assert.Count(1, await server.Database.Frames.ListBySequenceAsync(yard.Id));
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task AnUploadCompletedAfterItsPhotoWasDeletedDoesNotBringItBack()
|
||||
{
|
||||
// Owner review 2, R6: the start checked the tombstones, but the photo can be deleted while its bytes are on the way
|
||||
// (here a repair of a copy whose original is gone, deleted from the Library meanwhile).
|
||||
await using var server = await TestServer.StartAsync(context, o => o.ChunkSizeBytes = Chunk, ArchiveModule.Register);
|
||||
var browser = await Browser.SignInAsync(server);
|
||||
var sequence = await CreateAsync(browser, "Roof");
|
||||
var photo = Photo(13);
|
||||
var sha = Uploads.Sha(photo);
|
||||
var keeper = await UploadAsync(browser, sequence.Id, "IMG_0000.JPG", Photo(10));
|
||||
var first = await UploadAsync(browser, sequence.Id, "IMG_0001.JPG", photo);
|
||||
await UntilIdleAsync(server);
|
||||
var frame = Assert.NotNull(await server.Database.Frames.GetAsync(first.FrameId));
|
||||
File.Delete(Path.Combine(server.Options.ArchivePath, frame.RelativePath));
|
||||
|
||||
var repair = await StartAsync(browser, sequence.Id, "IMG_0001.JPG", photo, HttpStatusCode.Created);
|
||||
var id = Assert.NotNull(repair.UploadId);
|
||||
await SendChunksAsync(browser, id, photo);
|
||||
await DeleteAsync(server, browser, sequence.Id, first.FrameId);
|
||||
|
||||
using (var complete = await CompleteAsync(browser, id))
|
||||
await complete.ExpectErrorAsync(HttpStatusCode.Conflict, UploadErrors.PhotoRemoved);
|
||||
Assert.SequenceEqual([keeper.FrameId], (await server.Database.Frames.ListBySequenceAsync(sequence.Id)).Select(f => f.Id));
|
||||
Assert.True(await server.Database.FrameRemovals.IsRemovedAsync(sequence.Id, sha), "the tombstone stays");
|
||||
Assert.False(server.Services.GetRequiredService<UploadStore>().Exists(id), "the partial file is gone");
|
||||
// Completing again changes nothing: the upload is over.
|
||||
using (var late = await CompleteAsync(browser, id))
|
||||
await late.ExpectErrorAsync(HttpStatusCode.Conflict, "upload_state");
|
||||
Assert.SequenceEqual([keeper.FrameId], (await server.Database.Frames.ListBySequenceAsync(sequence.Id)).Select(f => f.Id));
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task AStaleTombstoneBesideAFrameOfTheSameContentRefusesNoRepair()
|
||||
{
|
||||
// A tombstone left beside a frame that still holds the content (the other copy of a duplicate, a file put back by
|
||||
// hand) does not stand for a deletion: the frame's original can still be repaired by uploading it.
|
||||
await using var server = await TestServer.StartAsync(context, o => o.ChunkSizeBytes = Chunk, ArchiveModule.Register);
|
||||
var browser = await Browser.SignInAsync(server);
|
||||
var sequence = await CreateAsync(browser, "Roof");
|
||||
var photo = Photo(14);
|
||||
var sha = Uploads.Sha(photo);
|
||||
var first = await UploadAsync(browser, sequence.Id, "IMG_0001.JPG", photo);
|
||||
await UntilIdleAsync(server);
|
||||
await server.Database.FrameRemovals.SyncSequenceAsync(sequence.Id, [new FrameRemoval(sequence.Id, sha, null, RemovalReason.Deleted, DateTimeOffset.UtcNow)]);
|
||||
var frame = Assert.NotNull(await server.Database.Frames.GetAsync(first.FrameId));
|
||||
var original = Path.Combine(server.Options.ArchivePath, frame.RelativePath);
|
||||
File.Delete(original);
|
||||
|
||||
var repaired = await UploadAsync(browser, sequence.Id, "IMG_0001.JPG", photo);
|
||||
Assert.Equal(first.FrameId, repaired.FrameId);
|
||||
await Http.BytesEqualAsync(photo, File.ReadAllBytesAsync(original));
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task TheDialogsCaptureTimeDatesAPhotoThatRecordsNone()
|
||||
{
|
||||
// Owner review 2, R3: a photo without Exif (a PNG, a JPEG stripped on export) takes the time the dialog sends, as local
|
||||
// wall time, not the moment the server staged it; frames sent out of order still end up in capture order.
|
||||
await using var server = await TestServer.StartAsync(context, o => o.ChunkSizeBytes = Chunk, ArchiveModule.Register);
|
||||
var browser = await Browser.SignInAsync(server);
|
||||
var sequence = await CreateAsync(browser, "Timelapse");
|
||||
var later = await UploadAsync(browser, sequence.Id, "frame_0002.jpg", Photo(16), "2025-05-01T06:31:00.000");
|
||||
var earlier = await UploadAsync(browser, sequence.Id, "frame_0001.jpg", Photo(15), "2025-05-01T06:30:00");
|
||||
|
||||
var frames = await server.Database.Frames.ListBySequenceAsync(sequence.Id);
|
||||
Assert.SequenceEqual([earlier.FrameId, later.FrameId], frames.OrderBy(f => f.Index).Select(f => f.Id));
|
||||
Assert.Equal(new DateTime(2025, 5, 1, 6, 30, 0), frames.Single(f => f.Id == earlier.FrameId).CapturedAt);
|
||||
Assert.Equal(new DateTime(2025, 5, 1, 6, 31, 0), frames.Single(f => f.Id == later.FrameId).CapturedAt);
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task HiddenDotFilesAreRefusedLikeTheScannerLeavesThemOut()
|
||||
{
|
||||
// The AppleDouble "._" companion a Mac leaves on an exFAT card, and any hidden dot file, is no photo for the scanner:
|
||||
// the dialog leaves them out, and the server refuses them from any client.
|
||||
await using var server = await TestServer.StartAsync(context, o => o.ChunkSizeBytes = Chunk, ArchiveModule.Register);
|
||||
var browser = await Browser.SignInAsync(server);
|
||||
var sequence = await CreateAsync(browser, "Card");
|
||||
foreach (var name in new[] { "._IMG_0001.JPG", ".IMG_0002.JPG" })
|
||||
{
|
||||
using var start = await browser.SendAsync(HttpMethod.Post, "/api/v1/uploads", StartBody(sequence.Id, name, Uploads.Photo(4096, 21)));
|
||||
await start.ExpectErrorAsync(HttpStatusCode.BadRequest, "invalid_file_name");
|
||||
}
|
||||
var upload = await server.Client.GetStringAsync("/admin/upload.js");
|
||||
Assert.Contains(@"file.name.startsWith(""."")", upload);
|
||||
}
|
||||
|
||||
[Test]
|
||||
public async Task TheDialogKeepsItselfOpenAndSettlesWhatAnotherCompletionFiles()
|
||||
{
|
||||
// Owner review 2, R3 (upload.js): Esc is stopped at its keydown during a run, since Chromium lets a dialog's cancel be
|
||||
// prevented only once per user activation; a completion refused while another one files the upload (409
|
||||
// upload_state) waits for it instead of failing the photo; the same photo picked twice waits for its first copy.
|
||||
await using var server = await TestServer.StartAsync(context);
|
||||
var upload = await server.Client.GetStringAsync("/admin/upload.js");
|
||||
Assert.True(Regex.IsMatch(upload, @"addEventListener\(""keydown"", \(e\) => \{\s*if \(running && e\.key === ""Escape""\) e\.preventDefault\(\);"), "Esc at keydown");
|
||||
Assert.Contains(@"err.code === ""upload_state""", upload);
|
||||
Assert.Contains(@"api.get(`/api/v1/uploads/${uploadId}`)", upload);
|
||||
Assert.Contains("sameContent.get(sha256)", upload);
|
||||
Assert.Contains(@"?restore=true", upload);
|
||||
Assert.Contains(@"err.code === ""photo_removed""", upload);
|
||||
Assert.Contains(@"import { captureTime } from ""./capture-time.js"";", upload);
|
||||
var capture = await server.Client.GetStringAsync("/admin/capture-time.js");
|
||||
Assert.Contains("export function exifCaptureTime(bytes)", capture);
|
||||
}
|
||||
|
||||
/// <summary>The bootstrap administrator signed in to the panel (a cookie session over HTTPS), calling as <c>api.js</c> does.</summary>
|
||||
private sealed class Browser(HttpClient client, string csrfToken)
|
||||
{
|
||||
@@ -165,17 +338,80 @@ public sealed class BrowserUploadTests(TestContext context)
|
||||
return content;
|
||||
}
|
||||
|
||||
/// <summary>The start body upload.js writes: no capture time (the server reads it from the photo), no session key.</summary>
|
||||
private static StringContent StartBody(Guid? sequenceId, string fileName, byte[] data) =>
|
||||
Json($$"""{"sequenceId":{{(sequenceId is { } id ? $"\"{id}\"" : "null")}},"fileName":"{{fileName}}","size":{{data.Length}},"sha256":"{{Uploads.Sha(data)}}","capturedAt":null,"sessionKey":null}""");
|
||||
/// <summary>
|
||||
/// The start body upload.js writes: the capture time capture-time.js read (local wall time without an offset, or null),
|
||||
/// no session key, and restore true only after the user chose Upload again on the photo's row.
|
||||
/// </summary>
|
||||
private static StringContent StartBody(Guid? sequenceId, string fileName, byte[] data, string? capturedAt = null, bool restore = false) =>
|
||||
Json($$"""{"sequenceId":{{(sequenceId is { } id ? $"\"{id}\"" : "null")}},"fileName":"{{fileName}}","size":{{data.Length}},"sha256":"{{Uploads.Sha(data)}}","capturedAt":{{(capturedAt is null ? "null" : $"\"{capturedAt}\"")}},"sessionKey":null,"restore":{{(restore ? "true" : "false")}}}""");
|
||||
|
||||
private static async Task<StartUploadResponse> StartAsync(Browser browser, Guid sequenceId, string fileName, byte[] data, HttpStatusCode expected)
|
||||
private static async Task<StartUploadResponse> StartAsync(Browser browser, Guid sequenceId, string fileName, byte[] data, HttpStatusCode expected,
|
||||
string? capturedAt = null, bool restore = false)
|
||||
{
|
||||
using var response = await browser.SendAsync(HttpMethod.Post, "/api/v1/uploads", StartBody(sequenceId, fileName, data));
|
||||
using var response = await browser.SendAsync(HttpMethod.Post, "/api/v1/uploads", StartBody(sequenceId, fileName, data, capturedAt, restore));
|
||||
await response.ExpectAsync(expected);
|
||||
return await response.ReadAsync(ProtocolJson.Default.StartUploadResponse);
|
||||
}
|
||||
|
||||
/// <summary>A sequence made as the dialog's New sequence makes it.</summary>
|
||||
private static async Task<SequenceDto> CreateAsync(Browser browser, string title)
|
||||
{
|
||||
using var created = await browser.SendAsync(HttpMethod.Post, "/api/v1/sequences", Json($$"""{"title":"{{title}}","cameraHint":null}"""));
|
||||
await created.ExpectAsync(HttpStatusCode.Created);
|
||||
return await created.ReadAsync(ProtocolJson.Default.SequenceDto);
|
||||
}
|
||||
|
||||
/// <summary>Every chunk of an upload the dialog started, one after the other.</summary>
|
||||
private static async Task SendChunksAsync(Browser browser, Guid id, byte[] data)
|
||||
{
|
||||
for (var n = 0; n < (data.Length + Chunk - 1) / Chunk; n++)
|
||||
{
|
||||
using var put = await browser.SendAsync(HttpMethod.Put, $"/api/v1/uploads/{id}/chunks/{n}", Bytes(data, n));
|
||||
await put.ExpectAsync(HttpStatusCode.OK);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>The dialog's completion: a body of <c>{}</c> (api.post), <c>?restore=true</c> after Upload again.</summary>
|
||||
private static Task<HttpResponseMessage> CompleteAsync(Browser browser, Guid id, bool restore = false) =>
|
||||
browser.SendAsync(HttpMethod.Post, $"/api/v1/uploads/{id}/complete{(restore ? "?restore=true" : string.Empty)}", Json("{}"));
|
||||
|
||||
/// <summary>Started, sent and completed as the dialog does it; returns the frame.</summary>
|
||||
private static async Task<CompleteUploadResponse> UploadAsync(Browser browser, Guid sequenceId, string fileName, byte[] data, string? capturedAt = null,
|
||||
bool restore = false)
|
||||
{
|
||||
var start = await StartAsync(browser, sequenceId, fileName, data, HttpStatusCode.Created, capturedAt, restore);
|
||||
var id = Assert.NotNull(start.UploadId);
|
||||
await SendChunksAsync(browser, id, data);
|
||||
using var complete = await CompleteAsync(browser, id, restore);
|
||||
await complete.ExpectAsync(HttpStatusCode.OK);
|
||||
return await complete.ReadAsync(ProtocolJson.Default.CompleteUploadResponse);
|
||||
}
|
||||
|
||||
/// <summary>Deletes a photo from its sequence as the Library does, once no job is working on it.</summary>
|
||||
private static async Task DeleteAsync(TestServer server, Browser browser, Guid sequenceId, Guid frameId)
|
||||
{
|
||||
await UntilIdleAsync(server);
|
||||
var current = Assert.NotNull(await server.Database.Sequences.GetAsync(sequenceId));
|
||||
using var delete = await browser.SendAsync(HttpMethod.Post, $"/api/v1/sequences/{sequenceId}/frames/delete",
|
||||
Json($$"""{"frameIds":["{{frameId}}"],"expectedFramesVersion":{{current.FramesVersion}},"dryRun":false}"""));
|
||||
Assert.NotNull((await delete.ReadAsync(ProtocolJson.Default.FrameDeleteResultDto)).TrashEntry);
|
||||
}
|
||||
|
||||
/// <summary>Empties the whole trash as the Trash page does: the dry run, then the job with its fingerprint.</summary>
|
||||
private static async Task EmptyTrashAsync(TestServer server, Browser browser)
|
||||
{
|
||||
TrashEmptyPlanDto plan;
|
||||
using (var dry = await browser.SendAsync(HttpMethod.Post, "/api/v1/archive/trash/empty", Json("""{"names":null,"dryRun":true,"fingerprint":null}""")))
|
||||
plan = await dry.ReadAsync(ProtocolJson.Default.TrashEmptyPlanDto);
|
||||
Assert.True(plan.Entries.Count > 0, "the trash holds the deleted photo");
|
||||
using (var empty = await browser.SendAsync(HttpMethod.Post, "/api/v1/archive/trash/empty",
|
||||
Json($$"""{"names":null,"dryRun":false,"fingerprint":"{{plan.Fingerprint}}"}""")))
|
||||
await empty.ExpectAsync(HttpStatusCode.Accepted);
|
||||
await UntilIdleAsync(server);
|
||||
using var listed = await browser.SendAsync(HttpMethod.Get, "/api/v1/archive/trash");
|
||||
Assert.Empty((await listed.ReadAsync(ProtocolJson.Default.TrashListDto)).Entries);
|
||||
}
|
||||
|
||||
/// <summary>Waits until no job is queued or running (the thumbnails, proxies and analyses uploads queue).</summary>
|
||||
private static async Task UntilIdleAsync(TestServer server)
|
||||
{
|
||||
|
||||
Reference in new issue
Block a user