8ea25228ebeb3a3e88b1ad0950efd8bac849068f
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
8ea25228eb |
fix(convert): reserve the photos still writing, keep photo ids out of the job row, never copy a source twice
Stage A's space guard reserved, before each photo, only that photo and the ones after it, so at the end of a sequence it counted a single photo while up to twice the parallelism were still converting: a volume filling meanwhile could go into the reserve. It now reserves the photo and every photo started whose converted file has not landed yet (a file lands whole, by a rename, before it is verified), so nothing in flight is missed and nothing landed is counted twice, at the end of a sequence as anywhere. The confirmed photo ids (about 21 MB for a million photos) lived in the job's parameters, read by every dry run's busy check, by each sequence turn of the conversion and by the job lists. They are now written whole, under a job id chosen before the job is queued, into convert-<job>.photos beside the journal, and go with it; the row keeps each sequence's digest, so the job still converts exactly the photos confirmed (without the file, the whole plan made again must match). The busy check reads only ids, kinds and targets (and a verify's parameters), never the asking job's row; a conversion's turn reads only its creation time; the server's job list and dashboard and the engine's queue peeks read jobs without parameters. A retried New sequence conversion could make a second copy. The journal now names the copy as soon as its photos are filed; a retry finishes that copy whatever changed in the source since, never remakes one deleted since, and needs no space again for what it holds; and a source another conversion copied in the same format after this one was confirmed is done with, with no copy made (new outcome Superseded, with its job log line). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c669d0504d |
fix(convert): the job converts exactly the photos its preview listed, once
A conversion job could still do something its confirmed dry run did not show, or refuse what it should have converted: - An upload landing between the confirmation and the sequence's turn made the sequence Changed at every retry. Each ConvertSequence now names the photos the plan converts (Photos, their frame ids); the job takes those from its re-plan, checks their content against the confirmed digest and leaves photos that came in since as they are. A photo the plan did not list is never converted, whatever changed on disk or in the index since the dry run. - A file another program held during the dry run was planned as Missing and then converted or not depending on the re-plan. It is now its own reason (inaccessible, "held open or cannot be opened"); a truncated file reads as undecodable. At job time a confirmed photo that cannot be opened leaves the sequence Blocked for the retry; a held photo the job does not convert does not hide a real change. - The same sequences could be converted twice (two dialogs or tabs, a lost answer, no dry run at all). MassConverter.QueueAsync checks the queued, running and paused conversions and queues under one lock, in the caller's context (the desktop's enqueue shows a toast on the UI thread); a job's own re-plan treats sequences of conversions queued before it as busy. - The per-photo space check reserved a full batch of photos even when fewer were left, so a job whose preview had the space could stop at every retry. It now reserves the photo and those after it that may convert at once. - New sequence now says in the dry run, the panel's totals and confirmation, and a new desktop confirmation, how many photos each new sequence lacks; only sequences that are copied use those words. - On the server, a confirmation takes only the plan kept for the index as it is now (503 plan_dropped otherwise, never a plan made inside the request), and busy sequences and free space are as they are at every request. The panel's Apply asks the dry run again first, with Close still available, and confirms only the same fingerprint. - Maintenance keeps another host's conversion staging as long as its journal and purges both together; the unstaged-copy failure and blocked sequences are logged in catalog sentences. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit fefd9c34ebfeed1f9e84161610ef286026d38355) |
||
|
|
67a6d51cbe |
fix(archive): convert job applies exactly the confirmed plan, resumes safely and watches its space
The convert job planned every sequence again when it ran and checked only the frames version, so a verify, a repaired upload or a tier change between the dry run and the job let it replace photos the confirmed plan had listed as staying unchanged. - Each sequence of a convert job now carries the digest of the photos it converts, each with its SHA-256 (ConvertSequencePlan.Digest, also in the plan's fingerprint). Under the sequence's lease, its sidecar settled, the job plans it again and goes on only while the digest matches; it then pins those photos (convert-<job>.pins), so photos that come in or are verified later stay as they are, a retry included. Stage B checks under the lease that every pinned photo still holds the content converted instead of holding the frames version, so a sequence still receiving uploads converts what was confirmed. - A resumed job settles the journals a crash left in a sequence (a half-done replacement among them) before it plans, and reports a sequence its replacement completed as Converted, never Nothing or a partial swap. - The passes a converted or new sequence needs are queued as soon as its outcome is journaled; the journal records the statuses they follow from and whether they were queued, so a pause, yield, stop or crash no longer loses them. - Staged conversions of a blocked or failed sequence are kept for the retry, which re-verifies and takes them; maintenance keeps a convert job's staging while the job is queued, running or paused, and removes its journal and staging once the job is done or gone from the history. - A dry run counts the sequences of a queued conversion as busy, so the same conversion cannot be confirmed twice. - New sequence copies are owned by the source's owner, else by the requesting user's id (never the user name); a staged copy that vanished fails the sequence instead of leaving its photo out; the job log states how many photos stayed in the source alone. - Free space is measured where the conversion writes: the server takes the tighter of the space path's volume and the archive's own (Unraid cache), the desktop asks Windows (GetDiskFreeSpaceEx), which also measures UNC archives; the job checks again before each sequence and each photo, scaling estimates by the files staged so far, and stops before the volume fills with what it staged kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2d7d461c33b9f257bf57793cc7cf331d0425c14e) |
||
|
|
341fb77507 |
fix: conversions name, lock and settle as phase 4a's writers do
Phase 4b was built on phase 4a before its review fixes. Integrated on the reviewed 4a, the
conversion now follows the same rules as filing and the other writers.
- Names: a converted file is named as filing names a photo of its sequence (FrameNaming: the
archive's file template, the frame's context in the sequence with its camera, lens, number
and title, {name} the card's, {ext} the target's, a long name keeping its extension).
Replace through ConvertOptions.FileName; New sequence through the ingest itself
(StagedFrame.NamedAs), which numbers and names the copies in the new sequence. Names stay
unique against every file present, those leaving included, so a Replace never moves onto a
file; a test pins a stray file that keeps its bytes.
- Leases: stage A starts from the sidecar settled under the sequence's lease where the index
has it then (SequenceWriter.ReadLocked), and FrameTrash.ReplaceAsync reads the row again
under the lease, so a folder a Reorganize moved meanwhile is followed, never made again.
A Reorganize the archive waits for keeps the conversion off its sequences (pinned).
- Journals: the replace journal is settled with the others. MassRenamer steps, restores into
a sequence and sequence deletions wait for an unsettled replacement as they wait for a
restore, and so do photo deletions, whose frames version the recovery reads
(FrameTrash.EnsureReplaced); a scan settles it and reconciles nothing meanwhile; a restore
settles its sequence's replace journal first, at the folder the index has under the lease.
- The convert job tells each sequence converted or made as it completes, as a Reorganize
tells each step, so an editor open on it reads it again while the rest of the job runs.
- Desktop: JobKind.Convert gets its row in the settings' job priority list. A click on Apply
that commits the quality no longer starts the same dry run again under the click (as 4a
fixed for Reorganize).
- Panel: a dry run answered during the confirmation with the same plan no longer drops the
confirmed Apply; a different plan is shown with Admin.Convert.PlanChanged (en, it).
- Tests: TrashDialogTests.EmptyQueuesTheJobOfTheConfirmedPlanOnly waits for the command to
end before running it again (a running AsyncRelayCommand ignored the second Execute).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 2e0d1d6dcc10575734970bb9c77749f00777c859)
|
||
|
|
0e52929274 |
feat(archive): convert sequences to JPEG, TIFF-16 or linear DNG after import
Owner review 2, R7 asks for mass conversion after import on the web panel and the desktop, with a choice at every run between Replace and New sequence. This is the engine and the server API both UIs build on. - Core/Conversion: ConvertOptions (JPEG 4:4:4 at quality 1-100, default 95; TIFF-16 sRGB Deflate with the predictor; linear DNG, the archival choice; converted names keep the stem and take the target's extension in the old one's case). FrameConverter decodes at the native tier only, encodes with the in-house writers, writes Exif from the sidecar (capture time to the tick, offset, exposure) and the original (camera, serial, lens, modes, GPS), hashes the bytes as made, writes atomically and verifies: read back, the metadata again, a decode by the server's decoders. ExifWriter now writes the exposure program, metering mode, flash and white balance. ConvertPlanner: convertible frames, skips with reasons, size estimates per pixel, the plan's space and fingerprint. - FrameTrash.ReplaceAsync swaps the files of frames in place: same ids and indices, the old files and their XMP companions into one Replaced trash entry (records, replacement hashes, a copy of analysis.json), tombstones aliasing old content to new, proxies of the swapped frames dropped, under .chronos/replace.pending.json, which the next writer undoes or completes. Restoring a Replaced entry swaps back while every converted frame is unchanged (the converted files become a Replaced entry in turn), else the originals come back as a new sequence. - MassConverter and JobKind.Convert (19): the dry run (busy, offline, unreadable, space against ArchiveServiceOptions.FreeSpace), stage A converting into staging/convert-<job>/ in parallel with each source re-hashed and each staged file appended to journal/convert-<job>.staged, stage B per sequence all or nothing under the lease at the planned frames version: Replace, or a new sequence through the ingest with the source's tags, rating, notes and project. Resumes keep verified staged files; a sequence changed, busy or with a failed photo is left as it was and fails the job at the end, the journal kept for a retry. - POST /api/v1/archive/convert: dry run with the plan, its space line and fingerprint, or a 202 convert job; formats the server cannot read at full resolution are skipped as convert_on_pc; validated, audited (archive_converted), sequenceChanged converted/created. The server measures the space volume for the check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 809210d924cfc3f597ecdea26106fb3ff62d3762) (cherry picked from commit e13d4c0bb1bd9441933b6f3e5b6220e865a7228c) |