Commit Graph
7 Commits
Author SHA1 Message Date
Alby96andClaude Opus 5.5 b471a33bfc fix(convert): a conversion never changes what Chronos reads from a photo
Owner review 2, phase 4b, fix round 3 (codecs).

Chronos's decoders never read an ICC profile: they take code values through
the sRGB curve and float samples as they are. Round 2 wrote a float TIFF's
values with its profile's own curves, so after a Replace the renders,
thumbnails, analyses and deflicker of an Adobe RGB, ProPhoto or gamma-1.0
float TIFF saw other pixels (up to 2.7 EV darker, shadows lifted). Every
converted file now holds the values the decoders read from the original, as
they read them back, and says truthfully what they are:

- A JPEG or TIFF-16 keeps the codes of code values and carries the original's
  profile. Float values are written as sRGB codes with a profile made for
  them: the original's colorants and media white with the sRGB curve. A gray
  profile's codes are kept, its curve carried as an RGB profile (they were
  converted to sRGB before, which changed them too).
- A Linear DNG, whose matrices Chronos develops it with as well, takes only
  values that are sRGB's as the decoders read them; any other profile is
  refused as color_profile instead of converted to colours Chronos would then
  show differently.
- A profile with colorant tags that convert nothing is never sRGB by its name.
- A float file with NaN or infinite samples fails as damaged data instead of
  being written black (and spending a DNG's headroom).

The plan and the decoders behind it:

- TooLarge and WorstBytes answer any size, int.MaxValue sides included,
  without a product that wraps around; the worst case saturates.
- A frame another host flagged unreadable (the server, for a CMYK or CIELab
  TIFF its reader refuses) is asked again with a small decode where the
  conversion has other decoders (WIC on a PC), so a PC still converts it.
  The too_large sentences say "or Chronos reads back" (en, it).
- WIC's TIFF decoder decodes an unknown compression as zeros, does not undo
  the floating-point predictor of Photoshop's 32-bit files and reads 32-bit
  integers as floats, all without an error: such TIFFs are left to the
  managed reader, on the desktop as in conversions, never black or noise.
- A conversion's WIC probe reads the headers (decoder, first frame's oriented
  size, pixel format), so the size bound applies to what only WIC reads; a
  read error stays a read error (inaccessible), never a verdict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:36:00 +01:00
Alby96 c669d0504d fix(convert): the job converts exactly the photos its preview listed, once
A conversion job could still do something its confirmed dry run did not show,
or refuse what it should have converted:

- An upload landing between the confirmation and the sequence's turn made the
  sequence Changed at every retry. Each ConvertSequence now names the photos
  the plan converts (Photos, their frame ids); the job takes those from its
  re-plan, checks their content against the confirmed digest and leaves photos
  that came in since as they are. A photo the plan did not list is never
  converted, whatever changed on disk or in the index since the dry run.
- A file another program held during the dry run was planned as Missing and
  then converted or not depending on the re-plan. It is now its own reason
  (inaccessible, "held open or cannot be opened"); a truncated file reads as
  undecodable. At job time a confirmed photo that cannot be opened leaves the
  sequence Blocked for the retry; a held photo the job does not convert does
  not hide a real change.
- The same sequences could be converted twice (two dialogs or tabs, a lost
  answer, no dry run at all). MassConverter.QueueAsync checks the queued,
  running and paused conversions and queues under one lock, in the caller's
  context (the desktop's enqueue shows a toast on the UI thread); a job's own
  re-plan treats sequences of conversions queued before it as busy.
- The per-photo space check reserved a full batch of photos even when fewer
  were left, so a job whose preview had the space could stop at every retry.
  It now reserves the photo and those after it that may convert at once.
- New sequence now says in the dry run, the panel's totals and confirmation,
  and a new desktop confirmation, how many photos each new sequence lacks;
  only sequences that are copied use those words.
- On the server, a confirmation takes only the plan kept for the index as it
  is now (503 plan_dropped otherwise, never a plan made inside the request),
  and busy sequences and free space are as they are at every request. The
  panel's Apply asks the dry run again first, with Close still available,
  and confirms only the same fingerprint.
- Maintenance keeps another host's conversion staging as long as its journal
  and purges both together; the unstaged-copy failure and blocked sequences
  are logged in catalog sentences.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit fefd9c34ebfeed1f9e84161610ef286026d38355)
2026-11-02 10:33:34 +01:00
Alby96 5b5acaf307 fix(convert): plan, decode and colour only what a conversion really does
Round-2 fixes for the codecs part of owner review 2, phase 4b (mass
conversion), so the dry run never promises a conversion that then fails
or writes wrong pixels.

- The managed TIFF reader refuses samples it does not render (CMYK,
  CIELab, palette, LogL, RGB of fewer than three samples, float
  min-is-white) before reading pixels, instead of decoding them as the
  gray of their first plane. The conversion chain then asks WIC on a PC;
  the server leaves them to a PC. Impossible strip/tile counts are
  refused as data, not as an OverflowException that escaped the chain.
- A conversion's managed provider reads JPEG, PNG and TIFF headers in
  its probe (ReadsHeaders): what its decoders refuse there (compression,
  sample format, photometric, MaxSamples, MaxPixels, arithmetic coding)
  is not planned as convertible on the server, and is asked of WIC on a
  PC. The probe gives the real size, so TooLarge sees it.
- Float TIFFs are linear light (SourceColor.IsLinear): a JPEG or TIFF
  carrying their profile gets codes from the profile's own curves, a DNG
  gets them through the matrix only (no curve, no clip, highlights
  kept). IsSrgb is decided by the numbers for matrix/TRC profiles, so a
  linear profile named sRGB is converted, not taken as sRGB. The TIFF
  profile is read from the image the decoders read.
- TooLarge is a bound: the target's limits and what the server's
  decoders read back, which every converted file must (JPEG 2^28 pixels,
  TIFF-16 and DNG 2^29 samples), and the incompressible worst case
  against the 2 GB array. A MemoryStream overflow is TooLarge, never a
  generic file failure.
- Colorants that convert nothing (a white without positive cone
  responses, a non-finite or singular matrix) are dropped when a profile
  is parsed: the photo is carried into a JPEG or TIFF and planned out of
  a DNG as colour_profile, instead of throwing in BradfordAdaptation at
  every retry.

Tests: header probes, refused photometrics, the size bound against the
verifier's header reads, float TIFF colour, a fuzz of damaged profiles,
and a Windows test converting a CMYK TIFF through WIC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 18cf9133087aada2db06c8310d2c69ff804c0115)
2026-11-02 10:33:33 +01:00
Alby96 67a6d51cbe fix(archive): convert job applies exactly the confirmed plan, resumes safely and watches its space
The convert job planned every sequence again when it ran and checked only the frames
version, so a verify, a repaired upload or a tier change between the dry run and the job
let it replace photos the confirmed plan had listed as staying unchanged.

- Each sequence of a convert job now carries the digest of the photos it converts, each
  with its SHA-256 (ConvertSequencePlan.Digest, also in the plan's fingerprint). Under the
  sequence's lease, its sidecar settled, the job plans it again and goes on only while the
  digest matches; it then pins those photos (convert-<job>.pins), so photos that come in
  or are verified later stay as they are, a retry included. Stage B checks under the lease
  that every pinned photo still holds the content converted instead of holding the frames
  version, so a sequence still receiving uploads converts what was confirmed.
- A resumed job settles the journals a crash left in a sequence (a half-done replacement
  among them) before it plans, and reports a sequence its replacement completed as
  Converted, never Nothing or a partial swap.
- The passes a converted or new sequence needs are queued as soon as its outcome is
  journaled; the journal records the statuses they follow from and whether they were
  queued, so a pause, yield, stop or crash no longer loses them.
- Staged conversions of a blocked or failed sequence are kept for the retry, which
  re-verifies and takes them; maintenance keeps a convert job's staging while the job is
  queued, running or paused, and removes its journal and staging once the job is done or
  gone from the history.
- A dry run counts the sequences of a queued conversion as busy, so the same conversion
  cannot be confirmed twice.
- New sequence copies are owned by the source's owner, else by the requesting user's id
  (never the user name); a staged copy that vanished fails the sequence instead of leaving
  its photo out; the job log states how many photos stayed in the source alone.
- Free space is measured where the conversion writes: the server takes the tighter of the
  space path's volume and the archive's own (Unraid cache), the desktop asks Windows
  (GetDiskFreeSpaceEx), which also measures UNC archives; the job checks again before each
  sequence and each photo, scaling estimates by the files staged so far, and stops before
  the volume fills with what it staged kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 2d7d461c33b9f257bf57793cc7cf331d0425c14e)
2026-10-27 22:06:55 +01:00
Alby96 f86a151a5d fix(convert): keep what the original holds and plan only what converts
Phase 4b review, codec findings. A Replace makes the converted file the
photo, so a conversion must not lose what the original held, and the dry
run must promise only what the job will do.

- Linear DNG from a conversion is scene-referred: the pixels are stored in
  linear ProPhoto primaries (colours outside sRGB stay positive) and
  divided by the power of two that brings their peak under the white
  level (at most 4 stops), with BaselineExposure giving it back and
  ColorMatrix1/ForwardMatrix1 describing that space. Highlights a RAW
  develops above 1 and its wide-gamut colours now read back as written.
  Exports keep the clipped linear-sRGB DNG.
- A conversion reads with the managed decoders first for every format they
  read natively (RAWs developed at float precision, as on the server) and
  with WIC only for what they cannot. A RAW only WIC's Raw Image Extension
  develops (8 bits) is refused for TIFF-16 and linear DNG and planned out
  as "eight_bit"; JPEG still converts it. Providers say so through
  IImageDecoderProvider.DevelopsRawAt8Bits.
- The original's colour space is kept: a JPEG (APP2) or TIFF (tag 34675)
  carries its ICC profile, with the values unchanged; a linear DNG gets the
  values converted through a matrix/TRC profile; a gray profile converts
  to sRGB. Exif Adobe RGB (uncalibrated + R03) and HEIF nclx P3 get an
  in-house profile. A space nothing here can keep (CMYK, a lookup-table
  profile into a DNG, uncalibrated without R03, HDR nclx) is planned out
  as "color_profile". New in-house ICC parser/writer and header reader
  for JPEG, PNG, TIFF, HEIF/AVIF and WebP.
- The planner asks every file, whatever tier was recorded (unknown before
  the light analysis, or by another host), with the conversion's decoders,
  headers only and eight at a time; a RAW the server cannot develop is
  "convert_on_pc" in the plan instead of failing its sequence. Photos a
  target cannot hold (a JPEG side above 65535, more samples than one
  buffer) are planned out as "too_large", and an overflow fails its photo
  rather than the job.
- A photo that fails reads in the job log as one catalog sentence by cause
  (Archive.Convert.Failed.*), the file named once; the exception's English
  text goes to the diagnostics log only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 0ca1e91ea09946d7a55f45695ee58e1a0ff5f779)
2026-10-27 22:04:34 +01:00
Alby96 341fb77507 fix: conversions name, lock and settle as phase 4a's writers do
Phase 4b was built on phase 4a before its review fixes. Integrated on the reviewed 4a, the
conversion now follows the same rules as filing and the other writers.

- Names: a converted file is named as filing names a photo of its sequence (FrameNaming: the
  archive's file template, the frame's context in the sequence with its camera, lens, number
  and title, {name} the card's, {ext} the target's, a long name keeping its extension).
  Replace through ConvertOptions.FileName; New sequence through the ingest itself
  (StagedFrame.NamedAs), which numbers and names the copies in the new sequence. Names stay
  unique against every file present, those leaving included, so a Replace never moves onto a
  file; a test pins a stray file that keeps its bytes.
- Leases: stage A starts from the sidecar settled under the sequence's lease where the index
  has it then (SequenceWriter.ReadLocked), and FrameTrash.ReplaceAsync reads the row again
  under the lease, so a folder a Reorganize moved meanwhile is followed, never made again.
  A Reorganize the archive waits for keeps the conversion off its sequences (pinned).
- Journals: the replace journal is settled with the others. MassRenamer steps, restores into
  a sequence and sequence deletions wait for an unsettled replacement as they wait for a
  restore, and so do photo deletions, whose frames version the recovery reads
  (FrameTrash.EnsureReplaced); a scan settles it and reconciles nothing meanwhile; a restore
  settles its sequence's replace journal first, at the folder the index has under the lease.
- The convert job tells each sequence converted or made as it completes, as a Reorganize
  tells each step, so an editor open on it reads it again while the rest of the job runs.
- Desktop: JobKind.Convert gets its row in the settings' job priority list. A click on Apply
  that commits the quality no longer starts the same dry run again under the click (as 4a
  fixed for Reorganize).
- Panel: a dry run answered during the confirmation with the same plan no longer drops the
  confirmed Apply; a different plan is shown with Admin.Convert.PlanChanged (en, it).
- Tests: TrashDialogTests.EmptyQueuesTheJobOfTheConfirmedPlanOnly waits for the command to
  end before running it again (a running AsyncRelayCommand ignored the second Execute).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 2e0d1d6dcc10575734970bb9c77749f00777c859)
2026-10-27 22:03:07 +01:00
Alby96 0e52929274 feat(archive): convert sequences to JPEG, TIFF-16 or linear DNG after import
Owner review 2, R7 asks for mass conversion after import on the web panel and the desktop,
with a choice at every run between Replace and New sequence. This is the engine and the
server API both UIs build on.

- Core/Conversion: ConvertOptions (JPEG 4:4:4 at quality 1-100, default 95; TIFF-16 sRGB
  Deflate with the predictor; linear DNG, the archival choice; converted names keep the
  stem and take the target's extension in the old one's case). FrameConverter decodes at
  the native tier only, encodes with the in-house writers, writes Exif from the sidecar
  (capture time to the tick, offset, exposure) and the original (camera, serial, lens,
  modes, GPS), hashes the bytes as made, writes atomically and verifies: read back, the
  metadata again, a decode by the server's decoders. ExifWriter now writes the exposure
  program, metering mode, flash and white balance. ConvertPlanner: convertible frames,
  skips with reasons, size estimates per pixel, the plan's space and fingerprint.
- FrameTrash.ReplaceAsync swaps the files of frames in place: same ids and indices, the old
  files and their XMP companions into one Replaced trash entry (records, replacement
  hashes, a copy of analysis.json), tombstones aliasing old content to new, proxies of the
  swapped frames dropped, under .chronos/replace.pending.json, which the next writer undoes
  or completes. Restoring a Replaced entry swaps back while every converted frame is
  unchanged (the converted files become a Replaced entry in turn), else the originals come
  back as a new sequence.
- MassConverter and JobKind.Convert (19): the dry run (busy, offline, unreadable, space
  against ArchiveServiceOptions.FreeSpace), stage A converting into
  staging/convert-<job>/ in parallel with each source re-hashed and each staged file
  appended to journal/convert-<job>.staged, stage B per sequence all or nothing under the
  lease at the planned frames version: Replace, or a new sequence through the ingest with
  the source's tags, rating, notes and project. Resumes keep verified staged files; a
  sequence changed, busy or with a failed photo is left as it was and fails the job at the
  end, the journal kept for a retry.
- POST /api/v1/archive/convert: dry run with the plan, its space line and fingerprint, or a
  202 convert job; formats the server cannot read at full resolution are skipped as
  convert_on_pc; validated, audited (archive_converted), sequenceChanged converted/created.
  The server measures the space volume for the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 809210d924cfc3f597ecdea26106fb3ff62d3762)
(cherry picked from commit e13d4c0bb1bd9441933b6f3e5b6220e865a7228c)
2026-10-27 22:01:23 +01:00