Files
Chronos/src/Chronos.Core/Conversion/ConvertPlanner.cs
T
Alby96andClaude Opus 5.5 efa87f68ce
CI / Build and test (push) Failing after 3m1s
CI / PostgreSQL 16 integration (push) Skipped
CI / PostgreSQL 17 integration (push) Skipped
CI / Docker image (push) Skipped
CI / Windows tests and installer (push) Skipped
CI / YAML check (push) Successful in 10s
Release / Build, test and publish (push) Successful in 15m31s
Release / Server image (push) Successful in 1m13s
feat: owner review 2 phase 4b: mass conversion after import (#13)
Convert photos already in the archive to JPEG, TIFF-16 or Linear DNG, from the panel's Library and the desktop's Archive and Library. Each run asks Replace (same frames, originals to the trash, recoverable) or New sequence. A dry run lists what converts, what is skipped and why, size before and after, and free space; the job converts exactly the confirmed photos, resumes, and commits all or nothing per sequence. A conversion never changes what Chronos reads from a photo. Replaced originals are erased only on the archive's own word that their conversion is gone. Desktop TIFFs that WIC showed black or as noise now go to the managed reader; the Edit picker no longer spins a core.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Alberto Balbo <alberto.balbo.96@gmail.com>
2026-11-03 10:41:50 +01:00

385 lines
21 KiB
C#
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
using System.Globalization;
using System.Security.Cryptography;
using System.Text;
using Chronos.Core.Archiving;
using Chronos.Core.Codecs.Icc;
using Chronos.Core.Codecs.Jpeg;
using Chronos.Core.Codecs.Tiff;
using Chronos.Core.Export;
using Chronos.Core.Imaging;
using Chronos.Core.Imaging.Decoders;
using Chronos.Core.Model;
namespace Chronos.Core.Conversion;
/// <summary>Why a frame of a sequence is not converted (owner review 2, R7); nothing of it is touched.</summary>
public enum ConvertSkip
{
/// <summary>The photo is in the target's format already (a TIFF of 8 bits included).</summary>
SameFormat,
/// <summary>
/// This host has no full-resolution decoder for the photo's format (a server and a CR3 or HEIC), or its decoders do not
/// read this file at full resolution (a CMYK, CIELab or palette TIFF, a compression or size they do not take, a RAW body
/// they do not develop and without a preview): a PC may have one.
/// </summary>
NoNativeDecoder,
/// <summary>This host reads the photo only from its embedded preview (a camera body its RAW decoder does not support).</summary>
PreviewOnly,
/// <summary>
/// The photo is flagged unreadable, damaged or corrupt: no decoder produced its pixels. A flag set where other decoders
/// read the file (the server's, for a CMYK TIFF that WIC reads on a PC) holds only where a decode says so too.
/// </summary>
Undecodable,
/// <summary>The photo's file is not there.</summary>
Missing,
/// <summary>The photo was never hashed (adopted, not verified yet): its replacement could not name it. Verify it first.</summary>
Unhashed,
/// <summary>A 16-bit target, and this host develops the RAW at 8 bits only (WIC's Raw Image Extension): JPEG converts it.</summary>
EightBit,
/// <summary>The photo's colour profile can be neither carried into the target nor converted to it (CMYK, an unknown space…).</summary>
ColorProfile,
/// <summary>
/// The photo is larger than the target holds (a JPEG side above 65535) or than Chronos reads back: the server's decoders,
/// which every converted file must decode with (<see cref="ConvertPlanner.TooLarge"/>).
/// </summary>
TooLarge,
/// <summary>
/// The photo's file is there but could not be opened when the plan was made: another program holds it without sharing
/// (a sync client, an editor), or access is denied. A plan made once it is free converts it.
/// </summary>
Inaccessible,
}
/// <summary>Why a whole sequence is not converted now; nothing of it is touched.</summary>
public enum ConvertBlock
{
/// <summary>A running or paused job works on the sequence or its whole archive, or a conversion of it waits in the queue.</summary>
Busy,
/// <summary>The archive is offline or read-only.</summary>
Offline,
/// <summary>The sequence's sidecar cannot be read, or belongs to another sequence.</summary>
Unreadable,
/// <summary>
/// The photos the sequence converts are no longer those of the plan confirmed (found by the job, which then leaves it as
/// it is).
/// </summary>
FramesChanged,
}
/// <summary>Frames of a sequence that are not converted, and why.</summary>
public sealed record ConvertSkipped(ConvertSkip Reason, int Frames);
/// <summary>
/// One frame a conversion converts: its id and index, its file and format, its size, the converted file's estimated size,
/// and the SHA-256 of the content converted.
/// </summary>
public sealed record ConvertFramePlan(Guid FrameId, int Index, string File, ImageFileFormat Format, long Size, long EstimatedBytes, string? Sha256 = null);
/// <summary>
/// One sequence of a conversion plan: the frames converted, those skipped with why, the frames version the plan was made at,
/// and why the whole sequence cannot be converted now (<see cref="Blocked"/>).
/// </summary>
public sealed record ConvertSequencePlan(Guid SequenceId, string Title, string Folder, int FramesVersion, IReadOnlyList<ConvertFramePlan> Frames,
IReadOnlyList<ConvertSkipped> Skipped, IReadOnlyList<ConvertBlock> Blocked)
{
/// <summary>Whether the sequence is converted: something to convert and nothing blocking it.</summary>
public bool Converts => Blocked.Count == 0 && Frames.Count > 0;
/// <summary>Size of the originals converted.</summary>
public long BytesBefore => Frames.Sum(f => f.Size);
/// <summary>Estimated size of the converted files.</summary>
public long BytesAfter => Frames.Sum(f => f.EstimatedBytes);
public int SkippedFrames => Skipped.Sum(s => s.Frames);
/// <summary>
/// A digest of the photos the sequence converts: each frame in order with the SHA-256 of the content converted. A
/// conversion job converts the sequence only while its plan there still has the digest the user confirmed.
/// </summary>
public string Digest() => DigestOf(Frames.Select(f => (f.FrameId, f.Sha256)));
/// <summary>The digest of frames, each with the SHA-256 of its content, in order (<see cref="Digest"/>).</summary>
public static string DigestOf(IEnumerable<(Guid FrameId, string? Sha256)> frames)
{
ArgumentNullException.ThrowIfNull(frames);
var text = new StringBuilder();
foreach (var (id, sha) in frames) text.Append(CultureInfo.InvariantCulture, $"{id:N}|{sha?.ToLowerInvariant()}\n");
return Convert.ToHexStringLower(SHA256.HashData(Encoding.UTF8.GetBytes(text.ToString())));
}
}
/// <summary>
/// The free space a conversion needs on the archive's volume (<see cref="Required"/>: the estimate with a margin and a
/// reserve, <see cref="ConvertPlanner.Required"/>) and what it has; <see cref="Free"/> null when it cannot be measured.
/// </summary>
public sealed record ConvertSpace(long Required, long? Free)
{
public bool Enough => Free is not { } free || free >= Required;
}
/// <summary>
/// The dry run of a mass conversion (owner review 2, R7): the options and mode, each sequence's plan, the space it needs,
/// and the fingerprint the confirmation sends back (<see cref="Fingerprint"/>).
/// </summary>
public sealed record ConvertPlan(Guid ArchiveId, ConvertOptions Options, ConvertMode Mode, IReadOnlyList<ConvertSequencePlan> Sequences, ConvertSpace Space)
{
/// <summary>The sequences that are converted.</summary>
public IEnumerable<ConvertSequencePlan> Converted => Sequences.Where(s => s.Converts);
public int Frames => Converted.Sum(s => s.Frames.Count);
public long BytesBefore => Converted.Sum(s => s.BytesBefore);
public long BytesAfter => Converted.Sum(s => s.BytesAfter);
public int Skipped => Sequences.Sum(s => s.SkippedFrames);
/// <summary>
/// A digest of what the plan converts (options, mode, and each sequence in order with its frames version, the photos
/// converted with their content (<see cref="ConvertSequencePlan.Digest"/>), those skipped, and what blocks it), the
/// same for two plans of the same archive state.
/// </summary>
public string Fingerprint()
{
var text = new StringBuilder().Append(Options.Key).Append('|').Append(Mode).Append('\n');
foreach (var s in Sequences)
{
text.Append(CultureInfo.InvariantCulture, $"{s.SequenceId:N}|{s.FramesVersion}|{s.Frames.Count}|{s.BytesBefore}|{s.Digest()}|");
foreach (var skipped in s.Skipped) text.Append(CultureInfo.InvariantCulture, $"{skipped.Reason}:{skipped.Frames};");
foreach (var blocked in s.Blocked) text.Append(blocked).Append(';');
text.Append('\n');
}
return Convert.ToHexStringLower(SHA256.HashData(Encoding.UTF8.GetBytes(text.ToString())));
}
}
/// <summary>
/// Plans the conversion of a sequence's frames (owner review 2, R7): a frame is converted when it is not in the target's
/// format already, its file is there, it was hashed, it is not flagged unreadable (or another host's decoders flagged it and
/// this host's decode it), its size is one the target holds and the server's decoders read back (<see cref="TooLarge"/>),
/// this host decodes it at full resolution with the decoders the conversion uses (<see cref="FrameConverter.Decoders"/>: its
/// format has a native decoder here and the file itself probes native now, whatever tier another host or an earlier pass
/// recorded, its headers saying what those decoders take; for a 16-bit target, not through a RAW development at 8 bits),
/// and its colour profile can be kept (<see cref="FrameConverter.Treatment"/>). The plan asks the files what the conversion
/// will ask them, reading headers only (and decoding small the few frames flagged elsewhere), several at once; a file it
/// cannot read now is inaccessible, never a verdict on its content. Sizes after are estimated from the pixels
/// (<see cref="EstimateBytes"/>).
/// </summary>
public static class ConvertPlanner
{
/// <summary>Files asked at once while a sequence is planned: header reads, bound by the disk more than the cores.</summary>
private const int ProbesAtOnce = 8;
/// <summary>Long edge of the decode that asks a frame flagged unreadable elsewhere whether this host's decoders read it.</summary>
private const int ConfirmEdge = 64;
/// <summary>Margin over the estimate the free space must hold (estimates of compressed files vary with the content).</summary>
public const double SpaceMargin = 1.15;
/// <summary>Space left free on the volume after the conversion, for the index, sidecars and everything else writing there.</summary>
public const long SpaceReserve = 512L << 20;
/// <summary>Pixels assumed for a frame whose size is not known (a 24 MP sensor).</summary>
public const long UnknownPixels = 24_000_000;
/// <summary>
/// Bits per pixel of a 4:4:4 JPEG over a 4:2:0 one of the same quality: the chroma is coded at full resolution, about a
/// quarter more bits on photographs.
/// </summary>
private const double FullChroma = 1.25;
/// <summary>
/// Bits per pixel of pixels that do not compress at all (noise) in the costliest target: three 16-bit samples, 48 bits
/// as Deflate stores them, and less than a bit more each as lossless JPEG codes them (an optimal Huffman code exceeds
/// the entropy of what it codes by less than one bit); a 4:4:4 JPEG of noise at quality 100 takes about 22.
/// </summary>
public const int WorstBitsPerPixel = 51;
/// <summary>
/// Margin on <see cref="WorstBitsPerPixel"/>: Deflate's stored blocks and zlib framing per strip, lossless JPEG's byte
/// stuffing (a 0xFF byte in 256 of noise) and headers per tile stay below 1 %.
/// </summary>
private const double WorstMargin = 1.01;
/// <summary>What a converted file holds beyond its pixels at most: an ICC profile carried whole, the Exif block, the directory and its strip or tile tables.</summary>
private const long WorstOverhead = IccProfile.MaxBytes + (4L << 20);
/// <summary>The plan of one sequence from its sidecar.</summary>
/// <param name="present">Every file in the folders of the sequence's frames, relative to the sequence folder
/// (<see cref="FileRenames.ListPresent"/>): the frames whose file is not among them are missing.</param>
/// <param name="decoder">This host's decoders.</param>
public static ConvertSequencePlan Plan(SequenceManifest manifest, SequencePaths folder, IReadOnlyCollection<string> present, ConvertOptions options,
ImageDecoder decoder, string title)
{
ArgumentNullException.ThrowIfNull(manifest);
ArgumentNullException.ThrowIfNull(folder);
ArgumentNullException.ThrowIfNull(present);
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(decoder);
var onDisk = present as IReadOnlySet<string> ?? new HashSet<string>(present, StringComparer.OrdinalIgnoreCase);
var reasons = new ConvertSkip?[manifest.Frames.Count];
Parallel.For(0, reasons.Length, new ParallelOptions { MaxDegreeOfParallelism = ProbesAtOnce },
i => reasons[i] = Skip(manifest.Frames[i], folder, onDisk, options, decoder));
var skipped = new Dictionary<ConvertSkip, int>();
var frames = new List<ConvertFramePlan>();
for (var i = 0; i < reasons.Length; i++)
{
var frame = manifest.Frames[i];
if (reasons[i] is { } reason)
{
skipped[reason] = skipped.GetValueOrDefault(reason) + 1;
continue;
}
frames.Add(new ConvertFramePlan(frame.Id, frame.I, frame.File, frame.Format, frame.Size, EstimateBytes(options, frame.W, frame.H), frame.Sha256));
}
return new ConvertSequencePlan(manifest.Id, title, folder.RelativeFolder, manifest.Versions.Frames, frames,
[.. skipped.OrderBy(s => s.Key).Select(s => new ConvertSkipped(s.Key, s.Value))], []);
}
/// <summary>Why <paramref name="frame"/> is not converted, or null when it is.</summary>
/// <param name="decoder">This host's decoders; the conversion's are made from them (<see cref="FrameConverter.Decoders"/>).</param>
public static ConvertSkip? Skip(SidecarFrame frame, SequencePaths folder, IReadOnlySet<string> present, ConvertOptions options, ImageDecoder decoder)
{
ArgumentNullException.ThrowIfNull(frame);
ArgumentNullException.ThrowIfNull(folder);
ArgumentNullException.ThrowIfNull(present);
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(decoder);
if (frame.Format == options.Format) return ConvertSkip.SameFormat;
if ((frame.Flags & FrameFlags.Missing) != 0 || !present.Contains(frame.File)) return ConvertSkip.Missing;
if (frame.Sha256 is null) return ConvertSkip.Unhashed;
if ((frame.Flags & FrameFlags.Corrupt) != 0) return ConvertSkip.Undecodable;
var decoders = FrameConverter.Decoders(decoder, options, frame.Format);
// Flagged unreadable by the host that analysed it last: on a server, whose managed decoders refuse what they do not
// read (a CMYK or CIELab TIFF), that holds for a PC only where its own decoders (WIC) do not read it either.
var flagged = (frame.Flags & FrameFlags.Undecodable) != 0;
if (flagged && !OthersRead(decoders, frame.Format)) return ConvertSkip.Undecodable;
if (TooLarge(options, frame.W, frame.H)) return ConvertSkip.TooLarge;
var eightBit = FrameConverter.OnlyAt8Bits(decoder, options, frame.Format);
if (decoders.TierFor(frame.Format) != DecodeTier.Native) return eightBit ? ConvertSkip.EightBit : ConvertSkip.NoNativeDecoder;
// Whatever tier was recorded (unknown before the light analysis, or by another host with other decoders, or for a
// body this host's RAW decoder does not support), the file itself is asked: headers only, and nothing at all for the
// formats every file of which decodes alike.
var path = folder.File(frame.File);
try
{
// Not read at full resolution here, although the format is: what the file holds (its encoding, its size, a RAW
// body) is beyond these decoders, and only a preview, or nothing, is left.
var probe = decoders.Probe(path, frame.Format);
if (probe.Tier != DecodeTier.Native)
{
return flagged ? ConvertSkip.Undecodable
: eightBit ? ConvertSkip.EightBit
: probe.Tier == DecodeTier.EmbeddedPreview ? ConvertSkip.PreviewOnly
: ConvertSkip.NoNativeDecoder;
}
if (TooLarge(options, probe.Width, probe.Height)) return ConvertSkip.TooLarge;
// The flag was another host's verdict, whose decoders failed where the headers say nothing: a decode here says
// whether these read its pixels.
if (flagged && !Decodes(decoders, path)) return ConvertSkip.Undecodable;
return FrameConverter.Treatment(SourceColor.Read(path, frame.Format), options.Target) == ColorTreatment.Refuse ? ConvertSkip.ColorProfile : null;
}
catch (Exception ex) when (ex is FileNotFoundException or DirectoryNotFoundException)
{
// Gone since the folder was listed.
return ConvertSkip.Missing;
}
catch (EndOfStreamException)
{
// Opened, and shorter than its own headers say: no decoder gets its pixels.
return ConvertSkip.Undecodable;
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
// There, but held by another program or denied, or a read failed: not missing, and not planned (a conversion job
// converts only the photos its plan listed), so the dry run says why. Nothing is concluded of its content.
return ConvertSkip.Inaccessible;
}
}
/// <summary>
/// Whether <paramref name="decoders"/> have, for <paramref name="format"/>, a native decoder besides the managed ones
/// every host has (WIC on a PC): a frame those flagged unreadable may still decode here.
/// </summary>
private static bool OthersRead(ImageDecoder decoders, ImageFileFormat format) =>
decoders.Providers.Any(p => p is not ManagedDecoderProvider && p.TierFor(format) == DecodeTier.Native);
/// <summary>
/// Whether <paramref name="decoders"/> produce the pixels of the file at <paramref name="path"/> at full-resolution tier,
/// asked with a small decode.
/// </summary>
/// <exception cref="IOException">The file cannot be read.</exception>
private static bool Decodes(ImageDecoder decoders, string path)
{
using var decoded = decoders.Decode(new DecodeRequest(path, ConfirmEdge, DecodePurpose.Thumbnail, ApplyOrientation: false, AllowPreview: false));
return decoded is { Tier: DecodeTier.Native, ByChoice: false };
}
/// <summary>
/// Whether a frame of <paramref name="width"/> × <paramref name="height"/> (unknown: 0) cannot be converted to
/// <paramref name="options"/>, a bound and never an estimate: the target does not hold it (a JPEG side above
/// <see cref="FrameConverter.MaxJpegSide"/>), Chronos does not read it back (the server's decoders, which every converted
/// file must decode with, <see cref="FrameConverter.Verify"/>: a JPEG of more pixels than
/// <see cref="JpegDecodeOptions.MaxPixels"/>, a TIFF-16 or DNG of more samples than <see cref="TiffDecodeOptions.MaxSamples"/>),
/// or its file could outgrow the array it is made in (<see cref="WorstBytes"/>, which those sizes keep below it). What it
/// lets through fits one image buffer, its file the array it is made in, and the server's decoders read it back. Any
/// size is answered, a damaged header's included: nothing is multiplied beyond what a long holds.
/// </summary>
public static bool TooLarge(ConvertOptions options, int width, int height)
{
ArgumentNullException.ThrowIfNull(options);
if (width <= 0 || height <= 0) return false;
// At most (2^31 - 1)^2, below 2^62.
var pixels = (long)width * height;
var readBack = options.Target == ConvertTarget.Jpeg
? Math.Max(width, height) <= FrameConverter.MaxJpegSide && pixels <= JpegDecodeOptions.Default.MaxPixels
: pixels <= TiffDecodeOptions.Default.MaxSamples / 3;
return !readBack || WorstBytes(width, height) > Array.MaxLength;
}
/// <summary>
/// The largest a converted file of <paramref name="width"/> × <paramref name="height"/> pixels can be when its pixels do
/// not compress at all, in any target: <see cref="WorstBitsPerPixel"/> with its margin, and the most a file holds beyond
/// its pixels; <see cref="long.MaxValue"/> for sizes beyond it.
/// </summary>
public static long WorstBytes(int width, int height) =>
Saturated(Math.Ceiling((double)Math.Max(0, width) * Math.Max(0, height) * WorstBitsPerPixel / 8 * WorstMargin) + WorstOverhead);
/// <summary><paramref name="bytes"/> (whole, not negative) as a long, <see cref="long.MaxValue"/> when beyond it.</summary>
private static long Saturated(double bytes) => bytes >= long.MaxValue ? long.MaxValue : (long)bytes;
/// <summary>
/// Estimated size of a converted file of <paramref name="width"/> × <paramref name="height"/> pixels (unknown:
/// <see cref="UnknownPixels"/>), from the bits per pixel the Export tab estimates with (<see cref="ExportEstimator"/>):
/// JPEG by quality (×1.25 for 4:4:4 chroma), TIFF-16 30, linear DNG 26.
/// </summary>
public static long EstimateBytes(ConvertOptions options, int width, int height)
{
ArgumentNullException.ThrowIfNull(options);
var pixels = width > 0 && height > 0 ? (double)width * height : UnknownPixels;
var bits = options.Target switch
{
ConvertTarget.Jpeg => ExportEstimator.JpegBitsPerPixel(options.Quality) * FullChroma,
ConvertTarget.Tiff16 => 30,
_ => 26,
};
return Saturated(Math.Ceiling(pixels * bits / 8));
}
/// <summary>The free space a conversion whose files are estimated at <paramref name="estimated"/> bytes needs.</summary>
public static long Required(long estimated) => (long)Math.Ceiling(Math.Max(0, estimated) * SpaceMargin) + SpaceReserve;
}