refactor!: rename Titano to Chronos and move CI and releases to Gitea (#6)
CI / Build and test (push) Successful in 8m14s
CI / Windows tests and installer (push) Skipped
CI / YAML check (push) Successful in 7s
Release / Build, test and publish (push) Successful in 13m19s
CI / PostgreSQL 16 integration (push) Successful in 4m49s
CI / PostgreSQL 17 integration (push) Successful in 5m28s
CI / Docker image (push) Successful in 53s
Release / Server image (push) Failing after 10s

The owner renamed the product and moved the repository to the self-hosted
Gitea (https://gitea.encke-hake.ts.net/BalboSoft/Chronos) for more control,
and asked for Actions and a first automatic release with the server package
and the Windows exe (docs/requests/2026-10-23-chronos-gitea.md).

- Chronos everywhere: a byte-exact rename of contents and paths, then a
  per-language review of the forms the old name's ending had dictated.
- Gitea instead of GitHub: the update check, licence, issue and support
  links read ChronosReleasesUrl; the update check takes the highest stable
  version, offers a pre-release build its final release and reports a
  repository it cannot see as a failed check. EULA, security policy and
  Gitea issue forms live at the repository root.
- Gitea Actions for the Linux act_runner (.gitea/workflows): CI, release,
  nightly image and performance gates in the .NET SDK container with git
  checkout and the system SQLite. The release cross-compiles the Windows
  programs, checks their icon, version and DPI manifest, packs installers
  and zips, publishes the Gitea release and pushes the server image
  (REGISTRY_TOKEN). Windows-only tests run in an opt-in job on a Windows
  runner (WINDOWS_RUNNER).
- New Chronos.Tools commands: zip, gitea-release, postgres-ci (replaces the
  docker-exec script, which Gitea's service containers do not allow; the
  throw-away server runs without durability) and pe-check.
- Timed tests take their budget from TestTiming (CHRONOS_TEST_TIME_FACTOR,
  3 on the shared runner; the weekly performance gates keep theirs).

BREAKING CHANGE: programs, settings, data folders, file types, environment
variables and the Docker image use the new name; nothing is migrated from
Titano, which never had a release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Alberto Balbo <alberto.balbo.96@gmail.com>
This commit was merged in pull request #6.
This commit is contained in:
Alby96andClaude Opus 5.5 authored and Alby96 committed 2026-10-23 12:45:18 +02:00
1 parent c99842cc86
commit f2b58522b9
1801 files changed
+11804 -10676

No files matched your search

+705
View File
@@ -0,0 +1,705 @@
using System.Collections.Concurrent;
using System.Globalization;
using System.Net;
using Chronos.Core.Archiving;
using Chronos.Core.Localization;
using Chronos.Core.Model;
using Chronos.Core.Project;
using Chronos.Data.Archiving;
using Chronos.Data.Archiving.Jobs;
using Chronos.Data.Jobs;
using Chronos.Data.Processing;
using Chronos.Protocol;
using Chronos.Protocol.Client;
using SequenceFrames = Chronos.Data.Processing.SequenceFrames;
using SequenceManifest = Chronos.Protocol.SequenceManifest;
namespace Chronos.Data.Sync;
/// <summary>
/// The work of upload and download jobs. Uploads send every local original the server lacks through the chunked,
/// resumable protocol after hashing it again (an original that no longer matches its archived hash is never sent),
/// creating the server sequence first and sending its settings when the upload made it, then the proxies the server asks
/// for (originals it cannot decode, BRIEF §2.3), made here from this PC's originals. Downloads fetch originals the
/// local copy lacks into the archive's staging folder (resuming with Range), require the manifest's SHA-256 (or, for an
/// original the server has not hashed yet, its size, and hash it here) and commit them through the archive ingest,
/// which names them and writes the sidecar; the first original creates the local
/// sequence, which then takes the server's settings. Frames of the local copy not hashed yet are hashed first (so the
/// server's copies of them are recognised), and originals flagged missing here are restored to their recorded place.
/// Proxy downloads fill the proxy cache. Renders travel on request: this PC's to the server (hash-verified chunked
/// upload) and the server's into the local copy's exports. A resumed job skips what is already on the other side, so
/// pausing or a crash loses at most the files in flight.
/// </summary>
internal sealed class SyncTransfers(SyncService sync)
{
private const long SpaceReserve = 512L << 20;
/// <summary>Space a cached proxy is assumed to take before it is downloaded (1920 px JPEG, BRIEF §8.5).</summary>
private const long ProxyEstimate = 3L << 19;
public async Task UploadAsync(JobContext context, ISyncRemote remote, Guid localId)
{
var ct = context.Cancellation;
var db = sync.Database;
var local = await db.Sequences.GetAsync(localId, ct).ConfigureAwait(false) ?? throw new InvalidOperationException(Loc.L("Errors.Sync.LocalMissing"));
var archive = await db.Archives.GetAsync(local.ArchiveId, ct).ConfigureAwait(false) ?? throw new InvalidOperationException(Loc.L("Errors.Sync.NoArchive"));
var paths = new ArchivePaths(archive.RootPath);
var (remoteId, created) = await TargetAsync(context, remote, local).ConfigureAwait(false);
var frames = (await SequenceFrames.ListAsync(db, local.Id, ct).ConfigureAwait(false)).Where(f => (f.Flags & FrameFlags.Missing) == 0).ToList();
var manifest = remoteId is { } known ? await remote.GetManifestAsync(known, ct).ConfigureAwait(false) : null;
var present = manifest?.Frames.Where(f => f.Sha256.Length > 0).Select(f => f.Sha256.ToLowerInvariant()).ToHashSet(StringComparer.Ordinal) ?? [];
// Originals the server holds without a hash (adopted there, not verified) are the ones of their size and capture
// time: the server could not recognise them by content, so they are not sent again.
var adopted = manifest is not null && manifest.Frames.Any(f => f.Sha256.Length == 0)
? await UnhashedAsync(remote, manifest, ct).ConfigureAwait(false)
: new FramePhotos([]);
var pending = new Queue<Frame>(frames.Where(f => (f.Sha256 is null || !present.Contains(f.Sha256.ToLowerInvariant()))
&& adopted.IndexOf(new FramePhoto(f.Sha256, f.Size, f.CapturedAt)) is null));
var progress = new TransferProgress(context, frames.Count, frames.Count - pending.Count);
await progress.ReportAsync().ConfigureAwait(false);
var skipped = 0;
// The SHA-256 of each original the index has none for (a folder adopted and not verified), as its upload computes
// it: the proxies the server asks for are matched to this PC's originals by content. Every such original is
// pending, so each one still here is hashed.
var hashes = new ConcurrentDictionary<Guid, string>();
// Without a server sequence the originals go out one by one under this sequence's session key until the
// server has filed one, which creates the sequence (an original it already holds elsewhere creates nothing).
while (remoteId is null && pending.TryDequeue(out var first))
{
var sent = await UploadFrameAsync(context, remote, null, SyncService.SessionKey(local.Id), paths.FromRelative(first.RelativePath), first, hashes, ct)
.ConfigureAwait(false);
if (sent is null) skipped++;
else if (sent.SequenceId != Guid.Empty)
{
remoteId = sent.SequenceId;
await context.CheckpointAsync(sent.SequenceId.ToString(), progress.Done + 1, frames.Count).ConfigureAwait(false);
await sync.LinkAsync(remote.Server, sent.SequenceId, local, ct).ConfigureAwait(false);
}
await progress.StepAsync().ConfigureAwait(false);
}
if (remoteId is not { } target) throw new InvalidOperationException(Loc.L("Errors.Sync.NothingSent"));
await Parallel.ForEachAsync(pending, Workers(sync.Options.Uploads, ct), async (frame, token) =>
{
if (await UploadFrameAsync(context, remote, target, null, paths.FromRelative(frame.RelativePath), frame, hashes, token).ConfigureAwait(false) is null)
Interlocked.Increment(ref skipped);
await progress.StepAsync().ConfigureAwait(false);
}).ConfigureAwait(false);
await SendProxiesAsync(context, remote, target, local, paths, frames, hashes).ConfigureAwait(false);
if (created) await sync.PushAllAsync(remote, target, ct).ConfigureAwait(false);
await sync.RecompareAsync(remote, target, ct).ConfigureAwait(false);
if (skipped > 0) throw new InvalidDataException(Loc.Plural("Errors.Sync.Unreadable", skipped));
}
/// <summary>The server's frames it has not hashed yet, by size and capture time, at their indices.</summary>
private static async Task<FramePhotos> UnhashedAsync(ISyncRemote remote, SequenceManifest manifest, CancellationToken cancellationToken)
{
var captured = (await remote.ListFramesAsync(manifest.SequenceId, cancellationToken).ConfigureAwait(false)).ToDictionary(f => f.Id, f => f.CapturedAt);
return new FramePhotos(manifest.Frames.Where(f => f.Sha256.Length == 0).Select(f => (f.Index, new FramePhoto(null, f.Size, captured.GetValueOrDefault(f.Id)))));
}
/// <summary>
/// The proxies the server asks for (<see cref="ManifestFrame.WantsProxy"/>) that this PC's decoders can make from its
/// originals (by their archived SHA-256, or the one <paramref name="hashes"/> holds for an original the index has none
/// for; by size and capture time for a frame the server has not hashed, <see cref="SyncRules.ProxyOriginal"/>), made as
/// the server's proxy policy says with the server's capture data, and sent. One the server refuses is
/// counted in the job log; a server that makes no proxies, or an account that may not send them, ends the pass; a
/// server busy or out of reach fails the job like an original would.
/// </summary>
private async Task SendProxiesAsync(JobContext context, ISyncRemote remote, Guid remoteId, Sequence local, ArchivePaths paths, IReadOnlyList<Frame> frames,
IReadOnlyDictionary<Guid, string> hashes)
{
var ct = context.Cancellation;
var manifest = await remote.GetManifestAsync(remoteId, ct).ConfigureAwait(false);
var wanted = manifest.Frames.Where(f => f.WantsProxy).ToList();
if (manifest.Proxies is not { } spec || wanted.Count == 0) return;
var originals = new Dictionary<string, Frame>(StringComparer.Ordinal);
foreach (var frame in frames)
{
if ((frame.Sha256 ?? hashes.GetValueOrDefault(frame.Id)) is { } sha) originals.TryAdd(sha.ToLowerInvariant(), frame);
}
var records = (await remote.ListFramesAsync(remoteId, ct).ConfigureAwait(false)).ToDictionary(f => f.Id);
var photos = SyncRules.Photos(frames);
var camera = new SidecarCamera { Make = local.CameraMake, Model = local.CameraModel, Serial = local.CameraSerial };
var decoder = sync.Options.Archive.Decoder;
var progress = new TransferProgress(context, wanted.Count, 0);
await progress.ReportAsync().ConfigureAwait(false);
int sent = 0, unsent = 0;
using var refused = CancellationTokenSource.CreateLinkedTokenSource(ct);
try
{
await Parallel.ForEachAsync(wanted, Workers(sync.Options.Uploads, refused.Token), async (frame, token) =>
{
records.TryGetValue(frame.Id, out var record);
// A frame the server has not hashed yet is this PC's original of its size and capture time.
var original = frame.Sha256.Length == 0
? SyncRules.ProxyOriginal(frame, record?.CapturedAt, frames, photos)
: originals.GetValueOrDefault(frame.Sha256.ToLowerInvariant());
var jpeg = original is not null && record is not null
? await Task.Run(() => MakeProxy(decoder, paths.FromRelative(original.RelativePath), RenderNodeClient.ToFrame(record), camera, spec, frame.FileName,
token), token).ConfigureAwait(false)
: null;
try
{
if (jpeg is not null)
{
await remote.UploadProxyAsync(frame.Id, jpeg, token).ConfigureAwait(false);
Interlocked.Increment(ref sent);
}
else
{
Interlocked.Increment(ref unsent);
}
}
catch (Exception ex) when (ex is NotSupportedException or ChronosApiException { Status: HttpStatusCode.Conflict or HttpStatusCode.Forbidden })
{
// The server takes no proxies (or none from this account): nothing more to send.
Core.Diagnostics.Log.Info("sync", $"Proxies of {local.DisplayTitle} not sent: {ex.Message}");
await refused.CancelAsync().ConfigureAwait(false);
}
catch (ChronosApiException ex) when (!ex.IsTransient)
{
// This proxy only (not a JPEG of that frame, a frame gone meanwhile): the others still go.
Core.Diagnostics.Log.Warning("sync", $"The proxy of {frame.FileName} was refused", ex);
Interlocked.Increment(ref unsent);
}
await progress.StepAsync().ConfigureAwait(false);
}).ConfigureAwait(false);
}
catch (OperationCanceledException) when (refused.IsCancellationRequested && !ct.IsCancellationRequested)
{
unsent = wanted.Count - sent;
}
context.Log(Loc.L("Archive.Log.ProxiesSent", sent, unsent));
}
/// <summary>A proxy made here for the server; null (logged) when this PC cannot decode or read the original either.</summary>
private static byte[]? MakeProxy(Core.Imaging.ImageDecoder decoder, string path, Frame frame, SidecarCamera camera, ProxySpec spec, string name,
CancellationToken cancellationToken)
{
try
{
return ArchiveImages.MakeProxy(decoder, path, frame, camera, spec.LongEdge, spec.Quality, cancellationToken);
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or InvalidDataException or NotSupportedException)
{
Core.Diagnostics.Log.Warning("sync", $"No proxy of {name} could be made", ex);
return null;
}
}
public async Task DownloadOriginalsAsync(JobContext context, ISyncRemote remote, Guid remoteId, Guid? targetArchive)
{
var ct = context.Cancellation;
var db = sync.Database;
var link = await sync.Links.GetAsync(remote.Server, remoteId, ct).ConfigureAwait(false);
var local = link?.LocalId is { } id ? await db.Sequences.GetAsync(id, ct).ConfigureAwait(false) : null;
var archive = (local?.ArchiveId ?? targetArchive) is { } archiveId ? await db.Archives.GetAsync(archiveId, ct).ConfigureAwait(false) : null;
if (archive is null) throw new InvalidOperationException(Loc.L("Errors.Sync.NoArchive"));
var paths = new ArchivePaths(archive.RootPath);
ArchiveVolume.EnsureWritable(paths);
var manifest = await remote.GetManifestAsync(remoteId, ct).ConfigureAwait(false);
var captured = (await remote.ListFramesAsync(remoteId, ct).ConfigureAwait(false)).ToDictionary(f => f.Id, f => f.CapturedAt);
var (have, restore, photos) = local is null ? ([], [], new FramePhotos([])) : await LocalContentAsync(archive, local, ct).ConfigureAwait(false);
// An original the server has not hashed yet (adopted there, not verified) is here when an original of its size and
// capture time is; each one is fetched on its own and hashed here.
var wanted = manifest.Frames
.Where(f => f.Sha256.Length == 0
? photos.IndexOf(new FramePhoto(null, f.Size, captured.GetValueOrDefault(f.Id))) is null
: !have.Contains(f.Sha256.ToLowerInvariant()))
.DistinctBy(f => f.Sha256.Length == 0 ? f.Id.ToString("N") : f.Sha256.ToLowerInvariant()).ToList();
EnsureSpace(paths.Root, wanted.Sum(f => f.Size));
var staging = Path.Combine(paths.StagingDirectory, "sync-" + remoteId.ToString("N"));
Directory.CreateDirectory(staging);
var progress = new TransferProgress(context, manifest.Frames.Count, manifest.Frames.Count - wanted.Count);
await progress.ReportAsync().ConfigureAwait(false);
var created = false;
var rest = wanted;
if (local is null && wanted.Count > 0)
{
// The first original creates the local sequence (grouped by the server id); the others join it.
var (staged, sha) = await FetchOriginalAsync(remote, wanted[0], staging, ct).ConfigureAwait(false);
var first = await CommitAsync(remoteId, archive.Id, null, wanted[0], staged, sha, captured, ct).ConfigureAwait(false);
local = await db.Sequences.GetAsync(first.SequenceId, ct).ConfigureAwait(false)
?? throw new InvalidOperationException(Loc.L("Errors.Sync.LocalMissing"));
await sync.LinkAsync(remote.Server, remoteId, local, ct).ConfigureAwait(false);
await sync.Links.SetOriginAsync(local.Id, SequenceOrigin.Server, ct).ConfigureAwait(false);
created = true;
rest = wanted.GetRange(1, wanted.Count - 1);
await progress.StepAsync().ConfigureAwait(false);
}
var sequenceId = local?.Id;
var restored = 0;
await Parallel.ForEachAsync(rest, Workers(sync.Options.Downloads, ct), async (frame, token) =>
{
var (staged, sha) = await FetchOriginalAsync(remote, frame, staging, token).ConfigureAwait(false);
if (restore.TryGetValue(sha, out var place))
{
Restore(staged, place);
Interlocked.Increment(ref restored);
}
else
{
await CommitAsync(remoteId, archive.Id, sequenceId, frame, staged, sha, captured, token).ConfigureAwait(false);
}
await progress.StepAsync().ConfigureAwait(false);
}).ConfigureAwait(false);
TryDeleteDirectory(staging);
// Restored originals are checked against their records, which clears their missing flag.
if (restored > 0) await VerifyAsync(archive, local!.Id, ct).ConfigureAwait(false);
if (created) await sync.TakeAllAsync(remote, remoteId, ct).ConfigureAwait(false);
await sync.RecompareAsync(remote, remoteId, ct).ConfigureAwait(false);
if (sequenceId is { } downloaded && wanted.Count > restored)
{
var label = local!.DisplayTitle;
await EnqueueFollowUpAsync(context, ArchiveJobs.Thumbnails([downloaded], label)).ConfigureAwait(false);
await EnqueueFollowUpAsync(context, ProcessingJobs.LightAnalysis([downloaded], label)).ConfigureAwait(false);
}
}
/// <summary>
/// The originals a local copy holds (lower-case SHA-256, and their photos for server frames not hashed yet) and where
/// each original flagged missing belongs. Frames not hashed yet (an adopted folder never verified) are hashed first
/// through the archive's verify, so the server's copies of them are recognised instead of committed again as duplicates.
/// </summary>
private async Task<(HashSet<string> Have, Dictionary<string, string> Restore, FramePhotos Photos)> LocalContentAsync(Archive archive, Sequence local,
CancellationToken ct)
{
var frames = await SequenceFrames.ListAsync(sync.Database, local.Id, ct).ConfigureAwait(false);
if (frames.Any(f => f.Sha256 is null && (f.Flags & FrameFlags.Missing) == 0))
{
await VerifyAsync(archive, local.Id, ct).ConfigureAwait(false);
frames = await SequenceFrames.ListAsync(sync.Database, local.Id, ct).ConfigureAwait(false);
}
var have = frames.Where(f => (f.Flags & FrameFlags.Missing) == 0 && f.Sha256 is not null).Select(f => f.Sha256!.ToLowerInvariant())
.ToHashSet(StringComparer.Ordinal);
var paths = new ArchivePaths(archive.RootPath);
var restore = new Dictionary<string, string>(StringComparer.Ordinal);
foreach (var frame in frames.Where(f => (f.Flags & FrameFlags.Missing) != 0 && f.Sha256 is not null))
{
var sha = frame.Sha256!.ToLowerInvariant();
if (!have.Contains(sha)) restore.TryAdd(sha, paths.FromRelative(frame.RelativePath));
}
var photos = SyncRules.Photos([.. frames.Where(f => (f.Flags & FrameFlags.Missing) == 0)]);
return (have, restore, photos);
}
private Task VerifyAsync(Archive archive, Guid sequenceId, CancellationToken ct) =>
sync.Archives.Verifier.VerifyAsync(archive, [sequenceId], default, (_, _, _) => ValueTask.CompletedTask, cancellationToken: ct);
/// <summary>Puts a verified copy of an original flagged missing back where its record says; a file that reappeared meanwhile is kept.</summary>
private static void Restore(string staged, string place)
{
if (File.Exists(place))
{
File.Delete(staged);
return;
}
Directory.CreateDirectory(Path.GetDirectoryName(place)!);
File.Move(staged, place, overwrite: false);
}
public async Task SendExportsAsync(JobContext context, ISyncRemote remote, Guid localId)
{
var ct = context.Cancellation;
var db = sync.Database;
var local = await db.Sequences.GetAsync(localId, ct).ConfigureAwait(false) ?? throw new InvalidOperationException(Loc.L("Errors.Sync.LocalMissing"));
var archive = await db.Archives.GetAsync(local.ArchiveId, ct).ConfigureAwait(false) ?? throw new InvalidOperationException(Loc.L("Errors.Sync.NoArchive"));
var link = await sync.Links.FindByLocalAsync(remote.Server, local.Id, ct).ConfigureAwait(false)
?? throw new InvalidOperationException(Loc.L("Errors.Sync.NotLinked"));
var there = (await remote.GetManifestAsync(link.RemoteId, ct).ConfigureAwait(false)).Renders.Select(r => r.Id).ToHashSet();
var renders = (await db.Renders.ListBySequenceAsync(local.Id, ct).ConfigureAwait(false)).Where(r => !there.Contains(r.Id)).ToList();
var progress = new TransferProgress(context, renders.Count, 0);
await progress.ReportAsync().ConfigureAwait(false);
var gone = 0;
foreach (var render in renders)
{
var file = SyncService.RenderFile(archive, render);
if (File.Exists(file))
{
var sha = await ChronosClient.HashFileAsync(file, ct).ConfigureAwait(false);
await remote.UploadRenderAsync(link.RemoteId, file, new RenderUploadRequest(render.Id, Path.GetFileName(file), new FileInfo(file).Length, sha,
render.PresetName, render.Codec, render.Width, render.Height, render.Fps, render.DurationSeconds, render.ProducedBy), ct).ConfigureAwait(false);
}
else
{
gone++;
}
await progress.StepAsync().ConfigureAwait(false);
}
await sync.RecompareAsync(remote, link.RemoteId, ct).ConfigureAwait(false);
if (gone > 0) throw new FileNotFoundException(Loc.Plural("Errors.Sync.ExportsGone", gone));
}
public async Task GetExportsAsync(JobContext context, ISyncRemote remote, Guid remoteId)
{
var ct = context.Cancellation;
var db = sync.Database;
var link = await sync.Links.GetAsync(remote.Server, remoteId, ct).ConfigureAwait(false);
var local = (link?.LocalId is { } id ? await db.Sequences.GetAsync(id, ct).ConfigureAwait(false) : null)
?? throw new InvalidOperationException(Loc.L("Errors.Sync.LocalMissing"));
var archive = await db.Archives.GetAsync(local.ArchiveId, ct).ConfigureAwait(false) ?? throw new InvalidOperationException(Loc.L("Errors.Sync.NoArchive"));
var paths = new ArchivePaths(archive.RootPath);
ArchiveVolume.EnsureWritable(paths);
var here = (await db.Renders.ListBySequenceAsync(local.Id, ct).ConfigureAwait(false)).Select(r => r.Id).ToHashSet();
var wanted = (await remote.GetManifestAsync(remoteId, ct).ConfigureAwait(false)).Renders.Where(r => !here.Contains(r.Id)).ToList();
EnsureSpace(paths.Root, wanted.Sum(r => r.Bytes));
var staging = Path.Combine(paths.StagingDirectory, "exports-" + remoteId.ToString("N"));
var exports = paths.Sequence(local.Folder).Exports;
Directory.CreateDirectory(staging);
Directory.CreateDirectory(exports);
var progress = new TransferProgress(context, wanted.Count, 0);
await progress.ReportAsync().ConfigureAwait(false);
foreach (var render in wanted)
{
var (part, name) = await FetchRenderAsync(remote, render, staging, ct).ConfigureAwait(false);
var target = Unique(exports, name);
File.Move(part, target);
await db.Renders.InsertAsync(new Render
{
Id = render.Id,
SequenceId = local.Id,
PresetName = render.PresetName,
OutputPath = paths.ToRelative(target),
Codec = render.Codec,
Width = render.Width,
Height = render.Height,
Fps = render.Fps,
DurationSeconds = render.DurationSeconds,
Bytes = new FileInfo(target).Length,
Created = render.Created,
ProducedBy = render.ProducedBy,
}, ct).ConfigureAwait(false);
await progress.StepAsync().ConfigureAwait(false);
}
TryDeleteDirectory(staging);
await sync.RecompareAsync(remote, remoteId, ct).ConfigureAwait(false);
}
/// <summary>A render's file in staging (resumed with Range; started over once if the partial copy no longer fits) and the name the server gives it.</summary>
private static async Task<(string Part, string Name)> FetchRenderAsync(ISyncRemote remote, RenderDto render, string staging, CancellationToken cancellationToken)
{
var part = Path.Combine(staging, render.Id.ToString("N") + ".part");
for (var attempt = 1; ; attempt++)
{
try
{
string? name;
await using (var file = new FileStream(part, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None))
name = await remote.DownloadRenderAsync(render.Id, file, cancellationToken).ConfigureAwait(false);
return (part, SafeFileName(name) ?? render.Id.ToString("N"));
}
catch (ChronosApiException ex) when (ex.Status == HttpStatusCode.RequestedRangeNotSatisfiable && attempt == 1)
{
File.Delete(part);
}
}
}
private static string? SafeFileName(string? name)
{
var file = Path.GetFileName(name?.Trim());
return string.IsNullOrEmpty(file) || file.IndexOfAny(Path.GetInvalidFileNameChars()) >= 0 || file is "." or ".." ? null : file;
}
/// <summary>"name.mp4", else "name (2).mp4", … — an existing export is never replaced.</summary>
private static string Unique(string folder, string fileName)
{
var candidate = Path.Combine(folder, fileName);
var stem = Path.GetFileNameWithoutExtension(fileName);
var extension = Path.GetExtension(fileName);
for (var n = 2; File.Exists(candidate); n++) candidate = Path.Combine(folder, string.Create(CultureInfo.InvariantCulture, $"{stem} ({n}){extension}"));
return candidate;
}
public async Task DownloadProxiesAsync(JobContext context, ISyncRemote remote, Guid remoteId)
{
var ct = context.Cancellation;
var dto = await remote.GetSequenceAsync(remoteId, ct).ConfigureAwait(false);
var manifest = await remote.GetManifestAsync(remoteId, ct).ConfigureAwait(false);
var frames = await remote.ListFramesAsync(remoteId, ct).ConfigureAwait(false);
var assets = manifest.Frames.ToDictionary(f => f.Id);
var folder = sync.Proxies.Folder(remote.Server, remoteId);
Directory.CreateDirectory(folder);
var cachedKinds = sync.Proxies.Kinds(remote.Server, remoteId);
ProxyCache.Prune(folder, frames.Select(f => ProxyCache.FileName(f.Id)).ToHashSet(StringComparer.Ordinal));
EnsureSpace(folder, frames.Count(f => !File.Exists(Path.Combine(folder, ProxyCache.FileName(f.Id)))) * ProxyEstimate);
var files = new string?[frames.Count];
var kinds = new string?[frames.Count];
var progress = new TransferProgress(context, frames.Count, 0);
await progress.ReportAsync().ConfigureAwait(false);
await Parallel.ForEachAsync(Enumerable.Range(0, frames.Count), Workers(sync.Options.Downloads, ct), async (i, token) =>
{
IReadOnlyList<string> offered = assets.TryGetValue(frames[i].Id, out var asset) ? Kinds(asset) : [];
var fetched = await FetchProxyAsync(remote, frames[i].Id, offered, cachedKinds.GetValueOrDefault(frames[i].Id), folder, token).ConfigureAwait(false);
files[i] = fetched?.Name;
kinds[i] = fetched?.Kind;
await progress.StepAsync().ConfigureAwait(false);
}).ConfigureAwait(false);
sync.Proxies.Save(remote.Server, remoteId, new ProxyCacheIndex(dto, manifest.SettingsJson, manifest.FramesVersion, frames, files, kinds));
var cached = files.Count(f => f is not null);
var standIns = Enumerable.Range(0, frames.Count).Count(i => kinds[i] != ProxyCache.Proxy && !NeverProxied(frames[i]));
await sync.SetProxiesAsync(remote.Server, dto, cached, standIns, manifest.FramesVersion, ct).ConfigureAwait(false);
if (cached < frames.Count) context.Log($"{frames.Count - cached} frame(s) have neither proxy nor thumbnail on the server");
}
/// <summary>
/// The server sequence an upload fills and whether this job creates it: the linked one, the one an interrupted
/// attempt created, or none yet (the first original creates it).
/// </summary>
private async Task<(Guid? RemoteId, bool Created)> TargetAsync(JobContext context, ISyncRemote remote, Sequence local)
{
var ct = context.Cancellation;
Guid? started = Guid.TryParse(context.Checkpoint, out var saved) ? saved : null;
var link = await sync.Links.FindByLocalAsync(remote.Server, local.Id, ct).ConfigureAwait(false);
foreach (var candidate in new[] { link?.RemoteId, started })
{
if (candidate is not { } id) continue;
if (await ExistsAsync(remote, id, ct).ConfigureAwait(false))
{
if (link?.RemoteId != id) await sync.LinkAsync(remote.Server, id, local, ct).ConfigureAwait(false);
return (id, id == started);
}
// Deleted on the server meanwhile: the originals go into a new sequence.
if (id == link?.RemoteId) await sync.ForgetGoneAsync(remote.Server, id, ct).ConfigureAwait(false);
}
return (null, true);
}
private static async Task<bool> ExistsAsync(ISyncRemote remote, Guid id, CancellationToken cancellationToken)
{
try
{
await remote.GetSequenceAsync(id, cancellationToken).ConfigureAwait(false);
return true;
}
catch (ChronosApiException ex) when (ex.Status == HttpStatusCode.NotFound)
{
return false;
}
}
/// <summary>
/// Hashes the original again and sends it into <paramref name="sequenceId"/> (or the sequence of
/// <paramref name="sessionKey"/>); null (logged) when it is gone or no longer the archived file. The hash of an
/// original the index has none for goes into <paramref name="hashes"/>.
/// </summary>
private static async Task<CompleteUploadResponse?> UploadFrameAsync(JobContext context, ISyncRemote remote, Guid? sequenceId, string? sessionKey, string path,
Frame frame, ConcurrentDictionary<Guid, string> hashes, CancellationToken cancellationToken)
{
if (!File.Exists(path))
{
context.Log($"{frame.FileName}: missing, not uploaded");
return null;
}
var sha = await ChronosClient.HashFileAsync(path, cancellationToken).ConfigureAwait(false);
if (frame.Sha256 is null) hashes[frame.Id] = sha;
if (frame.Sha256 is { } archived && !archived.Equals(sha, StringComparison.OrdinalIgnoreCase))
{
context.Log($"{frame.FileName}: changed since it was archived, not uploaded");
return null;
}
return await remote.UploadFileAsync(path, sequenceId, sessionKey, sha, frame.CapturedAt, cancellationToken).ConfigureAwait(false)
?? new CompleteUploadResponse(Guid.Empty, sequenceId ?? Guid.Empty, -1);
}
/// <summary>Commits one original from staging into the local sequence (created from the server id when null).</summary>
private async Task<IngestResult> CommitAsync(Guid remoteId, Guid archiveId, Guid? sequenceId, ManifestFrame frame, string staged, string sha,
IReadOnlyDictionary<Guid, DateTime?> captured, CancellationToken cancellationToken)
{
var request = new IngestRequest(archiveId, sequenceId, null, sequenceId is null ? SyncService.SessionKey(remoteId) : null, staged, frame.FileName,
frame.Size, sha, captured.GetValueOrDefault(frame.Id), null);
var result = await sync.Archives.Ingest.CommitAsync(request, cancellationToken).ConfigureAwait(false);
// A duplicate stays in staging (the sequence already holds that content).
if (result.Duplicate) File.Delete(staged);
return result;
}
/// <summary>
/// A verified copy in staging and its lower-case SHA-256: resumed from a partial download, accepted only with the
/// manifest's SHA-256 (two attempts). A part as long as the file (a job stopped while hashing it) is only hashed again;
/// a part the server can no longer resume (416) starts over.
/// </summary>
private static async Task<(string Path, string Sha256)> FetchOriginalAsync(ISyncRemote remote, ManifestFrame frame, string staging,
CancellationToken cancellationToken)
{
if (frame.Sha256.Length == 0) return await FetchUnhashedAsync(remote, frame, staging, cancellationToken).ConfigureAwait(false);
var sha = frame.Sha256.ToLowerInvariant();
if (sha.Length != 64 || !sha.All(char.IsAsciiHexDigitLower)) throw new InvalidDataException(Loc.L("Errors.Sync.Hash", frame.FileName));
var target = Path.Combine(staging, sha + Path.GetExtension(frame.FileName).ToLowerInvariant());
if (File.Exists(target)) return (target, sha);
var part = target + ".part";
for (var attempt = 1; ; attempt++)
{
if (!await FetchPartAsync(remote, frame, part, attempt, cancellationToken).ConfigureAwait(false)) continue;
if (await ChronosClient.HashFileAsync(part, cancellationToken).ConfigureAwait(false) == sha)
{
File.Move(part, target, overwrite: true);
return (target, sha);
}
File.Delete(part);
if (attempt == 2) throw new InvalidDataException(Loc.L("Errors.Sync.Hash", frame.FileName));
}
}
/// <summary>
/// An original the server has not hashed yet (a folder it adopted and did not verify): nothing to check it against
/// but its size, so a copy of another size is fetched again once; it is hashed here, and the archive ingest files it
/// by that hash like any other. Kept in staging under the frame's id, as several such originals may share content.
/// </summary>
private static async Task<(string Path, string Sha256)> FetchUnhashedAsync(ISyncRemote remote, ManifestFrame frame, string staging,
CancellationToken cancellationToken)
{
var target = Path.Combine(staging, frame.Id.ToString("N") + Path.GetExtension(frame.FileName).ToLowerInvariant());
var part = target + ".part";
for (var attempt = 1; !File.Exists(target); attempt++)
{
if (!await FetchPartAsync(remote, frame, part, attempt, cancellationToken).ConfigureAwait(false)) continue;
if (new FileInfo(part).Length == frame.Size)
{
File.Move(part, target, overwrite: true);
break;
}
File.Delete(part);
if (attempt == 2) throw new InvalidDataException(Loc.L("Errors.Sync.Hash", frame.FileName));
}
return (target, await ChronosClient.HashFileAsync(target, cancellationToken).ConfigureAwait(false));
}
/// <summary>
/// Downloads an original into <paramref name="part"/>, resuming after what it holds; a part as long as the file is
/// kept as it is. False when the server can no longer resume it (416) on the first attempt: the part is gone and the
/// next attempt starts over.
/// </summary>
private static async Task<bool> FetchPartAsync(ISyncRemote remote, ManifestFrame frame, string part, int attempt, CancellationToken cancellationToken)
{
if (File.Exists(part) && new FileInfo(part).Length >= frame.Size) return true;
try
{
await using (var file = new FileStream(part, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None))
await remote.DownloadFrameAsync(frame.Id, "original", file, cancellationToken).ConfigureAwait(false);
return true;
}
catch (ChronosApiException ex) when (ex.Status == HttpStatusCode.RequestedRangeNotSatisfiable && attempt == 1)
{
File.Delete(part);
return false;
}
}
/// <summary>
/// The cached file of a frame and what it is, downloading the first asset kind the server has
/// (<paramref name="offered"/>, the proxy first); null when it has none. A cached proxy stays. A cached thumbnail,
/// or a file of a cache that did not record its kind (<paramref name="cachedKind"/> null), stands in only until the
/// server has a proxy of the frame, which then replaces it; until then it stays as the thumbnail.
/// </summary>
private static async Task<(string Name, string Kind)?> FetchProxyAsync(ISyncRemote remote, Guid frameId, IReadOnlyList<string> offered, string? cachedKind,
string folder, CancellationToken cancellationToken)
{
var name = ProxyCache.FileName(frameId);
var path = Path.Combine(folder, name);
var cached = new FileInfo(path) is { Exists: true, Length: > 0 };
if (cached && (cachedKind == ProxyCache.Proxy || !offered.Contains(ProxyCache.Proxy))) return (name, cachedKind ?? ProxyCache.Thumb);
var part = path + ".part";
foreach (var kind in offered)
{
// The cached file is as good as the server's thumbnail.
if (cached && kind == ProxyCache.Thumb) break;
try
{
await using (var file = new FileStream(part, FileMode.Create, FileAccess.ReadWrite, FileShare.None))
await remote.DownloadFrameAsync(frameId, kind, file, cancellationToken).ConfigureAwait(false);
File.Move(part, path, overwrite: true);
return (name, kind);
}
catch (ChronosApiException ex) when (ex.Status == HttpStatusCode.NotFound)
{
File.Delete(part);
}
}
return cached ? (name, ProxyCache.Thumb) : null;
}
/// <summary>
/// A frame missing or damaged on the server: its manifest never asks for its proxy and a pass makes none of a missing
/// one, so the thumbnail (or the lack of a file) that stands in for it does not keep the cache stale.
/// </summary>
private static bool NeverProxied(FrameDto frame) => ((FrameFlags)frame.Flags & (FrameFlags.Missing | FrameFlags.Corrupt)) != 0;
private static string[] Kinds(ManifestFrame frame) => (frame.HasProxy, frame.HasThumb) switch
{
(true, true) => [ProxyCache.Proxy, ProxyCache.Thumb],
(true, false) => [ProxyCache.Proxy],
(false, true) => [ProxyCache.Thumb],
_ => [],
};
/// <summary>Queues a follow-up on this job's engine for the same user; skipped when the engine has no handler for it.</summary>
private static Task<Job?> EnqueueFollowUpAsync(JobContext context, ArchiveJobRequest request) =>
context.EnqueueFollowUpAsync(request.Kind, request.Targets, request.Label, request.ParametersJson, request.Priority);
private static void EnsureSpace(string root, long bytes)
{
long? free;
try
{
var full = Path.GetFullPath(root);
// Windows wants a drive root; Unix accepts any directory and reports the volume it is mounted on.
free = new DriveInfo(OperatingSystem.IsWindows() ? Path.GetPathRoot(full)! : full).AvailableFreeSpace;
}
catch (Exception ex) when (ex is ArgumentException or IOException or UnauthorizedAccessException)
{
free = null; // network shares: the copy itself reports a full disk
}
if (free is { } available && available < bytes + SpaceReserve)
throw new IOException(Loc.L("Errors.Sync.DiskFull", ((bytes + SpaceReserve - available) / 1e9).ToString("0.0", CultureInfo.CurrentCulture)));
}
private static void TryDeleteDirectory(string folder)
{
try
{
Directory.Delete(folder, recursive: true);
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
Core.Diagnostics.Log.Warning("sync", $"Staging folder {folder} could not be removed", ex);
}
}
private static ParallelOptions Workers(int degree, CancellationToken cancellationToken) =>
new() { MaxDegreeOfParallelism = Math.Max(1, degree), CancellationToken = cancellationToken };
/// <summary>Files done out of the total, reported in order from parallel workers.</summary>
private sealed class TransferProgress(JobContext context, long total, long done)
{
private readonly SemaphoreSlim _gate = new(1, 1);
public async Task StepAsync()
{
await _gate.WaitAsync(CancellationToken.None).ConfigureAwait(false);
try
{
done++;
await context.ReportAsync(done, total).ConfigureAwait(false);
}
finally
{
_gate.Release();
}
}
public long Done => done;
public ValueTask ReportAsync() => context.ReportAsync(done, total);
}
}