fix: logging never fails a job, one backup retention, first copy after the upgrade

Owner review 2, phase 2 review findings (scheduler core).

- Log.Write caught only IOException. Microsoft.Extensions.Logging wraps
  provider failures in an AggregateException, so a log line that could not
  be written (in tests: the process-wide sink of a stopped server whose log
  folder was gone) escaped JobEngine.RunAsync's catch before the failure was
  stored: the job stayed running and the worker faulted. This was the cause
  of the intermittent JobApiTests/ArchiveModuleTests failures. Log.Write now
  drops any non-fatal sink failure, TeeLogSink keeps a line from the other
  sinks, FileLogSink drops a line it cannot write and reopens the file at the
  next one, and LoggerSink drops what the providers throw. The scheduler's
  loop relies on the same guarantee.
- The daily backup loop (RunDailyAsync, IsDue, Interval, CheckInterval) had
  no caller since the index-backup schedule replaced it; it is removed and
  the DatabaseBackups summary describes the schedule.
- The first copy after the upgrade no longer waits a day (or for 03:00):
  SystemSchedule.LastDone seeds a new row due at once when its work was last
  done a period or more before, and both hosts pass the newest copy in the
  backups folder for index-backup.
- Back up now (Settings and Archive), the copy kept before a restore and a
  backup job that keeps fewer prune to the largest Copies kept of the backup
  schedules (DatabaseBackups.KeptAsync), not a fixed three. The four
  tooltips that promised three daily copies are reworded (en, it) and their
  stale translations deleted.
- New tests for the scheduler loop itself: a tick, Wake, a pass that threw,
  and dispose during a pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit c5ad0c4c3f636129afe5dd1ce039185516da5621)
This commit is contained in:
Alby96 committed 2026-10-27 09:56:47 +01:00
1 parent 422f33d5f3
commit e384564784
59 files changed
+668 -264

No files matched your search

@@ -64,6 +64,9 @@ public sealed class ArchiveService : IJobHandlerProvider
/// </summary>
public IReadOnlyList<Housekeeping.ICleanupStep> CleanupSteps { get; }
/// <summary>Where this engine's <see cref="JobKind.Backup"/> jobs put their copies (<see cref="ArchiveServiceOptions.BackupDirectory"/>; null: next to the index).</summary>
public string? BackupDirectory => _writer.Options.BackupDirectory;
/// <summary>
/// Handlers for scan, rebuild, verify, import, thumbs, proxies and maintenance jobs, the read-only duplicates and orphans
/// reports, index backups and clean-ups (owner review 2, R10); register them on the <see cref="JobEngine"/>.
+46 -37
View File
@@ -13,22 +13,21 @@ public sealed record DatabaseBackup(string Path, DateTimeOffset Created, long By
public sealed record DatabaseSummary(int SchemaVersion, int Archives, int Sequences, long Photos);
/// <summary>
/// Backups of the SQLite index (BRIEF §6.2): a <c>VACUUM INTO</c> copy made once a day while the host runs (at start
/// when the newest copy is a day old, then checked every hour), the three newest kept; and their restore from
/// Settings › Advanced. A restore never swaps the file under a running host: the chosen copy is staged next to the
/// index and replaces it at the next start, before the host opens the index (so it also recovers an index that no
/// longer opens), once the index it replaces has been kept.
/// Backups of the SQLite index (BRIEF §6.2): <c>VACUUM INTO</c> copies in the backups folder, made by the
/// <see cref="Core.Model.JobKind.Backup"/> jobs of the hosts' <c>index-backup</c> system schedule (owner review 2, R10: every
/// day on the desktop, at 03:00 on the server; its recurrence and Copies kept can be edited, three by default), by Back up
/// now, and of the index a restore replaces. Every copy keeps as many as the backup schedules keep
/// (<see cref="KeptAsync"/>). A restore, from Settings › Advanced, never swaps the file under a running host: the chosen
/// copy is staged next to the index and replaces it at the next start, before the host opens the index (so it also
/// recovers an index that no longer opens), once the index it replaces has been kept.
/// </summary>
public static partial class DatabaseBackups
{
/// <summary>Copies kept (BRIEF §6.2: ×3).</summary>
/// <summary>Copies kept by default (BRIEF §6.2: ×3).</summary>
public const int Keep = 3;
/// <summary>Age of the newest copy from which the next one is due.</summary>
public static readonly TimeSpan Interval = TimeSpan.FromDays(1);
/// <summary>How often a running host checks whether a copy is due.</summary>
public static readonly TimeSpan CheckInterval = TimeSpan.FromHours(1);
/// <summary>The hosts' system schedule that makes the copies.</summary>
public const string ScheduleKey = "index-backup";
/// <summary>Name of the folder next to the index that holds its copies (the desktop's and the server's).</summary>
public const string DirectoryName = "backups";
@@ -80,39 +79,47 @@ public static partial class DatabaseBackups
return backups;
}
/// <summary>True when there is no copy yet or the newest one is at least <see cref="Interval"/> old.</summary>
public static bool IsDue(IReadOnlyList<DatabaseBackup> backups, DateTimeOffset now) => backups.Count == 0 || now - backups[0].Created >= Interval;
/// <summary>
/// Makes the daily copy of a SQLite index into <paramref name="directory"/> until <paramref name="cancellationToken"/>
/// is cancelled: now when one is due, then whenever an hourly check finds one due. A failed copy is logged and tried
/// again at the next check. Returns at once for PostgreSQL, which its own server tools back up.
/// When the newest copy of <paramref name="db"/>'s index file in <paramref name="directory"/> (null: <see cref="DefaultDirectory"/>)
/// was made; null when there is none, the index has no file, or the folder cannot be read. The hosts seed their index
/// backup schedule with it (<see cref="Scheduling.SystemSchedule.LastDone"/>), so the first copy after an upgrade comes
/// as the former daily loop would have made it.
/// </summary>
public static async Task RunDailyAsync(ChronosDatabase db, string directory, TimeProvider? time = null, CancellationToken cancellationToken = default)
public static DateTimeOffset? Newest(ChronosDatabase db, string? directory)
{
if (db.Database is not SqliteDatabase { Path: { } path }) return;
var clock = time ?? TimeProvider.System;
using var timer = new PeriodicTimer(CheckInterval, clock);
ArgumentNullException.ThrowIfNull(db);
if (db.Database is not SqliteDatabase { Path: { } path }) return null;
try
{
do
{
try
{
if (IsDue(List(directory, path), clock.GetUtcNow())) await db.BackupAsync(directory, Keep, cancellationToken).ConfigureAwait(false);
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or DatabaseException)
{
Log.Warning("backup", $"The daily index backup into {directory} failed; it is tried again within the hour", ex);
}
}
while (await timer.WaitForNextTickAsync(cancellationToken).ConfigureAwait(false));
return List(directory ?? DefaultDirectory(path), path) is [var newest, ..] ? newest.Created : null;
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
return null;
}
}
/// <summary>
/// How many copies the backups folder keeps: the largest Copies kept of the backup schedules, the <see cref="ScheduleKey"/>
/// system schedule's among them, enabled or not (<see cref="Keep"/> with none). Every copy prunes to it, so Back up now, the
/// copy kept before a restore and a schedule that keeps fewer never delete copies another schedule keeps.
/// </summary>
public static async Task<int> KeptAsync(ChronosDatabase db, CancellationToken cancellationToken = default) =>
await ScheduledKeepAsync(db, cancellationToken).ConfigureAwait(false) is > 0 and var kept ? kept : Keep;
/// <summary>The largest Copies kept of the backup schedules (1 to <see cref="Housekeeping.BackupJobHandler.MaxKeep"/>); 0 without any.</summary>
internal static async Task<int> ScheduledKeepAsync(ChronosDatabase db, CancellationToken cancellationToken)
{
ArgumentNullException.ThrowIfNull(db);
var kept = 0;
foreach (var schedule in await db.Schedules.ListAsync(cancellationToken).ConfigureAwait(false))
{
if (schedule.Action == Scheduling.ScheduleActions.Backup && Scheduling.ScheduleMapping.Parameters(schedule).Keep is { } keep)
kept = int.Max(kept, int.Clamp(keep, 1, Housekeeping.BackupJobHandler.MaxKeep));
}
return kept;
}
/// <summary>Counts what the index holds (the current one, for the restore's comparison).</summary>
public static async Task<DatabaseSummary> SummarizeAsync(ChronosDatabase db, CancellationToken cancellationToken = default)
{
@@ -196,7 +203,7 @@ public static partial class DatabaseBackups
/// Carries out a staged restore before the host opens the index at <paramref name="databasePath"/>, so it also
/// recovers an index that no longer opens. The staged copy is checked first (<see cref="SummarizeBackupAsync"/>:
/// <c>quick_check</c>, open, migration). The index as it is now is kept in <paramref name="backupDirectory"/>: as a
/// fresh <c>VACUUM INTO</c> copy among the daily ones, or, when it cannot be read, copied there whole (with its WAL) as
/// fresh <c>VACUUM INTO</c> copy among the others, or, when it cannot be read, copied there whole (with its WAL) as
/// <c>&lt;name&gt;-replaced-&lt;stamp&gt;.db</c>, outside the rotation, so it never pushes a good copy out. Only then is
/// the staged copy renamed over the index: the rename consumes it, so it is never applied twice. The host's open then
/// migrates it to this build's schema. False when none was staged.
@@ -231,14 +238,16 @@ public static partial class DatabaseBackups
/// <summary>
/// The index about to be replaced, kept in <paramref name="backupDirectory"/>: a <c>VACUUM INTO</c> copy when it opens,
/// else the file itself with its WAL, set aside under a name the pruning of the daily copies never matches.
/// pruning to what its backup schedules keep (<see cref="KeptAsync"/>), else the file itself with its WAL, set aside under
/// a name the pruning of the copies never matches.
/// </summary>
private static async Task<string> KeepAsync(string path, string backupDirectory, CancellationToken cancellationToken)
{
try
{
await using var current = await ChronosDatabase.OpenAsync(DatabaseOptions.Sqlite(path), cancellationToken).ConfigureAwait(false);
return await current.BackupAsync(backupDirectory, Keep, cancellationToken).ConfigureAwait(false);
var keep = await KeptAsync(current, cancellationToken).ConfigureAwait(false);
return await current.BackupAsync(backupDirectory, keep, cancellationToken).ConfigureAwait(false);
}
catch (Exception ex) when (ex is DatabaseException or SchemaTooNewException)
{
@@ -10,8 +10,9 @@ namespace Chronos.Data.Housekeeping;
/// <summary>
/// <see cref="JobKind.Backup"/> (owner review 2, R10): a <c>VACUUM INTO</c> copy of the SQLite index into
/// <paramref name="directory"/> (by default the <c>backups</c> folder next to the index, <see cref="DatabaseBackups.DefaultDirectory"/>),
/// keeping the newest <see cref="BackupParameters.Keep"/>. A PostgreSQL index, or one without a file, cannot be copied this
/// way (<see cref="DatabaseBackups.Unavailable"/>): the job fails with that reason, and schedules do not queue it.
/// keeping the newest <see cref="BackupParameters.Keep"/>, or more when a backup schedule keeps more (one folder, one
/// retention: <see cref="DatabaseBackups.KeptAsync"/>). A PostgreSQL index, or one without a file, cannot be
/// copied this way (<see cref="DatabaseBackups.Unavailable"/>): the job fails with that reason, and schedules do not queue it.
/// </summary>
internal sealed class BackupJobHandler(ChronosDatabase db, string? directory) : IJobHandler
{
@@ -26,7 +27,8 @@ internal sealed class BackupJobHandler(ChronosDatabase db, string? directory) :
throw new NotSupportedException(DatabaseBackups.Unavailable(db) == Scheduling.ScheduleUnavailable.Postgres
? "Backups of a PostgreSQL database are made with the server's own tools."
: "An index without a file cannot be backed up.");
var keep = Math.Clamp(ArchiveJobJson.Read(context.Job, ArchiveJobJson.Default.BackupParameters, new BackupParameters(DatabaseBackups.Keep)).Keep, 1, MaxKeep);
var own = Math.Clamp(ArchiveJobJson.Read(context.Job, ArchiveJobJson.Default.BackupParameters, new BackupParameters(DatabaseBackups.Keep)).Keep, 1, MaxKeep);
var keep = int.Max(own, await DatabaseBackups.ScheduledKeepAsync(db, context.Cancellation).ConfigureAwait(false));
await context.ReportAsync(0, 1).ConfigureAwait(false);
var copy = await db.BackupAsync(directory ?? DatabaseBackups.DefaultDirectory(path), keep, context.Cancellation).ConfigureAwait(false);
context.Log(Loc.L("Archive.Log.BackupDone", copy, CleanupSteps.Megabytes(new FileInfo(copy).Length), keep));
@@ -65,13 +65,17 @@ public sealed class ScheduleRepository
/// <summary>
/// Inserts each system schedule whose key no row has yet (<c>INSERT … ON CONFLICT (system_key) DO NOTHING</c>), with
/// no next run (the scheduler computes it); rows that exist keep whatever the user changed. Rows added together are
/// stamped a microsecond apart, so listings keep the host's order. Returns how many were added.
/// no next run (the scheduler computes it), or due at <paramref name="now"/> when it is enabled and its work was last
/// done a period or more before (<see cref="SystemSchedule.LastDone"/>); rows that exist keep whatever the user changed.
/// Rows added together are stamped a microsecond apart, so listings keep the host's order. Returns how many were added.
/// </summary>
public Task<int> EnsureSystemAsync(IEnumerable<SystemSchedule> schedules, DateTimeOffset now, CancellationToken cancellationToken = default)
{
ArgumentNullException.ThrowIfNull(schedules);
var rows = schedules.Select((s, i) => FromSystem(s, now.AddTicks(i * TimeSpan.TicksPerMicrosecond))).ToList();
var rows = schedules.Select((s, i) => FromSystem(s, now.AddTicks(i * TimeSpan.TicksPerMicrosecond)) with
{
NextRun = s.Enabled && s.LastDone is { } done && now - done >= s.Recurrence.Period ? now : null,
}).ToList();
return rows.Count == 0 ? Task.FromResult(0) : _db.InTransactionAsync(async session =>
{
var added = 0;
+6 -2
View File
@@ -45,7 +45,10 @@ public interface IScheduleHost
/// <summary>
/// A host's system schedule: <paramref name="SystemKey"/> identifies it across starts (<c>archive-scan</c>,
/// <c>index-backup</c>, <c>housekeeping</c>…); the rest is what it is seeded with and reset to. <paramref name="Name"/> is
/// stored as given; clients show system schedules by their key in the user's language.
/// stored as given; clients show system schedules by their key in the user's language. <paramref name="LastDone"/> is when
/// the host last did this work before the schedule existed (the newest index copy, made by the timer the schedule
/// replaced): seeded a whole period or more later, the row is due at once, so the upgrade's first start does the work
/// as that timer would have; otherwise, and without it, the first run is the recurrence's first occurrence.
/// </summary>
public sealed record SystemSchedule(
string SystemKey,
@@ -55,4 +58,5 @@ public sealed record SystemSchedule(
ScheduleParametersDto? Parameters = null,
bool Enabled = true,
bool CatchUp = true,
JobPriority Priority = JobPriority.Background);
JobPriority Priority = JobPriority.Background,
DateTimeOffset? LastDone = null);