fix(projects): sidecars stay the truth when they cannot be read, are missing or newer

A project record that could not be read once, or a missing .chronos/projects, marked
projects deleted for good and made a second Main; an index meeting Main with no mirror
state took its own stale settings column as an outside change and reverted the pushed
head; a newer-format or unreadable head made every project call fail or rewrote the
record's head over the real history. Now:

- Deletion is never inferred from absence or a read failure: an unreadable record holds
  its project back (listed as indexed, nothing written for it, no second Main), a row
  marked deleted is revived when its live record says so, and projects the archive has
  no folder for are written back from the index.
- Appends re-read the record under the lease and refuse when its head is not the
  index's; Main is only made when the sequence has none anywhere.
- An index with no mirror state decides from project.json and the record's new
  mirrored member (older records: the head's ancestry), never from its own column.
- A newer-format Main head is listed and pulled but never mirrored, merged or built on;
  pushes must parse strictly and fit the one 64 MB version limit (TooLarge otherwise).
- Duplicate version numbers and name/Main clashes are contained per project, so a scan
  or rebuild never stops.
- Delete moves the folder first, then writes the deleted record in the trash.
- The home's read-back for workspace pruning and Remove local copy reads the version
  files (IProjectHome.HoldsAsync), not the index.
- A read-only archive is read without writing: its projects open from the sidecars and
  a sequence with no Main gets one in the index only, written back once writable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Alby96andClaude Opus 5.5 committed 2026-11-03 10:13:40 +01:00
1 parent 50e28fd63d
commit d940fc7c61
6 files changed
+1247 -158

No files matched your search

+236 -82
View File
@@ -5,50 +5,97 @@ using Chronos.Core.Model;
namespace Chronos.Data.Projects;
/// <summary>What a reconcile of a sequence's projects met (<see cref="ProjectIndexer.ReconcileAsync"/>).</summary>
/// <param name="Missing">The sequence has no projects folder: nothing was read, and nothing taken for gone.</param>
/// <param name="Seen">The projects the archive has a folder for, live or in the projects' trash, whether or not its record read.</param>
/// <param name="HeldBack">The live projects the index could not be brought in line with: a record that cannot be read (damaged,
/// of a newer Chronos, a read failure), a head whose version this Chronos cannot read, or a row the index refused. Each stays
/// as the index has it, and nothing is written for it until it reads again.</param>
/// <param name="Live">The records of the live projects read, by id (their head as moved forward).</param>
internal sealed record ProjectReconcile(bool Missing, IReadOnlySet<Guid> Seen, IReadOnlySet<Guid> HeldBack, IReadOnlyDictionary<Guid, ProjectRecord> Live)
{
/// <summary>Every live project of the folder was read and indexed: a sequence without a Main may be given one.</summary>
public bool Complete => HeldBack.Count == 0;
}
/// <summary>
/// Brings the index's projects of one sequence in line with their sidecars (owner review 2, R8), which are the truth: a
/// project or version the index lacks is added (an index rebuilt, a sequence restored from the trash), a head the index is
/// behind on moves forward, a name, Main flag or deletion recorded since is taken, and a live index row whose folder is gone
/// is marked deleted. A version file written past its record's head by a push a crash cut short is adopted when it
/// follows the head (its file is complete: versions are written atomically, then the record); any other file past the
/// head is moved aside to <c>versions/.stray</c>, never deleted. Runs under the sequence's lease.
/// behind on moves forward, a name or deletion recorded since is taken, and a row the index marked deleted is listed again
/// when its live record says it is not. Only a record that says so, live or in the projects' trash, deletes a project:
/// never a folder that is not there (moved for a moment, a backup put back, a sync tool) nor a record that cannot be read,
/// whose project stays as the index has it (<see cref="ProjectReconcile.HeldBack"/>). A version file written past its
/// record's head by a push a crash cut short is adopted when it follows the head (its file is complete: versions are
/// written atomically, then the record); any other file past the head, or a second file of a number the record's chain
/// names another for, is moved aside to <c>versions/.stray</c>, never deleted. Runs under the sequence's lease, or without
/// writing to the archive at all when it is read-only.
/// </summary>
internal sealed class ProjectIndexer(ChronosDatabase db)
{
private const string StrayFolder = ".stray";
/// <summary>Reconciles the projects of <paramref name="sequenceId"/> in <paramref name="folder"/>; the caller holds the sequence's lease.</summary>
public async Task ReconcileAsync(SequencePaths folder, Guid sequenceId, CancellationToken cancellationToken)
/// <param name="writable">False for a read-only archive: the sidecars are read as they are and only the index is written
/// (a version past its record's head is indexed, not adopted into the record; nothing is moved aside).</param>
public async Task<ProjectReconcile> ReconcileAsync(SequencePaths folder, Guid sequenceId, CancellationToken cancellationToken, bool writable = true)
{
var seen = new HashSet<Guid>();
if (Directory.Exists(folder.ProjectsDirectory))
var heldBack = new HashSet<Guid>();
var live = new Dictionary<Guid, ProjectRecord>();
if (!Directory.Exists(folder.ProjectsDirectory)) return new ProjectReconcile(true, seen, heldBack, live);
foreach (var (directory, trashed) in ProjectFolders(folder))
{
foreach (var (directory, trashed) in ProjectFolders(folder))
cancellationToken.ThrowIfCancellationRequested();
ProjectRecord record;
try
{
cancellationToken.ThrowIfCancellationRequested();
ProjectRecord record;
try
{
record = ProjectSidecars.ReadRecord(Path.Combine(directory, ArchiveLayout.ProjectFile));
}
catch (Exception ex) when (ex is IOException or InvalidDataException or UnauthorizedAccessException)
{
Log.Warning("projects", $"The project in {directory} cannot be read", ex);
continue;
}
// A sequence folder copied by hand carries the projects of its source, which stay that sequence's.
if (record.SequenceId != sequenceId || !seen.Add(record.Id)) continue;
if (trashed && record.Deleted is null) record = record with { Deleted = Directory.GetLastWriteTimeUtc(directory) };
if (record.Deleted is null) record = RollForward(folder, record, directory);
await IndexAsync(record, directory, cancellationToken).ConfigureAwait(false);
record = ProjectSidecars.ReadRecord(Path.Combine(directory, ArchiveLayout.ProjectFile));
}
catch (Exception ex) when (ex is IOException or InvalidDataException or UnauthorizedAccessException)
{
Log.Warning("projects", $"The project in {directory} cannot be read; it stays as the index has it", ex);
// Its folder names it (<id>, or <id>-<time> in the trash): its row stays as it is, never taken for gone nor written
// again, and a live one is held back (it may be Main).
var name = Path.GetFileName(directory);
if (Guid.TryParseExact(trashed && name.Length > 32 ? name[..32] : name, "N", out var named) && seen.Add(named) && !trashed) heldBack.Add(named);
continue;
}
// A sequence folder copied by hand carries the projects of its source, which stay that sequence's.
if (record.SequenceId != sequenceId || !seen.Add(record.Id)) continue;
if (trashed && record.Deleted is null) record = record with { Deleted = Directory.GetLastWriteTimeUtc(directory) };
if (record.Deleted is null)
{
record = RollForward(folder, record, directory, writable);
live[record.Id] = record;
}
try
{
if (!await IndexAsync(record, directory, trashed, writable, cancellationToken).ConfigureAwait(false)) heldBack.Add(record.Id);
}
catch (DatabaseException ex) when (ex.IsUniqueViolation || ex.IsForeignKeyViolation)
{
// One project the index refuses never stops the others, nor a scan of the archive: it is read again next time.
Log.Warning("projects", $"Project {record.Id} of sequence {sequenceId} could not be indexed", ex);
if (record.Deleted is null) heldBack.Add(record.Id);
}
}
foreach (var gone in await db.Projects.ListBySequenceAsync(sequenceId, includeDeleted: false, cancellationToken).ConfigureAwait(false))
return new ProjectReconcile(false, seen, heldBack, live);
}
/// <summary>
/// <paramref name="name"/>, or "<paramref name="name"/> (2)", "(3)"… (within <see cref="ProjectService.MaxNameLength"/>):
/// the first no project of <paramref name="live"/> but <paramref name="projectId"/> has.
/// </summary>
internal static string FreeName(IReadOnlyList<SequenceProject> live, string name, Guid? projectId)
{
var taken = live.Where(p => p.Id != projectId).Select(p => p.Name).ToHashSet(StringComparer.OrdinalIgnoreCase);
var candidate = name;
for (var n = 2; taken.Contains(candidate); n++)
{
if (seen.Contains(gone.Id)) continue;
Log.Warning("projects", $"Project {gone.Id} of sequence {sequenceId} has no folder any more; it is marked deleted");
await db.Projects.UpdateAsync(gone with { DeletedAt = DateTimeOffset.UtcNow, UpdatedAt = DateTimeOffset.UtcNow }, cancellationToken).ConfigureAwait(false);
var suffix = $" ({n.ToString(CultureInfo.InvariantCulture)})";
candidate = (name.Length + suffix.Length > ProjectService.MaxNameLength ? name[..(ProjectService.MaxNameLength - suffix.Length)].TrimEnd() : name) + suffix;
}
return candidate;
}
/// <summary>The project folders of a sequence: the live ones, then those in the projects' trash (<c>&lt;id&gt;-&lt;time&gt;</c>).</summary>
@@ -67,25 +114,26 @@ internal sealed class ProjectIndexer(ChronosDatabase db)
/// <summary>
/// The record with the version files that follow its head adopted (a push that wrote its version but not the record),
/// and the files past the head that do not follow it moved aside; the record is written again when its head moved.
/// and the files past the head that do not follow it moved aside; the record is written again when its head moved. A
/// file past the head that cannot be read proves nothing: then none is moved aside.
/// </summary>
private static ProjectRecord RollForward(SequencePaths folder, ProjectRecord record, string directory)
private static ProjectRecord RollForward(SequencePaths folder, ProjectRecord record, string directory, bool writable)
{
var versions = Path.Combine(directory, ArchiveLayout.ProjectVersionsFolder);
var files = ProjectSidecars.ListVersions(versions);
var head = record.Head;
var moved = false;
var unread = false;
while (true)
{
var next = (head?.Number ?? 0) + 1;
var candidates = files.Where(f => f.Number == next).ToList();
ProjectVersion? follower = null;
foreach (var candidate in candidates)
foreach (var candidate in files.Where(f => f.Number == next))
{
try
{
var version = ProjectSidecars.ReadVersion(candidate.Path);
if (version.ProjectId == record.Id && version.ParentId == head?.Id)
if (version.ProjectId == record.Id && version.Id == candidate.Id && version.ParentId == head?.Id)
{
follower = version;
break;
@@ -94,86 +142,192 @@ internal sealed class ProjectIndexer(ChronosDatabase db)
catch (Exception ex) when (ex is IOException or InvalidDataException or UnauthorizedAccessException)
{
Log.Warning("projects", $"Version file {candidate.Path} cannot be read", ex);
unread = true;
}
}
if (follower is null) break;
head = new ProjectHead(follower.Id, follower.Number);
moved = true;
}
foreach (var stray in files.Where(f => f.Number > (head?.Number ?? 0)))
if (writable && !unread)
{
try
{
var aside = Path.Combine(versions, StrayFolder);
Directory.CreateDirectory(aside);
File.Move(stray.Path, Path.Combine(aside, Path.GetFileName(stray.Path) + "." + DateTime.UtcNow.ToString("yyyyMMddHHmmssfff", CultureInfo.InvariantCulture)));
Log.Warning("projects", $"Version file {stray.Path} does not follow the head of project {record.Id}; moved aside");
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
Log.Warning("projects", $"Version file {stray.Path} could not be moved aside", ex);
}
foreach (var stray in files.Where(f => f.Number > (head?.Number ?? 0)))
SetAside(versions, stray.Path, $"does not follow the head of project {record.Id}");
}
if (!moved) return record;
var rolled = record with { Head = head, Updated = DateTimeOffset.UtcNow };
ProjectSidecars.WriteRecord(folder.ProjectRecordPath(record.Id), rolled);
if (!writable) return rolled;
try
{
ProjectSidecars.WriteRecord(folder.ProjectRecordPath(record.Id), rolled);
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
// The index takes the versions all the same; the record is written with them before anything else is (ProjectService).
Log.Warning("projects", $"The record of project {record.Id} could not be moved forward to version {head!.Number}", ex);
}
return rolled;
}
/// <summary>Writes what the index lacks of one project's record and versions.</summary>
private async Task IndexAsync(ProjectRecord record, string directory, CancellationToken cancellationToken)
/// <summary>A version file moved aside into <c>versions/.stray</c> (kept, never deleted).</summary>
private static void SetAside(string versions, string path, string why)
{
try
{
var aside = Path.Combine(versions, StrayFolder);
Directory.CreateDirectory(aside);
File.Move(path, Path.Combine(aside, Path.GetFileName(path) + "." + DateTime.UtcNow.ToString("yyyyMMddHHmmssfff", CultureInfo.InvariantCulture)));
Log.Warning("projects", $"Version file {path} {why}; moved aside");
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
Log.Warning("projects", $"Version file {path} could not be moved aside", ex);
}
}
/// <summary>
/// Writes what the index lacks of one project's record and versions. False when its head could not be reached: the
/// record names a version this Chronos cannot read (damaged, of a newer Chronos), and the index's head stays on the
/// newest one it holds below it.
/// </summary>
private async Task<bool> IndexAsync(ProjectRecord record, string directory, bool trashed, bool writable, CancellationToken cancellationToken)
{
var row = await db.Projects.GetAsync(record.Id, cancellationToken).ConfigureAwait(false);
if (row is null)
{
var project = record.ToProject();
var live = await db.Projects.ListBySequenceAsync(record.SequenceId, includeDeleted: false, cancellationToken).ConfigureAwait(false);
if (project.DeletedAt is null && project.IsMain && live.Any(p => p.IsMain))
if (project.DeletedAt is null)
{
Log.Warning("projects", $"Project {record.Id} says it is Main, but sequence {record.SequenceId} has one; indexed as another project");
project = project with { IsMain = false };
var live = await db.Projects.ListBySequenceAsync(record.SequenceId, includeDeleted: false, cancellationToken).ConfigureAwait(false);
if (project.IsMain && live.Any(p => p.IsMain))
{
Log.Warning("projects", $"Project {record.Id} says it is Main, but sequence {record.SequenceId} has one; indexed as another project");
project = project with { IsMain = false };
}
project = project with { Name = FreeName(live, project.Name, project.Id) };
}
if (project.DeletedAt is null && live.Any(p => string.Equals(p.Name, project.Name, StringComparison.OrdinalIgnoreCase)))
project = project with { Name = project.Name + " (" + project.Id.ToString("N")[..6] + ")" };
// The head follows its versions below, so a version the index cannot take never leaves a head without its row.
await db.Projects.InsertAsync(project with { HeadVersionId = null, HeadNumber = 0 }, cancellationToken).ConfigureAwait(false);
row = project with { HeadVersionId = null, HeadNumber = 0 };
await db.Projects.InsertAsync(row, cancellationToken).ConfigureAwait(false);
}
var indexed = await db.Projects.ListVersionIdsAsync(record.Id, cancellationToken).ConfigureAwait(false);
var headNumber = row.HeadNumber;
Guid? headId = row.HeadVersionId;
var limit = record.Head?.Number ?? 0;
foreach (var file in ProjectSidecars.ListVersions(Path.Combine(directory, ArchiveLayout.ProjectVersionsFolder)))
await IndexVersionsAsync(record, directory, writable, cancellationToken).ConfigureAwait(false);
var (headId, headNumber) = (row.HeadVersionId, row.HeadNumber);
var reached = true;
if (record.Head is { } head && (head.Number > headNumber || (head.Number == headNumber && head.Id != headId)))
{
if (file.Number > limit || indexed.ContainsKey(file.Number)) continue;
ProjectVersion version;
if (await db.Projects.GetVersionAsync(head.Id, cancellationToken).ConfigureAwait(false) is { } indexed && indexed.ProjectId == record.Id
&& indexed.Number == head.Number)
{
(headId, headNumber) = (head.Id, head.Number);
}
else
{
reached = false;
Log.Warning("projects", $"The newest version of project {record.Id} ({head.Number}) cannot be read; the project is shown as far as it can be, and not changed");
var below = await db.Projects.ListVersionsAsync(record.Id, head.Number, 1, cancellationToken).ConfigureAwait(false);
if (below.Count > 0 && below[0].Number > headNumber) (headId, headNumber) = (below[0].Id, below[0].Number);
}
}
var updated = row with { HeadVersionId = headId, HeadNumber = headNumber };
if (record.Deleted is not null || trashed)
{
updated = updated with { DeletedAt = row.DeletedAt ?? record.Deleted };
}
else if (row.DeletedAt is not null || row.Name != record.Name)
{
var others = (await db.Projects.ListBySequenceAsync(record.SequenceId, includeDeleted: false, cancellationToken).ConfigureAwait(false))
.Where(p => p.Id != record.Id).ToList();
if (row.DeletedAt is not null)
{
// Its live record says it is not deleted: listed again, Main again unless the sequence has another.
Log.Warning("projects", $"Project {record.Id} of sequence {record.SequenceId} is live in its sidecars; it is listed again");
updated = updated with { DeletedAt = null, IsMain = row.IsMain && !others.Any(p => p.IsMain) };
}
// The record's name, else (another project has it) the name the index has, else one free.
updated = updated with
{
Name = FreeName(others, record.Name, record.Id) == record.Name ? record.Name
: FreeName(others, row.Name, record.Id) == row.Name ? row.Name
: FreeName(others, record.Name, record.Id),
};
}
if (updated != row) await db.Projects.UpdateAsync(updated, cancellationToken).ConfigureAwait(false);
return reached || record.Deleted is not null || trashed;
}
/// <summary>
/// The version files up to the record's head the index lacks, taken in. Of several files of one number, the one the
/// record's chain names is taken and the others are moved aside; when the chain cannot tell (a version on it cannot be
/// read), none is taken.
/// </summary>
private async Task IndexVersionsAsync(ProjectRecord record, string directory, bool writable, CancellationToken cancellationToken)
{
var versions = Path.Combine(directory, ArchiveLayout.ProjectVersionsFolder);
var files = ProjectSidecars.ListVersions(versions);
var limit = record.Head?.Number ?? 0;
var indexed = new Dictionary<int, Guid>(await db.Projects.ListVersionIdsAsync(record.Id, cancellationToken).ConfigureAwait(false));
Dictionary<int, Guid>? chain = null;
foreach (var number in files.Where(f => f.Number <= limit && !indexed.ContainsKey(f.Number)).GroupBy(f => f.Number))
{
var candidates = number.ToList();
if (candidates.Count > 1)
{
chain ??= Chain(record, files);
var named = candidates.Where(c => chain.TryGetValue(c.Number, out var id) && id == c.Id).ToList();
if (named.Count == 0)
{
Log.Warning("projects", $"Project {record.Id} has {candidates.Count} version files numbered {number.Key} and its record's chain names none; none is indexed");
continue;
}
if (writable)
{
foreach (var other in candidates.Where(c => c.Id != named[0].Id))
SetAside(versions, other.Path, $"has the number of another version of project {record.Id}");
}
candidates = named;
}
foreach (var file in candidates)
{
ProjectVersion version;
try
{
version = ProjectSidecars.ReadVersion(file.Path);
}
catch (Exception ex) when (ex is IOException or InvalidDataException or UnauthorizedAccessException)
{
Log.Warning("projects", $"Version file {file.Path} cannot be read; it is not indexed", ex);
continue;
}
if (version.ProjectId != record.Id || version.Number != file.Number || version.Id != file.Id) continue;
if (await db.Projects.GetVersionAsync(version.Id, cancellationToken).ConfigureAwait(false) is not null)
{
Log.Warning("projects", $"Version file {file.Path} holds a version the index has under another number; it is not indexed");
continue;
}
await db.Projects.InsertVersionAsync(version, cancellationToken).ConfigureAwait(false);
indexed[version.Number] = version.Id;
}
}
}
/// <summary>The version ids by number along the record's chain, from its head back through each parent, as far as their files read.</summary>
private static Dictionary<int, Guid> Chain(ProjectRecord record, IReadOnlyList<(int Number, Guid Id, string Path)> files)
{
var chain = new Dictionary<int, Guid>();
var byId = files.GroupBy(f => f.Id).ToDictionary(g => g.Key, g => g.First());
for (var id = record.Head?.Id; id is { } current && byId.TryGetValue(current, out var file) && !chain.ContainsKey(file.Number);)
{
chain[file.Number] = file.Id;
try
{
version = ProjectSidecars.ReadVersion(file.Path);
id = ProjectSidecars.ReadVersion(file.Path).ParentId;
}
catch (Exception ex) when (ex is IOException or InvalidDataException or UnauthorizedAccessException)
{
Log.Warning("projects", $"Version file {file.Path} cannot be read; it is not indexed", ex);
continue;
Log.Warning("projects", $"Version file {file.Path} cannot be read", ex);
break;
}
if (version.ProjectId != record.Id || version.Number != file.Number) continue;
await db.Projects.InsertVersionAsync(version, cancellationToken).ConfigureAwait(false);
}
if (record.Head is { } head && head.Number > headNumber && await db.Projects.GetVersionAsync(head.Id, cancellationToken).ConfigureAwait(false) is not null)
(headId, headNumber) = (head.Id, head.Number);
var updated = row with
{
Name = row.DeletedAt is null && record.Deleted is null && row.Name != record.Name && !await NameTakenAsync(record, cancellationToken).ConfigureAwait(false)
? record.Name
: row.Name,
HeadVersionId = headId,
HeadNumber = headNumber,
DeletedAt = row.DeletedAt ?? record.Deleted,
};
if (updated != row) await db.Projects.UpdateAsync(updated, cancellationToken).ConfigureAwait(false);
return chain;
}
private async Task<bool> NameTakenAsync(ProjectRecord record, CancellationToken cancellationToken) =>
(await db.Projects.ListBySequenceAsync(record.SequenceId, includeDeleted: false, cancellationToken).ConfigureAwait(false))
.Any(p => p.Id != record.Id && string.Equals(p.Name, record.Name, StringComparison.OrdinalIgnoreCase));
}