The desktop keyed every server by the address it was reached at, so one
NAS reached by name and by IP, or after an address change, looked like two
servers, and all servers shared one workspace and one cache. Each server
now announces a stable id, and the desktop's folders are sorted into one
home per server, so that every server can have its own archive of
downloaded originals (or5 answers Q1).
- The server makes a Guid v7 once, keeps it as the app setting server.id
(so it survives restarts and restores, on SQLite and Postgres) and
announces it as ServerInfo.ServerId. Older clients ignore the field, and
older servers read as null.
- ServerIdentity keys a server as id:<N> when it announces an id, and by
its address otherwise. Address keys are kept as aliases.
- HomeFolders names the homes: local, servers/<id> or
servers/addr-<hash16>. The hash equals the 1.0 proxy cache folder name.
It also names the per-server originals archive root.
- WorkspaceHomes splits the workspace into one format-1 store per home
under a format-2 parent manifest, which 1.0 refuses. A 1.0 workspace
stays one store until it is sorted.
- HomeLayoutMigration plans as a dry run and applies through a journal:
- only same-folder renames, each checked by inventory;
- a clash keeps the copy saved last and sends the other to the trash;
- it resumes after a crash at any step;
- it skips leased projects;
- it defers an offline workspace or one being moved, while the cache
part still runs.
- RekeyAsync moves an alias home to the id home: whole folders, or
project-by-project merges. It also rewrites working states under their
lease, and moves sync_links (the newer row wins a clash).
- ProxyCache, the job root resolver, WorkspaceMover and WorkspaceCleaner
understand the split layout. The new layout is opt-in through
SyncOptions.ServersDirectory, so the app sees no change until it is
wired.
Refs #66
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
802 lines
42 KiB
C#
802 lines
42 KiB
C#
using System.Text.Json;
|
|
using System.Text.Json.Serialization;
|
|
using Chronos.Core.Archiving;
|
|
using Chronos.Core.Diagnostics;
|
|
using Chronos.Core.IO;
|
|
using Chronos.Core.Model;
|
|
using Chronos.Core.Project;
|
|
using Chronos.Data.Projects;
|
|
using Chronos.Data.Sync;
|
|
|
|
namespace Chronos.Data.Workspaces;
|
|
|
|
/// <summary>The manifest of a workspace folder (<c>workspace.json</c>).</summary>
|
|
/// <param name="Layout">
|
|
/// <see cref="WorkspaceHomes.HomesLayout"/> in the manifest of format <see cref="WorkspaceHomes.ParentFormatVersion"/> of a
|
|
/// workspace split into homes (owner review 5, #66): the folder holds no projects of its own but one store per home; null
|
|
/// for a store.
|
|
/// </param>
|
|
public sealed record WorkspaceManifest(string Format, int FormatVersion, Guid Id, DateTimeOffset Created, string? Layout = null);
|
|
|
|
/// <summary>
|
|
/// A workspace's sync state of one project besides its versions (<c>sync.json</c>): a restore to record, a conflict to
|
|
/// settle, and the journal of a replay or a save-both under way, which the next sync finishes or undoes after a crash.
|
|
/// </summary>
|
|
/// <param name="RestoredFrom">The version a restore brought back; the next commit is a "restore" of it.</param>
|
|
/// <param name="Conflict">The conflict a pull or push stopped at, until the user settles it.</param>
|
|
/// <param name="Replay">A replay (or a base moved onto local commits the home holds) written in steps; null when none is under way.</param>
|
|
/// <param name="SaveBoth">A save-both that made its copy at the home and has not finished; null when none is under way.</param>
|
|
public sealed record WorkspaceSyncState(Guid? RestoredFrom, WorkspaceConflict? Conflict, WorkspaceReplay? Replay = null, WorkspaceSaveBoth? SaveBoth = null)
|
|
{
|
|
public static WorkspaceSyncState None { get; } = new(null, null);
|
|
|
|
/// <summary>Nothing to keep: the file can go.</summary>
|
|
[JsonIgnore]
|
|
public bool IsEmpty => RestoredFrom is null && Conflict is null && Replay is null && SaveBoth is null;
|
|
}
|
|
|
|
/// <summary>
|
|
/// The journal of a replay (<see cref="WorkspaceSync"/>): the base it moves the project onto, the local versions it
|
|
/// replaces, the replayed ones it writes and the local versions the home already holds (their answer lost), whose files
|
|
/// take the home's copy. The bundle's base is the commit point: once it is <see cref="OntoBase"/> the replay is finished
|
|
/// (the replaced set aside, the held ones adopted), before it the replayed ones are set aside and nothing else moved.
|
|
/// </summary>
|
|
/// <param name="Settles">
|
|
/// The replay settles the conflict recorded beside it (a pull, a push or a choice that merged): once its base is
|
|
/// <see cref="OntoBase"/> that conflict is over, the choice made applied. False for a base moved onto versions the home
|
|
/// held, which leaves the conflict to settle (and for a journal an earlier build wrote).
|
|
/// </param>
|
|
public sealed record WorkspaceReplay(Guid OntoBase, IReadOnlyList<Guid> Replaced, IReadOnlyList<Guid> Replayed, IReadOnlyList<Guid> Adopted,
|
|
bool Settles = false);
|
|
|
|
/// <summary>
|
|
/// A save-both under way (<see cref="WorkspaceSync"/>): the copy made at the home and its first version (the base the
|
|
/// local versions are sent on top of), the base this project had when it started, and the id each local version is sent
|
|
/// under, so a retry sends to the same copy what it lacks and the home answers "already stored" for the rest. Before the
|
|
/// copy is asked of the home its name and the version it is made from are journaled (<see cref="Intent"/>, the copy's ids
|
|
/// still empty): a retry after a cut finds the copy the home made by them instead of making another.
|
|
/// </summary>
|
|
public sealed record WorkspaceSaveBoth(Guid CopyId, Guid CopyBaseId, int CopyBaseNumber, Guid? FromBase, IReadOnlyList<WorkspaceMove> Moves,
|
|
WorkspaceCopyIntent? Intent = null)
|
|
{
|
|
/// <summary>The copy is asked of the home but not known yet: only its <see cref="Intent"/> is.</summary>
|
|
[JsonIgnore]
|
|
public bool Intended => CopyId == Guid.Empty;
|
|
}
|
|
|
|
/// <summary>
|
|
/// The copy a save-both asks the home for, journaled before it asks: its name, the version it is made from (null: Main's
|
|
/// head) with that version's content hash, and when it was asked. A project of that name made since, whose only version
|
|
/// holds that content, is that copy.
|
|
/// </summary>
|
|
public sealed record WorkspaceCopyIntent(string Name, Guid? From, string? FromHash, DateTimeOffset Asked);
|
|
|
|
/// <summary>A local version (<paramref name="LocalId"/>) and the id it is sent under to a save-both's copy.</summary>
|
|
public sealed record WorkspaceMove(Guid LocalId, Guid CopyVersionId);
|
|
|
|
/// <summary>A version file of a workspace that cannot be read (<see cref="WorkspaceProject.UnreadableVersions"/>): the number and id its name carries, and why.</summary>
|
|
public sealed record UnreadableVersion(int Number, Guid Id, string Path, string Reason);
|
|
|
|
/// <summary>
|
|
/// A pull or push that met real conflicts (owner review 2, R8): the home's head it met, the version the local commits were
|
|
/// made from, the local commits (oldest first) and the members both sides changed (mine, theirs). Settled by keeping mine,
|
|
/// taking the server's (each member or all), or saving both (<see cref="WorkspaceSync.ResolveAsync"/>).
|
|
/// </summary>
|
|
public sealed record WorkspaceConflict(Guid TheirsVersionId, int TheirsNumber, Guid BaseVersionId, IReadOnlyList<Guid> Mine,
|
|
IReadOnlyList<SettingsDifference> Differences, DateTimeOffset Found)
|
|
{
|
|
/// <summary>
|
|
/// The home's head holds settings of a newer Chronos (a difference at <see cref="ProjectMerge.FormatPath"/>): nothing of
|
|
/// this workspace's is merged into them. Only Save both settles it (this workspace's versions go to a new project in this
|
|
/// Chronos's format), or a Chronos that reads them; keeping mine or taking the server's is refused.
|
|
/// </summary>
|
|
[JsonIgnore]
|
|
public bool NewerFormat => Differences.Any(d => d.Path == ProjectMerge.FormatPath);
|
|
}
|
|
|
|
/// <summary>
|
|
/// Parameters of a <see cref="JobKind.WorkspaceCopy"/> job: target = the home's sequence; the workspace folder, the
|
|
/// project whose media to copy and what to copy (<see cref="BundleMedia.Proxies"/>, or <see cref="BundleMedia.Originals"/>:
|
|
/// the proxies plus the originals, phase 5c).
|
|
/// </summary>
|
|
/// <param name="Server">The server the home's sequence is on (<see cref="SyncService.ServerKey"/>, phase 5b); null for a sequence of this PC's archive.</param>
|
|
public sealed record WorkspaceCopyParameters(string WorkspaceRoot, Guid ProjectId, string Media, string? Server = null);
|
|
|
|
/// <summary>
|
|
/// Parameters of a <see cref="JobKind.WorkspaceRender"/> job (phase 5c): target = the home's sequence; the workspace folder,
|
|
/// the project rendered from the originals its media hold, the version rendered (null: the head version the workspace
|
|
/// holds), and the render's export settings as Protocol <c>RenderJobParameters</c> JSON (portable settings, preset,
|
|
/// overrides, batch), as <see cref="Processing.ProcessingJobs"/> writes them for a render.
|
|
/// </summary>
|
|
/// <param name="ProjectVersionId">The version rendered and recorded with each output; null renders the head version the workspace holds.</param>
|
|
/// <param name="RenderJson">Protocol <c>RenderJobParameters</c> as JSON.</param>
|
|
/// <param name="Server">The server the home's sequence is on (<see cref="SyncService.ServerKey"/>); null for a sequence of this PC's archive.</param>
|
|
public sealed record WorkspaceRenderParameters(string WorkspaceRoot, Guid ProjectId, Guid? ProjectVersionId, string RenderJson, string? Server = null);
|
|
|
|
/// <summary>
|
|
/// Parameters of a <see cref="JobKind.WorkspaceSend"/> job (phase 5c): target = the home's sequence; the workspace folder,
|
|
/// the project, and the ledger entries (<see cref="WorkspaceRenderEntry.RenderId"/>) whose files go home.
|
|
/// </summary>
|
|
/// <param name="RenderIds">The renders sent; empty sends every pending entry of the project's ledger.</param>
|
|
/// <param name="Server">The server the home's sequence is on (<see cref="SyncService.ServerKey"/>); null for a sequence of this PC's archive.</param>
|
|
public sealed record WorkspaceSendParameters(string WorkspaceRoot, Guid ProjectId, IReadOnlyList<Guid> RenderIds, string? Server = null);
|
|
|
|
/// <summary>
|
|
/// Parameters of a <see cref="JobKind.WorkspaceMove"/> job (phase 5c): no target; the workspace folder moved
|
|
/// (<paramref name="From"/>), the folder it goes to, and the fingerprint of the plan the user confirmed (the job refuses a
|
|
/// workspace that changed since).
|
|
/// </summary>
|
|
public sealed record WorkspaceMoveParameters(string From, string To, string PlanFingerprint);
|
|
|
|
[JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase, DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
|
|
UseStringEnumConverter = true, WriteIndented = true)]
|
|
[JsonSerializable(typeof(WorkspaceManifest))]
|
|
[JsonSerializable(typeof(WorkspaceMedia))]
|
|
[JsonSerializable(typeof(WorkspaceSyncState))]
|
|
[JsonSerializable(typeof(WorkspaceCopyParameters))]
|
|
[JsonSerializable(typeof(WorkspaceRenderParameters))]
|
|
[JsonSerializable(typeof(WorkspaceSendParameters))]
|
|
[JsonSerializable(typeof(WorkspaceMoveParameters))]
|
|
[JsonSerializable(typeof(WorkspaceRenders))]
|
|
[JsonSerializable(typeof(ServerHomeRecord))]
|
|
internal sealed partial class WorkspaceJson : JsonSerializerContext;
|
|
|
|
/// <summary>
|
|
/// The workspace of this PC (owner review 2, R8): a folder on a fast disk where projects are edited, never inside the
|
|
/// library. <c><root>/workspace.json</c> names it; each project has a folder named by its id
|
|
/// (<see cref="WorkspaceProject"/>). There is no index: the store lists the folders holding a <c>project.chronos</c>. The
|
|
/// store takes any folder; the desktop's workspace (<c>%LOCALAPPDATA%\Chronos\workspace</c> unless set otherwise) is split
|
|
/// into one store per home since owner review 5 (<see cref="WorkspaceHomes"/>: <c>local</c>, <c>servers\<id></c>).
|
|
/// </summary>
|
|
public sealed class WorkspaceStore
|
|
{
|
|
public const string ManifestFile = "workspace.json";
|
|
public const string FormatName = "chronos-workspace";
|
|
public const int FormatVersion = 1;
|
|
|
|
/// <summary>The folder name of the workspace inside the program's data folder.</summary>
|
|
public const string DefaultFolderName = "workspace";
|
|
|
|
/// <summary>
|
|
/// Opens the workspace at <paramref name="root"/>, making the folder and its manifest when there is none. The manifest
|
|
/// is written whole to a temporary file and moved into place, so a crash never leaves half of one; a manifest left
|
|
/// empty, or unreadable in a folder that holds no project yet (an earlier build's interrupted first write), is made again.
|
|
/// </summary>
|
|
/// <exception cref="InvalidDataException">The folder's <c>workspace.json</c> is not a workspace manifest, or a newer one.</exception>
|
|
/// <exception cref="IOException">The folder cannot be made or read.</exception>
|
|
public WorkspaceStore(string root)
|
|
{
|
|
ArgumentException.ThrowIfNullOrWhiteSpace(root);
|
|
Root = FullRoot(root);
|
|
Directory.CreateDirectory(Root);
|
|
var path = Path.Combine(Root, ManifestFile);
|
|
if (!File.Exists(path)) Create(path, replace: false);
|
|
else if (Unfinished(path)) Create(path, replace: true);
|
|
Manifest = Read(path);
|
|
}
|
|
|
|
/// <summary>
|
|
/// The projects a workspace split into homes still holds where 1.0 kept them (<c><parent>/<project id></c>: one in
|
|
/// use when the folders were sorted, owner review 5): a view of the parent with its manifest, which makes and writes nothing
|
|
/// (<see cref="WorkspaceHomes"/>).
|
|
/// </summary>
|
|
internal WorkspaceStore(string root, WorkspaceManifest manifest)
|
|
{
|
|
Root = FullRoot(root);
|
|
Manifest = manifest;
|
|
}
|
|
|
|
public string Root { get; }
|
|
|
|
public WorkspaceManifest Manifest { get; }
|
|
|
|
public Guid Id => Manifest.Id;
|
|
|
|
/// <summary>The folder of a project, whether it exists or not.</summary>
|
|
public WorkspaceProject Project(Guid projectId) => new(this, projectId);
|
|
|
|
/// <summary>The project's folder when it holds a working state; null otherwise.</summary>
|
|
public WorkspaceProject? Find(Guid projectId)
|
|
{
|
|
var project = Project(projectId);
|
|
return project.Exists ? project : null;
|
|
}
|
|
|
|
/// <summary>The folder a workspace at <paramref name="root"/> has as its <see cref="Root"/> (the store opened or not).</summary>
|
|
internal static string FullRoot(string root) => LongPath.Normalize(Path.TrimEndingDirectorySeparator(Path.GetFullPath(root)));
|
|
|
|
/// <summary>
|
|
/// The projects of the workspace: every folder named by a project id that holds a <c>project.chronos</c>. A folder named
|
|
/// by a project id that holds nothing but what a lease no one holds left (a removal that met a lease taken meanwhile)
|
|
/// goes (<see cref="IsLeftover"/>).
|
|
/// </summary>
|
|
public IReadOnlyList<WorkspaceProject> List()
|
|
{
|
|
var found = new List<WorkspaceProject>();
|
|
foreach (var directory in Directory.EnumerateDirectories(Root).Order(StringComparer.Ordinal))
|
|
{
|
|
if (!Guid.TryParseExact(Path.GetFileName(directory), "N", out var id)) continue;
|
|
if (File.Exists(Path.Combine(directory, WorkspaceProject.BundleFile))) found.Add(Project(id));
|
|
else RemoveLeftover(directory, DateTime.UtcNow);
|
|
}
|
|
return found;
|
|
}
|
|
|
|
/// <summary>
|
|
/// A project's folder that is no project and holds nothing a user or a writer has: no <c>project.chronos</c>, no folder,
|
|
/// only lease files (the lease and those set aside as stale), none of them held, and nothing written there for as long
|
|
/// as a lease lasts (a lease taken this moment makes its folder first).
|
|
/// </summary>
|
|
internal static bool IsLeftover(string folder, DateTime nowUtc)
|
|
{
|
|
var info = new DirectoryInfo(folder);
|
|
if (!info.Exists || File.Exists(Path.Combine(folder, WorkspaceProject.BundleFile)) || info.LastWriteTimeUtc + LeaseFile.DefaultDuration > nowUtc) return false;
|
|
foreach (var entry in info.EnumerateFileSystemInfos())
|
|
{
|
|
if (entry is not FileInfo file || !WorkspaceProject.IsLeaseFile(file.Name) || file.LastWriteTimeUtc + LeaseFile.DefaultDuration > nowUtc) return false;
|
|
if (string.Equals(file.Name, WorkspaceProject.LeaseFileName, StringComparison.OrdinalIgnoreCase) && LeaseFile.Read(file.FullName) is { } held
|
|
&& held.Expires >= nowUtc)
|
|
return false;
|
|
}
|
|
return true;
|
|
}
|
|
|
|
private static void RemoveLeftover(string folder, DateTime nowUtc)
|
|
{
|
|
try
|
|
{
|
|
if (!IsLeftover(folder, nowUtc)) return;
|
|
Directory.Delete(folder, recursive: true);
|
|
Log.Info("workspace", $"{folder} held only an old lease and was removed");
|
|
}
|
|
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
|
|
{
|
|
Log.Debug("workspace", $"{folder} stays: {ex.Message}");
|
|
}
|
|
}
|
|
|
|
/// <summary>The working states of the workspace's projects (one unreadable is left out, and logged).</summary>
|
|
public IReadOnlyList<(WorkspaceProject Project, ProjectBundle Bundle)> ListBundles()
|
|
{
|
|
var bundles = new List<(WorkspaceProject, ProjectBundle)>();
|
|
foreach (var project in List())
|
|
{
|
|
try
|
|
{
|
|
bundles.Add((project, project.ReadBundle()));
|
|
}
|
|
catch (Exception ex) when (ex is IOException or InvalidDataException or UnauthorizedAccessException)
|
|
{
|
|
Log.Warning("workspace", $"The project in {project.Folder} cannot be read", ex);
|
|
}
|
|
}
|
|
return bundles;
|
|
}
|
|
|
|
/// <summary>
|
|
/// Writes a new manifest. Without <paramref name="replace"/> another program opening the same folder at the same moment
|
|
/// may have made one first: that one wins (the move never replaces a file).
|
|
/// </summary>
|
|
private static void Create(string path, bool replace)
|
|
{
|
|
var manifest = new WorkspaceManifest(FormatName, FormatVersion, Guid.CreateVersion7(), DateTimeOffset.UtcNow);
|
|
var text = JsonSerializer.Serialize(manifest, WorkspaceJson.Default.WorkspaceManifest);
|
|
if (replace)
|
|
{
|
|
Log.Warning("workspace", $"{path} was left unfinished; a new manifest is written");
|
|
AtomicFile.WriteAllText(path, text);
|
|
return;
|
|
}
|
|
var temp = AtomicFile.TempPathFor(path);
|
|
try
|
|
{
|
|
using (var stream = new FileStream(temp, FileMode.CreateNew, FileAccess.Write, FileShare.None))
|
|
{
|
|
stream.Write(System.Text.Encoding.UTF8.GetBytes(text));
|
|
stream.Flush(flushToDisk: true);
|
|
}
|
|
try
|
|
{
|
|
File.Move(temp, path, overwrite: false);
|
|
}
|
|
catch (IOException) when (File.Exists(path))
|
|
{
|
|
// Another program's manifest came first.
|
|
}
|
|
}
|
|
finally
|
|
{
|
|
if (File.Exists(temp)) File.Delete(temp);
|
|
}
|
|
}
|
|
|
|
/// <summary>
|
|
/// A manifest an interrupted first write left (an earlier build wrote it in place): empty, or not JSON in a folder that
|
|
/// holds no project yet. A readable manifest, or any in a folder with projects, is never written over.
|
|
/// </summary>
|
|
private bool Unfinished(string path)
|
|
{
|
|
string text;
|
|
try
|
|
{
|
|
text = AtomicFile.ReadAllText(path);
|
|
}
|
|
catch (IOException)
|
|
{
|
|
return false;
|
|
}
|
|
if (string.IsNullOrWhiteSpace(text.Trim('\0'))) return true;
|
|
try
|
|
{
|
|
using var _ = JsonDocument.Parse(text);
|
|
return false;
|
|
}
|
|
catch (JsonException)
|
|
{
|
|
return List().Count == 0;
|
|
}
|
|
}
|
|
|
|
/// <summary>The manifest of the folder <paramref name="path"/> names, of any format (a store's or a parent's of homes); null when it is not one.</summary>
|
|
internal static WorkspaceManifest? TryRead(string path)
|
|
{
|
|
try
|
|
{
|
|
return JsonSerializer.Deserialize(AtomicFile.ReadAllText(path), WorkspaceJson.Default.WorkspaceManifest) is { Format: FormatName } manifest && manifest.Id != Guid.Empty
|
|
? manifest
|
|
: null;
|
|
}
|
|
catch (Exception ex) when (ex is JsonException or IOException or UnauthorizedAccessException)
|
|
{
|
|
return null;
|
|
}
|
|
}
|
|
|
|
/// <summary>Writes <paramref name="manifest"/> whole to a temporary file and moves it into place.</summary>
|
|
internal static void Write(string path, WorkspaceManifest manifest) =>
|
|
AtomicFile.WriteAllText(path, JsonSerializer.Serialize(manifest, WorkspaceJson.Default.WorkspaceManifest));
|
|
|
|
private static WorkspaceManifest Read(string path)
|
|
{
|
|
WorkspaceManifest? manifest;
|
|
try
|
|
{
|
|
manifest = JsonSerializer.Deserialize(AtomicFile.ReadAllText(path), WorkspaceJson.Default.WorkspaceManifest);
|
|
}
|
|
catch (JsonException ex)
|
|
{
|
|
throw new InvalidDataException($"{path} is not a workspace manifest.", ex);
|
|
}
|
|
if (manifest is null || manifest.Format != FormatName || manifest.Id == Guid.Empty) throw new InvalidDataException($"{path} is not a workspace manifest.");
|
|
if (manifest.Layout == WorkspaceHomes.HomesLayout)
|
|
throw new InvalidDataException($"{path} is the manifest of a workspace split into homes, whose projects are in its home folders ({HomeFolders.Local}, {HomeFolders.Servers}).");
|
|
if (manifest.FormatVersion > FormatVersion)
|
|
throw new InvalidDataException($"{path} uses format {manifest.FormatVersion}, newer than this Chronos ({FormatVersion}).");
|
|
return manifest;
|
|
}
|
|
}
|
|
|
|
/// <summary>
|
|
/// One project in the workspace (owner review 2, R8): <c><root>/<project id>/</c> with
|
|
/// <c>project.chronos</c> (the working state: a project file of format 2, its settings placed on the media's frames),
|
|
/// <c>media/media.json</c> with <c>media/proxies/<frame id>.jpg</c>, <c>media/originals/<frame id>-<sha8><ext></c>
|
|
/// (media of kind originals, phase 5c) and <c>media/analysis.json</c> (the home's deep report),
|
|
/// <c>versions/<number>-<id>.json</c> (commits not pushed yet, and the versions pulled), <c>sync.json</c>,
|
|
/// <c>exports/</c> with the local renders and their ledger <c>exports/renders.json</c> (phase 5c), and the lease
|
|
/// <c>.lock.lease</c> that one writer at a time holds.
|
|
/// </summary>
|
|
public sealed class WorkspaceProject
|
|
{
|
|
public const string BundleFile = "project.chronos";
|
|
public const string MediaFolder = "media";
|
|
public const string MediaIndexFile = "media.json";
|
|
public const string ProxiesFolder = "proxies";
|
|
public const string OriginalsFolder = "originals";
|
|
public const string VersionsFolder = "versions";
|
|
public const string ExportsFolder = "exports";
|
|
public const string SyncFile = "sync.json";
|
|
|
|
/// <summary>The name of the ledger of the project's local renders (<see cref="WorkspaceRenders"/>) in <see cref="ExportsFolder"/>.</summary>
|
|
public const string RendersFileName = "renders.json";
|
|
|
|
/// <summary>The ledger of the project's local renders (phase 5c), relative to <see cref="Folder"/>.</summary>
|
|
public const string RendersFile = ExportsFolder + "/" + RendersFileName;
|
|
|
|
/// <summary>
|
|
/// Old local commits replaced by their replay on top of the home's head, files the home's copy replaced, and the working
|
|
/// states a take of the server's values replaced: kept up to a month, the newest 20 at most (<see cref="PruneReplaced"/>).
|
|
/// </summary>
|
|
public const string ReplacedFolder = ".replaced";
|
|
|
|
/// <summary>
|
|
/// Beside <see cref="ReplacedFolder"/> in <c>versions/</c>: when this Chronos first set a file aside there or pruned it
|
|
/// (an ISO-8601 time). A file dated before it was set aside by an earlier build, which kept the time it was written: it
|
|
/// counts as set aside then, at the earliest (<see cref="PruneReplaced"/>).
|
|
/// </summary>
|
|
public const string ReplacedSinceFile = ".replaced.since";
|
|
|
|
private const string LeaseName = ".lock";
|
|
|
|
/// <summary>The file of the project's lease (<see cref="LockAsync"/>) in <see cref="Folder"/>.</summary>
|
|
internal const string LeaseFileName = LeaseName + ".lease";
|
|
|
|
/// <summary>The project's lease, or one set aside as stale (<c>.lock.lease.<tag>.stale</c>).</summary>
|
|
internal static bool IsLeaseFile(string name) => name.StartsWith(LeaseFileName, StringComparison.OrdinalIgnoreCase);
|
|
|
|
internal WorkspaceProject(WorkspaceStore store, Guid projectId)
|
|
{
|
|
Store = store;
|
|
ProjectId = projectId;
|
|
Folder = Path.Combine(store.Root, projectId.ToString("N"));
|
|
}
|
|
|
|
public WorkspaceStore Store { get; }
|
|
public Guid ProjectId { get; }
|
|
public string Folder { get; }
|
|
public string BundlePath => Path.Combine(Folder, BundleFile);
|
|
public string MediaDirectory => Path.Combine(Folder, MediaFolder);
|
|
public string MediaIndexPath => Path.Combine(MediaDirectory, MediaIndexFile);
|
|
public string ProxiesDirectory => Path.Combine(MediaDirectory, ProxiesFolder);
|
|
public string OriginalsDirectory => Path.Combine(MediaDirectory, OriginalsFolder);
|
|
public string AnalysisPath => Path.Combine(MediaDirectory, ArchiveLayout.AnalysisFile);
|
|
public string VersionsDirectory => Path.Combine(Folder, VersionsFolder);
|
|
public string ExportsDirectory => Path.Combine(Folder, ExportsFolder);
|
|
public string SyncPath => Path.Combine(Folder, SyncFile);
|
|
|
|
/// <summary>The ledger of the project's local renders (<see cref="RendersFile"/>, phase 5c).</summary>
|
|
public string RendersPath => Path.Combine(ExportsDirectory, RendersFileName);
|
|
|
|
public bool Exists => File.Exists(BundlePath);
|
|
|
|
/// <summary>The file a frame's proxy is copied to, relative to <see cref="MediaDirectory"/>.</summary>
|
|
public static string ProxyName(Guid frameId) => ProxiesFolder + "/" + ProxyCache.FileName(frameId);
|
|
|
|
/// <summary>
|
|
/// The file a frame's proxy of other bytes is copied to while the media name <see cref="ProxyName(Guid)"/> for the frame
|
|
/// (a conversion since): the editor keeps reading that one until the copy commits.
|
|
/// </summary>
|
|
public static string ProxyName(Guid frameId, string sha256)
|
|
{
|
|
ArgumentException.ThrowIfNullOrEmpty(sha256);
|
|
return ProxiesFolder + "/" + frameId.ToString("N") + "-" + sha256[..Math.Min(16, sha256.Length)].ToLowerInvariant() + ".jpg";
|
|
}
|
|
|
|
/// <summary>
|
|
/// The file a frame's original is copied to, relative to <see cref="MediaDirectory"/> (phase 5c):
|
|
/// <c>originals/<frame id>-<sha8><ext></c>. The first 8 hex digits of its SHA-256 tell two encodings of one
|
|
/// frame apart (a conversion since: the editor keeps the one it reads until the copy commits), and the original's own
|
|
/// extension is kept, lower case, since decoders go by it.
|
|
/// </summary>
|
|
/// <param name="extension">The original's extension, with or without its dot (".CR3", "dng", or the file name's <see cref="Path.GetExtension(string)"/>); empty or null for none.</param>
|
|
public static string OriginalName(Guid frameId, string sha256, string? extension)
|
|
{
|
|
ArgumentException.ThrowIfNullOrEmpty(sha256);
|
|
var ext = (extension ?? string.Empty).Trim().TrimStart('.').ToLowerInvariant();
|
|
if (ext.IndexOfAny(Path.GetInvalidFileNameChars()) >= 0 || ext.Contains('/') || ext.Contains('\\')) ext = string.Empty;
|
|
return OriginalsFolder + "/" + frameId.ToString("N") + "-" + sha256[..Math.Min(8, sha256.Length)].ToLowerInvariant() + (ext.Length > 0 ? "." + ext : string.Empty);
|
|
}
|
|
|
|
/// <summary>The ledger of the project's local renders; empty while it has none (<see cref="WorkspaceRenders.Read"/>).</summary>
|
|
/// <exception cref="InvalidDataException">The ledger is damaged or newer.</exception>
|
|
/// <exception cref="IOException">The ledger cannot be read.</exception>
|
|
public WorkspaceRenders ReadRenders() => WorkspaceRenders.Read(RendersPath);
|
|
|
|
/// <summary>Writes the ledger of the project's local renders whole and atomically (<see cref="WorkspaceRenders.Write"/>); the caller holds the project's lease.</summary>
|
|
public void WriteRenders(WorkspaceRenders renders) => WorkspaceRenders.Write(RendersPath, renders);
|
|
|
|
/// <summary>The absolute path of a file named relative to <see cref="MediaDirectory"/>.</summary>
|
|
public string MediaFile(string relative) => LongPath.FromPortableRelative(MediaDirectory, relative);
|
|
|
|
/// <summary>The absolute path of a local render named relative to <see cref="ExportsDirectory"/> (<see cref="WorkspaceRenderEntry.File"/>).</summary>
|
|
public string ExportFile(string relative) => LongPath.FromPortableRelative(ExportsDirectory, relative);
|
|
|
|
/// <summary>
|
|
/// The absolute path of a local render named relative to <see cref="ExportsDirectory"/>, only when it lies strictly inside
|
|
/// that folder and is not its ledger; null for a name that is empty, rooted, climbs out (<c>..</c>), names the folder
|
|
/// itself (<c>.</c>) or the ledger. What reads, sends or deletes a ledger's file goes through this, so a damaged or
|
|
/// hand-edited ledger never points it at anything else.
|
|
/// </summary>
|
|
public string? ContainedExportFile(string? relative)
|
|
{
|
|
if (string.IsNullOrWhiteSpace(relative)) return null;
|
|
var portable = relative.Replace('\\', '/');
|
|
if (Path.IsPathRooted(relative) || portable.StartsWith('/') || portable.Contains(':', StringComparison.Ordinal)
|
|
|| portable.Split('/').Any(part => part is "" or "." or "..")) return null;
|
|
string full, folder;
|
|
try
|
|
{
|
|
full = Path.GetFullPath(ExportFile(relative));
|
|
folder = Path.TrimEndingDirectorySeparator(Path.GetFullPath(ExportsDirectory)) + Path.DirectorySeparatorChar;
|
|
}
|
|
catch (Exception ex) when (ex is ArgumentException or NotSupportedException or PathTooLongException)
|
|
{
|
|
return null;
|
|
}
|
|
var comparison = OperatingSystem.IsWindows() ? StringComparison.OrdinalIgnoreCase : StringComparison.Ordinal;
|
|
if (!full.StartsWith(folder, comparison) || full.Length <= folder.Length) return null;
|
|
return string.Equals(full, Path.GetFullPath(RendersPath), comparison) ? null : full;
|
|
}
|
|
|
|
/// <summary>Holds the project for one writer (this program's sync, a copy job) until disposed.</summary>
|
|
/// <exception cref="LeaseUnavailableException">Another writer kept it for longer than <paramref name="wait"/>.</exception>
|
|
public async Task<IDisposable> LockAsync(TimeSpan wait, CancellationToken cancellationToken = default)
|
|
{
|
|
Directory.CreateDirectory(Folder);
|
|
return await LeaseFile.AcquireAsync(Folder, LeaseName, "workspace@" + Environment.MachineName, wait, cancellationToken: cancellationToken).ConfigureAwait(false);
|
|
}
|
|
|
|
/// <summary>
|
|
/// Holds the project as <see cref="LockAsync"/> does, never making its folder: taken by a writer that only clears what is
|
|
/// there, which a removal of the folder may race.
|
|
/// </summary>
|
|
/// <exception cref="DirectoryNotFoundException">The folder is not there, or went while the lease was awaited.</exception>
|
|
/// <exception cref="LeaseUnavailableException">Another writer kept it for longer than <paramref name="wait"/>.</exception>
|
|
internal async Task<IDisposable> LockExistingAsync(TimeSpan wait, CancellationToken cancellationToken = default) =>
|
|
await LeaseFile.AcquireAsync(Folder, LeaseName, "workspace@" + Environment.MachineName, wait, cancellationToken: cancellationToken, createDirectory: false)
|
|
.ConfigureAwait(false);
|
|
|
|
/// <exception cref="InvalidDataException">Not a project file of a workspace, damaged or newer.</exception>
|
|
/// <exception cref="IOException">The file cannot be read.</exception>
|
|
public ProjectBundle ReadBundle()
|
|
{
|
|
var bundle = ProjectBundle.Load(BundlePath);
|
|
if (bundle.Workspace is not { } workspace || workspace.ProjectId != ProjectId)
|
|
throw new InvalidDataException($"{BundlePath} is not the working state of project {ProjectId}.");
|
|
return bundle;
|
|
}
|
|
|
|
public void WriteBundle(ProjectBundle bundle)
|
|
{
|
|
ArgumentNullException.ThrowIfNull(bundle);
|
|
if (bundle.Workspace?.ProjectId != ProjectId) throw new ArgumentException("The project file is another project's.", nameof(bundle));
|
|
Directory.CreateDirectory(Folder);
|
|
bundle.Save(BundlePath);
|
|
}
|
|
|
|
/// <summary>The media index; null while the media were never copied, or were removed.</summary>
|
|
/// <exception cref="InvalidDataException">The index is damaged or newer.</exception>
|
|
public WorkspaceMedia? ReadMedia()
|
|
{
|
|
if (!File.Exists(MediaIndexPath)) return null;
|
|
WorkspaceMedia? media;
|
|
try
|
|
{
|
|
media = JsonSerializer.Deserialize(AtomicFile.ReadAllText(MediaIndexPath), WorkspaceJson.Default.WorkspaceMedia);
|
|
}
|
|
catch (JsonException ex)
|
|
{
|
|
throw new InvalidDataException($"{MediaIndexPath} cannot be read.", ex);
|
|
}
|
|
if (media is null || media.Format != WorkspaceMedia.FormatName) throw new InvalidDataException($"{MediaIndexPath} is not a media index.");
|
|
if (media.FormatVersion > WorkspaceMedia.CurrentFormatVersion) throw new InvalidDataException($"{MediaIndexPath} is newer than this Chronos.");
|
|
return media;
|
|
}
|
|
|
|
internal void WriteMedia(WorkspaceMedia media)
|
|
{
|
|
Directory.CreateDirectory(MediaDirectory);
|
|
AtomicFile.WriteAllText(MediaIndexPath, JsonSerializer.Serialize(media with { FormatVersion = media.WrittenFormatVersion }, WorkspaceJson.Default.WorkspaceMedia));
|
|
}
|
|
|
|
/// <summary>
|
|
/// The versions this workspace holds, oldest first. One that cannot be read is left out here, logged, and listed by
|
|
/// <see cref="UnreadableVersions"/>: the sync treats one above the base as a gap in the chain of versions to send, never
|
|
/// as a version that is not there (<see cref="WorkspaceSync"/>).
|
|
/// </summary>
|
|
public IReadOnlyList<ProjectVersion> Versions() => ReadVersions(log: true).Versions;
|
|
|
|
/// <summary>
|
|
/// The version files of the workspace that cannot be read now (damaged, held open, over the size a version may be, of
|
|
/// a newer Chronos), by the number and id their names carry, oldest first.
|
|
/// </summary>
|
|
public IReadOnlyList<UnreadableVersion> UnreadableVersions() => ReadVersions(log: false).Unreadable;
|
|
|
|
/// <summary>
|
|
/// <see cref="Versions"/> and <see cref="UnreadableVersions"/> from one read of the folder: each file (up to
|
|
/// <see cref="ProjectSidecars.MaxVersionBytes"/>) is parsed once.
|
|
/// </summary>
|
|
internal (IReadOnlyList<ProjectVersion> Versions, IReadOnlyList<UnreadableVersion> Unreadable) ReadVersionFiles() => ReadVersions(log: true);
|
|
|
|
private (List<ProjectVersion> Versions, List<UnreadableVersion> Unreadable) ReadVersions(bool log)
|
|
{
|
|
var versions = new List<ProjectVersion>();
|
|
var unreadable = new List<UnreadableVersion>();
|
|
foreach (var (number, id, path) in ProjectSidecars.ListVersions(VersionsDirectory))
|
|
{
|
|
try
|
|
{
|
|
var version = ProjectSidecars.ReadVersion(path);
|
|
if (version.ProjectId == ProjectId) versions.Add(version);
|
|
}
|
|
catch (Exception ex) when (ex is IOException or InvalidDataException or UnauthorizedAccessException)
|
|
{
|
|
// Said once per file as it is now: every sync step lists the versions again.
|
|
if (log && (!Reported.TryGetValue(path, out var said) || said != ex.Message))
|
|
{
|
|
Reported[path] = ex.Message;
|
|
Log.Warning("workspace", $"Version file {path} cannot be read", ex);
|
|
}
|
|
unreadable.Add(new UnreadableVersion(number, id, path, ex.Message));
|
|
}
|
|
}
|
|
return (versions, unreadable);
|
|
}
|
|
|
|
/// <summary>The version files whose read failure was logged, with why (logged again when the reason changes).</summary>
|
|
private static readonly System.Collections.Concurrent.ConcurrentDictionary<string, string> Reported = new(StringComparer.Ordinal);
|
|
|
|
/// <summary>The path of a version in <see cref="VersionsDirectory"/>.</summary>
|
|
public string VersionPath(ProjectVersion version) => Path.Combine(VersionsDirectory, ArchiveLayout.ProjectVersionFileName(version.Number, version.Id));
|
|
|
|
internal void WriteVersion(ProjectVersion version, bool replace = false) => ProjectSidecars.WriteVersion(VersionPath(version), version, replace);
|
|
|
|
/// <summary>
|
|
/// Moves a version file aside into <c>versions/.replaced</c> (a local commit replayed under another id), trying again
|
|
/// for a moment while another program (an indexer, an antivirus) holds it open.
|
|
/// </summary>
|
|
internal void SetAside(ProjectVersion version) => SetAside(VersionPath(version));
|
|
|
|
/// <summary>Moves the version file at <paramref name="path"/> aside into <c>versions/.replaced</c> (see <see cref="SetAside(ProjectVersion)"/>).</summary>
|
|
internal void SetAside(string path)
|
|
{
|
|
if (!File.Exists(path)) return;
|
|
var aside = Path.Combine(VersionsDirectory, ReplacedFolder);
|
|
Directory.CreateDirectory(aside);
|
|
ReplacedSince(DateTime.UtcNow);
|
|
var target = Path.Combine(aside, Path.GetFileName(path));
|
|
if (File.Exists(target)) target += "." + DateTime.UtcNow.Ticks.ToString(System.Globalization.CultureInfo.InvariantCulture);
|
|
for (var attempt = 1; ; attempt++)
|
|
{
|
|
try
|
|
{
|
|
File.Move(path, target);
|
|
break;
|
|
}
|
|
catch (IOException) when (attempt < SetAsideAttempts && File.Exists(path) && !File.Exists(target))
|
|
{
|
|
Thread.Sleep(10 * attempt);
|
|
}
|
|
}
|
|
// Dated as set aside, which the pruning goes by (a move keeps the time the version was written).
|
|
TryTouch(target);
|
|
}
|
|
|
|
/// <summary>
|
|
/// Writes a version straight into <c>versions/.replaced</c>: kept, never part of the chain of versions to send (the
|
|
/// working state a take of the server's values replaced).
|
|
/// </summary>
|
|
internal void WriteAside(ProjectVersion version)
|
|
{
|
|
ArgumentNullException.ThrowIfNull(version);
|
|
ProjectSidecars.WriteVersion(Path.Combine(VersionsDirectory, ReplacedFolder, Path.GetFileName(VersionPath(version))), version);
|
|
ReplacedSince(DateTime.UtcNow);
|
|
}
|
|
|
|
/// <summary>Files <see cref="PruneReplaced"/> keeps in <c>versions/.replaced</c> whatever their age.</summary>
|
|
public const int ReplacedKept = 20;
|
|
|
|
/// <summary>The age past which <see cref="PruneReplaced"/> removes a file of <c>versions/.replaced</c>.</summary>
|
|
public static readonly TimeSpan ReplacedMaxAge = TimeSpan.FromDays(30);
|
|
|
|
/// <summary>
|
|
/// Removes from <c>versions/.replaced</c> the files set aside more than <see cref="ReplacedMaxAge"/> ago and those past
|
|
/// the newest <see cref="ReplacedKept"/>, never one of a version in <paramref name="referenced"/> (a journal's). Every
|
|
/// file there is a copy of what the chain or the home holds, or what a take of the server's values replaced (its undo):
|
|
/// called only after a sync that ended well. A file is set aside when this Chronos dated it (it touches what it sets
|
|
/// aside), and no earlier than this Chronos first knew the folder (<see cref="ReplacedSinceFile"/>): one an earlier build
|
|
/// set aside keeps the time its version was written, which says nothing of when it went aside. How many went.
|
|
/// </summary>
|
|
internal int PruneReplaced(IReadOnlySet<Guid> referenced, DateTime nowUtc)
|
|
{
|
|
ArgumentNullException.ThrowIfNull(referenced);
|
|
var aside = Path.Combine(VersionsDirectory, ReplacedFolder);
|
|
if (!Directory.Exists(aside)) return 0;
|
|
var since = ReplacedSince(nowUtc);
|
|
var files = new DirectoryInfo(aside).EnumerateFiles()
|
|
.Select(f => (File: f, Aside: f.LastWriteTimeUtc > since ? f.LastWriteTimeUtc : since))
|
|
.OrderByDescending(f => f.Aside).ThenByDescending(f => f.File.LastWriteTimeUtc).ThenByDescending(f => f.File.Name, StringComparer.Ordinal).ToList();
|
|
var removed = 0;
|
|
for (var i = 0; i < files.Count; i++)
|
|
{
|
|
var (file, set) = files[i];
|
|
if (i < ReplacedKept && nowUtc - set <= ReplacedMaxAge) continue;
|
|
if (VersionIdOf(file.Name) is { } id && referenced.Contains(id)) continue;
|
|
try
|
|
{
|
|
file.Delete();
|
|
removed++;
|
|
}
|
|
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
|
|
{
|
|
Log.Warning("workspace", $"{file.FullName} could not be removed", ex);
|
|
}
|
|
}
|
|
return removed;
|
|
|
|
// "<number>-<id>.json", maybe with a suffix a second file of the same version took.
|
|
static Guid? VersionIdOf(string name)
|
|
{
|
|
var end = name.IndexOf(".json", StringComparison.OrdinalIgnoreCase);
|
|
return end > 0 && ProjectSidecars.TryParseName(name[..(end + 5)], out _, out var id) ? id : null;
|
|
}
|
|
}
|
|
|
|
/// <summary>
|
|
/// When this Chronos first knew <c>versions/.replaced</c> (<see cref="ReplacedSinceFile"/>): read, else written as
|
|
/// <paramref name="nowUtc"/> (also over a file that cannot be read). One that cannot be written counts as now: nothing
|
|
/// set aside by an earlier build goes by its age then.
|
|
/// </summary>
|
|
private DateTime ReplacedSince(DateTime nowUtc)
|
|
{
|
|
var path = Path.Combine(VersionsDirectory, ReplacedSinceFile);
|
|
try
|
|
{
|
|
if (File.Exists(path)
|
|
&& DateTime.TryParse(File.ReadAllText(path).Trim(), System.Globalization.CultureInfo.InvariantCulture, System.Globalization.DateTimeStyles.RoundtripKind,
|
|
out var known)
|
|
&& known.Kind == DateTimeKind.Utc && known <= nowUtc)
|
|
return known;
|
|
Directory.CreateDirectory(VersionsDirectory);
|
|
AtomicFile.WriteAllText(path, nowUtc.ToString("O", System.Globalization.CultureInfo.InvariantCulture));
|
|
}
|
|
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
|
|
{
|
|
Log.Debug("workspace", $"{path} could not be kept: {ex.Message}");
|
|
}
|
|
return nowUtc;
|
|
}
|
|
|
|
private static void TryTouch(string path)
|
|
{
|
|
try
|
|
{
|
|
if (File.Exists(path)) File.SetLastWriteTimeUtc(path, DateTime.UtcNow);
|
|
}
|
|
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
|
|
{
|
|
Log.Debug("workspace", $"{path} keeps the time it was written: {ex.Message}");
|
|
}
|
|
}
|
|
|
|
/// <summary>Moves tried, some 1 s in all, before a version file held open fails to move aside.</summary>
|
|
private const int SetAsideAttempts = 15;
|
|
|
|
public WorkspaceSyncState ReadSync()
|
|
{
|
|
if (!File.Exists(SyncPath)) return WorkspaceSyncState.None;
|
|
try
|
|
{
|
|
return JsonSerializer.Deserialize(AtomicFile.ReadAllText(SyncPath), WorkspaceJson.Default.WorkspaceSyncState) ?? WorkspaceSyncState.None;
|
|
}
|
|
catch (JsonException ex)
|
|
{
|
|
throw new InvalidDataException($"{SyncPath} cannot be read.", ex);
|
|
}
|
|
}
|
|
|
|
/// <summary>
|
|
/// The conflict <c>sync.json</c> records that still waits for a choice, as <see cref="WorkspaceSync.StatusAsync"/> reads
|
|
/// it: one a replay that settles it took past its commit point (its base moved onto the replay's) is over, the choice
|
|
/// made applied, even before the next sync clears it.
|
|
/// </summary>
|
|
/// <exception cref="InvalidDataException"><c>sync.json</c> or the working state cannot be read.</exception>
|
|
/// <exception cref="IOException">A file cannot be read.</exception>
|
|
public WorkspaceConflict? OpenConflict() => ReadSync() is { Conflict: not null } sync ? Unsettled(sync, ReadBundle()) : null;
|
|
|
|
/// <summary>The conflict of <paramref name="sync"/> that waits for a choice with the working state <paramref name="bundle"/> (<see cref="OpenConflict"/>).</summary>
|
|
internal static WorkspaceConflict? Unsettled(WorkspaceSyncState sync, ProjectBundle bundle) =>
|
|
sync.Replay is { Settles: true } replay && bundle.Workspace?.Base?.VersionId == replay.OntoBase ? null : sync.Conflict;
|
|
|
|
internal void WriteSync(WorkspaceSyncState state)
|
|
{
|
|
ArgumentNullException.ThrowIfNull(state);
|
|
if (state.IsEmpty)
|
|
{
|
|
if (File.Exists(SyncPath)) File.Delete(SyncPath);
|
|
return;
|
|
}
|
|
Directory.CreateDirectory(Folder);
|
|
AtomicFile.WriteAllText(SyncPath, JsonSerializer.Serialize(state, WorkspaceJson.Default.WorkspaceSyncState));
|
|
}
|
|
}
|