Skip to content

[#99] Add --build-history option to analyze #109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SkowronskiAndrew merged 4 commits into from
Jul 23, 2026
Merged

[#99] Add --build-history option to analyze #109

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
137933a
Initial implementation
SkowronskiAndrew Jul 22, 2026
f933273
Improvements based on review
SkowronskiAndrew Jul 22, 2026
ccfc50a
establish BuildReportSummary.cs in UnityDataModels
SkowronskiAndrew Jul 22, 2026
5ed3eec
[#99] Normalize BuildManifestHash casing when reading it for matching
SkowronskiAndrew Jul 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 54 additions & 51 deletions Analyzer/AnalyzerTool.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
using System.Diagnostics;
using System.IO;
using System.Linq;
using Newtonsoft.Json;
using UnityDataTools.Analyzer.SQLite.Handlers;
using UnityDataTools.Analyzer.SQLite.Parsers;
using UnityDataTools.Analyzer.SQLite.Writers;
Expand Down Expand Up @@ -46,6 +45,9 @@ public class AnalyzeOptions
public bool SkipCrc { get; init; }
public bool Verbose { get; init; }
public bool NoRecursion { get; init; }
// Build history folder (e.g. Library/BuildHistory); when set, the folder of the analyzed
// build is located in it and its ContentLayout.json and build report join the input.
public string BuildHistoryPath { get; init; }
}

public int Analyze(AnalyzeOptions options)
Expand Down Expand Up @@ -183,44 +185,21 @@ public int Analyze(AnalyzeOptions options)
}

// Validates the ContentDirectory-related inputs and prepares the file list (issue #99):
// enforces that a single build is analyzed, selects the ContentLayout.json whose
// enforces that a single build is analyzed, adds the files of the matching build history
// folder when --build-history is used, selects the ContentLayout.json whose
// BuildManifestHash matches that build (dropping any others), and moves it to the front of
// the list so it is imported before the content files whose references it resolves. Returns
// false, after printing an error, when the input combination is invalid.
bool PrepareContentDirectoryInputs(List<(string FullPath, string DisplayRoot)> files)
{
const string hashFileName = "BuildManifestHash.txt";

var layoutCandidates = files.Where(f => ContentLayoutParser.IsContentLayoutFile(f.FullPath)).ToList();
var hashFiles = files
.Where(f => string.Equals(Path.GetFileName(f.FullPath), hashFileName, StringComparison.OrdinalIgnoreCase))
.Select(f => f.FullPath)
.ToList();

// ContentDirectory output is recognized by its .cf content files, or by its archive when
// built compressed. The BuildManifestHash.txt identifying the build sits next to them; it
// is not always on the input (e.g. when specific files are passed), so pick it up from
// the directories containing the content.
var contentDirectories = files
.Where(f => HasExtension(f.FullPath, ".cf") || HasExtension(f.FullPath, ".archive"))
.Select(f => Path.GetDirectoryName(Path.GetFullPath(f.FullPath)))
.Distinct(StringComparer.OrdinalIgnoreCase);

if (hashFiles.Count == 0)
{
hashFiles.AddRange(contentDirectories
.Select(dir => Path.Combine(dir, hashFileName))
.Where(File.Exists));
}

List<string> buildHashes;
try
{
buildHashes = hashFiles.Select(f => File.ReadAllText(f).Trim()).Distinct().ToList();
buildHashes = BuildHistoryHelper.FindBuildHashes(files.Select(f => f.FullPath));
}
catch (Exception e)
{
Console.Error.WriteLine($"Error reading {hashFileName}: {e.Message}");
Console.Error.WriteLine($"Error reading {BuildHistoryHelper.HashFileName}: {e.Message}");
return false;
}

Expand All @@ -231,7 +210,14 @@ bool PrepareContentDirectoryInputs(List<(string FullPath, string DisplayRoot)> f
}

var buildHash = buildHashes.Count == 1 ? buildHashes[0] : null;
var hasContentDirectory = buildHash != null || files.Any(f => HasExtension(f.FullPath, ".cf"));

if (m_Options.BuildHistoryPath != null && !AddBuildHistoryFiles(buildHash, files))
{
return false;
}

var layoutCandidates = files.Where(f => ContentLayoutParser.IsContentLayoutFile(f.FullPath)).ToList();
var hasContentDirectory = buildHash != null || BuildHistoryHelper.HasContentFiles(files.Select(f => f.FullPath));

if (layoutCandidates.Count == 0)
{
Expand Down Expand Up @@ -259,7 +245,7 @@ bool PrepareContentDirectoryInputs(List<(string FullPath, string DisplayRoot)> f

// The hash match guarantees the layout describes exactly this build; a stale or
// unrelated layout would silently produce misleading results.
selected = layoutCandidates.FirstOrDefault(c => TryReadBuildManifestHash(c.FullPath) == buildHash);
selected = layoutCandidates.FirstOrDefault(c => BuildHistoryHelper.TryReadBuildManifestHash(c.FullPath) == buildHash);

Comment thread
SkowronskiAndrew marked this conversation as resolved.
if (selected.FullPath == null)
{
Expand All @@ -282,7 +268,10 @@ bool PrepareContentDirectoryInputs(List<(string FullPath, string DisplayRoot)> f
{
if (candidate != selected)
{
Console.Error.WriteLine($"Ignoring \"{candidate.FullPath}\": its BuildManifestHash does not match the analyzed build.");
var reason = BuildHistoryHelper.TryReadBuildManifestHash(candidate.FullPath) == buildHash
? "it duplicates the selected layout"
: "its BuildManifestHash does not match the analyzed build";
Console.Error.WriteLine($"Ignoring \"{candidate.FullPath}\": {reason}.");
Comment thread
SkowronskiAndrew marked this conversation as resolved.
files.Remove(candidate);
}
}
Expand All @@ -295,34 +284,48 @@ bool PrepareContentDirectoryInputs(List<(string FullPath, string DisplayRoot)> f
return true;
}

static bool HasExtension(string path, string extension)
// --build-history: locate the analyzed build's folder inside the build history and add its
// ContentLayout.json and build report to the input. Purely additive — the added layout goes
// through the same candidate validation as a manually passed one.
bool AddBuildHistoryFiles(string buildHash, List<(string FullPath, string DisplayRoot)> files)
{
return string.Equals(Path.GetExtension(path), extension, StringComparison.OrdinalIgnoreCase);
}
var historyPath = m_Options.BuildHistoryPath;

// Reads the top-level BuildManifestHash of a ContentLayout.json without parsing the whole
// file (layouts of large builds are big; the hash is one of the first properties). Returns
// null when the value cannot be found or the file is not valid json.
static string TryReadBuildManifestHash(string path)
{
try
if (!Directory.Exists(historyPath))
{
using var reader = new JsonTextReader(File.OpenText(path));
Console.Error.WriteLine($"--build-history path not found: {historyPath}");
return false;
}

for (int i = 0; i < 64 && reader.Read(); ++i)
{
if (reader.TokenType == JsonToken.PropertyName && reader.Depth == 1 &&
"BuildManifestHash".Equals(reader.Value))
{
return reader.ReadAsString();
}
}
if (buildHash == null)
{
Console.Error.WriteLine("--build-history requires a ContentDirectory build in the input: no BuildManifestHash.txt was found to identify the build to match.");
return false;
}
catch (Exception)

var buildFolder = BuildHistoryHelper.LocateBuildFolder(historyPath, buildHash);
if (buildFolder == null)
{
Console.Error.WriteLine($"No build in \"{historyPath}\" matches the analyzed build (BuildManifestHash {buildHash}).");
return false;
}

return null;
Console.WriteLine($"Using build history folder \"{buildFolder}\".");

// Skip files that are already on the input (e.g. the layout was also passed explicitly).
var newEntries = new List<(string FullPath, string DisplayRoot)>();
foreach (var file in BuildHistoryHelper.CollectBuildFiles(buildFolder))
{
var alreadyIncluded = files.Any(f => string.Equals(
Path.GetFullPath(f.FullPath), Path.GetFullPath(file), StringComparison.OrdinalIgnoreCase));

if (!alreadyIncluded)
newEntries.Add((file, buildFolder));
}

files.InsertRange(0, newEntries);

return true;
}

// Expands the input paths into the concrete files to analyze. Each result pairs the file with the
Expand Down
138 changes: 138 additions & 0 deletions Analyzer/Util/BuildHistoryHelper.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using Newtonsoft.Json;
using UnityDataTools.Models;

namespace UnityDataTools.Analyzer.Util;

// Identifies the ContentDirectory build present in the analyze input and locates its folder
// inside a build history (Library/BuildHistory, one folder per build). The folder of a build is
// found by matching the BuildManifestHash of its ContentLayout.json against the build's
// BuildManifestHash.txt. Pure file lookup; no knowledge of the parsers or the database.
public static class BuildHistoryHelper
{
public const string HashFileName = "BuildManifestHash.txt";
public const string LayoutFileName = "ContentLayout.json";
public const string SummaryFileName = "BuildReportSummary.json";

// Reads the BuildManifestHash.txt values identifying the ContentDirectory build(s) in the
// input files: taken from the input itself when present, otherwise picked up next to the
// .cf/.archive content files (the hash file is not always on the input, e.g. when specific
// files are passed). More than one distinct hash means more than one build.
public static List<string> FindBuildHashes(IEnumerable<string> files)
{
var fileList = files.ToList();

var hashFiles = fileList
.Where(f => string.Equals(Path.GetFileName(f), HashFileName, StringComparison.OrdinalIgnoreCase))
.ToList();

if (hashFiles.Count == 0)
{
var contentDirectories = fileList
.Where(f => HasExtension(f, ".cf") || HasExtension(f, ".archive"))
.Select(f => Path.GetDirectoryName(Path.GetFullPath(f)))
.Distinct(StringComparer.OrdinalIgnoreCase);

hashFiles.AddRange(contentDirectories
.Select(dir => Path.Combine(dir, HashFileName))
.Where(File.Exists));
}

// Hashes are lowercase hex; normalizing here (and in TryReadBuildManifestHash) keeps all
// hash comparisons simple, exact matches.
return hashFiles.Select(f => File.ReadAllText(f).Trim().ToLowerInvariant()).Distinct().ToList();
}

// Fallback signal used when no BuildManifestHash.txt is available: a .cf file marks
// ContentDirectory output. This is only a rough sanity check — the reverse does not hold, as
// content files can be built without the extension or sit inside archives.
public static bool HasContentFiles(IEnumerable<string> files)
{
return files.Any(f => HasExtension(f, ".cf"));
}

// Searches the history root and its direct child folders for the ContentLayout.json whose
// BuildManifestHash matches the analyzed build. A build history is flat, so there is no
// recursion — which also keeps the scan bounded if a large unrelated folder is passed. When
// several folders match (rebuilds of identical content), the most recent build wins.
// Returns the containing folder, or null when nothing matches.
public static string LocateBuildFolder(string historyRoot, string buildHash)
{
var matches = Directory.EnumerateDirectories(historyRoot)
.Prepend(historyRoot)
.Where(dir => File.Exists(Path.Combine(dir, LayoutFileName)))
.Where(dir => TryReadBuildManifestHash(Path.Combine(dir, LayoutFileName)) == buildHash);

string selected = null;
var selectedTime = default(DateTime);
Comment thread
SkowronskiAndrew marked this conversation as resolved.
foreach (var folder in matches)
{
var startTime = TryReadBuildStartTime(folder);
if (selected == null || startTime > selectedTime)
{
selected = folder;
selectedTime = startTime;
}
}

return selected;
}

// Reads the start time of the build from the BuildReportSummary.json of a build history
// folder. The file is expected to always be present; returns a zero timestamp when it is
// missing or unreadable.
public static DateTime TryReadBuildStartTime(string buildFolder)
{
try
{
var json = File.ReadAllText(Path.Combine(buildFolder, SummaryFileName));
var summary = JsonConvert.DeserializeObject<BuildReportSummary>(json);
return summary?.BuildStartedAt ?? default;
}
catch (Exception)
{
return default;
}
}

// The files of a build history folder that analyze imports: the layout and the build report.
public static List<string> CollectBuildFiles(string buildFolder)
{
var files = new List<string> { Path.Combine(buildFolder, LayoutFileName) };
files.AddRange(Directory.EnumerateFiles(buildFolder, "*.buildreport"));
return files;
}

// Reads the top-level BuildManifestHash of a ContentLayout.json without parsing the whole
// file (layouts of large builds are big; the hash is one of the first properties). Returns
// null when the value cannot be found or the file is not valid json.
public static string TryReadBuildManifestHash(string contentLayoutPath)
{
try
{
using var reader = new JsonTextReader(File.OpenText(contentLayoutPath));

for (int i = 0; i < 64 && reader.Read(); ++i)
{
if (reader.TokenType == JsonToken.PropertyName && reader.Depth == 1 &&
"BuildManifestHash".Equals(reader.Value))
{
return reader.ReadAsString()?.ToLowerInvariant();
}
}
}
catch (Exception)
{
}

return null;
}

static bool HasExtension(string path, string extension)
{
return string.Equals(Path.GetExtension(path), extension, StringComparison.OrdinalIgnoreCase);
}
}
26 changes: 23 additions & 3 deletions Documentation/command-analyze.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ UnityDataTool analyze <paths>... [options]
| `-v, --verbose` | Show more information during analysis | `false` |
| `--no-recurse` | Do not recurse into sub-directories when scanning directories | `false` |
| `-d, --typetree-data <file>` | Load an external TypeTree data file before processing (Unity 6.5+) | — |
| `--build-history <folder>` | Build history folder of the project (e.g. `Library/BuildHistory`). The build folder matching the analyzed build is located automatically and its `ContentLayout.json` and build report are included in the analysis. See [ContentDirectory builds](#contentdirectory-builds) | — |

There is no way to append to an existing database, so every file you want in the results must be
included in a single `analyze` invocation. Pass multiple paths to combine files from more than one
Expand All @@ -40,6 +41,12 @@ Combine a build output directory with a build report file kept in a separate loc
UnityDataTool analyze /path/to/build/output /path/to/Library/LastBuild.buildreport
```

Analyze a ContentDirectory build together with its build history (the matching layout and build
report are located automatically):
```bash
UnityDataTool analyze /path/to/ContentDirectory --build-history /path/to/Library/BuildHistory
```

Analyze only `.bundle` files and specify a custom database name:
```bash
UnityDataTool analyze /path/to/asset/bundles -o my_database.db -p "*.bundle"
Expand Down Expand Up @@ -79,7 +86,16 @@ command works with the following types of input:
## ContentDirectory builds

Analyze a ContentDirectory build together with its `ContentLayout.json` by passing the build output
folder and the build report folder (or the layout file itself) in one call:
folder and the project's build history with `--build-history`:

```bash
UnityDataTool analyze /path/to/ContentDirectory --build-history /path/to/Library/BuildHistory -o database.db
```

Analyze locates the build's own folder inside the build history and includes its
`ContentLayout.json` and `.buildreport` in the analysis, so there is no need to identify the
correct build folder by hand. Alternatively, pass the build folder (or the layout file itself) as a
regular input:

```bash
UnityDataTool analyze /path/to/ContentDirectory /path/to/Library/BuildHistory/<build-directory> -o database.db
Expand All @@ -94,8 +110,10 @@ resolved and are recorded in `dangling_refs`.
A `ContentLayout.json` is only used when its `BuildManifestHash` matches the analyzed build's
`BuildManifestHash.txt` (a file guaranteed to exist in the build output folder, which analyze picks
up automatically). This exact-match rule prevents a stale or unrelated layout from silently
producing misleading results, and it means you can point analyze at a folder containing several
layouts (such as `Library/BuildHistory`) and the matching one is selected.
producing misleading results. It is also how `--build-history` finds the build's folder: the folder
whose `ContentLayout.json` matches the hash is used (the most recent one, if the same content was
built several times). `--build-history` does not search recursively — the build folders must be
direct children of the given path (or the path may be a build folder itself).

How the input combinations behave:

Expand All @@ -104,6 +122,8 @@ How the input combinations behave:
| ContentDirectory only | Analyzed, with a warning that the analysis is incomplete (broken references, no source mapping) |
| Subset of files from a ContentDirectory | Same as above; recognized by the `.cf` extension |
| ContentDirectory + matching ContentLayout.json | Analyzed with resolved references and source mapping |
| ContentDirectory + `--build-history` | The build's folder is located in the history; its layout and build report are analyzed too. Error when no folder matches |
| `--build-history` without a ContentDirectory build in the input | Error — only ContentDirectory builds can be matched against a build history |
| Subset of ContentDirectory files + ContentLayout.json | Works when a `BuildManifestHash.txt` can be found for the hash match; error when the layout cannot be validated |
| ContentDirectory + multiple ContentLayout.json | The one matching the build is used, the rest are ignored; error when none match |
| ContentLayout.json only | Imported on its own (layout tables only) |
Expand Down
Loading
Loading