Add shopify validate, the unified Shopify skill, and per-topic agent files

[craigmichaelmartin]

What

Prototypes the CLI side of the AI Toolkit consolidation: instead of ~20 per-topic skills that each bundle their own search/validate scripts, agents get one umbrella skill backed by first-class CLI validators and per-topic reference files. Three pieces:

1. shopify validate — a new top-level command family for offline, deterministic validation of Shopify code against bundled schemas and rules.
2. The shopify umbrella skill (.agents/skills/shopify/SKILL.md) — a single validate-first skill replacing the per-topic skills in the AI Toolkit.
3. 19 topic agent files (.agents/skills/shopify/topics/*.md) — dense, doc-grounded primers the skill has agents read before doing any work on a topic.

The skill and topic files will ultimately be served from shopify.dev (shopify.dev/skill.md and adjacent paths); they live in this repo while we prototype.

1) shopify validate

Ports the four validation scripts from ai-toolkit-source's shopify-dev-tools package into CLI subcommands:

Subcommand	Validates	Engine
shopify validate theme	Liquid/theme code (full-app + stateless codeblock modes)	@shopify/theme-check-* (already shipped by the CLI)
shopify validate graphql	GraphQL operations against bundled Admin/Storefront/Customer/Partner/Payments schemas; reports required offline access scopes	graphql + bundled introspection JSON
shopify validate functions	Shopify Functions input queries, per Function kind (--api functions_*)	shared GraphQL engine
shopify validate components	UI-extension component code (TSX/JSX/HTML) for polaris-app-home, polaris-admin-extensions, polaris-checkout-extensions, polaris-customer-account-extensions, pos-ui, hydrogen	virtual TypeScript environment + bundled @shopify/ui-extensions type defs

All four sit on one shared engine under packages/cli/src/cli/services/validate/engine/:

• A dependency-free result contract (ValidationResult const-union, ValidationResponse) and a barrel-decoupled formatter — the non-component validators never pull in the TypeScript compiler.
• Pure, injectable version resolution over a bundled version catalog.
• One lazy gunzip walk-up data-loader that resolves bundled assets in dev (tsx/vitest), the split esbuild bundle, and the published /assets alike.
• Pure-core + thin-orchestrator split per subcommand (command parses flags → service holds all logic → renders/records/exits).

Behavior is preserved from the source tools (flags, --json shape {success, responses, resolvedVersion?}, exit codes, stdin) while adapting to CLI conventions: cli-kit rendering, outputResult + AbortSilentError for --json, telemetry via addPublicMetadata (new cmd_validate_* fields). Agent-only plumbing (random-UUID artifact IDs, shopify.dev network instrumentation) is dropped for deterministic output. Bad --file/empty stdin return a structured FAILED payload instead of crashing.

Reference data (~13.7 MB compressed: GraphQL schemas + UI-extension .d.ts.gz) is bundled for fully offline validation and copied into dist/ via a recursive esbuild rule. typescript becomes a lazily-loaded, esbuild-external runtime dependency (used only by validate components).

2) The shopify umbrella skill

One skill spanning every surface (GraphQL APIs, custom data, Functions, Hydrogen, themes, Polaris/extension UI, onboarding, CLI operations, UCP, App Store review). Its required flow:

1. Read the topic agent file for the family the task falls into (mapped in the skill's table).
2. Do the work from the file plus the model's own knowledge — CLI generators over hand-written boilerplate.
3. Validate the outcome with the matching shopify validate topic — the mandatory ground-truth gate, applied to chat-reply examples (via temp file) and touched files alike. Prose-only replies have nothing to validate.
4. On validation failure — and only then, as a requirement — search the docs (shopify doc search, shopify doc fetch sparingly), fix, and re-validate (max 3 retries).

Search before validating is deliberately optional, never a prerequisite: a passing validation proves no search was needed, and a mid-write shopify doc search to pin an uncertain name is a judgment call. Outcomes with no validator (merchant onboarding, App Store review analysis, UCP flows) are the one case that requires an up-front search. The skill also encodes operational guardrails learned from eval runs: no API spelunking through node_modules//package stores, --help checked only for a command you're about to run, and exact --api/--api-name value enumerations matching this branch's commands.

3) Topic agent files

Each of the 19 topic skills being retired from the AI Toolkit is distilled into one ≤10,000-char file at .agents/skills/shopify/topics/<topic>.md (admin, storefront-graphql, customer, partner, payments-apps, custom-data, functions, hydrogen, liquid, polaris-app-home, polaris-admin-extensions, polaris-checkout-extensions, polaris-customer-account-extensions, pos-ui, onboarding-dev, onboarding-merchant, use-shopify-cli, ucp, app-store-review). shopify-dev is intentionally not ported — it is the generic doc-search skill, which is exactly the umbrella skill's fallback path.

• Content: a "Key facts" section (current API versions, deprecations and replacements, exact enum values, required scopes), a surface map of the most-used operations/components, config shapes, complete validated examples, gotchas, and a short docs list.
• Stateless by design: facts are written as timeless statements of current API state (version-anchored where the docs are), never as corrections relative to a model's training data or the present moment.
• Provenance: built from shopify.dev via shopify doc search/doc fetch, not model recall; every embedded example was proven with this branch's shopify validate where a surface exists; each file passed an adversarial verification pass (byte cap, example re-validation, fact spot-checks against the docs).

Test plan

• nx type-check cli ✅ · nx lint cli ✅
• nx vitest cli for the validate suites: 81/81 pass (real temp dirs, no fs mocking)
• nx bundle cli ✅ — bundled binary smoke-tested for all four subcommands:
  • validate graphql --api admin --code 'query { shop { name } }' → success, exit 0
  • validate graphql with an invalid field → failed, exit 1 (Cannot query field "nope" on type "Shop")
  • validate functions --api functions_discount (valid input query) → success
  • validate theme (codeblock) → success
  • validate components --api polaris-app-home --code '<s-text>hello</s-text>' --language html → success, validatedComponents: ["s-text"]
• Topic files: all 19 verified ≤10,000 bytes; embedded examples validated against the dev CLI; per-file adversarial fact-check pass came back clean.
• Regenerated oclif.manifest.json, README.md, and dev-docs; changeset added for the command (skill/topic files are agent-facing and not changelogged).

Notes / reviewer callouts

• Package size: adds ~13.7 MB of compressed schema/type data (Admin schema alone ~4.8 MB). This is the faithful-offline tradeoff; the loader decompresses lazily per requested version, so it is on-disk weight only. Happy to trim historical Admin versions if preferred.
• New runtime dep: typescript (lazy, esbuild-external) for validate components.
• A scoped packages/cli/assets/validate/.gitignore re-includes the vendored type packages' dist/ folders (otherwise the repo-wide dist ignore would drop ~320 .d.ts.gz reference files).
• The skill references topic files as topics/<slug>.md relative to SKILL.md, so serving them next to skill.md on shopify.dev keeps the references working as-is.

🤖 Generated with Claude Code

[binks-code-reviewer]

🤖 Code Review · Skipped — PR has too many files for review (limit: 1000)

[craigmichaelmartin]

/snapit

[github-actions]

🫰✨ Thanks @craigmichaelmartin! Your snapshot has been published to npm.

Test the snapshot by installing your package globally:

pnpm i -g --@shopify:registry=https://registry.npmjs.org @shopify/cli@0.0.0-snapshot-20260721185120

Caution

After installing, validate the version by running shopify version in your terminal.
If the versions don't match, you might have multiple global instances installed.
Use which shopify to find out which one you are running and uninstall it.

[alfonso-noriega]

• Announce background skill updates on the next CLI run #8166
• Keep the installed Shopify skill up to date daily #8164
• Auto-install the Shopify skill in the background on CLI runs #8160
• Add Shopify skill install command #8140
• Add shopify validate, the unified Shopify skill, and per-topic agent files #8142 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[craigmichaelmartin]

/snapit

[github-actions]

🫰✨ Thanks @craigmichaelmartin! Your snapshot has been published to npm.

Test the snapshot by installing your package globally:

pnpm i -g --@shopify:registry=https://registry.npmjs.org @shopify/cli@0.0.0-snapshot-20260723175937

Caution

After installing, validate the version by running shopify version in your terminal.
If the versions don't match, you might have multiple global instances installed.
Use which shopify to find out which one you are running and uninstall it.

[github-actions]

Differences in type declarations

We detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:

• Some seemingly private modules might be re-exported through public modules.
• If the branch is behind main you might see odd diffs, rebase main into this branch.

New type declarations

We found no new type declarations in this PR

Existing type declarations

packages/cli-kit/dist/public/node/metadata.d.ts

@@ -34,7 +34,7 @@ export type SensitiveSchema<T> = T extends RuntimeMetadataManager<infer _TPublic
  * @returns A container for the metadata.
  */
 export declare function createRuntimeMetadataContainer<TPublic extends AnyJson, TSensitive extends AnyJson = Record<string, never>>(defaultPublicMetadata?: Partial<TPublic>): RuntimeMetadataManager<TPublic, TSensitive>;
-type CmdFieldsFromMonorail = PickByPrefix<MonorailEventPublic, 'cmd_all_'> & PickByPrefix<MonorailEventPublic, 'cmd_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_create_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_theme_'> & PickByPrefix<MonorailEventPublic, 'store_'> & PickByPrefix<MonorailEventPublic, 'env_auto_upgrade_'>;
+type CmdFieldsFromMonorail = PickByPrefix<MonorailEventPublic, 'cmd_all_'> & PickByPrefix<MonorailEventPublic, 'cmd_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_create_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_theme_'> & PickByPrefix<MonorailEventPublic, 'cmd_validate_'> & PickByPrefix<MonorailEventPublic, 'store_'> & PickByPrefix<MonorailEventPublic, 'env_auto_upgrade_'>;
 declare const coreData: RuntimeMetadataManager<CmdFieldsFromMonorail, {
     commandStartOptions: {
         startTime: number;

packages/cli-kit/dist/public/node/monorail.d.ts

@@ -77,6 +77,11 @@ export interface Schemas {
             cmd_app_validate_valid?: Optional<boolean>;
             cmd_app_validate_issue_count?: Optional<number>;
             cmd_app_validate_file_count?: Optional<number>;
+            cmd_validate_subcommand?: Optional<string>;
+            cmd_validate_result?: Optional<string>;
+            cmd_validate_api?: Optional<string>;
+            cmd_validate_api_version?: Optional<string>;
+            cmd_validate_json?: Optional<boolean>;
             cmd_dev_tunnel_type?: Optional<string>;
             cmd_dev_tunnel_custom_hash?: Optional<string>;
             cmd_dev_urls_updated?: Optional<boolean>;