Skip to content

test(e2e): finalize settings suite stabilization#5885

Open
BillLeoutsakosvl346 wants to merge 2 commits into
e2e/settings-playwrightfrom
e2e/07-stabilization
Open

test(e2e): finalize settings suite stabilization#5885
BillLeoutsakosvl346 wants to merge 2 commits into
e2e/settings-playwrightfrom
e2e/07-stabilization

Conversation

@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor

Summary

  • complete the Step 7 objective-to-proof audit and acceptance dossier for all settings navigation, authorization, persona, and workflow contracts
  • harden the pre-0266 SSO migration audit with a real database rehearsal and rollout inputs for GitHub environments, Docker Compose, and Helm
  • make auth failure diagnostics fail closed, retain safe failure artifacts, and graduate Settings E2E to a required provider-neutral CI job
  • correct the free-organization admin fixture and model post-lapse workspace detachment from persisted state
  • make unscoped --repeat-each run every Playwright project sequentially on one stack, rather than repeating only the terminal dependency project
  • fix the docs chat tool's cross-Zod schema type failure while preserving local runtime validation

Verification

  • frozen install, format, lint, architecture/policy checks, migration safety, schema drift, type-check, Helm/Compose validation, and sandbox bundle integrity passed
  • monorepo tests passed, including 13,761 Sim tests, 50 DB tests, and the new docs schema tests
  • real pre/post-0266 SSO migration rehearsal passed
  • production Sim build passed
  • clean E2E cache miss: 256/256 passed; verified build stored
  • cache-hit repeat qualification: 512/512 passed across all six projects
  • definitive retry-free five-repeat gate: 1,280/1,280 passed against one guarded stack
  • leak scans completed and Stripe/MCP fakes recorded no unexpected requests

Manual follow-up after umbrella merge

  • exercise this required workflow on the later umbrella-to-staging PR before changing branch protection
  • perform the documented protected SSO quiescence/audit/backfill/TXT rollout before enabling migration 0266 and domain verification in staging or production

Co-authored-by: Cursor <cursoragent@cursor.com>
@BillLeoutsakosvl346
BillLeoutsakosvl346 requested a review from a team as a code owner July 23, 2026 08:16
@vercel

vercel Bot commented Jul 23, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview, Comment Jul 23, 2026 2:01pm

Request Review

@cursor

cursor Bot commented Jul 23, 2026

Copy link
Copy Markdown

PR Summary

High Risk
Production/staging migrations can fail closed on SSO preflight secrets and legacy data, and making Settings E2E blocking plus removing PR path filters increases CI blast radius and cost on every protected PR.

Overview
Settings E2E is now a required CI job (no continue-on-error), uses the same Blacksmith/GitHub runner switch as the app build, runs the full suite plus db:rehearse-sso-migration, and PR workflows no longer skip docs/markdown-only changes so the check context stays valid.

Migration 0266 (SSO hardening) gains a one-shot preflight in migrate.ts: advisory lock, SSO_PROVIDER_WRITES_QUIESCED, optional SSO_AUDIT_APPROVED_PROVIDER_IDS, and shared audit logic in sso-provider-audit.ts (also used by db:audit-sso-providers and a new db:rehearse-sso-migration rehearsal). Those env vars are wired through GitHub protected environments, Docker Compose prod, and Helm (migrations.ssoPreflight + chart tests). Enterprise SSO docs describe the operator rollout.

E2E harness fixes the lapsed-org persona (freeOrganizationAdmin, explicit grant, billing hidden), models post-lapse workspace detachment from persisted DB state in scenario validation/seed invariants, runs unscoped --repeat-each across all six Playwright projects on one stack, clears passwords before auth failure screenshots, and selectively disables network traces for people/SSO while keeping failure traces for other workflows. Adds STABILIZATION.md and expanded runbook/gate documentation.

Docs Ask AI replaces inline Zod on searchDocs with jsonSchema + validation tests (Vitest on the docs package).

Reviewed by Cursor Bugbot for commit 66eb9a5. Bugbot is set up for automated code reviews on this repo. Configure here.

@greptile-apps

greptile-apps Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Completes the Settings E2E stabilization work and hardens the SSO migration rollout.

  • Promotes the provider-neutral Settings E2E suite to a required CI job with safer diagnostics and repeat orchestration.
  • Adds fail-closed SSO migration auditing, rehearsal coverage, and rollout inputs for GitHub Actions, Docker Compose, and Helm.
  • Makes Helm’s one-shot SSO Secret keys optional at pod admission while preserving the migration runner’s pending-migration guard.
  • Extracts and tests the docs chat search-tool schema to resolve cross-Zod type incompatibility.
  • Updates settings contracts, personas, fixtures, workflows, and stabilization documentation.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains within the scope of the previous review thread.

The Helm fix admits pods when retired one-shot Secret keys are absent while the migration runner still fails closed whenever migration 0266 is pending without an explicit quiescence acknowledgement.

Important Files Changed

Filename Overview
helm/sim/templates/deployment-app.yaml Makes Secret-backed one-shot SSO migration inputs optional at pod admission while the migration runner continues to reject a pending migration without explicit quiescence.
packages/db/scripts/migrate.ts Adds the pending-0266 SSO audit gate before normal migration execution.
packages/db/sso-provider-audit.ts Implements provider-state validation and explicit approval handling for the SSO migration preflight.
.github/workflows/test-build.yml Makes the complete Settings E2E suite required and provider-neutral, and adds the SSO migration rehearsal.
apps/sim/e2e/scripts/run.ts Stabilizes full-project repeat execution and guarded single-stack orchestration.
apps/docs/lib/search-tool-schema.ts Replaces the route-local Zod tool schema with an AI SDK JSON schema and local validator.

Reviews (2): Last reviewed commit: "fix(helm): tolerate retired SSO prefligh..." | Re-trigger Greptile

Comment thread helm/sim/templates/deployment-app.yaml
Co-authored-by: Cursor <cursoragent@cursor.com>
@BillLeoutsakosvl346

Copy link
Copy Markdown
Contributor Author

@greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant