Plugin Directory: Add automated plugin review job by obenland · Pull Request #586 · WordPress/wordpress.org

obenland · 2026-04-02T14:59:20Z

Summary

Introduces a 3-phase automated review system (triage → batch → synthesis) that reviews plugin submissions against directory guidelines, security checklists, and code quality standards using the WordPress AI Client API
Bundled prompt references with review rules encoding experienced reviewer judgment, plus fallback to bundled guidelines when DevHub is unreachable
Admin UI button gated to review-eligible statuses with proper nonce/cap checks; integrates with existing Plugin Check (PCP) results for cross-referencing
Plugin source files wrapped in XML tags with randomized boundary tokens to mitigate prompt injection
39 unit tests covering verdict logic, fallback aggregation, batch building, file collection, triage heuristics, PCP integration, and output formatting

Test plan

Verify automated review triggers on plugin upload (cron job queued after 1 hour)
Verify "Run Automated Review" button appears only on draft/pending/new plugins when wp_supports_ai() is available
Verify AJAX handler rejects missing slug, invalid nonce, and unauthorized users
Verify review results appear in Internal Notes with correct verdict/findings format
Verify fallback to bundled guidelines when DevHub is unreachable
Verify verdict enforcement: blockers always produce "reject" regardless of AI response
Verify incomplete batch coverage (skipped/failed batches) prevents "approve"
Run unit tests: cd environments && npm run plugins:test

🤖 Generated with Claude Code

github-actions · 2026-04-02T14:59:35Z

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Core Committers: Use this line as a base for the props when committing in SVN:

Props obenland.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

Copilot

Pull request overview

Adds an automated (AI-assisted) review job to the Plugin Directory submission workflow, aiming to produce a structured verdict and findings for reviewer internal notes using a triage → batch → synthesis loop.

Changes:

Introduces a new cron/AJAX-driven job (Plugin_Automated_Review) that extracts the uploaded ZIP, batches source files, calls the WordPress AI Client API, and logs/stores results.
Wires the job into existing job infrastructure (colon-based cron hooks) and plugin upload flow, plus adds an admin metabox button to trigger reviews on-demand.
Adds bundled prompt/reference markdown files, including a guidelines fallback when DevHub content can’t be fetched.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
`jobs/class-plugin-automated-review.php`	Implements the multi-phase automated review loop, batching, AI calls, result storage, and audit logging.
`jobs/class-manager.php`	Registers the new colon-based cron task handler mapping and queues review on `plugin_upload`.
`jobs/automated-review/triage-prompt.md`	Triage system prompt template.
`jobs/automated-review/batch-prompt.md`	Batch review system prompt template + security/guideline rules.
`jobs/automated-review/synthesis-prompt.md`	Synthesis system prompt template for final report.
`jobs/automated-review/guidelines.md`	Bundled guidelines fallback when DevHub is unreachable.
`admin/metabox/class-review-tools.php`	Adds “Run Automated Review” button + AJAX call in the review metabox.
`admin/class-customizations.php`	Registers the new admin-ajax action for on-demand review runs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...s.org/public_html/wp-content/plugins/plugin-directory/jobs/class-plugin-automated-review.php

...ess.org/public_html/wp-content/plugins/plugin-directory/admin/metabox/class-review-tools.php

...s.org/public_html/wp-content/plugins/plugin-directory/jobs/class-plugin-automated-review.php

Introduces a 3-phase automated review system (triage → batch → synthesis) that reviews plugin submissions against directory guidelines, security checklists, and code quality standards using the WordPress AI Client API. - New `Plugin_Automated_Review` job class with cron and AJAX entry points - Bundled prompt references: security checklist, guidelines, 13 micro-prompts encoding experienced reviewer judgment rules - Integrates with existing Plugin Check (PCP) results for cross-referencing - Falls back to bundled guidelines when DevHub is unreachable - Enforces verdict consistency (blockers always mean reject) - Admin UI button gated to review-eligible statuses with proper nonce/cap checks - WPCS formatting cleanup in class-review-tools.php (short → long array syntax) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Pass slug args to wp_next_scheduled() so duplicate detection works - Check wp_supports_ai() in cron_trigger() and async AJAX path - Cast file_get_contents() returns to string to avoid TypeError with strict_types - Wrap AI client call in try/finally to ensure timeout filter is always removed Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...s.org/public_html/wp-content/plugins/plugin-directory/jobs/class-plugin-automated-review.php

...s.org/public_html/wp-content/plugins/plugin-directory/jobs/automated-review/triage-prompt.md

...s.org/public_html/wp-content/plugins/plugin-directory/jobs/class-plugin-automated-review.php

- Escape file paths in XML tag attributes with esc_attr() - Add untrusted-content warning to triage system prompt - Cast batch-prompt.md file_get_contents to string for strict_types safety Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Fix collect_files() to skip vendor/ and node_modules/ inside nested plugin directories (e.g., my-plugin/vendor/) not just at the root - Extract determine_verdict() as single source of truth for verdict logic, used by both the AI synthesis path and the fallback path - Remove unreachable verdict check (build_fallback_result always sets it) - Incomplete batch coverage now adds a warning finding so determine_verdict() naturally produces needs_changes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...s.org/public_html/wp-content/plugins/plugin-directory/jobs/class-plugin-automated-review.php

...rg/public_html/wp-content/plugins/plugin-directory/jobs/automated-review/synthesis-prompt.md

wordpress.org/public_html/wp-content/plugins/plugin-directory/admin/class-customizations.php

- Use end() for ZIP attachment selection instead of max(array_keys()) for more reliable "newest attachment" semantics - Include vendor/node_modules paths in all_files list (still excluded from source_files) so structure analysis can detect dev artifacts - Pass full file list to batch prompts for disallowed-file detection - Rewrite synthesis prompt to request JSON output matching the schema, fixing mismatch with as_json_response() that could force fallback path - Restore leading backslash on use import to match file convention Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...s.org/public_html/wp-content/plugins/plugin-directory/jobs/class-plugin-automated-review.php

...ss.org/public_html/wp-content/plugins/plugin-directory/jobs/automated-review/batch-prompt.md

…tches - Inject incomplete-coverage warning in run_review() before determine_verdict(), covering both AI synthesis and fallback paths - Remove duplicate incomplete-coverage logic from build_fallback_result() - Cast guidelines fallback file_get_contents to (string) - Fix batch prompt: replace per-guideline PASS/FAIL/WARN/N/A instruction with "only create findings for violations" to match JSON schema - Mark common-issues section as internal reference only to prevent fix suggestions from leaking into findings output Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...s.org/public_html/wp-content/plugins/plugin-directory/jobs/class-plugin-automated-review.php

...ess.org/public_html/wp-content/plugins/plugin-directory/admin/metabox/class-review-tools.php

39 tests covering the deterministic logic methods: - determine_verdict(): blockers→reject, warnings→needs_changes, clean→approve - build_fallback_result(): severity bucketing, field defaults, multi-batch aggregation, unknown severity routing - normalize_file_priorities(): array-of-objects and map form handling - build_default_triage(): extension-based priority, PCP error promotion - build_batches(): skip filtering, oversized isolation, priority sorting, batch size splitting - collect_files(): vendor/node_modules skipping, extension filtering - summarize_pcp_results(): error/warning counting, formatted output - format_as_note(): verdict display, HTML output, finding rendering - format_pcp_for_file(): forward and reverse path matching - find_readme_content(): regex matching Pure-logic methods made public for direct testability. Uses Yoast PHPUnit Polyfills XTestCase for PHPUnit 10 compatibility. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...s.org/public_html/wp-content/plugins/plugin-directory/jobs/class-plugin-automated-review.php

Catch missing or invalid attachment file paths early with an explicit check, rather than relying on Filesystem::unzip() to fail gracefully. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings April 2, 2026 14:59

Copilot started reviewing on behalf of obenland April 2, 2026 15:00 View session

Copilot AI reviewed Apr 2, 2026

View reviewed changes

obenland force-pushed the add/plugin-directory-automated-review branch from 0c2e01a to c87848e Compare April 2, 2026 15:09

obenland requested a review from Copilot April 2, 2026 15:17

Copilot started reviewing on behalf of obenland April 2, 2026 15:18 View session

Remove leading backslash from use import

4acfadc

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot AI reviewed Apr 2, 2026

View reviewed changes

obenland and others added 2 commits April 2, 2026 11:26

obenland force-pushed the add/plugin-directory-automated-review branch from f76caa7 to a32e8c5 Compare April 2, 2026 15:37

obenland requested a review from Copilot April 2, 2026 15:37

Copilot started reviewing on behalf of obenland April 2, 2026 15:38 View session

Copilot AI reviewed Apr 2, 2026

View reviewed changes

obenland force-pushed the add/plugin-directory-automated-review branch from 22f0274 to bb7004e Compare April 2, 2026 16:49

obenland force-pushed the add/plugin-directory-automated-review branch from bb7004e to ec81148 Compare April 2, 2026 16:50

obenland requested a review from Copilot April 2, 2026 16:51

Copilot started reviewing on behalf of obenland April 2, 2026 16:51 View session

Copilot AI reviewed Apr 2, 2026

View reviewed changes

obenland requested a review from Copilot April 2, 2026 17:24

Copilot started reviewing on behalf of obenland April 2, 2026 17:25 View session

Copilot AI reviewed Apr 2, 2026

View reviewed changes

obenland requested a review from Copilot April 3, 2026 11:59

Copilot started reviewing on behalf of obenland April 3, 2026 12:00 View session

Copilot AI reviewed Apr 3, 2026

View reviewed changes

...s.org/public_html/wp-content/plugins/plugin-directory/jobs/class-plugin-automated-review.php Outdated Show resolved Hide resolved

obenland force-pushed the add/plugin-directory-automated-review branch from 411facf to d245ca6 Compare April 3, 2026 15:51

Validate ZIP path before passing to Filesystem::unzip()

965c707

Catch missing or invalid attachment file paths early with an explicit check, rather than relying on Filesystem::unzip() to fail gracefully. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

obenland force-pushed the add/plugin-directory-automated-review branch from d245ca6 to 965c707 Compare April 3, 2026 15:51

Conversation

obenland commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

obenland commented Apr 2, 2026 •

edited

Loading