Skip to content

Add reusable reviewing-openmc-code skill and Copilot Review agent#3842

Merged
paulromano merged 12 commits intoopenmc-dev:developfrom
paulromano:review-skill
Mar 6, 2026
Merged

Add reusable reviewing-openmc-code skill and Copilot Review agent#3842
paulromano merged 12 commits intoopenmc-dev:developfrom
paulromano:review-skill

Conversation

@paulromano
Copy link
Contributor

Description

This PR introduces AI-assisted code review tooling so that the review guidelines live in a single, tool-agnostic skill that can be used by Copilot, Claude Code, Codex, and similar agents. It makes the following updates:

  • .claude/skills/openmc-code-review/SKILL.md: A reusable skill containing the code review procedure. Supports both reviewer mode (PR/branch review) and self-review mode (author preparing changes before submission). Note that I've chosen to put this in the .claude directory because it is recognized by both Claude Code and Copilot.
  • .github/agents/Review.agent.md: A thin Github Copilot custom-agent wrapper that delegates to the openmc-code-review skill, providing a named "Review" persona in Copilot's agent dropdown.
  • .github/copilot-instructions.md: Instructs Copilot to use the openmc-code-review skill when performing automated code reviews triggered directly on the GitHub repository.
  • AGENTS.md (updated): Replaced the inline code-review guidance with a pointer to the openmc-code-review skill, avoiding duplication.

Checklist

  • I have performed a self-review of my own code
  • I have run clang-format (version 15) on any C++ source files (if applicable)
  • I have followed the style guidelines for Python source files (if applicable)
  • I have made corresponding changes to the documentation (if applicable)
  • I have added tests that prove my fix is effective or that my feature works (if applicable)

@paulromano paulromano requested a review from jtramm February 27, 2026 02:38
Copy link
Contributor

@jtramm jtramm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great @paulromano!!! I had a lot of fun reviewing this and experimenting with making custom skills. I'm guessing this will be the first of many agentic skills for OpenMC in the near future.

I made a few suggestions for you to consider, but overall it seems like a great starting point for us.

For posterity, I've been benchmarking the claude code agentic review on my upcoming shared secondary branch draft PR (https://github.com/jtramm/openmc/tree/another_shared_secondary_branch). There's at least 10 legitimate logic bugs in there (not style/maintainability issues), but the default claude code review and this skill only find about 1 in 10 of the bugs on any given run. So, my own personal baseline expectation is that the agent will likely flag super obvious issues, but may only find about 10% of other more complex bugs. The default claude code /review skill doesn't do any better.

One trick I found is that due to random seeding, for complex PRs there is actually a huge divergence in the results you get (provided you run the review from clean/fresh contexts that aren't polluted from a previous review tool). As such, I've found that if I run multiple fully independent agentic reviews I'm able to expose way more issues than just running one. There's a paper here that shows this effect: https://arxiv.org/html/2509.01494v1 where you see significant improvements in PR reviewing if you collate 3-5 independent reviews, with greatly diminished returns going up to 10 or 20 reviews.

To that end, I may follow this up with a multi-agent skill that launches multiple such reviews in parallel so as to potentially uncover more bugs (as empirically this seems to be helpful, as discussed in https://arxiv.org/html/2509.01494v1, and in my findings as well). Sounds complex but claude already has an agent tool, so it's not too hard to get it to spawn a bunch of subagents to do parallel reviews (though it is more expensive!)

@paulromano paulromano changed the title Add reusable openmc-code-review skill and Copilot Review agent Add reusable reviewing-openmc-code skill and Copilot Review agent Mar 6, 2026
@paulromano paulromano enabled auto-merge (squash) March 6, 2026 03:39
@paulromano paulromano merged commit 908e631 into openmc-dev:develop Mar 6, 2026
6 checks passed
@paulromano paulromano deleted the review-skill branch March 6, 2026 03:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants