--- title: "How to Use AI for Code Review" subtitle: "A Practical Playbook for Faster, Higher-Quality Reviews" author: "Kelly Price" date: "2026-04-21" description: "The developer's field guide to integrating AI into the code review process — from setup to team adoption — without losing the signal that matters." tags: [ai, developer-tools, productivity] --- # How to Use AI for Code Review ## A Practical Playbook for Faster, Higher-Quality Reviews *Kelly Price* --- ## About This Guide Code review is one of the highest-leverage activities in software development. It's also one of the most inconsistent. The same pull request reviewed by two different engineers can yield wildly different feedback — not because one of them is bad at their job, but because review quality depends on attention, context, energy level, and familiarity with the codebase in ways that vary day to day and person to person. AI tools do not fix this. What they do is raise the floor. An AI reviewer will catch that you forgot to handle a null case on line 47 even when it's 4 PM on a Friday and your human reviewers are mentally checked out. It will notice that your new function duplicates logic that already exists in a utility file, even if the person reviewing your PR hasn't touched that module in eight months. It will flag that your SQL query is vulnerable to injection before the code ever ships. That's the value proposition. Not replacing human reviewers — replacing the mechanical, exhausting, inconsistent parts of review that humans are bad at anyway. This guide is written for working developers: people who write and review code professionally, who want to add AI to their workflow without adopting a half-broken process that creates more noise than signal. It assumes you know how to code. It does not assume you've worked with AI tools beyond basic chat interfaces. Each chapter covers one slice of the problem: what AI actually does during review, how to set up a workflow that fits into your existing process, how to write prompts that produce useful output, how to filter that output intelligently, where AI genuinely outperforms humans, where it falls short, and how to get your team to actually use it. The practical exercises at the end of each chapter are not optional reading. They are the point. Reading about code review without reviewing code is like reading about swimming. At some point you have to get in the water. By the end of this guide, you will have a working AI review pipeline integrated into your pull request process, a set of prompts tuned to your codebase and team standards, and the judgment to know when to trust AI output and when to ignore it. That last part matters more than the tools. --- ## Table of Contents 1. [What AI Code Review Actually Does (and What It Does Not)](#chapter-1) 2. [Setting Up Your AI Review Workflow](#chapter-2) 3. [Writing Prompts That Get Useful Feedback](#chapter-3) 4. [Filtering AI Suggestions: Signal vs. Noise](#chapter-4) 5. [Security and Bug Detection: Where AI Shines](#chapter-5) 6. [Style, Consistency, and Standards Enforcement](#chapter-6) 7. [Integrating AI Review into Pull Request Workflows](#chapter-7) 8. [Team Adoption: Getting Buy-In Without a Fight](#chapter-8) 9. [Measuring Review Quality Over Time](#chapter-9) - [Conclusion](#conclusion) - [Appendix A: Glossary](#appendix-a) - [Appendix B: Tools and Resources](#appendix-b) - [Appendix C: Further Reading](#appendix-c) --- ## Chapter 1: What AI Code Review Actually Does (and What It Does Not) {#chapter-1} Before you integrate any tool into your workflow, you need an accurate model of what it actually does. Miscalibrated expectations are the primary reason AI review workflows fail. Teams either expect too much — thinking AI will catch every bug and make human review optional — or too little, dismissing the tools before they've been configured properly. AI code review tools work by analyzing your diff (or full file, depending on the tool) and generating natural language feedback using a large language model. The underlying model has been trained on enormous volumes of code from public repositories, documentation, and other sources, which means it carries statistical knowledge of common patterns, common mistakes, and common idioms across dozens of languages and frameworks. When you submit a pull request, an AI reviewer does roughly the following: it tokenizes the changed code, builds context from surrounding lines and sometimes the broader file, and generates a response that identifies potential issues, suggests improvements, or asks clarifying questions. It is doing pattern matching at scale, not executing your code or running tests. This distinction is critical. AI review is static analysis with a natural language interface and a much broader pattern vocabulary than traditional linters. It cannot: - Run your code and observe runtime behavior - Detect race conditions that only appear under specific scheduling - Know whether your business logic is correct for your domain - Understand the political context of why a particular decision was made six months ago - Catch bugs that require understanding your full production data shape What it can do is impressive in its own right. A well-configured AI reviewer catches: missing error handling, common injection vulnerabilities, off-by-one errors, unhandled edge cases in conditional logic, violations of naming conventions, duplicate code, suspicious type coercions, and deprecated API usage. It can explain what a block of code does, suggest a cleaner implementation, and flag when a function is doing too many things. > **Key Insight:** AI review is best understood as an extremely well-read junior engineer who has read every public repository on GitHub but has never run your code in production. Intelligent, fast, broadly knowledgeable — and completely blind to runtime and domain context. The tools in this space fall into a few categories. First, there are native integrations built into platforms like GitHub (Copilot code review), GitLab Duo, and Bitbucket. These have the advantage of living where your PRs already are. Second, there are standalone tools like CodeRabbit, Qodo (formerly CodiumAI), and Sourcery that connect to your repository via webhooks and post review comments automatically. Third, there is the manual approach: pasting diffs into a chat interface like Claude or ChatGPT with a custom prompt. Each has different tradeoffs in setup cost, configurability, and signal quality. A critical piece of context: AI reviewers are not deterministic. Run the same diff twice and you may get slightly different feedback. This is not a bug — it reflects the probabilistic nature of language models — but it does mean you should not treat AI output as authoritative. Treat it as a second pass from a fast, knowledgeable colleague who might occasionally hallucinate a concern that doesn't exist. > **Warning:** AI tools will sometimes confidently flag code that is correct and intentional. If your reviewer says "this loop could be replaced with `map()`" but you deliberately used a loop for readability, that is not a bug in your code. It is a feature suggestion from a tool that doesn't know your team's style preferences. Always verify before acting on AI feedback. The most productive mental model is this: AI review handles the mechanical layer of review so your human reviewers can focus on the architectural and domain layers. Instead of spending review cycles asking "did you handle the error case?", humans can ask "is this the right abstraction for this problem?" That shift in cognitive load is where the real value comes from. One more thing worth stating plainly: AI code review does not make you a better programmer on its own. The engineers who get the most out of these tools are the ones who read the feedback critically, understand why the tool flagged something, and internalize the pattern. Blindly accepting or dismissing AI suggestions produces neither better code nor better engineers. **Key Takeaways** - AI review is static analysis with natural language output — it does not execute your code - It excels at mechanical checks: error handling, common vulnerabilities, duplication, style - It cannot evaluate business logic correctness, runtime behavior, or domain context - Treat output as probabilistic feedback from a well-read peer, not authoritative judgment - The value is freeing human reviewers to focus on architecture and domain concerns **Practical Exercise** Take a recent merged pull request from your codebase — one where you remember what review comments were actually made. Paste the diff into Claude or ChatGPT with the prompt: "Review this code diff. Identify potential bugs, missing error handling, and style issues." Compare the AI output to the comments your human reviewers made. Note what the AI caught that humans missed, what humans caught that the AI missed, and what the AI flagged that was not actually a problem. Keep this comparison — you'll reference it in Chapter 4. --- ## Chapter 2: Setting Up Your AI Review Workflow {#chapter-2} Setup is where most teams get stuck. Not because the tools are hard to install, but because "installing a tool" and "having a workflow" are completely different things. This chapter gets you from zero to a repeatable, configurable AI review process. The first decision is where AI review fits in your pipeline. There are three insertion points: pre-commit (on your local machine before you push), pre-merge (on the PR before it can be merged), or post-merge (for retrospective analysis). For most teams, the right answer is pre-merge. Pre-commit adds friction to the development loop. Post-merge is too late to act on. Pre-merge catches issues at exactly the right moment — before code ships, but after the developer has done their own review. Start with a GitHub Actions workflow if your code lives on GitHub. Here is a working configuration using the Claude API directly: ```yaml # .github/workflows/ai-review.yml name: AI Code Review on: pull_request: types: [opened, synchronize] jobs: ai-review: runs-on: ubuntu-latest permissions: pull-requests: write contents: read steps: - name: Checkout uses: actions/checkout@v4 with: fetch-depth: 0 - name: Get diff id: diff run: | git diff origin/${{ github.base_ref }}...HEAD > /tmp/pr.diff echo "diff_size=$(wc -c < /tmp/pr.diff)" >> $GITHUB_OUTPUT - name: Run AI review if: steps.diff.outputs.diff_size < '50000' env: ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} run: | python3 .github/scripts/ai_review.py \ --diff /tmp/pr.diff \ --output /tmp/review.json - name: Post review comment uses: actions/github-script@v7 with: script: | const fs = require('fs'); const review = JSON.parse(fs.readFileSync('/tmp/review.json', 'utf8')); await github.rest.pulls.createReview({ owner: context.repo.owner, repo: context.repo.repo, pull_number: context.issue.number, body: review.summary, event: 'COMMENT', comments: review.line_comments }); ``` The Python script that generates the review: ```python # .github/scripts/ai_review.py import anthropic import argparse import json import sys def review_diff(diff_content: str) -> dict: client = anthropic.Anthropic() system_prompt = """You are a code reviewer. Analyze the provided diff and return a JSON object with: - "summary": A 2-3 sentence overall assessment - "line_comments": An array of objects, each with: - "path": file path - "line": line number (integer) - "body": your comment Focus on: bugs, missing error handling, security issues, and unclear logic. Skip: formatting preferences, minor style issues, and suggestions the author clearly already considered. Return only valid JSON. No markdown wrapping.""" message = client.messages.create( model="claude-opus-4-7", max_tokens=4096, messages=[ { "role": "user", "content": f"Review this diff:\n\n```diff\n{diff_content}\n```" } ], system=system_prompt ) try: return json.loads(message.content[0].text) except json.JSONDecodeError: return {"summary": message.content[0].text, "line_comments": []} def main(): parser = argparse.ArgumentParser() parser.add_argument("--diff", required=True) parser.add_argument("--output", required=True) args = parser.parse_args() with open(args.diff) as f: diff_content = f.read() if not diff_content.strip(): result = {"summary": "No changes detected.", "line_comments": []} else: result = review_diff(diff_content) with open(args.output, "w") as f: json.dump(result, f) if __name__ == "__main__": main() ``` > **Try This:** Run this script locally before wiring it into CI. Generate a diff with `git diff main...HEAD > test.diff` and call the script directly: `python3 ai_review.py --diff test.diff --output result.json`. Read the JSON output. This is also the fastest way to debug prompt issues without burning CI minutes. If you're using a managed tool instead of a custom script, the configuration is different but the principle is the same. CodeRabbit, for instance, uses a `.coderabbit.yaml` file at the root of your repository: ```yaml # .coderabbit.yaml language: "en-US" tone_instructions: "Be direct and concise. Skip compliments." early_access: false reviews: profile: "chill" request_changes_workflow: false high_level_summary: true poem: false review_status: true collapse_walkthrough: false auto_review: enabled: true drafts: false base_branches: - main - develop path_filters: - "!**/*.lock" - "!**/vendor/**" - "!**/*.min.js" path_instructions: - path: "src/auth/**" instructions: "Focus on security. Check for injection, insecure token handling, and privilege escalation." - path: "src/api/**" instructions: "Check input validation, rate limiting, and error response formats." ``` Three configuration decisions matter most regardless of which tool you use. First, set a diff size limit. Reviews of 10,000-line diffs are useless. If the diff is too large, either skip the AI review or break the PR. Second, define path-specific instructions — your authentication code needs different review criteria than your UI components. Third, configure which events trigger review: opened and synchronize are usually sufficient; you probably don't want AI comments on every comment or label event. > **Warning:** Do not store your API key in your repository. Use GitHub Secrets (`${{ secrets.ANTHROPIC_API_KEY }}`), environment variables injected at runtime, or a secrets manager. Rotating a leaked API key is annoying. Paying for unauthorized API usage is worse. The last setup step is creating a feedback loop for tuning. Add a reaction to AI review comments — a thumbs up for useful, thumbs down for noise — and periodically audit which comments are being acted on. This is the data you'll use to improve your prompts in Chapter 3 and your filtering logic in Chapter 4. **Key Takeaways** - Pre-merge is the right insertion point for AI review in most workflows - A working GitHub Actions setup requires: diff extraction, API call, and comment posting - Diff size limits prevent useless reviews of massive PRs - Path-specific instructions dramatically improve review relevance - Never commit API keys; use secrets management from day one **Practical Exercise** Set up the GitHub Actions workflow above in a repository you control — a personal project or a test repo works fine. Open a small PR, observe the AI review comment, and verify the JSON output is structured correctly. If you get a JSON parse error, add a fallback that posts the raw text as a comment. Debugging this locally before relying on CI saves hours. --- ## Chapter 3: Writing Prompts That Get Useful Feedback {#chapter-3} The quality of AI review output is determined almost entirely by the quality of your prompt. This is not a metaphor. A vague prompt produces vague output. A precise prompt that tells the model exactly what to look for, what to ignore, and what format to use produces output that is immediately actionable. Most developers who are disappointed with AI code review have never written a real prompt. They paste a diff into a chat window and type "review this." The model does its best, produces a generic response, and the developer concludes the tool isn't useful. The tool isn't the problem. Start with the structure of a good review prompt. It has four components: role definition, focus areas, exclusions, and output format. **Role definition** tells the model what kind of reviewer to be: ``` You are a senior backend engineer reviewing a Python pull request. You prioritize correctness, security, and maintainability. You are direct and do not soften criticism. ``` **Focus areas** tell it what to look for: ``` Check for: - Unhandled exceptions and missing error cases - SQL injection and other injection vulnerabilities - Race conditions or non-thread-safe patterns - Functions that exceed a single responsibility - Any use of deprecated stdlib APIs ``` **Exclusions** prevent noise: ``` Do not comment on: - Line length or whitespace (handled by our formatter) - Import ordering (handled by isort) - Docstring presence (we enforce this separately) - Suggestions to rename variables unless the name is actively misleading ``` **Output format** ensures the output is usable: ``` For each issue found, output: - File and line number - Severity: CRITICAL | HIGH | MEDIUM | LOW - One sentence describing the issue - One sentence explaining the fix If no issues are found, say so in one sentence. Do not fabricate issues. ``` Here is a complete prompt template for a Python backend service: ```python REVIEW_PROMPT = """You are a senior Python backend engineer. Review the following diff with these priorities in order: 1. Security vulnerabilities (injection, authentication bypass, exposed secrets) 2. Correctness bugs (wrong logic, missing error handling, uncaught exceptions) 3. Performance problems (N+1 queries, unnecessary blocking calls, large memory allocations) 4. Maintainability concerns (single responsibility violations, unclear naming) Skip: formatting, import style, docstring presence, minor refactors with no behavior impact. For each issue, output exactly this format: ISSUE: : [] PROBLEM: FIX: If no issues are found, output: NO ISSUES FOUND Diff: {diff} """ ``` > **Key Insight:** Telling the model what to ignore is as important as telling it what to check. Without exclusions, the model optimizes for appearing thorough by covering everything it can see. With exclusions, it focuses on what you actually need. The prompt above works for a general Python review. For domain-specific reviews, you need domain-specific context. If you're reviewing database migration code, add context about your schema conventions. If you're reviewing API endpoints, add your authentication requirements. The model does not know your team's decisions — you have to tell it. For a Django API endpoint review: ```python DJANGO_API_PROMPT = """You are reviewing a Django REST Framework view. Project conventions: - All views require authentication via `@permission_classes([IsAuthenticated])` - Serializer validation errors must be returned as 400, not 500 - Database writes require explicit transaction management for multi-step operations - Pagination is required on any endpoint returning more than one object - We use `select_related` and `prefetch_related` explicitly — no implicit joins Check this diff against those conventions and standard DRF security practices. Flag any deviation from the above as HIGH severity. {diff} """ ``` > **Try This:** Take your worst AI review output from Chapter 1's exercise — the one with the most noise — and rewrite the prompt using the four-component structure above. Run the same diff through the new prompt. Count how many low-value comments disappear. In most cases, exclusions alone cut noise by 40-60%. Temperature matters too, though most interfaces don't expose it directly. If you're using the API, set `temperature=0` for review tasks. You want consistency and precision, not creative variation. A review that gives different output for the same diff on repeated runs is a review you can't build process around. One pattern that reliably produces better output is chain-of-thought prompting: asking the model to reason before it responds. This works especially well for security-focused review: ```python SECURITY_REVIEW_PROMPT = """You are a security-focused code reviewer. Before listing issues, reason through these questions: 1. What does this code receive as input from external sources? 2. How is that input validated or sanitized? 3. What resources (files, database, network) does this code access? 4. What authentication or authorization checks are present? 5. What is the worst-case scenario if this code has a bug? After reasoning, list specific issues you found. Be concrete about file and line. Diff: {diff} """ ``` The reasoning step forces the model to build a mental model of the code before generating feedback, which produces more coherent and accurate analysis than jumping straight to output. Finally: version your prompts. Store them in your repository alongside your CI configuration. When you change a prompt, treat it like a code change — review it, understand the tradeoffs, and be able to roll it back if the output quality degrades. A prompt is executable logic. Treat it accordingly. **Key Takeaways** - Prompt quality determines output quality — vague prompts produce vague reviews - Every review prompt needs: role definition, focus areas, exclusions, and output format - Domain-specific context dramatically improves relevance for non-generic code - Set `temperature=0` when using the API for consistent, reproducible review output - Store prompts in version control and treat prompt changes like code changes **Practical Exercise** Write three prompts for your codebase: one general-purpose review prompt, one focused on security, and one focused on performance. Use the four-component structure from this chapter. Run each against the same test diff and compare the output. Keep the one that produces the most actionable feedback. You'll use these prompts in Chapter 7 when you wire them into your PR workflow. --- ## Chapter 4: Filtering AI Suggestions: Signal vs. Noise {#chapter-4} Even a well-configured AI reviewer produces noise. Not all noise is equal. Some of it is irrelevant (style suggestions that conflict with your formatter). Some of it is wrong (flagging intentional behavior as a bug). Some of it is low-priority (valid observations that don't warrant a blocking comment). Filtering effectively is a skill, and it is what separates teams that get value from AI review from teams that disable it after a week because "it's annoying." The first filtering principle: distinguish between suggestions that require developer judgment and suggestions that can be automatically verified. A comment saying "this function might throw a NullPointerException" requires you to look at the calling code to evaluate. A comment saying "this import is unused" can be verified in three seconds with your IDE. Automatically-verifiable suggestions are almost always worth checking. Judgment-call suggestions require more scrutiny. The second principle: categorize by severity and act on categories, not individual comments. Process CRITICAL and HIGH items immediately — these are potential bugs and security holes. Review MEDIUM items during normal human review. Treat LOW items as optional reading, not blocking feedback. If your prompt isn't generating severity labels, add them. They are the single most important structural element in AI review output. Here is a Python script that parses AI review output and filters by severity: ```python #!/usr/bin/env python3 # filter_review.py — parse AI review output and filter by minimum severity import re import sys from dataclasses import dataclass from typing import List SEVERITY_ORDER = {"CRITICAL": 0, "HIGH": 1, "MEDIUM": 2, "LOW": 3} @dataclass class ReviewComment: location: str severity: str problem: str fix: str def parse_review(text: str) -> List[ReviewComment]: pattern = re.compile( r"ISSUE:\s+(.+?)\s+\[(\w+)\]\s*\n" r"PROBLEM:\s+(.+?)\s*\n" r"FIX:\s+(.+?)(?=\nISSUE:|\Z)", re.DOTALL ) comments = [] for match in pattern.finditer(text): comments.append(ReviewComment( location=match.group(1).strip(), severity=match.group(2).strip().upper(), problem=match.group(3).strip(), fix=match.group(4).strip() )) return comments def filter_by_severity(comments: List[ReviewComment], min_severity: str) -> List[ReviewComment]: threshold = SEVERITY_ORDER.get(min_severity, 3) return [c for c in comments if SEVERITY_ORDER.get(c.severity, 99) <= threshold] if __name__ == "__main__": min_sev = sys.argv[1] if len(sys.argv) > 1 else "MEDIUM" raw = sys.stdin.read() comments = parse_review(raw) filtered = filter_by_severity(comments, min_sev) for c in filtered: print(f"[{c.severity}] {c.location}") print(f" Problem: {c.problem}") print(f" Fix: {c.fix}") print() print(f"Showing {len(filtered)} of {len(comments)} issues (threshold: {min_sev})") ``` Usage: ```bash python3 ai_review.py --diff pr.diff --output raw_review.txt python3 filter_review.py HIGH < raw_review.txt ``` > **Warning:** Do not let AI review block merges based on LOW or MEDIUM severity issues without human approval. Automated gates on probabilistic output will create bottlenecks on legitimate PRs. Reserve hard blocks for CRITICAL issues only, and even then, give engineers a documented escape hatch for false positives. The third principle: build a false positive log. Every time AI review flags something that is correct and intentional, record it. After a few weeks, you'll see patterns. Maybe your codebase uses a particular pattern that the model consistently misreads. Add that pattern to your prompt's exclusion list. Over time your false positive rate drops toward zero. A simple false positive log format in JSON: ```json [ { "date": "2026-03-15", "file": "src/auth/tokens.py", "ai_concern": "Hardcoded string 'secret' may be a credential", "actual": "Variable name 'secret' refers to OAuth client secret parameter, not a value", "prompt_fix": "Added exclusion: 'Do not flag parameter names containing the word secret'" } ] ``` Maintain this log in your repository. When your AI review pipeline produces a false positive, add an entry and update the prompt the same day. Deferred prompt fixes accumulate into "AI review is useless" sentiment. > **Key Insight:** The false positive rate of AI review is roughly inversely proportional to the specificity of your prompt. Generic prompts produce generic output with high noise. Prompts that describe your specific codebase, conventions, and known patterns produce targeted output with low noise. The fourth principle: treat human override as a valid outcome. If an engineer on your team reviews an AI comment and determines it's wrong, that's the correct result — not a failure of the process. The engineer's judgment is the final filter. AI review is an input to that judgment, not a replacement for it. Teams that treat AI output as authoritative turn their engineers into rubber stamps, which is worse than having no AI review at all. **Key Takeaways** - Distinguish automatically-verifiable suggestions from judgment-call suggestions - Severity labels are the most important structural element in AI review output - Build and maintain a false positive log; use it to improve your prompts - Never auto-block merges on LOW or MEDIUM severity AI findings - Human judgment is the final filter — treat it as such, not as a rubber stamp **Practical Exercise** From the comparison you built in Chapter 1's exercise, categorize every AI comment as: (a) correct and actionable, (b) correct but not actionable (low priority), or (c) incorrect/misleading. Calculate your false positive rate. If it's above 20%, your prompt needs work. Apply the exclusion principle from Chapter 3 to cut the false positive rate below 15%. --- ## Chapter 5: Security and Bug Detection: Where AI Shines {#chapter-5} If there is one category where AI code review consistently outperforms casual human review, it is security. Not because the model has magical insight into your threat model — it doesn't — but because security vulnerabilities are often instances of well-known patterns, and pattern matching is what language models do well. Consider SQL injection. The pattern is straightforward: user-controlled input concatenated into a database query string. A human reviewer who is tired, or who doesn't have security top-of-mind, will miss this. An AI reviewer will catch it every time if your prompt asks it to look. Same with command injection, path traversal, and cross-site scripting. These are pattern-matching problems with decades of documented examples in training data. Here is a concrete example. Given this Python code: ```python def get_user(username: str) -> dict: query = f"SELECT * FROM users WHERE username = '{username}'" result = db.execute(query) return result.fetchone() ``` A well-prompted AI reviewer will flag this immediately: ``` ISSUE: src/users.py:2 [CRITICAL] PROBLEM: f-string interpolation of username directly into SQL query creates injection vulnerability. FIX: Use parameterized query: db.execute("SELECT * FROM users WHERE username = ?", (username,)) ``` The fix is correct. This is exactly the category where you want AI review: catching a security mistake that is easy to make and easy to miss in a code review where the reviewer is focused on logic rather than query construction. Other high-confidence security detections include: **Hardcoded credentials:** ```python # AI will flag this API_KEY = "sk-prod-a8f3k2..." DATABASE_URL = "postgresql://admin:hunter2@localhost/prod" ``` **Insecure deserialization:** ```python import pickle def load_session(data: bytes): return pickle.loads(data) # AI flags: arbitrary code execution via crafted payload ``` **Missing authentication checks:** ```python @app.route("/admin/delete-user", methods=["POST"]) def delete_user(): user_id = request.json["user_id"] User.query.filter_by(id=user_id).delete() # AI flags: no auth check db.session.commit() return {"deleted": True} ``` > **Key Insight:** AI review catches security issues at the code pattern level. It cannot catch issues at the architectural level — for example, a correctly-implemented authentication system that is attached to the wrong endpoint. That requires a human who understands your authorization model. Bug detection follows a similar pattern. The bugs AI catches reliably are the ones that have clear structural signatures: missing null checks, off-by-one errors in range expressions, unreachable code, swapped arguments, and missing `await` in async functions. Here are real examples: ```python # Missing await — AI catches this async def fetch_data(url: str) -> dict: response = requests.get(url) # Should be: await session.get(url) return response.json() # Off-by-one — AI catches this def get_last_n_items(items: list, n: int) -> list: return items[len(items) - n:] # Correct, but AI checks edge case: what if n > len(items)? # Swapped arguments — AI catches this (if the types are suggestive) def set_dimensions(height: int, width: int): self.width = height # AI flags: argument names suggest these are swapped self.height = width ``` The category where AI bug detection gets less reliable is logic errors that require understanding the intended behavior. If a function sorts a list in the wrong order, AI might not catch it unless the sort direction is implied by the variable name or surrounding comments. The model cannot know what the function was supposed to do — only what it does. > **Try This:** Run a security-focused prompt against three months of your merged PRs. Use the prompt from Chapter 3's security template. Look for patterns in what it flags. If it consistently surfaces a particular class of issue (e.g., missing input validation on API endpoints), that's a signal to add input validation to your PR checklist or your linting rules — so AI review catches new instances automatically. For bug detection specifically, the signal-to-noise ratio improves dramatically when you include type information. Add type hints to your function signatures. AI models use type annotations as strong signals for detecting mismatches — a function that accepts `str` being called with what the model can infer is an `int` will be flagged. Without types, the model is guessing. One class of bugs worth special mention: resource leaks. AI reliably catches file handles, database connections, and network sockets that are opened but not properly closed: ```python def write_report(filename: str, data: str): f = open(filename, "w") f.write(data) # AI flags: file handle not closed. Use: with open(filename, "w") as f: ``` This is the kind of bug that is obvious in isolation but easy to miss when you're reading a 50-file diff and this is in file 34. **Key Takeaways** - AI excels at security detection for well-known vulnerability patterns: injection, hardcoded credentials, missing auth, insecure deserialization - Bug detection is most reliable for structural patterns: missing awaits, null checks, resource leaks, swapped arguments - AI cannot detect logic errors that require understanding intended behavior - Type annotations significantly improve bug detection accuracy - Use AI security findings to identify systemic gaps, not just individual issues **Practical Exercise** Write a security-focused prompt using the template from Chapter 3 and run it against five recently merged PRs. For each issue flagged, determine: (1) Is it a real issue? (2) Was it caught in human review? (3) If it was missed by human review, what would have happened if it shipped? This exercise calibrates your sense of how much risk AI review is actually preventing. --- ## Chapter 6: Style, Consistency, and Standards Enforcement {#chapter-6} Style enforcement is the domain where developers are most skeptical of AI review, and for good reason. Most style issues should be handled by automated formatters — `black` for Python, `prettier` for JavaScript, `gofmt` for Go. If your formatter runs in CI, you should not be getting style comments in code review at all. AI review should never be your first line of defense for formatting. But style enforcement is broader than formatting. It includes naming conventions, architectural patterns, API design consistency, error handling conventions, and test structure requirements. These are exactly the things formatters cannot check, and where AI review adds genuine value. Consider naming conventions. If your team has a rule that database model methods that return multiple rows use the `_list` suffix, and a developer adds a method called `get_active_users()` that returns a queryset, AI review can catch this — but only if your prompt includes the convention: ``` Team naming conventions: - Methods returning multiple rows from the database must end in _list (e.g., get_active_users_list, not get_active_users) - Methods returning a single object must end in _or_none if they return None on miss - Boolean methods must start with is_ or has_ ``` With this context, the model knows what to look for. Without it, it has no basis for flagging the deviation. The same principle applies to error handling conventions: ``` Error handling standards: - All exceptions must be caught and re-raised as domain exceptions from src/exceptions.py - Never let raw database exceptions propagate to the API layer - Every caught exception must be logged before re-raising - Use our custom AppError class, not generic Exception subclasses ``` > **Key Insight:** AI style and consistency review is only as good as the documentation you give it. The model cannot read your team's unwritten rules. If your conventions aren't in the prompt, they won't be enforced. This is actually a useful side effect: building AI review prompts forces you to document conventions that previously lived only in senior engineers' heads. Architectural pattern enforcement is another high-value use case. If your service uses a strict layered architecture — controllers call services, services call repositories, repositories call the database — and a PR adds a database call in a controller, AI can catch it: ``` Architecture rules: - Controllers (src/api/) must not import from src/db/ directly - Only repository classes (src/repositories/) interact with the database - Service classes (src/services/) contain business logic - Flag any import of db or models from non-repository code ``` For test quality, AI review can check that tests follow your coverage expectations: ``` Test requirements: - Every public method needs at least one test for the happy path and one for the error path - Tests must not use time.sleep() — use mock_time or freeze_gun - Each test must have exactly one assertion focus (multiple assertions are allowed but must test the same concern) - Tests in src/tests/integration/ must use real database connections, not mocks ``` > **Warning:** Do not use AI review to enforce rules that should be enforced by static analysis tools. If you want to ban `eval()`, use a linter rule. If you want to enforce import ordering, use `isort`. AI review is for checks that require contextual understanding, not for checks that can be expressed as a regex or an AST rule. Using AI for mechanical rule enforcement wastes tokens and creates noise. One practical technique: build a "team standards" document in your repository and reference it in your prompt. This keeps the prompt concise and lets you update standards without modifying the prompt template: ```python # Load team standards from a file in the repo with open(".github/review-standards.md") as f: standards = f.read() prompt = f"""You are reviewing a pull request for compliance with our team standards. Standards: {standards} Check the diff for violations of the above standards only. Ignore anything not covered by the standards. Diff: {diff} """ ``` Your `review-standards.md` becomes a living document that the team maintains and the AI enforces: ```markdown # Code Review Standards ## Naming - Database query methods returning multiple rows: `*_list` suffix - Nullable return methods: `*_or_none` suffix - Boolean predicates: `is_*` or `has_*` prefix ## Error Handling - All exceptions caught and re-raised as domain exceptions - Raw database exceptions never reach API layer - All caught exceptions logged before re-raising ## Architecture - Controllers: no direct db imports - Repositories: only layer that calls db - Services: business logic only, no HTTP concepts ``` This pattern turns style enforcement into a documentation problem, which is much easier to manage than a prompt engineering problem. When standards change, you update the document. The AI behavior updates automatically. **Key Takeaways** - Formatting should be handled by automated tools, not AI review - AI style review adds value for naming conventions, architectural patterns, and team-specific standards - The model cannot enforce rules you haven't documented — writing AI prompts forces conventions to be explicit - Store review standards in a versioned file and reference it from your prompt - Never use AI for checks that can be expressed as linter rules **Practical Exercise** Document five team conventions that are currently enforced informally (via human reviewer memory) but not in any tool or written document. Add them to a `review-standards.md` file. Write a prompt that checks a diff against those five conventions specifically. Run it against a PR where you know one of those conventions was violated. Verify the AI catches it. --- ## Chapter 7: Integrating AI Review into Pull Request Workflows {#chapter-7} A working AI review script and a working PR workflow are different things. The script produces output. The workflow determines where that output lands, who sees it, how it's acted on, and what happens when the automated review disagrees with human judgment. This chapter covers the full integration. The baseline integration posts AI review as a single PR comment. That's functional but not optimal. Better is posting inline comments at the specific lines where issues were found. The GitHub API supports this via the `pull_request_reviews` endpoint, which accepts both a summary body and an array of line-level comments with file path, line number, and comment body. Here is a complete integration script that posts structured inline comments: ```python #!/usr/bin/env python3 # post_review.py import anthropic import json import os import subprocess import sys from github import Github def get_diff(base_sha: str, head_sha: str) -> str: result = subprocess.run( ["git", "diff", f"{base_sha}...{head_sha}"], capture_output=True, text=True, check=True ) return result.stdout def parse_diff_positions(diff: str) -> dict: """Map file:line to diff position for GitHub API.""" positions = {} current_file = None position = 0 line_num = 0 for line in diff.split("\n"): if line.startswith("+++ b/"): current_file = line[6:] position = 0 elif line.startswith("@@"): import re match = re.search(r"\+(\d+)", line) if match: line_num = int(match.group(1)) - 1 position += 1 elif current_file: position += 1 if not line.startswith("-"): line_num += 1 positions[f"{current_file}:{line_num}"] = position return positions def run_ai_review(diff: str, standards_path: str = ".github/review-standards.md") -> dict: client = anthropic.Anthropic() standards = "" if os.path.exists(standards_path): with open(standards_path) as f: standards = f.read() system = f"""You are a code reviewer. Return JSON only. Output format: {{ "summary": "2-3 sentence overall assessment", "issues": [ {{ "file": "path/to/file.py", "line": 42, "severity": "CRITICAL|HIGH|MEDIUM|LOW", "comment": "Specific, actionable feedback in one or two sentences." }} ] }} Standards to enforce: {standards} Rules: - Only flag real issues, not stylistic preferences not in standards - severity=CRITICAL: security vulnerability or data loss risk - severity=HIGH: likely bug or standards violation - severity=MEDIUM: possible issue requiring judgment - severity=LOW: minor improvement opportunity - Return empty issues array if no issues found - Return valid JSON only""" response = client.messages.create( model="claude-opus-4-7", max_tokens=4096, temperature=0, system=system, messages=[{"role": "user", "content": f"Review:\n```diff\n{diff}\n```"}] ) try: return json.loads(response.content[0].text) except json.JSONDecodeError: return {"summary": response.content[0].text, "issues": []} def post_review(token: str, repo_name: str, pr_number: int, review: dict, positions: dict): g = Github(token) repo = g.get_repo(repo_name) pr = repo.get_pull(pr_number) commit = pr.get_commits().reversed[0] comments = [] for issue in review.get("issues", []): key = f"{issue['file']}:{issue['line']}" position = positions.get(key) if position: comments.append({ "path": issue["file"], "position": position, "body": f"**[{issue['severity']}]** {issue['comment']}" }) pr.create_review( commit=commit, body=f"🤖 AI Review\n\n{review['summary']}", event="COMMENT", comments=comments ) if __name__ == "__main__": token = os.environ["GITHUB_TOKEN"] repo_name = os.environ["GITHUB_REPOSITORY"] pr_number = int(os.environ["PR_NUMBER"]) base_sha = os.environ["BASE_SHA"] head_sha = os.environ["HEAD_SHA"] diff = get_diff(base_sha, head_sha) if len(diff) > 50000: print("Diff too large for AI review. Skipping.") sys.exit(0) positions = parse_diff_positions(diff) review = run_ai_review(diff) post_review(token, repo_name, pr_number, review, positions) print(f"Posted review with {len(review.get('issues', []))} comments.") ``` > **Try This:** Before deploying to production, test your integration using GitHub's REST API directly with `curl`. Create a test PR, get the PR number, and verify your script posts comments at the right lines. Debugging line number positioning issues in CI is painful; debugging locally takes five minutes. Beyond the mechanics, the workflow design matters. A few decisions that affect team adoption: **Timing**: AI review should complete before human review begins. Add a required status check named `ai-review` and set it to non-blocking — it must complete, but it cannot block merge on its own. This tells reviewers the AI pass is done and they can read its output while doing their review. **Threading**: Post AI comments as a single review event, not individual comments. Single review events appear in the PR timeline as one collapsible block, which is much less noisy than twenty separate comments. Engineers can expand the block when they want to read AI feedback and ignore it otherwise. **Resolution**: Decide who is responsible for resolving AI review comments. The cleanest pattern: the PR author resolves them with a brief note ("fixed in latest commit" or "intentional — see discussion in #123"). This creates an audit trail without requiring a separate review round. > **Warning:** If you make AI review a required check with the ability to block merges, you will create a process that breaks whenever the AI API has an outage or rate limiting kicks in. Add a timeout and a fallback: if the review job fails to complete within five minutes, mark the status check as passing with a note that AI review was skipped. Never let third-party API availability determine whether your team can ship code. The last workflow consideration is re-review on push. When a developer pushes new commits to a PR, should AI review run again? In most cases, yes — you want to catch issues introduced in subsequent commits. But scope the re-review to the new commits only, not the full PR diff, to avoid re-flooding the PR with comments on already-reviewed code. **Key Takeaways** - Inline line-level comments require mapping diff positions to GitHub's position API - Post all AI comments as a single review event to reduce timeline noise - Make AI review a required non-blocking status check — it must complete but cannot independently block merges - Add timeouts and fallbacks so API outages never block deploys - Re-review on push should scope to new commits only **Practical Exercise** Deploy the integration script above to a real repository. Open a PR with at least one intentional issue (a hardcoded credential, a missing error handler). Verify the AI posts an inline comment at the right line. Then simulate an API failure by temporarily setting an invalid `ANTHROPIC_API_KEY` and verify your workflow fails gracefully rather than blocking the PR. --- ## Chapter 8: Team Adoption: Getting Buy-In Without a Fight {#chapter-8} Technical setup is the easy part. Getting a team to actually use AI review — to read the output, act on what's valid, and push back on what isn't — requires more care. Teams that get good value from AI review treat it as a shared infrastructure investment. Teams that fight it or ignore it usually had it imposed on them without explanation or input. The mistake most leads make is deploying AI review silently and hoping the team notices the value. This produces one of two outcomes: developers ignore the AI comments because they don't know what they're supposed to do with them, or developers resent the AI comments because they feel surveilled or second-guessed. Neither outcome is what you want. Start with a team conversation before you ship the integration. Not a presentation — a conversation. Bring an example diff with real AI output. Show what the tool catches. Show what it gets wrong. Explicitly say: "This is not a replacement for human review. It's a first pass that we think will save us time on the mechanical checks. Human judgment still runs the show." That framing matters. Engineers respond well to tools that augment their judgment and poorly to tools that seem to replace it. The moment someone feels like AI review is treating them as a quality problem to be corrected, adoption collapses. > **Key Insight:** Frame AI review as a tool that makes your human reviewers faster and better, not as a tool that compensates for human reviewer weaknesses. The first framing creates allies. The second creates adversaries. Once the tool is deployed, run a two-week trial with explicit feedback collection. Ask engineers to rate each AI comment as "useful," "not applicable," or "wrong." Collect this data. At the end of the two weeks, share the results with the team: "We got 73 AI comments. Engineers rated 41 as useful, 18 as not applicable, and 14 as wrong. Here's what we're changing in the prompt to improve that ratio." This shows the team that their feedback changes the system, which creates investment. Addressing common objections: **"It's going to slow down code review with noise."** This is a legitimate concern and the answer is configuration, not dismissal. Show them the noise-reduction techniques from Chapter 4. If after configuration the noise rate is still above 20%, keep tuning. Don't declare victory until the tool is actually useful. **"I don't want a robot telling me my code is wrong."** Acknowledge this directly. The AI is not judging their engineering ability — it's doing a fast scan for common patterns. Senior engineers get value from it because it frees them from scanning for mechanical errors and lets them focus on architecture. Make this point with a senior engineer the team respects, not just from the team lead. **"What if it misses something and we ship a bug because we thought AI caught it?"** This is the most important objection and it deserves a direct answer: AI review does not reduce the requirement for human review. It adds a layer. The existing process is still intact. Nothing that AI review catches would have previously been a human's sole responsibility to catch. > **Warning:** Do not use AI review findings as ammunition in performance reviews or code quality discussions about individual engineers. The moment AI review data becomes a measurement tool for people rather than a quality tool for code, you will lose the team's trust in the process permanently. For remote or async teams, AI review has an amplifying benefit that is worth calling out explicitly: it provides immediate feedback to developers in time zones where human reviewers are asleep. A developer in Singapore who opens a PR at 9 AM their time gets AI feedback immediately, rather than waiting for reviewers in San Francisco to wake up eight hours later. This reduces the "waiting for review" blocking time that is one of the most common complaints in distributed teams. Getting senior engineers invested is the fastest path to broad adoption. Find one senior engineer who is willing to champion the tool for a month — to read AI output carefully, respond to AI comments in the PR thread, and tell the team when AI caught something they would have missed. This social proof is more powerful than any number of statistics. Onboarding new engineers is simpler: for someone learning your codebase and your team's conventions, AI review is an immediate resource. It flags when their code deviates from team patterns before they face a human review. New engineers often become the strongest advocates for AI review because it gives them faster feedback with lower social friction than asking a senior colleague to explain why something is wrong. **Key Takeaways** - Deploy after a team conversation, not before — frame it as augmentation, not surveillance - Run an explicit two-week trial with feedback collection and share results publicly - Address the noise objection with configuration, not dismissal - Never use AI review data as a people measurement tool - Senior engineer champions drive adoption faster than any policy or mandate **Practical Exercise** Before deploying AI review to your team, write a one-page document that explains: what the tool does, what it does not do, how to interpret its output, and how to give feedback when it's wrong. Share this document before the tool goes live. Track how often team members engage with (resolve or respond to) AI comments in the first month. That engagement rate is your leading indicator of whether adoption is working. --- ## Chapter 9: Measuring Review Quality Over Time {#chapter-9} You cannot improve what you don't measure. Most teams run AI code review for a month, have a vague sense that it's "kind of useful," and never systematically improve it. This chapter gives you the measurement framework to move from "kind of useful" to quantifiably better. There are four metrics worth tracking. Understand what each measures and what it doesn't before you start collecting. **Review coverage** is the percentage of PRs that received at least one AI review comment. This is a baseline metric — low coverage means the tool isn't running or is being bypassed, not that it's working well. Target 95%+ coverage on all PRs above a minimum size threshold. **Actionability rate** is the percentage of AI comments that developers acted on — either fixing the flagged issue or explicitly rejecting it with a documented reason. This measures whether AI output is relevant. An actionability rate below 40% means your prompts are producing too much noise. Above 70% is excellent. **Defect escape rate** is the percentage of bugs found in production that could have been caught by static analysis. This is the hardest metric to collect but the most meaningful. It requires a bug post-mortem process that includes asking "could AI review have caught this?" for each production incident. **Review cycle time** is how long it takes from PR opened to PR merged. AI review should reduce this by catching mechanical issues before human review starts, so human reviewers spend less time on iteration cycles. If cycle time increases after AI review deployment, something is wrong — either the tool is creating noise that causes unnecessary back-and-forth, or the review-commenting workflow is adding friction. Here is a Python script that extracts these metrics from GitHub's API: ```python #!/usr/bin/env python3 # review_metrics.py from github import Github from datetime import datetime, timedelta import os import statistics def get_pr_metrics(token: str, repo_name: str, days: int = 30) -> dict: g = Github(token) repo = g.get_repo(repo_name) since = datetime.utcnow() - timedelta(days=days) prs = list(repo.get_pulls(state="closed", sort="updated", direction="desc")) recent_prs = [pr for pr in prs if pr.merged_at and pr.merged_at > since] total_prs = len(recent_prs) prs_with_ai_review = 0 ai_comment_counts = [] cycle_times = [] for pr in recent_prs: reviews = list(pr.get_reviews()) ai_reviews = [r for r in reviews if "🤖 AI Review" in (r.body or "")] if ai_reviews: prs_with_ai_review += 1 comments = list(pr.get_review_comments()) ai_comments = [c for c in comments if c.user.login == "github-actions[bot]"] ai_comment_counts.append(len(ai_comments)) if pr.merged_at and pr.created_at: cycle_hours = (pr.merged_at - pr.created_at).total_seconds() / 3600 cycle_times.append(cycle_hours) return { "period_days": days, "total_prs": total_prs, "coverage_pct": round(prs_with_ai_review / total_prs * 100, 1) if total_prs else 0, "avg_ai_comments": round(statistics.mean(ai_comment_counts), 1) if ai_comment_counts else 0, "median_cycle_hours": round(statistics.median(cycle_times), 1) if cycle_times else 0, } if __name__ == "__main__": token = os.environ["GITHUB_TOKEN"] repo = os.environ["GITHUB_REPOSITORY"] metrics = get_pr_metrics(token, repo, days=30) print(f"AI Review Metrics — Last {metrics['period_days']} days") print(f" PRs analyzed: {metrics['total_prs']}") print(f" Coverage: {metrics['coverage_pct']}%") print(f" Avg AI comments/PR: {metrics['avg_ai_comments']}") print(f" Median cycle time: {metrics['median_cycle_hours']}h") ``` > **Try This:** Run this script weekly and track trends over time. Put the output in a shared channel or dashboard. A rising actionability rate over the first three months is the clearest signal that your prompts are improving. A rising cycle time is the clearest signal that AI review is creating friction rather than reducing it. Qualitative measurement matters as much as quantitative. Once a month, ask three engineers a direct question: "Did AI review catch anything useful in the last month that human review might have missed?" Keep a log of the answers. If engineers are consistently saying yes, the tool is earning its place. If they consistently say no, the tool is not configured correctly for your codebase. The defect escape metric requires the most discipline to collect. In your post-mortem template, add a standard field: ```markdown ## Post-Mortem: [Incident Name] ...existing fields... ## AI Review Assessment - Could this bug have been caught by static analysis? [Yes/No/Maybe] - If yes, what pattern would have identified it? - Did AI review run on the PR that introduced this bug? [Yes/No] - If yes, did it flag this issue? [Yes/No/NotApplicable] ``` Over time, this data answers the most important question about AI review: is it actually reducing the defects that reach production? If your defect escape rate for statically-analyzable bugs trends down, the tool is doing its job. If it stays flat, you need better prompts focused on the specific patterns that match your historical bugs. > **Key Insight:** Measure review quality, not review volume. The number of AI comments per PR is a vanity metric. The defect escape rate and the actionability rate are the metrics that tell you whether the tool is making your software better. Set a quarterly review cadence: look at the metrics, read the qualitative feedback, update the prompts, and share the results with the team. Treating AI review as an ongoing practice rather than a set-and-forget deployment is what separates teams that sustain value from teams that abandon the tool in six months. **Key Takeaways** - Track four metrics: coverage, actionability rate, defect escape rate, and cycle time - Actionability rate below 40% means prompts need work; above 70% is excellent - Add AI review assessment to post-mortem templates to measure defect escape - Qualitative monthly check-ins with engineers are as valuable as quantitative metrics - Set a quarterly prompt review cycle; treat it like a standing improvement sprint **Practical Exercise** Run the metrics script above against your repository for the past 30 days. If AI review hasn't been running, set the baseline before you deploy. After one month of AI review, compare. Write down three hypotheses for what's causing any differences you observe. Test one of those hypotheses by making a specific prompt change and measuring the effect over the next two weeks. --- ## Conclusion {#conclusion} Code review is a quality lever that most teams pull inconsistently. Some PRs get exhaustive review. Others get rubber-stamped because the reviewer is overloaded, unfamiliar with the code, or just trying to get through their queue before the standup. This inconsistency is where defects hide. AI review doesn't eliminate inconsistency — the human layer will always vary. But it adds a consistent baseline that runs on every PR, every time, regardless of time zone, workload, or how well the reviewer slept. That consistent baseline is the primary value. Everything else — the security catches, the style enforcement, the documentation of your team's conventions — is downstream of that floor-raising function. The teams that get the most from AI review are the ones that treat it as infrastructure, not a feature. They configure it carefully, measure it systematically, improve it over time, and integrate it into their culture the same way they integrated code review itself: as something expected and valuable, not something imposed and tolerated. Getting to that state takes three to six months. In the first month, you're configuring the tool and fighting noise. In the second month, you're tuning prompts and seeing the actionability rate rise. By the third month, engineers are reading AI output as a natural part of their PR process. By the sixth month, you have data showing whether defect escape rates are trending down, and you're making evidence-based decisions about where to focus next. The skills you've built in this guide — prompt engineering, output filtering, metric collection, workflow integration — transfer beyond code review. The same pattern applies to AI-assisted documentation review, architecture review, and security auditing. Once you understand how to configure an AI reviewer for your specific context, the technique generalizes. A few final principles to carry forward: **Stay skeptical of every specific claim.** AI output is probabilistic. A CRITICAL flag means "the model thinks this is likely a serious issue," not "this is definitely a bug." Verify before acting. **Keep humans in the loop on judgment.** AI review handles pattern matching. Humans handle intent. Architecture, business logic, and strategic decisions require engineers who understand why the code exists, not just what it does. **Improve continuously.** A prompt that works well for your codebase today will drift as your codebase evolves. Add new standards when you add new conventions. Update exclusions when patterns change. Treat your prompts like living code. **Measure the right things.** Coverage is vanity. Actionability and defect escape are signal. Optimize for the metrics that connect to code quality, not the ones that look impressive on a dashboard. The goal has never been to review more code faster. It's to ship better software. AI review is one tool in that direction. Use it deliberately, measure it honestly, and keep the humans in charge. --- ## Appendix A: Glossary {#appendix-a} **Actionability Rate** The percentage of AI review comments that developers act on — either by fixing the flagged issue or explicitly rejecting it with a documented reason. A measure of signal quality. **Callout Severity** A label applied to AI review comments to indicate urgency: CRITICAL (security/data loss risk), HIGH (likely bug or standards violation), MEDIUM (judgment call), LOW (minor improvement). **Chain-of-Thought Prompting** A prompting technique that instructs the model to reason through a problem before producing output. Produces more coherent analysis for complex tasks like security review. **Coverage** The percentage of PRs that received at least one AI review pass. A baseline operational metric, not a quality metric. **Defect Escape Rate** The percentage of bugs found in production that were not caught in review. Used to measure whether AI review is reducing the classes of defects it is capable of catching. **Diff** The textual representation of changes between two versions of code. AI reviewers typically analyze the diff rather than the full file. **Diff Position** GitHub's internal numbering system for lines in a pull request diff, used when posting inline review comments. Different from line number in the source file. **False Positive** An AI review comment that flags correct, intentional code as a problem. High false positive rates indicate prompts need refinement. **Inline Comment** A review comment attached to a specific file and line number in a pull request, as opposed to a general comment on the PR as a whole. **Large Language Model (LLM)** The type of AI model underlying AI code review tools. Generates text by predicting likely next tokens based on training data and the provided input. **Prompt** The instruction given to an AI model. In code review, it includes role definition, focus areas, exclusions, and output format instructions. **Prompt Temperature** A parameter controlling the randomness of AI output. Temperature 0 produces the most deterministic output; higher values produce more varied responses. Set to 0 for review tasks. **Review Standards Document** A versioned file in your repository that documents team-specific coding conventions used as context in AI review prompts. **Static Analysis** Code analysis performed without executing the code. AI review is a form of static analysis with a natural language interface. **Status Check** A required or optional signal on a GitHub pull request that must pass before merging. AI review can be configured as a non-blocking required status check. **Webhook** An HTTP callback that fires when an event occurs. Many AI review tools use repository webhooks to trigger reviews when PRs are opened or updated. --- ## Appendix B: Tools and Resources {#appendix-b} ### AI Review Platforms **GitHub Copilot Code Review** Native GitHub integration. Requires GitHub Copilot Enterprise subscription. Reviews PRs automatically with configurable focus areas. Best for teams already on GitHub Enterprise. **CodeRabbit** Standalone AI review tool with GitHub, GitLab, and Bitbucket integrations. Configurable via `.coderabbit.yaml`. Free tier available for open source. Strongest out-of-the-box configuration options. **Qodo (formerly CodiumAI)** Focus on test generation alongside code review. Strong for teams with test coverage goals. IDE plugin available in addition to PR integration. **Sourcery** Python-focused AI review tool with GitHub integration and CLI. Good for teams with Python-heavy codebases. Integrates with `pre-commit` hooks. **Ellipsis** AI review with Jira and Linear ticket integration. Strong for teams that track code review against tickets. ### AI APIs for Custom Integrations **Anthropic Claude API** Used in the examples throughout this guide. Claude claude-opus-4-7 produces strong code review output. Python and TypeScript SDKs available. Requires `ANTHROPIC_API_KEY`. Installation: ```bash pip install anthropic ``` **OpenAI API** GPT-4o produces comparable code review output to Claude for most use cases. Well-documented Python SDK. Requires `OPENAI_API_KEY`. ```bash pip install openai ``` ### Supporting Tools **PyGitHub** Python library for the GitHub REST API. Used in the integration scripts in this guide for posting inline review comments. ```bash pip install PyGithub ``` **pre-commit** Framework for managing git pre-commit hooks. Useful for running AI review or linters before local commits. Configuration via `.pre-commit-config.yaml`. ```bash pip install pre-commit pre-commit install ``` **actionlint** Static checker for GitHub Actions workflow files. Run this on your AI review workflow before deploying to catch common CI configuration errors. ```bash # Install via GitHub releases or homebrew brew install actionlint actionlint .github/workflows/ai-review.yml ``` ### Supplementary Libraries ```bash # For parsing and processing code diffs programmatically pip install whatthepatch unidiff # For structured logging in review scripts pip install structlog # For rate limiting API calls in high-volume repositories pip install tenacity ``` --- ## Appendix C: Further Reading {#appendix-c} ### On Code Review Practice **"Code Review Best Practices" — Palantir Engineering Blog** A detailed treatment of code review philosophy, scope, and mechanics. Particularly strong on the difference between blocking and non-blocking feedback — a distinction that matters when integrating AI review. **"How to Do Code Review" — Google Engineering Practices** Google's internal code review guide, published publicly. Covers review speed, what to look for, the CL author's perspective, and handling pushback. The section on navigating disagreements applies directly to AI vs. human reviewer conflicts. **"Helping Yourself by Helping Reviewers" — the Morning Paper (summary of research)** Academic research on what makes code reviews effective. The finding that reviewers catch fewer defects as diff size increases is directly relevant to AI review scope decisions. ### On Prompt Engineering **"Prompt Engineering Guide" — promptingguide.ai** Comprehensive resource on prompting techniques including chain-of-thought, few-shot, and structured output. The sections on structured output and role prompting apply directly to review prompt design. **Anthropic Documentation: "Prompt Engineering Overview"** Official documentation on prompting Claude effectively. Covers system prompts, formatting guidance, and avoiding common failure modes. Required reading before writing production review prompts. ### On Secure Code Review **"OWASP Code Review Guide"** The definitive reference for what to check in security-focused code review. Organized by vulnerability type. Use this to build your security-focused prompt's focus area list. **"The Art of Software Security Assessment" — Dowd, McDonald, Schuh** Deep reference on source code security review methodology. Chapter 2 on the code review process and Chapter 7 on auditing design apply even when the reviewer is AI-assisted. ### On Measurement and Quality **"Accelerate: The Science of Lean Software and DevOps" — Forsgren, Humble, Kim** The empirical research behind DORA metrics. Change failure rate and mean time to restore are the production-level metrics that connect to defect escape rate. Understanding this context makes the measurement framework in Chapter 9 more meaningful. **"How Effective Is Code Review?" — Fagan, 1976 (via ACM Digital Library)** The original inspection research. Dated in tooling but foundational in methodology. The finding that code review effectiveness depends heavily on reviewer preparation and scope is as relevant to AI review configuration as it was to human review in 1976. --- *© 2026 Pyckle. All rights reserved. This guide may be shared freely for personal and educational use. Commercial reproduction or redistribution requires written permission. Contact kellyprice@pyckle.co.*