coderabbit-prod-checklist

CodeRabbit Production Checklist

Overview

Complete checklist for deploying CodeRabbit as a production merge gate. Covers the transition from "optional AI review" to "required status check that blocks merges." Ensures configuration is tuned, team is onboarded, and fallback procedures are documented.

Prerequisites

• CodeRabbit installed and running in non-blocking mode for 1-2 weeks
• Team has seen CodeRabbit reviews and provided feedback
• .coderabbit.yaml tuned based on pilot feedback
• GitHub organization admin access

Instructions

Step 1: Pre-Launch Configuration Audit

# Run through each item before going live:

## App Installation
- [ ] CodeRabbit GitHub App installed on all target repos
- [ ] Repository access scoped correctly (all repos vs select)
- [ ] Seat assignment policy set (active committers vs manual)
- [ ] Bot accounts excluded from seats (dependabot, renovate)

## Configuration
- [ ] `.coderabbit.yaml` committed to main branch
- [ ] YAML syntax validated (no parse errors)
- [ ] `auto_review.enabled: true`
- [ ] `auto_review.drafts: false` (skip draft PRs)
- [ ] `base_branches` includes all target branches (main, develop)
- [ ] `path_filters` excludes generated/lock/vendor files
- [ ] `path_instructions` configured for key directories
- [ ] `chat.auto_reply: true`

## Review Behavior
- [ ] Profile set to "assertive" (or team's preferred level)
- [ ] `request_changes_workflow` decision documented:
  - `true`: CodeRabbit blocks merge when issues found
  - `false`: CodeRabbit only comments (non-blocking)
- [ ] Org-level defaults in `.github/.coderabbit.yaml` if multi-repo

Step 2: Validate Configuration

set -euo pipefail
echo "=== CodeRabbit Production Readiness Check ==="

# 1. YAML syntax
echo "--- Config Validation ---"
if [ -f .coderabbit.yaml ]; then
  python3 -c "
import yaml, sys
try:
    config = yaml.safe_load(open('.coderabbit.yaml'))
    print('YAML syntax: PASS')

    reviews = config.get('reviews', {})
    auto = reviews.get('auto_review', {})

    checks = {
        'auto_review.enabled': auto.get('enabled', False) == True,
        'auto_review.drafts': auto.get('drafts', True) == False,
        'path_filters configured': len(reviews.get('path_filters', [])) > 0,
        'path_instructions configured': len(reviews.get('path_instructions', [])) > 0,
        'chat.auto_reply': config.get('chat', {}).get('auto_reply', False) == True,
    }

    for name, passed in checks.items():
        status = 'PASS' if passed else 'WARN'
        print(f'  {name}: {status}')

    if all(checks.values()):
        print('Overall: READY FOR PRODUCTION')
    else:
        print('Overall: Review warnings above before going live')

except yaml.YAMLError as e:
    print(f'YAML syntax: FAIL ({e})')
    sys.exit(1)
" 2>&1
else
  echo "FAIL: .coderabbit.yaml not found"
fi

Step 3: Verify Review History

set -euo pipefail
OWNER="${1:-your-org}"
REPO="${2:-your-repo}"

echo ""
echo "--- Review History Check ---"
REVIEWED=0
TOTAL=0

for PR_NUM in $(gh api "repos/$OWNER/$REPO/pulls?state=all&per_page=20" --jq '.[].number'); do
  TOTAL=$((TOTAL + 1))
  CR=$(gh api "repos/$OWNER/$REPO/pulls/$PR_NUM/reviews" \
    --jq '[.[] | select(.user.login=="coderabbitai[bot]")] | length' 2>/dev/null || echo "0")
  [ "$CR" -gt 0 ] && REVIEWED=$((REVIEWED + 1))
done

echo "Coverage: $REVIEWED/$TOTAL PRs reviewed by CodeRabbit"
if [ "$TOTAL" -gt 0 ] && [ "$((REVIEWED * 100 / TOTAL))" -lt 80 ]; then
  echo "WARNING: Coverage below 80%. Check base_branches and ignore_title_keywords."
else
  echo "Coverage: PASS"
fi

Step 4: Team Readiness Checklist

## Team Onboarding
- [ ] Team briefed on CodeRabbit (what it does, how to interact)
- [ ] Quick reference shared:
  - `@coderabbitai full review` — re-review from scratch
  - `@coderabbitai summary` — regenerate walkthrough
  - `@coderabbitai resolve` — mark all comments resolved
  - `@coderabbitai help` — list all commands
- [ ] Team knows to reply to comments to train learnings
- [ ] "WIP" in PR title skips review (documented)
- [ ] Escalation path defined for false positives

## Fallback Procedures
- [ ] Admin merge bypass documented (for emergencies)
- [ ] Process for temporarily disabling CodeRabbit:
  1. Remove from branch protection required checks
  2. Set `auto_review.enabled: false` in config
  3. Or uninstall the GitHub App from the repo
- [ ] Contact info for CodeRabbit support documented

Step 5: Enable as Required Check

set -euo pipefail
OWNER="${1:-your-org}"
REPO="${2:-your-repo}"

echo ""
echo "--- Enabling CodeRabbit as Required Check ---"

# Update branch protection to require CodeRabbit
gh api "repos/$OWNER/$REPO/branches/main/protection" \
  --method PUT \
  --field 'required_status_checks={"strict":true,"contexts":["coderabbitai"]}' \
  --field 'required_pull_request_reviews={"required_approving_review_count":1}' \
  --field 'enforce_admins=false' \
  --field 'restrictions=null'

echo "DONE: CodeRabbit is now a required check on $OWNER/$REPO main branch"
echo ""
echo "To revert (emergency):"
echo "  gh api repos/$OWNER/$REPO/branches/main/protection --method DELETE"

Step 6: Update .coderabbit.yaml for Production

# .coderabbit.yaml - Production configuration
language: "en-US"
early_access: false

reviews:
  profile: "assertive"
  request_changes_workflow: true      # NOW blocking (was false during pilot)
  high_level_summary: true
  high_level_summary_in_walkthrough: true
  review_status: true
  collapse_walkthrough: false
  sequence_diagrams: true
  poem: false

  auto_review:
    enabled: true
    drafts: false
    base_branches:
      - main
      - develop
    ignore_title_keywords:
      - "WIP"
      - "DO NOT MERGE"
      - "chore: bump"

  path_filters:
    - "!**/*.lock"
    - "!**/*.snap"
    - "!**/generated/**"
    - "!dist/**"
    - "!vendor/**"

  path_instructions:
    - path: "src/api/**"
      instructions: "Review for input validation, auth middleware, error handling."
    - path: "src/db/**"
      instructions: "Review for parameterized queries, transactions, N+1 patterns."
    - path: "**/*.test.*"
      instructions: "Review for edge cases and assertion completeness. Skip style."

chat:
  auto_reply: true

Step 7: Post-Launch Monitoring

# First 48 hours after go-live:

## Monitor:
- [ ] All PRs to main are getting CodeRabbit reviews
- [ ] No PRs blocked by CodeRabbit timeout (> 15 min)
- [ ] Team is not overwhelmed by review volume
- [ ] No legitimate emergency PRs blocked

## Week 1 review:
- [ ] Review coverage > 90%
- [ ] No team complaints about false positives
- [ ] Learnings being created from feedback
- [ ] Emergency bypass has not been needed (or was used correctly)

Output

• Configuration audited and validated
• Review history confirms adequate coverage
• Team onboarded with quick reference guide
• CodeRabbit enabled as required status check
• Fallback and emergency procedures documented
• Post-launch monitoring plan in place

Error Handling

Issue	Cause	Solution
PRs stuck waiting for review	CodeRabbit outage	Check status.coderabbit.ai; use admin bypass
All PRs blocked	request_changes_workflow: true too aggressive	Temporarily set to false
Team pushback	Too many comments	Switch to chill profile
Emergency PR blocked	Required check blocking	Admin merge bypass or remove required check
Config not loading	YAML error	Run @coderabbitai configuration to diagnose

Resources

• CodeRabbit Configuration Reference
• GitHub Branch Protection
• CodeRabbit Status Page

Next Steps

For ongoing monitoring, see coderabbit-observability.