ci-analysis

Azure DevOps and Helix CI Analysis

Analyze CI build status and test failures in Azure DevOps and Helix for dotnet repositories (runtime, sdk, aspnetcore, roslyn, and more).

🚨 NEVER use gh pr review --approve or --request-changes. Only --comment is allowed. Approval and blocking are human-only actions.

📝 AI-generated content disclosure: When posting any content to GitHub (PR comments, retry commands, analysis summaries) under a user's credentials — i.e., the account is not a dedicated "copilot" or "bot" account/app — you MUST include a concise, visible note (e.g. a > [!NOTE] alert) indicating the content was AI/Copilot-generated. Skip this if the user explicitly asks you to omit it.

Workflow: Gather PR context (Step 0) → run the script → read the human-readable output + [CI_ANALYSIS_SUMMARY] JSON → synthesize recommendations yourself. The script collects data; you generate the advice. For supplementary investigation beyond the script, MCP tools (AzDO, Helix, GitHub) provide structured access when available; the script and gh CLI work independently when they're not.

When to Use This Skill

Use this skill when:

• Checking CI status on a PR ("is CI passing?", "what's the build status?", "why is CI red?")
• Investigating CI failures or checking why a PR's tests are failing
• Determining if a PR is ready to merge based on CI results
• Debugging Helix test issues or analyzing build errors
• Given URLs containing dev.azure.com, helix.dot.net, or GitHub PR links with failing checks
• Asked questions like "why is this PR failing", "analyze the CI", "is CI green", "retry CI", "rerun tests", or "test failures"
• Investigating canceled or timed-out jobs for recoverable results

Script Limitations

The Get-CIStatus.ps1 script targets Azure DevOps + Helix infrastructure specifically. It won't help with:

• GitHub Actions workflows (different API, different log format)
• Repos not using Helix for test distribution (no Helix work items to query)
• Pure build performance questions (use MSBuild binlog analysis instead)

However, the analysis patterns in this skill (interpreting failures, correlating with PR changes, distinguishing infrastructure vs. code issues) apply broadly even outside AzDO/Helix.

Quick Start

# Analyze PR failures (most common) - defaults to dotnet/runtime
./scripts/Get-CIStatus.ps1 -PRNumber 123445 -ShowLogs

# Analyze by build ID
./scripts/Get-CIStatus.ps1 -BuildId 1276327 -ShowLogs

# Query specific Helix work item
./scripts/Get-CIStatus.ps1 -HelixJob "4b24b2c2-..." -WorkItem "System.Net.Http.Tests"

# Other dotnet repositories
./scripts/Get-CIStatus.ps1 -PRNumber 12345 -Repository "dotnet/aspnetcore"
./scripts/Get-CIStatus.ps1 -PRNumber 67890 -Repository "dotnet/sdk"
./scripts/Get-CIStatus.ps1 -PRNumber 11111 -Repository "dotnet/roslyn"

Key Parameters

Parameter	Description
-PRNumber	GitHub PR number to analyze
-BuildId	Azure DevOps build ID
-ShowLogs	Fetch and display Helix console logs
-Repository	Target repo (default: dotnet/runtime)
-MaxJobs	Max failed jobs to show (default: 5)
-SearchMihuBot	Search MihuBot for related issues

Three Modes

The script operates in three distinct modes depending on what information you have:

You have...	Use	What you get
A GitHub PR number	-PRNumber 12345	Full analysis: all builds, failures, known issues, structured JSON summary
An AzDO build ID	-BuildId 1276327	Single build analysis: timeline, failures, Helix results
A Helix job ID (optionally a specific work item)	-HelixJob "..." [-WorkItem "..."]	Deep dive: list work items for the job, or with -WorkItem, focus on a single work item's console logs, artifacts, and test results

❌ Don't guess the mode. If the user gives a PR URL, use -PRNumber. If they paste an AzDO build link, extract the build ID. If they reference a specific Helix job, use -HelixJob.

What the Script Does

PR Analysis Mode (-PRNumber)

1. Discovers AzDO builds associated with the PR (from GitHub check status; for full build history, query AzDO builds on refs/pull/{PR}/merge branch)
2. Fetches Build Analysis for known issues
3. Gets failed jobs from Azure DevOps timeline
4. Separates canceled jobs from failed jobs (canceled may be dependency-canceled or timeout-canceled)
5. Extracts Helix work item failures from each failed job
6. Fetches console logs (with -ShowLogs)
7. Searches for known issues with "Known Build Error" label
8. Correlates failures with PR file changes
9. Emits structured summary — [CI_ANALYSIS_SUMMARY] JSON block with all key facts for the agent to reason over

After the script runs, you (the agent) generate recommendations. The script collects data; you synthesize the advice. See Generating Recommendations below.

Build ID Mode (-BuildId)

1. Fetches the build timeline directly (skips PR discovery)
2. Performs steps 3–7 from PR Analysis Mode, but does not fetch Build Analysis known issues or correlate failures with PR file changes (those require a PR number). Still emits [CI_ANALYSIS_SUMMARY] JSON.

Helix Job Mode (-HelixJob [and optional -WorkItem])

1. With -HelixJob alone: enumerates work items for the job and summarizes their status
2. With -HelixJob and -WorkItem: queries the specific work item for status and artifacts
3. Fetches console logs and file listings, displays detailed failure information

Interpreting Results

Known Issues section: Failures matching existing GitHub issues - these are tracked and being investigated.

Build Analysis check status: The "Build Analysis" GitHub check is green only when every failure is matched to a known issue. If it's red, at least one failure is unaccounted for — do NOT claim "all failures are known issues" just because some known issues were found. You must verify each failing job is covered by a specific known issue before calling it safe to retry.

Canceled/timed-out jobs: Jobs canceled due to earlier stage failures or AzDO timeouts. Dependency-canceled jobs don't need investigation. Timeout-canceled jobs may have all-passing Helix results — the "failure" is just the AzDO job wrapper timing out, not actual test failures. To verify: use hlx_status on each Helix job in the timed-out build (include passed work items). If all work items passed, the build effectively passed.

❌ Don't dismiss timed-out builds. A build marked "failed" due to a 3-hour AzDO timeout can have 100% passing Helix work items. Check before concluding it failed.

PR Change Correlation: Files changed by PR appearing in failures - likely PR-related.

Build errors: Compilation failures need code fixes.

Helix failures: Test failures on distributed infrastructure.

Local test failures: Some repos (e.g., dotnet/sdk) run tests directly on build agents. These can also match known issues - search for the test name with the "Known Build Error" label.

Per-failure details (failedJobDetails in JSON): Each failed job includes errorCategory, errorSnippet, and helixWorkItems. Use these for per-job classification instead of applying a single recommendationHint to all failures.

Error categories: test-failure, build-error, test-timeout, crash (exit codes 139/134/-4), tests-passed-reporter-failed (all tests passed but reporter crashed — genuinely infrastructure), unclassified (investigate manually).

⚠️ crash does NOT always mean tests failed. Exit code -4 often means the Helix work item wrapper timed out after tests completed. Always check testResults.xml before concluding a crash is a real failure. See Recovering Results from Crashed/Canceled Jobs.

⚠️ Be cautious labeling failures as "infrastructure." Only conclude infrastructure with strong evidence: Build Analysis match, identical failure on target branch, or confirmed outage. Exception: tests-passed-reporter-failed is genuinely infrastructure.

❌ Missing packages on flow PRs ≠ infrastructure. Flow PRs can cause builds to request different packages. Check which package and why before assuming feed delay.

Recovering Results from Crashed/Canceled Jobs

When an AzDO job is canceled (timeout) or Helix work items show Crash (exit code -4), the tests may have actually passed. Follow this procedure:

1. Find the Helix job IDs — Read the AzDO "Send to Helix" step log and search for lines containing Sent Helix Job. Extract the job GUIDs.

2. Check Helix job status — Get pass/fail summary for each job. Look at failedCount vs passedCount.

3. For work items marked Crash/Failed — Check if tests actually passed despite the crash. Try structured test results first (TRX parsing), then search for pass/fail counts in result files without downloading, then download as last resort:

   • Parse the XML: total, passed, failed attributes on the <assembly> element
   • If failed=0 and passed > 0, the tests passed — the "crash" is the wrapper timing out after test completion

4. Verdict:

   • All work items passed or crash-with-passing-results → Tests effectively passed. The failure is infrastructure (wrapper timeout).
   • Some work items have failed > 0 in testResults.xml → Real test failures. Investigate those specific tests.
   • No testResults.xml uploaded → Tests may not have run at all. Check console logs for errors.

This pattern is common with long-running test suites (e.g., WasmBuildTests) where tests complete but the Helix work item wrapper exceeds its timeout during result upload or cleanup.

Generating Recommendations

After the script outputs the [CI_ANALYSIS_SUMMARY] JSON block, you synthesize recommendations. Do not parrot

---

Content truncated.