cli-e2e-testing

Aspire CLI End-to-End Testing with Hex1b

This skill provides patterns and practices for writing end-to-end tests for the Aspire CLI using the Hex1b terminal automation library.

Overview

CLI E2E tests use the Hex1b library to automate terminal sessions, simulating real user interactions with the Aspire CLI. Tests run in CI with asciinema recordings for debugging.

Location: tests/Aspire.Cli.EndToEnd.Tests/

Supported Platforms: Linux only. Hex1b requires a Linux terminal environment. Tests are configured to skip on Windows and macOS in CI.

Key Components

Core Classes

• Hex1bTerminal: The main terminal class from the Hex1b library for terminal automation
• Hex1bTerminalAutomator: Async/await API for driving a Hex1bTerminal — the preferred approach for new tests
• Hex1bAutomatorTestHelpers (shared helpers): Async extension methods on Hex1bTerminalAutomator (WaitForSuccessPromptAsync, AspireNewAsync, etc.)
• CliE2EAutomatorHelpers (Helpers/CliE2EAutomatorHelpers.cs): CLI-specific async extension methods on Hex1bTerminalAutomator (PrepareDockerEnvironmentAsync, InstallAspireCliInDockerAsync, etc.)
• CellPatternSearcher: Pattern matching for terminal cell content
• SequenceCounter (Helpers/SequenceCounter.cs): Tracks command execution count for deterministic prompt detection
• CliE2ETestHelpers (Helpers/CliE2ETestHelpers.cs): Environment variable helpers and terminal factory methods
• TemporaryWorkspace: Creates isolated temporary directories for test execution
• Hex1bTerminalInputSequenceBuilder (legacy): Fluent builder API for building sequences of terminal input/output operations. Prefer Hex1bTerminalAutomator for new tests.

Test Architecture

Each test:

1. Creates a TemporaryWorkspace for isolation
2. Builds a Hex1bTerminal with headless mode and asciinema recording
3. Creates a Hex1bTerminalAutomator wrapping the terminal
4. Drives the terminal with async/await calls and awaits completion

Test Structure

public sealed class SmokeTests(ITestOutputHelper output)
{
    [Fact]
    public async Task MyCliTest()
    {
        var workspace = TemporaryWorkspace.Create(output);
        var installMode = CliE2ETestHelpers.DetectDockerInstallMode();

        using var terminal = CliE2ETestHelpers.CreateDockerTestTerminal();
        var pendingRun = terminal.RunAsync(TestContext.Current.CancellationToken);

        var counter = new SequenceCounter();
        var auto = new Hex1bTerminalAutomator(terminal, defaultTimeout: TimeSpan.FromSeconds(500));

        await auto.PrepareDockerEnvironmentAsync(counter, workspace);
        await auto.InstallAspireCliInDockerAsync(installMode, counter);

        await auto.TypeAsync("aspire --version");
        await auto.EnterAsync();
        await auto.WaitForSuccessPromptAsync(counter);

        await auto.TypeAsync("exit");
        await auto.EnterAsync();
        await pendingRun;
    }
}

SequenceCounter and Prompt Detection

The SequenceCounter class tracks the number of shell commands executed. This enables deterministic waiting for command completion via a custom shell prompt.

How It Works

1. PrepareDockerEnvironmentAsync() configures the shell with a custom prompt: [N OK] $ or [N ERR:code] $
2. Each command increments the counter
3. WaitForSuccessPromptAsync(counter) waits for a prompt showing the current count with OK

var counter = new SequenceCounter();
var auto = new Hex1bTerminalAutomator(terminal, defaultTimeout: TimeSpan.FromSeconds(500));

await auto.PrepareDockerEnvironmentAsync(counter, workspace);  // Sets up prompt, counter starts at 1

await auto.TypeAsync("echo hello");
await auto.EnterAsync();
await auto.WaitForSuccessPromptAsync(counter);  // Waits for "[1 OK] $ ", then increments to 2

await auto.TypeAsync("ls -la");
await auto.EnterAsync();
await auto.WaitForSuccessPromptAsync(counter);  // Waits for "[2 OK] $ ", then increments to 3

await auto.TypeAsync("exit");
await auto.EnterAsync();

This approach is more reliable than arbitrary timeouts because it deterministically waits for each command to complete.

Pattern Searching with CellPatternSearcher

Use CellPatternSearcher to find text patterns in terminal output:

// Simple text search (literal string matching - PREFERRED)
var waitingForPrompt = new CellPatternSearcher()
    .Find("Enter the project name");

// Literal string with special characters (use Find, not FindPattern!)
var waitingForTemplate = new CellPatternSearcher()
    .Find("> Starter App (FastAPI/React)");  // Parentheses and slashes are literal

// Regex pattern (only when you need wildcards/regex features)
var waitingForAnyStarter = new CellPatternSearcher()
    .FindPattern("> Starter App.*");  // .* matches anything

// Chained patterns (find "b", then scan right until "$", then right of " ")
var waitingForShell = new CellPatternSearcher()
    .Find("b").RightUntil("$").Right(' ').Right(' ');

// Use in WaitUntilAsync
await auto.WaitUntilAsync(
    snapshot => waitingForPrompt.Search(snapshot).Count > 0,
    TimeSpan.FromSeconds(30),
    description: "waiting for prompt");

Find vs FindPattern

• Find(string): Literal string matching. Use this for most cases.
• FindPattern(string): Regex pattern matching. Use only when you need regex features like wildcards.

Important: If your search string contains regex special characters like (, ), /, ., *, +, ?, [, ], {, }, ^, $, |, or \, use Find() instead of FindPattern() to avoid regex interpretation.

Extension Methods

Hex1bAutomatorTestHelpers Extensions (Shared — Automator API)

Method	Description
WaitForSuccessPromptAsync(counter, timeout?)	Waits for [N OK] $ prompt and increments counter
WaitForAnyPromptAsync(counter, timeout?)	Waits for any prompt (OK or ERR) and increments counter
WaitForErrorPromptAsync(counter, timeout?)	Waits for [N ERR:code] $ prompt and increments counter
WaitForSuccessPromptFailFastAsync(counter, timeout?)	Waits for success prompt, fails immediately if error prompt appears
DeclineAgentInitPromptAsync()	Declines the aspire agent init prompt if it appears
AspireNewAsync(projectName, counter, template?, useRedisCache?)	Runs aspire new interactively, handling template selection, project name, output path, URLs, Redis, and test project prompts

See AspireNew Helper below for detailed usage.

CliE2EAutomatorHelpers Extensions on Hex1bTerminalAutomator

Method	Description
PrepareDockerEnvironmentAsync(counter, workspace)	Sets up Docker container environment with custom prompt and command tracking
InstallAspireCliInDockerAsync(installMode, counter)	Installs the Aspire CLI inside the Docker container
ClearScreenAsync(counter)	Clears the terminal screen and waits for prompt

SequenceCounterExtensions

Method	Description
IncrementSequence(counter)	Manually increments the counter

Legacy Builder Extensions

The following extensions on Hex1bTerminalInputSequenceBuilder are still available but should not be used in new tests:

Method	Description
WaitForSuccessPrompt(counter, timeout?)	(legacy) Waits for [N OK] $ prompt and increments counter
PrepareEnvironment(workspace, counter)	(legacy) Sets up custom prompt with command tracking
InstallAspireCliFromPullRequest(prNumber, counter)	(legacy) Downloads and installs CLI from PR artifacts
SourceAspireCliEnvironment(counter)	(legacy) Adds ~/.aspire/bin to PATH

DO: Use CellPatternSearcher for Output Detection

Wait for specific output patterns rather than arbitrary delays:

var waitingForMessage = new CellPatternSearcher()
    .Find("Project created successfully.");

await auto.TypeAsync("aspire new");
await auto.EnterAsync();
await auto.WaitUntilAsync(
    s => waitingForMessage.Search(s).Count > 0,
    TimeSpan.FromMinutes(2),
    description: "waiting for project created message");

DO: Use WaitForSuccessPromptAsync After Commands

After running shell commands, use WaitForSuccessPromptAsync() to wait for the command to complete:

await auto.TypeAsync("dotnet build");
await auto.EnterAsync();
await auto.WaitForSuccessPromptAsync(counter);  // Waits for prompt, verifies success

await auto.TypeAsync("dotnet run");
await auto.EnterAsync();
await auto.WaitForSuccessPromptAsync(counter);

AspireNew Helper

The AspireNew extension method centralizes the multi-step aspire new interactive flow. Use it instead of manually building the prompt sequence.

AspireTemplate Enum

Value	Template	Arrow Keys
Starter (default)	Starter App (Blazor)	None (first option)
JsReact	Starter App (ASP.NET Core/React)	Down ×1
PythonReact	Starter App (FastAPI/React)	Down ×2
ExpressReact	Starter App (Express/React)	Down ×3
EmptyAppHost	Empty AppHost	Down ×4

Parameters

Parameter	Default	Description
projectName	(required)	Project name typed at the prompt
counter	(required)	SequenceCounter for prompt tracking
template	AspireTemplate.Starter	Which template to select
useRedisCache	true	Accept Redis (Enter) or decline (Down+Enter). Only applies to Starter, JsReact, PythonReact.

Usage Examples

// Starter template with defaults (Redis=Yes, TestProject=No)
await auto.AspireNewAsync("MyProject", counter);

// Starter template, no Redis
await auto.AspireNewAsync("MyProject", counter, useRedisCache: false);

// JsReact template, no Redis
await auto.AspireNewAsync("MyProject", counter, t

---

*Content truncated.*