development-tools

Development Tools

Development commands for the multigres project.

Command Structure

/mt-dev [test-type] [args...]

For Claude Code

When executing commands:

• Always run make build before integration tests
• Show the actual command being executed before running it
• Summarize test results (passed/failed counts, execution time)
• If tests fail, offer to show detailed output or logs
• For verbose output, provide a summary rather than dumping everything

---

Unit Tests

Unit tests are fast, isolated tests for individual functions and packages. They don't require external services or build artifacts. Note: plain go test ./go/... will also traverse go/test/endtoend/...; use -short for unit-focused runs.

Command Syntax

Run all unit tests:

/mt-dev unit all

Executes: go test -short ./go/...

Run specific package:

/mt-dev unit <package-path>

Examples:

• /mt-dev unit ./go/services/multipooler/... - All multipooler package tests
• /mt-dev unit ./go/multigateway/... - All multigateway package tests
• /mt-dev unit ./go/pgprotocol/... - All pgprotocol package tests

Run specific test:

/mt-dev unit <package-path> <TestName>

Examples:

• /mt-dev unit ./go/services/multipooler TestConnectionPool
• /mt-dev unit ./go/pgprotocol TestParseQuery

Run with pattern matching:

/mt-dev unit <package-path> <TestPattern>

Examples:

• /mt-dev unit ./go/services/multipooler TestConn.* - All tests starting with TestConn
• /mt-dev unit ./go/multigateway Test.*Route.* - All tests with "Route" in name

Common Flags

• -v - Verbose output (shows all test names as they run)
• -race - Enable race detector (slower, catches concurrency bugs)
• -cover - Show coverage percentage
• -coverprofile=coverage.out - Generate coverage report
• -count=N - Run tests N times (useful for flaky test detection); -count=1 also forces re-run and bypasses test cache
• -timeout=30s - Set timeout (default: 10m)
• -short - Skip long-running tests
• -parallel=N - Run N tests in parallel (default: GOMAXPROCS)

Examples

# Quick test run
/mt-dev unit all

# Verbose with race detection
/mt-dev unit ./go/services/multipooler/... -v -race

# Coverage report
/mt-dev unit ./go/pgprotocol/... -cover

# Test for flakiness
/mt-dev unit ./go/multigateway TestRouting -count=10

# Fast tests only
/mt-dev unit all -short

# Specific test with verbose output
/mt-dev unit ./go/services/multipooler TestConnectionPool -v

Natural Language Support

• "run unit tests" → /mt-dev unit all
• "test the multipooler package" → /mt-dev unit ./go/services/multipooler/...
• "run TestConnectionPool" → /mt-dev unit ./go/services/multipooler TestConnectionPool
• "run all unit tests with coverage" → /mt-dev unit all -cover
• "check for race conditions in multigateway" → /mt-dev unit ./go/multigateway/... -race

---

Integration Tests

Integration tests are end-to-end tests that start real components (MultiGateway, MultiPooler, PostgreSQL) and test their interactions. These tests are slower and require building the project first.

IMPORTANT: Integration tests always run make build first.

Available Test Packages

• all - Run all integration tests
• multipooler - Connection pooling, pool lifecycle, connection management
• multiorch - Orchestration, failover, leader election, consensus protocol
• queryserving - Query routing, execution, transaction handling
• localprovisioner - Local cluster provisioning and setup
• shardsetup - Shard configuration and management
• pgregresstest - PostgreSQL regression tests (opt-in, comprehensive)

Integration Test Command Syntax

Run all integration tests:

/mt-dev integration all

Executes: make build && go test ./go/test/endtoend/...

Run specific package:

/mt-dev integration <package-name>

Examples:

• /mt-dev integration multipooler → make build && go test ./go/test/endtoend/multipooler/...
• /mt-dev integration multiorch → make build && go test ./go/test/endtoend/multiorch/...
• /mt-dev integration queryserving → make build && go test ./go/test/endtoend/queryserving/...

Run specific test:

/mt-dev integration <package-name> <TestName>

Examples:

• /mt-dev integration multiorch TestFixReplication → make build && go test -run TestFixReplication ./go/test/endtoend/multiorch/...
• /mt-dev integration multipooler TestConnCache → make build && go test -run TestConnCache ./go/test/endtoend/multipooler/...

Run specific test in all packages:

/mt-dev integration all <TestName>

Example:

• /mt-dev integration all TestBootstrap → make build && go test -run TestBootstrap ./go/test/endtoend/...

Run with pattern matching:

/mt-dev integration <package-name> <TestPattern>

Examples:

• /mt-dev integration queryserving Test.*Transaction.* - All transaction tests
• /mt-dev integration multipooler TestConn.* - All connection tests

Integration Test Flags

Same flags as unit tests, plus:

• -timeout=30m - Integration tests often need longer timeouts (default: 10m)
• -p=1 - Run packages sequentially (useful if tests conflict on resources)
• -count=N - Run tests N times (useful to detect flakes); -count=1 also forces re-run and bypasses test cache

Integration Test Examples

# Run all integration tests
/mt-dev integration all

# Test multipooler with verbose output
/mt-dev integration multipooler -v

# Test specific failure scenario
/mt-dev integration multiorch TestFixReplication

# Check for race conditions in query serving
/mt-dev integration queryserving -race

# Test for flakiness (run 10 times)
/mt-dev integration multipooler TestConnCache -count=10

# Run with extended timeout
/mt-dev integration all -timeout=45m

# Sequential execution to avoid resource conflicts
/mt-dev integration all -p=1

Integration Test Natural Language

• "run integration tests" → /mt-dev integration all
• "run multipooler tests" → /mt-dev integration multipooler
• "test multiorch TestFixReplication" → /mt-dev integration multiorch TestFixReplication
• "run all integration tests with race detector" → /mt-dev integration all -race
• "test query serving" → /mt-dev integration queryserving

---

Interpreting Test Results

Success Output

PASS
ok      github.com/multigres/multigres/go/services/multipooler    2.456s

• All tests passed
• Shows package path and execution time

Failure Output

--- FAIL: TestConnectionPool (0.15s)
    pool_test.go:45: expected 10 connections, got 8
FAIL
FAIL    github.com/multigres/multigres/go/services/multipooler    2.456s

• Shows which test failed
• Shows file, line number, and failure message
• Claude should summarize: "TestConnectionPool failed in pool_test.go:45"

Build Failure

# github.com/multigres/multigres/go/services/multipooler
./connection.go:123:45: undefined: somethingMissing
FAIL    github.com/multigres/multigres/go/services/multipooler [build failed]

• Compilation error before tests could run
• Claude should highlight the build error and suggest checking the code

Race Condition Detected

==================
WARNING: DATA RACE
Read at 0x00c0001a2080 by goroutine 7:
  ...
==================

• Race detector found a potential concurrency bug
• Claude should flag this as critical and recommend investigation

Timeout

panic: test timed out after 10m0s

• Test exceeded timeout
• Claude should suggest increasing timeout or investigating hanging test

---

Common Workflows

Before Committing Code

# 1. Run unit tests (fast feedback)
/mt-dev unit all

# 2. If unit tests pass, run integration tests
/mt-dev integration all

# 3. Check for race conditions
/mt-dev integration all -race

Debugging a Failing Test

# 1. Run the specific test with verbose output
/mt-dev integration multipooler TestConnCache -v

# 2. Check if it's flaky (intermittent failure)
/mt-dev integration multipooler TestConnCache -count=10

# 3. Run with race detector
/mt-dev integration multipooler TestConnCache -race -v

Testing a Specific Package After Changes

# Unit tests first (fast)
/mt-dev unit ./go/services/multipooler/... -v

# Integration tests second
/mt-dev integration multipooler -v

---

Troubleshooting

"build failed" during integration tests

• Run make build manually to see detailed error
• Check for uncommitted generated files (protobuf)
• Verify all dependencies are installed

Flaky tests (pass sometimes, fail others)

• Run with -count=10 to reproduce
• Enable race detector: -race
• Check for timing-dependent code or shared state

---

Debugging Integration Test Failures

When integration tests fail, logs are automatically preserved in a temp directory. Follow this systematic approach to debug failures:

Step 1: Find the Log Location

After a test failure, look for this message in the test output:

==== TEST LOGS PRESERVED ====
Logs available at: /tmp/shardsetup_test_XXXXXXXXXX
Set TEST_PRINT_LOGS=1 to print log contents
===========================

The temp directory contains all component logs from the failed test.

Step 2: Understand the Directory Structure

Integration test log directories follow this structure:

/tmp/shardsetup_test_XXXXXXXXXX/
├── multigateway.log              # Multigateway service logs
├── temp-multiorch/               # Temporary multiorch used during bootstrap
│   └── multiorch.log
├── pooler-1/                     # First multipooler instance (pooler-1, pooler-2, etc.)
│   ├── multipooler.log           # Multipooler service logs
│   ├── pgctld.log                # PostgreSQL control daemon logs
│   └── data/
│       ├── pg_data/
│       │   └── postgresql.log    # PostgreSQL server logs
│       ├── pg_sockets/

---

*Content truncated.*