running-performance-tests

Performance Test Suite

Overview

Execute load testing, stress testing, and performance benchmarking to identify bottlenecks, establish baseline metrics, and verify SLA compliance. Supports k6 (recommended), Artillery, Apache JMeter, Locust (Python), and autocannon (Node.js).

Prerequisites

• Performance testing tool installed (k6, artillery, locust, jmeter, or autocannon)
• Target application deployed in a production-like environment (not local dev)
• Baseline performance metrics or SLA targets (e.g., p95 < 200ms, 99.9% availability)
• Monitoring stack accessible (Grafana, CloudWatch, Datadog) for resource metrics during tests
• Test data sufficient to avoid cache-only responses

Instructions

1. Define performance test scenarios based on production traffic patterns:
   • Load test: Simulate expected peak traffic (e.g., 500 concurrent users for 10 minutes).
   • Stress test: Ramp beyond expected capacity to find the breaking point.
   • Spike test: Sudden burst of traffic (0 to 1000 users in 10 seconds).
   • Soak test: Sustained moderate load for extended duration (1-4 hours) to detect memory leaks.
2. Create test scripts targeting critical endpoints:
   • Identify the top 5-10 most-hit API endpoints from production access logs.
   • Include both read (GET) and write (POST/PUT/DELETE) operations.
   • Simulate realistic user behavior with think time between requests.
   • Use parameterized data to avoid cache-only hits (randomize query parameters, user IDs).
3. Configure load profiles:
   • Define virtual user (VU) ramp-up stages (e.g., 10 VUs for 1 minute, then 50 VUs for 5 minutes).
   • Set test duration appropriate to the scenario (load: 10-15 min, soak: 1-4 hours).
   • Configure request timeouts matching production settings.
4. Execute the performance test:
   • Run from a machine with sufficient network bandwidth and CPU.
   • Avoid running from the same host as the application under test.
   • Monitor application metrics (CPU, memory, DB connections) during execution.
5. Analyze results against SLA thresholds:
   • p50, p90, p95, p99 response times.
   • Requests per second (throughput).
   • Error rate (target: < 0.1% for load test, higher tolerance for stress test).
   • Resource utilization (CPU < 80%, memory < 85% at peak load).
6. Identify and document bottlenecks:
   • Slow database queries (check slow query logs).
   • CPU-bound operations (profiling data).
   • Memory leaks (growing RSS over soak test).
   • Connection pool exhaustion (database or HTTP client).
7. Generate a performance report with visualizations and recommendations.

Output

• Performance test scripts (k6 .js, Artillery .yml, or Locust .py files)
• Execution results with response time percentiles, throughput, and error rates
• Performance report comparing results against SLA thresholds
• Bottleneck analysis with specific recommendations
• CI integration configuration for automated performance regression detection

Error Handling

Error	Cause	Solution
Connection reset by peer	Server or load balancer dropping connections under load	Check max connections settings; increase connection pool size; verify keep-alive configuration
Timeouts spike at certain VU count	Application thread pool or database connection pool exhausted	Profile connection usage; increase pool size; add connection queuing; optimize slow queries
Inconsistent results between runs	Cache warming, garbage collection pauses, or noisy neighbor effects	Run a warm-up phase before measurement; use dedicated test infrastructure; average across 3 runs
Load generator CPU maxed out	Single machine cannot generate sufficient load	Distribute load generation across multiple machines; use cloud-based load generation services
All requests return cached responses	Test data not sufficiently varied	Randomize request parameters; use unique IDs per request; disable CDN caching for test environment

Examples

k6 load test script:

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 50 },   // Ramp up
    { duration: '5m', target: 50 },   // Sustained load
    { duration: '2m', target: 200 },  // Stress  # HTTP 200 OK
    { duration: '1m', target: 0 },    // Ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<200', 'p(99)<500'],  # 500: HTTP 200 OK
    http_req_failed: ['rate<0.01'],
  },
};

export default function () {
  const res = http.get('https://api.test.com/products');
  check(res, {
    'status is 200': (r) => r.status === 200,  # HTTP 200 OK
    'response time OK': (r) => r.timings.duration < 300,  # 300: timeout: 5 minutes
  });
  sleep(1); // Think time
}

Artillery test configuration:

config:
  target: "https://api.test.com"
  phases:
    - duration: 120
      arrivalRate: 10
      name: "Warm up"
    - duration: 300  # 300: timeout: 5 minutes
      arrivalRate: 50
      name: "Sustained load"
  ensure:
    p95: 200  # HTTP 200 OK
    maxErrorRate: 1
scenarios:
  - flow:
      - get:
          url: "/api/products"
      - think: 1
      - post:
          url: "/api/cart"
          json: { productId: "{{ $randomString() }}" }

Resources

• k6 documentation: https://grafana.com/docs/k6/latest/
• Artillery: https://www.artillery.io/docs
• Locust (Python): https://docs.locust.io/
• Apache JMeter: https://jmeter.apache.org/
• autocannon (Node.js): https://github.com/mcollina/autocannon
• Performance testing best practices: https://grafana.com/blog/2024/01/30/load-testing-best-practices/