databricks-multi-env-setup

Databricks Multi-Environment Setup

Overview

Configure Databricks across development, staging, and production environments with isolated API keys, environment-specific settings, and proper secret management. Each environment gets its own credentials and configuration to prevent cross-environment data leakage.

Prerequisites

• Separate Databricks API keys per environment
• Secret management solution (environment variables, Vault, or cloud secrets)
• CI/CD pipeline with environment-aware deployment
• Application with environment detection logic

Environment Strategy

Environment	Purpose	API Key Source	Settings
Development	Local development	.env.local	Debug enabled, relaxed limits
Staging	Pre-production testing	CI/CD secrets	Production-like settings
Production	Live traffic	Secret manager	Optimized, hardened

Instructions

Step 1: Configuration Structure

config/
  databricks/
    base.ts           # Shared defaults
    development.ts    # Dev overrides
    staging.ts        # Staging overrides
    production.ts     # Prod overrides
    index.ts          # Environment resolver

Step 2: Base Configuration

// config/databricks/base.ts
export const baseConfig = {
  timeout: 30000,  # 30000: 30 seconds in ms
  maxRetries: 3,
  cache: {
    enabled: true,
    ttlSeconds: 300,  # 300: timeout: 5 minutes
  },
};

Step 3: Environment-Specific Configs

// config/databricks/development.ts
import { baseConfig } from "./base";

export const developmentConfig = {
  ...baseConfig,
  apiKey: process.env.DATABRICKS_TOKEN_DEV,
  debug: true,
  cache: { enabled: false, ttlSeconds: 60 },
};

// config/databricks/staging.ts
import { baseConfig } from "./base";

export const stagingConfig = {
  ...baseConfig,
  apiKey: process.env.DATABRICKS_TOKEN_STAGING,
  debug: false,
};

// config/databricks/production.ts
import { baseConfig } from "./base";

export const productionConfig = {
  ...baseConfig,
  apiKey: process.env.DATABRICKS_TOKEN_PROD,
  debug: false,
  timeout: 60000,  # 60000: 1 minute in ms
  maxRetries: 5,
  cache: { enabled: true, ttlSeconds: 600 },  # 600: timeout: 10 minutes
};

Step 4: Environment Resolver

// config/databricks/index.ts
import { developmentConfig } from "./development";
import { stagingConfig } from "./staging";
import { productionConfig } from "./production";

type Environment = "development" | "staging" | "production";

const configs = {
  development: developmentConfig,
  staging: stagingConfig,
  production: productionConfig,
};

export function detectEnvironment(): Environment {
  const env = process.env.NODE_ENV || "development";
  if (env === "production") return "production";
  if (env === "staging" || process.env.VERCEL_ENV === "preview") return "staging";
  return "development";
}

export function getDatabricksConfig() {
  const env = detectEnvironment();
  const config = configs[env];

  if (!config.apiKey) {
    throw new Error(`DATABRICKS_TOKEN not set for environment: ${env}`);
  }

  return { ...config, environment: env };
}

Step 5: Secret Management

# Local development (.env.local - git-ignored)
DATABRICKS_TOKEN_DEV=your-dev-key

# GitHub Actions
# Settings > Environments > staging/production > Secrets
# Add DATABRICKS_TOKEN_STAGING and DATABRICKS_TOKEN_PROD

# AWS Secrets Manager
aws secretsmanager create-secret \
  --name databricks/production/api-key \
  --secret-string "your-prod-key"

# GCP Secret Manager
echo -n "your-prod-key" | gcloud secrets create databricks-api-key-prod --data-file=-

# .github/workflows/deploy.yml
jobs:
  deploy-staging:
    environment: staging
    env:
      DATABRICKS_TOKEN_STAGING: ${{ secrets.DATABRICKS_TOKEN_STAGING }}

  deploy-production:
    environment: production
    env:
      DATABRICKS_TOKEN_PROD: ${{ secrets.DATABRICKS_TOKEN_PROD }}

Error Handling

Issue	Cause	Solution
Wrong environment	Missing NODE_ENV	Set environment variable in deployment
Secret not found	Wrong secret path	Verify secret manager configuration
Cross-env data leak	Shared API key	Use separate keys per environment
Config validation fail	Missing field	Add startup validation with Zod schema

Examples

Quick Environment Check

const config = getDatabricksConfig();
console.log(`Running in ${config.environment}`);
console.log(`Cache enabled: ${config.cache.enabled}`);

Startup Validation

import { z } from "zod";

const configSchema = z.object({
  apiKey: z.string().min(1, "DATABRICKS_TOKEN is required"),
  environment: z.enum(["development", "staging", "production"]),
  timeout: z.number().positive(),
});

const config = configSchema.parse(getDatabricksConfig());

Resources

• Databricks Asset Bundles
• Databricks Environments

Next Steps

For deployment, see databricks-deploy-integration.

Output

• Configuration files or code changes applied to the project
• Validation report confirming correct implementation
• Summary of changes made and their rationale