Mock Mode Guide

Test and demonstrate S9S functionality without requiring a real SLURM cluster using the built-in mock mode.

🎭 Overview

Mock mode provides:

Simulated SLURM cluster environment
Realistic job and node data
Full S9S functionality without backend dependencies
Safe environment for testing and training
Customizable cluster topologies and workloads

🚀 Getting Started

Enable Mock Mode


# Start S9S in mock mode
s9s --mock

# Or set environment variable
export S9S_MOCK=true
s9s

# Enable in config file
# ~/.s9s/config.yaml
mock:
  enabled: true

Quick Mock Setup


# Generate default mock cluster
s9s mock generate

# Start with predefined scenarios
s9s --mock --scenario=small-cluster
s9s --mock --scenario=gpu-cluster  
s9s --mock --scenario=mixed-workload

⚙️ Mock Configuration

Basic Mock Settings


# ~/.s9s/config.yaml
mock:
  enabled: true
  
  # Cluster configuration
  cluster:
    name: "mock-cluster"
    nodes: 100
    partitions: ["cpu", "gpu", "highmem"]
    
  # Job simulation
  jobs:
    initialCount: 500
    arrivalRate: 0.2  # jobs per second
    completionRate: 0.1  # fraction completing per minute
    failureRate: 0.05    # fraction failing
    
  # Performance simulation
  performance:
    cpuUtilization: 0.75
    memoryUtilization: 0.60
    networkLatency: "1-5ms"

Advanced Mock Configuration


mock:
  # Cluster topology
  topology:
    racks: 4
    nodesPerRack: 25
    coresPerNode: 32
    memoryPerNode: "128GB"
    
  # Node types
  nodeTypes:
    cpu:
      count: 60
      cores: 32
      memory: "128GB"
      features: ["infiniband"]
      
    gpu:
      count: 20
      cores: 32  
      memory: "256GB"
      gpus: 4
      features: ["gpu", "cuda", "infiniband"]
      
    highmem:
      count: 20
      cores: 64
      memory: "1TB" 
      features: ["highmem", "infiniband"]
      
  # Partition configuration
  partitions:
    cpu:
      nodes: "node[001-060]"
      timeLimit: "7-00:00:00"
      maxNodes: 32
      
    gpu:
      nodes: "node[061-080]"
      timeLimit: "2-00:00:00"
      maxNodes: 8
      priority: 1000
      
    highmem:
      nodes: "node[081-100]"
      timeLimit: "1-00:00:00"
      maxNodes: 4
      
  # User simulation
  users:
    count: 50
    accounts: ["research", "teaching", "industry"]
    activityPattern: "realistic"  # or "heavy", "light"

📋 Mock Scenarios

Predefined Scenarios

Small Cluster:


# scenarios/small-cluster.yaml
name: "Small Research Cluster"
cluster:
  nodes: 32
  totalCores: 1024
  totalMemory: "4TB"
jobs:
  active: 50-80
  pending: 10-30
  daily: 200
workload: "cpu-intensive"

GPU Cluster:


# scenarios/gpu-cluster.yaml  
name: "GPU-Accelerated Cluster"
cluster:
  nodes: 64
  gpuNodes: 32
  totalGPUs: 128
jobs:
  active: 100-150
  gpuJobs: 60%
  longRunning: 40%
workload: "ml-training"

Mixed Workload:


# scenarios/mixed-workload.yaml
name: "Production HPC Cluster"
cluster:
  nodes: 500
  partitions: 8
  users: 200
jobs:
  active: 1000-2000
  pending: 200-500
  array_jobs: 30%
workload: "mixed-hpc"

Custom Scenarios

Create custom mock scenarios:


# ~/.s9s/mock/scenarios/my-scenario.yaml
name: "My Custom Scenario"
description: "Simulates our production workload"

cluster:
  nodes: 256
  racks: 8
  totalCores: 8192
  totalMemory: "32TB"
  interconnect: "infiniband"
  
partitions:
  - name: "normal"
    nodes: 200
    timeLimit: "7-00:00:00"
    
  - name: "express" 
    nodes: 32
    timeLimit: "4:00:00"
    priority: 2000
    
  - name: "gpu"
    nodes: 24
    gpus: 4
    timeLimit: "2-00:00:00"
    
jobPatterns:
  - type: "short"
    percentage: 60
    runtime: "5m-2h"
    resources: "1-4 cores"
    
  - type: "medium"
    percentage: 30  
    runtime: "2h-12h"
    resources: "4-32 cores"
    
  - type: "long"
    percentage: 10
    runtime: "12h-7d"
    resources: "32-256 cores"

Load custom scenario:


s9s --mock --scenario=my-scenario

🎬 Job Simulation

Simulated Job Types

Mock mode simulates various job types:


# CPU-intensive jobs
array_job_001  alice  RUNNING   node[001-008]  2h   
cpu_benchmark  bob    PENDING   -              -    

# GPU machine learning jobs  
ml_training    carol  RUNNING   gpu001         18h  
tensorflow_job dave   RUNNING   gpu[002-003]   4h   

# Memory-intensive jobs
big_data_proc  eve    RUNNING   highmem001     1d   
genome_assembly frank PENDING   -              -    

# Array jobs
parameter_sweep[1-100] alice RUNNING node[010-020] 30m

Job State Transitions

Observe realistic job lifecycle:


# Job progression simulation
12:00  job_001  PENDING   → Waiting in queue
12:05  job_001  RUNNING   → Resources allocated  
14:30  job_001  COMPLETED → Job finished successfully

# Failed job simulation
12:15  job_002  PENDING   → Queued
12:20  job_002  RUNNING   → Started
13:45  job_002  FAILED    → Out of memory error

# Job suspension/resumption
15:00  job_003  RUNNING   → Normal execution
15:30  job_003  SUSPENDED → Higher priority job preempted
16:00  job_003  RUNNING   → Resumed after preemption

Interactive Job Control

Test job management operations:


# Submit mock jobs
:submit --mock cpu-job --cores=16 --time=2h
:submit --mock gpu-job --gpus=2 --partition=gpu

# Cancel jobs (simulated)
:cancel 12345
Job 12345 cancelled successfully (mock)

# Hold/release jobs
:hold 12346 --reason="Testing hold functionality"
:release 12346

📊 Performance Simulation

Resource Utilization

Mock mode simulates realistic resource usage:


Cluster Utilization (Mock):

CPU:     ██████████████░░ 72.5%
Memory:  ██████████░░░░░░ 58.3%
GPU:     █████████████░░░ 84.7%
Network: █████░░░░░░░░░░░ 32.1%

Queue Statistics:
- Pending Jobs: 47
- Average Wait: 12.3 minutes  
- Throughput: 145 jobs/hour

Simulated Metrics


# Performance metrics simulation
:metrics --mock

Simulated Performance Metrics:

┌─────────────────────────────────────────────┐
│ Node Health:                                   │
│   Online: 98 nodes (98%)                      │
│   Offline: 2 nodes (maintenance)              │
│   Load Average: 2.1 (normal)                  │
│                                               │
│ Job Efficiency:                               │
│   CPU Efficiency: 82.3% (good)                │
│   Memory Efficiency: 67.8% (fair)             │
│   Runtime Efficiency: 91.2% (excellent)       │
└─────────────────────────────────────────────┘

🔧 Testing and Development

API Testing

Mock mode provides realistic API responses:


# Test API endpoints
curl http://localhost:8080/api/v1/jobs
# Returns mock job data

curl http://localhost:8080/api/v1/nodes  
# Returns mock node data

# Mock API behaves like real SLURM REST API

Feature Testing

Test all S9S features safely:


# Test filtering
/state:RUNNING user:alice
# Shows mock jobs matching filter

# Test batch operations
:cancel /state:PENDING priority:<100
# Simulates cancelling low-priority jobs

# Test exports
:export csv --output mock-data.csv
# Exports mock job data

Training and Demos

Use mock mode for training:


# Demo scenario with realistic data
s9s --mock --demo-mode

# Training scenario with guided tour
s9s --mock --training --interactive

# Presentation mode with large fonts
s9s --mock --presentation

📊 Data Generation

Mock Data Patterns

Customize mock data generation:


mock:
  dataGeneration:
    # Job name patterns
    jobNames:
      - "simulation_{id}"
      - "analysis_{user}_{timestamp}"
      - "ml_training_{model}_{epoch}"
      
    # User patterns  
    users:
      patterns:
        - "student{01-50}"  # student01, student02, ...
        - "prof_{lastname}"  # prof_smith, prof_jones, ...
        - "admin"            # admin user
        
    # Realistic resource requests
    resources:
      cpuDistribution: "normal(16, 8)"  # mean=16, stddev=8
      memoryDistribution: "lognormal(32, 2)"  # GB
      runtimeDistribution: "exponential(2h)"  # hours

Historical Data Simulation


# Generate historical data
:mock generate-history --days=30 --output=history.json

# Load historical data
:mock load-history history.json

# Replay historical patterns
:mock replay --speed=100x --start="2023-12-01"

🔍 Mock Debugging

Mock State Inspection


# View mock cluster state
:mock status

Mock Cluster Status:
┌─────────────────────────────────────────────┐
│ Mock Engine: Active                            │
│ Scenario: mixed-workload                       │
│ Simulation Time: 2023-12-15 14:30:15          │
│ Time Acceleration: 1x (real-time)             │
│                                               │
│ Generated Data:                               │
│   Jobs: 1,247 total, 156 active              │
│   Nodes: 100 simulated                       │
│   Users: 50 active                           │
└─────────────────────────────────────────────┘

# Inspect mock data sources
:mock inspect --detail

# View simulation parameters
:mock config

Mock Event Simulation


# Simulate cluster events
:mock event node-failure node001
:mock event job-completion 12345
:mock event maintenance-start rack01

# Simulate load patterns
:mock load-pattern peak --duration=1h
:mock load-pattern weekend --quiet

🔄 Mock vs Real Comparison

Feature Parity

Feature	Mock Mode	Real SLURM	Notes
Job listing	✓	✓	Full parity
Job submission	✓*	✓	*Simulated
Node monitoring	✓	✓	Realistic data
SSH access	✗	✓	Not applicable
Performance metrics	✓	✓	Simulated data
Export/reports	✓	✓	Mock data
Filtering	✓	✓	Full support
Batch operations	✓*	✓	*Simulated

Switching Modes


# Switch from mock to real
:config set mock.enabled false
:reconnect

# Switch from real to mock
:config set mock.enabled true
:reconnect

# Hybrid mode (mock + real data overlay)
:config set mock.hybrid true

🚀 Best Practices

Mock Mode Usage

Start with mock - Learn S9S without cluster impact
Test configurations - Validate settings safely
Demo preparations - Practice with realistic data
Development testing - Test new features
Training users - Safe environment for learning

Data Realism

Configure realistic scenarios - Match your cluster profile
Use appropriate scales - Similar node/job counts
Simulate real patterns - Match actual workloads
Include edge cases - Test failure scenarios
Update regularly - Keep mock data current

Testing Strategy

Mock first - Test all functionality in mock mode
Gradual transition - Move to real cluster incrementally
Keep mock updated - Sync with cluster changes
Document scenarios - Share mock configurations
Automate testing - Use mock mode in CI/CD

🎓 Training Scenarios

Beginner Tutorial


# Start guided tutorial
s9s --mock --tutorial=beginner

# Covers:
# - Basic navigation
# - Job monitoring  
# - Simple filtering
# - Basic operations

Advanced Workshop


# Advanced features workshop
s9s --mock --tutorial=advanced

# Covers:
# - Complex filtering
# - Batch operations
# - Performance analysis
# - Automation

Administrator Training


# Admin-focused training
s9s --mock --tutorial=admin

# Covers:
# - Node management
# - User management
# - System monitoring
# - Troubleshooting

🚀 Next Steps

Practice with Job Management in mock mode
Test Advanced Filtering with mock data
Explore Batch Operations safely
Learn Performance Monitoring concepts