Skip to main content

CF app JVM CPU stress

Last updated on

CF app JVM CPU stress is a Cloud Foundry chaos fault that drives high CPU usage inside the JVM process of one or more instances of a Java-based app in organization/space. The stress lasts for duration seconds and is then released.

Use this fault to validate how the application, the CF platform, and downstream consumers behave under sustained JVM CPU pressure: whether response latencies stay within SLO, whether the JVM's autoscaling decisions are sensible, whether the platform scales additional instances, and whether alerts fire only when the workload actually degrades.

Run your first experiment

If you have not configured the chaos infrastructure yet, go to Quickstart to install the Linux chaos infrastructure and run an experiment end to end.


Use cases

  • Throughput limits: Measure the application's behavior at the edge of its compute envelope.
  • Autoscaler validation: Confirm horizontal scaling rules trigger correctly and pull traffic away from the stressed instance.
  • Thread pool tuning: Check whether thread pools handle requests gracefully under contention.
  • Alert thresholds: Distinguish transient spikes from sustained pressure that should page on-call.

Before you begin

  • Chaos infrastructure: A Linux chaos infrastructure (LCI) installed in one of the supported deployment models.
  • CF and BOSH credentials: The LCI host has CF_*, UAA_SERVER_ENDPOINT, and BOSH_* credentials configured.
  • Target identifiers: You know the organization, space, app, and the boshDeployment that manages the cluster.
  • Java app: The target app is a Java workload. The fault relies on the JVM exposing a debug agent on port (default 9091).
  • javaHome: Either the JAVA_HOME environment variable is set on the container, or you provide javaHome explicitly.

Supported environments

PlatformSupport status
Java apps deployed to Cloud Foundry (TAS, PCF, open-source)Supported
Non-Java workloads (Node.js, Python, Go)Not supported (use CF app network latency for protocol-level stress)

Permissions required

ActionRequirement
List apps the CF user can accessSpaceDeveloper, SpaceAuditor, OrgManager, or OrgAuditor; scopes cloud_controller.read or cloud_controller.admin
List BOSH deploymentsBOSH user with bosh.read scope
SSH to a Diego cell via BOSHBOSH UAA token with bosh.ssh or bosh.admin scope
Attach the JVM agent to the target containerOperator with sudo or root on the cell host

Authentication

LayerWhere to provideTunables
Cloud Foundry API + BOSH director/etc/linux-chaos-infrastructure/cf.env on the LCI hostCF_API_ENDPOINT, CF_USERNAME, CF_PASSWORD, UAA_SERVER_ENDPOINT, BOSH_CLIENT, BOSH_CLIENT_SECRET, BOSH_CA_CERT, BOSH_ENVIRONMENT
vSphere (only when faultInjectorLocation: vSphere)/etc/linux-chaos-infrastructure/vsphere.envGOVC_URL, GOVC_USERNAME, GOVC_PASSWORD, GOVC_INSECURE, VM_NAME, VM_USERNAME, VM_PASSWORD

Fault tunables

Required parameters

TunableDescriptionDefault
deploymentModelLCI placement model. One of model-1 or model-2. For model-1, boshDeployment and faultInjectorLocation are not required.(required)
organizationCF organization that owns the app.(required)
spaceCF space within the organization.(required)
appJava app to stress.(required)

Chaos parameters

TunableDescriptionDefault
cpuNumber of CPU cores to saturate inside the JVM.2
portPort exposed by the JVM agent inside the container.9091
javaHomeValue of JAVA_HOME. Not required if the Java binary is already on the container's PATH.""
instanceAffectedPercentagePercentage of instances to target. 0 targets exactly one.0
boshDeploymentBOSH deployment name. Required for deploymentModel: model-2.""
faultInjectorLocationlocal or vSphere. Required for deploymentModel: model-2.local
faultInjectorPortLocal port used by the fault-injector.50320
durationTotal chaos duration.30s
skipSSLValidationSkip SSL validation when calling CF APIs.false
rampTimeWait period in seconds before and after the fault.0

Tunables that apply to every fault are documented in common tunables for all faults.


Fault execution in brief

Authenticates to Cloud Foundry and BOSH, locates the target app instance(s), attaches an agent to the JVM process via the JVM debug port, and drives cpu cores worth of CPU usage inside the JVM for duration seconds. The agent is detached on completion and CPU usage returns to baseline.


Expected behavior during fault execution

  • Process CPU on the affected instance rises toward 100% of cpu cores.
  • Application response latencies typically rise; throughput may dip.
  • Autoscalers may scale the app out if scale-up thresholds are reached.
  • After the fault ends, CPU returns to baseline and latencies recover.

Signals to watch

  • Latency: Use an HTTP probe and assert P95 stays within SLO.
  • Instance count: Use a command probe running cf app <name> and verify autoscaling behavior.

Recovery and cleanup

  • The JVM agent is detached at the end of duration, releasing CPU pressure.
  • If the experiment is aborted, the fault still attempts to detach the agent before exiting.

Limitations

  • Targets the JVM process inside the container, not the host. Other workloads on the same Diego cell are unaffected.
  • Requires JVM debug port (port) reachable inside the container.
  • Recovery time depends on the workload re-establishing its steady state after the stress is removed.

Troubleshooting

CF app JVM CPU stress fails with 'JAVA_HOME not found' in Harness Chaos Engineering

Set the javaHome tunable to the absolute path of the JDK on the app container (for example, /usr/lib/jvm/openjdk). If you do not know the path, run cf ssh <app> -c 'echo $JAVA_HOME' from the LCI host.

Cannot attach to JVM debug port

Confirm the app exposes a JVM agent on the port you configured (default 9091). For Java buildpack apps, set JBP_CONFIG_DEBUG: enabled=true,port=9091 as an app env var, restage the app, and retry.

CPU usage does not return to baseline after the experiment

The agent may have failed to detach. Restart the affected app instance: cf restart-app-instance <app> <index>. If the issue persists, restage the app to release any leaked threads.


Common configurations

Stress multiple cores

apiVersion: litmuchaos.io/v1alpha1
kind: LinuxFault
metadata:
name: cf-app-jvm-cpu-stress
labels:
name: app-jvm-cpu-stress
spec:
cfAppJVMChaos/inputs:
duration: 60s
deploymentModel: model-2
faultInjectorLocation: vSphere
app: cf-app
organization: dev-org
space: dev-space
boshDeployment: cf
cpu: 4

Target multiple instances

apiVersion: litmuchaos.io/v1alpha1
kind: LinuxFault
metadata:
name: cf-app-jvm-cpu-stress
labels:
name: app-jvm-cpu-stress
spec:
cfAppJVMChaos/inputs:
duration: 30s
deploymentModel: model-2
faultInjectorLocation: vSphere
app: cf-app
organization: dev-org
space: dev-space
boshDeployment: cf
cpu: 2
instanceAffectedPercentage: 50

CF secrets

The following Cloud Foundry secrets reside on the same machine where the chaos infrastructure is executed. These secrets are provided in the /etc/linux-chaos-infrastructure/cf.env file in the following format:

CF_API_ENDPOINT=XXXXXXXXXXXXXXXXXXX
CF_USERNAME=XXXXXXXXXXXXXXXXXXXXXXX
CF_PASSWORD=XXXXXXXXXXXXXXXXXXXXXXX
UAA_SERVER_ENDPOINT=XXXXXXXXXXXXXXX
BOSH_CLIENT=XXXXXXXXXXXXXXXXXXXXXXX
BOSH_CLIENT_SECRET=XXXXXXXXXXXXXXXX
BOSH_CA_CERT=XXXXXXXXXXXXXXXXXXXXXX
BOSH_ENVIRONMENT=XXXXXXXXXXXXXXXXXX
info

If the secrets file is not provided, the secrets are attempted to be derived from environment variables and the config file by the fault-injector.

ENV nameDescriptionExample
CF_API_ENDPOINTAPI endpoint for the CF setuphttps://api.system.cf-setup.com
CF_USERNAMEUsername for the CF userusername
CF_PASSWORDPassword for the CF userpassword
UAA_SERVER_ENDPOINTAPI endpoint for the UAA server for the CF setuphttps://uaa.system.cf-setup.com
BOSH_CLIENTUsed by the bosh CLI, the BOSH clientadmin
BOSH_CLIENT_SECRETUsed by the bosh CLI, the BOSH client secretUBu9Fu3oW35sO6fw12auPH76gsRTy7
BOSH_CA_CERTUsed by the bosh CLI, the file path for BOSH CA certificate/root/root_ca_certificate
BOSH_ENVIRONMENTUsed by the bosh CLI, the BOSH environmentbosh.corp.local

Fault injector ENVs and config file

If /etc/linux-chaos-infrastructure/cf.env file is not provided, fault-injector attempts to derive the secrets from environment variables or a configuration file. Any secret that is re-declared will be overridden in the following order of decreasing precedence:

  1. /etc/linux-chaos-infrastructure/cf.env file
  2. Environment variables
  3. Configuration file

The configuration file should be provided at /etc/linux-chaos-infrastructure/cf-fault-injector.yaml:

cf-api-endpoint: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
username: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
password: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
uaa-server-endpoint: XXXXXXXXXXXXXXXXXXXXXXXXXX
bosh-client: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
bosh-client-secret: XXXXXXXXXXXXXXXXXXXXXXXXXXX
bosh-ca-cert: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
bosh-environment: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX

A mapping between all the three formats for providing the secrets is as follows:

cf.envENVcf-fault-injector.yaml
CF_API_ENDPOINTCF_API_ENDPOINTcf-api-endpoint
CF_USERNAMEUSERNAMEusername
CF_PASSWORDPASSWORDpassword
UAA_SERVER_ENDPOINTUAA_SERVER_ENDPOINTuaa-server-endpoint
BOSH_CLIENTBOSH_CLIENTbosh-client
BOSH_CLIENT_SECRETBOSH_CLIENT_SECRETbosh-client-secret
BOSH_CA_CERTBOSH_CA_CERTbosh-ca-cert
BOSH_ENVIRONMENTBOSH_ENVIRONMENTbosh-environment

vSphere secrets

These secrets are provided only if vSphere is used as the deployment platform for CF.

The following vSphere secrets reside on the same machine where the chaos infrastructure is executed. These secrets are provided in the /etc/linux-chaos-infrastructure/vsphere.env file in the following format:

GOVC_URL=XXXXXXXXXXXXXXXXXXXXXX
GOVC_USERNAME=XXXXXXXXXXXXXXXXX
GOVC_PASSWORD=XXXXXXXXXXXXXXXXX
GOVC_INSECURE=XXXXXXXXXXXXXXXXX
VM_NAME=XXXXXXXXXXXXXXXXXXXXXXX
VM_USERNAME=XXXXXXXXXXXXXXXXXXX
VM_PASSWORD=XXXXXXXXXXXXXXXXXXX
ENV Name Description Notes
GOVC_URL Endpoint for vSphere For example, 192.168.214.244
GOVC_USERNAME Username for the vSphere user For example, username
GOVC_PASSWORD Password for the vSphere user For example, password
GOVC_INSECURE Skip SSL validation for govc commands For example, true
VM_NAME Name of the vSphere VM where the fault-injector utility is installed For example, cf-vm
VM_USERNAME Username for the VM guest user For example, root
VM_PASSWORD Password for the VM guest user For example, password