CF app JVM trigger GC
CF app JVM trigger GC is a Cloud Foundry chaos fault that forces a full garbage-collection cycle inside the JVM of one or more instances of a Java-based app. The cycle runs once per affected instance; the fault holds the chaos window for duration seconds before exiting.
Use this fault to surface tail-latency regressions caused by long GC pauses, validate that long pauses do not trigger health-check failures or cascading retries, and test alert thresholds tuned around GC behavior.
If you have not configured the chaos infrastructure yet, go to Quickstart to install the Linux chaos infrastructure and run an experiment end to end.
Use cases
- Pause-time validation: Measure the worst-case stop-the-world pause and assert it stays within the latency SLO.
- Health check tuning: Confirm liveness/readiness probes do not falsely fail during a full GC.
- Tail latency: Quantify the P99/P999 latency cost of a full GC in production-shaped traffic.
- Memory leak triage: Combined with metrics, distinguish a true leak from healthy memory churn.
Before you begin
- Chaos infrastructure: A Linux chaos infrastructure (LCI) installed in one of the supported deployment models.
- CF and BOSH credentials: The LCI host has
CF_*,UAA_SERVER_ENDPOINT, andBOSH_*credentials configured. - Target identifiers: You know the
organization,space,app, and theboshDeploymentthat manages the cluster. - Java app: The target app exposes a JVM agent on
port(default9091).
Supported environments
| Platform | Support status |
|---|---|
| Java apps deployed to Cloud Foundry | Supported |
| Non-Java workloads | Not supported |
Permissions required
| Action | Requirement |
|---|---|
| List apps the CF user can access | SpaceDeveloper, SpaceAuditor, OrgManager, or OrgAuditor; scopes cloud_controller.read or cloud_controller.admin |
| List BOSH deployments | BOSH user with bosh.read scope |
| SSH to a Diego cell via BOSH | BOSH UAA token with bosh.ssh or bosh.admin scope |
| Attach the JVM agent to the target container | Operator with sudo or root on the cell host |
Authentication
| Layer | Where to provide | Tunables |
|---|---|---|
| Cloud Foundry API + BOSH director | /etc/linux-chaos-infrastructure/cf.env on the LCI host | CF_API_ENDPOINT, CF_USERNAME, CF_PASSWORD, UAA_SERVER_ENDPOINT, BOSH_CLIENT, BOSH_CLIENT_SECRET, BOSH_CA_CERT, BOSH_ENVIRONMENT |
vSphere (only when faultInjectorLocation: vSphere) | /etc/linux-chaos-infrastructure/vsphere.env | GOVC_URL, GOVC_USERNAME, GOVC_PASSWORD, GOVC_INSECURE, VM_NAME, VM_USERNAME, VM_PASSWORD |
Fault tunables
Required parameters
| Tunable | Description | Default |
|---|---|---|
deploymentModel | LCI placement model. One of model-1 or model-2. | (required) |
organization | CF organization that owns the app. | (required) |
space | CF space within the organization. | (required) |
app | Java app whose JVM is targeted. | (required) |
Chaos parameters
| Tunable | Description | Default |
|---|---|---|
port | JVM agent port inside the container. | 9091 |
javaHome | Value of JAVA_HOME. Not required if the Java binary is on the container's PATH. | "" |
instanceAffectedPercentage | Percentage of instances to target. 0 targets exactly one. | 0 |
boshDeployment | BOSH deployment name. Required for deploymentModel: model-2. | "" |
faultInjectorLocation | local or vSphere. Required for deploymentModel: model-2. | local |
faultInjectorPort | Local port used by the fault-injector. | 50320 |
duration | Total chaos duration. | 30s |
skipSSLValidation | Skip SSL validation when calling CF APIs. | false |
rampTime | Wait period in seconds before and after the fault. | 0 |
Tunables that apply to every fault are documented in common tunables for all faults.
Fault execution in brief
Authenticates to Cloud Foundry and BOSH, locates the target app instance(s), attaches an agent to the JVM via the debug port, and invokes a full garbage collection cycle on the affected instance. The fault then holds the chaos window for duration seconds before exiting.
Expected behavior during fault execution
- The JVM pauses briefly (typically tens to hundreds of milliseconds depending on heap size and collector).
- Tail latencies on in-flight requests rise during the pause.
- After the GC completes, heap usage drops sharply and latencies return to baseline.
Signals to watch
- GC pause: Inspect JVM GC logs or metrics for a full collection event matching the experiment window.
- Tail latency: Use an HTTP probe and assert P99 stays within SLO.
Recovery and cleanup
- The JVM resumes normal operation immediately after the GC cycle completes.
- The agent is detached at the end of
duration.
Limitations
- Triggers a single full GC per affected instance; not a sustained pressure load (use CF app JVM memory stress for that).
- Pause time is determined by the JVM's collector and heap size, not by the fault.
Troubleshooting
CF app JVM trigger GC fails to attach to the JVM
Confirm the app exposes a JVM agent on the configured port (default 9091). For Java buildpack apps, set JBP_CONFIG_DEBUG: enabled=true,port=9091 as an app env var and restage.
No GC pause was observed
Some collectors (for example, G1, ZGC) may complete a full GC quickly enough that no perceptible pause appears in your probes. Verify the cycle ran by checking the JVM GC logs.
Common configurations
Trigger GC on multiple instances
apiVersion: litmuchaos.io/v1alpha1
kind: LinuxFault
metadata:
name: cf-app-jvm-trigger-gc
labels:
name: app-jvm-trigger-gc
spec:
cfAppJVMChaos/inputs:
duration: 30s
deploymentModel: model-2
faultInjectorLocation: vSphere
app: cf-app
organization: dev-org
space: dev-space
boshDeployment: cf
instanceAffectedPercentage: 100
CF secrets
The following Cloud Foundry secrets reside on the same machine where the chaos infrastructure is executed. These secrets are provided in the /etc/linux-chaos-infrastructure/cf.env file in the following format:
CF_API_ENDPOINT=XXXXXXXXXXXXXXXXXXX
CF_USERNAME=XXXXXXXXXXXXXXXXXXXXXXX
CF_PASSWORD=XXXXXXXXXXXXXXXXXXXXXXX
UAA_SERVER_ENDPOINT=XXXXXXXXXXXXXXX
BOSH_CLIENT=XXXXXXXXXXXXXXXXXXXXXXX
BOSH_CLIENT_SECRET=XXXXXXXXXXXXXXXX
BOSH_CA_CERT=XXXXXXXXXXXXXXXXXXXXXX
BOSH_ENVIRONMENT=XXXXXXXXXXXXXXXXXX
If the secrets file is not provided, the secrets are attempted to be derived from environment variables and the config file by the fault-injector.
| ENV name | Description | Example |
|---|---|---|
| CF_API_ENDPOINT | API endpoint for the CF setup | https://api.system.cf-setup.com |
| CF_USERNAME | Username for the CF user | username |
| CF_PASSWORD | Password for the CF user | password |
| UAA_SERVER_ENDPOINT | API endpoint for the UAA server for the CF setup | https://uaa.system.cf-setup.com |
| BOSH_CLIENT | Used by the bosh CLI, the BOSH client | admin |
| BOSH_CLIENT_SECRET | Used by the bosh CLI, the BOSH client secret | UBu9Fu3oW35sO6fw12auPH76gsRTy7 |
| BOSH_CA_CERT | Used by the bosh CLI, the file path for BOSH CA certificate | /root/root_ca_certificate |
| BOSH_ENVIRONMENT | Used by the bosh CLI, the BOSH environment | bosh.corp.local |
Fault injector ENVs and config file
If /etc/linux-chaos-infrastructure/cf.env file is not provided, fault-injector attempts to derive the secrets from environment variables or a configuration file. Any secret that is re-declared will be overridden in the following order of decreasing precedence:
/etc/linux-chaos-infrastructure/cf.envfile- Environment variables
- Configuration file
The configuration file should be provided at /etc/linux-chaos-infrastructure/cf-fault-injector.yaml:
cf-api-endpoint: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
username: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
password: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
uaa-server-endpoint: XXXXXXXXXXXXXXXXXXXXXXXXXX
bosh-client: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
bosh-client-secret: XXXXXXXXXXXXXXXXXXXXXXXXXXX
bosh-ca-cert: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
bosh-environment: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
A mapping between all the three formats for providing the secrets is as follows:
| cf.env | ENV | cf-fault-injector.yaml |
|---|---|---|
| CF_API_ENDPOINT | CF_API_ENDPOINT | cf-api-endpoint |
| CF_USERNAME | USERNAME | username |
| CF_PASSWORD | PASSWORD | password |
| UAA_SERVER_ENDPOINT | UAA_SERVER_ENDPOINT | uaa-server-endpoint |
| BOSH_CLIENT | BOSH_CLIENT | bosh-client |
| BOSH_CLIENT_SECRET | BOSH_CLIENT_SECRET | bosh-client-secret |
| BOSH_CA_CERT | BOSH_CA_CERT | bosh-ca-cert |
| BOSH_ENVIRONMENT | BOSH_ENVIRONMENT | bosh-environment |
vSphere secrets
These secrets are provided only if vSphere is used as the deployment platform for CF.
The following vSphere secrets reside on the same machine where the chaos infrastructure is executed. These secrets are provided in the /etc/linux-chaos-infrastructure/vsphere.env file in the following format:
GOVC_URL=XXXXXXXXXXXXXXXXXXXXXX
GOVC_USERNAME=XXXXXXXXXXXXXXXXX
GOVC_PASSWORD=XXXXXXXXXXXXXXXXX
GOVC_INSECURE=XXXXXXXXXXXXXXXXX
VM_NAME=XXXXXXXXXXXXXXXXXXXXXXX
VM_USERNAME=XXXXXXXXXXXXXXXXXXX
VM_PASSWORD=XXXXXXXXXXXXXXXXXXX
| ENV Name | Description | Notes |
|---|---|---|
| GOVC_URL | Endpoint for vSphere | For example, 192.168.214.244 |
| GOVC_USERNAME | Username for the vSphere user | For example, username |
| GOVC_PASSWORD | Password for the vSphere user | For example, password |
| GOVC_INSECURE | Skip SSL validation for govc commands | For example, true |
| VM_NAME | Name of the vSphere VM where the fault-injector utility is installed | For example, cf-vm |
| VM_USERNAME | Username for the VM guest user | For example, root |
| VM_PASSWORD | Password for the VM guest user | For example, password |
Related faults
- CF app JVM memory stress: Apply sustained memory pressure that may itself trigger GCs.
- CF app JVM CPU stress: Apply CPU pressure inside the JVM.