Skip to main content

CF app stop

Last updated on

CF app stop is a Cloud Foundry chaos fault that stops the target app (app in organization/space) for duration seconds, then starts it back up. It exercises how the Cloud Foundry platform, upstream routers, and downstream consumers behave when an app is gracefully taken offline and brought back.

Use this fault to validate disaster-recovery behaviors such as the platform restarting the app, the router rejecting traffic during the outage, blue/green peers absorbing requests, and alerting firing inside SLA.

Run your first experiment

If you have not configured the chaos infrastructure yet, go to Quickstart to install the Linux chaos infrastructure and run an experiment end to end.


Use cases

  • Outage absorption: Validate that consumers (gateways, other apps) handle the stopped app cleanly: retries, fallbacks, circuit breakers.
  • Platform recovery: Confirm Cloud Foundry brings the app back to a healthy state when the fault re-starts it.
  • Alert fidelity: Verify health-check based alerts (/health, route probes) fire inside the alerting SLA.
  • Runbook validation: Practice the on-call response to a CF app outage in a controlled window.

Before you begin

  • Chaos infrastructure: A Linux chaos infrastructure (LCI) installed on a machine that can reach the Cloud Foundry API and UAA endpoints. Go to Cloud Foundry chaos deployment to read the supported deployment models.
  • CF credentials: CF_API_ENDPOINT, CF_USERNAME, CF_PASSWORD, and UAA_SERVER_ENDPOINT available to the chaos infrastructure (see Authentication).
  • Target identifiers: You know the organization, space, and app name as registered in CF.
  • Permissions: The CF user holds the roles described in Permissions required.

Supported environments

PlatformSupport status
Cloud Foundry on TAS, PCF, or open-source CF deploymentsSupported
Kubernetes-only deployments (no Cloud Foundry control plane)Not supported (use the Kubernetes pod faults)

Permissions required

On Cloud Foundry. The CF user authenticating as CF_USERNAME needs both read and write scopes on the target app:

ActionRequired role (any)Required OAuth scope (any)
List apps the user can accessSpaceDeveloper, SpaceAuditor, OrgManager, or OrgAuditorcloud_controller.read, cloud_controller.global_auditor, cloud_controller.admin
Stop the target app (set state to STOPPED)SpaceDeveloper in the app's spacecloud_controller.write, cloud_controller.admin
Start the target app (set state to STARTED)SpaceDeveloper in the app's spacecloud_controller.write, cloud_controller.admin

Authentication

LayerWhere to provideTunables
Cloud Foundry API/etc/linux-chaos-infrastructure/cf.env on the LCI host (or environment / config file)CF_API_ENDPOINT, CF_USERNAME, CF_PASSWORD, UAA_SERVER_ENDPOINT
vSphere (only when faultInjectorLocation: vSphere)/etc/linux-chaos-infrastructure/vsphere.envGOVC_URL, GOVC_USERNAME, GOVC_PASSWORD, GOVC_INSECURE, VM_NAME, VM_USERNAME, VM_PASSWORD

Full key/format reference is in CF secrets and vSphere secrets below.


Fault tunables

Required parameters

TunableDescriptionDefault
organizationCloud Foundry organization that owns the target app.(required)
spaceCF space within the organization.(required)
appName of the app to stop.(required)

Chaos parameters

TunableDescriptionDefault
durationTotal chaos duration. The app stays stopped for this period before it is restarted.30s
faultInjectorLocationWhere the fault-injector runs relative to the LCI. Supports local and vSphere. Go to Fault injector location.local
faultInjectorPortLocal port used by the fault-injector. If unavailable, a random port in 50320-51320 is picked. Go to Fault injector port.50320
skipSSLValidationSkip SSL validation when calling CF APIs. Go to Skip SSL validation.false
rampTimeWait period in seconds before and after the fault. Go to ramp time.0

Tunables that apply to every fault are documented in common tunables for all faults.


Fault execution in brief

Authenticates to the Cloud Foundry API, locates the app in organization/space, sets its state to STOPPED, waits duration seconds, and then sets the state back to STARTED. Recovery completes when CF reports the app STARTED and healthy.


Expected behavior during fault execution

  • The app's CF status transitions to STOPPED; its instances are drained and removed.
  • Requests routed to the app fail (typically 502/503 from the CF router) until the fault re-starts the app.
  • After the fault ends, CF starts the app and health checks return successful before traffic is restored.

Signals to watch

  • App availability: Use an HTTP probe against a public route on the app.
  • Downstream errors: Probe a consumer that depends on the app; assert it returns its expected fallback response, not a 5xx.

Recovery and cleanup

  • The fault re-starts the app automatically when duration elapses.
  • If the experiment is aborted, the fault still attempts to start the app before exiting.

Limitations

  • Stops the entire app, not individual instances. To affect a subset, use CF app container kill with instanceAffectedPercentage.
  • The app must already be in STARTED state at experiment start.

Troubleshooting

CF app stop fails with 'app not found' in Harness Chaos Engineering

Verify the organization, space, and app values match what is registered in Cloud Foundry (case-sensitive). Run cf app <name> from the LCI host to confirm visibility under the same CF user.

CF API call fails with TLS certificate error

If your CF API uses a self-signed or internal CA certificate, set skipSSLValidation: true on the fault, or install the trusted CA on the chaos infrastructure host.

App is still STOPPED after the experiment ends

The fault attempts to restart the app on cleanup. If the start failed (for example, due to insufficient memory in the org/space quota), start the app manually with cf start <app> and investigate the platform logs.


Common configurations

Fault injector location

The faultInjectorLocation input determines where the fault-injector utility runs:

  • local: Runs on the same host as the LCI. Use this when the LCI host can reach the CF API directly.
  • vSphere: Runs on a remote vSphere-managed VM. Use this when the chaos infrastructure is deployed separately from the network path that reaches CF.
apiVersion: litmuchaos.io/v1alpha1
kind: LinuxFault
metadata:
name: cf-app-stop
labels:
name: app-stop
spec:
cfAppStop/inputs:
duration: 30s
faultInjectorLocation: vSphere
app: cf-app
organization: dev-org
space: dev-space

Skip SSL validation

Set skipSSLValidation: true when calling a CF API that uses a self-signed or internal CA certificate.

apiVersion: litmuchaos.io/v1alpha1
kind: LinuxFault
metadata:
name: cf-app-stop
labels:
name: app-stop
spec:
cfAppStop/inputs:
duration: 30s
faultInjectorLocation: vSphere
app: cf-app
organization: dev-org
space: dev-space
skipSSLValidation: true

Fault injector port

Set faultInjectorPort to override the default 50320. If the chosen port is already in use, the fault-injector picks a random port in 50320-51320.

apiVersion: litmuchaos.io/v1alpha1
kind: LinuxFault
metadata:
name: cf-app-stop
labels:
name: app-stop
spec:
cfAppStop/inputs:
duration: 30s
faultInjectorLocation: local
app: cf-app
organization: dev-org
space: dev-space
faultInjectorPort: 50331

CF secrets

The following Cloud Foundry secrets reside on the same machine where the chaos infrastructure is executed. These secrets are provided in the /etc/linux-chaos-infrastructure/cf.env file in the following format:

CF_API_ENDPOINT=XXXXXXXXXXXXXXXXXXX
CF_USERNAME=XXXXXXXXXXXXXXXXXXXXXXX
CF_PASSWORD=XXXXXXXXXXXXXXXXXXXXXXX
UAA_SERVER_ENDPOINT=XXXXXXXXXXXXXXX
info

If the secrets file is not provided, the secrets are attempted to be derived from environment variables and the config file by the fault-injector.

ENV nameDescriptionExample
CF_API_ENDPOINTAPI endpoint for the CF setuphttps://api.system.cf-setup.com
CF_USERNAMEUsername for the CF userusername
CF_PASSWORDPassword for the CF userpassword
UAA_SERVER_ENDPOINTAPI endpoint for the UAA server for the CF setuphttps://uaa.system.cf-setup.com

Fault injector ENVs and config file

If /etc/linux-chaos-infrastructure/cf.env file is not provided, fault-injector attempts to derive the secrets from environment variables or a configuration file. Any secret that is re-declared will be overridden in the following order of decreasing precedence:

  1. /etc/linux-chaos-infrastructure/cf.env file
  2. Environment variables
  3. Configuration file

The configuration file should be provided at /etc/linux-chaos-infrastructure/cf-fault-injector.yaml:

cf-api-endpoint: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
username: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
password: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
uaa-server-endpoint: XXXXXXXXXXXXXXXXXXXXXXXXXX

A mapping between all the three formats for providing the secrets is as follows:

cf.envENVcf-fault-injector.yaml
CF_API_ENDPOINTCF_API_ENDPOINTcf-api-endpoint
CF_USERNAMEUSERNAMEusername
CF_PASSWORDPASSWORDpassword
UAA_SERVER_ENDPOINTUAA_SERVER_ENDPOINTuaa-server-endpoint

vSphere secrets

These secrets are provided only if vSphere is used as the deployment platform for CF.

The following vSphere secrets reside on the same machine where the chaos infrastructure is executed. These secrets are provided in the /etc/linux-chaos-infrastructure/vsphere.env file in the following format:

GOVC_URL=XXXXXXXXXXXXXXXXXXXXXX
GOVC_USERNAME=XXXXXXXXXXXXXXXXX
GOVC_PASSWORD=XXXXXXXXXXXXXXXXX
GOVC_INSECURE=XXXXXXXXXXXXXXXXX
VM_NAME=XXXXXXXXXXXXXXXXXXXXXXX
VM_USERNAME=XXXXXXXXXXXXXXXXXXX
VM_PASSWORD=XXXXXXXXXXXXXXXXXXX
ENV Name Description Notes
GOVC_URL Endpoint for vSphere For example, 192.168.214.244
GOVC_USERNAME Username for the vSphere user For example, username
GOVC_PASSWORD Password for the vSphere user For example, password
GOVC_INSECURE Skip SSL validation for govc commands For example, true
VM_NAME Name of the vSphere VM where the fault-injector utility is installed For example, cf-vm
VM_USERNAME Username for the VM guest user For example, root
VM_PASSWORD Password for the VM guest user For example, password