Monitor a Kubernetes CronJob

The Kubernetes documentation is explicit that CronJobs have limitations, and the sharp one is the missed-start-time counter. For each CronJob the controller counts how many schedules were missed since the last scheduled time. If that count passes 100, it stops starting the Job and logs "too many missed start times", which is a controller log line, not a Kubernetes Event most alerting will notice.

This is reachable in ordinary operation. With startingDeadlineSeconds unset and a job scheduled every minute, a control plane outage of under two hours is enough to exceed 100 missed schedules and stop the job permanently. Setting startingDeadlineSeconds changes the counting window to that many seconds instead of since the last run, which is why the Kubernetes docs recommend setting it.

The other silent case is concurrencyPolicy: Forbid. A run that outlives its own schedule causes the next start to be skipped and counted as missed. A job that gradually slows down can go from occasionally skipping to never running, and every individual Job that did run succeeded.

Put the token in a Secret and reference it as an environment variable. The ping goes at the end of the command, joined with && so it only runs when the work succeeded.

startingDeadlineSeconds is set here deliberately, both because it bounds the missed-schedule counting window and because it keeps a controller restart from replaying a backlog. Keep it above 10 seconds: the controller polls every 10 seconds, and a smaller deadline can cause the job to be missed entirely.

apiVersion: v1
kind: Secret
metadata:
  name: alplus-heartbeat
stringData:
  token: hb_YOUR_TOKEN
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "17 3 * * *"
  timeZone: "Etc/UTC"
  # Bounds the missed-schedule window and avoids a catch-up storm.
  startingDeadlineSeconds: 300
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: report
              image: your-registry/report:1.4.0
              env:
                - name: POSTDEPLOY_HEARTBEAT_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: alplus-heartbeat
                      key: token
              command: ["/bin/sh", "-c"]
              args:
                - >-
                  /app/bin/report &&
                  curl -fsS "https://ingest.postdeploy.dev/h/$POSTDEPLOY_HEARTBEAT_TOKEN"

The && form stays silent on failure. To alert as soon as a run fails, report the exit code: run the work, capture $?, and send it as a path segment. A 0 is a success and anything else opens an incident immediately.

Note the image needs curl available. If your job image is distroless or scratch, use an available HTTP client or add curl. Use an external watchdog when a missing CronJob run needs an alert.

args:
  - >-
    /app/bin/report; code=$?;
    curl -fsS "https://ingest.postdeploy.dev/h/$POSTDEPLOY_HEARTBEAT_TOKEN/$code";
    exit $code

Trigger a run immediately rather than waiting for the schedule, then confirm the monitor is passing in PostDeploy.

# Run it now, without waiting for the schedule.
kubectl create job --from=cronjob/nightly-report nightly-report-manual

# Watch it, then confirm the ping was sent.
kubectl logs -f job/nightly-report-manual

# Check the controller's view of missed schedules.
kubectl get cronjob nightly-report -o wide

# If runs stopped, this is the line to look for.
kubectl -n kube-system logs -l component=kube-controller-manager \
  | grep -i "too many missed start times"

A 404 from the ping means the token is wrong. Confirm the Secret is in the same namespace as the CronJob and that the key name matches secretKeyRef.

If the job runs but the ping never arrives, check egress: a NetworkPolicy or a mesh sidecar that blocks outbound traffic will stop the ping while the work itself succeeds.

If LAST SCHEDULE in kubectl get cronjob is stale, the controller is not creating Jobs. Check for the missed-start-times message, and set startingDeadlineSeconds if it is unset.

If the pings stop only sometimes, look at whether runs are overlapping their own schedule under Forbid. Longer runs skip more starts.