Monitor a Kubernetes CronJob
A Kubernetes CronJob can stop creating Jobs entirely and leave nothing behind to alert on. The controller skips a schedule when the previous run is still going under concurrencyPolicy: Forbid, and stops scheduling altogether after 100 missed start times. A heartbeat catches both.
By Paul, published , last verified
How a CronJob stops without telling you
The Kubernetes documentation is explicit that CronJobs have limitations, and the sharp one is the missed-start-time counter. For each CronJob the controller counts how many schedules were missed since the last scheduled time. If that count passes 100, it stops starting the Job and logs "too many missed start times", which is a controller log line, not a Kubernetes Event most alerting will notice.
This is reachable in ordinary operation. With startingDeadlineSeconds unset and a job scheduled every minute, a control plane outage of under two hours is enough to exceed 100 missed schedules and stop the job permanently. Setting startingDeadlineSeconds changes the counting window to that many seconds instead of since the last run, which is why the Kubernetes docs recommend setting it.
The other silent case is concurrencyPolicy: Forbid. A run that outlives its own schedule causes the next start to be skipped and counted as missed. A job that gradually slows down can go from occasionally skipping to never running, and every individual Job that did run succeeded.
A CronJob that reports in
Put the token in a Secret and reference it as an environment variable. The ping goes at the end of the command, joined with && so it only runs when the work succeeded.
startingDeadlineSeconds is set here deliberately, both because it bounds the missed-schedule counting window and because it keeps a controller restart from replaying a backlog. Keep it above 10 seconds: the controller polls every 10 seconds, and a smaller deadline can cause the job to be missed entirely.
apiVersion: v1
kind: Secret
metadata:
name: alplus-heartbeat
stringData:
token: hb_YOUR_TOKEN
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: nightly-report
spec:
schedule: "17 3 * * *"
timeZone: "Etc/UTC"
# Bounds the missed-schedule window and avoids a catch-up storm.
startingDeadlineSeconds: 300
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: report
image: your-registry/report:1.4.0
env:
- name: POSTDEPLOY_HEARTBEAT_TOKEN
valueFrom:
secretKeyRef:
name: alplus-heartbeat
key: token
command: ["/bin/sh", "-c"]
args:
- >-
/app/bin/report &&
curl -fsS "https://ingest.postdeploy.dev/h/$POSTDEPLOY_HEARTBEAT_TOKEN"
Reporting the exit code instead
The && form stays silent on failure and waits for the grace period. To alert as soon as a run fails, report the exit code: run the work, capture $?, and send it as a path segment. A 0 is a success and anything else opens an incident immediately.
Note the image needs curl available. If your job image is distroless or scratch, this is the tradeoff: either add curl, or accept detection at the grace period rather than at failure time.
args:
- >-
/app/bin/report; code=$?;
curl -fsS "https://ingest.postdeploy.dev/h/$POSTDEPLOY_HEARTBEAT_TOKEN/$code";
exit $code
Verify it, including the failure path
Trigger a run immediately rather than waiting for the schedule, then confirm the monitor is passing in PostDeploy.
# Run it now, without waiting for the schedule.
kubectl create job --from=cronjob/nightly-report nightly-report-manual
# Watch it, then confirm the ping was sent.
kubectl logs -f job/nightly-report-manual
# Check the controller's view of missed schedules.
kubectl get cronjob nightly-report -o wide
# If runs stopped, this is the line to look for.
kubectl -n kube-system logs -l component=kube-controller-manager \
| grep -i "too many missed start times"
Troubleshooting
A 404 from the ping means the token is wrong. Confirm the Secret is in the same namespace as the CronJob and that the key name matches secretKeyRef.
If the job runs but the ping never arrives, check egress: a NetworkPolicy or a mesh sidecar that blocks outbound traffic will stop the ping while the work itself succeeds.
If LAST SCHEDULE in kubectl get cronjob is stale, the controller is not creating Jobs. Check for the missed-start-times message, and set startingDeadlineSeconds if it is unset.
If the pings stop only sometimes, look at whether runs are overlapping their own schedule under Forbid. Longer runs skip more starts.
Questions
Why did my CronJob stop creating Jobs?
Most often the controller exceeded 100 missed start times and stopped scheduling, which it reports in the kube-controller-manager log rather than as an Event. Setting startingDeadlineSeconds changes the counting window to that many seconds and prevents the permanent stop.
Do I need a sidecar to send the heartbeat?
No. The ping is one outbound HTTPS request from the job container itself, so a curl call in the command is enough. A sidecar would add a container that has to be reasoned about on every run.
What should startingDeadlineSeconds be?
Above 10 seconds, because the controller polls on a 10-second cycle and a smaller value can cause the job to be skipped. Beyond that, set it to how late a run can start and still be worth running.
My job image has no curl. What are my options?
Use whatever HTTP client the image already has, since GET and POST are equivalent and no headers are required. Otherwise install curl in the image, or accept that failures are detected when the grace period expires rather than immediately.
Ship your next project with one toolkit, not three.
Sign up and every feature described on this page is yours on the trial. No credit card.
Create a heartbeat