Skip to content

A Kubernetes CronJob that runs every minute starts new Jobs while the old ones still run

February 6, 2025

A CronJob scheduled every minute filled up the pod limit. I first blamed the failed pods, but a pod that has finished (failed or succeeded) no longer counts. The ResourceQuota docs only count pods in a non-terminal state, and the scheduler ignores finished pods as well.

What piles up are Jobs that haven’t finished. The default concurrencyPolicy is Allow, so the CronJob starts a new Job every minute, even while earlier ones are still running. A pod that hangs on a dead downstream service stays in Running and keeps its place in the limit. So does one that restarts in backoff under restartPolicy: OnFailure. And one more joins every minute.

The fix is to stop the overlap and put a deadline on each run:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: my-job
spec:
  schedule: "* * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      activeDeadlineSeconds: 50
      backoffLimit: 0
      ttlSecondsAfterFinished: 300
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: my-job
              image: my-image

Forbid skips a run while the previous one is still active. activeDeadlineSeconds below the schedule interval ends a hanging run before the next one is due. Keep the resource requests, though. Removing them just hides the pods from the scheduler, and the pods don’t get any smaller.

—

Back to TIL

Next: Tailwind 4 theme values are CSS custom properties that JavaScript can read

Phil-Bastian Berndt

Phil-Bastian Berndt
Tech Lead at Naymspace