Troubleshooting & Errors

How Do You Fix Common Argo Workflows Errors?

Diagnose and fix common Argo Workflows errors — RBAC and service accounts, pending pods, OOMKilled, image pulls, artifacts, parameter resolution, DAG cycles, UI login, CronWorkflows and controller issues.

10 min read 2,010 wordsUpdated By Argo Workflows Editorial Team

When an Argo Workflow fails, the cause is almost always one of a small number of categories: validation, permissions, scheduling, the container itself, data passing or the platform. This guide gives you a systematic debugging process and then a catalogue of the most common error messages with their causes and fixes. Bookmark it — most problems you meet in production are listed here.

A Systematic Debugging Process

Before searching for an error message, gather the facts in a consistent order. Five commands answer most questions:

argo get -n <ns> <workflow>                 # node tree, phases and messages
argo logs -n <ns> <workflow> --follow       # logs from every pod
kubectl -n <ns> describe pod <pod-name>     # scheduling, image and container state
kubectl -n <ns> get events --sort-by=.lastTimestamp
kubectl -n argo logs deploy/workflow-controller --tail=200

Then work through these questions:

  1. Did the workflow start? If argo submit or kubectl apply failed, it is a validation or permissions problem with the manifest or your user account.
  2. Was a pod created? If the node is Pending without a pod, look at the controller logs and quotas.
  3. Did the pod start? If the pod exists but is Pending or stuck initialising, look at describe pod and events.
  4. Did the main container fail? Read its logs and exit code; this is your code or image.
  5. Did saving outputs fail? The wait container logs show artifact and task result problems.

Classifying the problem this way narrows the search immediately.

Validation and Manifest Errors

"spec.entrypoint ... template not found"

Cause: the entrypoint does not match any template name. Fix: correct the spelling; names are case-sensitive. Our validator flags this.

"templates.main.steps[0].x template name 'y' undefined"

Cause: a step or task references a template that does not exist. Fix: add the template or correct the reference. If it lives in a WorkflowTemplate, use templateRef instead of template.

"Unable to resolve: {{inputs.parameters.foo}}"

Cause: the template uses a parameter that is not declared in its inputs, or the caller did not supply a value. Fix: declare the input, give it a default, or pass it in arguments.

"Unable to resolve: {{steps.a.outputs.result}}"

Cause: a is not a script template, or it is in the same stage as the referencing step, or the name is wrong. Fix: use outputs.parameters for container templates, and move the consumer to a later stage.

"dependency cycle detected"

Cause: DAG tasks depend on each other in a loop. Fix: remove one edge. Visualise the graph with the DAG visualizer to see the cycle.

"did not find expected key" or "mapping values are not allowed here"

Cause: YAML syntax — indentation, tabs or unquoted {{ at the start of a value. Fix: use spaces, quote templated values, and run the file through the YAML formatter.

"unknown field" errors on version 4

Cause: fields removed in 4.0, such as CronWorkflow schedule or singular mutex/semaphore. Fix: use schedules, mutexes and semaphores lists.

Permission and RBAC Errors

"workflowtaskresults.argoproj.io is forbidden"

Cause: the workflow's service account cannot create or patch task results, which the executor needs to report outputs. Fix: bind a Role granting create and patch on workflowtaskresults to the service account, and set serviceAccountName in the workflow. The Kubernetes guide has the full manifest.

"pods is forbidden: User 'system:serviceaccount:...' cannot create resource"

Cause: a resource template or a tool inside a step tries to create Kubernetes objects without permission. Fix: grant the exact verbs and resources required to a dedicated service account used only by that template.

"workflows.argoproj.io is forbidden" when submitting

Cause: your own user or token lacks permission in the target namespace. Fix: ask an administrator for a role allowing create on workflows, or submit to a namespace you own.

UI shows "Unauthorized" or keeps returning to login

Cause: an expired or invalid token in client mode, or SSO misconfiguration. Fix: generate a fresh token, check the server's --auth-mode arguments, and verify the SSO redirect URL matches the external address exactly.

Scheduling and Pod Start-Up Problems

Pod stuck in Pending: "Insufficient cpu" or "Insufficient memory"

Cause: no node has enough free capacity for the requests. Fix: reduce requests, lower parallelism, or allow the autoscaler to add nodes. Check that requests are realistic rather than copied defaults.

"exceeded quota"

Cause: the namespace ResourceQuota is full. Remember the init and wait containers also consume quota. Fix: raise the quota, set executor resources in the controller ConfigMap, or reduce concurrency.

"node(s) didn't match Pod's node affinity/selector" or "had untolerated taint"

Cause: node selectors or tolerations do not match any node. Fix: correct labels and tolerations, or create a node pool that matches.

ImagePullBackOff or ErrImagePull

Cause: wrong image name or tag, private registry without credentials, or registry rate limits. Fix: verify the image exists, add imagePullSecrets, and mirror public images to your own registry to avoid rate limits.

Pod stuck in PodInitializing

Cause: the init container is slow or failing, usually while downloading input artifacts. Fix: run kubectl logs <pod> -c init and check artifact repository access and network policies.

Container Failures

"OOMKilled" or exit code 137

Cause: the container exceeded its memory limit and was killed by the kernel. Fix: increase the memory limit, process data in smaller chunks, or fix the leak. Exit code 137 without OOMKilled can also mean the pod was evicted or the node was reclaimed.

Exit code 1 or other non-zero code

Cause: your program failed. Fix: read the main container's logs. Add set -euo pipefail to shell scripts so errors are reported where they happen.

"Pod was active on the node longer than the specified deadline"

Cause: activeDeadlineSeconds on the template or workflow was exceeded. Fix: raise the deadline if the work is legitimately long, or investigate why the step hung.

"Step exceeded its deadline" during retries

Cause: the retry backoff's maxDuration elapsed. Fix: increase maxDuration or the retry limit, or fix the underlying flakiness.

Data Passing and Artifact Errors

"You need to configure artifact storage"

Cause: a template declares artifacts but no artifact repository is configured. Fix: configure artifactRepository in the controller ConfigMap or reference one with artifactRepositoryRef.

"failed to save outputs: ... no such file or directory"

Cause: the output path does not exist when the container exits. Fix: make sure the program writes to that path, or mark the artifact optional: true. For parameters, add valueFrom.default.

"Access Denied" or "NoCredentialProviders" from S3

Cause: the pod cannot authenticate to the object store. Fix: check IRSA or Workload Identity annotations on the service account, or the referenced access key Secret, and the bucket policy.

Output parameter contains unexpected whitespace

Cause: a trailing newline from echo. Fix: use echo -n or printf, which avoids breaking conditions and comparisons.

"workflow is longer than maximum allowed size"

Cause: the workflow status exceeds the etcd object size limit, typically with thousands of nodes or large outputs. Fix: enable node status offloading with a database, reduce output sizes, and split very large workflows into child workflows.

Controller and Platform Issues

Workflows are created but never start

Cause: the controller is not watching the namespace, is down, or another controller with a different instance ID owns the workflow. Fix: check controller logs, the namespaced or managed-namespace configuration and the workflows.argoproj.io/controller-instanceid label.

Workflows remain Running after all pods finish

Cause: the controller cannot update status, often due to API server throttling or executor RBAC failures. Fix: check controller logs for errors, verify task result permissions, and scale controller resources.

Pods deleted before you could debug them

Cause: an aggressive podGC strategy. Fix: use OnPodSuccess so failed pods remain, and ship logs to central storage.

CronWorkflow Issues

  • Not running at the expected time: set timezone; the default is the controller's timezone.
  • Runs skipped: concurrencyPolicy: Forbid with a previous run still active; add deadlines.
  • Duplicate runs after an outage: catch-up behaviour; set startingDeadlineSeconds.

See the CronWorkflow guide for details.

Reading Exit Codes Quickly

The exit code of the main container is often the fastest clue. These are the ones you will see most:

Exit code Usual meaning First thing to check
0 Success Output saving, if the node still failed
1 Generic application error Application logs
2 Misuse of a shell built-in or bad arguments Command and args quoting
126 Command found but not executable File permissions in the image
127 Command not found Image contents and PATH
137 Killed by SIGKILL Memory limit (OOMKilled), eviction or node loss
139 Segmentation fault Native library or binary compatibility
143 Terminated by SIGTERM Deadline, stop or terminate command

Exit codes 126 and 127 are especially common after switching to a minimal or distroless image, which may not contain a shell. If your template uses command: [sh, -c], make sure the image actually ships sh.

Debugging With the Argo UI

The UI turns many of the steps above into clicks. Selecting a node shows its phase, message, start and finish times, inputs, outputs and the pod name. The logs tab streams the main container by default and lets you switch to the init and wait containers, which is where artifact problems appear. The "Events" view lists Kubernetes events for the workflow's pods, so scheduling problems are visible without kubectl. For archived workflows, the UI shows the stored status even after the objects are deleted from the cluster, which is invaluable for investigating failures that happened overnight.

A Worked Example

Suppose a nightly workflow fails and argo get shows the load node in Error with the message "failed to save outputs". The node is Error rather than Failed, which points to something outside the application. The main container's logs end normally, so the application finished. The wait container's logs show "AccessDenied" from S3. The cause is now clear: the artifact upload failed because credentials changed. Checking the service account reveals that an IAM role annotation was removed during an unrelated refactor. Restoring the annotation and running argo retry completes the workflow without rerunning the earlier, expensive steps. Five commands and two minutes of reasoning were enough, because each step of the process eliminated a whole category of causes.

Preventing Problems Before They Happen

  1. Lint every manifest in CI with argo lint.
  2. Use dedicated service accounts with least-privilege RBAC.
  3. Set resource requests, memory limits and deadlines everywhere.
  4. Configure ttlStrategy and podGC defaults.
  5. Monitor controller metrics and alert on failure rates.
  6. Pin image tags and mirror critical images.
  7. Run new templates in a staging namespace first.

Run your manifests through the Configuration Checker to catch many of these gaps automatically.

Frequently Asked Questions

Where are the logs of a deleted pod?

Gone from Kubernetes. Enable archiveLogs to store logs as artifacts, or ship logs to a central system.

How do I rerun only the failed part?

Use argo retry <workflow>. Add --restart-successful --node-field-selector options to rerun selected successful nodes too.

Why does argo logs show nothing?

The pod may not have started yet, may have been garbage collected, or your account may lack pods/log permission. Check the node phase with argo get first, then permissions.

Why is my workflow stuck in Pending with no pods?

The controller may be throttled by a parallelism or semaphore limit, waiting for a mutex, or not watching that namespace. The workflow's status message usually says which.

How can I debug interactively?

Add a temporary sleep to the command, then kubectl exec into the pod. Remove it afterwards.

What is the difference between Failed and Error?

Failed means your container exited non-zero or a condition failed. Error means something outside your code failed, such as pod deletion or artifact upload.

Key Takeaways

  • Classify the failure first: validation, permissions, scheduling, container, data or platform.
  • Error and Failed phases point to different causes.
  • The wait container logs reveal most output and artifact problems.
  • Prevent recurring failures with linting, sensible defaults, monitoring and staging namespaces rather than fixing the same error twice.

Conclusion

Most Argo Workflows failures fall into predictable categories. Debug systematically — workflow, node, pod, container, outputs — and you will usually find the cause within minutes. Combine that habit with linting, least-privilege RBAC, resource requests and good cleanup defaults, and many errors never reach production at all. When you meet a new error, add it to your team's runbook so the next person fixes it faster.

Argo Workflows Editorial Team

Written and reviewed according to our editorial policy. Spotted an error? Tell us and we will fix it. This site is independent and not affiliated with the Argo Project.