How to Capture Application Logs When Using Amazon EKS on AWS Fargate

Capturing and persisting logs reliably is critical in containerized environments. In scenarios where Amazon EKS runs workloads on AWS Fargate, traditional node-level logging and DaemonSet-based collectors aren’t available. Instead, modern solutions leverage built-in Fluent Bit routing or sidecar patterns to funnel application logs to centralized destinations like CloudWatch, OpenSearch, or third-party systems.

This blog demonstrates best practices, walkthroughs, and considerations for logging application output when operating EKS pods on Fargate.

Kubernetes Logging Fundamentals

Kubernetes logging typically spans three levels:

• Basic (Pod-level): kubectl logs to view stdout/stderr.

• Node-level: Container runtime writes logs to host file system (e.g., /var/log/pods/*.log).

• Cluster-level aggregation: Agents like Fluent Bit, Fluentd run on nodes to collect and ship logs.

On Fargate, there are no EC2 nodes available to host DaemonSets. As such, alternative mechanisms must be used to capture logs.

Option 1: Built-in Fluent Bit Log Router (Recommended)

AWS recently introduced a native Fluent Bit–based logging router for EKS on Fargate, eliminating the need for sidecar containers.

How to Use It

• Create the Namespace:

kind: Namespace 
apiVersion: v1 
metadata: 
  name: aws-observability 
  labels: 
    aws-observability: enabled

• Define a ConfigMap named aws-logging in this namespace using Fluent Bit's config syntax. Example to send logs to CloudWatch:

kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
  data:
    output.conf: |
      [OUTPUT]
         Name cloudwatch_logs
         Match *
         region us-east-1
         log_group_name eks-fargate-app-logs
         log_stream_prefix from-fluent-bit-
         auto_create_group On

• Apply the config:

kubectl apply -f aws-observability-namespace.yaml kubectl apply -f fluentbit-config.yaml

• Ensure Permissions: Attach an IAM policy granting CloudWatch (or chosen destination) write access to the Fargate Pod Execution Role.

Benefits:

• Simple, scalable, and fully managed by AWS.

• Supports destinations like CloudWatch, OpenSearch, Kinesis Firehose/Streams, S3, and via intermediaries, third-party services.

Note: ConfigMap edits only apply to pods created after the update. Existing pods remain unaffected.

Option 2: Fluentd / Fluent Bit Sidecar + File Logging

If native Fluent Bit router isn’t suitable or older clusters require manual control:

1. Configure your application to log to both stdout and a file (e.g. /var/log/containers/app.log) using tee.

2. Add a Fluentd/Fluent Bit sidecar in the pod to tail these logs and ship them to CloudWatch.

3. Optionally include a logrotate sidecar to rotate logs, preventing disk overflow. Fargate pods have a 20 GB ephemeral storage limit.

This pattern requires modifying the application and pod spec but lets you maintain kubectl logs compatibility while ensuring persistent logging. For more details refer: Amazon Web Services

Option 3: Persist Logs via Amazon EFS (Durable Storage)

When applications need durability beyond ephemeral storage:

• Use Amazon EFS via the CSI driver to mount persistent volumes in Fargate pods and write logs to it.

• Logs remain accessible even after pod termination, and you can have Fluent Bit forward them to desired sinks. For more details refer: Amazon Web Services

This is particularly useful in failure scenarios or when logs must be retained locally for extended periods before shipping.

Option 4: Forward Logs to AWS CloudTrail Lake via S3

For long-term analytics, immutability, and audit scenarios:

1. Use Fluent Bit to push logs to S3.

2. Trigger ingestion pipelines that load logs into AWS CloudTrail Lake for SQL queries, security analysis, and forensic use cases. For more details refer: Amazon Web Services

This architecture provides centralized, queryable, immutable storage of container logs.

Summary Table: Option Comparison

Option	Setup Effort	Durability	Kubernetes Logging Experience	Best For
Built-in Fluent Bit Router	Low	Ephemeral	Native stdout capture	General logging with minimal overhead
Fluentd/Fluent Bit Sidecar + File	Medium	Ephemeral	kubectl logs + file support	Custom control, older clusters
EFS Persistence	Medium–High	Durable	Requires sidecar or router	Long-term local retention
S3 → CloudTrail Lake	High	Immutable	Indirect access	Auditing, analytics, compliance use cases

Best Practices (from AWS Prescriptive Guidance)

• Centralize logging (e.g., CloudWatch, OpenSearch).

• Enable structured logs (JSON, metadata, timestamp, log levels).

• Log to stdout/stderr for portability.

• Implement log rotation to manage storage.

• Set retention policies and secure logs (encryption, IAM).

• Monitor ingestion health (failures, delays).

• Use sampling for high-volume logs.

• Document your logging strategy. Refer AWS Documentation

Sample Fluent Bit Config - Sending to OpenSearch

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  output.conf: |
    [OUTPUT]
      Name es
      Match *
      Host your-opensearch-domain.endpoint
      Port 443
      tls On
      AWS_Auth On
      Index pod-logs
      Type _doc

Then apply, ensure IAM permissions, and your logs stream directly into OpenSearch.

Conclusion

For Amazon EKS on Fargate, the most streamlined logging path is to use AWS’s built-in Fluent Bit log router. It requires minimal setup, scales effortlessly, and supports a variety of output paths.

If durability, auditability, or custom control are priorities, consider using EFS-based persistence, sidecar collectors, or structured log pipelines into S3 and CloudTrail Lake.

Ready to level up your Kubernetes observability?

Let the cloud-native experts at Ananta Cloud help you architect scalable, secure, and cost-effective logging solutions for Amazon EKS on Fargate.

---

Schedule a Call | Contact us | LinkedIn