Basic Docker Container Monitoring with Datadog

The advent of containerization has transformed the deployment landscape, offering scalability and efficiency. However, effective monitoring is paramount to ensuring optimal performance and troubleshooting issues. In this blog post, we'll explore the basics of container monitoring using the DataDog Agent, covering essential steps from setup to creating a sample dashboard and configuring alerts.

Setting Up the DataDog Platform

To embark on your container monitoring journey, begin by signing up for the DataDog platform. Navigate to Integrations -> Agent -> Docker, where you'll find detailed steps for installing the Docker agent. Before configuring the agent, ensure you set up an API key, a crucial component for seamless integration.

Configuring the DataDog Agent for Container Monitoring

To monitor containerized applications, you have the option of installing the DataDog agent directly on the host machine or deploying a containerized agent to monitor other Docker containers. The agent installation process offers various options, such as enabling Traces (APM), Log collection, Tag extraction, and Prometheus integrations. In this blog post, we'll focus on the fundamental log collection method.

Follow these steps to install the agent for log collection:

docker run -d --name datadog-agent \
  --cgroupns host \
  --pid host \
  -e DD_API_KEY=<DATADOG_API_KEY> \
  -e DD_LOGS_ENABLED=true \
  -e DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true \
  -e DD_CONTAINER_EXCLUDE="name:datadog-agent" \
  -e DD_SITE=<DD_SITE>
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /var/lib/docker/containers:/var/lib/docker/containers:ro \
  -v /proc/:/host/proc/:ro \
  -v /opt/datadog-agent/run:/opt/datadog-agent/run:rw \
  -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
  gcr.io/datadoghq/agent:latest

Replace <DATADOG_API_KEY> and <DD_SITE> with your API key and site name, respectively. Once executed, the agent will start sending data to the DataDog platform.

Sample Dashboard Setup

Easily set up a sample dashboard by cloning DataDog's default container dashboard. Configure it to the host running Docker containers, providing an instant visual representation of container metrics.

Basic Metrics for Monitoring

Initiate container monitoring with a few key metrics:

• docker.cpu.limit: Limit on CPU available to the container (percentage of a core).

• docker.mem.rss: Non-cache memory belonging to the container's processes.

• docker.containers.running: Number of running containers on the host (tagged by image).

• docker.containers.stopped: Number of stopped containers on the host (tagged by image).

• docker.mem.limit: Memory limit for the container, if set.

• docker.mem.in_use: Fraction of used memory to available memory (if the limit is set).

• docker.uptime: Container uptime.

Additionally, consider monitoring Docker events such as Delete Image, Die, Error, Fail, Kill, Out of memory (oom), Pause, Restart container, Restart Daemon, and Update.

Alerts for Metrics:

1. Navigate to Monitoring -> New Monitor -> Metric: Begin by accessing the DataDog dashboard and head to the Monitoring section. Choose "New Monitor" and select the option for creating a metric alert.

2. Define Metric: In the Metric Dashboard, choose the metric you want to monitor. This could be any of the basic metrics discussed earlier, such as docker.cpu.limit, docker.mem.rss, or others depending on your monitoring requirements.

3. Set Alert Conditions: On the Set Alert Conditions section, define the thresholds for triggering alerts. Analyze the monitoring graphs displayed on the screen to determine appropriate values for Alert and Warn thresholds. These values signify when an alert should be triggered based on the metric's behavior.

4. Notify Your Team: Configure the notification details for your team. Specify the message, team members who need to be alerted, and any additional information that is crucial for understanding and responding to the alert effectively.

5. Create Alert: Finalize the alert configuration and create the alert. This ensures that the DataDog platform will actively monitor the selected metric, and whenever the defined conditions are met, it will notify the specified team members.

Alerts for Events:

1. Navigate to Monitoring -> New Monitor -> Events: To set up alerts for events, go to the Monitoring section, select "New Monitor," and choose the option for creating an event alert.

2. Define the Query: On the Events screen, define the query by selecting the host and source relevant to your containerized application. Set the 'Show Count of * by' to Container Name, Status, and Message to gain insights into specific details surrounding the events.

3. Set Alert Conditions: In this section, determine the conditions that should trigger an alert. For events, this typically involves setting the Alert threshold value. If the count of certain events surpasses this threshold, an alert will be triggered.

4. Notify Your Team: Similar to metric alerts, configure the notification settings for your team. Include information about the event, the team members to be notified, and any other critical details.

5. Create Event: Once the alert conditions and notifications are set, create the event. This action ensures that DataDog actively monitors events, and if the specified conditions are met, it will promptly alert the designated team members.

By following these steps, you establish a basic alerting system that keeps you informed about critical metrics and events within your Docker containerized environment. Adjust these settings based on the specific needs of your application and organization for effective monitoring and incident response.