Tuesday, September 3, 2024

How to Monitoring Your On-Premise Servers with AWS CloudWatch Agent

  September 03, 2024 Ihsanuddin Idharo   , , ,   No comments   Edit

In today's hybrid cloud environment, managing your on-premise servers can present unique challenges. You might want to consolidate data monitoring from your on-premise servers with the data you collect in the AWS Cloud. A great solution for this is using the AWS CloudWatch Agent.

This article will walk you through the steps of installing and configuring the AWS CloudWatch Agent on your Debian-based on-premise servers using AWS Systems Manager (SSM). For ease of understanding, we will use an EC2 instance in a separate AWS account to represent your on-premise server.

Objectives:

  • Understand how to register Debian-based on-premise servers to AWS Systems Manager using Hybrid Activation.

  • Configure how to install and set up the CloudWatch Agent on the server.

  • Send logs from your on-premise server to AWS CloudWatch.

Prerequisites:

  • An AWS account with the necessary permissions.

  • A Debian-based server (in this example, we will use an EC2 instance in a different AWS account).

Step 1: Setting Up Hybrid Activation in SSM

Before registering your on-premise server, you need to create a Hybrid Activation in SSM. This process will give you the Activation Code and Activation ID needed to register your server.

  1. Access the Systems Manager Console: Login to the AWS Management Console and open the Systems Manager console.

  2. Create a New Hybrid Activation:

    • In the Systems Manager navigation pane, choose Hybrid Activations under Node Management.

    • Click Create Activation.

  3. Configure the Hybrid Activation:

    • Activation Description: Provide a clear description, for example, "On-Premise Server Registration".

    • Instance Limit: Specify the number of on-premise servers you want to register.

    • IAM Role: Choose or create an IAM role that has the necessary permissions for Systems Manager.

    • Registration Expiration Date: Set an expiration date for this activation; after this date, the activation will no longer be usable.

    • Click Create Activation.

  4. Save the Activation Code and Activation ID: After creating the activation, you will receive an Activation Code and Activation ID. Carefully note this information as you will need it later to register your on-premise server.

Step 2: Registering Your On-Premise Server to AWS Systems Manager

With your Hybrid Activation in place, you can now register your Debian server to AWS Systems Manager.

  1. Update the Package List: Start by updating your server's package list:

          sudo apt-get update

  2. Install SSM Agent: Download and install the SSM Agent:

          mkdir /tmp/ssm
cd /tmp/ssm
wget https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/debian_amd64/amazon-ssm-agent.deb
sudo dpkg -i amazon-ssm-agent.deb

  3. Register the Server to SSM: Register your server using the Activation Code, Activation ID, and your AWS region:

          sudo amazon-ssm-agent -register -code "<your-activation-code>" -id "<your-activation-id>" -region "<your-region>"

    Example:

          sudo amazon-ssm-agent -register -code "h7FfWBbOrDCeXexxxxxx" -id "914e2266-e1c1-4c3a-b638-2azzzzzzzzzz" -region "eu-central-1"

  4. Start the SSM Agent: Once registered, start the SSM Agent:

          sudo systemctl start amazon-ssm-agent

  5. Enable the SSM Agent to Start on Boot: Ensure the agent starts automatically on boot:

          sudo systemctl enable amazon-ssm-agent

  6. Verify the SSM Agent Status: Finally, confirm that the agent is running:

          sudo systemctl status amazon-ssm-agent

Your Debian server is now successfully registered with AWS Systems Manager and can be managed through the AWS Management Console.

Step 3: Install and Configure Nginx (For Log Collection)

To generate logs for the CloudWatch Agent, let's install Nginx on your Debian server.

  1. Install Nginx:

    sudo apt update
sudo apt install nginx
sudo systemctl status nginx

  2. Nginx Log Files: The Nginx log files we will send to Cloudwatch are:

    • /var/log/nginx/error.log

    • /var/log/nginx/access.log

Step 4: Install CloudWatch Agent Using SSM

Now, let's leverage the SSM Agent to install the CloudWatch Agent on our server.

  1. Access the Systems Manager Console: Go to the Systems Manager console in your AWS Management Console.

  2. Run Command to Install CloudWatch Agent:

    • Navigate to Run Command.

    • Choose AWS-ConfigureAWSPackage from the list of command documents.

    • Select your on-premise server as the target.

    • Set Action to Install.

    • Enter AmazonCloudWatchAgent in the Name box.

    • Leave the Version field blank to install the latest version.

    • Choose Run.

The CloudWatch Agent will now be installed on your server.

Step 5: Configure CloudWatch Agent

To enable the CloudWatch Agent to send logs from your on-premise server to AWS CloudWatch, you need to set up an IAM user with the required permissions, configure your server to use this IAM user's credentials, and ensure that the CloudWatch Agent is configured correctly to use these credentials.

  1. Create an IAM User with Necessary Permissions:

    • First, you need to create an IAM user that has permissions to send logs to CloudWatch.

    • Steps to create IAM user:

      1. Log in to the AWS Management Console and open the IAM console.

      2. Create a New User:

        • Navigate to Users and click Add user.

        • Enter a user name (e.g., CloudWatchAgentUser).

        • Under Access type, select Programmatic access to generate an access key ID and secret access key for this user.

      3. Attach Permissions:

        • Click Attach policies directly.

        • Attach the following managed policies to the user:

          • CloudWatchAgentServerPolicy

          • CloudWatchAgentAdminPolicy

          • AmazonSSMManagedInstanceCore
            These policies provide the necessary permissions to send logs to CloudWatch, access SSM, and interact with the CloudWatch Agent.

      4. Finish Creating the User:

        • Proceed to review and create the user.

        • On the final page, ensure you download the .csv file containing the Access Key ID and Secret Access Key, or copy them to a secure location. You will need these credentials in the next step.

  2. Configure the Server with the IAM User Credentials:

    • Now that you have the Access Key ID and Secret Access Key, you need to configure your server to use these credentials by creating an AWS CLI profile named AmazonCloudWatchAgent.

    • Configure AWS CLI with IAM User Credentials:

      • On your on-premise server, run the following command to configure the AWS CLI with the IAM user credentials:

              sudo aws configure --profile AmazonCloudWatchAgent

      • When prompted, enter the following details:

        • AWS Access Key ID: Enter the Access Key ID you obtained earlier.

        • AWS Secret Access Key: Enter the Secret Access Key.

        • Default region name: Enter the region where you want the logs to be sent (e.g., eu-central-1).

        • Default output format: Leave this field blank or enter json.
          This creates a profile named AmazonCloudWatchAgent on your server that the CloudWatch Agent will use to send logs to AWS CloudWatch.

  3. Update the CloudWatch Agent Configuration:

    • If you are simulating an on-premise environment using an EC2 instance, you might need to update the CloudWatch Agent configuration file (common-config.toml) to use the newly created profile.

    • Update the Configuration File:

      • Open the common-config.toml file:

              sudo nano /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

      • Uncomment and update the following section to include the profile name and credentials file:

              [credentials]
shared_credential_profile = "AmazonCloudWatchAgent"
shared_credential_file = "/root/.aws/credentials"

      • Save and exit the file.

Step 6: Start CloudWatch Agent with the Pre-Created Configuration File Using SSM

In this step, we will use a pre-created configuration file stored in the AWS Systems Manager Parameter Store to start the CloudWatch Agent.

  1. Create and Store the Configuration File in SSM Parameter Store:

    • First, create the CloudWatch Agent configuration file on your local machine. Here is an example configuration:

            {
    "agent": {
        "metrics_collection_interval": 60,
        "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log"
    },
    "logs": {
        "logs_collected": {
            "files": {
                "collect_list": [
                    {
                        "file_path": "/var/log/nginx/access.log",
                        "log_group_class": "INFREQUENT_ACCESS",
                        "log_group_name": "{instance_id}-nginx-access.log",
                        "log_stream_name": "{instance_id}",
                        "retention_in_days": 7
                    },
                    {
                        "file_path": "/var/log/nginx/error.log",
                        "log_group_class": "INFREQUENT_ACCESS",
                        "log_group_name": "{instance_id}-nginx-error.log",
                        "log_stream_name": "{instance_id}",
                        "retention_in_days": 7
                    }
                ]
            }
        }
    }
}

    • Once your configuration file is ready, store it in the AWS Systems Manager Parameter Store:

            aws ssm put-parameter --name "CloudWatchAgentConfig" --type "String" --value file://configuration_file_pathname

      Replace configuration_file_pathname with the actual path to your configuration file.

    • For more information on how to create a cloudwatch agent configuration file, visit [Link].

  2. Access the Systems Manager Console:

    • Return to the Systems Manager console.

  3. Run Command to Start CloudWatch Agent:

    • Navigate to Run Command.

    • Choose AmazonCloudWatch-ManageAgent from the Command documents.

    • Select your on-premise server as the target.

    • Set Action to configure.

    • Set Mode to onPremise.

    • In the Optional Configuration Location box, enter the name of the configuration file stored in the Parameter Store (e.g., CloudWatchAgentConfig).

    • Choose Run.

The CloudWatch Agent will now start with the specified configuration.

Verifying CloudWatch Agent Logs

You can monitor the CloudWatch Agent logs by running:

      sudo tail -f /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

Notes on KMS Encryption for Log Groups

Currently, CloudWatch Agent does not support KMS encryption for log groups during creation. A workaround is to allow the agent to create the log groups first and then manually associate them with a KMS key using the following command:

      aws logs associate-kms-key --log-group-name LOG_GROUP_NAME --kms-key-id KEY_ARN

There is an ongoing feature request regarding this issue which you can track here.

Conclusion

This article provides a practical guide for anyone needing to register their on-premise servers to AWS Systems Manager (SSM) and install the CloudWatch Agent using SSM. By following the steps outlined here, you can quickly and efficiently integrate your servers with SSM and configure CloudWatch Agent to monitor and log your system activity, ensuring you maintain visibility and control over your infrastructure, both on-premise and in the cloud.

0 comments:

Post a Comment

Subscribe to: Post Comments (Atom)