Hello Devops Amigos!!! Today We will construct a powerful log aggregation system using AWS Lambda, CloudWatch, and Amazon OpenSearch. By the end of this blog, you'll have a sophisticated setup that can scale with your needs, offering valuable insights into system usage, aiding in debugging, and enhancing overall operational efficiency. So, let's dive in!

This blog covers everything from setting up Amazon OpenSearch to creating visualizations and dashboards using Kibana, empowering you to manage and monitor your distributed systems effectively.

Architecture

Before

After

Lab Step 1: Creating Logs Using AWS Lambda

Welcome to the AWS Lambda playground! We're about to execute a Lambda function and witness the magic of log generation sent straight to CloudWatch.

1. Navigating to AWS Lambda

   • Open the AWS Management Console.

   • Enter "Lambda" in the search bar and click on the Lambda result under Services.

1. Accessing the Lambda Function

   • Locate and click on the cloudacademylabs-DynamoLambda function in the Functions list.

1. Exploring Lambda Function Console

   • Take a moment to explore the Lambda function console, understanding the Designer and Function code sections.

1. Configuring a Test Event

   • Scroll down to the Code source section and click Test to configure a test event.

2. Configuring Test Event Details

   • Enter TestPutEvent as the Event name.

   • Enter the provided JSON in the code editor.

   • Click Save.

1. Running the Function with Test Event

   • Run the function by clicking Test again.

   • View the Execution results tab for function logs.

1. Viewing CloudWatch Logs

   • Click the Monitor tab.

   • Click View CloudWatch Logs.

Conclusion
In this step, you executed a Lambda function, triggered a test event, and viewed the generated logs in Amazon CloudWatch. This process showcases how AWS Lambda seamlessly sends logs to CloudWatch, laying the foundation for effective log analysis.

Lab Step 2: Manually Viewing Logs in Amazon CloudWatch

Now, let's delve into CloudWatch Logs exploration, manually inspecting logs to gain insights into log streams and filtering capabilities.

1. Observing Log Streams

   • Observe Log Streams in the CloudWatch log group for the Lambda function.

1. Viewing Log Stream

   • Click on the latest Log Stream to explore its log events.

1. Filtering Events

   • Enter "PUT" into the Filter events search bar and press enter.

1. Expanding Event Details

   • Click the triangle to expand the event that matches the filter.

2. Custom Time Range Filter

   • Click custom to display the custom time range filter.

Conclusion

In this step, you explored log filtering capabilities in Amazon CloudWatch Logs, understanding how logs are organized into Log Groups, Log Streams, and Events.

Lab Step 3: Launching the OpenSearch Domain

Now, let's set up your Amazon OpenSearch domain, the backbone of our advanced log aggregation system.

1. Navigate to Amazon OpenSearch Service

   • In the AWS Management Console, enter "OpenSearch" in the search bar.

   • Click on the Amazon OpenSearch Service result.

     

2. Start Creating the Domain

   • On the welcome page, click Create domain.

     

3. Fill in Domain Details

   • Enter a unique Domain name.

   • Choose Standard create as the Domain creation method.

   • Select Dev/test in the Templates section.

   • Choose Domain without standby and Availability Zone(s): 1-AZ in the Deployment Option(s) section.

     

1. Configure Engine Options and Data Nodes

   • Choose Elasticsearch 6.8 under Engine options.

   • Specify instance type, number of nodes, and EBS storage size per node.

1. Network Configuration

   • Select Public access in the Network section.

2. Fine-grained Access Control

   • Uncheck Enable fine-grained access control in the Fine-grained access control section.

1. Domain Access Policy

   • Configure domain access policy using the visual editor, allowing access based on your IP address.

   • Get IP address from here -> checkip.amazonaws.com

1. Create the Domain

   • Scroll to the bottom and click Create.

   • Monitor the provisioning status; it usually takes 20-30 minutes.

Conclusion

Successfully configured an Amazon OpenSearch domain, the foundation of our advanced log aggregation system. Once it's active, we'll move on to streaming logs into OpenSearch and exploring its capabilities.

Lab Step 4: Sending CloudWatch Logs to OpenSearch

Now, let's establish the integration between CloudWatch Logs and your OpenSearch domain. Follow these steps to create a subscription filter and seamlessly stream logs into OpenSearch.

1. Navigate to CloudWatch

   • In the AWS Management Console, enter "CloudWatch" in the search bar.

   • Click on the CloudWatch result.

     

2. Access Log Groups

   • In the left-hand menu, under Logs, click on Log groups.

     

3. Select Log Group

   • Select the log group for your Lambda function.

4. Create Subscription Filter

   • Click Actions, and under Subscription filters, click Create Amazon OpenSearch Service subscription filter.

1. Choose Destination

   • Ensure This account is selected as the Select account.

   • Choose the OpenSearch cluster you created earlier.

1. Select Lambda IAM Execution Role

   • In the Lambda IAM Execution Role drop-down, select LambdaElasticSearch.

1. Configure Log Format and Filters

   • Select Amazon Lambda as the Log Format.

   • Enter ca-lab-filter as the Subscription filter name.

1. Start Streaming

   • Click Start streaming at the bottom.

     

Conclusion

Successfully created an Amazon OpenSearch Service subscription filter, connecting your Lambda function's log group to automatically stream logs into your OpenSearch domain. The groundwork is set for us to discover and search events in OpenSearch.

Lab Step 5: Discovering and Searching Events

Now, let's create more test events and explore the discovery and search functionality in Kibana.

1. Create More Test Events

   • Return to the Lambda function and click Test to submit more PUT events.

2. Configure Test Event

   • Click the arrow on Test and then click Configure test event.

     

3. Create New Test Event

   • Create new test events with the provided JSON for GET events.

{
  "fn": "GET",
  "id": "12345"
}

• Save the test event.

1. Make GET Events

   • Click Test several times to generate GET events.

2. Access Kibana

   • Return to the Amazon OpenSearch Search Console.

   • Click the link under Kibana URL.

1. Connect to Elasticsearch Index

   • In the Add Data to Kibana section, click Connect to your Elasticsearch index.

     

2. Create Index Pattern

   • Enter cwl-* as the Index pattern in the Create an index pattern wizard.

   • Click Create index pattern.

   • Index pattern: Select timestamp in Next Step

1. Explore the Discover Interface

   • Click Discover in the sidebar menu to explore the Discover interface.

2. Search and Filter

   • Enter PUT 12345 in the search bar and observe the highlighted events.

Conclusion

You've learned how to use Kibana's discover capabilities to explore and search through data stored in Amazon OpenSearch. The journey continues as we proceed to visualize aggregated events in Kibana.

Lab Step 6: Visualizing Aggregated Events

Great job! Now, let's harness Kibana's visualization capabilities to create an area chart showcasing different log request types over time.

1. Access Visualize in Kibana

   • Click Visualize in the Kibana sidebar menu.

2. Create a Visualization

   • Click Create a visualization.

     

3. Select Area Chart

   • Choose Area chart visualization.

     

4. Choose Index

   • Select the cwl-* index name.

     

5. Configure X-Axis

   • Configure the X-Axis with Date Histogram.

1. Add Sub-Buckets

   • Add Sub-Buckets for Split Series, using Terms on $event.data.fn.keyword.

2. Apply Changes

   • Click the play button to apply changes and produce the visualization.

     

3. Save Visualization

   • Save the visualization with the name "PUTs and GETs Over Time."

Conclusion

In this step, you created an area chart visualization in Kibana, offering insights into different log request types over time. Kibana provides a plethora of visualization tools that we'll explore further.

Lab Step 7: Creating a Kibana Dashboard

Let's tie everything together by creating Kibana dashboard, combining visualizations to provide an overview of the entire system.

1. Access Dashboard in Kibana

   • Click Dashboard in the sidebar menu.

2. Create a Dashboard

   • Click Create a dashboard.

     

3. Add Saved Visualization

   • Add the saved visualization "PUTs and GETs Over Time" to the dashboard.

   • 

4. Adjust Visualization Size

   • Adjust the size of the visualization as needed.

5. Save Dashboard

   • Save the dashboard with the title "Log Dashboard" and a description like "Lambda API Logs."

1. Generate Test Events

   • Return to the Lambda console and create test events.

2. Refresh Kibana Dashboard

   • Refresh the Kibana dashboard to see new requests in the visualization.

   • Configure Auto-refresh for real-time updates.

     

Conclusion

Congratulations! You've completed the task, constructing a sophisticated log aggregation system using Kibana running on the Amazon OpenSearch Service. This system is a valuable tool for your team, offering insights into system usage, aiding in debugging, and providing operational visibility.

Resources

This is lab from CloudAcademy:
https://cloudacademy.com/lab/aws-devops-pro-monitoring-build-log-aggregation-system/
How to Implement & Enable Logging Across AWS Services (Part 1 of 2)
How to Implement & Enable Logging Across AWS Services (Part 2 of 2)
Understanding AWS Lambda to Run & Scale Your Code
https://docs.aws.amazon.com/cloudwatch/
https://docs.aws.amazon.com/lambda/latest/operatorguide/monitoring-observability.html
https://docs.aws.amazon.com/opensearch-service/latest/developerguide/admin-options.html

Thank you for joining this journey! I hope you've gained valuable skills in CloudWatch Logs, Amazon OpenSearch, and Kibana. If you have any questions or need further assistance, feel free to reach out me at Linkedin. Happy logging! 🚀🔍

If you enjoy content like this, please hit the follow button and subscribe to my newsletter to stay updated on future blogs.