Analyze logs with Dynatrace Davis AI Engine utilizing Amazon Kinesis Knowledge Firehose HTTP endpoint supply


This weblog publish is co-authored with Erick Leon, Sr. Technical Alliance Supervisor from Dynatrace.

Amazon Kinesis Knowledge Firehose is the best method to reliably load streaming knowledge into knowledge lakes, knowledge shops, and analytics companies. With only a few clicks, you’ll be able to create fully-managed supply streams that auto scale on demand to match the throughput of your knowledge. Clients already use Kinesis Knowledge Firehose to ingest uncooked knowledge from numerous knowledge sources, together with logs from AWS companies. Kinesis Knowledge Firehose now helps delivering streaming knowledge to Dynatrace. Dynatrace begins analyzing incoming knowledge inside minutes of Amazon CloudWatch knowledge era.

Beginning at this time, you should utilize Kinesis Knowledge Firehose to ship CloudWatch Metrics and Logs on to the Dynatrace observability platform to carry out your explorations and evaluation. Dynatrace, an AWS Accomplice Community (APN) has supplied full observability into AWS Companies by ingesting CloudWatch metrics which are revealed by AWS companies. Dynatrace ingests this knowledge to carry out root-cause evaluation utilizing the Dynatrace Davis AI engine.

On this publish, we describe the Kinesis Knowledge Firehose and associated Dynatrace integration.

Conditions

For this walkthrough, it’s best to have the next conditions:

Dynatrace API Token

Determine 1 – Dynatrace API Token

The way it works

Amazon Kinesis Data Firehose HTTP endpoint delivery

Determine 2 – Amazon Kinesis Knowledge Firehose HTTP endpoint supply

Merely create a log stream in your Amazon companies to ship your context wealthy logs to the Amazon CloudWatch Logs service. Subsequent, choose your Dynatrace HTTP endpoint to reinforce your logs streams with the facility of the Dynatrace Intelligence Platform. Lastly, it’s also possible to again up your logs to an Amazon Easy Storage Service (Amazon S3) bucket.

Setup directions

So as to add a service to monitoring, comply with these steps:

  1. Within the Dynatrace menu, go to Settings > Cloud and virtualization, and choose AWS.
  2. On the AWS overview web page, scroll down and choose the specified AWS occasion. Choose the Edit button.
  3. Scroll down and choose Add service. Select the service title from the drop-down, and choose Add service.
  4. Choose Save adjustments.

To course of and ship AWS CloudWatch Metrics to Dynatrace, comply with these steps.

  1. Log in to the AWS console and kind “Kinesis” within the textual content search bar. Choose Kinesis
AWS Console

Determine 3 – AWS Console

  1. On the Amazon Kinesis companies web page, choose the radio button for Kinesis Knowledge Firehose and choose the Create supply stream button.
Amazon Kinesis

Determine 4 – Amazon Kinesis

  1. Select the “Direct PUT” from the drop down, and from Vacation spot drop down, select “Dynatrace”.
Amazon Kinesis Data Firehose

Determine 5 – Amazon Kinesis Knowledge Firehose

  1. Supply stream title – Give your stream a brand new title, for instance: – “KFH-StreamToDynatrace”

Determine 6 – Supply stream title

  1. Within the part “Vacation spot settings”:
Destination settings

Determine 7 – Vacation spot settings

  1. HTTP endpoint title – “Dynatrace”.
  2. HTTP endpoint URL – From the drop down, choose “Dynatrace – US”.
  3. API token – Enter Dynatrace API TOKEN created within the prerequisite part.
  4. API URL – enter the Dynatrace URL in your tenant, for instance: https://xxxxx.reside.dynatrace.com
  5. Again Up Settings – Both choose an current S3 bucket or create a brand new one and add particulars and choose the Create supply stream button.
Backup settings

Determine 8 – Backup settings

As soon as profitable, your AWS Console will appear to be the next:

Amazon Kinesis Data Firehose

Determine 9 – Amazon Kinesis Knowledge Firehose

The Dynatrace Expertise

As soon as the preliminary setups are accomplished in each Dynatrace and the AWS Console, comply with these steps to visualise your new KHF stream knowledge within the Dynatrace console.

  1. Log in to the Dynatrace Console, and on the left aspect menu increase the “infrastructure” part, and choose “AWS
  2. From the display screen, choose the AWS account that you just need to add the KFH stream to.
  3. Subsequent, you’ll see a virtualization of your AWS property for the account chosen. Choose the field marked “Supporting Companies”.
  4. Subsequent, press the “Configure companies” button.
  5. Subsequent, choose “Add service”.
  6. From the drop down, choose “Kinesis Knowledge Firehose”.
  7. Subsequent, choose the “Add metric” button, and choose the metrics that you just need to see for this stream. Dynatrace has a complete record of metrics that may be chosen from the UI. The record will be discovered on this hyperlink.

Troubleshooting

  1. After configuration, load to the brand new KFH stream no knowledge within the Dynatrace Console.
    1. Test the Error Logs tab verify to be sure that the Vacation spot URL is appropriate for the Dynatrace Tenant.
Destination error logs

Determine 10 – Vacation spot error logs

  1. Invalid or misconfigured Dynatrace API token or scope isn’t correctly set.
Destination error logs

Determine 11 – Vacation spot error logs

Conclusion

On this publish, we display the Kinesis Knowledge Firehose and associated Dynatrace integration. As well as, engineers can use CloudWatch Metrics to discover their manufacturing programs alongside occasions in Dynatrace. This supplies a seamless, present view of your system (from logs to occasions and traces) in a single knowledge retailer.

To be taught extra about CloudWatch Service, see the Amazon CloudWatch residence web page. You probably have any questions, publish them on the AWS CloudWatch service discussion board.

For those who haven’t but signed up for Dynatrace, then you’ll be able to check out Kinesis Knowledge Firehose with Dynatrace with a free Dynatrace trial.


In regards to the Authors

Erick Leon is a Technical Alliances Sr. Supervisor at Dynatrace, Observability Follow Architect, and Buyer Advocate. He promotes robust technical integrations with a concentrate on AWS. With over 15 years as a Dynatrace buyer, his real-world experiences and classes realized carry precious insights into the Dynatrace Clever Observability Platform.

Shashiraj Jeripotula (Raj) is a San Francisco-based Sr. Accomplice Options Architect at AWS. He works with numerous unbiased software program distributors (ISVs), and companions who focus on cloud administration instruments and DevOps to develop joint options and speed up cloud adoption on AWS.

Leave a Reply