Github Repo: https://github.com/pedropcamellon/medical-calls-analysis-aws

Introduction

In the previuos article of this series, we created a simple yet extensible event-driven architecture using S3 triggers, AWS Lambda Amazon Transcribe and Amazon Bedrock. As applications scale up, manually tracking metrics and monitoring system performance becomes increasingly challenging and time-consuming. With numerous API calls, data processing tasks, and model invocations happening simultaneously, it's crucial to implement automated monitoring solutions. This is where a robust logging and monitoring framework becomes essential.

AWS services offer comprehensive monitoring through seamless integration with CloudWatch. Amazon Bedrock's monitoring features work alongside Lambda function logs and S3 trigger tracking to provide a complete monitoring solution. Through CloudWatch's centralized platform, you can track usage metrics, create custom dashboards, set up real-time alerts, and monitor AI model interactions. This integration enables detailed logging, helps maintain data privacy and compliance, and provides valuable audit trails for debugging in serverless environments. By leveraging these capabilities, you can maintain a unified view of your system's performance and ensure your AI-powered application operates reliably at scale.

Additionally, CloudWatch enables you to create alarms based on specific metric value thresholds or anomalous metric behavior detected through machine learning algorithms. These alarms can be configured to notify you if triggered, enabling swift action to mitigate any potential issues or disruptions. Automated actions can also be set up, such as temporarily blocking model access when usage thresholds are exceeded, helping prevent potential abuse while maintaining application security and cost controls.

In this article, we'll set up monitoring for our Lambda function that summarizes medical call transcripts. We'll configure the necessary permissions to enable model invocation logging in CloudWatch and create an alarm to track the number of model invocations, helping us maintain visibility into our application's usage patterns.

Configuring model invocation logging

To configure logging, navigate to the Settings page in the Bedrock console, from the left navigation bar. Then toggle the Model invocation logging button which will present you with several fields that will need to be filled out before logging can be enabled. First, select the data types to include with logs. You can choose text, image and embedding.

Now you'll need to choose where to send your logs. You have three options:

S3 Only: Logs are only sent to an S3 storage bucket that you choose.
CloudWatch Logs only: Logs are sent to CloudWatch. If any log data is very large (over 100kb) or contains files like images, you can choose to have it sent to S3 instead.
Both S3 & CloudWatch Logs: Logs are sent to both services. Like option 2, very large files or binary data will only go to S3.

No matter which option you pick, you stay in control of your data. You can encrypt it using KMS and decide how long to keep the logs. I decided to use CloudWatch Logs only for my setup. If you don't have a log group already, you'll need to create one in CloudWatch and specify its name here.

Next, select the Create and use a new role option and provide a name for your role. I chose BedrockCloudWatchLogs. Then, in the S3 bucket for large data delivery field, select the same bucket we created earlier for storing medical call audio files. Click Save Settings to complete the configuration.

Inspecting Log Data

Now that we have setup logging in Bedrock, triggering the summarize Lambda function will create log streams where you can observe both the model invocation details and Lambda function output. This combination of logs provides valuable debugging information about how your application is processing requests and interacting with the model.