在本文中，小李哥将介绍如何使用Amazon Bedrock上的亚马逊云科技自研模型Amazon Nova构建聊天应用WhatsApp的AI助手，以处理聊天软件对话中中图像、视频、文档和音频等多媒体消息内容，同时该方案也可以移植到微信应用中处理用户的消息。该无服务器架构解决方案使用亚马逊云科技End User Messaging服务进行直接集成。
背景
Amazon Bedrock现在可以通过Amazon Nova模型处理多种内容类型，帮助大家能够创建理解多模态媒体格式数据上下文的AI助手。本文将演示如何通过Agent代理构建一个WhatsApp AI助手，用于分析图像、处理视频、从文档中提取信息以及转录音频消息，并在与Amazon Bedrock代理的对话中始终保持对上下文的记忆。
在本文中，大家将学习如何将Amazon Bedrock与亚马逊云科技End User Messaging相结合，实现直接的WhatsApp集成，并构建一个无服务器架构解决方案，而不再需要额外的API中间件实现。聊天应用中的的数据将安全地存储在大家的亚马逊云科技账户中，不会被共享或用于模型训练。但云平台之外的WhatsApp为数据的安全性保障我们不是很了解，因此本实验中将不会共享私人信息。
在聊天软件中利用AI对话并分析图片/视频的场景展示
实验前提条件
- 亚马逊云科技账户
- 具备Python的基础知识
- 配置了用户秘钥和必要权限的亚马逊云科技CLI命令行工具。
- Python 3.8或更高版本。
- 亚马逊云科技亚马逊云科技Cloud Development Kit (CDK) v2.172.0或更高版本。
- 拥有或创建Meta Business账户
方案整体架构图
方案架构详解
这个项目使用亚马逊云科技亚马逊云科技Cloud Development Kit (CDK)来定义并部署以下资源:
- 亚马逊云科技Lambda（包括以下代码文件）:
- whatsapp_in: 处理传入的WhatsApp消息。
- transcriber_done: 处理已完成的转录作业。
- bedrock_agent: 调用Amazon Bedrock AI代理。
2. Amazon Simple Storage Service(Amazon S3):
- 用于存储媒体文件（语音、图像、视频、文档）的存储桶。
3. Amazon DynamoDB （NoSQL数据库）:
- messages:存储WhatsApp消息数据
- agenthistory:在处理图像、文档和视频时，为Amazon Bedrock Converse API存储对话历史
4. Amazon Simple Notification Service(SNS):
- 用于接收WhatsApp事件的主题。
5. 亚马逊云科技Identity and Access Management:
- Lambda函数和Bedrock Agent的角色和策略。
6. Amazon Bedrock:
- 用于处理消息的代理配置。
- 调用Converse API处理文档、图像和视频。
- 为代理版本控制提供Agent别名。
7. Amazon Transcribe:
- 用于转录音频消息。
- 亚马逊云科技End User Messaging
- 原生地连接WhatsApp Business账户(WABA)和亚马逊云科技账户。
这些基础设施的代码都在private_assistant_v2_stack.py文件中的PrivateAssistantV2Stack类中定义。
项目源代码
本项目方案和源代码全部在代码库 “build-on-aws/building-gen-ai-whatsapp-assistant-with-amazon-bedrock-and-python”中。
构建云基础设施的脚本代码如下：
from aws_cdk import (
# Duration,
Stack,
CfnOutput,
aws_iam as iam,
aws_s3 as s3,
RemovalPolicy,
aws_dynamodb as ddb,
aws_s3_notifications,
aws_s3_deployment as s3deploy # Add this import
# aws_sqs as sqs,
)
from constructs import Construct
from sns_topic import Topic
from lambdas import Lambdas
from databases import Tables
from agent_bedrock import CreateAgentSimple
import json
model_id = 'amazon.nova-pro-v1:0'
model_id_multimodal = "us.amazon.nova-pro-v1:0"
agent_name = 'DemoMultimodalAssistant'
# "foundation_model": "anthropic.claude-3-5-sonnet-20240620-v1:0"
file_path_agent_data = './private_assistant_v2/agent_data.json'
class PrivateAssistantV2Stack(Stack):
def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
stk = Stack.of(self)
region = stk.region
account_id = stk.account
with open(file_path_agent_data, 'r') as file:
agent_data = json.load(file)
agent_simple = CreateAgentSimple(self, "agentsimple", agent_name, model_id, agent_data["agent_instruction"], agent_data["description"])
agent_id = agent_simple.agent.attr_agent_id
agent_alias_id = agent_simple.agent_alias.attr_agent_alias_id
#https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_bedrock/CfnAgentAlias.html
Fn = Lambdas(self, "L")
# amazonq-ignore-next-line
Bucket = s3.Bucket(self, "S3", removal_policy=RemovalPolicy.DESTROY)
Bucket.grant_read_write(Fn.whatsapp_in)
Bucket.grant_read_write(Fn.bedrock_agent)
Bucket.grant_read_write(Fn.transcriber_done)
# Create empty folders (prefixes) in the bucket
s3deploy.BucketDeployment(self, "CreateFolders",
sources=[s3deploy.Source.data("voice/placeholder.txt", "")], # Creates voice_ folder
destination_bucket=Bucket,
retain_on_delete=False,
)
s3deploy.BucketDeployment(self, "CreateImageFolder",
sources=[s3deploy.Source.data("image/placeholder.txt", "")], # Creates image_ folder
destination_bucket=Bucket,
retain_on_delete=False,
)
s3deploy.BucketDeployment(self, "CreateTranscribeFolder",
sources=[s3deploy.Source.data("transcribe_response/placeholder.txt", "")], # Creates image_ folder
destination_bucket=Bucket,
retain_on_delete=False,
)
s3deploy.BucketDeployment(self, "CreateVideoFolder",
sources=[s3deploy.Source.data("video/placeholder.txt", "")], # Creates image_ folder
destination_bucket=Bucket,
retain_on_delete=False,
)
s3deploy.BucketDeployment(self, "CreateDocumentFolder",
sources=[s3deploy.Source.data("document/placeholder.txt", "")], # Creates image_ folder
destination_bucket=Bucket,
retain_on_delete=False,
)
Bucket.add_event_notification(s3.EventType.OBJECT_CREATED,
aws_s3_notifications.LambdaDestination(Fn.transcriber_done),
s3.NotificationKeyFilter(prefix="transcribe_response/"))
Tb = Tables(self, "Table")
Tb.messages.grant_read_write_data(Fn.whatsapp_in)
Tb.agenthistory.grant_read_write_data(Fn.bedrock_agent)
Tb.messages.grant_read_write_data(Fn.transcriber_done)
Tb.messages.add_global_secondary_index(index_name = 'jobnameindex',
partition_key = ddb.Attribute(name="jobName",type=ddb.AttributeType.STRING),
projection_type=ddb.ProjectionType.KEYS_ONLY)
Tp = Topic(self, "WhatsappEventsDestination", lambda_function=Fn.whatsapp_in)
Tp.topic.add_to_resource_policy(
iam.PolicyStatement(
effect=iam.Effect.ALLOW,
principals=[iam.ServicePrincipal("social-messaging.amazonaws.com")],
actions=["sns:Publish"],
resources=[Tp.topic.topic_arn]
)
)
# Grant permissions for WhatsApp messaging
Fn.whatsapp_in.add_to_role_policy(
iam.PolicyStatement(
actions=["social-messaging:SendWhatsAppMessage", "social-messaging:GetWhatsAppMessageMedia"],
resources=[f"arn:aws:social-messaging:{region}:{account_id}:phone-number-id/*"]
)
)
# Grant permissions for WhatsApp messaging
Fn.bedrock_agent.add_to_role_policy(
iam.PolicyStatement(
actions=["social-messaging:SendWhatsAppMessage", "social-messaging:GetWhatsAppMessageMedia"],
resources=[f"arn:aws:social-messaging:{region}:{account_id}:phone-number-id/*"]
)
)
# Grant permissions for Bedrock agent interaction
Fn.whatsapp_in.add_to_role_policy(
iam.PolicyStatement(
actions=[
'bedrock:InvokeAgent',
'bedrock:InvokeModel',
'bedrock-agent-runtime:InvokeAgent'
],
resources=['*']
)
)
# Add environment variables for the Bedrock agent
Fn.whatsapp_in.add_to_role_policy(
iam.PolicyStatement(
actions=["transcribe:*"],
resources=["*"]
)
)
Fn.whatsapp_in.add_environment("TABLE_NAME", Tb.messages.table_name)
Fn.whatsapp_in.add_environment("BUCKET_NAME", Bucket.bucket_name)
Fn.whatsapp_in.add_environment("VOICE_PREFIX", "voice/voice_")
Fn.whatsapp_in.add_environment("IMAGE_PREFIX", "image/image_")
Fn.whatsapp_in.add_environment("VIDEO_PREFIX", "video/video_")
Fn.whatsapp_in.add_environment("DOC_PREFIX", "document/document_")
Fn.whatsapp_in.add_environment("ENV_TRANSCRIBE_PREFIX", "transcribe_response")
Fn.bedrock_agent.grant_invoke(Fn.whatsapp_in)
Fn.whatsapp_in.add_environment( key='ENV_LAMBDA_BEDROCK_AGENT', value=Fn.bedrock_agent.function_name)
#Fn.whatsapp_in.add_environment( key='ENV_TRANSCRIBE_PREFIX', value=Fn.transcriber_done.function_name)
Fn.bedrock_agent.add_to_role_policy(iam.PolicyStatement( actions=["bedrock:*"], resources=['*']))
Fn.bedrock_agent.add_environment("ENV_AGENT_ID", agent_id)
Fn.bedrock_agent.add_environment("BUCKET_NAME", Bucket.bucket_name)
Fn.bedrock_agent.add_environment("ENV_ALIAS_ID", agent_alias_id)
Fn.bedrock_agent.add_environment("ENV_MODEL_ID", model_id_multimodal)
Fn.bedrock_agent.add_environment("TABLE_NAME", Tb.agenthistory.table_name)
Fn.bedrock_agent.add_environment(key='ENV_KEY_NAME', value="phone_number")
Fn.transcriber_done.add_to_role_policy(iam.PolicyStatement( actions=["dynamodb:*"], resources=[f"{Tb.messages.table_arn}",f"{Tb.messages.table_arn}/*"]))
Fn.transcriber_done.add_environment(key='ENV_INDEX_NAME', value="jobnameindex")
Fn.transcriber_done.add_environment(key='ENV_KEY_NAME', value="id")
Fn.transcriber_done.add_environment("TABLE_NAME", Tb.messages.table_name)
Fn.transcriber_done.add_environment( key='ENV_LAMBDA_BEDROCK_AGENT', value=Fn.bedrock_agent.function_name)
Fn.bedrock_agent.grant_invoke(Fn.transcriber_done)
CfnOutput(self, "TopicArn", value=Tp.topic.topic_arn)总结：
这篇文章展示了如何使用Amazon Bedrock和Amazon Nova模型构建WhatsApp AI助手，并支持移植到微信，从而处理图像、视频、文档和音频等多媒体内容，并通过亚马逊云科技End User Messaging实现直接集成。该无服务器架构方案能够安全地将大家的数据存储在大家的亚马逊云科技账户中，有效保护了数据的安全和隐私。希望本文对大家构建自己的多媒体处理WhatsApp AI助手有所帮助。