aws(学习笔记第二十五课)
- 使用
aws batch
学习内容：
aws batch整体架构aws batch的hands on
1. aws batch整体架构
AWS batch的整体架构
相对online开发而言，batch也是一个另外的一个开发领域。AWS的batch开发，既可以采用EC2，也可以采用Fargate。- 用户通过各种方式将
job放入到job queue AWS batch将job queue里面的job取出- 之后将
job通过Fargate或者EC2进行执行
接下来，hands on采用Fargate的方式进行练习。
- 用户通过各种方式将
AWS batch的创建步骤- 创建
compute environment - 创建
job queue，这时候会绑定建立好的compute environment - 创建
job definition - 最后创建
job
- 创建
2.aws batch的hands on
- 首先定义计算环境
compute environment-
定义计算环境
-
使用
Fargate和默认的role- 这里的
role，使用默认的，之后查看AWS设定该role的权限和信任关系。 - 查看
AWSServiceRoleForBatch
- 权限设定
可以看出都是EC2和ECS以及autoscaling相关的权限。{ "Version": "2012-10-17", "Statement": [ { "Sid": "AWSBatchPolicyStatement1", "Effect": "Allow", "Action": [ "ec2:DescribeAccountAttributes", "ec2:DescribeInstances", "ec2:DescribeInstanceStatus", "ec2:DescribeInstanceAttribute", "ec2:DescribeSubnets", "ec2:DescribeSecurityGroups", "ec2:DescribeKeyPairs", "ec2:DescribeImages", "ec2:DescribeImageAttribute", "ec2:DescribeSpotInstanceRequests", "ec2:DescribeSpotFleetInstances", "ec2:DescribeSpotFleetRequests", "ec2:DescribeSpotPriceHistory", "ec2:DescribeSpotFleetRequestHistory", "ec2:DescribeVpcClassicLink", "ec2:DescribeLaunchTemplateVersions", "ec2:RequestSpotFleet", "autoscaling:DescribeAccountLimits", "autoscaling:DescribeAutoScalingGroups", "autoscaling:DescribeLaunchConfigurations", "autoscaling:DescribeAutoScalingInstances", "autoscaling:DescribeScalingActivities", "eks:DescribeCluster", "ecs:DescribeClusters", "ecs:DescribeContainerInstances", "ecs:DescribeTaskDefinition", "ecs:DescribeTasks", "ecs:ListClusters", "ecs:ListContainerInstances", "ecs:ListTaskDefinitionFamilies", "ecs:ListTaskDefinitions", "ecs:ListTasks", "ecs:DeregisterTaskDefinition", "ecs:TagResource", "ecs:ListAccountSettings", "logs:DescribeLogGroups", "iam:GetInstanceProfile", "iam:GetRole" ], "Resource": "*" }, { "Sid": "AWSBatchPolicyStatement2", "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream" ], "Resource": "arn:aws:logs:*:*:log-group:/aws/batch/job*" }, { "Sid": "AWSBatchPolicyStatement3", "Effect": "Allow", "Action": [ "logs:PutLogEvents" ], "Resource": "arn:aws:logs:*:*:log-group:/aws/batch/job*:log-stream:*" }, { "Sid": "AWSBatchPolicyStatement4", "Effect": "Allow", "Action": [ "autoscaling:CreateOrUpdateTags" ], "Resource": "*", "Condition": { "Null": { "aws:RequestTag/AWSBatchServiceTag": "false" } } }, { "Sid": "AWSBatchPolicyStatement5", "Effect": "Allow", "Action": "iam:PassRole", "Resource": [ "*" ], "Condition": { "StringEquals": { "iam:PassedToService": [ "ec2.amazonaws.com", "ec2.amazonaws.com.cn", "ecs-tasks.amazonaws.com" ] } } }, { "Sid": "AWSBatchPolicyStatement6", "Effect": "Allow", "Action": "iam:CreateServiceLinkedRole", "Resource": "*", "Condition": { "StringEquals": { "iam:AWSServiceName": [ "spot.amazonaws.com", "spotfleet.amazonaws.com", "autoscaling.amazonaws.com", "ecs.amazonaws.com" ] } } }, { "Sid": "AWSBatchPolicyStatement7", "Effect": "Allow", "Action": [ "ec2:CreateLaunchTemplate" ], "Resource": "*", "Condition": { "Null": { "aws:RequestTag/AWSBatchServiceTag": "false" } } }, { "Sid": "AWSBatchPolicyStatement8", "Effect": "Allow", "Action": [ "ec2:TerminateInstances", "ec2:CancelSpotFleetRequests", "ec2:ModifySpotFleetRequest", "ec2:DeleteLaunchTemplate" ], "Resource": "*", "Condition": { "Null": { "aws:ResourceTag/AWSBatchServiceTag": "false" } } }, { "Sid": "AWSBatchPolicyStatement9", "Effect": "Allow", "Action": [ "autoscaling:CreateLaunchConfiguration", "autoscaling:DeleteLaunchConfiguration" ], "Resource": "arn:aws:autoscaling:*:*:launchConfiguration:*:launchConfigurationName/AWSBatch*" }, { "Sid": "AWSBatchPolicyStatement10", "Effect": "Allow", "Action": [ "autoscaling:CreateAutoScalingGroup", "autoscaling:UpdateAutoScalingGroup", "autoscaling:SetDesiredCapacity", "autoscaling:DeleteAutoScalingGroup", "autoscaling:SuspendProcesses", "autoscaling:PutNotificationConfiguration", "autoscaling:TerminateInstanceInAutoScalingGroup" ], "Resource": "arn:aws:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/AWSBatch*" }, { "Sid": "AWSBatchPolicyStatement11", "Effect": "Allow", "Action": [ "ecs:DeleteCluster", "ecs:DeregisterContainerInstance", "ecs:RunTask", "ecs:StartTask", "ecs:StopTask" ], "Resource": "arn:aws:ecs:*:*:cluster/AWSBatch*" }, { "Sid": "AWSBatchPolicyStatement12", "Effect": "Allow", "Action": [ "ecs:RunTask", "ecs:StartTask", "ecs:StopTask" ], "Resource": "arn:aws:ecs:*:*:task-definition/*" }, { "Sid": "AWSBatchPolicyStatement13", "Effect": "Allow", "Action": [ "ecs:StopTask" ], "Resource": "arn:aws:ecs:*:*:task/*/*" }, { "Sid": "AWSBatchPolicyStatement14", "Effect": "Allow", "Action": [ "ecs:CreateCluster", "ecs:RegisterTaskDefinition" ], "Resource": "*", "Condition": { "Null": { "aws:RequestTag/AWSBatchServiceTag": "false" } } }, { "Sid": "AWSBatchPolicyStatement15", "Effect": "Allow", "Action": "ec2:RunInstances", "Resource": [ "arn:aws:ec2:*::image/*", "arn:aws:ec2:*::snapshot/*", "arn:aws:ec2:*:*:subnet/*", "arn:aws:ec2:*:*:network-interface/*", "arn:aws:ec2:*:*:security-group/*", "arn:aws:ec2:*:*:volume/*", "arn:aws:ec2:*:*:key-pair/*", "arn:aws:ec2:*:*:launch-template/*", "arn:aws:ec2:*:*:placement-group/*", "arn:aws:ec2:*:*:capacity-reservation/*", "arn:aws:ec2:*:*:elastic-gpu/*", "arn:aws:elastic-inference:*:*:elastic-inference-accelerator/*", "arn:aws:resource-groups:*:*:group/*" ] }, { "Sid": "AWSBatchPolicyStatement16", "Effect": "Allow", "Action": "ec2:RunInstances", "Resource": "arn:aws:ec2:*:*:instance/*", "Condition": { "Null": { "aws:RequestTag/AWSBatchServiceTag": "false" } } }, { "Sid": "AWSBatchPolicyStatement17", "Effect": "Allow", "Action": [ "ec2:CreateTags" ], "Resource": [ "*" ], "Condition": { "StringEquals": { "ec2:CreateAction": [ "RunInstances", "CreateLaunchTemplate", "RequestSpotFleet" ] } } } ] } - 信任关系设定
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "batch.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }
- 这里的
-
设定
VPC等网络环境- 设定公网
Public VPC
这里为了方便练习，设定了Public VPC，实际的生产环境中，为了安全考虑，应该是在Private VPC中，但是需要设定NAT。
- 设定公网
-
创建好的
compute environment的执行环境
等待一会，状态就会变成成功。 -
定义
job queue
定义job queue很简单，这里需要绑定compute environment。
-
定义
job（job definition）
Fargate的job定义的思想就是docker image+entrance command的设定。job名字的设定
- 为执行
ECS Task设定role- 设定
Service
- 设定权限
这里选择AWS托管的AmazonECSTaskExecutionRolePolicy。
- 为
job definition设定role
- 为
job definition定义docker image和entrance command
- 设定
- 为执行
-
执行
job（job execution）- 提交新作业
- 关联
job queue和job definition
- 可以将
enterance command覆盖job definition的enterance command - 提交新作业
- 提交新作业
-
执行结果
- 执行成功
- 查看
log
可以看出命令行的覆盖已经成功，job上定义的hello,world from job已经打印出来，job definition里面定义的hello,world from job definition没有生效。
这里，为了测试，使用的是echo命令，但是真实的生产环境应该是镜像的入口执行command
- 执行成功
-
