Bootstrap

aws(学习笔记第十四课) 面向NoSQL DB的DynamoDB

aws(学习笔记第十四课)

  • 面向NoSQL DB的DynamoDB

学习内容：

  • 开发一个任务TODO管理器

1. 主键，分区键和排序键

  1. DynamoDB的表定义和属性定义
    • 表定义（简单主键）

      • 表定义的命名需要系统名 + _ + 表名的形式，提前规划好前缀。因为DynamoDB直接建立表，没有建立数据库，之后在个别的数据库里面建立表的过程，所以注意提前规划。

      • 分区键 -简单的主键，由一个称为分区键 的属性组成。DynamoDB 使用分区键的值作为内部散列函数的输入。来自散列函数的输出决定了项目将存储到的分区 (DynamoDB 内部的物理存储)。在只有分区键的表中，任何两个项目都不能有相同的分区键值
        在这里插入图片描述

      • 表定义代码
        注意，建表语句区分大小写
        注意，这里建表的时候，建表的时候id作为主键，不参与主键的name和gender不能定义

        aws dynamodb create-table --table-name stu_system_student \
--attribute-definitions AttributeName=id,AttributeType=S \
--key-schema AttributeName=id,KeyType=HASH \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

        定义之后如下所示：

        $ aws dynamodb create-table --table-name stu_system_student \
--attribute-definitions AttributeName=id,AttributeType=S \
--key-schema AttributeName=id,KeyType=HASH \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5
{
    "TableDescription": {
        "AttributeDefinitions": [
            {
                "AttributeName": "id",
                "AttributeType": "S"
            }
        ],
        "TableName": "stu_system_student",
        "KeySchema": [
            {
                "AttributeName": "id",
                "KeyType": "HASH"
            }
        ],
        "TableStatus": "CREATING",
        "CreationDateTime": "2024-11-16T16:37:57.967000+08:00",
        "ProvisionedThroughput": {
            "NumberOfDecreasesToday": 0,
            "ReadCapacityUnits": 5,
            "WriteCapacityUnits": 5
        },
        "TableSizeBytes": 0,
        "ItemCount": 0,
        "TableArn": "arn:aws:dynamodb:ap-northeast-1:081353481087:table/stu_system_student",
        "TableId": "316d0a81-801b-451a-8fe1-57959f1906c6",
        "DeletionProtectionEnabled": false
    }
}

        执行aws cli查看表建立的状态

        $ aws dynamodb describe-table --table-name stu_system_student
{
    "Table": {
        "AttributeDefinitions": [
            {
                "AttributeName": "id",
                "AttributeType": "S"
            }
        ],
        "TableName": "stu_system_student",
        "KeySchema": [
            {
                "AttributeName": "id",
                "KeyType": "HASH"
            }
        ],
        "TableStatus": "ACTIVE",
        "CreationDateTime": "2024-11-16T16:37:57.967000+08:00",
        "ProvisionedThroughput": {
            "NumberOfDecreasesToday": 0,
            "ReadCapacityUnits": 5,
            "WriteCapacityUnits": 5
        },
        "TableSizeBytes": 0,
        "ItemCount": 0,
        "TableArn": "arn:aws:dynamodb:ap-northeast-1:081353481087:table/stu_system_student",
        "TableId": "316d0a81-801b-451a-8fe1-57959f1906c6",
        "DeletionProtectionEnabled": false
    }
}

        在这里插入图片描述

    • 表定义（复合主键）
      • 分区键和排序键 - 称为复合主键，此类型的键由两个属性组成。第一个属性是分区键，第二个属性是排序键。
      • DynamoDB 使用分区键值作为对内部散列函数的输入。来自散列函数的输出决定了项目将存储到的分区 (DynamoDB 内部的物理存储)。具有相同分区键的所有项目按排序键值的排序顺序存储在一起。
      • 在具有分区键和排序键的表中，两个项目可以具有相同的分区键值，但是，这两个项目必须具有不同的排序键值。
      • 定义复合主键表
        aws dynamodb create-table --table-name stu_system_score \
--attribute-definitions AttributeName=id,AttributeType=S \
AttributeName=lesson,AttributeType=S \
--key-schema AttributeName=id,KeyType=HASH \
AttributeName=lesson,KeyType=RANGE \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5
      • 复合建表结果
        $ aws dynamodb create-table --table-name stu_system_score \
--attribute-definitions AttributeName=id,AttributeType=S \
AttributeName=lesson,AttributeType=S \
--key-schema AttributeName=id,KeyType=HASH \
AttributeName=lesson,KeyType=RANGE \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5
{
    "TableDescription": {
        "AttributeDefinitions": [
            {
                "AttributeName": "id",
                "AttributeType": "S"
            },
            {
                "AttributeName": "lesson",
                "AttributeType": "S"
            }
        ],
        "TableName": "stu_system_score",
        "KeySchema": [
            {
                "AttributeName": "id",
                "KeyType": "HASH"
            },
            {
                "AttributeName": "lesson",
                "KeyType": "RANGE"
            }
        ],
        "TableStatus": "CREATING",
        "CreationDateTime": "2024-11-16T16:55:16.830000+08:00",
        "ProvisionedThroughput": {
            "NumberOfDecreasesToday": 0,
            "ReadCapacityUnits": 5,
            "WriteCapacityUnits": 5
        },
        "TableSizeBytes": 0,
        "ItemCount": 0,
        "TableArn": "arn:aws:dynamodb:ap-northeast-1:081353481087:table/stu_system_score",
        "TableId": "795a7ce6-8ecf-46c8-90f7-5c31e615886f",
        "DeletionProtectionEnabled": false
    }
}
        这里，AttributeName=id这个属性是分区键，不是主键。AttributeName=lesson这个属性是排序键，他们两个属性组成联合主键。在这里插入图片描述

2. 利用DynamoDB编写一个任务管理器的例子在这里插入图片描述

3. 创建表

  1. 创建两个表todo-user和todo-task
    • todo-user
      aws dynamodb create-table --table-name todo-user \
--attribute-definitions AttributeName=uid,AttributeType=S \
--key-schema AttributeName=uid,KeyType=HASH \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5
    • todo-task
      aws dynamodb create-table --table-name todo-task \
--attribute-definitions AttributeName=uid,AttributeType=S \
AttributeName=tid,AttributeType=N \
--key-schema AttributeName=uid,KeyType=HASH \
AttributeName=tid,KeyType=RANGE \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

4. docopt命令行代码框架以及node整体程序

  1. docopt的cli模板文件
    nodetodo

Usage:
  nodetodo user-add <uid> <email> <phone>
  nodetodo user-rm <uid>
  nodetodo user-ls [--limit=<limit>] [--next=<id>]
  nodetodo user <uid>
  nodetodo task-add <uid> <description> [<category>] [--dueat=<yyyymmdd>] 
  nodetodo task-rm <uid> <tid>
  nodetodo task-ls <uid> [<category>] [--overdue|--due|--withoutdue|--futuredue|--dueafter=<yyyymmdd>|--duebefore=<yyyymmdd>] [--limit=<limit>] [--next=<id>]
  nodetodo task-la <category> [--overdue|--due|--withoutdue|--futuredue|--dueafter=<yyyymmdd>|--duebefore=<yyyymmdd>] [--limit=<limit>] [--next=<id>]
  nodetodo task-done <uid> <tid>
  nodetodo -h | --help
  nodetodo --version

Options:
  -h --help        Show this screen.
  --version        Show version.
  --limit=<limit>  Maximum number of results [default: 10].
  2. node程序todo-task的整体结构
    • 整体代码
      var fs = require('fs');
var docopt = require('docopt');
var moment = require("moment");
var AWS = require('aws-sdk');
var db = new AWS.DynamoDB({
	"region": "ap-northeast-1"
});

var cli = fs.readFileSync('./cli.txt', {"encoding": "utf8"});
var input = docopt.docopt(cli, {"version": "1.0", "argv": process.argv.splice(2)});

function getValue(attribute, type) {
	if (attribute === undefined) {
		return null;
	}
	return attribute[type];
}

function mapTaskItem(item) {
	return {
		"tid": item.tid.N,
		"description": item.description.S,
		"created": item.created.N,
		"due": getValue(item.due, 'N'),
		"category": getValue(item.category, 'S'),
		"completed": getValue(item.completed, 'N')
	};
}

function mapUserItem(item) {
	return {
		"uid": item.uid.S,
		"email": item.email.S,
		"phone": item.phone.S
	};
}

if (input['user-add'] === true) {
	var params = {
		"Item": {
			"uid": {
				"S": input['<uid>']
			},
			"email": {
				"S": input['<email>']
			},
			"phone": {
				"S": input['<phone>']
			}
		},
		"TableName": "todo-user",
		"ConditionExpression": "attribute_not_exists(uid)"
	};
	db.putItem(params, function(err) {
		if (err) {
			console.error('error', err);
		} else {
			console.log('user added with uid ' + input['<uid>']);
		}
	});
} else if (input['user-rm'] === true) {
	var params = {
		"Key": {
			"uid": {
				"S": input['<uid>']
			}
		},
		"TableName": "todo-user"
	};
	db.deleteItem(params, function(err) {
		if (err) {
			console.error('error', err);
		} else {
			console.log('user removed with uid ' + input['<uid>']);
		}
	});
} else if (input['user-ls'] === true) {
	var params = {
		"TableName": "todo-user",
		"Limit": input['--limit']
	};
	if (input['--next'] !== null) {
		params.ExclusiveStartKey = {
			"uid": {
				"S": input['--next']
			}
		};
	}
	db.scan(params, function(err, data) {
		if (err) {
			console.error('error', err);
		} else {
			console.log('users', data.Items.map(mapUserItem));
			if (data.LastEvaluatedKey !== undefined) {
				console.log('more users available with --next=' + data.LastEvaluatedKey.uid.S);
			}
		}
	});
} else if (input['user'] === true) {
	var params = {
		"Key": {
			"uid": {
				"S": input['<uid>']
			}
		},
		"TableName": "todo-user"
	};
	db.getItem(params, function(err, data) {
		if (err) {
			console.error('error', err);
		} else {
			if (data.Item) {
				console.log('user with uid ' + input['<uid>'], mapUserItem(data.Item));
			} else {
				console.error('user with uid ' + input['<uid>'] + ' not found');
			}
		}
	});
} else if (input['task-add'] === true) {
	var tid = Date.now();
	var params = {
		"Item": {
			"uid": {
				"S": input['<uid>']
			},
			"tid": {
				"N": tid.toString()
			},
			"description": {
				"S": input['<description>']
			},
			"created": {
				"N": moment().format("YYYYMMDD")
			}
		},
		"TableName": "todo-task",
		"ConditionExpression": "attribute_not_exists(uid) and attribute_not_exists(tid)"
	};
	if (input['--dueat'] !== null) {
		params.Item.due = {
			"N": input['--dueat']
		};
	}
	if (input['<category>'] !== null) {
		params.Item.category = {
			"S": input['<category>']
		};
	}
	db.putItem(params, function(err) {
		if (err) {
			console.error('error', err);
		} else {
			console.log('task added with tid ' + tid);
		}
	});
} else if (input['task-rm'] === true) {
	var params = {
		"Key": {
			"uid": {
				"S": input['<uid>']
			},
			"tid": {
				"N": input['<tid>']
			}
		},
		"TableName": "todo-task"
	};
	db.deleteItem(params, function(err) {
		if (err) {
			console.error('error', err);
		} else {
			console.log('task removed with tid ' + input['<tid>']);
		}
	});
} else if (input['task-ls'] === true) {
	var params = {
		"KeyConditionExpression": "uid = :uid",
		"ExpressionAttributeValues": {
			":uid": {
				"S": input['<uid>']
			}
		},
		"TableName": "todo-task",
		"Limit": input['--limit']
	};
	if (input['--next'] !== null) {
		params.KeyConditionExpression += ' AND tid > :next';
		params.ExpressionAttributeValues[':next'] = {
			"N": input['--next']
		};
	}
	if (input['--overdue'] === true) {
		params.FilterExpression = "due < :yyyymmdd";
		params.ExpressionAttributeValues[':yyyymmdd'] = {"N": moment().format("YYYYMMDD")};
	} else if (input['--due'] === true) {
		params.FilterExpression = "due = :yyyymmdd";
		params.ExpressionAttributeValues[':yyyymmdd'] = {"N": moment().format("YYYYMMDD")};
	} else if (input['--withoutdue'] === true) {
		params.FilterExpression = "attribute_not_exists(due)";
	} else if (input['--futuredue'] === true) {
		params.FilterExpression = "due > :yyyymmdd";
		params.ExpressionAttributeValues[':yyyymmdd'] = {"N": moment().format("YYYYMMDD")};
	} else if (input['--dueafter'] !== null) {
		params.FilterExpression = "due > :yyyymmdd";
		params.ExpressionAttributeValues[':yyyymmdd'] = {"N": input['--dueafter']};
	} else if (input['--duebefore'] !== null) {
		params.FilterExpression = "due < :yyyymmdd";
		params.ExpressionAttributeValues[':yyyymmdd'] = {"N": input['--duebefore']};
	}
	if (input['<category>'] !== null) {
		if (params.FilterExpression === undefined) {
			params.FilterExpression = '';
		} else {
			params.FilterExpression += ' AND ';
		}
		params.FilterExpression += 'category = :category';
		params.ExpressionAttributeValues[':category'] = {
			"S": input['<category>']
		};
	}
	db.query(params, function(err, data) {
		if (err) {
			console.error('error', err);
		} else {
			console.log('tasks', data.Items.map(mapTaskItem));
			if (data.LastEvaluatedKey !== undefined) {
				console.log('more tasks available with --next=' + data.LastEvaluatedKey.tid.N);
			}
		}
	});
} else if (input['task-la'] === true) {
	var params = {
		"KeyConditionExpression": "category = :category",
		"ExpressionAttributeValues": {
			":category": {
				"S": input['<category>']
			}
		},
		"TableName": "todo-task",
		"IndexName": "category-index",
		"Limit": input['--limit']
	};
	if (input['--next'] !== null) {
		params.KeyConditionExpression += ' AND tid > :next';
		params.ExpressionAttributeValues[':next'] = {
			"N": input['--next']
		};
	}
	if (input['--overdue'] === true) {
		params.FilterExpression = "due < :yyyymmdd";
		params.ExpressionAttributeValues[':yyyymmdd'] = {"N": moment().format("YYYYMMDD")};
	} else if (input['--due'] === true) {
		params.FilterExpression = "due = :yyyymmdd";
		params.ExpressionAttributeValues[':yyyymmdd'] = {"N": moment().format("YYYYMMDD")};
	} else if (input['--withoutdue'] === true) {
		params.FilterExpression = "attribute_not_exists(due)";
	} else if (input['--futuredue'] === true) {
		params.FilterExpression = "due > :yyyymmdd";
		params.ExpressionAttributeValues[':yyyymmdd'] = {"N": moment().format("YYYYMMDD")};
	} else if (input['--dueafter'] !== null) {
		params.FilterExpression = "due > :yyyymmdd";
		params.ExpressionAttributeValues[':yyyymmdd'] = {"N": input['--dueafter']};
	} else if (input['--duebefore'] !== null) {
		params.FilterExpression = "due < :yyyymmdd";
		params.ExpressionAttributeValues[':yyyymmdd'] = {"N": input['--duebefore']};
	}
	db.query(params, function(err, data) {
		if (err) {
			console.error('error', err);
		} else {
			console.log('tasks', data.Items.map(mapTaskItem));
			if (data.LastEvaluatedKey !== undefined) {
				console.log('more tasks available with --next=' + data.LastEvaluatedKey.tid.N);
			}
		}
	});
} else if (input['task-done'] === true) {
	var params = {
		"Key": {
			"uid": {
				"S": input['<uid>']
			},
			"tid": {
				"N": input['<tid>']
			}
		},
		"UpdateExpression": "SET completed = :yyyymmdd",
		"ExpressionAttributeValues": {
			":yyyymmdd": {
				"N": moment().format("YYYYMMDD")
			}
		},
		"TableName": "todo-task"
	};
	db.updateItem(params, function(err) {
		if (err) {
			console.error('error', err);
		} else {
			console.log('task completed with tid ' + input['<tid>']);
		}
	});
}

5. 进行操作以及代码解析

  1. 增加用户
    • 命令执行
      node index.js user-add '001' '[email protected]' '1234567'
    • 实现代码
      	var params = {
		"Item": {
			"uid": {
				"S": input['<uid>']
			},
			"email": {
				"S": input['<email>']
			},
			"phone": {
				"S": input['<phone>']
			}
		},
		"TableName": "todo-user",
		"ConditionExpression": "attribute_not_exists(uid)"
	};
  2. 查询用户
    • 执行命令
       node index.js user-ls
    • 实现代码
      } else if (input['user-ls'] === true) {
	var params = {
		"TableName": "todo-user",
		"Limit": input['--limit']
	};
	if (input['--next'] !== null) {
		params.ExclusiveStartKey = {
			"uid": {
				"S": input['--next']
			}
		};
	}
	db.scan(params, function(err, data) {
		if (err) {
			console.error('error', err);
		} else {
			console.log('users', data.Items.map(mapUserItem));
			if (data.LastEvaluatedKey !== undefined) {
				console.log('more users available with --next=' + data.LastEvaluatedKey.uid.S);
			}
		}
	});

6. 查询用户表数据

  1. 执行查询用户命令
    • 查询用户
      $ node index.js user-ls
users [ { uid: '001', email: '[email protected]', phone: '13512345678' } ]
    • 实现代码
      } else if (input['user-ls'] === true) {
	var params = {
		"TableName": "todo-user",
		"Limit": input['--limit']
	};
	if (input['--next'] !== null) {
		params.ExclusiveStartKey = {
			"uid": {
				"S": input['--next']
			}
		};
	}
	db.scan(params, function(err, data) {
		if (err) {
			console.error('error', err);
		} else {
			console.log('users', data.Items.map(mapUserItem));
			if (data.LastEvaluatedKey !== undefined) {
				console.log('more users available with --next=' + data.LastEvaluatedKey.uid.S);
			}
		}
	});

7. 追加task表数据

  1. 执行追加命令
    • 执行追加task命令
      node index.js task-add 001 "this is a emergency job" --dueat=20241201
    • 实现代码
      } else if (input['task-add'] === true) {
	var tid = Date.now();
	var params = {
		"Item": {
			"uid": {
				"S": input['<uid>']
			},
			"tid": {
				"N": tid.toString()
			},
			"description": {
				"S": input['<description>']
			},
			"created": {
				"N": moment().format("YYYYMMDD")
			}
		},
		"TableName": "todo-task",
		"ConditionExpression": "attribute_not_exists(uid) and attribute_not_exists(tid)"
	};
	if (input['--dueat'] !== null) {
		params.Item.due = {
			"N": input['--dueat']
		};
	}
	if (input['<category>'] !== null) {
		params.Item.category = {
			"S": input['<category>']
		};
	}
	db.putItem(params, function(err) {
		if (err) {
			console.error('error', err);
		} else {
			console.log('task added with tid ' + tid);
		}
	});

8. 查询task表数据

  1. 执行task查询命令
    • 执行查询task命令
      node index.js task-ls 001
    • 代码实现
      } else if (input['task-ls'] === true) {
	var params = {
		"KeyConditionExpression": "uid = :uid",
		"ExpressionAttributeValues": {
			":uid": {
				"S": input['<uid>']
			}
		},
		"TableName": "todo-task",
		"Limit": input['--limit']
	};
	if (input['--next'] !== null) {
		params.KeyConditionExpression += ' AND tid > :next';
		params.ExpressionAttributeValues[':next'] = {
			"N": input['--next']
		};
	}
	if (input['--overdue'] === true) {
		params.FilterExpression = "due < :yyyymmdd";
		params.ExpressionAttributeValues[':yyyymmdd'] = {"N": moment().format("YYYYMMDD")};
	} else if (input['--due'] === true) {
		params.FilterExpression = "due = :yyyymmdd";
		params.ExpressionAttributeValues[':yyyymmdd'] = {"N": moment().format("YYYYMMDD")};
	} else if (input['--withoutdue'] === true) {
		params.FilterExpression = "attribute_not_exists(due)";
	} else if (input['--futuredue'] === true) {
		params.FilterExpression = "due > :yyyymmdd";
		params.ExpressionAttributeValues[':yyyymmdd'] = {"N": moment().format("YYYYMMDD")};
	} else if (input['--dueafter'] !== null) {
		params.FilterExpression = "due > :yyyymmdd";
		params.ExpressionAttributeValues[':yyyymmdd'] = {"N": input['--dueafter']};
	} else if (input['--duebefore'] !== null) {
		params.FilterExpression = "due < :yyyymmdd";
		params.ExpressionAttributeValues[':yyyymmdd'] = {"N": input['--duebefore']};
	}
	if (input['<category>'] !== null) {
		if (params.FilterExpression === undefined) {
			params.FilterExpression = '';
		} else {
			params.FilterExpression += ' AND ';
		}
		params.FilterExpression += 'category = :category';
		params.ExpressionAttributeValues[':category'] = {
			"S": input['<category>']
		};
	}
	db.query(params, function(err, data) {
		if (err) {
			console.error('error', err);
		} else {
			console.log('tasks', data.Items.map(mapTaskItem));
			if (data.LastEvaluatedKey !== undefined) {
				console.log('more tasks available with --next=' + data.LastEvaluatedKey.tid.N);
			}
		}
	});

9. 查询表数据的三种方式

  1. 提供键来获取数据

    • 指定主键的属性
      在这里插入图片描述

    • 使用分区键加上过滤条件查询
      在这里插入图片描述

  2. 二级索引查询
    • 二级索引的原理
      下面的例子category作为索引的例子，通过再做出一个索引表，DynamoDB增加一个原表的另一个只读索引表。
      在这里插入图片描述
    • 作成二级索引
      注意，json如果在命令行里面，命令行对于json这里就不能使用\换行符！
      aws dynamodb update-table --table-name todo-task \
--attribute-definitions AttributeName=uid,AttributeType=S \
AttributeName=tid,AttributeType=N \
AttributeName=category,AttributeType=S \
--global-secondary-index-updates '[{"Create": {"IndexName": "category-index","KeySchema":[{"AttributeName":"category","KeyType":"HASH"},{"AttributeName": "tid", "KeyType": "RANGE"}],"Projection": {"ProjectionType": "ALL"},"ProvisionedThroughput":{"ReadCapacityUnits":5,"WriteCapacityUnits":5}}}]'

在这里插入图片描述

  1. 扫描查询
    如果实在没有主键，分区键，排序键进行查找，那么选择full table scan也是没有办法的事情。
    在这里插入图片描述

悦读

道可道，非常道；名可名，非常名。 无名，天地之始，有名，万物之母。 故常无欲，以观其妙，常有欲，以观其徼。 此两者，同出而异名，同谓之玄，玄之又玄，众妙之门。

最新收录
汽车大灯尾灯划痕裂缝破洞破损掉角崩角等如何修复？根本没必要换车灯换总成，使用无痕修UV树脂胶液即可轻松搞定。
使用SQL Server 2017 Docker容器在.NET Core中进行本地Web API开发
微服务-istio
好奇怪的游戏（BFS）
巧用for循环绘制各种图形
【Java Web】PostMan业务接口测试工具
C++ 教程（最新、最全、3.5万字总结），强烈建议收藏！！！
【qt】 TCP编程小项目
随机生成1000个1~100之内的整数，统计出不重复的数值和其重复的次数
c++ error: ‘*’ does not name a type； did you mean ‘**’?
;