Hadoop的内存配置有两种方法:
1)利用手动安装hadoop的帮助脚本;
2)手动计算yarn和mapreduce内存大小进行配置。
此处只记录脚本计算方法:
1、用wget命令从hortonworks下载脚本
wget http://public-repo-1.hortonworks.com/HDP/tools/2.1.1.0/hdp_manual_install_rpm_helper_files-2.1.1.385.tar.gz
2、解压文件
tar -xvf hdp_manual_install_rpm_helper_files-2.1.1.385.tar.gz hdp_manual_install_rpm_helper_files-2.1.1.385
3、进入脚本所在目录 cd hdp_manual_install_rpm_helper_files-2.1.1.385/scripts
运行脚本:python hdp-configuration-utils.py,具体参数如下:
Option | Description |
---|---|
-c CORES | The number of cores on each host. |
-m MEMORY | The amount of memory on each host in GB. |
-d DISKS | Thenumber of disks on each host. |
-k HBASE | "True"if HBase is installed, “False” if not. |
其中:Core的数量可以通过nproc命令计算;内存大小可以通过free –m命令来计算;磁盘的数量可以同过lsblk –s或sudo fdisk –l命令来查看。
[root@server31 scripts]# nproc
48
[root@server31 scripts]# free -m
total used free shared buff/cache available
Mem: 128658 26607 739 37 101311 101165
Swap: 4095 611 3484
[root@server31 scripts]# lsblk -s
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdm1 8:193 0 1G 0 part /boot
└─sdm 8:192 0 228.8G 0 disk
cl-root 253:0 0 50G 0 lvm /
└─sdn1 8:209 0 50.1G 0 part
└─sdn 8:208 0 50.1G 0 disk
cl-swap 253:1 0 4G 0 lvm [SWAP]
└─sda1 8:1 0 1.8T 0 part
└─sda 8:0 0 1.8T 0 disk
cl-home 253:2 0 22.1T 0 lvm /home
├─sda1 8:1 0 1.8T 0 part
│ └─sda 8:0 0 1.8T 0 disk
├─sdb1 8:17 0 1.8T 0 part
│ └─sdb 8:16 0 1.8T 0 disk
├─sdc1 8:33 0 1.8T 0 part
│ └─sdc 8:32 0 1.8T 0 disk
├─sdd1 8:49 0 1.8T 0 part
│ └─sdd 8:48 0 1.8T 0 disk
├─sde1 8:65 0 1.8T 0 part
│ └─sde 8:64 0 1.8T 0 disk
├─sdf1 8:81 0 1.8T 0 part
│ └─sdf 8:80 0 1.8T 0 disk
├─sdg1 8:97 0 1.8T 0 part
│ └─sdg 8:96 0 1.8T 0 disk
├─sdh1 8:113 0 1.8T 0 part
│ └─sdh 8:112 0 1.8T 0 disk
├─sdi1 8:129 0 1.8T 0 part
│ └─sdi 8:128 0 1.8T 0 disk
├─sdj1 8:145 0 1.8T 0 part
│ └─sdj 8:144 0 1.8T 0 disk
├─sdk1 8:161 0 1.8T 0 part
│ └─sdk 8:160 0 1.8T 0 disk
├─sdl1 8:177 0 1.8T 0 part
│ └─sdl 8:176 0 1.8T 0 disk
├─sdm2 8:194 0 227.8G 0 part
│ └─sdm 8:192 0 228.8G 0 disk
└─sdn1 8:209 0 50.1G 0 part
└─sdn 8:208 0 50.1G 0 disk
故:python hdp-configuration-utils.py -c 48 -m 128 -d 16 -k True
返回结果如下:
[root@server31 scripts]# python hdp-configuration-utils.py -c 48 -m 128 -d 16 -k True
Using cores=48 memory=128GB disks=16 hbase=True
Profile: cores=48 memory=81920MB reserved=48GB usableMem=80GB disks=16
Num Container=29
Container Ram=2560MB
Used Ram=72GB
Unused Ram=48GB
yarn.scheduler.minimum-allocation-mb=2560
yarn.scheduler.maximum-allocation-mb=74240
yarn.nodemanager.resource.memory-mb=74240
mapreduce.map.memory.mb=2560
mapreduce.map.java.opts=-Xmx2048m
mapreduce.reduce.memory.mb=2560
mapreduce.reduce.java.opts=-Xmx2048m
yarn.app.mapreduce.am.resource.mb=2560
yarn.app.mapreduce.am.command-opts=-Xmx2048m
mapreduce.task.io.sort.mb=1024
tez.am.resource.memory.mb=2560
tez.am.java.opts=-Xmx2048m
hive.tez.container.size=2560
hive.tez.java.opts=-Xmx2048m
hive.auto.convert.join.noconditionaltask.size=671088000
本篇文章来源于 Linux公社网站(www.linuxidc.com) 原文链接:https://www.linuxidc.com/Linux/2014-09/106520.htm