最近业务中涉及到远程服务器的日志收集需求, 出于限制技术栈扩大的想法,使用PHP进行了实现.
实现过程中有些小小需要注意的点,记录如下:
1. 主动获取. 由于服务器较多, 如果使用Flume之类的架构, 需要在每台服务器上安装软件, 这就产生了运维成本 . 所以我们使用 收集端主动获取的方式. 不需要在生产者(服务端)安装软件.
2.SSH连接. 每台服务器都配置了SSH连接权限,使用PHP的 ssh2扩展即可远程连接并访问服务器内容.
3.服务器日志结构统一. 每台服务器上的日志文件都按同一目录 规则放置,以简化程序逻辑.
4.CLI运行. 收集是持续运行的程序,使用CLI模式,要注意,此时所使用的INI文件问题.
5.SSH连接异常. 有时,由于网络问题,导致SSH连接或验证失败, 延时重试即可.
6.日志截断与压缩. 通常,我们的运维会在每天的固定时间对日志进行截断和压缩, 这就有了两种类型的文件需要读取:压缩与未压缩的日志, 需要分别处理.
7.日志中的时间戳. 以秒为单位 的时间戳不足以区分请求, 我们增加$msec以毫秒计量, 同一毫秒内,同一IP来源,同一UA的可以认为是一个请求.
8.读取目录. 使用readdir即可读取SSH格式的远程目录, readdir("ssh2.sft://......"); 过滤掉不需要的文件后, 按文件创建时间排序,逐个处理.
9.读取压缩文件. 如果用file_get_contents会导致界面长期无响应, 我使用了fopen, fread 分步读取. 一次读取8K(再大也没有用了). 每读取一定次数后,输出一个进度显示.
10.压缩文件缓存. 读取成功后, 保存到缓存目录 , 以便备份以及下次使用. 如果程序出错或重新运行时, 先检查缓存目录, 如果有缓存文件,就不用从网络上读取了.
11.解压缩. 使用gzdecode即可. 这会导致PHP内存需要暴增, 调整PHP.INI吧, 把内存限制扩大.
12.压缩日志处理完成记录. 处理完成一个压缩文件后, 在数据库中记录下来, 以后PHP程序运行后,就不用重复处理了.
13.未压缩日志处理. 未压缩的日志表明,此日志仍在增长中. 不需要缓存. 使用数据库记录,当前文件指针(使用ftell,fseek). 记录文件创建日期.
14.未压缩日志判断. 当文件日期与记录的日期不同时, 或文件小于记录中的文件大小, 说明 此文件被更新了, 需要重置文件指针.
否则可以直接定位(fseek),以继续从上次处理的位置进行.
15.日志行分解. 使用正则即可,根据空格及定界符进行区分. 也可使用logParser第三方类库来处理. 为节省内存开销.可使用Iterator 协程模式, 逐行返回.
16.日志判重. 事先读取每个服务器的最后 日志时间戳(毫秒)以及IP,UA.
17.日志保存. 我是使用了MYSQL来保存日志. 每一行日志执行一次MYSQL会极大浪费运行时间, 可以累积4000行再一次性插入.
18.错误处理. 除了SSH连接失败外, 还会读取半行日志,导致分解失败, 此时也抛出异常. 由主程序捕获,并重新运行即可.
源程序如下:
<?php
/**
* Created by IcePHP Framework.
* User: 蓝冰大侠
* Date: 2018/4/11
* Time: 15:09
*/
class MLogImport
{
/**
* 当前正在处理的站点名称
* @var string
*/
private $feed;
/**
* 当前正在处理的站点主机地址
* @var string
*/
private $host;
/**
* 当前站点的登录名/站点名称
* @var string
*/
private $user;
/**
* 当前站点的登录密码
* @var string
*/
private $pass;
/**
* 当前站点的日志所在目录
* @var string
*/
private $logPath;
//用于本地保存服务器日志的目录,仅备份压缩后的日志
const CACHE_PATH = DIR_ROOT . 'run/serverLog/';
/**
* 当前站点的SSH连接
* @var resource
*/
private $sftp;
/**
* 当前正在处理的文件名
* @var string
*/
private $file;
/**
* 本站最后一条日志
* @var array
*/
private $lastRow;
private $agentPatterns;
private $agentReplaces;
public function __construct()
{
$maps = [
'/AppleWebKit\/[\d\.]*/i' => 'AppleWebKit/...',
'/Mobile\/[\d\w]*/i' => 'Mobile/...',
'/Safari\/[\d\.]*/i' => 'Safari/...',
'/CriOS\/[\d\.]*/i' => 'CriOS/...',
'/GSA\/[\d\.]*/i' => 'GSA/...',
'/Version\/[\d\.]*/i' => 'Version/...',
'/Chrome\/[\d\.]*/i' => 'Chrome/...',
'/Edge\/[\d\.]*/i' => 'Edge/...',
'/Firefox\/[\d\.]*/i' => 'Firefox/...',
'/SamsungBrowser\/[\d\.]*/i' => 'SamsungBrowser/...',
'/build\/[\d\w\-\.]*/i' => 'build/...',
'/Silk\/[\d\.]*/i' => 'Silk/...',
'/Crosswalk\/[\d\.]*/i' => 'Crosswalk/...',
'/Gecko\/[\d\.]*/i' => 'Gecko/...',
'/NTENTBrowser\/[\.\d]*/i' => 'NTENTBrowser/...',
'/Snapchat\/[\d\w\-\.]*/i' => 'Snapchat/...',
'/Java\/[\d\.]*/i' => 'Java/...',
'/UCBrowser\/[\d\.]*/i' => 'UCBrowser',
'/\(Linux[^\)]*SAMSUNG[^\)]*\)/i' => 'SAMSUNG...',
'/\([^\)]*IPAD[^\)]*\)/i' => 'IPAD...',
'/\([^\)]*SM-[^\)]*\)/i' => 'SM...',
'/LG\-[\w\d]*/i' => 'LG...',
'/LGL\d[\w\d]*/i' => 'LGL...',
'/itel it\d*/i' => 'itel...',
'/XT\d*/i' => 'XT...',
'/TECNO\-[\w\d]*/i' => 'TECNO...',
'/RCT[\d\w]*/i' => 'RCT...',
'/Micromax\s[\w\d]*/i' => 'Micromax...',
'/LGMS[\d]*/i' => 'LGMS...',
'/GT\-[\w\d]*/i' => 'GT...',
'/HUAWEI\s[\-\w\d]*/i' => 'HUAWEI...',
'/Lenovo\s[\-\w\d]*/i' => 'Lenovo...',
'/SCH\-[\w\d]*/i' => 'SCH...',
'/rv\:[\d\.]*/i' => 'rv:...',
'/Lumia\s\d+/i' => 'Lumia...',
'/Instagram\s[\d\.]*/i' => 'Instagram...',
'/iPhone OS 5[_\d]*/i' => 'iOS 5...',
'/iPhone OS 6[_\d]*/i' => 'iOS 6...',
'/iPhone OS 7[_\d]*/i' => 'iOS 7...',
'/iPhone OS 8[_\d]*/i' => 'iOS 8...',
'/iPhone OS 9[_\d]*/i' => 'iOS 9...',
'/iPhone OS 10[_\d]*/i' => 'iOS 10...',
'/iPhone OS 11[_\d]*/i' => 'iOS 11...',
'/iOS 5[_\d]*/i' => 'iOS 5...',
'/iOS 6[_\d]*/i' => 'iOS 6...',
'/iOS 7[_\d]*/i' => 'iOS 7...',
'/iOS 8[_\d]*/i' => 'iOS 8...',
'/iOS 9[_\d]*/i' => 'iOS 9...',
'/iOS 10[_\d]*/i' => 'iOS 10...',
'/iOS 11[_\d]*/i' => 'iOS 11...',
'/Android 2[\.\d]*/i' => 'Android 2...',
'/Android 3[\.\d]*/i' => 'Android 3...',
'/Android 4[\.\d]*/i' => 'Android 4...',
'/Android 5[\.\d]*/i' => 'Android 5...',
'/Android 6[\.\d]*/i' => 'Android 6...',
'/Android 7[\.\d]*/i' => 'Android 7...',
'/Android 8[\.\d]*/i' => 'Android 8...',
'/QuantcastSDK[^\s]*(\s\(\d+\))?/i' => 'QuantcastSDK...',
];
$this->agentPatterns = array_keys($maps);
$this->agentReplaces = array_values($maps);
}
/**
* 记录一个站点的账号,密码,日志路径
* @param string $host 主机/账号
* @param string $user 站点名称/登录名
* @param string $pass 登录密码
* @param string $logPath 日志文件路径
*/
public function site(string $host, string $user, string $pass, string $logPath): void
{
$this->feed = $user;
$this->host = $host;
$this->user = $user;
$this->pass = $pass;
$this->logPath = $logPath;
//重新连接的间隔时间
$interval = 1;
$connect = null;
while (true) {
//连接主机
$connect = ssh2_connect($this->host, '22');
//账号密码验证成功
if (false !== ssh2_auth_password($connect, $this->user, $this->pass)) {
break;
}
//间隔时间2秒,4,8,...
$interval *= 2;
echo "auth wrong at $this->host, retry after $interval seconds\r\n";
//间隔指定 时间后,重新连接
sleep($interval);
}
//登录成功
echo "\r\nlogin $this->feed\r\n";
//读取文件列表
$this->sftp = ssh2_sftp($connect);
if(!$this->sftp){
throw new Exception('ssh2_sftp fail.');
}
$handle = opendir("ssh2.sftp://{$this->sftp}{$this->logPath}"); //ssh2.sftp://Resource #33/home/.....
if (!$handle) {
throw new Exception('open dir ssh2.sftp fail.');
}
$zippedFiles = [];
$unzippedFile = '';
while (false !== ($file = readdir($handle))) {
$filePath = "ssh2.sftp://{$this->sftp}{$this->logPath}/$file";
//必须是文件,目录的不要
if (!is_file($filePath)) continue;
//必须是访问日志
if (left($file, 10) !== 'access.log') continue;
//如果是压缩文件
if (substr($file, -3) === '.gz') {
//4.5之前的不处理(这天改格式了)
if (substr($file, 11, 8) < '20180405') continue;
$zippedFiles[] = $file;
} else {
$unzippedFile = $file;
}
}
closedir($handle);
//本站最后请求时间
$this->lastRow = table('log')->row('*', ['feedName' => $this->feed], 'id desc')->toArray();
//按创建时间正序排序
asort($zippedFiles);
//逐个文件处理压缩文件
foreach ($zippedFiles as $file) {
$this->file = $file;
$this->zipped();
}
//如果有非压缩日志,处理
if ($unzippedFile) {
$this->file = $unzippedFile;
$this->unzipped();
}
}
/**
* 读取远程 文件内容
* @param $indicator string 远程 文件指示器
* @param $size int 文件大小
* @return Iterator 遍历器
*/
private function readUnzipped(string $indicator, int $size): Iterator
{
echo "Begin read File:$this->file:" . STool::kmgt($size) . "\r\n";
//打开文件,指向上次读取的位置
$f = fopen($indicator, 'r');
if (!$f) {
return;
}
if ($this->offset) {
fseek($f, $this->offset);
echo "Seek to $this->offset\r\n";
}
//总行数
$lines = 0;
//逐行读取
while (!feof($f)) {
$lines++;
$line = fgets($f);
//更新偏移量
$this->offset = ftell($f);
//返回行数
yield $line;
//每200行输出一个显示
if ($lines % 500 == 0) {
echo "read $this->feed $this->file Lines:$lines\r\n";
}
}
fclose($f);
echo "read $this->feed $this->file Lines:$lines\r\n";
echo "End.\r\n";
}
/**
* 读取远程 文件内容
* @return string 缓存文件路径
*/
private function readZipped(): string
{
//构造远程文件地址
$indicator = "ssh2.sftp://$this->sftp$this->logPath/$this->file";
//文件大小
$fileSize = filesize($indicator);
$size = STool::kmgt($fileSize);
//如果有缓存文件且缓存文件大小一致,则使用缓存文件
$cacheFile = self::CACHE_PATH . $this->feed . '/' . $this->file;
if (is_file($cacheFile) and filesize($cacheFile) == $fileSize) {
echo "Read Zipped File From Cache:" . $this->file . ' ' . $size . "\r\n";
return $cacheFile;
}
//从服务器读文件
echo "Begin read File:{$this->file}:" . $size . "\r\n";
$fileHandle = fopen($indicator, 'rb');
if (!$fileHandle) {
dump($indicator, 'OPEN FAIL');
exit;
}
//读取远程文件内容
$content = '';
$i = 0;
while (!feof($fileHandle)) {
//每次能读回8K字节
$content .= fread($fileHandle, 65536);
//每128K显示一次读取进度
$i++;
if ($i % 16 == 0) {
echo "$this->feed $this->file Reading :" . STool::kmgt(strlen($content)) . "/$size\r\n";
}
}
fclose($fileHandle);
//保存到缓存文件中
echo "Save to cache:" . $cacheFile . " \r\n";
makeDir(dirname($cacheFile));
file_put_contents($cacheFile, $content);
//返回压缩文件内容
return $cacheFile;
}
/**
* 字符串分行
* @param string $content
* @return Iterator
*/
public function explode(string $content): Iterator
{
$size = strlen($content);
$pointer = 0;
while ($pointer < $size) {
$next = strpos($content, "\n", $pointer);
if ($next === false) {
$line = substr($content, $pointer);
$next = $size;
} else {
$line = substr($content, $pointer, $next - $pointer);
}
yield $line;
$pointer = $next + 1;
}
}
private function valid(string $url): bool
{
return false !== strpos($url, '/?s=') or false !== strpos($url, '/?ss=') or preg_match('/^\/.*\/.*\/$/i', $url);
}
/**
* 处理一个压缩日志文件
*/
private function zipped(): void
{
//检查文件已经处理过
$fileTable = table('zipped');
if ($fileTable->exist(['feedName' => $this->feed, 'fileName' => $this->file])) return;
//读取文件内容
$gz=gzopen($this->readZipped(),'r');
echo "\r\nBegin Process File\r\n";
//$memTable = $this->createTemporaryTable(uniqid('tmp_'));
//要插入的日志表
$logTable = table('log');
//要插入的行缓冲区
$rows = [];
$insertRowsCount = 0;
$content = null;
$key=0;
while(!gzeof($gz)) {
$line=gzgets($gz);
if ((++$key) % 30000 == 0) {
echo "Analysis LINES:$key\r\n";
}
//空行不处理
$line = trim($line);
if (!$line) continue;
//行分解
$parts = $this->explodeLine($line);
if (!$parts) continue;
//判断 是否是 搜索 行
if (!$this->valid($parts['url'])) continue;
//检查是否已经处理过
if ($this->lastRow) {
if ($parts['timestamp'] < $this->lastRow['timestamp']) continue;
if ($parts['timestamp'] == $this->lastRow['timestamp'] and $parts['url'] == $this->lastRow['url'] and $parts['ip'] == $this->lastRow['ip']) {
continue;
}
}
//加入缓冲 区
$parts['feedName'] = $this->feed;
$rows[] = $parts;
//每4000行执行一次插入,再多就会出现placeholder太多
if (count($rows) >= 4000) {
$logTable->inserts($rows);
$insertRowsCount += count($rows);
SDebug::clearMsgs();
$rows = [];
}
}
//处理最后剩余的行
if (count($rows)) {
$logTable->inserts($rows);
$insertRowsCount += count($rows);
SDebug::clearMsgs();
}
echo "insert LINES:$insertRowsCount\r\n";
//标记此文件已经处理过
//$fileTable->begin();
//$this->move($memTable);
$fileTable->insert(['feedName' => $this->feed, 'fileName' => $this->file]);
//$fileTable->commit();
}
/**
* 将临时表中的日志转移到正式表中
* @param STable $memTable 临时表对象
*/
private function move(STable $memTable)
{
$fields = ['feedName', 'accessTime', 'timestamp', 'ip', 'requestTime', 'responseTime', 'method', 'url', 'code', 'length', 'referrer', 'agentId', 'created', 'updated', 'forward'];;
$fieldsStr = implode(',', $fields);
$memTable->execute("Insert" . " Into log($fieldsStr) select $fieldsStr from " . $memTable->name());
$memTable->deleteAll();
}
/**
* 当前文件的偏移
* @var int
*/
private $offset;
/**
* 处理一个未压缩的日志文件
*/
private function unzipped(): void
{
//检查上次处理情况
$fileTable = table('unzipped');
//如果没有记录,则生成一条初始记录
if ($fileTable->notExist(['feedName' => $this->feed])) {
$fileTable->insert(['feedName' => $this->feed, 'offset' => 0, 'size' => 0, 'timestamp' => 0]);
}
//取出处理信息,其中包含 offset(上次文件指针位置),size(上次文件大小), lasttime(上次最后时间)
$info = $fileTable->row('*', ['feedName' => $this->feed]);
//构造远程文件地址
$indicator = "ssh2.sftp://$this->sftp$this->logPath/$this->file";
//文件大小
$fileSize = filesize($indicator);
//文件变小了, 说明是新文件
if ($fileSize < $info['size']) {
$this->offset = 0;
} else {
// 取首行
$f = fopen($indicator, 'r');
$firstLine = fgets($f);
fclose($f);
$first = $this->explodeLine($firstLine);
$timestamp = $first['timestamp'];
if ($timestamp > $info['timestamp']) {
$this->offset = 0;
} else {
$this->offset = $info['offset'];
}
}
echo "\r\nBegin Process File\r\n";
//要插入的日志表
$logTable = table('log');
//要插入的行缓冲区
$rows = [];
$insertedRowsCount = 0;
$iterator = $this->readUnzipped($indicator, $fileSize);
$lastTime = 0;
foreach ($iterator as $key => $line) {
//空行不处理
$line = trim($line);
if (!$line) continue;
//分解 日志行
$parts = $this->explodeLine($line);
if (!$parts) continue;
//判断 是否是 搜索 行
if (!$this->valid($parts['url'])) continue;
//判断是否已经导入
if ($this->lastRow and (floatval($parts['timestamp']) < floatval($this->lastRow['timestamp']))) continue;
$rows[] = array_merge($parts, [
'feedName' => $this->feed
]);
//最大的时间戳
$lastTime = $parts['timestamp'];
//批量插入
if (count($rows) >= 100) {
$insertedRowsCount += count($rows);
$logTable->inserts($rows);
$fileTable->update(['size' => $fileSize, 'offset' => $this->offset, 'timestamp' => $lastTime], ['feedName' => $this->feed]);
echo "Insert LINES:$insertedRowsCount\r\n";
SDebug::clearMsgs();
$rows = [];
}
}
//处理最后剩余的行
if (count($rows)) {
$insertedRowsCount += count($rows);
$logTable->inserts($rows);
$fileTable->update(['size' => $fileSize, 'offset' => $this->offset, 'timestamp' => $lastTime], ['feedName' => $this->feed]);
echo "Insert LINES:$insertedRowsCount\r\n";
SDebug::clearMsgs();
}
}
/**
* 分解一行日志
* @param $line string
* @return array
* @throws Exception 匹配失败
*/
private function explodeLine(string $line): array
{
//[08/Apr/2018:03:30:17 +0800] 1523129417.075 72.178.128.43 - 0.114 - "GET /index.php/blog/search/?s=lowering%20ldl%20cholesterol&subid=tgr_zhen_BFX0J1ILON6N__rmlwuf_73004751 HTTP/1.1" 499 0 "http://168634854.keywordblocks.com/Cholesterol_Hdl_Ldl_Ratio.cfm?&vsid=1661264105777118&vi=1523124812717930856&dytm=1523124813100&kbbq=%26sde%3D1%26adepth%3D1%26ddepth%3D3&tdAdd[]=%7C%40%7Csde%3D1%7C%40%7Cadepth%3D1%7C%40%7Cddepth%3D3&sbdrId=135&vgd_matchstr=CommercialUrlOn%7Chlid%3D2002&matchstring=CommercialUrlOn%7Chlid%3D2002&vgd_bdata=ss%3D320x568%7C%7CMM%3D1.0%7C%7Cbb%3D145%7C%7CMP%3D.*%2Fcholesterol-management%2F.*%7C%7Cfbb%3D0%7C%7CRB%3D34.18110604079318%7C%7Cbtd%3D2341877441767294977%7C%7Ccbid%3D34.18110604079318%7C%7CMB%3D15.0%7C%7CMC%3DAUTO%7C%7Curl_l%3D50%7C%7Chour_group_l%3D20%7C%7CRImp%3D9.0%7C%7Cbid%3D15.1%7C%7Cdevice_l%3D20%7C%7CisRef%3D0&verid=111299&acid=427573913652889251523124810846&hvsid=00001523124812870012196577713996&upk=1523124813.1380&sttm=1523124812870&=&kp=1&kbc=143697&bdrid=4&subBdr=135&kt=266&ki=5912010&ktd=274911461948&kbc2=rpc%3D0.14&fdkt=266&lkpgd=UUID%3Duuid_s8_3_1523124813_778621763%7C%7CSI%3D863%7C%7CMPTD%3D232%7C%7CPTD%3D6922032661652308480%7C%7CSID%3D14%7C%7CCI%3D863%7C%7CMN%3D8%7C%7Cerpm%3D-1.0%7C%7CMI%3D863%7C%7CKTGD%3D3866%7C%7CKSE%3D1523124813242%7C%7CAN%3D5%7C%7CHID%3D3%7C%7CPTD2%3D16896&&lktgd=3866&&fp=biwFab2EOSptF9Dp9P5pLIuIHpVTe2ha94T5u6HCtebISTPUlc1la6_ujtvHa-nb8nGHPkJ_EnIwZF7mo3KnR2p3XYd1wmF70O9szYDQ9ufyP0OyS-gxVg%3D%3D&c=O5LJq2Lix-2w0IdspaXDCw&cme=rs5xevxSmJb0u22ZZHKqUTjYupvdJAHcw4kmb0sBhK6UBgyb-EKIO8Yg8DI2Uv0ZcpIG4AQvPb75jBLoeAG5VMn2cBgcO0Er9uHnU2G2b5527aplb-EHrVG_De8s_c_9-9bkhpH6jUmk3eK5uGthWBagtuatdg2SBe72cEUSh9aPY9sVJrkoOPaGQsQOH5rqAz1TMLK3_fisF-ozH6JyNg%3D%3D%7C%7CNDHRnZ9Gz3KXlI-i9OnZqQ%3D%3D%7C5gDUJdTGiJzedmq9hanWYg%3D%3D%7CtrJ5NInYpv_AyRdJRHyQbAoA6iGqXTxu%7CRrUTbnOe6Nf9cTuAtIJVy9no3H-wuOVy%7CN7fu2vKt8_s%3D%7Cl44MelaykDW0jQJG6bjukdQlinX0DB9oV4Sm9gijr_bD43Zl1UaHw39JatxHgP46euFaB3PMdSqZJqb8JKnexHrlF_K3RJ5R%7CJf0d-WoAdPuDA6UD6Gc_F1zJX7Ucny7osFvXic8Z4MU%3D%7Cue9AR4Lxeuwq7AuXzY3UTfqQIZ7T1ETAepQ5ZjhMUrn8F4iL72pDJxv9w1vxSK2jeiEactQl6VTIdrnkiwcfmH0laLhDYgMhmFyUaT5z0ZmFu4kbMwh587f73k-Z2prl2NRyNqvZoZL_mL9UwcCaoUiGM916VV0SyiuEizF5kMH-PgMGZNtaVAulY1i6cP1h%7C&ib=0&cid=8CU12LGKP&crid=285618735&size=300x250&lpid=&tsid=1&ksu=233&chid=&https=0&kwdsMaxTm=400&ugd=3&maxProviderPixel1=%2F%2Fc.ad-srv.co%2Fpixel&maxProviderPixel2=%2F%2Fc.adyield.co%2Fpixel&rms=1523124813&&sc=TX&asn=11427&kals=base&kalog=SI%3D863%7C%7CTPTD%3D516%7C%7CCI%3D863%7C%7CUUID%3Duuid_s12_nc1b_4_1523124812_210054693%7C%7CSID%3D11%7C%7CHID%3D4%7C%7CMI%3D863%7C%7CMPTD%3D176&kasts=tstype%3DBASE_BAG%7C%7C&kata=8ce5&clsKb=2&ecref=w77E%3ASS7mE8NQ.BJGYO.NmYS7mE8NSuSTOj%2BTJeJjQ%2BImLY1j%2BD1zyJS%3Fx7YMN1YE18yzvJY4RuuT%26x7YM7JLYvBw17n8QnzmLY1jnjOjnNwmjJQ7JLmj%26x7YMQmxLNJv%26x7YMYJO8xYvG%26x7YMNmz7Jz7vfHFWiXW9FiuH%26yM7yv%26yM78vUBOofiiAXAf9uWuH%26yMOJvY%26yMOYv%26yM1Evu7f%26yMzBvy%26yMN8vu9HHHfuWWX%26yM18vX9hiuHFXXFA%26yMjEvi9fhX9A%26yMj8v%26UvBw17n8QnzmLY1jnjOjnNwmjJQ7JLmj%26yNj8Ov%3Dd9C%3DgdBfCqpRD%3DfKDVQK6rMLAkw9YDzi%20YAVJOGawyN077a0Ezk7z8903%20AuogkUFBTjI%20%3DK8jODdV1K8Qg4KTBMBNR&kct=20512&abpl=2" "Mozilla/5.0 (iPhone; CPU iPhone OS 11_2_6 like Mac OS X) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0 Mobile/15D100 Safari/604.1" "-"
$ns = '([^\s]*)';
$str = '"([^"]*)"';
$datetime = '\[([^\]]*)\]';
//正则匹配
$matched = preg_match("/$datetime $ns $ns \- $ns $ns $str $ns $ns $str $str $str/i", $line, $matches);
if (!$matched) {
throw new Exception('NOT MATCH');
}
//空格区别 MODE URL HTTP协议
list($mode, $url, $protocol) = explode(' ', $matches[6]);
return [
'accessTime' => datetime(strtotime($matches[1])), //访问时间(秒)
'timestamp' => floatval($matches[2]),//访问时间戳(带毫秒)
'ip' => $matches[3], //请求者IP
'requestTime' => floatval($matches[4]), //Nginx处理请求的时间
'responseTime' => floatval($matches[5]), //Nginx完成整个响应的时间
'method' => $mode, //GET/POST/...
'url' => $url, //请求地址
'code' => $matches[7], //响应代码
'length' => intval($matches[8]), //响应正文长度
'referrer' => left($matches[9], 250), //引用
'agentId' => $this->getAgentId($matches[10]), //用户代理
'forward' => $matches[11] //真实IP
];
}
//获取Agent与ID的对应关系
private function getAgentMap()
{
$rows = table('agent')->select('id,agent', null, 'agent')->toArray();
return array_column($rows, 'id', 'agent');
}
//根据一个Agent,获取对应ID,如果没有则创建一个对应关系
private function getAgentId($agent)
{
//如果UA为空
if (!$agent) {
return 0;
}
//静态内存缓存
static $maps;
if (!$maps) {
$maps = $this->getAgentMap();
}
//缩减[FBAN/FBIOS;...]
$sub = mid($agent, '[', ']');
if ($sub) {
$agent = str_replace('[' . $sub . ']', '[...]', $agent);
}
$agent = str_replace(' (KHTML, like Gecko)', '', $agent);
//变种归并
$agent = preg_replace($this->agentPatterns, $this->agentReplaces, $agent);
//Agent缩减到250个字符
$agent = left($agent, 191);
if (!isset($maps[$agent])) {
$id = table('agent')->insertIgnore(['agent' => $agent]);
$maps[$agent] = $id;
}
return $maps[$agent];
}
}————————————————
版权声明：本文为CSDN博主「蓝冰大侠」的原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接及本声明。
原文链接：https://blog.csdn.net/bluehire/article/details/79985203