更新時間:2019-10-16 來源:黑馬程序員 瀏覽量:
日志在計算機(jī)系統(tǒng)中是一個非常廣泛的概念,任何程序都有可能輸出日志:操作系統(tǒng)內(nèi)核、各種應(yīng)用服務(wù)器等等。日志的內(nèi)容、規(guī)模和用途也各不相同,很難一概而論。
Web日志中包含了大量人們——主要是產(chǎn)品分析人員會感興趣的信息,最簡單的,我們可以從中獲取網(wǎng)站每類頁面的PV值(PageView,頁面訪問量)、獨(dú)立IP數(shù)(即去重之后的IP數(shù)量)等;稍微復(fù)雜一些的,可以計算得出用戶所檢索的關(guān)鍵詞排行榜、用戶停留時間最高的頁面等;更復(fù)雜的,構(gòu)建廣告點(diǎn)擊模型、分析用戶行為特征等等。【推薦了解軟件測試課程】
今天給大家介紹一款日志分析工具:ELK
ELK由Elasticsearch、Logstash和Kibana三部分組件組成;
Elasticsearch 是基于 JSON 的分布式搜索和分析引擎,專為實現(xiàn)水平擴(kuò)展、高可用和管理便捷性而設(shè)計
Logstash 是開源的服務(wù)器端數(shù)據(jù)處理管道,能夠同時 從多個來源采集數(shù)據(jù)、轉(zhuǎn)換數(shù)據(jù),然后將數(shù)據(jù)發(fā)送到您最喜歡的 “存儲庫” 中。(我們的存儲庫當(dāng)然是 Elasticsearch。)
Kibana 能夠以圖表的形式呈現(xiàn)數(shù)據(jù),并且具有可擴(kuò)展的用戶界面,供您全方位配置和管理 Elastic Stack。
今天的試驗是:通過ELK分析線上所有Nginx的訪問日志。
一、試驗拓?fù)鋱D
二、軟件包獲得
Nginx下載http://nginx.org/en/download.html
Redis下載 https://redis.io/
Elasticsearch logstash kibana下載 https://www.elastic.co/downloads
三、開始部署
3.1)業(yè)務(wù)機(jī)部署A
業(yè)務(wù)機(jī):192.168.1.242/24
OS:rhel6.5
涉及軟件:nginx+logstash+redis+jdk
軟件包準(zhǔn)備:根據(jù)上述的提示下載軟件包
[root@242 opt]# ls
jdk-8u144-linux-x64.rpm logstash-5.5.1.tar.gz nginx-1.13.4.tar.gz redis-4.0.1.tar.gz
3.1.1)安裝JDK
[root@242 opt]# rpm -ivh jdk-8u144-linux-x64.rpm
Preparing... ####################################### [100%]
1:jdk1.8.0_144 ######################################## [100%]
Unpacking JAR files...
tools.jar...
plugin.jar...
javaws.jar...
deploy.jar...
rt.jar...
jsse.jar...
charsets.jar...
localedata.jar...
設(shè)置java環(huán)境變量
[root@242 opt]# vim /root/.bash_profile 末尾追加一下內(nèi)容
JAVA_HOME=/usr/java/jdk1.8.0_144
PATH=$JAVA_HOME/bin:$PATH:$HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
export PATH JAVA_HOME CLASSPATH CATALINA_HOME
生效配置并驗證
[root@242 opt]# source /root/.bash_profile
[root@242 opt]# java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
3.1.2)安裝redis
[root@242 opt]# tar xf redis-4.0.1.tar.gz
[root@242 opt]# cd redis-4.0.1
[root@242 redis-4.0.1]# make
[root@242 redis-4.0.1]# make install
配置redis
[root@242 redis-4.0.1]# sed -i -r '/^(bind)/s/127.0.0.1/0.0.0.0/' redis.conf
[root@242 redis-4.0.1]# sed -i -r '/^(daemonize)/s/no/yes/' redis.conf
啟動redis
[root@242 redis-4.0.1]# redis-server redis.conf
5789:C 30 Aug 11:09:58.584 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
5789:C 30 Aug 11:09:58.584 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=5789, just started
5789:C 30 Aug 11:09:58.584 # Configuration loaded
驗證啟動
[root@242 redis-4.0.1]# lsof -i :6379
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
redis-ser 5790 root 6u IPv4 18672 0t0 TCP *:6379 (LISTEN)
3.1.3)安裝nginx
[root@242 opt]# tar xf nginx-1.13.4.tar.gz
[root@242 opt]# cd nginx-1.13.4
[root@242 nginx-1.13.4]# yum -y install pcre-devel zlib-devel
[root@242 nginx-1.13.4]# ./configure --prefix=/usr/local/nginx
[root@242 nginx-1.13.4]# make
[root@242 nginx-1.13.4]# make install
修改nginx配置文件,重新定義log_format 以json格式輸出日志到access.log
[root@242 nginx-1.13.4]# cd /usr/local/nginx/conf/
[root@242 conf]# vim nginx.conf
啟動nginx并驗證
[root@242 conf]# /usr/local/nginx/sbin/nginx
[root@242 conf]# lsof -i :80
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nginx 10765 root 6u IPv4 24510 0t0 TCP *:http (LISTEN)
nginx 10766 nobody 6u IPv4 24510 0t0 TCP *:http (LISTEN)
訪問一次nginx 驗證日志格式是否正確
3.1.4)安裝logstash
[root@242 opt]# tar xf logstash-5.5.1.tar.gz -C /usr/local/
[root@242 opt]# cd /usr/local/logstash-5.5.1/
[root@242 logstash-5.5.1]# mkdir conf.d
[root@242 logstash-5.5.1]# vim conf.d/nginx_to_redis
input {
file {
path => ["/usr/local/nginx/logs/access.log"]
type => "nginx_log"
codec => json
}
}
output {
redis{
host => "192.168.1.242"
key => 'logstash:redis'
data_type => 'channel'
port => '6379'
}
stdout {
codec => rubydebug
}
}
啟動logstash 并測試是否成功收集nginx日志到redis
[root@242 ~]# /usr/local/logstash-5.5.1/bin/logstash -f /usr/local/logstash-5.5.1/conf.d/nginx_to_redis
查看啟動日志
[root@242 logstash-5.5.1]# tailf /usr/local/logstash-5.5.1/logs/logstash-plain.log
測試日志收集
Logstash收集日志輸出
開啟redis 監(jiān)控
3.2)業(yè)務(wù)機(jī)部署B(yǎng)
業(yè)務(wù)機(jī):192.168.1.241/24
OS:rhel6.5
涉及軟件:elasticsearch+logstash+kibana
[root@241 opt]# ls
elasticsearch-5.5.1.rpm
kibana-5.5.1-x86_64.rpm
jdk-8u144-linux-x64.rpm
logstash-5.5.1.tar.gz
3.2.1)安裝jdk
參考242設(shè)置
3.2.2)安裝elasticsearch
[root@241 opt]# rpm -ivh elasticsearch-5.5.1.rpm
warning: elasticsearch-5.5.1.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing... ######################################## [100%]
Creating elasticsearch group... OK
Creating elasticsearch user... OK
1:elasticsearch ######################################## [100%]
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using chkconfig
sudo chkconfig --add elasticsearch
### You can start elasticsearch service by executing
sudo service elasticsearch start
配置elasticsearch
[root@241 opt]# sed -i -r '/^(#network\.host:)/cnetwork.host: 0.0.0.0' /etc/elasticsearch/elasticsearch.yml
[root@241 opt]# sed -i -r '/^(#http\.port:)/chttp.port: 9200' /etc/elasticsearch/elasticsearch.yml
[root@241 opt]# sed -i -r '/^(#bootstrap\.memory_lock:)/c\bootstrap.memory_lock: false\nbootstrap.system_call_filter: false' /etc/elasticsearch/elasticsearch.yml
優(yōu)化系統(tǒng)
[root@241 opt]# vim /etc/security/limits.conf 末尾追加
elasticsearch soft nproc 10240
elasticsearch hard nproc 10240
* soft nofile 65540
* hard nofile 65540
重啟計算機(jī)生效
啟動elasticsearch
[root@241 opt]# /etc/init.d/elasticsearch start
Starting elasticsearch: [ OK ]
驗證
3.2.3)安裝logstash
[root@241 opt]# tar xf logstash-5.5.1.tar.gz -C /usr/local/
[root@241 opt]# cd /usr/local/logstash-5.5.1/
[root@241 logstash-5.5.1]# mkdir conf.d
[root@241 logstash-5.5.1]# vim conf.d/redis_to_elk
input {
redis {
port => "6379"
host => "192.168.1.242"
data_type => "channel"
key => "logstash:redis"
type => "redis-input"
}
}
output {
elasticsearch {
hosts => "192.168.1.241"
index => "logstash-%{+YYYY.MM.dd}"
action => "index"
}
stdout {
codec => rubydebug
}
}
啟動logstash
[root@241 logstash-5.5.1]#./bin/logstash -f conf.d/redis_to_elk
訪問一次測試數(shù)據(jù)是否有redis寫入到elk
3.2.4)安裝kibana
[root@241 opt]# rpm -ivh kibana-5.5.1-x86_64.rpm
warning: kibana-5.5.1-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing... ####################################### [100%]
1:kibana ####################################### [100%]
修改配置文件中的
[root@241 opt]# sed -i -r '/^(#server.host:)/c\server.host: "0.0.0.0"' /etc/kibana/kibana.yml
[root@241 opt]# /etc/init.d/kibana start
kibana started
驗證啟動
[root@241 opt]# netstat -ntpl |grep 5601
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 1993/node
測試通過瀏覽器