4.5、清理开机自启动服务
for i in `chkconfig --list|grep 3:on |awk '{print $1}'`;do chkconfig$i off;donefor i in crond network rsyslog sshd;do chkconfig --level 3 $ion;donechkconfig --list|grep 3:on4.6、服务器时间同步
echo '*/5 * * * * /usr/sbin/ntpdate time.windows.com > /dev/null2>&1' >>/var/spool/cron/root4.7、加大文件描述符
echo '* - nofile 65535' >> /etc/security/limits.conftail -1 /etc/security/limits.conf#重新登陆后生效(无需重启)ulimit -n(重新登陆后查看)4.8、内核参数调优(可不操作)
cp /etc/sysctl.conf /etc/sysctl.conf.bakcat>>/etc/sysctl.conf<<EOFnet.ipv4.tcp_timestamps = 0net.ipv4.tcp_synack_retries = 2net.ipv4.tcp_syn_retries = 2net.ipv4.tcp_mem = 94500000 915000000 927000000net.ipv4.tcp_max_orphans = 3276800net.core.wmem_default = 8388608net.core.rmem_default = 8388608net.core.rmem_max = 16777216net.core.wmem_max = 16777216net.ipv4.tcp_rmem=4096 87380 16777216net.ipv4.tcp_wmem=4096 65536 16777216net.core.netdev_max_backlog = 32768net.core.somaxconn = 32768net.ipv4.tcp_syncookies=1net.ipv4.tcp_tw_reuse = 1net.ipv4.tcp_tw_recycle = 1net.ipv4.tcp_fin_timeout=1net.ipv4.tcp_keepalive_time=1200net.ipv4.tcp_max_syn_backlog = 65536net.ipv4.ip_local_port_range = 1024 65535EOF/sbin/sysctl -p五、部署开始
5.1、更改nginx日志输出格式
5.1.1、定义日志格式
cat /etc/nginx/nginx.conflog_format main '$remote_addr - $remote_user [$time_local]"$request" ''$status$body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for"';5.1.2、加载日志格式到对应域名配置中
cat /etc/nginx/conf.d/vhost/api.mingongge.com.cn.confserver {listen 80;server_name  newtest-msp-api.mingongge.com.cn;access_log   /var/log/nginx/api.mingongge.com.cn.log main;}5.1.3、reload生效
nginx -s reload5.1.4、清空原输出文件，并查看输出的日志格式
:> /var/log/nginx/api.mingongge.com.cn.logtailf /var/log/nginx/api.mingongger.com.cn.log1xx.2xx.72.175 - - [18/Mar/2019:13:51:17 +0800] "GET/user/fund/113 HTTP/1.1" 200 673 "-" "Mozilla/5.0 (WindowsNT 10.0; Win64; x64) AppleWebKit/537.36 (Khtml, like Gecko) sun/1.5.6 Chrome/69.0.3497.106Electron/4.0.3 Safari/537.36" "-"5.2、配置kafka
测试环境使用docker起的kafka，kafka部署掠过,以下任选一种
5.2.1、方法一 创建kafka topic
./kafka-topics.sh --create --topic nginxlog --replication-factor 1--partitions 1 --zookeeper localhost:21815.2.2、方法二
auto.create.topics.enable=true开启kafka自动创建topic配置
5.2.3、filebeat部署完成后确认kafka topic中有数据
./kafka-console-consumer.sh --bootstrap-server 192.168.0.53:9091--from-beginning --topic nginxlog输出如下
{"@timestamp":"2019-03-14T07:16:50.140Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.3.2","topic":"nginxlog"},"fields":{"log_topics":"nginxlog"},"beat":{"version":"6.3.2","name":"test-kafka-web","hostname":"test-kafka-web"},"host":{"name":"test-kafka-web"},"source":"/var/log/nginx/newtest-msp-api.mingongge.com.cn-80.log","offset":114942,"message":"116.226.72.175- - [14/Mar/2019:15:16:49 +0800] newtest-msp-api.mingongge.com.cn POST"/upstream/page" "-" 200 6314"http://newtest-msp-crm.mingongge.com.cn/" 200 192.168.0.49:60070.024 0.024 "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36(KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36""-""}Processed a total of 7516 messages测试环境中kafka地址为
192.168.0.53:90915.3、配置filebeat收集nginx日志
5.3.1、安装filebeat
cd /opt/ && wget http://download.mingongge.com.cn/download/software/filebeat-6.3.2-x86_64.rpmyum localinstall filebeat-6.3.2-x86_64.rpm -y5.3.2、编辑配置文件
cat /etc/filebeat/filebeat.ymlfilebeat.prospectors:- input_type: logenabled: truepaths:- /var/log/nginx/api.mingongge.com.cn.log#收集日志路径fields:log_topics: nginxlog #kafka中topic名称json.keys_under_root: truejson.overwrite_keys: trueoutput.kafka:enabled: truehosts:["192.168.0.53:9091"] #kafka地址topic:'%{[fields][log_topics]}' #kafka中topic名称partition.round_robin:reachable_only: falsecompression: gzipmax_message_bytes: 1000000required_acks: 1