https://index.zshipu.com/geek001/tags/Logstash/ Recent content in Logstash on 知识铺的博客 Hugo -- gohugo.io zh-CN Thu, 23 May 2024 00:53:53 +0000 https://index.zshipu.com/geek001/post/20240507/%E4%BD%BF%E7%94%A8Logstash%E7%9A%84Multiline%E6%8F%92%E4%BB%B6%E6%94%B6%E9%9B%86%E5%A4%9A%E8%A1%8C%E6%97%A5%E5%BF%97--%E7%9F%A5%E8%AF%86%E9%93%BA/ Thu, 23 May 2024 00:53:53 +0000 https://index.zshipu.com/geek001/post/20240507/%E4%BD%BF%E7%94%A8Logstash%E7%9A%84Multiline%E6%8F%92%E4%BB%B6%E6%94%B6%E9%9B%86%E5%A4%9A%E8%A1%8C%E6%97%A5%E5%BF%97--%E7%9F%A5%E8%AF%86%E9%93%BA/ 收集跨多行日志，我们需要用到codec的multiline插件来实现，它可以将多行进行合并加入单个事件，例如收集Java exception就可以用到它。 官方文档：https://www.elastic.co/guide/en/logstash/current/plugins-co https://index.zshipu.com/geek001/post/20240507/Docker%E9%83%A8%E7%BD%B2Logstash%E5%B9%B6%E9%85%8D%E7%BD%AEMySQL%E8%BE%93%E5%85%A5%E5%88%B0Elasticsearch%E8%BE%93%E5%87%BA--%E7%9F%A5%E8%AF%86%E9%93%BA/ Tue, 07 May 2024 10:30:47 +0000 https://index.zshipu.com/geek001/post/20240507/Docker%E9%83%A8%E7%BD%B2Logstash%E5%B9%B6%E9%85%8D%E7%BD%AEMySQL%E8%BE%93%E5%85%A5%E5%88%B0Elasticsearch%E8%BE%93%E5%87%BA--%E7%9F%A5%E8%AF%86%E9%93%BA/ 1.先pull镜像 1 docker pull logstash:7.6.2 1 2 mkdir -p /usr/share/logstash/conf.d mkdir /usr/share/logstash/log 1 vim logstash.yml 1 2 path.config: /usr/share/logstash/conf.d/*.conf path.logs: /usr/share/logstash/log vim test.conf 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 input { stdin { } jdbc { jdbc_connection_string => "jdbc:mysql://url:port/database?useSSL=false&useUnicode=true&characterEncoding=utf-8&useLegacyDatetimeCode=false&allowPublicKeyRetrieval=true&serverTimezone=Asia/Shanghai" jdbc_driver_class => "com.mysql.cj.jdbc.Driver" jdbc_user => "root" jdbc_password => "root" jdbc_paging_enabled => "true" jdbc_page_size => "50000" jdbc_default_timezone => "Asia/Shanghai" jdbc_driver_library => "/usr/share/logstash/conf.d/mysql-connector-java-8.0.28.jar" statement => "SELECT * FROM table_name" schedule => "* * * * *" lowercase_column_names => false } } filter { date { match => ["time_stamp","yyyy-MM-dd HH:mm:ss.SSS"] target => "@time_stamp" } } output { elasticsearch { hosts => ["http://url:9200"] index => "your self index" document_id => https://index.zshipu.com/geek001/post/20240507/Linux%E7%B3%BB%E7%BB%9F%E4%B8%8BLogstash%E6%9C%8D%E5%8A%A1%E7%9A%84%E5%90%AF%E5%81%9C%E5%8F%8A%E9%85%8D%E7%BD%AE--%E7%9F%A5%E8%AF%86%E9%93%BA/ Tue, 07 May 2024 10:10:47 +0000 https://index.zshipu.com/geek001/post/20240507/Linux%E7%B3%BB%E7%BB%9F%E4%B8%8BLogstash%E6%9C%8D%E5%8A%A1%E7%9A%84%E5%90%AF%E5%81%9C%E5%8F%8A%E9%85%8D%E7%BD%AE--%E7%9F%A5%E8%AF%86%E9%93%BA/ 1、查看 1 systemctl status logstash 2、启动 1 systemctl start logstash 3、停止服务 1 systemctl stop logstash 4、添加config.reload.automatic命令参数，自动加载配置，不需要重新启动logstash 1 /usr/share/logstash/bin/logstash -f /home/fxd/logstash/tizi-fxd.conf --config.reload.automatic 5、通过配置文件启动logstash 1 /usr/share/logstash/bin/logstash -f /home/fxd/logstash/tizi-fxd.conf 6、后台运行logstash 加& 1 /usr/share/logstash/bin/logstash -f /home/fxd/logstash/tizi-fxd.conf --config.reload.automatic & 7、 查看后台运 https://index.zshipu.com/geek001/post/20240507/%E5%A6%82%E4%BD%95%E5%B0%86MySQL%E6%95%B0%E6%8D%AE%E5%90%8C%E6%AD%A5%E5%88%B0Elasticsearch--%E7%9F%A5%E8%AF%86%E9%93%BA/ Tue, 07 May 2024 10:09:47 +0000 https://index.zshipu.com/geek001/post/20240507/%E5%A6%82%E4%BD%95%E5%B0%86MySQL%E6%95%B0%E6%8D%AE%E5%90%8C%E6%AD%A5%E5%88%B0Elasticsearch--%E7%9F%A5%E8%AF%86%E9%93%BA/ 上一篇我们已经完成Elasticsearch 和logstash安装，现在可以进行把数据从mysql 数据库同步es索引上 1、下载java 数据库连接池 [root@localhost home]# cd /<span>home [root@localhost home]# </span><span>wget</span> https:<span>//</span><span>repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.13/mysql-connector-java-8.0.13.jar</span> 2、安装logstash-input-jdb （如果你是按上一篇说的安装的是7.1以上的logstash 版本,调过这一步 https://index.zshipu.com/geek001/post/20240507/%E4%BC%81%E4%B8%9A%E8%BF%90%E7%BB%B4%E5%AE%9E%E6%88%98--ELK%E6%97%A5%E5%BF%97%E5%88%86%E6%9E%90%E5%B9%B3%E5%8F%B0%E4%B9%8Blogstash%E6%95%B0%E6%8D%AE%E9%87%87%E9%9B%86--%E7%9F%A5%E8%AF%86%E9%93%BA/ Tue, 07 May 2024 10:08:47 +0000 https://index.zshipu.com/geek001/post/20240507/%E4%BC%81%E4%B8%9A%E8%BF%90%E7%BB%B4%E5%AE%9E%E6%88%98--ELK%E6%97%A5%E5%BF%97%E5%88%86%E6%9E%90%E5%B9%B3%E5%8F%B0%E4%B9%8Blogstash%E6%95%B0%E6%8D%AE%E9%87%87%E9%9B%86--%E7%9F%A5%E8%AF%86%E9%93%BA/ 企业运维实战–ELK日志分析平台之logstash数据采集 前言–logstash简介 logstash数据采集 安装 日志采集输出插件 前言–logstash简介 Logstash是一个开源数据收集引擎，具有实时管道功能。Logstash可以动态地将来自不同数据源的 https://index.zshipu.com/geek001/post/20240507/%E5%9C%A8CentOS-7.x%E4%B8%8A%E5%AE%89%E8%A3%85JavaElasticsearch%E5%92%8CLogstash--%E7%9F%A5%E8%AF%86%E9%93%BA/ Tue, 07 May 2024 10:07:47 +0000 https://index.zshipu.com/geek001/post/20240507/%E5%9C%A8CentOS-7.x%E4%B8%8A%E5%AE%89%E8%A3%85JavaElasticsearch%E5%92%8CLogstash--%E7%9F%A5%E8%AF%86%E9%93%BA/ 安装环境基于CentOS 7.x 1、安装java 环境 安装java 8 参考：https://www.cnblogs.com/myibm/p/9232744.html 查看是否安装成功 [root@localhost ~]# java -version openjdk version “1.8.0_312” 2、ElasticSearch 和 logstash 安装 获取密钥 [root@localhost ~]# rpm --import https:<span>//</span><span>artifacts.elastic.co/GPG-KEY-elasticsearch</span> 下载 ELK三个套件版本要一致，这里获取的