https://index.zshipu.com/geek001/tags/Elasticsearch/ Recent content in Elasticsearch on 知识铺的博客 Hugo -- gohugo.io zh-CN Wed, 28 Aug 2024 08:39:34 +0000 https://index.zshipu.com/geek001/post/20240828/%E9%98%BF%E9%87%8C%E4%BA%91%E9%AD%8F%E5%AD%90%E7%8F%BA-%E9%98%BF%E9%87%8C%E4%BA%91Elasticsearch-AI%E6%90%9C%E7%B4%A2%E5%AE%9E%E8%B7%B5--%E7%9F%A5%E8%AF%86%E9%93%BA/ Wed, 28 Aug 2024 08:39:34 +0000 https://index.zshipu.com/geek001/post/20240828/%E9%98%BF%E9%87%8C%E4%BA%91%E9%AD%8F%E5%AD%90%E7%8F%BA-%E9%98%BF%E9%87%8C%E4%BA%91Elasticsearch-AI%E6%90%9C%E7%B4%A2%E5%AE%9E%E8%B7%B5--%E7%9F%A5%E8%AF%86%E9%93%BA/ Tech Day 本文由阿里云技术专家魏子珺在【AI搜索TechDay】上的分享【阿里云Elasticsearch AI搜索实践】整理而成。 【AI搜索TechDay】是Elastic和阿里云联合主办的AI技术Meetup系列，聚焦企业级AI搜索应用和开发者动手实践，旨在帮助开发者在大模型浪潮下升 https://index.zshipu.com/geek001/post/20240507/%E7%99%BE%E4%BA%BF%E7%BA%A7%E6%95%B0%E6%8D%AE%E5%AD%98%E5%82%A8%E6%9E%B6%E6%9E%84%E8%B5%B7%E6%BA%90--%E7%9F%A5%E8%AF%86%E9%93%BA/ Fri, 14 Jun 2024 02:14:18 +0000 https://index.zshipu.com/geek001/post/20240507/%E7%99%BE%E4%BA%BF%E7%BA%A7%E6%95%B0%E6%8D%AE%E5%AD%98%E5%82%A8%E6%9E%B6%E6%9E%84%E8%B5%B7%E6%BA%90--%E7%9F%A5%E8%AF%86%E9%93%BA/ 尼恩：百亿级数据存储架构起源 在40岁老架构师 尼恩的读者交流群(50+)中，经常性的指导小伙伴们改造简历。 经过尼恩的改造之后，很多小伙伴拿到了一线互联网企业如得物、阿里、滴滴、极兔、有赞、希音、百度、网易、美团的面试机会，拿到了大厂机会。 这些机会的来源，主要是尼恩给小伙伴 改造了简历 https://index.zshipu.com/geek001/post/20240507/Docker%E9%83%A8%E7%BD%B2Logstash%E5%B9%B6%E9%85%8D%E7%BD%AEMySQL%E8%BE%93%E5%85%A5%E5%88%B0Elasticsearch%E8%BE%93%E5%87%BA--%E7%9F%A5%E8%AF%86%E9%93%BA/ Tue, 07 May 2024 10:30:47 +0000 https://index.zshipu.com/geek001/post/20240507/Docker%E9%83%A8%E7%BD%B2Logstash%E5%B9%B6%E9%85%8D%E7%BD%AEMySQL%E8%BE%93%E5%85%A5%E5%88%B0Elasticsearch%E8%BE%93%E5%87%BA--%E7%9F%A5%E8%AF%86%E9%93%BA/ 1.先pull镜像 1 docker pull logstash:7.6.2 1 2 mkdir -p /usr/share/logstash/conf.d mkdir /usr/share/logstash/log 1 vim logstash.yml 1 2 path.config: /usr/share/logstash/conf.d/*.conf path.logs: /usr/share/logstash/log vim test.conf 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 input { stdin { } jdbc { jdbc_connection_string => "jdbc:mysql://url:port/database?useSSL=false&useUnicode=true&characterEncoding=utf-8&useLegacyDatetimeCode=false&allowPublicKeyRetrieval=true&serverTimezone=Asia/Shanghai" jdbc_driver_class => "com.mysql.cj.jdbc.Driver" jdbc_user => "root" jdbc_password => "root" jdbc_paging_enabled => "true" jdbc_page_size => "50000" jdbc_default_timezone => "Asia/Shanghai" jdbc_driver_library => "/usr/share/logstash/conf.d/mysql-connector-java-8.0.28.jar" statement => "SELECT * FROM table_name" schedule => "* * * * *" lowercase_column_names => false } } filter { date { match => ["time_stamp","yyyy-MM-dd HH:mm:ss.SSS"] target => "@time_stamp" } } output { elasticsearch { hosts => ["http://url:9200"] index => "your self index" document_id => https://index.zshipu.com/geek001/post/20240507/Elasticsearch%E7%B4%A2%E5%BC%95%E6%98%A0%E5%B0%84%E5%BC%82%E5%B8%B8%E5%AD%97%E6%AE%B5%E5%90%8D%E4%B8%8D%E8%83%BD%E5%8C%85%E5%90%AB.--%E7%9F%A5%E8%AF%86%E9%93%BA/ Tue, 07 May 2024 10:15:47 +0000 https://index.zshipu.com/geek001/post/20240507/Elasticsearch%E7%B4%A2%E5%BC%95%E6%98%A0%E5%B0%84%E5%BC%82%E5%B8%B8%E5%AD%97%E6%AE%B5%E5%90%8D%E4%B8%8D%E8%83%BD%E5%8C%85%E5%90%AB.--%E7%9F%A5%E8%AF%86%E9%93%BA/ kafka队列如下： { "@timestamp" : "2017-06-23T09:09:02.887Z" , "beat" :{ "name" : "192.168.12.106" }, "input_type" : "log" , "message" :"[2016-05-25 12:39:04,744][DEBUG][action.bulk ] [Set] [***][3] failed to execute bulk item (index) index {[***][***][***], source [{***}}\n MapperParsingException[Field name [events.created] cannot contain '.' ]\n at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:273)\n at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:218)\n at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:193)\n at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:305)\n at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:218)\n at org.elasticsearch.index.mapper.object.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:139)\n at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:118)\n at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:99)\n at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:498)\n at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:257)\n at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230)\n at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468)\n at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)\n at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)\n at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n at java.lang.Thread.run(Thread.java:745)\n\n\n\n "," offset ":35737," source ":" /tmp/test .log "," type ":" log"} https://index.zshipu.com/geek001/post/20240507/%E5%A6%82%E4%BD%95%E5%B0%86MySQL%E6%95%B0%E6%8D%AE%E5%90%8C%E6%AD%A5%E5%88%B0Elasticsearch--%E7%9F%A5%E8%AF%86%E9%93%BA/ Tue, 07 May 2024 10:09:47 +0000 https://index.zshipu.com/geek001/post/20240507/%E5%A6%82%E4%BD%95%E5%B0%86MySQL%E6%95%B0%E6%8D%AE%E5%90%8C%E6%AD%A5%E5%88%B0Elasticsearch--%E7%9F%A5%E8%AF%86%E9%93%BA/ 上一篇我们已经完成Elasticsearch 和logstash安装，现在可以进行把数据从mysql 数据库同步es索引上 1、下载java 数据库连接池 [root@localhost home]# cd /<span>home [root@localhost home]# </span><span>wget</span> https:<span>//</span><span>repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.13/mysql-connector-java-8.0.13.jar</span> 2、安装logstash-input-jdb （如果你是按上一篇说的安装的是7.1以上的logstash 版本,调过这一步 https://index.zshipu.com/geek001/post/20240507/%E5%9C%A8CentOS-7.x%E4%B8%8A%E5%AE%89%E8%A3%85JavaElasticsearch%E5%92%8CLogstash--%E7%9F%A5%E8%AF%86%E9%93%BA/ Tue, 07 May 2024 10:07:47 +0000 https://index.zshipu.com/geek001/post/20240507/%E5%9C%A8CentOS-7.x%E4%B8%8A%E5%AE%89%E8%A3%85JavaElasticsearch%E5%92%8CLogstash--%E7%9F%A5%E8%AF%86%E9%93%BA/ 安装环境基于CentOS 7.x 1、安装java 环境 安装java 8 参考：https://www.cnblogs.com/myibm/p/9232744.html 查看是否安装成功 [root@localhost ~]# java -version openjdk version “1.8.0_312” 2、ElasticSearch 和 logstash 安装 获取密钥 [root@localhost ~]# rpm --import https:<span>//</span><span>artifacts.elastic.co/GPG-KEY-elasticsearch</span> 下载 ELK三个套件版本要一致，这里获取的