本文共 21061 字，大约阅读时间需要 70 分钟。

在 上一节中我们跑通了整个架构的数据链路，后面将持续更新，逐步细化其中的几个问题，比如：

1、Filebeat如何在采集数据时只提取有用的数据发给kafka？

2、logstash消费了kafka消息后，如何格式化数据发给elasticsearch？

3、kibana又如何用直观的显示我们希望看到的日志报表？

今天这篇文章将解决第一个问题，Filebeat如何在采集数据时只提取有用的数据发给kafka的。

核心操作就是配置filebeat.yml文件：

官方配置说明：（选择版本）

版本配置页面地址：

也可以参考我的

一、过滤我们需要的信息，日志文件内容假设如下：

2017/06/22 11:26:30 [ERROR] 26067#0: *17918 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.32.17, server: localhost, request: "GET /wss/ HTTP/1.1", upstream: "http://192.168.12.106:8010/", host: "192.168.12.106"2017/06/22 11:26:30 [info] 26067#0:2017/06/22 12:05:10 [error] 26067#0: *17922 open() "/data/programs/nginx/html/ws" failed (2: No such file or directory), client: 192.168.32.17, server: localhost, request: "GET /ws HTTP/1.1", host: "192.168.12.106"

 1、使用include_lines参数过滤error行信息：不区分大小写

# Include lines. A list of regular expressions to match. It exports the lines that are  # matching any regular expression from the list.  include_lines: ['[ERROR]']

2、查看kibana信息，可以看到error的两行被采集，且不区'error'分大小写

补充：今天测试的时候又发现filebeat里面区分大小写，而且还不能识别特殊字符，比如圆括号，中括号等，要改成下面的才行：

include_lines: ['\[ERROR\]','\[error\]']

3、对于java应用常见的错误是多行的，我们来一个多行的错误采集测试，使用multiline与include_lines结合使用,日志如下

[2019-11-20 11:26:30.573][ERROR] 26067#0: *17918 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.32.17, server: localhost, request: "GET /wss/ HTTP/1.1", upstream: "http://192.168.12.106:8010/", host: "192.168.12.106"[2019-11-20 11:26:30.573][info] 26067#0:[2019-11-20 12:05:10.123][error] 26067#0: *17922 open() "/data/programs/nginx/html/ws" failed (2: No such file or directory), client: 192.168.32.17, server: localhost, request: "GET /ws HTTP/1.1", host: "192.168.12.106"[2019-11-20 20:25:22.573][DubboServerHandler-192.168.0.5:20904-thread-3][ERROR][?][]:{conn-10005, pstmt-20003} execute error. select count(1)from (	select t.RETAIL_INVOICE_ID as retailInvoiceId, t.RETAIL_INVOICE_CODE as retailInvoiceCode, t.BILL_CODE as billCode, t.CUSTOMER_ID as customerId, t.CURRENCY as currency		, t.SALE_CHANNEL as saleChannel, t.REMARK as remark, t.COMPANY_ID as companyId, t.CREATED_BY as createdBy, t.CREATED_DATE as createdDate		, t.MODIFIED_BY as modifiedBy, t.MODIFIED_DATE as modifiedDate, t.RD_STATUS as rdStatus, t.DATA_ID as dataId, t.COMPANY_NAME as companyName		, case T.DATA_FROM			when 1 then 'TMS'			when 2 then '接口'			when 3 then '手动导入'			else null		end as dataFromName	where T.RD_STATUS = 1		and TSP.BL_SCM = 1		and t.DATE_INVOICE >= to_date('2019-11-13 00:00:00', 'yyyy-mm-dd hh24:mi:ss')		and t.DATE_INVOICE <= to_date('2019-11-20 23:59:59', 'yyyy-mm-dd hh24:mi:ss')	order by T.CREATED_DATE desc) TOTALjava.sql.SQLSyntaxErrorException: ORA-00920: invalid relational operator	at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:91) ~[mybatis-generator-core-1.3.2-ztd.jar:?]	at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133) ~[mybatis-generator-core-1.3.2-ztd.jar:?]	at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:206) ~[mybatis-generator-core-1.3.2-ztd.jar:?]	at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:455) ~[mybatis-generator-core-1.3.2-ztd.jar:?]	at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:413) ~[mybatis-generator-core-1.3.2-ztd.jar:?]	at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:1034) ~[mybatis-generator-core-1.3.2-ztd.jar:?]	at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:194) ~[mybatis-generator-core-1.3.2-ztd.jar:?]	at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:791) ~[mybatis-generator-core-1.3.2-ztd.jar:?]	at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:866) ~[mybatis-generator-core-1.3.2-ztd.jar:?]	at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1186) ~[mybatis-generator-core-1.3.2-ztd.jar:?]	at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3387) ~[mybatis-generator-core-1.3.2-ztd.jar:?]	at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3431) ~[mybatis-generator-core-1.3.2-ztd.jar:?]	at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491) ~[mybatis-generator-core-1.3.2-ztd.jar:?]	at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_executeQuery(FilterChainImpl.java:2830) ~[druid-1.1.5.jar:1.1.5]	at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_executeQuery(FilterEventAdapter.java:465) ~[druid-1.1.5.jar:1.1.5]	at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_executeQuery(FilterChainImpl.java:2827) ~[druid-1.1.5.jar:1.1.5]	at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_executeQuery(FilterEventAdapter.java:465) ~[druid-1.1.5.jar:1.1.5]	at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_executeQuery(FilterChainImpl.java:2827) ~[druid-1.1.5.jar:1.1.5]	at com.alibaba.druid.proxy.jdbc.PreparedStatementProxyImpl.executeQuery(PreparedStatementProxyImpl.java:181) ~[druid-1.1.5.jar:1.1.5]	at com.alibaba.druid.pool.DruidPooledPreparedStatement.executeQuery(DruidPooledPreparedStatement.java:228) ~[druid-1.1.5.jar:1.1.5]	at com.baomidou.mybatisplus.plugins.PaginationInterceptor.count(PaginationInterceptor.java:189) ~[mybatis-plus-2.0.2.jar:?]	at com.baomidou.mybatisplus.plugins.PaginationInterceptor.intercept(PaginationInterceptor.java:156) ~[mybatis-plus-2.0.2.jar:?]	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61) ~[mybatis-3.4.2.jar:3.4.2]	at com.sun.proxy.$Proxy184.query(Unknown Source) ~[?:?]	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148) ~[mybatis-3.4.2.jar:3.4.2]	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433) ~[mybatis-spring-1.3.1.jar:1.3.1]	at com.sun.proxy.$Proxy57.selectList(Unknown Source) ~[?:?]	at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:238) ~[mybatis-spring-1.3.1.jar:1.3.1]	at org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:135) ~[mybatis-3.4.2.jar:3.4.2]	at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:75) ~[mybatis-3.4.2.jar:3.4.2]	at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:59) ~[mybatis-3.4.2.jar:3.4.2]	at com.sun.proxy.$Proxy131.findPage(Unknown Source) ~[?:?]	at com.zhitengda.dolphin.modular.service.impl.invoice.retailinvoice.TabRetailInvoiceServiceImpl.findPage(TabRetailInvoiceServiceImpl.java:83) ~[classes/:?]	at com.zhitengda.dolphin.modular.service.impl.invoice.retailinvoice.TabRetailInvoiceServiceImpl$$FastClassBySpringCGLIB$$2020b273.invoke(
   
    ) ~[classes/:?]	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:47) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:62) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:47) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at com.zhitengda.dolphin.modular.service.impl.invoice.retailinvoice.TabRetailInvoiceServiceImpl$$EnhancerBySpringCGLIB$$7183f17b.findPage(
    
     ) ~[classes/:?]	at com.zhitengda.dolphin.modular.service.impl.invoice.retailinvoice.TabRetailInvoiceServiceImpl$$FastClassBySpringCGLIB$$2020b273.invoke(
     
      ) ~[classes/:?]	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at com.zhitengda.dolphin.modular.service.impl.invoice.retailinvoice.TabRetailInvoiceServiceImpl$$EnhancerBySpringCGLIB$$62d14ac0.findPage(
      
       ) ~[classes/:?]	at com.zhitengda.dolphin.modular.service.impl.invoice.retailinvoice.TabRetailInvoiceServiceImpl$$FastClassBySpringCGLIB$$2020b273.invoke(
       
        ) ~[classes/:?]	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]	at com.zhitengda.dolphin.modular.service.impl.invoice.retailinvoice.TabRetailInvoiceServiceImpl$$EnhancerBySpringCGLIB$$62d14ac0.findPage(
        
         ) ~[classes/:?] at com.alibaba.dubbo.common.bytecode.Wrapper21.invokeMethod(Wrapper21.java) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:64) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:61) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:132) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:98) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:98) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52) ~[dubbo-2.5.7.jar:2.5.7] at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:81) ~[dubbo-2.5.7.jar:2.5.7] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
        
       
      
     
    
   

4、增加multiline配置如下

# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [  #不以[开头的行都合并到上一行的末尾    multiline.pattern: ^\[  # Defines if the pattern set under pattern should be negated or not. Default is false.  #默认是false，匹配pattern的行合并到上一行；true，不匹配pattern的行合并到上一行  multiline.negate: true  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern  # that was (not) matched before or after or as long as a pattern is not matched based on negate.  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash  #after 或 before，合并到上一行的末尾或开头  multiline.match: after

5、清除registry目录后重启filebeat，检查kibana数据

我们主要查看最后一个error命中内容，内容较长，展开如下：

数据采集正确。

下面贴上我的filebeat.yml文件

#=========================== Filebeat inputs =============================filebeat.inputs:- type: log  #开启监视，不开不采集  enabled: true  # 指定采集的路径，基于全局的路径（可用通配符，但不递归）.  paths:    #- /var/log/*.log   (linux例子)    #- c:\programdata\elasticsearch\logs\*  (windows 例子)    - D:\localinstall\ELK\data-log\*.log      # 排除行, 要匹配的正则表达式的列表. 它将从列表中删除与任何正则表达式匹配的行  #exclude_lines: ['^DBG']  # 包含行. 要匹配的正则表达式的列表. 它从列表中导出与任何正则表达式匹配的行  include_lines: ['[ERROR]']  # 排除文件. 要匹配的正则表达式的列表。 Filebeat 将从列表中删除与任何正则表达式匹配的文件。默认情况下, 不会删除任何文  #exclude_files: ['.gz$']  # 可选的附加字段。可以随意选取这些字段, 以便将附加信息添加到已爬网日志文件中以进行筛选  #fields:  #  level: debug  #  review: 1  ### 多行选项  # 可用于跨越多行的日志消息。这对于 Java 堆栈跟踪或 C 行继续很常见   # 必须匹配的 regexp 模式。示例模式匹配所有开始 [  multiline.pattern: ^\[  # 定义在模式下设置的模式是否应该被否定。默认值为 false.  multiline.negate: true  # 匹配可以设置为 "后" 或 "之前"。它用于定义是否应将行追加到模式 (不匹配) 之前或之后, 或者只要模式不匹配 (基于否定.  # Note: 后是等同于前和前是等价于下 Logstash  multiline.match: after#============================= Filebeat modules ===============================filebeat.config.modules:  # Glob pattern for configuration loading  path: ${path.config}/modules.d/*.yml  # 设置为true来启用配置重载  reload.enabled: false  # 检查路径下的文件更改的期间（多久检查一次）  #reload.period: 10s#==================== Elasticsearch template setting ==========================setup.template.settings:  index.number_of_shards: 1  #index.codec: best_compression  #_source.enabled: false#================================ General =====================================#tags: ["service-X", "web-tier"]#fields:#  env: staging#============================== Dashboards =====================================# 这些设置控制将示例仪表板加载到 Kibana 索引。默认情况下, 加载仪表板被禁用, 可以通过在此处设置选项, 或者使用 "安装" CLI 标志或 "安装" 命令来启用。setup.dashboards.enabled: false# 从何处下载仪表板存档的 URL。默认情况下, 此 URL 具有一个基于节拍名称和版本计算的值。对于发布的版本, 此 URL 指向 artifacts.elastic.co 网站上的仪表板存档.#setup.dashboards.url:#============================== Kibana =====================================# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.# This requires a Kibana endpoint configuration.setup.kibana:  host: "localhost:5601"  # Kibana Space ID  # ID of the Kibana Space into which the dashboards should be loaded. By default,  # the Default Space will be used.  #space.id:#================================ Outputs =====================================# Configure what output to use when sending the data collected by the beat.#-------------------------- Kafka output ------------------------------output.kafka:  # Array of hosts to connect to.  hosts: ["127.0.0.1:9092","127.0.0.1:9093","127.0.0.1:9094"]  topic: filebeat_log  enabled: true#-------------------------- Elasticsearch output ------------------------------#output.elasticsearch:  # Array of hosts to connect to.  #hosts: ["localhost:9200"]  # Optional protocol and basic auth credentials.  #protocol: "https"  #username: "elastic"  #password: "changeme"#----------------------------- Logstash output --------------------------------#output.logstash:  # The Logstash hosts  #hosts: ["localhost:5044"]  # Optional SSL. By default is off.  # List of root certificates for HTTPS server verifications  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]  # Certificate for SSL client authentication  #ssl.certificate: "/etc/pki/client/cert.pem"  # Client Certificate Key  #ssl.key: "/etc/pki/client/cert.key"#================================ Processors =====================================processors:  - add_host_metadata: ~  - add_cloud_metadata: ~  - add_docker_metadata: ~  - add_kubernetes_metadata: ~

转载地址：http://pcfsi.baihongyu.com/