This topic gives an overview of the Fluent Bit package, which you can install in Tanzu Kubernetes Grid (TKG) workload clusters to provide log forwarding services for the cluster.
Fluent Bit is a fast, lightweight log processor and forwarder that lets you collect application data and logs from different sources, unify them, and send them to multiple destinations.
You can use Fluent Bit as a log forwarder for logs from a TKG cluster. You need to have a logging management server deployed for storing and analyzing logs. You can use one of the supported logging servers.
- Syslog
- HTTP
- Elastic Search
- Kafka
- Splunk
Installation: Install Fluent Bit on a workload cluster in one of the following ways, based on its deployment option:
See Also
Software-Oriented Architecture (SOA) Defined | SplunkData Mining: The Ultimate Introduction | SplunkWhat Cisco’s Splunk acquisition means for APAC customers | Computer WeeklySenior Software Development Engineer in Test (Sr. SDET) | Splunk-
TKG on Supervisor:
- Install Fluent Bit Using the Tanzu CLI
- Install Fluent Bit Using Kubectl
-
Standalone management cluster: Install Fluent Bit in Workload Clusters Deployed by a Standalone Management Cluster
Fluent Bit Components, Configuration, Data Values
The following sections describe Fluent Bit components and show how you can configure the Fluent Bit package.
Fluent Bit Components
The Fluent Bit package installs on the cluster the container listed in the table. For more information, see https://fluentbit.io/. The package pulls the container from the VMware public registry specified in Package Repository.
| Container | Resource Type | Replicas | Description |
|---|---|---|---|
| Fluent Bit | DaemonSet | 6 | Log collector, aggregator, forwarder |
Fluent Bit Data Values
The example provided with the installation can be used for Syslog. See Install Fluent Bit Using Kubectl.
The following example can be used for an HTTP logging endpoint.
... --- namespace: fluentbit-logging tkg: instance_name: "<TKG_INSTANCE_NAME>" cluster_name: "<CLUSTER_NAME>" fluentbit: pspNames: "vmware-system-restricted" output_plugin: "http" http: host: "<HTTP_HOST>" port: "<HTTP_PORT>" uri: "<URI>" header_key_value: "<HEADER_KEY_VALUE>" format: "json"The following example can be used for Elastic Search.
... --- namespace: fluentbit-logging tkg: instance_name: "<TKG_INSTANCE_NAME>" cluster_name: "<CLUSTER_NAME>" fluentbit: pspNames: "vmware-system-restricted" output_plugin: "elasticsearch" elasticsearch: host: "<ELASTIC_SEARCH_HOST>" port: "<ELASTIC_SEARCH_PORT>"The following example can be used for Kafka.
... --- namespace: fluentbit-logging tkg: instance_name: "<TKG_INSTANCE_NAME>" cluster_name: "<CLUSTER_NAME>" fluentbit: pspNames: "vmware-system-restricted" output_plugin: "kafka" kafka: broker_service_name: "<BROKER_SERVICE_NAME>" topic_name: "<TOPIC_NAME>"The following example can be used for Splunk.
... --- namespace: fluentbit-logging tkg: instance_name: "<TKG_INSTANCE_NAME>" cluster_name: "<CLUSTER_NAME>" fluentbit: pspNames: "vmware-system-restricted" output_plugin: "splunk" splunk: host: "<SPLUNK_HOST>" port: "<SPLUNK_PORT>" token: "<SPLUNK_TOKEN>"Fluent Bit Configuration Parameters (TKG on Supervisor)
The configuration values for are set in fluent-bit-data-values.yaml. The table lists and describes the available parameters.
| Parameter | Description | Type | Default |
|---|---|---|---|
| logging.namespace | Namespace where Fluent Bit will be deployed | string | tanzu-system-logging |
| logging.service_account_name | Name of Fluent Bit service account | string | fluent-bit |
| logging.cluster_role_name | Name of cluster role which grants get, watch and list permissions to fluent bit | string | fluent-bit-read |
| logging.image.name | Name of Fluent Bit image | string | fluent-bit |
| logging.image.tag | Fluent Bit image tag. This value may need to be updated if you are upgrading the version. | string | v1.6.9_vmware.1 |
| logging.image.repository | Location of the repository with the Fluent Bit image. The default is the public VMware registry. Change this value if you are using a private repository (e.g., air-gapped environment). | string | projects.registry.vmware.com/tkg |
| logging.image.pullPolicy | Fluent bit image pull policy | string | IfNotPresent |
| logging.update_strategy | Update strategy to be used when updating DaemonSet | string | RollingUpdate |
| tkg.cluster_name | Name of the Tanzu Kubernetes cluster | string | Null (Mandatory parameter) |
| tkg.instance_name | User-defined name of the TKG instance, shared by the Supervisor Cluster and all Tanzu Kubernetes clusters in one deployment. You can use any name related to the installation. | string | Null (Mandatory parameter)Note: This field is mandatory but arbitrary. It is a name that appears in the logs. |
| fluent_bit.log_level | Log level to use for Fluent Bit | string | info |
| fluent_bit.output_plugin | Set the backend to which Fluent Bit should flush the information it gathers | string | Null (Mandatory parameter) |
| fluent_bit.elasticsearch.host | IP address or hostname of the target Elasticsearch instance | string | Null (Mandatory parameter when output_plugin is elastic search) |
| fluent_bit.elasticsearch.port | TCP port of the target Elasticsearch instance | integer | Null (Mandatory parameter when output_plugin is elastic search) |
| fluent_bit.elasticsearch.buffer_size | Specify the buffer size used to read the response from Elasticsearch service. Sets to unlimited if False | string | False |
| fluent_bit.elasticsearch.tls | Specify the default setting for TLS for Elasticsearch | string | Off |
| fluent_bit.kafka.broker_service_name | Single of multiple list of Kafka Brokers, e.g., 192.168.1.3:9092 | string | Null (Mandatory parameter when output_plugin is kafka) |
| fluent_bit.kafka.topic_name | Single entry or list of topics separated by (,) that Fluent Bit will use to send messages to Kafka | string | Null (Mandatory parameter when output_plugin is kafka) |
| fluent_bit.splunk.host | IP address or hostname of the target Splunk Server | string | Null (Mandatory parameter when output_plugin is splunk) |
| fluent_bit.splunk.port | TCP port of the target Splunk Server | integer | Null (Mandatory parameter when output_plugin is splunk) |
| fluent_bit.splunk.token | Specify the Authentication Token for the HTTP Event Collector interface | string | Null (Mandatory parameter when output_plugin is splunk) |
| fluent_bit.http.host | IP address or hostname of the target HTTP Server | string | Null (Mandatory parameter when output_plugin is http) |
| fluent_bit.http.port | TCP port of the target HTTP Server | integer | Null (Mandatory parameter when output_plugin is http) |
| fluent_bit.http.mode | Specify an HTTP URI for the target web server | string | Null (Mandatory parameter when output_plugin is http) |
| fluent_bit.http.header_key_value | HTTP header key/value pair. Multiple headers can be set | string | Null (Mandatory parameter when output_plugin is http) |
| fluent_bit.http.format | Specify the data format to be used in the HTTP request body | string | Null (Mandatory parameter when output_plugin is http) |
| fluent_bit.syslog.host | Domain or IP address of the remote Syslog server | string | Null (Mandatory parameter when output_plugin is syslog) |
| fluent_bit.syslog.port | TCP or UDP port of the remote Syslog server | integer | Null (Mandatory parameter when output_plugin is syslog) |
| fluent_bit.syslog.mode | Specify the transport type from TCP, UDP and TLS | string | Null (Mandatory parameter when output_plugin is syslog) |
| fluent_bit.syslog.format | Specify the data format to be used in the HTTP request body | string | Null ( Mandatory parameter when output_plugin is syslog) |
| host_path.volume_1 | Directory path from the host node’s file system into the pod, for volume 1 | string | /var/log |
| host_path.volume_2 | Directory path from the host node’s file system into the pod, for volume 2 | string | /var/lib/docker/containers |
| host_path.volume_3 | Directory path from the host node’s file system into the pod, for volume 3 | string | /run/log |
| systemd.path | Path to the Systemd journal directory | string | /var/log/journal |
Fluent Bit Configuration Parameters (Standalone MC)
The table below lists configuration parameters of the Fluent Bit package and describes their default values. You can set the following configuration values in your fluent-bit-data-values.yaml file created in Deploy Fluent Bit on a Cluster.
| Parameter | Description | Type | Default |
|---|---|---|---|
namespace | Namespace where Fluent Bit will be deployed. | String | tanzu-system-logging |
fluent_bit.config.outputs | For information about the configuration for Fluent Bit outputs, see the Fluent Bit documentation. | Multiline YAML | Standard output |
fluent_bit.config.parsers | For information about the configuration for Fluent Bit parsers, see the Fluent Bit documentation. | Multiline YAML | JSON parser |
fluent_bit.config.plugins | Content for Fluent Bit plugins configuration. | String | <nil> |
fluent_bit.config.service | For information about the configuration for Fluent Bit service, see the Fluent Bit documentation. | Multiline YAML | Default Fluent Bit service config. |
fluent_bit.config.streams | Content for Fluent Bit streams file. | String | <nil> |
fluent_bit.config.filters | For information about the configuration for Fluent Bit filters, see the Fluent Bit documentation. | Multiline YAML | Default Kubernetes filter. |
fluent_bit.config.inputs | For information about the configuration for Fluent Bit inputs, see the Fluent Bit documentation. | String | Ingest Kubernetes container logs using the tail plugin and ingest systemd logs from kubelet. |
fluent_bit.daemonset.resources | For information about the configuration for Fluent Bit containers resource requirements, see the Fluent Bit documentation. | Multiline YAML | <nil> |
fluent_bit.daemonset.podAnnotations | The Fluent Bit deployments pod annotations. | List | <nil> |
fluent_bit.daemonset.podLabels | The Fluent Bit deployments pod labels. | List | <nil> |
