Amazon Web Services Posted November 15, 2022 Share Posted November 15, 2022 Introduction Today, Amazon Web Services (AWS) announced the support for Fluent Bit container images for Windows operating system. This support eliminates the need for Windows customers to implement any custom logging solutions in their application code or manage custom agents on their Windows nodes to scrape the logs. For more details about the supported Windows versions and the image tags for Fluent Bit, please visit our documentation here. Fluent Bit is a fast and flexible log processor and router supported by various operating systems. It’s used to route logs to various AWS destinations, such as Amazon CloudWatch, Amazon Kinesis Data Firehose destinations, Amazon Simple Storage Service (Amazon S3), and Amazon OpenSearch. In addition to the AWS destinations, Fluent Bit supports common partner solutions such as Datadog, Splunk, custom HTTP servers, and many more. For more details about container log routing, please visit the blog link here. AWS already supports the Fluent Bit container images based on Linux operating system, and this release allows the customers to have a centralized mechanism for processing and routing their logs across Amazon Elastic Container Service (Amazon ECS) and Amazon Elastic Kubernetes Service (Amazon EKS) for both Linux and Windows workloads. In this post, we cover how Amazon EKS customers can deploy Fluent Bit Windows images as a DaemonSet on their Windows nodes to stream Internet Information Services (IIS) logs generated in the Windows pods to Amazon CloudWatch logs as a way to centralize logging. We’ll configure Fluent Bit to route all the logs generated by pods in different namespaces to respective CloudWatch log groups. Additionally, each log entry would be enriched with additional Kubernetes metadata such as namespace, pod name, container name, image name, host name, etc. For more details on deploying Fluent Bit on Amazon ECS Windows cluster, please visit our blog post here. Prerequisites Prerequisites and assumptions: Amazon EKS cluster (1.20 or newer) up and running. See this step by step guide. Launch Amazon EKS Windows worker nodes. See this step by step guide. You have properly installed and configured Amazon Command Line Interface (AWS CLI), eksctl, and kubectl. For building Windows container image, you have created Amazon Elastic Compute Cloud (Amazon EC2) Windows instance with Docker installed, which is based on the same Windows version as the Amazon EKS Windows worker nodes. Alternatively, you can also use Amazon EC2 Image Builder for building your container image. You have created Amazon Elastic Container Registry (Amazon ECR) repository to host Windows container image. See this step by step tutorial. Solution overview In this post, we’ll complete the following tasks: Validate the Windows worker nodes are up and running. Create Kubernetes namespace amazon-cloudwatch in which Fluent Bit runs. Configure the AWS Identity and Access Management (AWS IAM) policies required for enabling Fluent Bit to send the logs to required destinations. Create a config map with the required configuration. [Optional] Build a Windows container image containing IIS and LogMonitor. Deploy Fluent Bit on Windows node as a DaemonSet. Deploy Windows container image containing IIS and LogMonitor. [Optional] Access the IIS pods to generate logs. Checking logs on Amazon CloudWatch logs. Cleanup the created resources. Architecture of setup used in this blog post Walkthrough 1. Validate the Windows worker nodes are up and running To follow along in this tutorial, you must have an existing Amazon EKS cluster with Windows nodes running in it. To create the same, please follow our guide here. To check that Windows worker nodes are ready, run the following command. kubectl get nodes -o wide You can expect the following output: NAME STATUS ROLES AGE VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-192-168-24-199.us-west-2.compute.internal Ready <none> 13h v1.23.12-eks-d2d28e7 192.168.24.199 Windows Server 2019 Datacenter 10.0.17763.3532 docker://20.10.17 ip-192-168-47-140.us-west-2.compute.internal Ready <none> 13h v1.23.12-eks-d2d28e7 192.168.47.140 Windows Server 2019 Datacenter 10.0.17763.3532 containerd://1.6.6 ip-192-168-9-130.us-west-2.compute.internal Ready <none> 13h v1.23.9-eks-ba74326 192.168.9.130 Amazon Linux 2 5.4.209-116.367.amzn2.x86_64 docker://20.10.17 As you can see, we have two Windows Server 2019 nodes based on Amazon EKS version 1.23, which are using docker and containerd runtime, respectively. 2. Create Kubernetes namespace amazon-cloudwatch for Fluent Bit Kubernetes namespaces provide a scope for names and organize your workload inside the cluster. For consolidating all the Fluent Bit resources, we’ll create a namespace named amazon-cloudwatch. 2.1 Create a file with the following content and name it namespace.yaml: apiVersion: v1 kind: Namespace metadata: name: amazon-cloudwatch labels: name: amazon-cloudwatch 2.2 Run the following command to create the namespace: kubectl apply -f namespace.yaml 3. Configure the IAM policies required for enabling Fluent Bit to send the logs to required destinations Fluent Bit requires specific IAM permissions to send the logs to AWS destinations. The permissions required for a Fluent Bit output plugin are mentioned in the documentation for that specific output plugin. For example, we use Amazon CloudWatch as the log destination. The following IAM permission applies: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogStream", "logs:CreateLogGroup", "logs:PutLogEvents" ], "Resource": "*" } ] } You can also check for additional details in the official Fluent Bit documentation. Going by the principle of least privilege, instead of creating and distributing your AWS credentials to the containers or using the Amazon EC2 instance’s role, you can use IAM Roles for Service Accounts (IRSA) and configure your pods to use the Kubernetes service account. For more details, please refer to the documentation here. Alternatively, Fluent Bit can use the permissions from the IAM role attached to the Amazon EC2 instance. For more details on how to attach specific IAM policies when creating nodegroups using eksctl, please visit the schema documentation here. You can also attach the IAM policies once the nodegroup has been created. Please refer to documentation here. 3.1 To configure IAM roles for service accounts (i.e., IRSA) on Amazon EKS, we need to associate an IAM OpenID Connect (OIDC) provider to the Amazon EKS cluster. Check if your Amazon EKS cluster has an existing IAM OIDC provider by executing the following command: # Please replace <CLUSTER_NAME> and <REGION> with the actual values # before running the command. aws eks describe-cluster --name <CLUSTER_NAME> --region <REGION> --query "cluster.identity.oidc.issuer" --output text 3.2 If you need to create an IAM OIDC provider run the following command: # Please replace <CLUSTER_NAME> and <REGION> with the actual values # before running the command. eksctl utils associate-iam-oidc-provider --region <REGION> --cluster <CLUSTER_NAME> --approve 3.3 Create a file named fluent-bit-policy.json with the policy mentioned above. Run the following command to create the IAM policy. aws iam create-policy --policy-name fluent-bit-policy --policy-document file://fluent-bit-policy.json 3.4 Create the IAM service account and attach the policy previously created. Run the following command to create IAM service account using eksctl: # Please replace <CLUSTER_NAME>, <REGION>, and # <FLUENT_BIT_POLICY_ARN> with the actual values before running the commands. eksctl create iamserviceaccount --cluster <CLUSTER_NAME> \ --region <REGION> \ --attach-policy-arn <FLUENT_BIT_POLICY_ARN> \ --name fluent-bit-windows \ --namespace amazon-cloudwatch \ --approve In order to configure the same using AWS CLI, please refer to the documentation here. 4. Create a config map with the required configuration We need to provide few details to Fluent Bit to configure it. These include the cluster name and region. We accomplish the same using Config Maps. 4.1 Run the following to create a config map for providing configuration options to Fluent Bit: # Please replace <CLUSTER_NAME> and <REGION> with the actual values before running the commands. ClusterName=<CLUSTER_NAME> RegionName=<REGION> FluentBitReadFromHead='Off' kubectl create configmap fluent-bit-cluster-info \ --from-literal=cluster.name=${ClusterName} \ --from-literal=logs.region=${RegionName} \ --from-literal=read.head=${FluentBitReadFromHead} -n amazon-cloudwatch By default, Fluent Bit reads log files from the tail, and captures only new logs after it’s deployed. If you want the opposite, set FluentBitReadFromHead='On' and it collects all logs in the file system. 5. [Optional] Build a Windows container image containing IIS and LogMonitor If you already have a Windows container image for your application, please feel free to skip this section. 5.1 To test the functionality explained in this post, we create a Windows container image containing IIS and LogMonitor. By default, the Windows containers send logs to Event Tracing for Windows (ETW), Event Log, and custom logs files. However, log processors such as Fluent Bit and fluentd fetch containers logs from a STDOUT pipeline which doesn’t exist on Windows containers. LogMonitor is an open-source plugin created by Microsoft to create a STDOUT pipeline within the Windows container so such tools can successfully fetch logs in the same way they do on a Linux environment. For more instructions on how to use LogMonitor, access the official GitHub repository or Microsoft blog post about the same. In the example below, we have a Dockerfile build the Windows container image containing IIS and LogMonitor. FROM mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019 #Set powershell as default shell SHELL ["powershell", "-NoLogo", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"] #Add X-Forward-For Header to IIS Default website log RUN Add-WebConfigurationProperty -pspath 'MACHINE/WEBROOT/APPHOST' -filter "system.applicationHost/sites/siteDefaults/logFile/customFields" -name "." "-value @{logFieldName='X-Forwarded-For';sourceName='X-Forwarded-For';sourceType='RequestHeader'}" #Add STDOUT LogMonitor binary and config in json format COPY LogMonitor.exe LogMonitorConfig.json 'C:\LogMonitor\' WORKDIR /LogMonitor ENTRYPOINT ["C:\\LogMonitor\\LogMonitor.exe", "powershell.exe"] CMD ["C:\\ServiceMonitor.exe w3svc;"] LogMonitorConfig.json This sample LogMonitorConfig configuration retrieves all the logs files with the extension .log saved in C:\inetpub\logs and sub directories, including the IIS access logs. { "LogConfig": { "sources": [ { "type": "EventLog", "startAtOldestRecord": true, "eventFormatMultiLine": false, "channels": [ { "name": "system", "level": "Error" } ] }, { "type": "File", "directory": "c:\\inetpub\\logs", "filter": "*.log", "includeSubdirectories": true }, { "type": "ETW", "providers": [ { "providerName": "IIS: WWW Server", "ProviderGuid": "3A2A4E84-4C21-4981-AE10-3FDA0D9B0F83", "level": "Information" }, { "providerName": "Microsoft-Windows-IIS-Logging", "ProviderGuid": "7E8AD27F-B271-4EA2-A783-A47BDE29143B", "level": "Information", "keywords": "0xFF" } ] } ] } } As the build completes, push the image to your Amazon ECR registry. 6. Deploy Fluent Bit on Windows nodes as a DaemonSet A DaemonSet ensures that all (or some) nodes run a copy of a pod. As nodes are added to the cluster, pods are also added to them. In order to make sure that all Windows worker nodes have a copy of the Windows Fluent Bit pod, we deploy a DaemonSet using the deployment file specified in the following steps. 6.1 Copy the following manifest into a file named fluent-bit-daemon-set.yaml. --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: fluent-bit-windows-role namespace: amazon-cloudwatch rules: - nonResourceURLs: - /metrics verbs: - get - apiGroups: [""] resources: - namespaces - pods - pods/logs - nodes - nodes/proxy verbs: ["get", "list", "watch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit-windows-role-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: fluent-bit-windows-role subjects: # Assuming that the Service Account was created earlier with name fluent-bit-windows. - kind: ServiceAccount name: fluent-bit-windows namespace: amazon-cloudwatch --- apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-windows-config namespace: amazon-cloudwatch labels: k8s-app: fluent-bit-windows data: fluent-bit.conf: | [SERVICE] Flush 5 Log_Level info Daemon off net.dns.resolver LEGACY Parsers_File parsers.conf @INCLUDE application-log.conf application-log.conf: | [INPUT] Name tail Tag application.* Exclude_Path C:\\var\\log\\containers\\fluent-bit* Path C:\\var\\log\\containers\\*.log Docker_Mode On Docker_Mode_Flush 5 Docker_Mode_Parser container_firstline Parser docker DB C:\\var\\fluent-bit\\state\\flb_container.db Read_from_Head ${READ_FROM_HEAD} [INPUT] Name tail Tag application.* Path C:\\var\\log\\containers\\fluent-bit* Parser docker DB C:\\var\\fluent-bit\\state\\flb_log.db Read_from_Head ${READ_FROM_HEAD} [FILTER] Name kubernetes Match application.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Kube_Tag_Prefix application.C.var.log.container. Merge_Log On Merge_Log_Key log_processed K8S-Logging.Parser On K8S-Logging.Exclude Off Labels Off Annotations Off Use_Kubelet Off Buffer_Size 0 [OUTPUT] Name cloudwatch_logs Match application.*_default_* region ${AWS_REGION} log_group_name /aws/containerinsights/${CLUSTER_NAME}/default log_stream_prefix ${HOST_NAME}- auto_create_group true extra_user_agent container-insights [OUTPUT] Name cloudwatch_logs Match application.*_amazon-cloudwatch_* region ${AWS_REGION} log_group_name /aws/containerinsights/${CLUSTER_NAME}/amazon-cloudwatch log_stream_prefix ${HOST_NAME}- auto_create_group true extra_user_agent container-insights parsers.conf: | [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%LZ [PARSER] Name container_firstline Format regex Regex (?<log>(?<="log":")\S(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=}) Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%LZ --- apiVersion: apps/v1 kind: DaemonSet metadata: name: fluent-bit-windows namespace: amazon-cloudwatch labels: k8s-app: fluent-bit-windows version: v1 kubernetes.io/cluster-service: "true" spec: selector: matchLabels: k8s-app: fluent-bit-windows template: metadata: labels: k8s-app: fluent-bit-windows version: v1 kubernetes.io/cluster-service: "true" spec: containers: - name: fluent-bit-windows image: public.ecr.aws/aws-observability/aws-for-fluent-bit:windowservercore-latest imagePullPolicy: Always env: - name: AWS_REGION valueFrom: configMapKeyRef: name: fluent-bit-cluster-info key: logs.region - name: CLUSTER_NAME valueFrom: configMapKeyRef: name: fluent-bit-cluster-info key: cluster.name - name: HOST_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: READ_FROM_HEAD valueFrom: configMapKeyRef: name: fluent-bit-cluster-info key: read.head resources: limits: memory: 600Mi requests: cpu: 500m memory: 600Mi volumeMounts: # Only read only access to the following mounts is required - name: fluentbitstate mountPath: C:\var\fluent-bit\state - name: varlog mountPath: C:\var\log readOnly: true - name: varlibdockercontainers mountPath: C:\ProgramData\docker\containers readOnly: true - name: fluent-bit-config mountPath: C:\fluent-bit\etc\ readOnly: true terminationGracePeriodSeconds: 30 volumes: - name: fluentbitstate hostPath: path: C:\var - name: varlog hostPath: path: C:\var\log - name: varlibdockercontainers hostPath: path: C:\ProgramData\docker\containers - name: fluent-bit-config configMap: name: fluent-bit-windows-config nodeSelector: kubernetes.io/os: windows serviceAccountName: fluent-bit-windows Based on the above configuration, Fluent Bit would: Use tail output plugin to find new log entries which are appended to the container specific log files maintained by container runtime. Use Kubernetes Fluent Bit filter to append additional metadata to each log entry. This is done by querying the Kube-API server. Use Amazon CloudWatch log group ending with default for sending application logs of pods running in default Kubernetes namespace. Use Amazon CloudWatch log group ending with amazon-cloudwatch for sending logs generated by Fluent Bit pods running in amazon-cloudwatch Kubernetes namespace. The tag with each log entry is of the format application.C.var.log.container.<POD_NAME>_<NAMESPACE_NAME>_<CONTAINER_NAME>-<DOCKER_ID>. In the above example Fluent Bit configuration, we route logs based on namespaces using the regex in Match settings for output plugins. For more details about how Fluent Bit can be configured to use Tag and Match, please visit the documentation here. You can also visit our documentation here for instructions on customizing your Amazon CloudWatch Log Group or Stream with Kubernetes metadata. Note: In this post, we configured Kubernetes filter to use Kube API Server instead of kubelet endpoint for querying the additional metadata. This is because kubelet runs in host network namespace on Windows where Fluent Bit runs in the container network. We can use kubelet endpoint if the Fluent Bit pods are launched as HostProcess pods. 6.2 Deploy this manifest using the following command: kubectl apply -f fluent-bit-daemon-set.yaml 7. Deploy Windows container image containing IIS and LogMonitor In this step, we will deploy Windows pods on the Windows worker nodes. 7.1 Create a deployment file named windows_manifest.yaml with the following content: # Please replace <IIS_CONTAINER_IMAGE> with the image created in step 5.1 --- apiVersion: v1 kind: Service metadata: name: winiis labels: run: winiis spec: ports: - port: 80 protocol: TCP selector: run: winiis --- apiVersion: apps/v1 kind: Deployment metadata: labels: run: winiis name: winiis namespace: default spec: replicas: 2 revisionHistoryLimit: 10 selector: matchLabels: run: winiis strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: run: winiis spec: containers: - image: <IIS_CONTAINER_IMAGE> imagePullPolicy: IfNotPresent name: winiis dnsPolicy: ClusterFirst nodeSelector: kubernetes.io/os: windows restartPolicy: Always terminationGracePeriodSeconds: 30 7.2 Deploy the manifest using the following command: kubectl apply -f windows_manifest.yaml 8. [Optional] Access the IIS pods to generate logs This is an optional step. You can either wait for actual traffic to hit your Windows pod hosting the IIS webserver. Alternatively, you can log into your container and force the logs. 8.1 Run the following command to log into your container: kubectl -it exec <your_winiis_pod_name> powershell 8.2 From inside the container, run the following command to hit the web server: Invoke-WebRequest winiis -UseBasicParsing 9. Checking logs on Amazon CloudWatch logs In this step, we login into the Amazon CloudWatch console and observe the logs that were generated. The Amazon CloudWatch console can be accessed via the link here. Access Log Group console from side panel via Logs/Log Groups. You can expect log groups with name: /aws/containerinsights/<CLUSTER_NAME>/default /aws/containerinsights/<CLUSTER_NAME>/amazon-cloudwatch Each log group would contain logs from the pods deployed in respective Kubernetes namespaces. Cleaning up When you have finished the tutorial in this post, clean up the resources associated with it to avoid incurring charges for resources that you aren’t using: Delete the Windows deployment kubectl delete -f windows_manifest.yaml Delete the Fluent Bit DaemonSet kubectl delete -f fluent-bit-daemon-set.yaml Delete the log groups created by Fluent Bit aws logs delete-log-group —log-group-name /aws/containerinsights/<CLUSTER_NAME>/default aws logs delete-log-group --log-group-name /aws/containerinsights/<CLUSTER_NAME>/amazon-cloudwatch Delete the Windows container image created in step 5 Delete the Service Account and IAM policy created in step 3 Delete the amazon-cloudwatch namespace Conclusion In this post, we showed you how to deploy Fluent Bit as a DaemonSet on Windows worker nodes in Amazon EKS. Using Fluent Bit as a log router is a beneficial way to centralize logging across Amazon EKS for both Linux and Windows workloads. Fluent Bit can send the logs to various destination solutions allowing flexibility for customers. View the full article Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.