Simplifying Helm Configurations For

July 09, 2026 — LiveStream

Deploying Grafana using Helm can be complex, especially when managing the configuration of the Kubernetes service. Ensuring your configurations are accurate is crucial for the security and functionality of your deployment. This guide will help you understand the key components of both old and new configurations, highlight common mistakes, and ensure a smooth transition to the new configuration.

The AWS EKS Load Balancer Controller (AWS LB Controller) can sometimes change settings due to several reasons related to its management of Elastic Load Balancers (ELBs) and Network Load Balancers (NLBs) within an Amazon EKS cluster. Here are some common reasons why settings might be changed:

1. Automatic Updates: The AWS LB Controller automatically manages and updates load balancer configurations based on changes in the Kubernetes resources (like Services and Ingresses) or changes in associated annotations and configurations.
2. Annotation Changes: Annotations on Kubernetes resources (such as Services or Ingresses) can trigger updates or changes in the load balancer configuration. For example, modifying annotations related to SSL certificates, health checks, or backend protocols can lead to corresponding changes in the AWS load balancer settings.
3. Resource Lifecycle Events: Events such as scaling events (scaling up or down pods or nodes), changes in service configurations (like changing service types or annotations), or updates to the Kubernetes resources that affect load balancer behavior can cause the AWS LB Controller to adjust settings to maintain consistency or reflect the updated configurations.
4. Integration with AWS Services: The AWS LB Controller integrates closely with AWS services like ACM (AWS Certificate Manager) for SSL certificates, IAM (Identity and Access Management) for permissions, and CloudFormation for resource provisioning. Changes or updates in these AWS services or their configurations can influence how the AWS LB Controller manages and updates load balancer settings.
5. Policy and Compliance: AWS LB Controller might enforce certain policies or best practices by automatically adjusting settings. For example, ensuring that SSL certificates are correctly configured and up-to-date, enforcing health check configurations, or adhering to AWS security group rules.
6. Bug Fixes and Enhancements: Updates to the AWS LB Controller itself, including bug fixes, security patches, or enhancements, may result in changes to load balancer settings to improve stability, security, or performance.
7. Operator Intervention: In some cases, manual interventions by administrators or operators to change load balancer configurations via Kubernetes resources or directly through AWS APIs can lead to changes managed by the AWS LB Controller.

Understanding these reasons helps in troubleshooting and managing configurations effectively when using the AWS LB Controller with Amazon EKS. It's essential to monitor changes and understand how annotations, Kubernetes resource configurations, and AWS service integrations impact load balancer settings to maintain the desired operational state and security posture.

When setting up Grafana with Helm, the old configuration involved several critical settings to establish a LoadBalancer service. Here’s a detailed look at the old configuration:

service:
  enabled: true
  type: LoadBalancer
  loadBalancerIP: ""
  loadBalancerClass: ""

  loadBalancerAnnotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-2:REDACTED:certificate/REDACTED"  # Replace Me******
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"  # Expose port 443 (HTTPS)
    service.beta.kubernetes.io/aws-load-balancer-ssl-protocol: "TLS"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"  # Forward traffic to port 80 internally
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "80"  # Health check on port 80
    service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-REDACTED"  # Replace Me******
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "grafana-service=true"
    service.beta.kubernetes.io/aws-load-balancer-security-policy: "ELBSecurityPolicy-TLS-1-2-2017-01"
  labels: {}
  portName: service
  appProtocol: ""

Key Components of the Old Configuration

1. enabled: Activates the service.
2. type: Specifies the service type as LoadBalancer.
3. loadBalancerIP and loadBalancerClass: Initially left empty, these can be set for specific IP addresses or classes.
4. loadBalancerSourceRanges: Defines the IP ranges allowed to access the service.
5. loadBalancerAnnotations: Contains various AWS Load Balancer settings such as SSL certificate ARN, scheme, ports, protocols, security groups, health checks, and resource tags.

Common Mistakes in the Old Configuration

1. Incorrect SSL Certificate ARN: Not replacing placeholder values.
2. Improper Security Groups: Failing to update security group IDs.
3. Misconfigured Source Ranges: Allowing unintended IP ranges access.

These mistakes can lead to security vulnerabilities and connectivity issues, making it essential to handle these configurations with care.

Transitioning to the New Configuration

The new configuration aims to streamline and improve the management of the LoadBalancer service. Here's the updated configuration:

service:
  enabled: true
  type: LoadBalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-2:REDACTED:certificate/REDACTED"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    service.beta.kubernetes.io/aws-load-balancer-ssl-protocol: "HTTPS"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
    service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-REDACTED"
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "grafana-service=true"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: "tcp"
  labels: {}
  ports:
    - name: http
      port: 80
      targetPort: 3000
      protocol: TCP
    - name: https
      port: 443
      targetPort: 3000
      protocol: TCP
extraExposePorts:
  - name: https
    port: 443
    targetPort: 3000
healthCheck:
  path: "/"  # Health check path for ALB to monitor Grafana health
  intervalSeconds: 10
  timeoutSeconds: 5
  unhealthyThresholdCount: 6
  healthyThresholdCount: 2
externalTrafficPolicy: Cluster  # Use Cluster to preserve client source IP

Key Improvements in the New Configuration

1. annotations: Consolidates LoadBalancer settings into annotations, making the configuration cleaner.
2. ports: Defines both HTTP and HTTPS ports, specifying target ports and protocols.
3. extraExposePorts: Ensures HTTPS traffic is correctly routed to the appropriate target port.
4. healthCheck: Configures detailed health check settings for better monitoring.
5. externalTrafficPolicy: Uses the Cluster policy to preserve the client source IP.

Benefits of the New Configuration

• Simplified Management: Consolidated settings reduce complexity.
• Enhanced Security: Explicit annotations improve clarity and reduce the risk of misconfiguration.
• Better Monitoring: Detailed health check settings allow for more effective monitoring of Grafana's health.

Conclusion

Transitioning from the old to the new configuration for Grafana deployment using Helm can significantly enhance your deployment’s security and manageability. By understanding and correctly implementing these configurations, you can avoid common pitfalls and ensure a robust and efficient setup.

Remember, always double-check sensitive data like SSL certificates and security group IDs to avoid common mistakes. With this guide, you’re now better equipped to handle these configurations and ensure a smooth and secure Grafana deployment.

https://github.com/grafana/helm-charts/issues/1251#issuecomment-2167881691

##
  
 ################################################################################################################
service:
  enabled: true
  
  type: LoadBalancer
  
  annotations:
  
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-2:851465799644:certificate/f1"
  
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
  
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
  
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
  
    service.beta.kubernetes.io/aws-load-balancer-ssl-protocol: "HTTPS"
  
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
  
    service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-0bxxxxx"
  
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "grafana-service=true"
  
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: "tcp"
  
  labels: {}
  
  ports:
  
    - name: http
  
      port: 80
  
      targetPort: 3000
  
      protocol: TCP
  
    - name: https
  
      port: 443
  
      targetPort: 3000
  
       protocol: TCP
extraExposePorts:
  - name: https
  
    port: 443
  
    targetPort: 3000
healthCheck:
  path: "/"  # Health check path for ALB to monitor Grafana health
  
  intervalSeconds: 10
  
  timeoutSeconds: 5
  
  unhealthyThresholdCount: 6
  
  healthyThresholdCount: 2
externalTrafficPolicy: Cluster  # Use Cluster to preserve client source IP

Let’s break down each section of the Helm chart configuration for the service component:

1. service Section

service:
  enabled: true
  type: LoadBalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-2:851465799644:certificate/f1"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    service.beta.kubernetes.io/aws-load-balancer-ssl-protocol: "HTTPS"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
    service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-0bxxxxx"
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "grafana-service=true"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: "tcp"
  labels: {}

• enabled: true: Indicates that the service should be enabled.
• type: LoadBalancer: Specifies that this service should be exposed externally using an AWS Elastic Load Balancer (ELB) configured by Kubernetes.
• annotations: This section provides specific instructions to Kubernetes on how to configure the AWS ELB associated with this service:
  • service.beta.kubernetes.io/aws-load-balancer-ssl-cert: Specifies the ACM (AWS Certificate Manager) ARN for the SSL certificate used for HTTPS.
  • service.beta.kubernetes.io/aws-load-balancer-scheme: Specifies whether the ELB should be internet-facing or internal.
  • service.beta.kubernetes.io/aws-load-balancer-ssl-ports: Specifies the ports on which SSL should be enabled.
  • service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: Enables the Proxy Protocol to pass the client's information to the backend service.
  • service.beta.kubernetes.io/aws-load-balancer-ssl-protocol: Specifies the SSL protocol to be used (HTTPS).
  • service.beta.kubernetes.io/aws-load-balancer-backend-protocol: Specifies the protocol used between the ELB and the backend service (HTTP).
  • service.beta.kubernetes.io/aws-load-balancer-security-groups: Specifies the AWS security group to associate with the ELB.
  • service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: Adds additional AWS tags to the ELB for easier management.
  • service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: Specifies the protocol used for health checks by the ELB (TCP).
• labels: {}: This section allows you to specify Kubernetes labels for the service, which can be used for querying and selecting objects.

2. ports Section

ports:
  - name: http
    port: 80
    targetPort: 3000
    protocol: TCP
  - name: https
    port: 443
    targetPort: 3000
    protocol: TCP

• ports: Defines the ports that the service exposes:
  • name: Friendly name for the port.
  • port: Port number exposed on the ELB.
  • targetPort: Port on the container where traffic is sent (in this case, Grafana's port).
  • protocol: Protocol used (TCP in this case).

3. extraExposePorts Section

extraExposePorts:
  - name: https
    port: 443
    targetPort: 3000

• extraExposePorts: Additional ports to expose beyond those defined in the ports section. In this case, it reiterates the HTTPS configuration, specifying port 443 to target Grafana's port 3000.

4. healthCheck Section

healthCheck:
  path: "/"
  intervalSeconds: 10
  timeoutSeconds: 5
  unhealthyThresholdCount: 6
  healthyThresholdCount: 2

• healthCheck: Configures health checks for the AWS ELB:
  • path: The endpoint path used for health checks (root path "/" in this case).
  • intervalSeconds: Interval between health checks.
  • timeoutSeconds: Timeout for each health check.
  • unhealthyThresholdCount: Number of consecutive health check failures before considering the instance unhealthy.
  • healthyThresholdCount: Number of consecutive health check successes before considering the instance healthy again.

5. externalTrafficPolicy Section

externalTrafficPolicy: Cluster

• externalTrafficPolicy: Specifies how the external traffic is routed to the service:
  • Cluster: Preserves the client source IP, ensuring that the source IP is maintained when the traffic hits the backend service.

Summary

This Helm chart configuration snippet effectively sets up a Grafana service on AWS EKS with an AWS ELB configured for HTTPS, SSL termination, health checks, and traffic preservation policies. Each parameter plays a crucial role in ensuring the secure and reliable operation of Grafana within a Kubernetes cluster on AWS.

This breakdown provides insights into configuring AWS EKS load balancer settings via Helm for deploying Grafana securely with HTTPS support.

Grafana Helm Chart

• Installs the web dashboarding system Grafana

Get Repo Info

 helm repo add grafana https://grafana.github.io/helm-charts
 helm repo update

_See helm repo for command documentation._

Installing the Chart

To install the chart with the release name my-release:

 helm install my-release grafana/grafana

Uninstalling the Chart

To uninstall/delete the my-release deployment:

 helm delete my-release

The command removes all the Kubernetes components associated with the chart and deletes the release.

Upgrading an existing Release to a new major version

A major chart version change (like v1.2.3 -> v2.0.0) indicates that there is an

incompatible breaking change needing manual actions.

To 4.0.0 (And 3.12.1)

This version requires Helm >= 2.12.0.

To 5.0.0

You have to add --force to your helm upgrade command as the labels of the chart have changed.

To 6.0.0

This version requires Helm >= 3.1.0.

To 7.0.0

For consistency with other Helm charts, the global.image.registry parameter was renamed

to global.imageRegistry. If you were not previously setting global.image.registry, no action

is required on upgrade. If you were previously setting global.image.registry, you will

need to instead set global.imageRegistry.

Configuration

Example ingress with path

With grafana 6.3 and above

grafana.ini:
  server:
     domain: monitoring.example.com
    root_url: "%(protocol)s://%(domain)s/grafana"
    serve_from_sub_path: true
 ingress:
  enabled: true
  hosts:
    - "monitoring.example.com"
  path: "/grafana"

Example of extraVolumeMounts and extraVolumes

Configure additional volumes with extraVolumes and volume mounts with extraVolumeMounts.

Example for extraVolumeMounts and corresponding extraVolumes:

extraVolumeMounts:
  - name: plugins
    mountPath: /var/lib/grafana/plugins
    subPath: configs/grafana/plugins
    readOnly: false
  - name: dashboards
    mountPath: /var/lib/grafana/dashboards
    hostPath: /usr/shared/grafana/dashboards
    readOnly: false
extraVolumes:
  - name: plugins
    existingClaim: existing-grafana-claim
  - name: dashboards
    hostPath: /usr/shared/grafana/dashboards

Volumes default to emptyDir. Set to persistentVolumeClaim,

hostPath, csi, or configMap for other types. For a

persistentVolumeClaim, specify an existing claim name with

existingClaim.

Import dashboards

There are a few methods to import dashboards to Grafana. Below are some examples and explanations as to how to use each method:

 dashboards:
  default:
    some-dashboard:
      json: |
        {
          "annotations":
          ...
          # Complete json file here
          ...
          "title": "Some Dashboard",
          "uid": "abcd1234",
           "version": 1
        }
    custom-dashboard:
      # This is a path to a file inside the dashboards directory inside the chart directory
      file: dashboards/custom-dashboard.json
    prometheus-stats:
      # Ref: https://grafana.com/dashboards/2
       gnetId: 2
      revision: 2
      datasource: Prometheus
    loki-dashboard-quick-search:
      gnetId: 12019
      revision: 2
      datasource:
      - name: DS_PROMETHEUS
        value: Prometheus
      - name: DS_LOKI
        value: Loki
    local-dashboard:
      url: https://raw.githubusercontent.com/user/repository/master/dashboards/dashboard.json

BASE64 dashboards

Dashboards could be stored on a server that does not return JSON directly and instead of it returns a Base64 encoded file (e.g. Gerrit)

A new parameter has been added to the url use case so if you specify a b64content value equals to true after the url entry a Base64 decoding is applied before save the file to disk.

If this entry is not set or is equals to false not decoding is applied to the file before saving it to disk.

Gerrit use case

Gerrit API for download files has the following schema: <https://yourgerritserver/a/{project-name}/branches/{branch-id}/files/{file-id}/content> where {project-name} and

{file-id} usually has '/' in their values and so they MUST be replaced by %2F so if project-name is user/repo, branch-id is master and file-id is equals to dir1/dir2/dashboard

the url value is <https://yourgerritserver/a/user%2Frepo/branches/master/files/dir1%2Fdir2%2Fdashboard/content>

Sidecar for dashboards

If the parameter sidecar.dashboards.enabled is set, a sidecar container is deployed in the grafana

pod. This container watches all configmaps (or secrets) in the cluster and filters out the ones with

a label as defined in sidecar.dashboards.label. The files defined in those configmaps are written

to a folder and accessed by grafana. Changes to the configmaps are monitored and the imported

dashboards are deleted/updated.

A recommendation is to use one configmap per dashboard, as a reduction of multiple dashboards inside

one configmap is currently not properly mirrored in grafana.

Example dashboard config:

apiVersion: v1
 kind: ConfigMap
 metadata:
  name: sample-grafana-dashboard
  labels:
     grafana_dashboard: "1"
 data:
  k8s-dashboard.json: |-
  [...]

Sidecar for datasources

If the parameter sidecar.datasources.enabled is set, an init container is deployed in the grafana

pod. This container lists all secrets (or configmaps, though not recommended) in the cluster and

filters out the ones with a label as defined in sidecar.datasources.label. The files defined in

those secrets are written to a folder and accessed by grafana on startup. Using these yaml files,

the data sources in grafana can be imported.

Should you aim for reloading datasources in Grafana each time the config is changed, set sidecar.datasources.skipReload: false and adjust sidecar.datasources.reloadURL to http://<svc-name>.<namespace>.svc.cluster.local/api/admin/provisioning/datasources/reload.

Secrets are recommended over configmaps for this usecase because datasources usually contain private

data like usernames and passwords. Secrets are the more appropriate cluster resource to manage those.

Example values to add a postgres datasource as a kubernetes secret:

apiVersion: v1
 kind: Secret
metadata:
  name: grafana-datasources
  labels:
    grafana_datasource: 'true' # default value for: sidecar.datasources.label
stringData:
  pg-db.yaml: |-
    apiVersion: 1
    datasources:
       - name: My pg db datasource
        type: postgres
        url: my-postgresql-db:5432
        user: db-readonly-user
        secureJsonData:
          password: 'SUperSEcretPa$$word'
         jsonData:
          database: my_datase
          sslmode: 'disable' # disable/require/verify-ca/verify-full
          maxOpenConns: 0 # Grafana v5.4+
          maxIdleConns: 2 # Grafana v5.4+
          connMaxLifetime: 14400 # Grafana v5.4+
          postgresVersion: 1000 # 903=9.3, 904=9.4, 905=9.5, 906=9.6, 1000=10
          timescaledb: false
        # <bool> allow users to edit datasources from the UI.
        editable: false

Example values to add a datasource adapted from Grafana:

datasources:
  datasources.yaml:
   apiVersion: 1
   datasources:
      # <string, required> name of the datasource. Required
    - name: Graphite
      # <string, required> datasource type. Required
      type: graphite
      # <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
      access: proxy
      # <int> org id. will default to orgId 1 if not specified
      orgId: 1
      # <string> url
      url: http://localhost:8080
      # <string> database password, if used
      password:
      # <string> database user, if used
      user:
      # <string> database name, if used
       database:
      # <bool> enable/disable basic auth
      basicAuth:
      # <string> basic auth username
      basicAuthUser:
      # <string> basic auth password
      basicAuthPassword:
      # <bool> enable/disable with credentials headers
      withCredentials:
      # <bool> mark as default datasource. Max one per org
      isDefault:
      # <map> fields that will be converted to json and stored in json_data
      jsonData:
         graphiteVersion: "1.1"
         tlsAuth: true
         tlsAuthWithCACert: true
      # <string> json object of data that will be encrypted.
      secureJsonData:
        tlsCACert: "..."
         tlsClientCert: "..."
        tlsClientKey: "..."
      version: 1
      # <bool> allow users to edit datasources from the UI.
      editable: false

Sidecar for notifiers

If the parameter sidecar.notifiers.enabled is set, an init container is deployed in the grafana

pod. This container lists all secrets (or configmaps, though not recommended) in the cluster and

filters out the ones with a label as defined in sidecar.notifiers.label. The files defined in

those secrets are written to a folder and accessed by grafana on startup. Using these yaml files,

the notification channels in grafana can be imported. The secrets must be created before

helm install so that the notifiers init container can list the secrets.

Secrets are recommended over configmaps for this usecase because alert notification channels usually contain

private data like SMTP usernames and passwords. Secrets are the more appropriate cluster resource to manage those.

Example datasource config adapted from Grafana:

 notifiers:
  - name: notification-channel-1
    type: slack
    uid: notifier1
    # either
    org_id: 2
    # or
    org_name: Main Org.
    is_default: true
    send_reminder: true
    frequency: 1h
    disable_resolve_message: false
    # See `Supported Settings` section for settings supporter for each
    # alert notification type.
    settings:
      recipient: 'XXX'
      token: 'xoxb'
      uploadImage: true
      url: https://slack.com
delete_notifiers:
  - name: notification-channel-1
    uid: notifier1
    org_id: 2
  - name: notification-channel-2
    # default org_id: 1

Sidecar for alerting resources

If the parameter sidecar.alerts.enabled is set, a sidecar container is deployed in the grafana

pod. This container watches all configmaps (or secrets) in the cluster (namespace defined by sidecar.alerts.searchNamespace) and filters out the ones with

a label as defined in sidecar.alerts.label (default is grafana_alert). The files defined in those configmaps are written

to a folder and accessed by grafana. Changes to the configmaps are monitored and the imported alerting resources are updated, however, deletions are a little more complicated (see below).

This sidecar can be used to provision alert rules, contact points, notification policies, notification templates and mute timings as shown in Grafana Documentation.

To fetch the alert config which will be provisioned, use the alert provisioning API (Grafana Documentation).

You can use either JSON or YAML format.

Example config for an alert rule:

apiVersion: v1
 kind: ConfigMap
 metadata:
  name: sample-grafana-alert
  labels:
     grafana_alert: "1"
 data:
  k8s-alert.yml: |-
    apiVersion: 1
    groups:
        - orgId: 1
          name: k8s-alert
          [...]

To delete provisioned alert rules is a two step process, you need to delete the configmap which defined the alert rule

and then create a configuration which deletes the alert rule.

Example deletion configuration:

apiVersion: v1
 kind: ConfigMap
metadata:
  name: delete-sample-grafana-alert
  namespace: monitoring
  labels:
    grafana_alert: "1"
 data:
  delete-k8s-alert.yml: |-
    apiVersion: 1
    deleteRules:
      - orgId: 1
         uid: 16624780-6564-45dc-825c-8bded4ad92d3

Statically provision alerting resources

If you don't need to change alerting resources (alert rules, contact points, notification policies and notification templates) regularly you could use the alerting config option instead of the sidecar option above.

This will grab the alerting config and apply it statically at build time for the helm file.

There are two methods to statically provision alerting configuration in Grafana. Below are some examples and explanations as to how to use each method:

 alerting:
  team1-alert-rules.yaml:
    file: alerting/team1/rules.yaml
  team2-alert-rules.yaml:
    file: alerting/team2/rules.yaml
  team3-alert-rules.yaml:
    file: alerting/team3/rules.yaml
  notification-policies.yaml:
    file: alerting/shared/notification-policies.yaml
  notification-templates.yaml:
    file: alerting/shared/notification-templates.yaml
  contactpoints.yaml:
    apiVersion: 1
    contactPoints:
      - orgId: 1
        name: Slack channel
        receivers:
          - uid: default-receiver
             type: slack
            settings:
              # Webhook URL to be filled in
              url: ""
              # We need to escape double curly braces for the tpl function.
               text: '{{ `{{ template "default.message" . }}` }}'
               title: '{{ `{{ template "default.title" . }}` }}'

The two possibilities for static alerting resource provisioning are:

• Inlining the file contents as shown for contact points in the above example.
• Importing a file using a relative path starting from the chart root directory as shown for the alert rules in the above example.

Important notes on file provisioning

• The format of the files is defined in the Grafana documentation on file provisioning.
• The chart supports importing YAML and JSON files.
• The filename must be unique, otherwise one volume mount will overwrite the other.
• In case of inlining, double curly braces that arise from the Grafana configuration format and are not intended as templates for the chart must be escaped.
• The number of total files under alerting: is not limited. Each file will end up as a volume mount in the corresponding provisioning folder of the deployed Grafana instance.
• The file size for each import is limited by what the function .Files.Get can handle, which suffices for most cases.

How to serve Grafana with a path prefix (/grafana)

In order to serve Grafana with a prefix (e.g., <http://example.com/grafana>), add the following to your values.yaml.

 ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/use-regex: "true"
  path: /grafana/?(.*)
  hosts:
    - k8s.example.dev
grafana.ini:
  server:
    root_url: http://localhost:3000/grafana # this host can be localhost

How to securely reference secrets in grafana.ini

This example uses Grafana file providers for secret values and the extraSecretMounts configuration flag (Additional grafana server secret mounts) to mount the secrets.

In grafana.ini:

grafana.ini:
  [auth.generic_oauth]
  enabled = true
  client_id = $__file{/etc/secrets/auth_generic_oauth/client_id}
  client_secret = $__file{/etc/secrets/auth_generic_oauth/client_secret}

Existing secret, or created along with helm:

 ---
apiVersion: v1
 kind: Secret
 metadata:
  name: auth-generic-oauth-secret
 type: Opaque
stringData:
  client_id: <value>
  client_secret: <value>

Include in the extraSecretMounts configuration flag:

 - extraSecretMounts:
  - name: auth-generic-oauth-secret-mount
    secretName: auth-generic-oauth-secret
    defaultMode: 0440
    mountPath: /etc/secrets/auth_generic_oauth
    readOnly: true

extraSecretMounts using a Container Storage Interface (CSI) provider

This example uses a CSI driver e.g. retrieving secrets using Azure Key Vault Provider

 - extraSecretMounts:
  - name: secrets-store-inline
    mountPath: /run/secrets
    readOnly: true
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "my-provider"
      nodePublishSecretRef:
        name: akv-creds

Image Renderer Plug-In

This chart supports enabling remote image rendering

imageRenderer:
  enabled: true

Image Renderer NetworkPolicy

By default the image-renderer pods will have a network policy which only allows ingress traffic from the created grafana instance

High Availability for unified alerting

If you want to run Grafana in a high availability cluster you need to enable

the headless service by setting headlessService: true in your values.yaml

file.

As next step you have to setup the grafana.ini in your values.yaml in a way

that it will make use of the headless service to obtain all the IPs of the

cluster. You should replace {{ Name }} with the name of your helm deployment.

grafana.ini:
  ...
  unified_alerting:
    enabled: true
    ha_peers: {{ Name }}-headless:9094
    ha_listen_address: ${POD_IP}:9094
    ha_advertise_address: ${POD_IP}:9094
  alerting:
    enabled: false