Configure persistent storage for anomaly detection

note

This feature is tech preview. Tech preview features may be subject to significant changes before they become GA.

Big picture

Configure persistent storage for training models created by anomaly detection. By default, training model storage is ephemeral.

Concepts

Before configuring a storage class for anomaly detection, the following terms will help you understand storage interactions.

Persistent volume

Used by pods to persist storage within the cluster. Combined with persistent volume claims, pods can persist data across restarts and rescheduling.

Persistent volume claim

Used by pods to request and mount storage volumes. The claim specifies the volume requirements for the request: size, access rights, and storage class.

Dynamic provisioner

Provisions types of persistent volumes on demand. Although most managed public-cloud clusters provide a dynamic provisioner using cloud-specific storage APIs (for example, Amazon EBS or Google persistent disks), not all clusters have a dynamic provisioner.

When a pod makes a persistent volume claim from a storage class that uses a dynamic provisioner, the volume is automatically created. If the storage class does not use a dynamic provisioner (for example the local storage class), the volumes must be created in advance. For help, see the Kubernetes documentation.

Storage class

The storage provided by the cluster. Storage classes can be used with dynamic provisioners to automatically provision persistent volumes on demand, or with manually-provisioned persistent volumes. Different storage classes provide different service levels.

Before you begin

Storage size requirements

Anomaly detection requires 10Gi for storage. Note that training models are relatively small (~2MB), so you should not need more than this.

Determine storage support

Determine the storage types that are available on your cluster. If you are using dynamic provisioning, verify it is supported. If you are using local disks, you may find the sig-storage local static provisioner useful. It creates and manages PersistentVolumes by watching for disks mounted in a configured directory.

caution

Do not use the host path storage provisioner. This provisioner is not suitable for production and results in scalability issues, instability, and data loss.

How to

Create a storage class

Create a storage class named, tigera-anomaly-detection.

Examples

Pre-provisioned local disks

In the following example, we create a StorageClass to use when explicitly adding PersistentVolumes for local disks. This can be performed manually, or using the sig-storage local static provisioner.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: tigera-anomaly-detection
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain

AWS EBS disks

In the following example for an AWS cloud provider integration, the StorageClass tells Calico Enterprise to use EBS disks for log storage.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: tigera-anomaly-detection
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  fsType: ext4
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

AKS Azure Files storage

In the following example for an AKS cloud provider integration, the StorageClass tells Calico Enterprise to use LRS disks for log storage.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: tigera-anomaly-detection
provisioner: kubernetes.io/azure-disk
parameters:
  cachingmode: ReadOnly
  kind: Managed
  storageaccounttype: StandardSSD_LRS
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

GCP Persistent Disks

In the following example for a GKE cloud provider integration, the StorageClass tells Calico Enterprise to use the GCE Persistent Disks for log storage.

note

There are currently two types available: pd-standard and pd-ssd. For production deployments, we recommend using the pd-ssd storage type.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: tigera-anomaly-detection
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
  replication-type: none
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

Configure anomaly detection to use the persistent storage

For anomaly detection components to recognize the persistent storage you created, set the StorageClassName field in the Intrusion Detection installation resource to the name of the StorageClass you have provisioned. If you have configured persistent storage and would like to revert these changes, simply set the StorageClassName field in the Intrusion Detection installation resource to an empty string.

note

The examples used throughout this doc assume that the value for StorageClassName is tigera-anomaly-detection.

Troubleshooting

Intrusion Detection is degraded

The Tigerastatus IntrusionDetection component will be set to degraded if it fails to query for the provided StorageClass.

Solution/workaround: If the field StorageClassName is provided, tigera-operator will query for the StorageClass with the same name as the field value. Most often the unavailability of the StorageClass is due to the absence of a PersistentVolume that matches the PersistentVolumeClaim. Also, check that there is a Kubernetes node with enough CPU and memory.