Bootstrap

Dell Container Storage Modules Operator pod always restart for OOM killed

Environment

Red Hat OpenShift Container Platform (RHOCP)
4
Dell Container Storage Modules Operator
1.3

Issue

Dell Container Storage Modules Operator operator pod is always getting restarted due to OOM issue.

Resolution

The solution is to increase the memory limit value of the operator pod. There are two possible ways to do it:

  1. Set it through the ClusterServiceVersion: the advantage is that this way it’s possible to change only the memory limit of the manager container, leaving kube-rbac-proxy container unchanged. The downside is that the setting is lost after each ACS upgrade and it needs to be set again.
  2. Set it through the Subscription: the advantage is that the memory setting is preserved through updates, the downside is that this will be applied to both containers in the rhacs-operator pod, although only the manager container needs those resources.

Set it through the ClusterServiceVersion

Get the name of rhacs-operator csv:

$   oc get csv -n <ns>
NAME                                           DISPLAY                                    VERSION               REPLACES                                       PHASE
dell-csm-operator-certified.v1.3.0             Dell Container Storage Modules             1.3.0                                                                Succeeded

Take a backup of rhacs-operator csv:

$ oc -n <ns> get csv dell-csm-operator-certified.v1.3.0 -o yaml > dell-csm-operator-certified.v1.3.0.yaml

Edit it:

$ oc -n  <ns> edit csv dell-csm-operator-certified.v1.3.0

Increase the resources.limits value of the manager container:


                resources:
                  limits:
                    cpu: 200m
                    memory: 256Mi  <-- value to increase
                  requests:
                    cpu: 100m
                    memory: 192Mi   

Note: both limits and requests need to be defined. In order to be on the safe side, the values reported here are the higher values from both the containers.

;