Elasticsearch operator managed cluster version upgrade

Upgrade elasticsearch cluster to 7.10.2

Elasticsearch (ECK) Operator

Elastic Cloud on Kubernetes (ECK) is the official operator by Elastic for automating the deployment, provisioning, management, and orchestration of Elasticsearch, Kibana, APM Server, Beats, Enterprise Search, Beats, Elastic Agent, and Elastic Maps Server on Kubernetes.

Follow the below link to install the elasticsearch operator

https://github.com/elastic/helm-charts/tree/master/elasticsearch

Here on this page, I am sharing some inputs to upgrade the elasticsearch version which is managed by elasticsearch operator. Follow the below instructions for a smooth elasticsearch version upgrade.

Adjust the elasticsearch version in values.yaml files

Check cluster version, health, and indices status on the cluster before performing the version upgrade.

kubectl exec -it services/elasticsearch --context <CONTEXT> --namespace <NAMESPACE> -- bash
curl -s localhost:9200
curl -s localhost:9200/_cluster/health?pretty
curl -s localhost:9200/_cat/indices?v

Also, verify the elasticsearch annotations

kubectl get elasticsearch --context <CONTEXT> -n <NAMESPACE>
kubectl describe --context <CONTEXT> elasticsearch <ELASTICSEARCH_CLUSTER_NAME> -n <NAMESPACE>

You can control rolling restarts during the upgrade:

Upgrading the elasticsearch version in operator results in a one-time update to existing managed resources in the cluster. This triggers a rolling restart of pods by Kubernetes to apply those changes. If you have a very large Elasticsearch cluster or multiple Elastic Stack deployments, this rolling restart might be disruptive or inconvenient. To have more control over when the pods belonging to a particular deployment should be restarted, you can add an annotation to the corresponding Elasticsearch, Kibana, or APM Server resources to temporarily exclude them from being managed by the operator. When the time is convenient, you can remove the annotation and let the rolling restart go through.

Add annotations to the elasticsearch cluster

ANNOTATION='eck.k8s.elastic.co/managed=false' 

# Exclude Elasticsearch resources
kubectl annotate --overwrite elasticsearch <ELASTICSEARCH_CLUSTER_NAME> --context <CONTEXT> -n <NAMESPACE> $ANNOTATION

# Exclude all resources in the current namespace
kubectl annotate --overwrite elastic --context <CONTEXT> -n <NAMESPACE> --all $ANNOTATION

# Exclude all resources in all of the namespaces:
for NS in $(kubectl get ns -o=custom-columns='NAME:.metadata.name' --no-headers); do kubectl annotate --overwrite elastic --all $ANNOTATION -n $NS; done

Reverify the ANNOTATIONS on the elasticsearch operator.

kubectl describe --context <CONTEXT> elasticsearch <ELASTICSEARCH_CLUSTER_NAME> -n <NAMESPACE>

How it looks:

Name:         elasticsearch-cluster
Namespace:    e2e
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  common.k8s.elastic.co/controller-version: 1.2.1           
              common.k8s.elastic.co/pause: true
              eck.k8s.elastic.co/managed: false
              elasticsearch.k8s.elastic.co/cluster-uuid: XXXXXXXXXX
              meta.helm.sh/release-name: elasticsearch
              meta.helm.sh/release-namespace: <NAMESPACE>

Annotations are applied on the elasticseach operator now
Upgrade elasticsearch

helm upgrade <ELASTICSEARCH_CLUSTER_NAME> -f values.yaml --kube-context <CONTEXT> -n <NAMESPACE>

Now you can remove the Annotations to let the pod restart. It affects the cluster version to the latest.

Resume Elastic resource management by the operator.

RM_ANNOTATION='eck.k8s.elastic.co/managed-' 

kubectl annotate elasticsearch <ELASTICSEARCH_CLUSTER_NAME> --context <CONTEXT> -n <NAMESPACE> $RM_ANNOTATION

Now all elasticsearch cluster nodes undergo restart.

Once all the pods are restarted and UP/Running, verify the cluster version and health.

curl -s localhost:9200
curl -s localhost:9200/_cluster/health?pretty

Useful docs:
https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-install-helm.html
https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-upgrading-eck.html#k8s-ga-upgrade