Skip to content

Commit be13113

Browse files
authored
Update instructions for Elasticsearch node replacements (#5338)
1 parent 178e660 commit be13113

File tree

1 file changed

+17
-4
lines changed

1 file changed

+17
-4
lines changed

documentation/meta/maintenance/elasticsearch_cluster.md

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -354,6 +354,9 @@ In the steps below, `<env>` refers to a subfolder in the `environments/`
354354
directory, and can be one of "dev" or "prod". This is different from
355355
`<environment>` which is part of the ES module name and can be one of "staging"
356356
or "production".
357+
358+
Consider pausing the `<environment>_elasticsearch_cluster_healthcheck` DAG to avoid
359+
getting redundant alerts during this process.
357360
```
358361

359362
1. Find the instance in the AWS management console using the ID from the
@@ -381,9 +384,13 @@ or "production".
381384
with this change in production. Apply this change. The plan should include:
382385

383386
- creation of a new Elasticsearch data node
384-
- creation of new alarms and changes to existing alarms
387+
- creation of new alarms and changes to existing alarms (for production)
385388
- addition of the new instance's IP to the Route53 record
386389

390+
```bash
391+
just tf <env> apply -target='module.staging-elasticsearch-8-8-2'
392+
```
393+
387394
4. Wait for a new instance to be provisioned by Terraform.
388395

389396
Record the public IPv4 DNS and Terraform index of the new instance, they will
@@ -394,7 +401,7 @@ or "production".
394401
pass the public IPv4 DNS step from step 3 to the `-l`/`--limit` flag.
395402

396403
```bash
397-
just ansible/playbook <env> elasticsearch/sync_config.yml -e apply=true -l <public_ipv4_dns>`
404+
just ansible/playbook <environment> elasticsearch/sync_config.yml -e apply=true -l <public_ipv4_dns>
398405
```
399406

400407
```{note}
@@ -413,7 +420,7 @@ or "production".
413420
`http://localhost:9220`.
414421

415422
```{tip}
416-
Use the Elasticvue extension or app because the web interface cannot connect
423+
Use the Elasticvue browser extension or app because the web interface cannot connect
417424
to Elasticsearch due to CORS protection.
418425
```
419426

@@ -422,7 +429,9 @@ or "production".
422429

423430
```json
424431
{
425-
"transient.cluster.routing.allocation.exclude.name": "<private_ipv4_address>"
432+
"transient": {
433+
"cluster.routing.allocation.exclude.name": "<private_ipv4_address>"
434+
}
426435
}
427436
```
428437

@@ -458,6 +467,10 @@ or "production".
458467
- subtraction of the retired instance's IP from the Route53 record
459468
- destruction of extra alarms and changes to existing alarms
460469
470+
```bash
471+
just tf <env> apply -target='module.staging-elasticsearch-8-8-2'
472+
```
473+
461474
14. If the deletion of the individual alarms fails due to them being part of a
462475
composite alarm, go into the AWS management console and remove the alarms
463476
that are supposed to be deleted from any composite alarms that they are a

0 commit comments

Comments
 (0)