Manage Elasticsearch index retention with curator

Download the curator package

wget -c
https://packages.elastic.co/curator/5/debian/pool/main/e/elasticsearch-curator/elasticsearch-curator_5.8.3_amd64.deb


sudo dpkg -i elasticsearch-curator_5.8.3_amd64.deb


Create curator config

mkdir ~/.curator

vim ~/.curator/curator.yml

---
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
client:
hosts:
- vpc-your-elasticsearch-address.es.amazonaws.com
port: 80
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
ssl_no_validate: False
username:
password:
timeout: 30
master_only: False

logging:
loglevel: INFO
logfile:
logformat: default
blacklist: ['elasticsearch', 'urllib3']


Create config for manage indices retention

vim ~/.curator/delete_indices.yml
---
actions:
  1:
    action: "delete_indices"
    description: >-
      Delete indices older than 7 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and If you want to change the retention Days then goto unit_count:<enter no of day>.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
      exclude:

Automate the job with cron

crontab -e

# m h  dom mon dow   command
0 7 * * * curator /home/ubuntu/.curator/delete_indices.yml