🚀 DevOps & SRE Certification Program 📅 Starting: 1st of Every Month 🤝 +91 8409492687 🔍 Contact@DevOpsSchool.com

Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!

We spend hours on Instagram and YouTube and waste money on coffee and fast food, but won’t spend 30 minutes a day learning skills to boost our careers.
Master in DevOps, SRE, DevSecOps & MLOps!

Learn from Guru Rajesh Kumar and double your salary in just one year.


Get Started Now!

AWS

Elasticsearch Cluster Setup and Configuration Tutorial step by step using AWS ec2

April 15, 2020 comments off

Elasticsearch Cluster Setup and Configuration Tutorial step by step in AWS ec2.

Step 1. File Descriptor

sudo vi /etc/security/limits.conf
ec2-user  -  nofile  65535

This change will only take effect the next time the ec2-user user opens a new session.

Step 2. Adjust vm.max_map count to recommended level

To make it persistent, you can add this line:

$ sudo vi /etc/sysctl.conf
vm.max_map_count=262144
$ sudo sysctl -p

Step 3. List down Public IP and Private IP and decide their roles.

Role Public IPs Private IPs
Master 13.234.59.125 172.31.20.87
Data 13.126.179.145 172.31.30.190
Client 35.154.20.21 172.31.23.201

Step 4. Change host name of each machine

In Master (Pub-13.234.59.125 Pvt-172.31.20.87)

$ sudo hostnamectl set-hostname es-master01
$ hostname

In Data (Pub-13.126.179.145 Pvt-172.31.30.190)

$ sudo hostnamectl set-hostname es-data01
$ hostname

In Client (Pub-35.154.20.21 Pvt-172.31.23.201)

$ sudo hostnamectl set-hostname es-client01
$ hostname

Step 5. Modify /etc/hosts with following in Master(s), Data(s) and Client(s) node

In 13.234.59.125(Master)

$ sudo vi /etc/hosts

172.31.20.87	es-master01
172.31.30.190	es-client01
172.31.23.201	es-data01

Step 6. download and Install Elasticsearch

Step 7. Set the minimum and maximum heap size

The more heap available to Elasticsearch, the more memory it can use for its internal caches, but the less

memory it leaves available for the operating system to use for the filesystem cache. Also, larger heaps can

cause longer garbage collection pauses.

Method 1 - in jvm.options file Here are examples of how to set the heap size via the jvm.options file:

-Xms1g 	# Set the minimum heap size to 1g.
-Xms2g 	# Set the minimum heap size to 2g.
-Xmx2g 	# Set the maximum heap size to 2g.

Method 2 - Set environment variable using ES_JAVA_OPTS

**Set the minimum 1000 MB and maximum heap size to 2000 MB.**
ES_JAVA_OPTS="-Xms1000m -Xmx2000m" ./bin/elasticsearch 

**Set the minimum and maximum heap size to 2 GB.**
ES_JAVA_OPTS="-Xms2g -Xmx2g" ./bin/elasticsearch 

**Set the minimum and maximum heap size to 4000 MB.**
ES_JAVA_OPTS="-Xms4000m -Xmx4000m" ./bin/elasticsearch

Recommended: For 1 CPU and 2 GB RAM for demo - this is recommended settings.

CLIENT
-Xms1g
-Xmx1g

Master
-Xms2g
-Xmx2g

DATA
-Xms1g
-Xmx1g

Step 8. Configure Elasticsearch for 3 node Cluster

In Master (Pub-13.234.59.125 Pvt-172.31.20.87)

cluster.name: devopsschool
node.name: es-master01
node.master: true
node.data: false
network.host: 172.31.20.87
cluster.initial_master_nodes: ["es-master01"]
#discovery.seed_hosts: ["172.31.0.107", "172.31.14.88", "172.31.13.85"]
#or
discovery.seed_hosts: ["es-master01", "es-client01", "es-data01"]

In Data (Pub-13.126.179.145 Pvt-172.31.30.190)

cluster.name: devopsschool
node.name: es-client01
node.master: false
node.data: false
network.host: 172.31.30.190
cluster.initial_master_nodes: ["es-master01"]
#discovery.seed_hosts: ["172.31.0.107", "172.31.14.88", "172.31.13.85"]
#or
discovery.seed_hosts: ["es-master01", "es-client01", "es-data01"]

In Client (Pub-35.154.20.21 Pvt-172.31.23.201)

cluster.name: devopsschool
node.name: es-client01
node.master: false
node.data: false
network.host: 172.31.23.201
cluster.initial_master_nodes: ["es-master01"]
#discovery.seed_hosts: ["172.31.0.107", "172.31.14.88", "172.31.13.85"]
#or
discovery.seed_hosts: ["es-master01", "es-client01", "es-data01"]

Step 9. Run it

view raw Elasticsearch-Cluster-Setup.md hosted with ❤ by GitHub

Certification Courses

DevOpsSchool has introduced a series of professional certification courses designed to enhance your skills and expertise in cutting-edge technologies and methodologies. Whether you are aiming to excel in development, security, or operations, these certifications provide a comprehensive learning experience. Explore the following programs:

DevOps Certification, SRE Certification, and DevSecOps Certification by DevOpsSchool

Explore our DevOps Certification, SRE Certification, and DevSecOps Certification programs at DevOpsSchool. Gain the expertise needed to excel in your career with hands-on training and globally recognized certifications.