🚀 DevOps & SRE Certification Program 📅 Starting: 1st of Every Month 🤝 +91 8409492687 🔍 Contact@DevOpsSchool.com

Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!

We spend hours on Instagram and YouTube and waste money on coffee and fast food, but won’t spend 30 minutes a day learning skills to boost our careers.
Master in DevOps, SRE, DevSecOps & MLOps!

Learn from Guru Rajesh Kumar and double your salary in just one year.


Get Started Now!

Uncategorised

Elasticsearch Lab#2 – Bulk indexing document using mget & bulk api

September 24, 2021 comments off

Table of Contents

Sample Data

$ wget https://gist.githubusercontent.com/devops-school/0a09b9e55efb45bcf397d501feb38965/raw/92179a491c6a717c70280619a8a6e3e64b2b0de3/customers_full.json
=======================================================================================
Elasticsearch Lab#2 – Bulk indexing document using mget & bulk api
=======================================================================================
Elasticsearch-mget-bulk
=======================================================================================
Multi-get
=======================================================================================
curl -XGET 'localhost:9200/products/_search?pretty=true&q=*:*'
curl -XGET 'localhost:9200/_mget?pretty' -H 'Content-Type: application/json' -d'
{
"docs" : [
{
"_index" : "products",
"_type" : "mobiles",
"_id" : "nL_Fd3EBQb932Ywx993J"
},
{
"_index" : "products",
"_type" : "mobiles",
"_id" : "2"
}
]
}
'
curl -XGET 'localhost:9200/_mget?pretty' -H 'Content-Type: application/json' -d'
{
"docs" : [
{
"_index" : "products",
"_type" : "mobiles",
"_id" : "1"
},
{
"_index" : "products",
"_type" : "mobiles",
"_id" : "2"
}
]
}
'
curl -XGET 'localhost:9200/products/_mget?pretty' -H 'Content-Type: application/json' -d'
{
"docs" : [
{
"_type" : "mobiles",
"_id" : "1"
},
{
"_type" : "mobiles",
"_id" : "2"
}
]
}
'
curl -XGET 'localhost:9200/products/mobiles/_mget?pretty' -H 'Content-Type: application/json' -d'
{
"docs" : [
{
"_id" : "1"
},
{
"_id" : "2"
}
]
}
'
=======================================================================================
Index multiple documents
=======================================================================================
curl -XPOST 'localhost:9200/_bulk?pretty' -H 'Content-Type: application/json' -d'
{ "index" : { "_index" : "products", "_type" : "mobiles", "_id" : "3" } }
{ "name": "Puma","size": 9,"color": "black" }
{ "index" : { "_index" : "products", "_type" : "mobiles", "_id" : "4" } }
{ "name": "New Balance","size": 8,"color": "black" }
'
curl -XPOST 'localhost:9200/products/_bulk?pretty' -H 'Content-Type: application/json' -d'
{ "index" : {"_type" : "shoes", "_id" : "3" } }
{ "name": "Puma","size": 9,"color": "black" }
{ "index" : {"_type" : "shoes", "_id" : "4" } }
{ "name": "New Balance","size": 8,"color": "black" }
'
curl -XPOST 'localhost:9200/products/shoes/_bulk?pretty' -H 'Content-Type: application/json' -d'
{ "index" : {"_id" : "3" } }
{ "name": "Puma","size": 9,"color": "black" }
{ "index" : {"_id" : "4" } }
{ "name": "New Balance","size": 8,"color": "black" }
'
=======================================================================================
Auto-generate ids
=======================================================================================
curl -XPOST 'localhost:9200/products/mobiles/_bulk?pretty' -H 'Content-Type: application/json' -d'
{ "index" : {} }
{ "name": "Puma","size": 9,"color": "black" }
{ "index" : {} }
{ "name": "New Balance","size": 8,"color": "black" }
'
=======================================================================================
Bulk operations in one go (paste these one operation at a time)
=======================================================================================
curl -XPOST 'localhost:9200/products/shoes/_bulk?pretty' -H 'Content-Type: application/json' -d'
{ "index" : {"_id" : "3" } }
{ "name": "Puma","size": 9,"color": "black" }
{ "index" : {"_id" : "4" } }
{ "name": "New Balance","size": 8,"color": "black" }
{ "delete" : {"_id" : "2" } }
{ "create" : {"_id" : "5" } }
{ "name": "Nike Power","size": 12,"color": "black" }
{ "update" : {"_id" : "1"} }
{ "doc" : {"color" : "orange"} }
'
=======================================================================================
Bulk indexing documents from a JSON file
=======================================================================================
How to upload BULK data for Query DSL?
=============================================
cd
sudo yum install wget -y
wget https://www.devopsschool.com/notes/elastic/elasticsearch/customers_full.json
$ curl -XPUT 'localhost:9200/customers?&pretty'
$ curl -H "Content-Type: application/x-ndjson" -XPOST 'localhost:9200/customers/personal/_bulk?pretty&refresh' --data-binary @"customers_full.json"
$ curl -H "Content-Type: application/x-ndjson" -XPOST 'localhost:9200/customers/personal/_bulk?pretty&refresh' --data-binary @"sam.json"
$ curl -H "Content-Type: application/x-ndjson" -XPOST 'localhost:9200/customers/personal/_bulk?pretty&refresh' --data-binary @"customers_full.json"
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "The bulk request must be terminated by a newline [\\n]"
}
],
"type" : "illegal_argument_exception",
"reason" : "The bulk request must be terminated by a newline [\\n]"
},
"status" : 400
}
FIX - Add a New line in customers_full.json or URL BODY
curl -XGET 'localhost:9200/_cat/indices?v&pretty'
view raw Elasticsearch-mget-bulk.txt hosted with ❤ by GitHub

Reference

  • Sample Data & Query – https://github.com/devopsschool-demo-labs-projects/elasticsearch

Certification Courses

DevOpsSchool has introduced a series of professional certification courses designed to enhance your skills and expertise in cutting-edge technologies and methodologies. Whether you are aiming to excel in development, security, or operations, these certifications provide a comprehensive learning experience. Explore the following programs:

DevOps Certification, SRE Certification, and DevSecOps Certification by DevOpsSchool

Explore our DevOps Certification, SRE Certification, and DevSecOps Certification programs at DevOpsSchool. Gain the expertise needed to excel in your career with hands-on training and globally recognized certifications.