What is Elasticsearch?

Suppose you want to locate a particular piece of information in a library that is the size of a city. This is a predicament that businesses have to deal with on a daily basis regarding their electronic data. They contain giant quantities of logs, documents, and user actions. Locating what is important is like trying to find a needle in a digital haystack. That is where Elasticsearch fits in. Think of it as a potent magnet that can find the necessary information in a mountain of data in a second.

Elasticsearch is a search and analytics engine that is fast and scalable. It is built to process contemporary data requirements at a phenomenal pace. We are going to discuss its potent capabilities in this guide. We will begin with its conventional power in full-text search, which searches data using precise keywords. Then we will enter the realm of AI and its sophisticated vector search, which knows the meaning of your queries.

To make these ideas a reality, we are going to go through the construction of a modern RAG app to develop smarter AI assistants. We will also build a full ETL pipeline to understand how raw data turns into useful information. By the end, you will understand how Elasticsearch works and why developers and data analysts rely on it today.

So, let us start with the very basics.

What is Elasticsearch?

Elasticsearch works like a highly organized digital library. Instead of books, it stores data as flexible JSON documents. It is based on a robust open source platform named Apache Lucene. This serves as the faster engine, making every search extremely fast. This design enables it to manage simple log files in addition to complicated and structured information with ease.

It is applied in numerous ways by people. It can drive the search box in an online store, real-time server logs to understand what is wrong with it, or it can develop interactive dashboards that represent business performance.

Elasticsearch Key Features

Let’s look at its key features.

• Full-text search: It is the standard keyword search, but made very intelligent. Elasticsearch divides the text into single words or tokens and indexes them on a special map. This is similar to the index put at the back of a book that allows it to locate documents with your keywords in near real-time. It also allows such advanced features as relevance ranking to present the best results first.
• Real-time analytics: Elasticsearch is not only good at searching for information. It can make you know of it as it arrives. Elasticsearch lets you run complex calculations like sums, averages, and histograms directly inside search queries. This makes it ideal for building live dashboards that track website traffic or monitor application performance in real time.
• Distributed and scalable: It is constructed to scale. Elasticsearch does not use a single, giant server but distributes data and workloads among a group of servers. Consider it to be a puzzle where every person works on a part of the puzzle. This enables it to scale to petabytes of data with milliseconds response times.
• Vector search: The feature introduces the power of AI to your data. Elasticsearch performs semantic and contextual searches by understanding the meaning of search terms, rather than relying only on keyword matching. It is the technology of the contemporary modern semantic search and recommendation engines, and it is a key to creating a successful RAG application.

All these characteristics combine to ensure that Elasticsearch is a single solution to almost any data problem. It is one trusted system to store, search, and analyze all your information.

Full-Text vs. Vector Search

To get a real feel of Elasticsearch, it is important to have knowledge of the two major search features of the product. They both have their purpose, and the possibility to intertwine them is the key to the power of the platform.

Full-Text Search: The Power of Keywords

Think of a database of recipes that you are searching through. In the example of typing in the search query of a garlic bread recipe, full-text search will scan the index of documents that contain those same words. It is exact, word-for-word, and it is exceptionally effective in locating particular information. It is good when you have the terms that you are searching for.

Here is a simple example. First, you add a document to Elasticsearch:

{
  "title": "Easy Garlic Bread",
  "ingredients": "Bread, garlic, butter, parsley"
}
Then, you run a full-text search query:
{
  "query": {
    "match": { "ingredients": "garlic butter" }
  }
}

This query is rapidly sent by the recipe being the words “garlic” and Butter are there in the ingredients field. This is the best search technique in logs, legal documents, or any other text in which keywords are significant.

Vector Search: The Art of Meaning

Now, suppose that you look up the same recipe database but put in savory side for pasta. A keyword search might fail. This is where the power of vector search is seen. It knows what you mean by your query. It understands that garlic bread is a savory side dish, usually served with pasta.

The concept of vector search is that text is transformed into a numerical form known as an embedding. This is the semantic meaning of the text, which is captured by this vector. Your query is also transformed into a vector. Elasticsearch then identifies the documents with vectors nearest to that of your query.

You begin with a mapping that contains these vectors:

{
  "mappings": {
    "properties": {
      "description": { "type": "text" },
      "description_vector": { "type": "dense_vector", "dims": 384 }
    }
  }
}
Then, you can perform a k-Nearest Neighbor (kNN) search to find the most relevant results by meaning: 

{
  "knn": {
    "field": "description_vector",
    "query_vector": [0.12, -0.45, 0.89, ...],
    "k": 5
  }
}

The true strength of Elasticsearch is that it can do hybrid searches, balancing the search accuracy of full-text search with the context-awareness of vector search.

Building a Simple RAG Application using Elasticsearch

Large language models (LLMs) are powerful, but they have a clear limitation: they only know what they were trained on and can sometimes hallucinate. Retrieval-Augmented Generation (RAG) addresses this problem by grounding responses in external, up-to-date data. A RAG application will supply the LLM with up-to-date and relevant information from a credible source prior to the creation of an answer. It is as though the AI gets an open-book test.

The best book on the subject to pass this exam is Elasticsearch due to its quick and precise search. A RAG application operates in the following way:

• User Query: A user asks a question, like “How do I reset my password?”
• Retrieve Context: Elasticsearch searches its document index for information related to password resets. This may be a full-text search of keywords or an attempt to search using semantic meaning as a vector.
• Augment Prompt: Elasticsearch finds the most relevant documents and combines them with the initial query into a prompt with more details for the LLM.
• Generate Answer: The user is presented with the prompt and asked: “Answer the question posed by the user with the help of the given documents only. This bases the model on factual data in responding.

This workflow ensures the answers are accurate, current, and trustworthy.

Elasticsearch in RAG Application

Now, let us explore a simplified Python example showing the retrieval step of a RAG application.

1. First, create a free Elastic Cloud deployment.

Go to https://cloud.elastic.co

Start a free trial.

Now copy the ENDPOINT and API KEY from the top right corner.

2. Now, let’s start with installing the Elastic Search library

!pip install elasticsearch

3. Connect to Elasticsearch

from elasticsearch import Elasticsearch, helpers

client = Elasticsearch(

“https://my-elasticsearch-project-da953b.es.us-central1.gcp.elastic.cloud:443”,

api_key=”YOUR_API_KEY”

)

This connects the notebook to an Elastic Serverless project. The URL points to your cloud deployment. The API key allows authorized access.

4. Now, create an Index

# -----------------------------
# Define the index name and create index (if it doesn't exist)
# -----------------------------
index_name = "national-parks" 
if not client.indices.exists(index=index_name):
 create_response = client.indices.create(index=index_name)
 print("Index created:", create_response)
else:
 print(f"Index '{index_name}' already exists.")

Output:

An index is like a database table. This block checks whether the index already exists. If not, it creates one. This gives you a place to store documents.

5. Now add a semantic_text field

# -----------------------------
# Add or update mappings for the index
# -----------------------------
mappings = {
   "properties": {
       "text": {
           "type": "semantic_text"
       }
   }
}
mapping_response = client.indices.put_mapping(
   index=index_name,
   body=mappings
)
print("Mappings updated:", mapping_response) 

Output:

The mapping tells Elasticsearch how to analyze each field. The semantic_text type activates Elasticsearch’s built-in machine learning model. When you later insert text into this field, Elasticsearch produces vector embeddings automatically and stores them for semantic search. You do not need to use your own embedding model.

6. Let’s add documents to the database

# -----------------------------
# Sample documents to ingest
# -----------------------------
docs = [
   {
       "text": "Yellowstone National Park is one of the largest national parks in the United States. It ranges from the Wyoming to Montana and Idaho, and contains an area of 2,219,791 acres across three different states. Its most famous for hosting the geyser Old Faithful and is centered on the Yellowstone Caldera, the largest super volcano on the American continent. Yellowstone is host to hundreds of species of animal, many of which are endangered or threatened. Most notably, it contains free-ranging herds of bison and elk, alongside bears, cougars and wolves. The national park receives over 4.5 million visitors annually and is a UNESCO World Heritage Site."
   },
   {
       "text": "Yosemite National Park is a United States National Park, covering over 750,000 acres of land in California. A UNESCO World Heritage Site, the park is best known for its granite cliffs, waterfalls and giant sequoia trees. Yosemite hosts over four million visitors in most years, with a peak of five million visitors in 2016. The park is home to a diverse range of wildlife, including mule deer, black bears, and the endangered Sierra Nevada bighorn sheep. The park has 1,200 square miles of wilderness, and is a popular destination for rock climbers, with over 3,000 feet of vertical granite to climb. Its most famous and cliff is the El Capitan, a 3,000 feet monolith along its tallest face."
   },
   {
       "text": "Rocky Mountain National Park is one of the most popular national parks in the United States. It receives over 4.5 million visitors annually, and is known for its mountainous terrain, including Longs Peak, which is the highest peak in the park. The park is home to a variety of wildlife, including elk, mule deer, moose, and bighorn sheep. The park is also home to a variety of ecosystems, including montane, subalpine, and alpine tundra. The park is a popular destination for hiking, camping, and wildlife viewing, and is a UNESCO World Heritage Site."
   }
]

These documents are plain text. Each document describes a national park. Elasticsearch will convert each text entry into a semantic embedding internally.

7. Now bulk ingest the created docs

# -----------------------------
# Bulk ingest documents
# ----------------------------- 
ingestion_timeout=300 # Allow time for semantic ML model to load 
bulk_response = helpers.bulk( 
   client.options(request_timeout=ingestion_timeout), 
   docs, 
   index=index_name, 
   refresh="wait_for" # Wait until indexed documents are visible for search before returning the response 
)
print(bulk_response)

Output:

Bulk indexing is faster than sending documents one at a time. The timeout is increased to give Elasticsearch time to load the built-in model on first use. The refresh option makes sure the documents are searchable before the function returns.

8. Now, let’s run a semantic search and test it

# -----------------------------
# Define semantic search query
# -----------------------------
retriever_object = {
   "standard": {
       "query": {
           "semantic": {
               "field": "text",
               "query": "Sierra Nevada"
           }
       }
   }
}
search_response = client.search( 
   index=index_name,
   retriever=retriever_object,
) 
display(search_response['hits']['hits'])

Output:

This is the retrieval step in a RAG pipeline. The query “Sierra Nevada” is not a keyword search. Elasticsearch creates an embedding for the query and compares it to the stored document embeddings. It then returns the document that is semantically closest. In this case, the Yosemite document mentions the Sierra Nevada region, so it becomes the top match.

From here, the retrieved text becomes the context for your language model. The model reads the retrieved document along with the user’s question and produces a grounded, factual answer.

Elasticsearch vs. Dedicated Vector Databases

There are numerous specialized vector databases that have been developed with the development of AI. The question that comes up then is when to consider using Elasticsearch or a dedicated vector database? Imagine the dilemma of a multi-purpose tool versus a power saw of a very specific nature.

• Elasticsearch is a multi-purpose multi-tool. It is the ideal selection in the event that your application requires more than merely searching by vectors. In case you need robust keyword filtering, log analysis, security, and interactive dashboards, Elasticsearch provides the entire integrated package. When dealing with a single system, it is sometimes easier and less expensive to manage than trying to use many specialized tools.
• The specialized power saw is known as a vector database. These tools are designed with a single objective, which is conducting the search for similarity of vectors at extreme velocities and vast scales. A specialized vector database could provide the lowest possible latency in case your core application is a real-time recommendation engine to millions of users.

One industry comparison states that Elasticsearch is optimized toward text search and analytics, whereas vector databases are scaled towards ultrafast search by vector similarity. Elasticsearch is the more sensible and more powerful choice in most projects that require a combination of the search types.

ETL Pipeline with the Elastic Stack

Raw data is usually sloppy and disordered. It has to be cleaned, processed, and loaded into a system such as Elasticsearch before it can be analyzed. This is referred to as an ETL (Extract, Transform, Load). The Elastic Stack is an easy-to-build one.

The key players in this ETL pipeline are:

• Logstash (The Factory Worker): It extracts data from various sources like files, databases, or message queues. It then transforms the data by parsing it into structured fields, enriching it with additional information, and cleaning it up.
• Elasticsearch (The Warehouse): This is where the transformed, structured data is loaded and indexed. Once inside, it is ready for high-speed search and analysis.
• Kibana (The Showroom): This is the visualization layer. Kibana connects to Elasticsearch and allows you to explore your data, create charts and graphs, and build interactive dashboards with just a few clicks.

Conclusion

We started our journey with a huge, digital haystack to seek what we want, and came across a strong magnet. Elasticsearch is that magnet. It has been much more than just a mere search box. It is an end-to-end platform that artfully combines the old-fashioned way of searching with keywords and the smart and context-aware world of AI.

We could observe its two strengths: the ability to search through the text and extract specific information, as well as the ability to search by meaning and the power of a vector search. We understood how all these features are combined in order to create the intelligent and trusted RAG application, which assures the factual and useful answers given by AI assistants. Another tool that we have built is a stable ETL pipeline, which enables us to convert raw data into clean and actionable data using the Elastic Stack.

So, it is easy to see how Elasticsearch is a scalable, universal, and critical data solution to contemporary problems. It is the basis of creating your first search with or without a complex AI system because it gives you the foundation to transform data into discovery.

Frequently Asked Questions