How to Generate Text Embeddings?

Welcome to this in-depth tutorial on generating text embeddings using Eden AI!

Whether you're a beginner in natural language processing (NLP) or an experienced developer looking to integrate AI-powered search and recommendation features into your applications, this guide will help you understand how to obtain and utilize text embeddings effectively.

The tutorial is based on our latest YouTube video, where we walk through the entire process.

In this tutorial, we will cover the following topics:

• What text embeddings are and why they are essential
• How to use Eden AI's API to generate embeddings
• Implementing an API to handle embedding generation and storage
• Using FAISS (Facebook AI Similarity Search) for efficient similarity search
• Running the service locally with FastAPI

By the end of this guide, you’ll have a fully functional system for generating text embeddings and performing similarity searches. For a more detailed, step-by-step approach, be sure to check out our YouTube tutorial, where we break down each step in an easy-to-follow manner, making the process even simpler to implement!

‍

Understanding Text Embeddings

Text embeddings are mathematical representations of text (such as words, sentences, or even entire documents) in the form of high-dimensional vectors.

These vectors capture the semantic meaning of the text. The key idea is that similar pieces of text will have embeddings that are close together in this high-dimensional space.

This property is what makes embeddings so useful for various tasks in natural language processing (NLP), including:

• Semantic Search: Searching for text that is similar in meaning rather than relying on exact keyword matches.
• Text Classification: Categorizing text into predefined classes by comparing their embeddings.
• Clustering and Recommendation Systems: Grouping similar items or recommending new content based on similar embeddings.

Eden AI simplifies the process by offering an API that connects to multiple AI providers like OpenAI, Cohere, and others, allowing you to easily generate text embeddings for your own use cases.

‍

Setting Up the Environment

Before you can begin working with text embeddings, you need to set up your development environment. This section walks you through the necessary steps for installation and configuration.

‍

Install Dependencies

The first step is installing the necessary Python libraries. These packages are used for various parts of the system, including handling the API, working with text embeddings, and performing similarity search. You can install them using the following command:

pip install fastapi uvicorn pydantic requests python-dotenv faiss-cpu numpy pickle5

Here's what each package does:

• FastAPI: Used to create the web API for generating text embeddings and performing search.
• Requests: Handles making API calls to Eden AI for generating text embeddings.
• FAISS: A library developed by Facebook for efficient similarity search, specifically designed to handle vector-based data like embeddings.
• Dotenv: Helps you manage environment variables (such as API keys) securely.
• NumPy: Essential for handling numerical data like embeddings as arrays.

These dependencies will provide the necessary tools to implement and run your text embedding service.

‍

Set Up API Keys

To interact with Eden AI’s API, you need an API key. This key authenticates your requests to the service. To obtain an API key, sign up on the Eden AI platform and create an account.

Once you have the API key, create a .env file in your project directory and add your key like this:

EDEN_AI_API_KEY=your_api_key_here

By using the .env file, you can keep sensitive information like API keys secure, ensuring they are not hardcoded into your source code.

‍

Implementing the API with FastAPI

Now, let's dive into building the API. We will use FastAPI to create an API server that can generate text embeddings, store them in a vector database (using FAISS), and allow for efficient similarity searches.

‍

Here’s the main structure of our Python script:

from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from typing import List, Dict, Any, Optional
import requests
import os
from dotenv import load_dotenv
import uuid
import numpy as np
import faiss
import pickle

These imports cover the essential libraries:

• FastAPI for creating the web server and defining the API.
  
  
• Pydantic for data validation and model definition.
  
  
• Requests for making API calls to Eden AI.
  
  
• FAISS for managing the vector database.
  
  
• Dotenv to load the API key from the .env file.
  
  
• NumPy for handling the numerical arrays (embeddings).
  
  
• Pickle for saving and loading the FAISS index to disk.

‍

Load Environment Variables

Before we can use the API key, we need to load it from the .env file. Here’s how to do it:

load_dotenv()
EDEN_AI_API_KEY = os.getenv("EDEN_AI_API_KEY")
if not EDEN_AI_API_KEY:
    raise ValueError("EDEN_AI_API_KEY environment variable not set")

This code loads the environment variables and checks if the API key is present. If it’s missing, the code raises an error, ensuring the key is available before proceeding.

‍

Define the FastAPI Application

Next, we set up the FastAPI application and enable CORS (Cross-Origin Resource Sharing), which allows the API to be accessed from web browsers hosted on different domains.

app = FastAPI(
    title="Embedding Search API", 
    description="API for text embeddings and similarity search"
)

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

CORS is important when developing APIs that are accessed by web applications running on different servers.

‍

Define Pydantic Models

Pydantic models help validate the data sent to and received from the API. For instance, we define models to handle the input and output of the embeddings generation and search requests:

class TextItem(BaseModel):
    text: str
    metadata: Optional[Dict[str, Any]] = {}

class TextItems(BaseModel):
    items: List[TextItem]
    provider: str = "openai"  # Default provider
    model: Optional[str] = None  # Optional specific model

class SearchQuery(BaseModel):
    query: str
    provider: str = "openai"
    model: Optional[str] = None
    top_k: int = 5

These models ensure that the data received by the API is structured correctly:

• TextItem represents a single piece of text and optional metadata.
• TextItems is used for batch requests, where multiple TextItem instances are provided.
• SearchQuery is used for semantic search requests, allowing users to query the system for similar text.

‍

Working with FAISS for Vector Search

FAISS is used to store the embeddings and perform similarity searches. Here's how to set it up and use it for search operations.

Initialize the FAISS Index

We initialize a FAISS index, which is essentially a data structure that allows us to store and search through the embeddings efficiently. We choose a dimensionality (1536 is a common size for OpenAI embeddings) and create an index to store these vectors.

‍

class FAISSVectorDB:
    def __init__(self, dimension=1536, index_file="faiss_index.pkl"):
        self.dimension = dimension
        self.index_file = index_file
        self.index = faiss.IndexFlatL2(self.dimension)
        self.metadata = []
        self.load_or_create_index()

‍

Storing Embeddings

To store embeddings, we add them to the FAISS index and associate metadata with them. Here’s how you can do that:

def store_embedding(self, item_id: str, text: str, embedding: List[float], metadata: Dict[str, Any]) -> str:
    embedding_np = np.array(embedding, dtype=np.float32).reshape(1, -1)
    self.index.add(embedding_np)
    self.metadata.append({
        "id": item_id, 
        "text": text, 
        "metadata": metadata
    })
    self.save_index()
    return item_id

‍

Searching for Similar Embeddings

When a user submits a search query, we generate its embedding and compare it to the stored embeddings in FAISS. We retrieve the closest matches using the following code:

def search(self, query_embedding: List[float], top_k: int = 5) -> List[Dict[str, Any]]:
    query_np = np.array(query_embedding, dtype=np.float32).reshape(1, -1)
    distances, indices = self.index.search(query_np, top_k)
    return [{"id": self.metadata[idx]["id"], "text": self.metadata[idx]["text"], "score": float(1.0 / (1.0 + distances[0][i]))} for i, idx in enumerate(indices[0]) if idx < len(self.metadata)]

‍

API Endpoints

Now, let’s define the API endpoints that will allow users to interact with the service.

Generate Embeddings

This endpoint generates embeddings for provided texts and stores them in FAISS:

@app.post("/embeddings")
async def create_embeddings(items: TextItems):
    results = []
    for item in items.items:
        embedding = await get_embedding_for_single_text(item.text, items.provider, items.model)
        item_id = str(uuid.uuid4())
        db.store_embedding(item_id, item.text, embedding, item.metadata)
        results.append({
            "id": item_id, 
            "text": item.text, 
            "embedding": embedding
        })
    return results

‍

Search for Similar Texts

This endpoint performs semantic search based on the query embedding and returns the most similar results:

@app.post("/search")
async def search(query: SearchQuery):
    query_embedding = await get_embedding_for_single_text(query.query, query.provider, query.model)
    results = db.search(query_embedding, query.top_k)
    return {"query": query.query, "results": results}

‍

Conclusion

This tutorial took you through the entire process of generating text embeddings with Eden AI. By the end, you should be able to:

• Set up your environment
• Use Eden AI to generate text embeddings
• Store embeddings efficiently with FAISS
• Perform semantic searches on stored embeddings

Eden AI provides an easy-to-use API that simplifies AI integration, allowing you to access various AI providers with a single key. If you want to learn more or explore further, be sure to check out additional tutorials and subscribe to our YouTube channel!