AI Search for Magento 2

User Guide & Documentation

Installation from Command Line

This extension can be installed in a few minutes, by going through the following steps:

• Download/purchase the extension
• Unzip the file in a temporary directory
• Copy directory contents and paste it in your Magento installation root directory. This extension will not replace existing code or file structure
• Enter the following at the command line:
  
  php bin/magento setup:upgrade
  php bin/magento setup:static-content:deploy (only if site is in production mode)

 

Installation and updating with Composer

You can install this extension via Composer only after purchasing a Composer support package

Run the following Composer commands

• composer require anowave/package
• composer require anowave/explore
• bin/magento setup:upgrade
• bin/magento setup:static-content:deploy (only if site is in production mode)

 

OpenAI Platform

In order to use AI you need to have an account at OpenAI Platform (https://platform.openai.com/)

 

OpenAI API KEY

To obtain an API key which is required for the extension to operate normally, follow the steps below: 

1. Go to OpenAI Platform (https://platform.openai.com/) ,then click the SETTINGS (GEAR) icon located in the top right corner of the screen
2. Select API Keys / Create New Secret Key
3. Add name to your key and click Create Secret Key. Save your KEY somewhere because you won't be able to see it later
4. Insert your API key in Magento 2 admin panel under Stores / Configuration / Anowave / AI Buddy / OpenAI API / OpenAI API Key

 

Configuring AI Search capabilities

The only action required here is to add the same API key in Stores / Configuration / Anowave / Explore / OpenAI API / OpenAI API Key

 

Embeddings Configuration

The extension is based on OpenAI Embeddings API. In order for AI search to operate normally, it has to populate initial embedding vectors for all products form your store.

Product range could vary between stores, some may have a few products, others may have thousands of products. To cover all scenarios, we have implelemented OpenAI Batch API combined with a custom bin/magento command.

 

Selecting Embeddings Content

By default the extension will embed only the product titles however, vector search works better with more information. You can add more attributes to the embedding vectors by editing your Product attributes at Stores -> Attributes -> Product. Each attribute has 2 new options called Use for AI Buddy and Explore AI. To include the attribute into the embedding vector, set these 2 attributes to Yes. It's recommended to add text attributes such as short_description, meta description, meta keywords or other text attributes. OpenAI models work best with text attributes.

 

Configuration steps

The initial configuration involves the following important steps

 

STEP 1 - Define attributes used for embedding data

Define your attributes that you want embedded, by default the text that gets embedded includes product title, price, short description. This can be extended by updating your attributes properties which now have a new option AI that indicates that the value of the attribute must be included in embedding calculation. This way you can add values such as size, color and other options to searchable text.

To mark an attribute as AI data source:

1. Go to Stores > Attributes > Product
2. Edit ANY attribute you want
3. Switch to Storefront properties tab
4. Find section called Explore and set AI to YES

 

STEP 2 - Embeddings retrieval

Fetching ALL embeddings from OpenAI API is done with a BATCH job. Most of the configuration here is handled in CLI (Command Line Interface) through few custom commands coming with the extension.

 

BATCH JOB

Batch Job is a delayed process, where OpenAI is provided with a file to process. It has 24 hours to process a file and return a response. In reality however this takes between 10 and 30 minutes depending on the size of the processed file. Batch Job allows the module to query the OpenAI Batch API and obtain embedding vectors for thousands of products at once at a fraction of the cost. Embedding tokens cost $0.02 per 1 million tokens.

 

To create a batch job, execute the following command in console (CLI).

This command will compile a JSONL file with ALL product names, descriptions and desired attributes and it will submit it for batch processing. The command will return a batch job ID which can be used later in time to obtain results, it will return a new parameter with the batch job id.

 

 

GETTING BATCH RESULTS

To get the results of the processed JSONL file, execute the following command in console (CLI)

 

 

LISTING AVAILABLE BATCH JOBS

To list all current batch jobs, you can use the following command. It is useful for tracking batch status

The parameter 'batch_job_id' is the same ID returned from the initial batch job creation. If the process is complete, the extension will then store embedding vectors for the entire product range and it's good to go.

 

STEP 3  - Elasticsearch / OpenSearch 

IMPORTANT: If you've opted for Pinecone support skip this step and scroll down to PINECONE CONFIGURATION in this documentation

Creating custom index in ElasticSearch or OpenSearch. 

Opensearch / Elasticsearch 8 Vector Support / (native) MySQL/MariaDB support

Latest version of ElasicSearch 8 and OpenSearch come with vector support and this is supported by the AI Buddy extension as well. It can be set to store embedding vectors into it's own index in OpenSearch. This opens huge possibilities for top peak performance, fast similarity calculation, instant results and much more.

 

Creating vector index

To create a vecor index run the following command:

Executing this command will create brand new index into OpenSearch or Elasticsearch aimed to store and retrieve embedding vectors.

 

 

Updating vector index

To update index run the following command:

Excuting this command will fetch the embedding vectors already stored in the local database into vector index in Opensearch or Elasticseach 8 in effect refreshing embeedings vectors.

 

 

MySQL / MariaDb

For customers with ElasticSearch 7 or lower, vector support is not possible. We have however added a native support for non-vector databases such as MySQL and MariaDb. It provides a performant cosine similarity search as well.

 

PINECONE CONFIGURATION 

Pinecone is a vector database and managed cloud service designed for working with embeddings (numerical representations of data such as text, images, or audio). It’s widely used in AI and machine learning applications, especially for semantic search, recommendation systems, and retrieval-augmented generation (RAG).

In case your stack does not support Elasticsearch8 or OpenSearch, or has very old version of Elasticsearch which does not support vectors or cosine similarity search, Pinecone is a good alternative. 

 

To get started, first create your account at PINECONE. Once you have your account created, then follow the steps below: 

1. Go to your Pinecone Dashboard and select Indexes
2. Click Create Index
3. Name your index as per your preferences
4. Under Configuration select text-embedding-3-small
5. Change Dimension dropdown from 512 to 1536
6. Leave remaining options as is and click Create

Once index is created, you need to configure the Pinecone options in the extension's config screen in Magento 2. 

1. Go to Stores > Configuration > Anowave > AI Search and expand Pinecone section
2. Expand Pinecone API
3. Insert your Pinecone API key. To obtain an API key go to Pinecone > API keys and create new key
4. Insert your index HOST (Host can be copied from the index list in Pinecone > Indexes)

 

 

Updating PINECONE vector index

To update index run the following command:

Excuting this command will fetch the embedding vectors already stored in the local database and upsert them into your Pinecone index. 

Warning

If you have more than 10000 (10K) products, operation may take a lot of time and consume your Pinecone WU (write units). The command will show approximate information about WUs consumption and time required for the index to update. This applies only on initial indexation. 

Creating or deleting Pinecone index via CLI isn't possible.

 

Enabling Hybrid Embedding Search

Hybrid Embedding search is a combination between vector search and AI reasoning. When configured the assistant will first user super fast embedding search and then it will feed the results to OpenAI model to further optimize relevance. This would allow the assistant to return more relevant results and answer questions such as: 'Show me Miter Saw under 2500'. With embedding vector search, it may return results that won't cover 100% of this request but in combination with a reasoning model, results would be more alined to user's query.