Search is a big deal for eCommerce sites because it helps customers find what they’re looking for fast. Many eCommerce websites have search capabilities that fall short, which can frustrate users, drive them away, and hurt sales.
Google Cloud’s recent stats show that businesses are losing a whopping $2 trillion a year due to search abandonment. On top of that, it can damage your brand’s reputation and affect how customers feel about shopping with you.
But did you know that poor search functionality doesn’t just impact your customers? It also affects your internal operations.
Having a powerful search capability is just as crucial for quickly managing and analyzing large volumes of data scraped from multiple sources. Without an efficient way to search through this data, you risk slowing down decision-making and missing key insights.
Recognizing this, Bitcot implemented Algolia advanced search to enhance the search capabilities of our client’s internal dashboard. We utilized Node.js eCommerce backend to configure indexes, manage dependencies, and execute search queries effectively.
Client’s Pain Points in the Dashboard Search Functionality
Our client is a growing eCommerce business specializing in products and components, serving thousands of customers worldwide. They have specific requirements and believe in owning their data and understanding the competition.
We initially developed a custom dashboard to help them manage and visualize their extensive scraped data, categorized into different types; it’s a massive dataset.
As their business grew, they encountered limitations with the dashboard’s existing search functionality, which was slow and lacked advanced filtering capabilities, often returning irrelevant results.
With their data growing from 500,000 to 2.5 million records in a single tab, finding what they needed quickly became a big problem. Previously, searches took between 19 and 25 seconds, but as their dataset expanded, response times increased to 19 to 32 seconds, and searches occasionally failed.
To address this, we integrated Algolia into our system, significantly improving search performance.
They were frustrated by long loading times and inaccurate search results, which negatively impacted their decision-making process and overall efficiency.
The company needed a solution that could provide real-time, relevant search results and handle a large volume of products efficiently. The goal was to enhance their ability to search through this data quickly and accurately.
Bitcot’s Algolia-Based eCommerce Search Solution
To provide a faster, more intuitive search experience for our client, Bitcot’s development team decided to integrate Algolia – a powerful search-as-a-service solution specifically for handling search functions – into their admin dashboard.
Algolia is often considered the best eCommerce search solution due to its ability to deliver incredibly fast and precise search results.
Given the large volume of data and the numerous columns that needed to be searchable, Algolia instant search provided the efficiency and scalability required. With its advanced search algorithms, we implemented features like real-time indexing, enhanced filtering, and relevance tuning to deliver precise results instantly.
Algolia also provides full-text search capabilities, allowing for quick and efficient searches across multiple columns and categories. We structured the data in Algolia to reflect the categories in the dashboard, ensuring that searches are both quick and relevant.
We used Node.js for the backend operations and PostgreSQL as our database.
Overall Implementation
- Data Scraping: We developed an RPA-based scraping module to gather data from multiple websites.
- Data Processing: Python scripts clean and process the scraped data, which is then imported into CSV files. We extracted the data from these files and stored it in the database.
- Algolia Configuration: For the client to view the stored data, we created and configured indexes in Algolia to match the categories and data fields in the dashboard.
Current Goals
- Improving Search Speed: Delivering search results in real-time to ensure a fast and responsive user experience.
- Advanced Filtering and Ranking: Enabling users to filter search results by multiple attributes and improve result relevancy.
- Scalable Index Configuration: Ensuring that the search indexes could handle a large volume of data and adapt to future growth.
Key Features
- Fast Search Response: Reduced search times from up to 32 seconds to 1.5 seconds.
- Comprehensive Indexing: Data is categorized and indexed to ensure accurate search results.
- Custom Ranking: Allows for sorting and ranking of search results based on relevance.
- Typo Tolerance: Handles minor errors in search queries to provide relevant results.
- Customized Search: Recognizes specific separator characters based on our use case requirements.
The Architecture of the Node.js Application With Algolia Integration
Now let’s go on to the architecture of the Node.js application deployed on AWS EC2, utilizing PostgreSQL as the primary database and Algolia as the search service.
To ensure that the Algolia search index is always up-to-date, the application employs AWS Lambda to periodically synchronize data between PostgreSQL and Algolia. This architecture is designed to deliver a fast, scalable, and efficient search experience while maintaining accurate and consistent data across all components.
Key Components of the Architecture
1. AWS EC2 (Node.js Application)
- The core of the application is hosted on an AWS EC2 instance, running a Node.js backend that handles all user requests, processes data, and communicates with external services like Algolia and the PostgreSQL database.
- This component is responsible for serving the application’s business logic, managing user interactions, and ensuring smooth operations.
2. PostgreSQL (Database)
- PostgreSQL serves as the primary database, storing structured data such as product listings, user profiles, and metadata.
- It is optimized for transactional operations and data integrity, ensuring that all data is stored and maintained in a consistent manner.
3. Algolia API (Search Service)
- Algolia provides a fast, scalable search engine that performs full-text searches on the data.
- By indexing relevant data fetched from PostgreSQL, Algolia offers quick and efficient search capabilities for end-users.
- The Algolia API exposes endpoints for searching data and indexing new or updated records, which is essential for keeping the search results accurate and responsive.
4. AWS Lambda (Data Syncing Service)
- AWS Lambda functions are employed to periodically synchronize the Algolia search index with the latest data from PostgreSQL.
- These functions run at scheduled intervals, querying the PostgreSQL database for new or updated records, and pushing any changes to Algolia to maintain a real-time, accurate search index.
How the Components Interact
1. Node.js Application ↔ Algolia API
- The Node.js backend communicates with the Algolia API to fetch data and display search results on the user dashboard.
- When a user performs a search, the query is sent directly to Algolia, which returns the matching records instantly. This ensures a seamless and fast search experience for the end-user.
2. AWS Lambda ↔ PostgreSQL
- AWS Lambda periodically queries the PostgreSQL database to identify any new or modified records.
- The data retrieved is then processed and prepared for synchronization with Algolia, ensuring that any updates are quickly reflected in the search index.
3. AWS Lambda ↔ Algolia API
- Once the data is processed, AWS Lambda uses the Algolia indexing API to send the new or updated data to Algolia.
- The data is either added, updated, or removed from the Algolia index based on the logic defined within the Lambda function. This helps maintain the consistency and accuracy of the search results.
Optimizing Search Efficiency: Two Approaches to Using Algolia
When it comes to leveraging Algolia for powerful searches, there are two main approaches to consider: direct use of Algolia APIs and a hybrid method that combines Algolia with your own database, such as PostgreSQL. Each approach has its own set of benefits and is suited to different needs.
The first method relies entirely on Algolia for both search and data storage, while the second uses Algolia only for quick search results and retrieves full data from a different database.
The choice between these two methods depends on your specific use case. For simple applications where speed and simplicity are key, directly using Algolia might be the best option.
However, for more complex applications that require flexible data management, cost efficiency, and a balanced performance, the hybrid approach can provide a more optimized solution.
Direct Search with Algolia API
In the direct-use scenario, we rely entirely on Algolia’s APIs for both searching and data retrieval. This means that all data needed for search results is stored directly within Algolia’s data source. When a search query is made, Algolia handles the request, searches its indexed data, and returns the relevant records instantly.
This approach is ideal for applications where:
- Speed is a priority: Since all data is indexed within Algolia, searches are extremely fast.
- Simplicity is needed: Managing only one data source (Algolia) simplifies architecture and reduces the need for complex data synchronization.
Our Project Challenges
When we initially integrated Algolia into our client’s system, we were relatively new to the platform. As with any new tool, we soon encountered challenges that required us to adapt and find solutions tailored to our specific use case.
Algolia is optimized for speed, but this can sometimes come at the expense of accuracy, particularly in cases involving filter counts. We noticed discrepancies in the filter counts, which we found were due to Algolia’s inherent speed optimizations which are designed to prioritize speed but can occasionally result in imprecise hit counts for filtering.
This issue is known and documented by Algolia, and it required us to rethink our approach.
Hybrid Search
To overcome this, we modified our search workflow to use Algolia as a search database rather than relying on it for the full response with detailed data.
This approach focuses on a hybrid model that combines the search capabilities of Algolia with the data management strengths of a traditional database like PostgreSQL.
Here’s how it works:
Step 1: Fast Search with Algolia: When a search is performed, Algolia is used to quickly find and return only the IDs of the matching records.
Step 2: Detailed Data Retrieval from PostgreSQL: Using the IDs returned from Algolia, a subsequent query is made to a PostgreSQL database to fetch the complete data, including any additional information that may not be stored in Algolia.
This approach offers several advantages:
- Cost Efficiency: By storing only minimal data in Algolia (e.g., IDs and searchable attributes) and keeping more extensive data in PostgreSQL, you can reduce storage costs.
- Flexibility and Control: You retain more control over your data, which remains in your database, allowing for custom data management, additional security measures, and better integration with other systems.
- Balanced Performance: You get the best of both worlds — fast search results via Algolia and detailed data retrieval from your database.
The Process Behind Our Lightning-Fast Search Solution
Step 1: Initial Setup
We started by setting up Algolia, which involved installing the necessary tools and configuring our search settings.
- AlgoliaSearch: To enable advanced search functionalities.
- Dotenv: To securely manage environment variables, like API keys, without exposing them in our code.
We installed these tools using the Node.js package manager, ensuring a secure and efficient setup.
This process was straightforward and involved integrating Algolia with the client’s existing systems.
Step 2: Configuration
With the setup in place, we focused on configuring Algolia. The search system was configured to organize and handle data in a way that suited the specific needs of our project.
The configuration involved:
Connecting to Algolia
Using AlgoliaSearch to connect our application to the Algolia service with API credentials managed securely by dotenv.
Setting Up Indexes
We created a script to set up and configure Algolia indexes. We established different indexes for each data category.
An index in Algolia functions similarly to a table in a traditional database. It’s a structured container where data is organized into different categories or types. Each index holds a collection of records related to a specific type of data.
The dashboard has different tabs, each focusing on a particular category of data.
For example, the data displayed when users search within the “Compatible SSD” tab comes from the “Compatible SSD” index in Algolia. This index is specifically set up to store and manage all records related to compatible SSDs, allowing for efficient and relevant search results in that category.
Customizing Settings
We tailored the search settings for each index. This included:
- Attributes for Faceting: Specifying which relevant attributes (like IDs, part numbers, and store names) should be used for filtering search results. We implemented faceting in a balanced way to enhance filtering without affecting performance.
- Searchable Attributes: Determining which product details (such as names, descriptions, and categories) should be searchable.
- Custom Ranking: Setting up ranking criteria to ensure the most relevant results appear first in search results.
- Pagination Limits: Configuring limits appropriate to manage large volumes of search results displayed per page and optimize performance. By default, it shows 20 records per page, though this can be adjusted based on user needs.
- Typo Tolerance: Algolia’s typo tolerance feature handles minor typos in search terms, ensuring that users still receive relevant results even if their input is not perfect.
Customizable API Responses
We also configure which attributes are included in the API responses to meet the needs of different tabs. For example, in the “Host App” tab, we ensure that “Configuration Notes” are included in the search results, providing users with comprehensive information.
Step 3: Running the Configuration
We modified the project’s configuration file to include a custom command for setting environment variables and executing the index configuration script. This streamlined the process of applying settings to Algolia.
Step 4: Implementing Search Functionality
With the indexes configured, we moved to the search functionality using Algolia’s API:
- Constructing a Query: We created a query system that allowed users to search and filter results in different scenarios. This involved setting up parameters like pagination, search terms, and filters (like product part numbers or store names).
- Executing the Search: We implemented the search feature to perform searches and retrieve results quickly based on user input.
The Client’s Major Impact and Success With Dashboard Search
Optimizing the search function in eCommerce means a business can quickly pull up product details, making it a lot simpler to track performance.
The integration of Algolia has transformed the search functionality within the client’s dashboard, offering users a much faster and more reliable search experience. Improving the dashboard search bar now helps them quickly find specific data and streamline their workflow, making it much easier to manage everything.
Here’s how the implementation transformed their eCommerce business:
Enhanced Speed and Performance
Algolia’s integration reduced search times from up to 32 seconds to just 1.5 seconds. This speed enhancement ensures that users receive near-instantaneous search results, drastically improving overall user experience.
Superior Search Accuracy
The advanced search algorithms and real-time indexing provided by Algolia deliver highly relevant results. Comprehensive and precise indexing across multiple data categories ensures users find exactly what they need quickly.
Advanced Filtering and Customization
The new system allows for sophisticated filtering and custom ranking, enhancing result relevancy. Users can now sort and filter search results by multiple attributes, with typo tolerance ensuring accurate results even with minor query errors.
Scalable and Future-Ready
Algolia’s scalable indexing capabilities accommodate large data volumes and adapt to future growth. This scalability ensures that the search functionality in eCommerce remains efficient and effective as the data expands.
Streamlined Integration
The integration with Node.js search function and PostgreSQL, coupled with secure setup tools like AlgoliaSearch and Dotenv, streamlined the implementation process. This efficient setup facilitated a smooth transition and enhanced the overall search experience.
Final Thoughts
For eCommerce owners who don’t optimize their search, the negative impact can be pretty severe and sometimes last a long time.
That makes total sense. What also makes perfect sense is that the focus for search has naturally been on customer-facing online stores with an eye toward maximizing revenue.
What’s less apparent is that having a strong internal search system can be just as impactful.
Having good search functionality in eCommerce makes it a breeze for your team to find what they’re looking for.
The algoliasearch API makes it super easy to add fast and accurate search to your eCommerce app, so users can find what they’re looking for quickly.
Here are the key takeaways:
- Invest in Advanced Search Solutions: Leveraging powerful search platforms like Algolia can significantly enhance the user experience and drive business growth.
- Customize for Relevance: Tailoring search settings to different product categories ensures that users receive the most relevant results.
- Automate Where Possible: Automating configuration processes helps maintain consistency and efficiency, saving time and reducing errors.
Yes, you can transform your internal search into an impressive experience that wows your operations team, boosts their efficiency, and frees up valuable time in ways that count.
Reach out to learn more about how you can move beyond chaotic data and overcome your employees’ search limitations through our eCommerce search solutions.
