Search Based Analytics: What is it & How to understand it better

Search technology can be deceitful. It appears deceptively simple to build and many companies claim they have it, but delivering accurate answers is extremely difficult. Even regular users can be confused by subpar search technology because they don’t realize what insights they're missing out on.

Search-based analytics: An introduction

Although everyone in the world has been using consumer search technology for over 2 decades now, enterprise companies haven’t adopted search to empower its business users. Here's an example that best explains the power of information for everyone at an enterprise: In the early 1990s, stock brokers had to call the back office to get a stock quote before placing an order. Charles Schwab, the largest brokerage firm in the US, innovated by providing quotes over a tele-prompting system instead.

After the new technology launched in 1996, Schwab realized that brokers who used to make 10-12 calls per day started making close to 200 calls! This was a direct result of self-exploration capabilities and removing dependency on operators. Applying the same analogy to analytics, self-service data exploration can be a huge business accelerator because users are no longer dependent on pre-defined dashboards.

Today, the business benefits of consumerizing data include helping users gain visibility into their business and speeding up their decision-making process based on real time facts. When users have specific questions in mind, they can enter search terms or ask direct questions like "What are my sales last quarter." Search-based analytics processes large volumes of data and returns answers in seconds.

Search-based analytics vs Consumer search

Search-based analytics is the fastest and easiest way to extract intelligence from enterprise data using a natural language interface. However, search-based enterprise analytics poses a complex problem. On one hand, the user experience should be as simple as possible, like a Google search experience. On the other hand, enterprise data search requires a completely different architecture from consumer search because computations need to be done on the fly for every search with several pre-processing and post-processing needs.

From my personal experience, the complexity of search-based analytics is at least 8 to 10 times higher than that of a consumer search engine. There is also a higher probability of failure due to multiple environments and business contexts in play.

Enterprise analytics solutions that offer search capabilities to help users explore data are commonly referred to as augmented analytics platforms, while traditional Business Intelligence platforms are dashboarding solutions.

Translating user queries, understanding their context, and presenting the most relevant answers are the superpowers of search-based analytics.

How does search-based analytics work? A technical analysis

For the technically-minded readers out there, let’s dive into how search-based analytics typically works.

1. The user asks a business query in a search box, which is then tokenized using a combination of natural language processing, language dictionaries, and other means.
2. This is followed by processes including ambiguity correction, context normalization, and phrase completion. This phase is extremely important and technically complex because the platform cannot assume that the user’s query is complete by any means.
3. Next comes the SQL generation (or SPL and other interfaces) phase where the tokens are used to create a logically complete, optimized, and syntactically compatible SQL query that gets sent to the data store for execution.
4. The response record set goes through a post processing sequence for summarization, ranking and other purposes.
5. This is followed by a sequence of clarifications to resolve the ambiguity and context in the user’s query.
6. As we get closer to presenting the results back to the user, the platform must be able to generate relevant charts, data tables, and summary text using natural language generation techniques. The charts and data tables may need to be rendered very differently based on the device layout where the user began their query, such as a desktop, tablet, or mobile device. These presentation formats will also need some additional analysis options such as drill down choices, tool tips, and the ability to customize charts.

6 aspects to better understand search-based analytics

To understand the inner workings of search-based analytics, let's explore its 6 technical aspects. I’ll use the helpful analogy of how enterprise search is different from a consumer search engine:

1. Conveying contextual answers

Since search analytics can almost look like magic, let’s try to dissect the basics. Before answering a question correctly, it's essential to determine if the question is understood correctly. It's also important to examine if the question is understood but not answered and why. For example, a question on the “population of New York” can be understood as the population of either “New York City” or “New York state”. While conveying the answers, the context in which the question was interpreted must be clearly presented along with the answer.

2. Completing incomplete phrases

The most striking difference between a consumer search engine (like Google) and an enterprise application (such as analytics) comes in the form of supporting incomplete phrases. For example, “sales of western region” is incomplete as the element of time is missing here. An additional phrase such as “last month” or “last quarter” can complete the search query and bring relevant results. Otherwise, it can result in serious errors for an inattentive user. Additionally, the phrase must be completed and the answer must also be presented with the complete context. For example, "1.2M sales revenue in the western region last quarter" includes the time element.

3. Answering feed-in recursively

In the consumer world, no one would ask questions like “What are the total sales of Jerry Bruckheimer's top 5 movies?" But in an enterprise, “What are the total sales of the top 5 account reps” is a very common question. To process this question, a search-based analytics platform has to first get the sales numbers of all account reps, sort them in order, find the top 5, and then sum them up to get the answer. This is again a non-obvious feature, but things can get even more complicated with a question such as “What are the most sold products of the top 5 account reps?" This adds 2 more steps to the sequence mentioned before.

4. Making do with what’s available

Unlike the consumer world, which is highly optimized for a very small set of content types (text, pdfs, documents etc.), enterprise search can vary massively in content format (different data stores, different cloud providers, different operating systems) as well as varying network latencies. An enterprise search platform must work on varying levels of operating conditions and resources. A simple "time-out" value can become very complicated to define based on just the compatibility matrix. From a user’s perspective, search results have to be accurate, and can’t be compared with the results of a consumer search.

5. Understanding expert users’ needs

In the consumer world, search providers can get away without implementing every feature because customers don’t pay for them. However, enterprise buyers have special needs, specific demands, and much more granular asks than most internet search users. Enterprise search platforms need to support not only business users, but also expert users. While Google supports only 10 advanced search techniques, a search-based analytics platform would need to support hundreds of such techniques.

6. Preparing for a single answer

Unlike consumer search, which can show multiple answers over dozens of pages, enterprise users expect one single answer. This requires an enormous amount of resources to ensure that users' queries are understood, and returned with clear constraints as well. For example, think of the difference between these 2 search phrases: “Sales for women’s shoes in California yesterday” vs. “Sales for Pegasus model shoes in California yesterday”. Assume that women's shoe sales were 0 yesterday, while the Pegasus model is not sold in California at all. How would the answers be different? Would the answer be $0 in each case? Whatever the approach, it's essential that the answers clarify the situation to the user.

Why should you choose search-based analytics?

BI is passé. Legacy dashboards take days or weeks to build, and are outdated and out of context immediately. Most importantly, they're backward- looking and answer only the questions you can think of. They're not scalable, not explorable, and don't help in testing different hypotheses or conducting business experiments.

Search-based analytics changes this paradigm. To achieve true "data literacy," every decisionmaker and support staff in an organization needs to be equipped with data intelligence. Today, some advanced business units already have use cases for modern or augmented capabilities. But search-based analytics is the most basic requirement in order to remain agile and make informed decisions quickly.

How MachEye offers the best of search-based analytics

MachEye is the most advanced enterprise search technology that exists today. We provide intuitive search-based analytics that any business user from any department can easily start using from day one. No complex syntax or dependency on analysts to explore data or construct search queries. Simply use the search box to ask questions in natural language.

MachEye delivers contextual decision intelligence and even intuits business context for queries with multiple interpretations, like "Sales in California." California could indicate a city as well as a state - but MachEye is automatically equipped to handle ambiguity. With search suggestions, prompts, and ambiguity correction, MachEye helps refine searches to present the exact answer instantly in an interactive audio-visual experience. Get started today with a 15 day free trial!