Modern Day Data Warehouse

Modern Day Data Warehouse

Data is the new Oil a term often coined these days whenever we talk in context of Data Analytics.

Data Analysis is a process of reducing the large number of collected data to make a sense of them. Business would not be willing to look at the whole spreadsheet or entire column of numbers. It is an exhausting and tedious job to look at those numbers or spreadsheet.

Exploratory Data analysis gives techniques to devise as an aid in this situation. Most of these techniques work in part by hiding certain aspects of the data while making other aspects clear. This is understanding the key performance indicators which in turn helps Business take the "Decision".

The traditional ways of doing analytics over a historic data warehouse is now going out of fashion and out of context. People want the analysis on the live data and understand the trends of growth to take business decisions. The new way of analyzing the data is not as straight forward as it used to be which used to depend on traditional sources like ERP , CRM and LOB applications. People now want their decisions to be based on a more complicated approach which involves the data generated in and around the products right from social networking to their in house surveys to all connected devices.

To solve this puzzle of data warehousing we need to do some ground breaking changes. According to Gartner reports  “Data warehousing has reached the most significant tipping point since its inception. The biggest, possibly most elaborate data management system in IT is changing.” To achieve this the traditional data warehouse has to evolve and take advantage of big data in real time. So we need a modern data warehouse for our present puzzle.

In theory the traditional data warehouse was designed to be a central repository for all the data in an organization. The data from the transactional systems like CRM, ERP and LOB was deduplicated, cleansed and loaded (ETL) into a relational schema. The reports were based on the batches that used to run on top of that data and analytics was based on these kind of reports.

Traditional Data Warehouse System

The new trends of data analytics like high data volume, real-time data, types of data and new deployment models in cloud and hybrid are putting the traditional warehouse into pressure. So the need of advanced analytics and machine learning are coming into the picture. These trends are forcing enterprise to identify approaches to evolve their existing systems to modern data warehouse.

High Data Volume which is flowing from all Social Network Media and IOT in addition to the existing business data has caused the traditional data warehouse based on SMP to fail catastrophically. They are not able to aid business to predict the correct business decisions. The basic architecture of this design is incapable to meet scale out or scale in demand. The vertical scaling approach is not a viable return on investment scenario.

Real time Data is the demand of the Business Analysts in contrary to traditional data warehouse where analysis was done on historical sanitized data. The velocity of the data captured is ever increasing and the organizations are using this real time data not only to build and optimize their business but also to transact and engage in dynamic event driven processes like market forecasting.

New sources and types of data have come into the picture of Analytics like mobile and social media channel, scanners, sensors, RFID tags, devices, feeds and sources outside of business.  These data types are not easily digestible by traditional data warehouse and do not fit in the business schema model directly. These data types have a lot of potential to benefit the business in optimizing the operations and predictions.

New deployment models in cloud and in hybrid has hit the industry and analytics is not untouched by the presence of these. The trend is the organizations is to invest in Big Data for the analytics and most of the infrastructure for the same is chosen in cloud. The cloud gives the infrastructure the cost effectiveness and the scalability to meet the new demands of the organizations. This only means that a lot of data is also "cloud-born" such as clickstreams, videos , social feeds, GPS, market and weather.

Various Organizations are implementing advanced analytics and predictive analysis to understand what may happen in the near future of their business from varied set of data sources and types. The traditional data warehouse was not designed these types of new analytics which is inductive or bottoms up in nature. Unlike working through a defined set of schema and data collection from requirement based model of traditional ware house advanced analytics  and data science uses experimentation approach to explore answers to vague questions or non-existent questions. This requires a lot of experimentation with the data sets before a schema can be created allowing the data to make sense to business.

Organization should be ready to look into the new modern data warehouse approach when they see the below phenomenon

·       Traditional data warehouse is not able to keep up with the volume surge in the data

·       To do better business they need to look into new data sources with real time data

·       Vertical scaling is not helping

·       Platform cost is not in coherence with ROI

·       Sluggish performance

Modern Data Warehouse lets you choose from a variety of techniques like business intelligence and advanced analytics from vivid types of data sets that are exploding in volume while making the insights real time with an ability to deliver correct data at precise time.

A modern data warehouse is capable of  delivering an exhaustive logical data and analytical platform with a complete set of fully supported, solutions and technologies that can cater to demand of the most sophisticated and oppressive modern enterprise—on-premises, in the cloud, or within any hybrid scenario.

Modern Day Data Warehouse System

Things are changing fast and we need to adapt ourselves to the new as much as we know it we need to follow the same now. Or as I say it like now now. 

Thanks,

Tushar Kanti