Modern Day Data
Warehouse
Data is the new Oil a term often coined these days whenever we
talk in context of Data Analytics.
Data
Analysis is a process of reducing the large number of collected data to make a
sense of them. Business would not be willing to look at the whole spreadsheet
or entire column of numbers. It is an exhausting and tedious job to look at
those numbers or spreadsheet.
Exploratory
Data analysis gives techniques to devise as an aid in this situation. Most of
these techniques work in part by hiding certain aspects of the data while
making other aspects clear. This is understanding the key performance indicators
which in turn helps Business take the "Decision".
The traditional ways of doing analytics over
a historic data warehouse is now going out of fashion and out of context.
People want the analysis on the live data and understand the trends of growth
to take business decisions. The new way of analyzing the data is not as
straight forward as it used to be which used to depend on traditional sources
like ERP , CRM and LOB applications. People now want their decisions to be
based on a more complicated approach which involves the data generated in and
around the products right from social networking to their in house surveys to
all connected devices.
To solve this puzzle of data warehousing we
need to do some ground breaking changes. According to Gartner reports “Data warehousing has reached the most
significant tipping point since its inception. The biggest, possibly most
elaborate data management system in IT is changing.” To achieve this the
traditional data warehouse has to evolve and take advantage of big data in real
time. So we need a modern data warehouse for our present puzzle.
In theory the traditional data warehouse was
designed to be a central repository for all the data in an organization. The
data from the transactional systems like CRM, ERP and LOB was deduplicated,
cleansed and loaded (ETL) into a relational schema. The reports were based on
the batches that used to run on top of that data and analytics was based on
these kind of reports.
 |
| Traditional Data Warehouse System |
The new trends of data analytics like high
data volume, real-time data, types of data and new deployment models in cloud
and hybrid are putting the traditional warehouse into pressure. So the need of
advanced analytics and machine learning are coming into the picture. These
trends are forcing enterprise to identify approaches to evolve their existing
systems to modern data warehouse.
High Data Volume which is flowing from all
Social Network Media and IOT in addition to the existing business data has
caused the traditional data warehouse based on SMP to fail catastrophically.
They are not able to aid business to predict the correct business decisions.
The basic architecture of this design is incapable to meet scale out or scale
in demand. The vertical scaling approach is not a viable return on investment
scenario.
Real time Data is the demand of the Business
Analysts in contrary to traditional data warehouse where analysis was done on
historical sanitized data. The velocity of the data captured is ever increasing
and the organizations are using this real time data not only to build and
optimize their business but also to transact and engage in dynamic event driven
processes like market forecasting.
New sources and types of data have come into
the picture of Analytics like mobile and social media channel, scanners,
sensors, RFID tags, devices, feeds and sources outside of business. These data types are not easily digestible by
traditional data warehouse and do not fit in the business schema model
directly. These data types have a lot of potential to benefit the business in
optimizing the operations and predictions.
New deployment models in cloud and in hybrid
has hit the industry and analytics is not untouched by the presence of these.
The trend is the organizations is to invest in Big Data for the analytics and
most of the infrastructure for the same is chosen in cloud. The cloud gives the
infrastructure the cost effectiveness and the scalability to meet the new
demands of the organizations. This only means that a lot of data is also
"cloud-born" such as clickstreams, videos , social feeds, GPS, market
and weather.
Various Organizations are implementing
advanced analytics and predictive analysis to understand what may happen in the
near future of their business from varied set of data sources and types. The
traditional data warehouse was not designed these types of new analytics which
is inductive or bottoms up in nature. Unlike working through a defined set of
schema and data collection from requirement based model of traditional ware
house advanced analytics and data
science uses experimentation approach to explore answers to vague questions or
non-existent questions. This requires a lot of experimentation with the data
sets before a schema can be created allowing the data to make sense to
business.
Organization should be
ready to look into the new modern data warehouse approach when they see the
below phenomenon
· Traditional data warehouse is not able to
keep up with the volume surge in the data
· To do better business they need to look into
new data sources with real time data
· Vertical scaling is not helping
· Platform cost is not in coherence with ROI
· Sluggish performance
Modern Data Warehouse lets you choose from a
variety of techniques like business intelligence and advanced analytics from
vivid types of data sets that are exploding in volume while making the insights
real time with an ability to deliver correct data at precise time.
A modern data warehouse is capable of delivering an exhaustive logical data and analytical
platform with a complete set of fully supported, solutions and technologies
that can cater to demand of the most sophisticated and oppressive modern
enterprise—on-premises, in the cloud, or within any hybrid scenario.
 |
| Modern Day Data Warehouse System |
Things are changing fast and we need to adapt ourselves to the new as much as we know it we need to follow the same now. Or as I say it like now now.
Thanks,
Tushar Kanti
