Data architecture is a big decision, especially if there is a digital transformation.
Among the possible options, two of the most popular ones are the usage of “data warehouses” and “data lakes”.
What is Data Warehouse?
A data warehouse is a blend of technologies and components which allows the strategic use of data. It is a technique for collecting and managing data from varied sources to provide meaningful business insights.
It is electronic storage of a large amount of information by a business which is designed for query and analysis instead of transaction processing. It is a process of transforming data into information.
Data Warehouse stores data in files or folders which helps to organize and use the data to take strategic decisions. This storage system also gives a multi-dimensional view of atomic and summary data. The important functions which are needed to perform are:
- Data Extraction (logical or physical)
- Data Cleaning (a major part of the so-called ETL process)
- Data Transformation ( process of converting data from one format or structure into another format or structure)
- Data Loading and Refreshing (Load describes adding new transactions. Refresh is a job that updates a dimension to the new state)
Data warehouse uses a traditional ETL (Extract Transform Load) process.
Storage Costs, Users and Key Benefits
Storing data in Data warehouse is costlier and time-consuming.
The data warehouse is ideal for operational users because of being well structured, easy to use and understand.
Most users in an organization are operational, so that they in most cases care about reports and key performance metrics.
What is Data Lake?
A Data Lake, instead, is a storage repository that can store large amount of structured, semi-structured, and unstructured data. It is a place to store every type of data in its native format with no fixed limits on account size or file. It offers high data quantity to increase analytic performance and native integration.
We can say that a Data Lake is like a large container which is very similar to real lake and rivers. A Data Lake can have structured data, unstructured data, machine to machine and logs flowing through in real-time.
A Data Lake is a large size storage repository that holds a large amount of raw data in its original format until the time it is needed. Every data element in a Data lake is given a unique identifier and tagged with a set of extended metadata tags. It offers wide varieties of analytic capabilities.
Data Lakes use of the ELT (Extract Load Transform) process.
Despite the benefits, there are a few cases where a data lake might not work as planned:
No business case. Without clearly articulating and understanding how a data lake will benefit the business, a user might fail to acquire the approvals and buy-in needed to move forward.
Poor integration. A data lake can supplement or, in some cases, replace a data warehouse. But unless there is a plan for integrated data management, an organization might not achieve the full value a data lake can deliver.
Technology choices that don’t fit. Selecting the wrong platform or tools can add significant complexity and cost to implementation and ongoing management.
Inadequate governance and security. Enterprise-grade governance and security strategies are critical for protecting sensitive information, maintaining compliance and enabling users to take full advantage of data.
No long-term vision. A data lake requires a long-term commitment plus planning to accommodate continued data growth.
Storage Costs, Users and Key Benefits
Data storing in big data technologies are relatively inexpensive then storing data in a data warehouse.
Data lake is ideal for the users who indulge in deep analysis. Such users include data scientists who need advanced analytical tools with capabilities such as predictive modeling and statistical analysis.
They integrate different types of data to come up with entirely new questions, since their users usually may need to go beyond its capabilities.
For a firm that’s looking to analyze large but structured data sets, a data warehouse is a good option. In fact, if the company is only interested in descriptive analytics (the process of merely summarizing the data one has), a data warehouse may be all it needs.
But for most companies embarking on big data initiatives, structured data is only part of the story.
Each year, businesses generate a staggering quantity of unstructured data: for those firms, data lakes are attractive options because of their ability to store vast quantities of such data.
Data lakes can contain also all data and data types; they empower users to access data prior the process of transformed, cleansed and structured. But attention:
Data lakes are not simply data warehouses revisited. They represent a unique approach for organizations to achieve major business goals, if implemented properly.