Data Analytics Overview(OLTP vs OLAP)

You’ve likely heard about the lucrative careers of high-level Data Analysts, but have you ever wondered how they actually handle "Big Data"? Of course, they aren’t just using Microsoft Excel. To understand the technology behind it, we must first distinguish between the two primary ways we process information: OLTP and OLAP.

1. OLTP

OLTP (Online Transactional Processing) is the DB used for company’s daily operations. Whenever you post on Reddit or place an order on Amazon, you are interacting with an OLTP database. It is triggered by the business's customer and handled by the backend server so that it can provide the business.

  • Fast: It handles millions of requests with millisecond response times.

  • Small&Frequent: It focuses on small, frequent transactions—usually just a few rows at a time.

  • Write-intensive: Its primary job is to record or update new data as it happens.

2. OLAP

OLAP (Online Analytical Processing) is the very technology we're looking for. These systems aren't built for the general customer; instead, they are designed for Data Analysts and Data Scientists.

Unlike transactional systems, there is no application "shield" here. These professionals interact with the data directly using SQL editors or Business Intelligence (BI) dashboards to ask deep questions about the business.

  • Complex: Queries are often much more complex, involving functions like EXTRACT or GROUP BY to summarize vast datasets.

  • Read-intensive: It hardly writes or updates. It almost entirely reads.

  • Patience: Unlike the instant nature of OLTP, OLAP workflows can take seconds, minutes, or even hours. This is expected when processing historical data at scale.

3. Where is the Data Kept?

Since the requirements of an analyst differ so much from those of a customers, we use some specialized storage architectures: Data Warehouses and Data Lakes.

FeatureData WarehouseData Lake
Data FormatHighly structured (Schema-based)Raw (Structured, semi, or unstructured)
OptimizationBuilt for fast analytical performanceBuilt for cost-effective, massive scale
FlexibilityStrict table definitions; high qualityMultiple engines can access the same files
AccessManaged via internal tools/governanceOpen access; requires metadata catalogs

The Data Warehouse

A Data Warehouse is like a meticulously organized library. Every piece of data must fit a specific "schema" (a table definition) before it enters. This makes exploration easy and ensures high data quality, though it is often less accessible to external, third-party tools.

The Data Lake

A Data Lake is more like a vast reservoir. It stores data in its native, raw format. This is highly cost-effective and flexible, as it allows various processing engines to study the same files. However, it requires careful metadata management and lineage tracking to prevent it from becoming an unorganized "data swamp."

댓글

댓글 쓰기

이 블로그의 인기 게시물

Interface of Java

 You can define(declare + initialize) variable like interface_name variable_name = new class_name() In such cases, the variable can only use method overriding the interfaces methods. Also, the language does not recognized it as the class; the language recognizes it as interface. Still, it does have member variables of the class and there are some libraries that enables the programmer to tell what class it is constructed(i.e. Reflection) Interface can be used as parameter of functions or return types of functions. When interface is a parameter, the class that implements the interface should be given to the function. When interface is a return type, it can be used for polymorphism. Also, variable of java(in case its not primitive type like int, float), it is rather closer to pointer of C++; it refers to the value, not contains value itself. The important point is that, when the class implements the interface, all of its public methods should be the interface's methods. If not, client...
자세한 내용 보기

Leetcode

이미지
1. Two Sum: find a pair with specific sum   class Solution : def twoSum ( self , nums : List [ int ] , target : int ) - > List [ int ] : numMap = { } n = len ( nums ) # Build the hash table for i in range ( n ) : numMap [ nums [ i ] ] = i # Find the complement for i in range ( n ) : complement = target - nums [ i ] if complement in numMap and numMap [ complement ] != i : return [ i , numMap [ complement ] ] return [ ] # No solution found You might wonder how does the code deals with duplicates. As you can see, in the iteration of building the hash table, always the latter(with bigger index) will be saved in the hash table. In the latter iteration finding the complement, the former one(with smaller index) always goes through the iteration earlier, so the [i, numMap[complement]] can return two different indexes, carrying duplicate...
자세한 내용 보기