Apache Iceberg - Data Lake İçin Open Table Format

Giriş

Açıklaması şöyle

Imagine you’re managing a vast library, but instead of organized books, you have stacks of loose pages scattered everywhere. Finding the right information is a nightmare, and sharing it with others is even harder. This chaos is the reality for many businesses drowning in data. While traditional data warehouses like Snowflake, Redshift, and BigQuery offer some order, they often come with hefty price tags and vendor lock-in.

Iceberg vs Parquet

Açıklaması şöyle

Iceberg is a table format while Parquet is a file format. Iceberg tables on built on Parquet files. They offer different levels of abstraction.

Kim Iceberg Kullanıyor

Açıklaması şöyle

Iceberg is the most widely supported by various open-source engines, including pure query engines (e.g., Trino), New SQL databases (e.g., StarRocks, Doris), and streaming frameworks (e.g., Flink, Spark), all of which support Iceberg.

Iceberg Problems 

Stream write işleminde dosyaları birleştirmek gerekiyor. Açıklaması şöyle

Iceberg faces several problems in streaming scenarios, the most serious one is the fragmentation of small files. Queries in data lakehouses rely heavily on file reads, and if a query has to scan many files at once, it will of course perform poorly.

To address this issue, an external orchestrator is required to regularly merge files. 

What’s Coming for Iceberg in 2025?

Açıklama burada

1. RBAC Catalog: Fixing Permissions at Scale

2. Change Data Capture (CDC): Iceberg’s Streaming Evolution

3. Materialized Views: Simplifying Derived Data

AWS S3

Açıklaması şöyle

Without Iceberg, trying to find specific information in your raw data files on S3 can be like searching for a needle in a haystack. Tools like AWS Athena can query files, but managing the structure of your data (schema) and controlling who has access (access control) requires manual setup. Iceberg transforms your S3 buckets into well-structured, queryable datasets with proper access controls, making them compatible with any modern query engine. By layering Iceberg on top of S3, businesses gain a cohesive way to organize and make sense of sprawling data lakes, which would otherwise remain chaotic and unmanageable.