Apache Paimon is a new data lakehouse format that focuses on solving the challenges of streaming scenarios, but also supports batch processing.
Paimon is designed with a built-in merge mechanism, and many other optimizations for mass writes, making it more adaptable to streaming scenarios.
Iceberg is the most widely supported by various open-source engines, including pure query engines (e.g., Trino), New SQL databases (e.g., StarRocks, Doris), and streaming frameworks (e.g., Flink, Spark), all of which support Iceberg.
Iceberg faces several problems in streaming scenarios, the most serious one is the fragmentation of small files. Queries in data lakehouses rely heavily on file reads, and if a query has to scan many files at once, it will of course perform poorly.To address this issue, an external orchestrator is required to regularly merge files.
Apache Arrow is a cross-language development framework for in-memory data. It provides a standardized columnar memory format for efficient data sharing and fast analytics. Arrow employs a language-agnostic approach, designed to eliminate the need for data serialization and deserialization, improving the performance and interoperability between complex data processes and systems.
The Apache Arrow format project began in February 2016, focusing on columnar in-memory analytics workload. Unlike file formats like Parquet or CSV, which specify how data is organized on disk, Arrow focuses on how data is organized in memory.
<dependency><groupId>org.apache.arrow</groupId><artifactId>arrow-memory</artifactId><version>6.0.1</version></dependency><dependency><groupId>org.apache.arrow</groupId><artifactId>arrow-vector</artifactId><version>6.0.1</version></dependency>
import org.apache.arrow.memory.RootAllocator;import org.apache.arrow.vector.*;import org.apache.arrow.vector.ipc.*;import org.apache.arrow.vector.util.*;// Set up the allocator and the schema for the vectortry (RootAllocator allocator = new RootAllocator(Integer.MAX_VALUE);VarCharVector vector = new VarCharVector("vector", allocator);ArrowWriter writer = new ArrowWriter(vector, new Schema(Collections. singletonList(vector.getField())))) {// Write data to the vectorvector.setSafe(0, "Apache".getBytes());vector.setSafe(1, "Arrow".getBytes());vector.setSafe(2, "Java".getBytes());vector.setValueCount(3);// Write vector to a filetry (FileOutputStream out = new FileOutputStream("arrow-data.arrow")) {writer.writeArrow(out.getChannel());}}
// Now, let's read the data we just wrotetry (RootAllocator allocator = new RootAllocator(Integer.MAX_VALUE);ArrowReader reader = new ArrowReader(new FileInputStream("arrow-data.arrow") .getChannel(), allocator)) {// Read schema and load the datareader.loadNextBatch();// Get the vectortry (VarCharVector vector = (VarCharVector) reader.getVectorSchemaRoot() .getVector("vector")) {// Iterate over the values in the vectorfor (int i = 0; i < vector.getValueCount(); i++) {System.out.println(new String(vector.get(i)));} } }
Apache CarbonData is an indexed columnar data format that is developed specifically for big data scenarios where fast analytics and real-time insights are critical.
CarbonData has been deeply integrated with Apache Spark, providing Spark SQL’s query optimization techniques and using its Code Generation capabilities. This makes it possible to directly query CarbonData files using Spark SQL, hence giving faster and more efficient query results.
Apache CarbonData is structured in multiple layers, which includes the table, segment, block, and page levels. This hierarchical structure allows efficient data retrieval by skipping irrelevant data during the query execution.Table: A table is a collection of segments, and each segment represents a set of data files.Segment: A segment contains multiple data blocks, where each block can store a significant amount of data.Block: A block is divided into blocklets. Each blocklet holds a series of column pages, which are organized column-wise.Page: The page level is where the actual data is stored. The data in these pages is encoded and compressed, making data retrieval efficient.
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-csv</artifactId>
<version>1.8</version>
</dependency>compile "org.apache.commons:commons-csv:1.8"- Firstly, Create an instance of HttpClient class. (The HttpClient uses a HttpUriRequest to send and receive data.)- Create an instance of HttpRequestBase class (e.g. HttpGet, HttpPost, HttpTrace, HttpDelete, etc.) and set the necessary headers and parameters.- Execute the request using HttpClient’s execute() method, and it returns an instance of HttpResponse class.- Finally, Extract the response content using HttpResponse’s getEntity() method and process it as necessary.
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.10</version>
</dependency>