Iceberg data storage format
简体中文 | English
Table of Contents
Repository Introduction
Apache Iceberg is a new open-source table format used for large-scale data analysis. It is designed to store large, slow-moving table data. It aims to improve the de facto standard table layout built into Hive, Trino (PrestoSQL), and Spark. Iceberg can mask differences in underlying data storage formats and provide a unified API for operations, allowing different engines to access through its provided API.
Core Features:
- Schema evolution: Supports adding, dropping, updating, renaming, and reordering table format definitions.
- Partition layout evolution: The layout of a table can be updated with changes in data volume or query patterns.
- Hidden partitioning: The query no longer depends on the physical layout of the table. By separating physical and logical data, Iceberg tables can develop partitioning schemes as the amount of data changes and time passes. Tables with incorrect configurations can be fixed without the need for expensive migration.
- Time travel: supports users to use identical snapshots for repeated queries, or allows users to easily check for changes.
- Version rollback: allows users to quickly correct issues by resetting tables to a good state.
Architecture Design:
This project offers pre-configured Iceberg data storage format images with Iceberg and its runtime environment pre-installed, along with deployment templates. Follow the guide to enjoy an "out-of-the-box" experience.
System Requirements:
- CPU: 2GHz or higher
- RAM: 4GB or more
- Disk: At least 40GB
Prerequisites
Register a Huawei account and activate Huawei Cloud
Image Specifications
| Image Version | Description | Notes |
|---|---|---|
| Iceberg1.6-kunpeng-v1.0 | Deployed on Kunpeng servers with Huawei Cloud EulerOS 2.0 64bit | |
| Iceberg1.6-kunpeng-v1.0 | Deployed on Kunpeng servers with Ubuntu24.04 64bit |
Getting Help
- Submit an issue
- Contact Huawei Cloud Marketplace product support
How to Contribute
- Fork this repository and submit a merge request.
- Update README.md synchronously based on your open-source mirror information.