User Guide

• Catalog
  • Filesystem Catalog
  • REST Catalog
• Schema
  • Example
  • Compatibility
  • DataField
  • Nullable MAP Keys
  • Update Schema
• Snapshot
  • Directory Layout
  • Snapshot File
• Manifest
  • Manifest List
  • Manifest
  • Data Manifest
  • Index Manifest
  • Appendix
    • SimpleStats
    • BinaryRow
• Manifest Cache
  • Overview
  • Configuration
  • Future Optimizations
• Manifest Entry Cache
  • Overview
  • Configuration
  • Limitations
  • Metrics
• Parquet Metadata Cache
  • Overview
  • Configuration
  • Future Optimizations
• Data Types
• Primary Key Table
  • Bucket
  • LSM Trees
    • Sorted Runs
• Append Only Table
• System Tables
  • Read-Optimized System Table
    • Limitations
    • Typical Usage
• Write
  • Bucketing Modes
  • RecordBatch Construction
  • Prepare Commit
    • Compatibility Goals
    • Interface Design in Paimon C++
    • CommitMessage Contract
    • Serialization and Deserialization
    • Operational Flow
• Commit
  • Commit Process Overview
  • Detailed Process
    • Collect File Changes
    • Compact (Merge) Manifest Files
      • Full Compaction
      • Minor Compaction
      • Compaction Rules
    • Generate the Base Manifest List
    • Generate New Manifest Files and the Delta Manifest List
    • Generate the Snapshot and HINT File
• Compaction
  • Append-Only Table Compaction
    • Auto Compaction
    • Full Compaction
    • Append-Only Table Compaction Options
  • Primary Key Table Compaction
    • Full Compaction
    • Lookup Compaction
    • Primary Key Table Compaction Options
      • Number of Sorted Runs to Pause Writing
      • Number of Sorted Runs to Trigger Compaction
• Read
  • Schema Evolution
    • Scope and Compatibility
    • Per-File Schema via Field IDs
    • Overflow Behavior Disclaimer
    • Type Change Support Matrix
    • Implementation Guidance
  • Deletion Vectors on Data-Evolution Tables
    • How the Deletion Vector Is Located
    • Limitations
• Metrics reference
  • Scan planning
  • Prefetch reader
  • Prefetch I/O
• Data Cleanup
  • Orphan File Cleanup
    • Description
    • Detailed Steps
    • Performance Considerations
  • Expiring Partitions
    • Description
    • Detailed Steps
  • Expiring Snapshots
    • Description
    • Detailed Steps
    • Performance Considerations
• Prefetch
• Read Range Splitting Strategy
• Parquet
• ORC
• Memory Format
  • Why Apache Arrow
  • Versioning and Dependency Concerns
  • Adopting the Arrow C Data Interface
  • Design Principles
  • Implementation Outline
    • Schema and buffers
    • Memory management
    • Nullability and validity
    • Conversion boundaries
• Global Index
  • Bitmap Index Compatibility
  • BTree Index
  • Range Bitmap Index
  • Lucene Index
  • Tantivy Full-Text Index (Experimental)
  • Vector Index (Lumina)
• Primary Key Global Index
  • Table requirements
  • Semantics
  • Current scope