ParquetDB: A Lightweight Database System Leveraging Apache Parquet for Efficient Data Storage and Retrieval

Published

February 6, 2025

Author(s)

Logan Lang, Eduardo Hernandez, Kamal Choudhary, Aldo Romero

Abstract

Traditional data storage formats and databases often introduce complexities and inefficiencies that hinder rapid iteration and adaptability. To address these challenges, we introduce ParquetDB, a Python-based database framework that leverages the Parquet file format's optimized columnar storage. ParquetDB offers efficient serialization and deserialization, native support for complex and nested data types, reduced dependency on indexing through predicate pushdown filtering, and enhanced portability due to its file-based storage system. Comprehensive benchmarks demonstrate that ParquetDB outperforms traditional databases like SQLite and MongoDB in managing large volumes of data, especially when using data formats compatible with PyArrow. We validate ParquetDB's practical utility by applying it to the Alexandria 3D Materials Database, efficiently handling approximately 4.8 million complex and nested records. By addressing the inherent limitations of existing data storage systems and continuously evolving to meet future demands, ParquetDB has the potential to significantly streamline data management processes and accelerate research development in data-driven fields.

Citation

(potentially a different journal, still TBD)

Pub Type

Journals

Download Paper

DOI Link

Citation

Lang, L. , Hernandez, E. , Choudhary, K. and Romero, A. (2025), ParquetDB: A Lightweight Database System Leveraging Apache Parquet for Efficient Data Storage and Retrieval, (potentially a different journal, still TBD), [online], https://doi.org/10.48550/arXiv.2502.05311 (Accessed September 14, 2026)

Additional citation formats