MAR–APR 2024 · DATA QUOTIENT
dq_
One database interface for a data-quality platform
Python · Pydantic · Oracle · MariaDB · CockroachDB · Vertica · PostgreSQL
I contributed to dq_db_manager, the open-source library that Data Quotient's data-quality platform uses to connect to databases and read their structure. It is published on PyPI, and each database is an optional extra.
One interface, many databases
I added support for Oracle, MariaDB and MySQL, CockroachDB, and Vertica. Each sits behind the same interface, so the platform can connect to a source without caring which database it is. CockroachDB reuses the PostgreSQL implementation and MySQL reuses MariaDB's, which kept each new backend small.
Metadata as typed models
The library reads a database's structure, such as its tables, columns, constraints, and indexes, and returns it as Pydantic models, so the rest of the platform works with typed metadata instead of raw query results. I added those models for every supported database and worked on the second version of the metadata format.