Parquet Files

The SQL Database Reader can answer questions from a folder of Parquet files on your own server. The files are queried where they are, by the on-premise Data Gateway using the DuckDB engine, so an agent can answer "how many shipments did each hub send in October?" from a data lake export without the files ever leaving your network.

When to use it

Choose Parquet files when your data already lands in Parquet — exports from Spark, Databricks, Fabric, BigQuery or a data pipeline — and you would rather not load it into a database just so an agent can read it.

Parquet is only available through the gateway. Scanning files takes real CPU and memory, and running it on your hardware keeps both the cost and the data on your side.

How files become tables

Point the gateway at one folder. Inside it:

/srv/parquet/
  shipments/                     → table "shipments"
    year=2025/month=10/*.parquet     partition folders become columns year and month
  vendors/vendors.parquet        → table "vendors"
  hubs.parquet                   → table "hubs"
  _temporary/   .cache/          ignored (names starting with _ or .)

• Every sub-folder that contains .parquet files, at any depth, is one table. Files with different but compatible columns are combined by column name.
• Every .parquet file directly in the folder is one table.
• Partition folders named like year=2025 become ordinary columns. Questions that filter on them read fewer files, so a year/month layout keeps answers fast.
• Table names are lower case, with anything other than letters, digits and _ turned into _ (Order Items becomes order_items).
• New files in an existing folder are picked up by the very next question. New folders or new columns appear after Refresh Schema on the Tables tab.

Describing the tables (optional)

Parquet files carry no descriptions. You can add them in Qlar on the Tables tab, or give the gateway a small tables file so they arrive with the schema:

{
  "tables": {
    "shipments": {
      "description": "One row per shipment, partitioned by year and month",
      "columns": { "cost_idr": "Shipping cost in Rupiah", "sla_days": "Promised delivery days" }
    },
    "branches": { "path": "hubs.parquet", "description": "Branch list" }
  },
  "exclude": ["staging"]
}

Save it in the Parquet folder and set PARQUET_TABLES_FILE=tables.json in the gateway's .env. It can also rename a table (path points at a folder, file or glob inside the folder), exclude one, or switch automatic discovery off.

Relations between tables

Parquet has no primary or foreign keys, so Qlar sees no relations on its own. Add them as virtual foreign keys — for example shipments.hub_id → hubs.hub_id — so questions that combine tables join them correctly.

Setting it up

1. Choose Parquet on your server

Open the SQL Database Reader plugin from Tools → Plugins. On Where does your data come from?, under Files, pick Parquet on your server (marked Stays on your network). The next step is the gateway step; there are no credentials to enter.

2. Connect a gateway

Click Connect a new gateway, choose Docker or Python (pip), and run the command shown on the server that holds the files. The command already contains the one-time enrolment code and installs the Parquet engine.

With Docker, replace /path/to/parquet with your folder. It is mounted read-only — the gateway never needs to write to it:

docker run -d --name qlar-gateway --restart unless-stopped \
  --env-file .env -v "$PWD/state:/state" -v "/srv/parquet:/data:ro" \
  ghcr.io/qlar-ai/qlar-data-gateway:0.3.0 enroll

With pip, qlar-gateway enroll asks for the source: answer parquet and give the folder.

3. Approve the fingerprint

Compare the fingerprint the gateway prints with the one shown in Qlar and click Approve, as for a database gateway. The gateway's Test button reports the engine and how many tables it found, for example DuckDB v1.2.0 - 3 tables in /data.

4. Load and describe the tables

On the Tables step, load the tables, tick the ones the agent may use, add descriptions and virtual foreign keys, and save. From here everything — function settings, query testing, row-level security and logs — works as for a database.

Resources and limits

The gateway runs DuckDB inside its own process. These settings in its .env decide how much of the server it may use:

A question whose query needs more memory than allowed, or runs past the timeout, is answered with a message that the request was too heavy — narrow the period or ask for a summary instead of every row.

Security

• The gateway decides which files exist and what each table is called. Qlar can only ask for those tables by name.
• Every query must be a single SELECT. Functions that read other files (read_csv, read_parquet, glob …), file paths, and statements that write, attach or install anything are refused before they run.
• The engine is locked to the Parquet folder: no other file can be opened and the lock cannot be switched off by a query.
• Give the gateway read-only access to the folder anyway (:ro in Docker, or a service account without write permission). The gateway warns at start-up when it could write there.
• Every query is written to the gateway's local audit log, on your server.

Related Pages

• On-Premise Data Gateway
• Virtual Foreign Keys
• Configure Tables and Columns
• Excel File Source