Query GCS Archive Data with BigQuery

Edge Delta archives logs to Google Cloud Storage on the write path only. Once the data is in GCS as Parquet, you can query it in place from BigQuery with an external table, without loading or copying the files. BigQuery reads the objects directly from the bucket, and new archive files are picked up automatically.

This guide covers configuring the archive so BigQuery can read it, the archive schema, and the SQL to create and query the external table.

The following diagram shows Edge Delta archiving to GCS while BigQuery queries the data in place:

graph LR
  A[Edge Delta Agent
write path only] --> B[(GCS bucket
Parquet archive)]
  B --> C[BigQuery
external table]
  D[Analyst] -->|SQL| C

Edge Delta is only on the write path. The external table and its schema are set up once in BigQuery, and analysts query the archive directly.

Prerequisites

• A GCS Destination configured to archive data. See Configure the archive for BigQuery for the required settings.
• A Google Cloud project with the BigQuery API enabled.
• A BigQuery dataset in the same location (region) as the GCS bucket. External tables can only read a bucket in a matching location.
• roles/bigquery.dataEditor (to create the table) and read access to the bucket for the account creating the external table.

Configure the archive for BigQuery

BigQuery reads Parquet with native, per-column compression. Edge Delta must be configured to write standalone Parquet files, otherwise the files are not readable by BigQuery.

nodes:
  - name: gcs_archive
    type: gcs_output
    bucket: your-archive-bucket
    encoding: parquet
    compression: zstd
    use_native_compression: true
    schema: Archive
    disable_metadata_ingestion: true

The settings that matter for BigQuery:

Warning: Avro is not supported for BigQuery external tables. Edge Delta writes Avro archives with an external zstd wrapper (Avro archives are always zstd-compressed), which is not a valid Avro Object Container File, so BigQuery cannot read these files. Use encoding: parquet for BigQuery. Avro archives are intended for Edge Delta rehydration, not direct warehouse querying.

File layout

With no path_prefix, objects are written under a time-based prefix:

gs://your-archive-bucket/YYYY/MM/DD/HH/<tag><uuid>.parquet.zst

The file extension reflects the compression value (for example .parquet.zst for zstd). BigQuery detects Parquet by content and by the format = 'PARQUET' option, so the extension does not need to be .parquet.

To enable Hive partitioning in BigQuery, configure a path_prefix with key=value directories:

    path_prefix:
      order: [Year, Month, Day, Hour]
      format: year=%s/month=%s/day=%s/hour=%s/

This produces gs://your-archive-bucket/year=YYYY/month=MM/day=DD/hour=HH/<uuid>.parquet.zst.

Archive schema

With schema: Archive, each row is one log record. Let BigQuery autodetect the schema; the columns are:

Two schema shapes need care when querying:

• custom_source_tags.* sub-fields are BYTES (the Parquet writer does not tag them as UTF-8). Convert them with SAFE_CONVERT_BYTES_TO_STRING().
• other_tags, tags_metadata, and log_fields are Parquet maps, which BigQuery surfaces as a RECORD with a repeated key_value array. Query them with UNNEST().

Create the external table

Autodetect (recommended for the Archive schema)

Because the Archive schema has nested and map columns, let BigQuery infer it:

CREATE OR REPLACE EXTERNAL TABLE `my-project.archive_ds.ed_archive`
OPTIONS (
  format = 'PARQUET',
  uris = ['gs://your-archive-bucket/*']
);

The single * matches objects at any depth, including the time-based subdirectories. New files added to the bucket are read automatically on the next query.

Hive-partitioned table

If you configured the key=value path_prefix for Hive partitioning, create the table with partition columns so BigQuery can prune by time:

CREATE OR REPLACE EXTERNAL TABLE `my-project.archive_ds.ed_archive`
WITH PARTITION COLUMNS
OPTIONS (
  format = 'PARQUET',
  uris = ['gs://your-archive-bucket/*'],
  hive_partition_uri_prefix = 'gs://your-archive-bucket',
  require_hive_partition_filter = false
);

BigQuery autodetects year, month, day, and hour as INTEGER partition columns. Set require_hive_partition_filter = true to force every query to include a partition filter (recommended for large archives to control cost).

Query the archive

Basic query:

SELECT timestamp, host, source_type, raw
FROM `my-project.archive_ds.ed_archive`
WHERE timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
ORDER BY timestamp DESC
LIMIT 100;

Read a custom_source_tags value (BYTES to STRING):

SELECT SAFE_CONVERT_BYTES_TO_STRING(custom_source_tags.k8s_namespace) AS k8s_namespace,
       COUNT(*) AS n
FROM `my-project.archive_ds.ed_archive`
GROUP BY k8s_namespace;

Read a map column (log_fields, other_tags):

SELECT kv.key, kv.value
FROM `my-project.archive_ds.ed_archive`,
     UNNEST(log_fields.key_value) AS kv
LIMIT 100;

Partition-pruned query (Hive-partitioned table):

SELECT COUNT(*) AS n
FROM `my-project.archive_ds.ed_archive`
WHERE year = 2026 AND month = 7 AND day = 23;

Flat columns with a custom schema

The Archive schema keeps the OTel structure (nested source tags and maps). If you want clean, flat, typed columns instead (for example event_id STRING, user_id INT64, payload JSON, event_ts TIMESTAMP), deotel the data and define a user_schema:

1. Add the Deotel processor to promote fields to the top level.
2. Set schema: Raw, encoding: parquet, use_native_compression: true, and a user_schema describing your columns.

The Parquet files then contain exactly those columns, and you can create the external table with an explicit schema and Parquet-specific options:

CREATE OR REPLACE EXTERNAL TABLE `my-project.archive_ds.events`
(
  event_id  STRING,
  user_id   INT64,
  payload   JSON,
  event_ts  TIMESTAMP
)
OPTIONS (
  format = 'PARQUET',
  uris = ['gs://your-archive-bucket/*'],

  -- Parquet LIST logical type: infer as a clean ARRAY<T>
  -- instead of the nested list.element wrapper struct
  enable_list_inference = TRUE,

  -- Parquet ENUM logical type: read as STRING instead of BYTES
  enum_as_string = TRUE
);

enable_list_inference and enum_as_string are the two main Parquet type-handling options. Other Parquet encoding details (Snappy/GZIP/ZSTD compression, dictionary encoding, RLE) are handled transparently; you do not declare compression for a Parquet external table the way you would for CSV or JSON.

Notes and limits

• An external table stores no data; every query reads the Parquet files from GCS. For repeated heavy analytics, consider loading the data into a native BigQuery table.
• The bucket and the BigQuery dataset must be in the same location.
• New archive files are picked up automatically by the wildcard uris; no reload step is needed.
• To reduce scan cost on large archives, use a Hive-partitioned table and filter on the partition columns, and set require_hive_partition_filter = true.

See Also

• GCS Destination - Configure the Google Cloud Storage output node, encoding, and compression options
• Send Data to Google Cloud Storage - Set up the service account and HMAC key for GCS archiving
• Deotel Processor - Promote nested OTLP fields to the top level for flat, typed columns