Archive Outputs

There can only be at most one workflow with archive destinations. Raw logs are compressed and sent periodically when one of these conditions hit:

  • Compressed logs size reaches 16MB (configurable via agent_settings.archive_max_byte_limit)
  • 30 minutes passes (configurable via agent_settings.archive_flush_interval)

The destination bucket will receive compressed logs as a gzip file every 5min (or more frequent on high volume environments). The max file size is 16MB. The gzip files have json lines containing raw log and other source attributes. The folder path format is {bucket}/{year}/{month}/{day}/{hour}/{tag}/{host}/{random id}.log.gz, for example:

testbucket/2021/05/14/21/prod/host-1/1sXpYZPs83808oGZPJExJXlLXrb.log.gz

Amazon S3

Archive data in Amazon S3.

Azure Blob Storage

Archive data in Azure Blob Storage.

Google Cloud Storage

Archive data in Google Cloud Storage.

Local Storage

Archive data in Local Storage.

Beta Archive Destinations

Archive destinations that are in beta.