Archive Outputs

There can only be at most one workflow with archive destinations. Raw logs are compressed and sent periodically when one of these conditions hit:

• Compressed logs size reaches 16MB (configurable via agent_settings.archive_max_byte_limit)
• 30 minutes passes (configurable via agent_settings.archive_flush_interval)

The destination bucket will receive compressed logs as a gzip file every 5min (or more frequent on high volume environments). The max file size is 16MB. The gzip files have json lines containing raw log and other source attributes. The folder path format is {bucket}/{year}/{month}/{day}/{hour}/{tag}/{host}/{random id}.log.gz, for example:

testbucket/2021/05/14/21/prod/host-1/1sXpYZPs83808oGZPJExJXlLXrb.log.gz