From 37f35fdcfe8f6715cfb67bf922d23af12fb122d3 Mon Sep 17 00:00:00 2001 From: Oksana Levchuk Date: Thu, 14 Nov 2024 14:04:14 +0100 Subject: [PATCH] split logs export guides --- documentation/cdn/logs/logs-uploader.md | 122 ++++++++++++++++++ ...xport-cdn-resource-logs-to-your-storage.md | 119 +++-------------- 2 files changed, 140 insertions(+), 101 deletions(-) create mode 100644 documentation/cdn/logs/logs-uploader.md diff --git a/documentation/cdn/logs/logs-uploader.md b/documentation/cdn/logs/logs-uploader.md new file mode 100644 index 00000000..3a0efe01 --- /dev/null +++ b/documentation/cdn/logs/logs-uploader.md @@ -0,0 +1,122 @@ +--- +title: logs-uploader +displayName: Logs uploader (near-realtime) +published: true +order: +toc: + --1--Configure Logs uploader: "configure-logs-uploader" + --2--For Amazon: "amazon" + --2--For Object Storage Service: "oss" + --2--For Gcore or other storage: "gcore-other" +pageTitle: Guide on Using the CDN Logs Uploader Feature | Gcore +pageDescription: A comprehensive guide on using the CDN Logs Uploader feature for exporting logs to your storage in near-realtime. +--- +# Logs uploader (near-realtime) + +Logs Uploader is a feature that enables an automatic export of CDN resource logs to your storage in near-realtime. + +Exported logs contain information about user requests sent to cache servers and pre-cache servers (if you have the Origin shielding feature enabled). + + + +This is a paid feature. To activate Logs Uploader, [contact the Gcore support team](mailto:support@gcore.com). + + + +## Logs uploader settings + +In this section, you can find general information about log settings, statuses, and how to configure logs exporting for different storage types. + +### Logs uploader statuses + +You can check the status of the Logs uploader in the Gcore Customer Portal: + +* **Pending**: A status for the time interval between the connection to storage and the first log export. + +* **OK**: Logs are exported from at least one CDN server. + +* **Failed**: An error occurred while connecting to a storage, or the service failed to export logs within 24 hours. + +* **Pause**: The Logs uploader feature is paused. + +### Add logs from Origin Shielding + +If you are using the Origin Shielding feature, you’ll see the **Add logs from origin shielding** checkbox when configuring Logs uploader. + +We recommend that you select this option as it ensures that the logs report will include both requests to cache services and requests to the pre-cache server. Thus, you’ll receive more detailed information on resource usage. + +When configuring Logs uploader, select the checkbox to enable **Add logs from origin shielding**. + +Logs uploader page with the checkbox highlighted + + + +If you don’t see the Origin Shielding option on the **Logs uploader** page, this feature is not activated for your account. For details on how to activate Origin Shielding, check our dedicated guide. + + + +## Export logs with Logs uploader + + + +### Amazon + +Follow these instructions to export logs to AWS storage: + +1\. Open the Gcore CDN page and navigate to **Logs** > **Logs uploader**. + +2\. Enable the feature and configure its settings as described above in the Logs uploader configuration section. + +3\. In the **Storage provider**, select **Amazon**. + +4\. Provide your access key ID and secret access key, which together form long-term AWS credentials. + +5\. (Optional). Choose a storage region. While the region is often determined automatically, we recommend specifying it to ensure that your logs are exported successfully. + +6\. Specify the name of a bucket where you want to export CDN logs. + +7\. (Optional). Enter a folder name if you want to export logs to a specific folder within a bucket. + +Receive logs to AWS + +### OSS + +Follow these instructions to export logs to configure logs for Alibaba Cloud Object Storage Service (OSS): + +1\. Open the Gcore CDN page and navigate to **Logs** > **Logs uploader**. + +2\. Enable the feature and configure its settings as described above in the Logs uploader configuration section. + +3\. In the **Storage provider**, select **OSS**. + +4\. Provide your access key ID and secret access key. Check the official OSS documentation for details. + +5\. (Optional). Choose a storage region. While the region is often determined automatically, we recommend specifying it to ensure that your logs are exported successfully. + +6\. Specify the name of a bucket where you want to export CDN logs. + +7\. (Optional). Enter a folder name if you want to export logs to a specific folder within a bucket. + +Receive logs to AWS + +### Gcore\Other + +Follow these instructions to export logs to Gcore Object Storage or any S3 storage of your choice: + +1\. Open the Gcore CDN page and navigate to **Logs** > **Logs uploader**. + +2\. Enable the feature and configure its settings as described above in the Logs uploader configuration section. + +3\. In the **Storage provider**, select **Gcore** or **Other**. + +4\. Specify a hostname—a name that’s assigned to a storage server within a network and is used instead of an IP address. In you’re using Gcore Storage, you can find your hostname in the storage details section. + +5\. Provide your access key ID and secret access key. You can find this information in the Details of the storage. + +6\. Specify the name of a bucket where you want to export CDN logs. + +7\. (Optional). Enter a folder name if you want to export logs to a specific folder within a bucket. + +Receive logs to AWS + + \ No newline at end of file diff --git a/documentation/cdn/logs/raw-logs-export-cdn-resource-logs-to-your-storage.md b/documentation/cdn/logs/raw-logs-export-cdn-resource-logs-to-your-storage.md index 112cd2c0..83e4c205 100644 --- a/documentation/cdn/logs/raw-logs-export-cdn-resource-logs-to-your-storage.md +++ b/documentation/cdn/logs/raw-logs-export-cdn-resource-logs-to-your-storage.md @@ -1,6 +1,6 @@ --- title: raw-logs-export-cdn-resource-logs-to-your-storage -displayName: Export CDN logs +displayName: Raw logs published: true order: 10 toc: @@ -18,20 +18,15 @@ toc: --2--Non-Amazon storage: "non-amazon-storage" --1--Export logs to FTP or SFTP storage: "export-logs-to-ftp-or-sftp-storage" --1--Deactivate Raw Logs: "deactivate-raw-logs" - --1--Logs uploader settings: "logs-uploader-configuration" - --1--Export logs with Logs uploader: "export-logs-with-logs-uploader" pageTitle: Guide on Using the CDN Raw Logs Feature | Gcore pageDescription: A comprehensive guide on using the CDN Raw Logs feature for exporting logs to your storage, understanding traffic calculation in log reports. --- -# Export CDN resource logs to a storage +# Raw logs: export CDN resource logs to a storage -Configure automatic export of CDN resource logs to the storage of your choice using either Raw logs or Logs uploader feature. - -## Raw Logs - -Logs contain information about user requests sent to cache servers and pre-cache servers (if the Origin Shielding is enabled). +Raw Logs is an option that enables an automatic export of CDN resource logs to your storage. Logs contain information about user requests sent to cache servers and pre-cache servers (if the Origin Shielding is enabled). Raw logs page in the Customer Portal + This is a paid feature. To activate Raw Logs, [contact the Gcore support team](mailto:support@gcore.com). @@ -39,11 +34,11 @@ After activation, enable Raw Logs in the Gcore Customer Portal and configure exp -### Raw Logs settings +## Raw Logs settings In this section, you can find general information about log settings, overview of log statuses, and details on how Raw Logs are exported and calculated. -#### Add logs from Origin Shielding +### Add logs from Origin Shielding If you are using the Origin Shielding feature, you’ll see the **Add logs from origin shielding** checkbox when configuring logs. We recommend that you select this option as it ensures that the logs report will include both requests to cache services and requests to the pre-cache server. Thus, you’ll receive more detailed information on resource usage. @@ -57,7 +52,7 @@ If you don’t see the Origin Shielding option on the Raw Logs page, this featur -#### Raw Logs statuses +### Raw Logs statuses You can check the status of the Raw Logs in the Gcore Customer Portal: @@ -69,13 +64,13 @@ You can check the status of the Raw Logs in the Gcore Customer Portal: * **Pause**: The Raw Logs feature is paused. -#### Export time intervals +### Export time intervals Logs are exported at the end of each hour. If you activate Raw Logs at 00:30, the first logs will be exported between 00:45 and 01:00 and the next ones—between 01:45 and 02:00. If CDN servers are not requested and you didn’t select the **Do not send empty logs** checkbox when configuring Raw Logs, an empty log file (± 20 bytes) will be sent to your storage. -#### Log traffic calculation +### Log traffic calculation Logs can generate various types of analytics, such as delivered traffic. To understand what the totals mean, we recommend that you familiarize yourself with the formulas for calculating logs. For details about CDN reports, check out the View statistics on CDN resources guide. @@ -122,13 +117,13 @@ The final value of log data may differ slightly from the billing statistics as t -#### Log path example +### Log path example ``` s3://log-bucket-name/2019/08/20/15/nodename_primarycname.domain.ru_access.log.gz ``` -#### Log example +### Log example ``` "0.0.0.0" "-" "-" "[26/Apr/2019:09:47:40 +0000]" "GET /ContentCommon/images/image.png HTTP/1.1" @@ -141,7 +136,7 @@ AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.23 "application/json" "21" "PL" "45" "10" "100700" ``` -#### Log format +### Log format ``` "$remote_addr" "-" "$remote_user" "[$time_local]" "$request" "$status" @@ -160,7 +155,7 @@ AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.23 It’s OK if you find a field that’s not listed in the example. We occasionally add new fields to the end of the line. If some fields are added to logs, you’ll receive an email about the update. -#### Log fields +### Log fields The following table contains a complete list of available log fields. Fields formatted in italics relate to our internal CDN system, so you can ignore them. @@ -430,11 +425,11 @@ You can check other fields—they can be helpful for traffic analysis or statist -### Export logs to S3 storage +## Export logs to S3 storage -#### Amazon storage +### Amazon storage 1\. In the Gcore Customer Portal, navigate to **CDN** > **Logs**. @@ -467,7 +462,7 @@ Enter a folder name if you want to export logs to a specific folder within a buc You've successfully configured Raw Logs export to Amazon S3. -#### Non-Amazon storage +### Non-Amazon storage 1\. In the Gcore Customer Portal, navigate to **CDN** > **Logs**. @@ -512,7 +507,7 @@ You've successfully configured Raw Logs export to S3 storage. -### Export logs to FTP or SFTP storage +## Export logs to FTP or SFTP storage 1\. In the Gcore Customer Portal, navigate to **CDN** > **Logs**. @@ -558,7 +553,7 @@ If you’re using the Gcore SFTP storage, you won’t be able to write to the ho You've successfully configured Raw Logs export to FTP or SFTP storage. -### Deactivate Raw Logs +## Deactivate Raw Logs To stop Raw Logs delivery, disable the **Receive Raw Logs** toggle. @@ -567,81 +562,3 @@ If you disable the logs, your settings will be saved. You can always activate th If you made any changes to the configuration, click Discard changes to cancel any changes made during the editing process. Raw logs page - -## Logs uploader - -Upload CDN logs to multiple storages in near real time (archives are exported every 5 minutes. - -### Logs uploader configuration - -The configuration of both **Raw logs** and **Logs uploader** features is almost identical. Refer to Raw logs settings for a comprehensive overview. - -The only difference is that the Logs uploader allows you to choose log fields from a prepopulated list, making it easier to customize what logs you want to upload. - -Logs uploader settings - -### Export logs with Logs uploader - - - -#### Amazon - -Follow these instructions to export logs to AWS storage: - -1\. Open the Gcore CDN page and navigate to **Logs** > **Logs uploader**. - -2\. Enable the feature and configure its settings as described above in the Logs uploader configuration section. - -3\. In the **Storage provider**, select **Amazon**. - -4\. Provide your access key ID and secret access key, which together form long-term AWS credentials. - -5\. (Optional). Choose a storage region. While the region is often determined automatically, we recommend specifying it to ensure that your logs are exported successfully. - -6\. Specify the name of a bucket where you want to export CDN logs. - -7\. (Optional). Enter a folder name if you want to export logs to a specific folder within a bucket. - -Receive logs to AWS - -#### OSS - -Follow these instructions to export logs to configure logs for Alibaba Cloud Object Storage Service (OSS): - -1\. Open the Gcore CDN page and navigate to **Logs** > **Logs uploader**. - -2\. Enable the feature and configure its settings as described above in the Logs uploader configuration section. - -3\. In the **Storage provider**, select **OSS**. - -4\. Provide your access key ID and secret access key. Check the official OSS documentation for details. - -5\. (Optional). Choose a storage region. While the region is often determined automatically, we recommend specifying it to ensure that your logs are exported successfully. - -6\. Specify the name of a bucket where you want to export CDN logs. - -7\. (Optional). Enter a folder name if you want to export logs to a specific folder within a bucket. - -Receive logs to AWS - -#### Gcore\Other - -Follow these instructions to export logs to Gcore Object Storage or any S3 storage of your choice: - -1\. Open the Gcore CDN page and navigate to **Logs** > **Logs uploader**. - -2\. Enable the feature and configure its settings as described above in the Logs uploader configuration section. - -3\. In the **Storage provider**, select **Gcore** or **Other**. - -4\. Specify a hostname—a name that’s assigned to a storage server within a network and is used instead of an IP address. In you’re using Gcore Storage, you can find your hostname in the storage details section. - -5\. Provide your access key ID and secret access key. You can find this information in the Details of the storage. - -6\. Specify the name of a bucket where you want to export CDN logs. - -7\. (Optional). Enter a folder name if you want to export logs to a specific folder within a bucket. - -Receive logs to AWS - - \ No newline at end of file