Logging is a crucial administrative task as it helps identify details of an event that occurred. Logs usually store the details such as the username, time, action of the user and metadata of the event. It is helpful for auditing and for forensic examination in the event of a crime.
Azure provides various monitoring tools that help identify resource usage and bottlenecks. Azure Storage provides a logging feature that gives information on the events that occurred on the storage account.
Azure Log Analytics Workspace
Log analytics workspace is a service provided in Azure that enables us to collect logs from multiple services like an Azure Storage account and Azure Virtual Machines. The logs collected based on events can then be queried using a custom language called KQL (Kusto Query Language). KQL also is known as ‘Log Analytics Query language’ is like SQL with the additional capability to render charts.
You can add various types of events for loading into the Log Analytics workspace, and then combine it in the dashboard tiles.
Thus, Log Analytics Workspace provides a single place where you can store logs from different services, query them and build a dashboard from it.
Azure Storage Account
Azure Storage Account provides a storage platform on the cloud enabling us to store various kinds of data. Data can be stored as blobs, tables or queues. Lots of read/write/delete operations usually occur on the storage, and you might need to keep track of who is doing what.
To enable logging on an Azure Storage account, open the respective storage account. Go to Monitoring (classic) – > Diagnostic Settings (classic), select the version and check the operations you need to log (read/write/delete).
Azure Storage provides two versions of logging and v2.0 is just a superset of the v1.0. The log generated contains the following details -> resource ids, request type, the operation performed, operation status and network information like header size, authentication. v2.0 contains more details with respect to Unique IDs (UUIDs) of all the entities tied to the event.
The logs generated can be seen in the Azure Storage explorer under ‘$logs’ in the corresponding storage account. At the time of publishing this blog, ‘$logs’ is not visible in the preview version of Azure Storage Explorer in the Azure portal.
The log files are organized in a year/month/day folder structure and the file contents are ‘;’ separated. The log files can be downloaded and analyzed in your favorite tool or can be automatically imported into the Log Analytics workspace.
Loading log files into Log Analytics Workspace
At the time of publishing this blog, there is no direct way to connect ‘$logs’ to the analytics workspace. Microsoft has provided a PowerShell script that can be run to fetch logs and post them in the workspace.
Steps to load data:
- Download the PowerShell program from the link provided above. Using ‘Powershell ISE’ to run the program is recommended.
- Insert your respective ids at the top of the program. The details of getting the ids have been provided in the comments on top of each variable.
- ‘$LogType’ is the name of the table that will be created for the logs from this storage account. The table will also append ‘_CL’ during creation.
- Once you have inserted the required fields, run the program and it should import all the logs to the workspace. You can automate this using the Azure DevOps.
Querying Log Analytics Workspace
Once the logs are imported, open the Log Analytics workspace, select ‘Logs’ in the left pane and you should see your logs under the Custom Logs hierarchy. To query, you need to use the KQL (Kusto Query Language) which is like SQL.
Consider gen2_logs_CL is my custom log table and I need to select Operation_Type. In SQL, we would write it as below:
SELECT Operation_Type FROM gen2_logs_CL
gen2_logs_CL | project Operation_Type
In the below image, we have grouped the Operation type and created a pie chart to see which operations are the most common in the storage account. The render command is specific to KQL and is used to produce a chart from the output of the query.
Dashboards in Log Analytics workspace allow us to add the various queries we create across different services to be added in a single place. This allows us to get a quick look at the logs of all the services.
After creating this, you can add the pie chart to a new dashboard or an existing dashboard within the Log Analytics Workspace which will automatically update every time the custom log table is updated.
Thus, by using Azure Storage Analytics and Log analytics workspace, we can derive useful insights into the events that happen in the Azure Storage Account.