Microsoft recently announced the general availability (GA) of Virtual Network flow logs, a new capability of the Network Watcher service in Azure.
Azure Network Watcher provides network monitoring and troubleshooting capabilities to increase observability and actionable insights. It offers out-of-the-box health metrics, topology visualization, connectivity monitoring, traffic monitoring, and a diagnostics suite. The virtual network flow logs capability was in public preview last year and is now generally available. It allows users to gather data about the IP traffic passing through their virtual networks. This data can be used to monitor and optimize usage, troubleshoot connectivity issues, ensure compliance, and analyze network security.
Virtual network flow logs are designed with user-friendliness and flexibility in mind. They're a breeze to deploy for specific networks, subnets, or interfaces, and they capture layer-4 IP traffic data without impacting performance. The captured traffic is stored in a convenient JSON format for analysis, and you even have the option to enrich logs with metadata using Traffic Analytics, providing valuable insights into user behavior and security threats.
John Savill, a principal technical architect at Microsoft, summarizes in a YouTube episode on the Virtual Network flow logs:
At the Virtual Network level, I can enable flow logs. It no longer skips things when no NSG is applied. It's really designed for that large-scale usage. It supports telling me hey, is it encrypted. It supports security admin rules and will log whether it is closest to a particular nick in the order of the application. I can hook in optionally into Traffic Analytics as well for that built-in visualization and use kQL (Kusto Query Language) against it to get that extra insight.
Traffic Analysis Example (Source: Microsoft Tech Community blog post)
Virtual Network flow logs can be accessed directly from storage accounts or integrated with out-of-box visualization such as Power BI. Finally, logs can be integrated with 3rd party applications for network and security analysis, like Cisco XDR, Darktrace, IBM QRadar, and Splunk.
Furthermore, Virtual Network flow logs have a wide range of practical applications. They can monitor traffic behavior, identify unknown or unwanted traffic, and track traffic levels. They also provide insights into application behavior through IP and port filtering. Additionally, they enable the analysis of cross-region traffic with GeoIP data and capacity forecasting. They play a crucial role in ensuring compliance with enterprise access rules. They can aid network forensics and security analysis by analyzing flows from compromised IPs and exporting logs to Intrusion Detection System (IDS) or Security Information and Event Management (SIEM) tools for further investigation.
Users can enable Virtual network flow logs on one or more virtual networks using Azure Portal, PowerShell, or AzCLI, and Network Security Groups (NSGs) are not required to be attached to those virtual networks.
AWS and Google Cloud (GPC) offer capabilities like virtual network flow logs, such as Virtual Private Cloud (VPC) flow logs in AWS. These logs capture IP traffic information within a VPC and allow publishing to Amazon CloudWatch Logs, S3, or Data Firehose for various tasks like security diagnostics and traffic monitoring. Similarly, GCP provides VPC Flow Logs for VM instances and Google Kubernetes Engine nodes, offering network monitoring, security analysis, and expense optimization functionalities accessible through Cloud Logging with options for log exportation.
Lastly, virtual network flow logs will be billed based on the number of logs generated. The Network Watcher pricing page, which specifically addresses the collection of network flow logs, provides more details.