Virtual Private Cloud (VPC) Flow logging provides built-in power to monitor information about how your network resources are operating in Amazon Web Services. Rather than the old days of collecting this critical data through add-on applications and services—which add overhead and use compute power—Amazon has brought native flow monitoring to the cloud. It is an equivalent of netflow monitoring in the on-premise world.
VPC Flow logging lets you capture and log data about network traffic in your VPC. VPC Flow logging records information about the IP data going to and from designated network interfaces, storing this raw data in Amazon CloudWatch where it can be retrieved and viewed. VPC Flow logging is critical for security and compliance in your AWS cloud environment.
Below is a look at the uses of VPC logs, how to enable VPC flow logging in AWS, how to view and use the data it collects, and its limitations.
Uses for VPC Logging
Performance. Use VPC flow logs to identify latencies, establish performance baselines and, and tweak applications. VPC flow logs can reveal flow duration and latency, bytes sent which allows you to identify performance issues quickly and deliver a better user experience.
Security. By logging all of the traffic from a given interface or an entire subnet, root cause analysis can reveal critical gaps in security where malicious traffic is moving around your network. Key in on suspicious traffic and tighten security loopholes using the information from VPC flow logs.
Catch the flow: Enable VPC logging
By default VPC is not enabled. There are two different methods for turning on logging and capturing your network flow logs in Amazon Cloudwatch:
- Command line. For more advanced users, flow logging can also be enabled and configured from the AWS Command Line Interface (CLI), a unified scripting tool for managing all of your AWS services.
Though it may be tempting to enable flow logs for each and every resource on your network, do so judicially. Flow logs can quickly swell into the hundreds of gigabytes and there is a capture and storage fee for this mountain of data. Work with your Devops/operations team to determine what flow logs are beneficial and check Amazon Cloudwatch pricing to plan your budget.
AWS Log Analytics App for Sumo Logic
See how the AWS Log Analytics App for Sumo Logic provides full-stack visibility.
Three Kinds of Flow Logs
After enabling VPC Flow logging in AWS, it’s important to understand what is monitored and how the logs compile data. Amazon offers flow logging at three separate levels:
Virtual private cloud. Monitor all the activity within your cloud environment for a bird’s eye view of your operations, but note the pricing above. Analysis of VPC logging should reveal popular or vulnerable resources to watch closely moving forward.
Subnet. VPCs are often divided into subnets spanning to multiple availability zone in the region. Subnet can be a private or a public subnet. Private subnets isolate internal resources from public-facing traffic, among other uses. Public subnets require an elastic IP to communicate to the Internet. You can create a flow log for a specific subnet where you may want to monitor all activity. In this example you want to monitor flow logs to ensure there is no internet traffic going to the private subnet.
Network Interface. One can monitor specific interface on EC2 instance and capture flow logs from an interface. Capture full flow logs from critical connection points in your network to stay ahead of issues like latency and malicious intrusions.
After choosing what resources you’re going to log, define the logging parameters. These include:
- Traffic type. You can filter by all, accepted, or rejected traffic.
- Log name and destination. Specify a functional name for the log and where to store it in CloudWatch.
- Necessary permissions. Make certain the log owner has identity access management (IAM) privileges to publish and work with the flow log.
After setting up a flow log for a given resource, scaling is simple. The rules you outline will automatically replicate to additional instances, saving you the time and trouble of duplicating flow logs.
[Learn More: AWS Monitoring]
Limits to the Flow
VPC flow logs can’t capture everything. Certain types of traffic are excluded from VPC flow logs. A few instances where you can’t rely on VPC logging:
- DNS traffic. If you’re running your own DNS server, request resolution traffic can be logged. But many users rely on internal AWS DNS servers, and activity between the servers and AWS DNS services will not be captured by VPC flow logs.
- DHCP. Similarly, dynamic host configuration protocol (DHCP) traffic is not recorded. Depending on the size of your VPC this can represent a notable amount of traffic.
- Multiple IP Addresses. Sometimes a virtual NIC will pool IP addresses for better performance. Flow logs only displays traffic on the primary address even though traffic is destined for the secondary IP address.
- Legacy limitations. AWS instances prior to December 2013 running in the EC2 Classic format are not compatible with VPC Flow logging. Consider migrating to the current AWS format.
Get Your VPC Flowing
With VPC Flow logging Amazon adds a powerful deep analysis tool for your AWS cloud, including in a DevOps environment. Knowing how to turn it on, what critical data to collect, and what you can’t find in your VPC logs is a step in the right direction toward mastering AWS VPC logging.
Learn more about about logging and security in AWS and create a free trial account today to get started.
