Traffic Engineering and Characterization of High-Rate Large-Sized Flows

Abstract

High-rate large-sized (α) flows have adverse effects on delay-sensitive flows. Research-and- education network providers are interested in identifying such flows within their networks, and directing these flows to virtual circuits. To achieve this goal, a design was proposed for a hybrid network traffic engineering system (HNTES) that would run on an external server, gather NetFlow records from routers, analyze these records to identify α-flow source/destination address prefixes, configure firewall filter rules at ingress routers to extract future α flows and redirect them to pro- visioned virtual circuits. This thesis presents an evaluation of this HNTES design using NetFlow records collected over a 7-month period from four ESnet routers. The results show that the HNTES effectiveness was above 90% for NetFlow records collected at edge routers, which corresponded to file downloads from Department of Energy (DOE) laboratories, while the effectiveness was lower for peering routers whose NetFlow records corresponded to file uploads. With further investigation, we found that uploads were less frequent and involved fewer source/destination pairs than downloads. The thesis also describes an algorithm for characterizing the size, duration, average rate, and frequency of α flows, from NetFlow records. The algorithm was validated using independently collected usage logs from application servers. This algorithm can be used in a network management system for providers interested in these types of flows, such as research-and-education network providers whose customers move large scientific datasets. We executed the algorithm on the same NetFlow records used in the HNTES evaluation. Flows moving datasets as large as 811 GB and at rates as high as 5.7 Gbps were observed. Some source-destination pairs were found to repeatedly create α flows. An analysis of the rates of the 1596 repeated α flows created by one pair showed considerable variance, with minimum rate of 100 Mbps, maximum rate of 536 Mbps, and a coefficient of variation of 30%.