Getting the ET/BWMGR product

The ET/BWMGR is sold as a complete, integrated hardware solution, or as a software image (AKA "Software Appliance") that can be installed on your own hardware.

If you've purchased a complete hardware solution, then the system is already licensed, the bridge has been set up and tested, and the setup program will run automatically when the unit is powered on. Once you have set up networking as per the Quick-Start Guide, you can skip over any portions of this manual that are marked "Software Appliance Only."

Product Demos

We provide a downloadable image that can be "burned" on to a USB flash drive, which can serve as a platform for evaluating the product before a purchase, as well as the installation base for a Software Appliance. For more information on system requirements and instructions for downloading and writing the USB image, see the Appliance USB Demo page.

Buying an Appliance

There are many advantages to an full Hardware Appliance vs the Software Appliance version of the product.

• Compatibility: Our appliances run on hardware that we've rigorously tested and fine-tuned. No need to worry if your particular network card or motherboard chipset will perform well (or work at all!) when purchasing a Hardware Appliance.
• Performance: Our systems are optimised to perform on our hardware - we've spent a lot of time tuning performance and drivers, which is critical at the high end of performance.
• Support: We have hundreds of customers using the same hardware and Operating System, drivers, and software, which makes it possible for us to replicate any issues that may arise without the complexity of eliminating untested hardware. With our Hardware Appliances, you can have peace of mind that such an important component of your network is a solid product with full support.

Software Appliance

The Software Appliance is an appliance image that is built from the ET/BWMGR Appliance USB Demo. The image is downloaded and "burned" to a USB Flash Drive. For initial testing, the product can be run entirely from the USB drive. If you decide to purchase the product, or wish to do longer-term testing with a Demo or 30-day License, you can install to hard drive. The USB Demo allows you to make sure that the product works on your particular hardware before committing to a purchase of a license, and also allows you to test the product features before purchasing a full Hardware Appliance.

Setting Up Bridging (Software Appliance Only)

The most efficient way to use the ET/BWMGR is as a dedicated bridge. Bridging allows you to put your ET/BWMGR appliance between any two segments without having to worry about renumbering your network. A system with 3 ethernet interfaces: 2 dedicated to the traffic flow, and 1 for the administrative port is recommended. All of our hardware appliances are sold with this configuration.

Note that when you first boot a demo appliance, bwmgr_setup will run automatically, and prompt you to enter your network settings, including selecting ports for bridging. You can re-run the bwmgr_setup utility to set up networking at any time. Typically, bridging doesn't need to be re-configured after the initial setup.

Bridge

The bridge is similar to the if_bridge bridge implementation which is included with the FreeBSD operating system, with some customization for use with the ET/BWMGR. It is loaded by default when you load the etbwmgr module.

The following section describes how to set up a bridge manually, so you'll know how it works if you need to make manual adjustments.

Configuring a Bridge manually

Configuring a bridge in FreeBSD requires the creation of a bridge virtual interface. While the OS allows for name flexibility, the ET/BWMGR requires that bridges be named bridge# where '#' is a number from 1 to 8, representing the bridge group number. While you could use ifconfig to create the bridge, the ET/BWMGR implements a command to do it which allows you to set the MAC address, allowing it to persist across reboots.

# bwmgr bridge0 createbridge 00:00:00:17:12:11

This is significant because in FreeBSD 7+, the bridge MAC address is used rather than the ethernet address for the local address. If you create the bridge with the ifconfig command, it will generate a random address, which has the consequence of giving the system a different address every time the machine is restarted. Using the ET/BWMGR command allows you to assign a static mac address for the system. The number of the bridge is also the bridge group number, which you'll need to add members to the bridge.

To save the bridge so that it will be set up on boot, store it in /etc/rc.bridge. To save the running configuration:

bwmgr rebuildbridging > /etc/rc.bridge

NOTE: If you create a bridge with no address, the ET/BWMGR will save the randomly generated address in the startup file, /etc/rc.bridge, and that same MAC will be used for subsequent boots.

To add members to a bridge, you can use the bwmgr command-line utility:

# bwmgr BRIDGE bridge addif IFAC [ ff | learn]

This example adds em2 and em3 to bridge0.

# bwmgr bridge0 bridge addif em2 ff
# bwmgr bridge0 bridge addif em3 ff

If you need to assign an address to the bridge, you assign it to the bridge virtual interface bridge0 that you created rather than a bridge member.

# ifconfig bridge0 192.168.1.25 netmask 255.255.255.0

There are only 2 flags supported; ff for FastFoward, and learn for Learning. STP support has been removed.

• Learning sets up a standard "learning" bridge, in which tables are maintained to match MAC addresses to bridge paths. Learning should be used on bridges with more than 2 ports, or bridges with a lot of local-only traffic on the same segment as the BWMGR, because a learning bridge won't process or forward the local traffic.
• FastFoward is a feature that we've added, and is the default setting for bridges with only 2 members, where the bridge is dedicated to forwarding across a single path. In this scenario, there's no reason to "learn" addresses, as we know that all traffic is to be forwarded. With no tables to be maintained, overhead is reduced.

System Configuration

Setting the Time Zone

The time zone must be set for both the Operating System, as well as in the PHP configuration. If the time zone is not properly set, usage graphs and other features like quotas and bandwidth usage reports may not work accurately.

System Time Zone

In FreeBSD, you can set the system time zone from the command line, using the "tzsetup" program.

# tzsetup

This will bring up a series of text dialog boxes, which can be navigated by the arrow keys. Choose the proper geographic area to narrow down the selection list. After confirming the time zone is correct for your location, check the system time and adjust it if necessary. Take note of the time zone selected, and the format (Region / City). This is the format you will need to use when setting the time zone for PHP in the next step.

Examples:

America/New_York
Africa/Dar_es_Salaam
Asia/Dhaka
Europe/Budapest

PHP Time Zone

Edit the file /usr/local/lib/php.ini, and find the line that sets the time zone. A factory-default time zone will be set to the US Eastern time zone like so:

date.timezone="America/New_York"

Change the time zone setting to match the System time zone. After you change the PHP time zone, you should restart Apache so it can register the new setting.

# apachectl restart

Setting the Time and Date

Click on the "System" tab from the main ET/Admin menu, then select the "System Time" link. You will see several fields where you can select the current Date, Month, Year, Hour, Minute, and even seconds. Make sure the date and time are current and click the "Apply" button to change the system time.

Setting the Indexing Level

The ET/BWMGR incorporates a powerful algorithm which indexes rules for ultra-fast lookup. There are 4 indexing levels available for non-firewall rules, both of which use the -addr (IP Address) part of the rule as the index. The default, level 4, should be used for most installations. Levels 1,2 and 3 have less granularity but can be used for rulesets which have a lot of subnet rules. Firewall setups with extensive blocks on a wide variety of rules can be made much more efficient by changing the indexing. Indexing can be summarized as follows.

Suppose your address rule is A.B.C.D:

Level 4 - Uses A.B.C.D
Level 3 - Uses A.B.C
Level 2 - Uses A.B
Level 1 - Uses A

Rules that have less precision than the index cannot be indexed and are therefore Global rules, all of which must be checked for each packet received.

This rule can be indexed for levels 1,2 and 3 (full 255 mask) but not for level 4.

# bwmgr em0 -addr 192.168.1.0 -addrmsk 255.255.255.0

Each interface can index at a different level, although best practice is to have all of your rules on a single interface - typically, the "outside" interface. Since the default Indexing Level is 4, no change is necessary for most installations. The GUI (and the "rebuild" command) will always generate a command line to set the index level. The syntax is as follows:

# bwmgr em0 indexlevel 4

This sets the index level on interface em0 to 4. Note that this setting MUST be applied before adding any rules to the interface. The driver will reject an attempt to change the level if there are rules presently defined.

Firewall rules are also indexed, with 5 index levels available. 1,2,3 and 4 are the same as for the regular rulesets, using the IP address. A 3rd choice is available, ports, which uses -port. Port indexing is useful if you are firewalling by port rather than by address. Address is more effective, so if you use both ports and addresses, there is no need to use the port indexlevel. You set the firewall indexlevel using the fwlevel keyword:

# bwmgr em0 fwlevel 1|2|3|4|ports

Analyzing your ruleset

You can check the efficiency of your ruleset with the analyze command:

# bwmgr em0 analyze

stdClass Object
(
[firewall_global] => 0
[firewall_indexed] => 0
[firewall_worst] => 0
[bandwidth_global] => 3
[bandwidth_indexed] => 20
[bandwidth_worst] => 3
[cmd] => analyze
)

This shows that the current ruleset has 3 global rules that have to be checked for every packet that arrives. Keeping the number of global rules down is the key to efficiency. The "Worst Case" is the rule which requires the most processing. Worst case is always higher than the number of global rules, which is "every case".

Note that the definition of the term "global" is important here. In this context, a global rule is any rule that is not indexed. So all rules that do not have an IP address or MAC address are global rules. For example, a rule that matches a protocol is a global rule, since there is no way to index it. Secondly, a rule tagged -global is NOT a global rule if it has an IP or MAC address. So in that sense, the Global rules showed have no direct correlation to rules tagged -global in your ruleset.

Setting the maxbuffers parameter

The maxbuffers parameter will put a cap on the number of buffers that the bandwidth manager will use to store managed data in its limiting or prioritization queues. The purpose of this parameter is to keep the bandwidth manager from exhausting your system of network buffers. It can be also used to limit the amount of backup traffic that the bandwidth manager will manage. Typically, this number should be somewhat lower than the number of system buffers you have available for network frames.

You can check to see how many buffers are allocated in your running kernel with the following command, which will print the number of mbuf clusters allocated in your system:

# sysctl -a | grep clusters
kern.ipc.nmbclusters: 40000

To change the setting, you can set the number of clusters to use at boot time by putting a line like the following in /boot/loader.conf

kern.ipc.nmbclusters="60000"

If you're unfamiliar with 'BSD architecture, BSD systems use mbufs for network traffic. mbufs are chained, which seemed convenient when we had 50Mhz memory busses, but now most high speed traffic use mbuf Clusters. Clusters are contiguous 2K blocks; when traffic arrived via ethernet, the data is always stored in a cluster.

Remember that each cluster is 2K, so you can't allocate an infinite number of buffers. Previous to FreeBSD 9, FreeBSD allocated a fixed amount of ram to the kernel; 1/3 of system ram by default in most cases. FreeBSD 9 is a bit smarter; but note that mbuf clusters are NOT pre-allocated. So if you allocate 10 million buffers it won't tell you that it can't do it. It will allocate them in bunches as needed, and if your system runs out of memory, bad things will happen.

After determining the buffer allocation in your system, you can then use "netstat -m" after initial boot to see how many buffers your system uses with no traffic (this will vary depending on what cards and devices you have in your system. Suppose your system uses 700 clusters after boot. You could set your maxbuffers setting to say 15000 (which is less than 16000 - 700).

You can see the maxbuffers setting using the 'bwmgr status' command in the CLI, or in the Settings tab in the GUI.

# bwmgr status

stdClass Object
(
[bwmgr_status] => stdClass Object
(
[version] => 5.0
[build] => 11
[bwmgr_running] => 1
[module_time] => 1359499554
[unburst] => 10
[track_all] => 1
[small_pkt_pri] => 0
[stats_period] => 20
[max_streams] => 25000
[hi_streams] => 585
[mem_used] => 1457408
[streams_in_use] => 25
[max_buffers] => 35000
[bufs_in_use] => 1800
[high_bufs] => 4631
[count_headers] => 0
[appliance_type] => R2800C
[license] => 1358034290
[license_limit] => 130
[expires] => 1388552400
[sniffing] => enabled
[urlparams] => enabled
[boottime] => 1359553483
[cpu] => Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz
)

[cmd] => status
)

The output shows the maxbuffers is set to 35000, and that 1800 buffers are currently in delay queues. It also shows high_bufs at 4631, which is the maximum number of buffers used at any time since the last boot or reset. You can use high bufs to determine if you need to tune the setting.

To reset high_bufs, issue a set command from the CLI with a setting of 0.

# bwmgr set max_buffers 0

Note that max_buffers will show 0 until the bwmgr is started; the queues are allocated dynamically.

Setting max_streams

"Streams" are traffic flows, similar to sessions; "conversations" tracked by the software. Streams are allocated dynamically, so they use up memory. They're not completely required (without streams every packet has to be processed more rigorously), so the max_streams setting allows you to set the maximum number of streams that can be allocated at once. A large ISP can easily have over 100K streams. If you see your high streams reaching your max streams setting, you can increase max_streams as long as you have memory available.

# bwmgr set max_streams 55000

The above would set Max Streams to 55000. Note that this also resets high streams, so if you simply want to reset high streams, you can do so by setting max_streams to the current value.

Starting the Bandwidth Manager (Software Appliance Only)

You can start the bandwidth manager process with the following syntax::

bwmgr_license start

This requires that you've installed a valid license in /usr/bwmgr/BWMGR_LICENSE. If you purchased a hardware appliance the license is pre-installed.

Running in Test mode (Software Appliance Only)

Before you purchase a license, you should test your target hardware with our USB Demo. You'll need to run the sofware in "test mode" by using the word test as the key, like so:

# bwmgr_license start test

Test mode will pass data and gather statistics but it will not do any traffic shaping, and it will run for one hour. It's meant for you to be able to verity that your hardware works with the software. Once you've verified your hardware, you can request a demo key, which is more functional and runs for a longer period of time.

Requesting a Demo License (Software Appliance Only)

Demo mode provides full functionality for a 100Mb/s license, but will only run for 4 hours before requiring a reboot. It's meant to allow you to test the software in production mode for long enough to determine if the product meets your needs

Before requesting a Demo License, you will want to install the product to hard drive. Once you've booted from the hard drive, you can generate a license request.

cd /usr/bwmgr
bwmgr_license license-request

This generates a file named BWMGR_LICENSE_REQUEST.######### in the current directory. You'll need to send the contents of this file to us using our Sales Ticket System, and we'll issue you a license with instructions to install it.

The demo will run for 4 hours. When the demo concludes, a message will be displayed indicating that it has completed. Once the demo completes, traffic will no longer be controlled, but bridging will continue to operate. In order to restart the software, you will have to reboot the system. You will be able to run the full demo for approximately 20 days. Once you've determined that you want to buy a full license, we can generate a license using the license request file you've already submitted. All you will need to do is place the order using our online store.

Installing Your License

On a new Hardware Appliance, the license is pre-installed. However, if you are installing an updated license, either because you have upgraded the license speed, or you are renewing your support subscription, then you will need to install the license when you receive it.

The license file is /usr/bwmgr/BWMGR_LICENSE. Your license will be text similar to the following:

######################################################################
Q19x9lq+XDp1Huwuc1gjgpJ0jt94mZ6av0oz5f3D/D3047OSww/VtlCdDqx1
6SfbIbXetvQrP64W2ujby823LuNg/hoUVH8lUu25Q6YlHXz8wVB9IiF72M
NoNh8NZ4gs9WuQ7VK1KDx+llo0xNCWnbZWGsbFHfgmkrblMCosXF0J+AgZnJ
k8d5mq13GOBUDnlEoXmzoN0LwZTi9PUTeD==
######################################################################

Paste the entire text into the file; create the file if necessary. Make sure your cut/paste doesn't add any strange characters to the file, including leading or trailing spaces.

Adding Rules

The bwmgr rule syntax is very extensive, and will be illustrated throughout this document in the appropriate section. The base syntax for adding rules can be summarized in the following:

bwmgr interface [-x rulenumber] criteria [bandwidth controls / actions] [options]
criteria is the method for matching specific traffic, the actions can refer to a bandwidth limit or priority, and options refers to such features as statistics.

Basic Criteria

-x INDEX
Index Number (precedence)

-add_before INDEX
Use first available index before rule number INDEX

-fw
Rule is a firewall rule

-name
Rule Name (required for stats collection)

-global
Global Rule

-i
Match incoming traffic

-o
Match outgoing traffic

-d
Match no traffic (disable rule)

-l
Enable packet logging on this rule (use with caution)

-addr IPADDRESS
Match Host IPADDRESS (source or destination)

-addrmsk MASK
Apply net mask to -addr IP setting

-daddr IPADDRESS
Match Destination Host IPADDRESS

-daddrmsk MASK
Apply net mask to destination IP Address

-saddr IPADDRESS
Match Source IP

-saddrmsk MASK
Apply mask to source IP address

-daddr IPADDRESS
Match Destination Host IPADDRESS

-daddrmsk MASK
Apply net mask to destination IP Address

-maddr MACADDRESS
Match MAC address

-smaddr MACADDRESS
Match Source MAC address

-dmaddr MACADDRESS
Match Destination MAC address

-port NUM
Match source or destination port NUM

-dport NUM
Match destination port NUM

-sport NUM
Match source port NUM

-portrange ARG1-ARG2
Match port range between ARG1 and ARG2 inclusive

-url TEXT
Match TEXT string in http request

-vlan ARG
Match VLAN ID ARG

-tos HEXARG
Match TOS Field

-ipprot protocol
Match protocol

-site sitename
Match site domain name

Throughout this document the above criteria will be illustrated along with details of their use. First, some basics.

Rule Precedence

Each rule has an "index", which is a number which defines its precedence. Rules are processed from low indexes to high, so the lower the index, the higher the precedence. Once a rule "hits", or matches a packet, rule processing stops, and the action (if any) specified by that rule is carried out. Exceptions to this behavior are rules with the "-global" option, or the "-logonly" option. A gobal rule allows for subsequent matches (and actions) for a given packet, which allows for some more powerful combinations of rules and actions. A log-only rule is a rule that performs no action, but simply enters the packet details in the log when it matches.

Use the -x parameter to specify an index:

# bwmgr em0 -x 300 [criteria]

The example above specifies an index of 300. If you add a rule without specifying an index, auto-indexing will assign an index numbered higher than the highest-numbered rule. Auto-indexing ignores rules numbered 1 million or higher - see the next paragraph on Large Rule Indexes for an explanation.

Large Rule Indexes

Auto-indexing will not consider rules with indexes of 1 million or greater when determining the next highest index. This allows you to specify rules that MUST be at the end of your ruleset, while still using auto-index. For example, suppose you have 2 rules:

# bwmgr em0 -x 100 -addr 10.1.1.1 -bwboth 256000
# bwmgr em0 -x 1000000 -ipprot allip -priority discard

Now, you add another rule using auto-index:

# bwmgr em0 -addr 10.1.1.2 -bwin 512000 -bwout 512000

This rule will be assigned index 200, so your rule 1000000 would still be at the end as required.

Global Rules

Before Global rules were introduced in v3.24, rule index precedence was absolute. With global rules, its possible for a data frame to match multiple rules, which is useful both for statistics and control. Global rules are set with the -global option.

# bwmgr fxp2 -x 100 -ipprot tcp -global [ matches all TCP traffic ]
# bwmgr fxp2 -x 110 -ipprot telnet -global [ matches Telnet traffic ]
# bwmgr fxp2 -x 120 -addr 10.1.1.1 [ matches traffic to or from IP address 10.1.1.1 ]

Now suppose a telnet packet originating from 10.1.1.1 is received. If -global was not specified on rule 100, the packet would "hit" and rule processing would stop there. But since 100 and 110 are both global rules, all three rules will "hit" on the same packet.

Note that is makes no difference where the global rule is located in the ruleset; a global rule will always hit on a match, so the following:

# bwmgr fxp2 -x 110 -ipprot telnet -global
# bwmgr fxp2 -x 120 -addr 10.1.1.1
# bwmgr fxp2 -x 300 -ipprot tcp -global

would make no difference; all 3 rules would "hit". Note however this next case:

# bwmgr fxp2 -x 300 -addr 10.1.1.0 -global
# bwmgr fxp2 -x 310 -addr 10.1.1.1
# bwmgr fxp2 -x 320 -ipprot tcp

In this case, the telnet packet would only hit on 300 and 310, since rule 320 is not global. Only the first non-global rule (rule 310) will hit, and it does so by rule precedence.

Naming Rules

Names function as aliases to rule numbers and are used to uniquely identify rules (out of the scope of the interface) for links with other rules. Names must be unique for the entire system. If you will be graphing the data from a particular rule (see the -stats option below), a unique name is required. Also note that names are case insensitive in the database, so if you have rules named "ACME" and "acme" they will be considered the same for graphing (and it will cause problems). Groups must also be named, as must reverse rules. In order to modify or view a particular rule, you need to specify the index number or the name that is assigned to the rule when you create it. To make things easier, you can name entries and use the name to modify or refer to specific rules without having to know the index number. To name an entry, use the -name option:

# bwmgr em0 -name GenMotors -x 300 ...

the above entry will associate the name GenMotors with this rule at index 300. So, for example, if you wanted to modify this rule in the future, you could simply do the following:

# bwmgr em0 -name GenMotors ...criteria...

Note that names must be no more than 16 characters long and cannot include spaces or any non-alphanumeric characters other than a dash or underscore. Using periods in rule names should be avoided.

Specifying Hosts and Network Addresses

You can create rules for host addresses and networks with the BWMGR to limit the amount of bandwidth that a specific host or network can use. This is done as follows:

# bwmgr em0 -addr 207.11.14.1 -bwout 56000 -bwin 56000

This would limit the traffic from and to host 207.11.14.1 to 56000bps. Similarly, an address mask can be used to specify an entire network or sub-network:

# bwmgr em0 -addr 207.11.14.0 -addrmsk 255.255.255.0 -bwout 56000 -bwout 56000

This would limit the bandwidth of all of the devices on network 207.11.14 to an aggregate of 56000bps. You can also use host/domain names for specific hosts. For example, to limit traffic of a specific host by name:

# bwmgr em0 -addr www.sumdomain.com -bwout 128000 -bwin 128000

Note that if you use internet host.domain names, the name is resolved to a host address at the time the rule is added. Assuming "domain.com" resolves to 10.1.1.1

# bwmgr em0 -addr domain.com

this rule will be equivalent to

# bwmgr em0 -addr 10.1.1.1

No attempt is made to resolve numbers to hosts, so the numeric host will be saved in your ruleset.

MAC Address Rules

You may want to lmit an entire physical machine, or a customer by his router's MAC address. Assuming that you know the MAC address of the specific machine, you can specify the MAC address using the -maddr option in the same way that you can limit by IP address. For example:

This rule would limit all of the traffic to and from the host with MAC address 00:00:00:c0:a2:bc to 64kb/s. MAC rules match all of the traffic from or to the specified MAC address, including non-IP traffic.

# bwmgr em0 -maddr 00:00:c0:a2:ca:bc -bwin 64000 -bwout 64000

Alternatively, you can use -smaddr and/or -dmaddr to control a specific connection.

# bwmgr em0 -smaddr 00:00:c0:a2:ca:bc -dmaddr 00:00:c0:01:01:01
# bwmgr em0 -maddr 00:00:c0:a2:ca:bc -bwin 64000 -bwout 64000

The above rule pair would limit all of the traffic for MAC 00:00:c0:a2:ca:bc except traffic going from this address to 00:00:c0:01:01:01.

Grouping Bandwidth Entries - Creating a Group

If you want to define a disjointed group of addresses, you can use the "Group" feature to create a group. Groups MUST be named when they are created. Rules can then be linked to the group by name:

Here, we define a group, and give it a name and a bandwidth setting (128kb/s symmetrical).
# bwmgr em0 -name losers -group -bwin 128000 -bwout 128000
# bwmgr em0 -addr 207.11.1.4 -bwlink losers
# bwmgr em0 -addr 207.11.17.2 -bwlink losers
# bwmgr em0 -addr 207.11.17.3 -bwlink losers

the above would create a group with the 3 specified addresses which would all have the same aggregate bandwidth limits, as if they were all in one rule. Note the use of the -name parameter for the initial (group) entry. You must use the name of the entry when linking to a group head (previous versions allowed the index to be used). Using the name allows for group members to be on different interfaces. Note that if you delete the Group entry the other entries will continue to exist with an unattached link, and they will re-attach when a group rule with the proper name is added. You should avoid deleting group entries when possible, as the remaining rules will work differently (in the above case they will get unlimited bandwidth) when they are not in the group.

More powerful use of groups is discussed in the Group Bandwidth Management section below.

Managing Traffic by Data Type

Traffic can be identified by "traffic type" as well as by previously described IP or MAC addresses. Traffic types can also be specified along with addresses to identify specific traffic sent by or to specific hosts or customers.

MAC Protocol Rules

The "MAC protocol" is the "ethernet type" in an ethernet frame. While virtually all of your "internet" traffic is IP, you may have many other types of traffic on your LAN. If you have co-located customers, if you are bridging your traffic to remote locations or if you just want to control certain types of LAN traffic, the MAC protocol can be used to identify such traffic. The syntax for specifying MAC traffic is as follows:

# bwmgr em0 -mprot Protocol

Where "Protocol" is the hexidecimal representation of the ethernet type (ie 0x800 or 800 for IP, 0x806 or 806 for ARP, etc). The more common MAC types (IP, ARP, IPX) may be specfied by keyword.

There are 2 special types. First is "-mprot vlan", which matches all vlan tagged traffic. Note that such rules will only match vlan encapsulated traffic that contain the vlan header. The second is "-mprot 802.3", which is "typeless" ethernet traffic. Some older protocols (and IPX implementations) may specify the length instead of the protocol type. Specifying 802.3 will match all such traffic, regardless of the higher level protocol.

VLAN (VPN) Rules

You can also match based on the vlan tag (ie the vlan ID number) using the -vlan option. For example:

# bwmgr em0 -vlan 128 -bwboth 256000

will limit traffic containing vlan header info for vlan 128 to 256000. Note that -vlan implies -mprot 0x8100 and that any other setting of -mprot is not valid.

IP Protocol Rules

All IP traffic has an IP protocol. The most common IP protocols are TCP, UDP and ICMP, but there are many more. Most likely, you will not encounter anything different than these "big 3". Other IP protocols are listed in the file /etc/protocols on most unix-like installations.

To match traffic with a specific IP protocol, use the -ipprot criteria. The following are examples of how to match "the big 3":

# bwmgr em0 -ipprot UDP
# bwmgr em0 -ipprot ICMP
# bwmgr em0 -ipprot TCP

Note that port-defined and special internally defined protocols can also be specified with the -ipprot setting. These are discussed in the protocol engine manual.

Other IP Protocols

It's not very useful to shape traffic based on their base IP protocols, so protocols are further decoded based on their content or what TCP/UDP port they're running on. Examples are http, ftp, p2p, video, voip and some derivatives of those protocols, such as http-image and bittorrent.

A more comprensive discussion of IP Protocols, Tags and how to map traffic to protocols is in the Protocol Engine document.

Tags

Tags are described in the Protocol Engine section. Tags are text representations of various traffic flows that have been identified using one or more criteria. To match a tag, use the -tag option with the desired tag:

# bwmgr em0 -tag admins

The would match the tag "admins", which may be the IP Addresses of authorized admins:

ip_address: 10.1.1.1 tag:admins
ip_address:10.1.1.64 tag:admins
ip_address:192.168.11.17 tag:admins

IP TOS Rules

You can use a mask to match bits in the TOS field of ipv4 packets with the -iptos option. Since TOS is used in many different and sometimes non-standard ways, it is your responsibility to set the mask according to whatever scheme is in place on your network. Note also that this HEX value Matches the entire TOS octet, meaning both TOS and priority fields. For example:

# bwmgr em0 -iptos 0x18 -global -priority 8

The above rule will set the global priority (explained later) to 8 for any packets containing either the LOW_DELAY (0x10) or THROUGHPUT (0x08) bits. This implies that 0x10, 0x08 and 0x18 and 0x1E will all match.

IP Protocol Port Rules

While you can use ports in rules to match traffic, its more efficient to use protocol mappings than to use ports. Virtually all port matching has been antiquated by the protocol engine. You can still use -port to match a port if the protocol isn't defined, but its recommended that you use -ipprot in most cases.

"Special" Protocol Definitions

There are several "special" ip protocols defined that arent really IP protocols at all, but they can be quite useful. "Special" protocols are specified with the -ipprot syntax, although some of them actually combine protocols and ports. They represent special cases that are common and they are not definable by protocol and port alone:

tcpconnect Matches SYN requests only

tcpsyn Matches any packet with a SYN (including ACKS)

tcpack Matches packets with an ACK flag

tcppush Matches TCP packets with the PUSH flag (containing data)

allip Matched all IP packets (NOT ARPS or other ethernet prots

ipfragments Matches packets that are fragments


URL Rules

URL rules are special rules that look into HTTP and FTP frames and match requests for specific pages or files. URL rules specifically match 2 types of traffic: HTTP GET or POST requests for specific URLs and FTP RETR requests. These can be useful in filtering known attacks (ie default.ida or root.exe), and also for controlling downloads of specific pages or specific types of pages. Suppose Microsoft just released IE 12.0. You could limit only downloads with a "-url iesetup12" rule possibly. Or "-url .exe" to limit all executables.

URL rules will "track" any session that is encountered using the URL. So, for example, if you wanted to limit the download or tarballs, you could use:

# bwmgr em0 -url .tgz -bwin 64000 -bwout 64000

This would be all that would be necessary to limit the entire download of any file containing ".tgz". ".mp3", ".mpg", ".avi", ".wmv" are other useful choices.

URL rules may contain spaces. If a url contains spaces, it must be enclosed with quotation marks:

# bwmgr em0 -addr 10.1.1.1 -url "The Quick Brown Fox"

There is a setting urldecode "URL Decode" that can be enabled to cause the system to read the parameters string as well as the url. So if you wanted to block an exploit such as "?file=http://www.url.com", with URL decoding enabled, you could use:

# bwmgr em0 -fw -url "file=http" -priority fw-deny

To set url decoding, use the command:

# bwmgr set urldecode on|off

where on enables and off disables this feature.

Site Rules

New in v5.0 is -site rules. These match HTTP host parameter returned in headers. It's important to understand that the site address is NOT contained in the url.

While you might type this in your browser:

http://www.domain.com/someurl.htm

only the "/someurl.htm" is received as data in a packet. The site name is resolved via DNS, so in order to get the site name for a GET request you must get it from the HTTP headers.

bwmgr em0 -site facebook.com

The above rule will match http traffic with a host header specifying facebook.com. Note that the site specifier will match any host that matches the fragment. So "facebook.com" would match both "www.facebook.com" and "advertising.facebook.com".

Enabling and Disabling Rules

You may want to disable a rule temporarily to allow someone into your network or allow someone out. To disable a rule, use the following syntax:

bwmgr em0 disable ####
bwmgr em0 disablefw ####

where #### is the rule number to disable. disablefw is used to disable firewall rules. Note that disable internally flags the rule as disabled, and you must use enable to re-enable the rule (unless you delete it and re-add it of course). To reenable a rule or to change its I/O flags, use the enable keyword:

bwmgr em0 enable 1000
bwmgr em0 enable 2000 I
bwmgr em0 enable 3000 O
bwmgr em0 enable 4000 IO

for the above commands, rule 1000 will be enabled as Incoming and Outgoing (the default), rule 2000 as incoming only, rule 3000 as outgoing and rule 4000 as both. There is no danger in enabling a rule which has not been disabled; in fact you can use enable to change the flags for any rule.

Enabling and Disabling Rule by Time of Day

For an in depth discussion of Time Profiles for rules, please see Time of Day Profiles.

Notes on Logging

Adding the "-l" option enables packet logging for all traffic that matches the rule. Due to its high demands on cpu time and disk usage, this is not a feature that should be enabled on a typical rule. It's a powerful tool for debugging or identifying unusual traffic, intrusion detection, and finding traffic that hasn't matched any previous rules. Excessive logging (such as applying logging to a rule that matches all traffic to a host or subnet) will consume system resources, as every frame that goes to or comes from that host/net will be logged,

More acceptable uses include scenarios like the following:

Suppose you wanted to know what connections a specific employee of yours was making. You could use the following rule:

# bwmgr em0 -addr 207.14.172.1 -prot tcpconnect -l

The above rule would log all TCP connect requests from host 207.14.172.1. By specifying only the tcpconnect IP protocol, you have significantly reduced the amount of hits for this rule.

The information from rules with logging enabled will be sent to LOG_INFO in FreeBSD (kern.info in syslogd). Typically, these are logged to the file /var/log/messages.

ET/BWMGR Protocol Engine

The ET/BWMGR protocol engine is described in detail in a separate document.

The ET/BWMGR Firewall

Setting up the Firewall - Theory of Operation

The ET/BWMGR software incorporates a firewall ruleset which is separate from the bandwidth management rules. Each manageable interface on your system can optionally be assigned firewall rules, which can help you to control the flow of traffic to and from your network.

The purpose of a firewall is to decide which packets are allowed to pass through the network and which packets are not allowed. The default is to allow all traffic. So, if you have no firewall rules enabled, all traffic will pass through the firewall. Typically, you will add rules for traffic that you want to "firewall", or filter, from getting into your network. For example, suppose you don't want anyone on the outside world to access your server whose IP address is 207.11.14.1. You could do the following:

bwmgr em0 -fw -addr 207.11.14.1 -priority fw-deny

the above syntax selects interface em0 (which would have to be the interface connected to the outside world), uses the -fw flag to specify a firewall rule, and selects the address as the match criteria. The "priority" for firewall rules can be fw-deny, fw-allow or log-only. log-only rules will be discussed later. For now, you can either set a rule to allow, or to deny passage through the firewall. The above rule will cause all traffic to or from 207.11.14.1 to be discarded. This results in this address being inaccessable from outside of interface em0.

Firewall Criteria

Firewall matching criteria are similar to bandwidth rules, and include IP address, MAC address (source and/or destination,) port, port range, and protocol. Combinations of criteria are allowed, to allow for very specific matches, such as:

• Destination IP address and Destination Port : match traffic to a port on this IP
• IP Address and MAC Address : Match only if the IP is using the provided MAC

Combinations of rules can also be used to achieve more complex goals. For example, suppose you wanted to deny access to all ports over 1024 on a particular system but allow access to port 4718.

# bwmgr em0 -x 99 -addr 207.11.14.2 -port 4718 -priority FW-Allow
# bwmgr em0 -x 100 -addr 207.11.14.2 -portrange 1024 65535 -priority FW-Deny

Because rule 99 is before rule 100, it will be checked before rule 100, so you have effectively excluded port 4718 from the range in rule 100.

Although Firewall rules use the same criteria as bandwidth rules, certain options, such as bandwidth management and statistical parameters are not allowed.

Viewing Firewall Rules with bwmgr

To view a firewall rule, use the getfwrule CLI command

# bwmgr em0 getfwrule 100

stdClass Object
(
[ruleinfo] => stdClass Object
(
[ifname] => em0
[incoming] => 1
[outgoing] => 1
[fw] => 1
[index] => 100
[idx] => 6870
[hits] => 0
[priority] => Discard
[drops] => 0
[addr] => 11.1.1.1
[addrmsk] => 255.255.255.255
[bwinfo] => stdClass Object
(
)

[burstmax] => 0
[bps] => stdClass Object
(
[bps_in] => 0
[bps_out] => 0
)

)

[cmd] => getrule
)

Log-only Firewall Rules

Firewall rules can be given a priority of "log-only", which means that a log entry will be written when the rule matches, but subsequent rules will still be checked. It is important to position log-only rules before allow/deny rules, because once an allow or deny rule hits, then no more rules will be checked. A good strategy is to put all of your log rules at the beginning.

# bwmgr em0 -x 10 -fw -prot tcp -port snmp -l -priority log-only

You must specify -l to enable logging in addition to log-only, it is not automatic.

Reverse Firewall Rules

Reverse firewall rules allow you to use a "trigger rule" to block host addresses based on some action. There rules are described in detail in the reverse rules section below.

Setting up Bandwidth Management Rules

Setting Bandwidth Rules and Limits with the bwmgr utility

Bandwidth managment rules allow you to customize the way that certain traffic flows through your network. The following section describes how to use the bwmgr utility to set up your bandwidth management criteria. We will start with a brief listing of all of the options that apply limits or bandwidth settings to a rule. See the ET/BWMGR v5.0 CLI Manual for brief descriptions of each setting.

-bwin ARG
-bwout ARG
-bwboth ARG
-bwmin ARG
-burstmax ARG
-bwburst ARG
-bwburst_in ARG
-bwburst_out ARG
-bwprofile ARG
-burstthresh ARG
-burstthresh_in ARG
-burstthresh_out ARG
-burstperiod_in ARG
-burstperiod_out ARG
-min_window ARG
-autothresh_in ARG
-autothresh_out ARG
-autoperiod_in ARG
-autoperiod_out ARG
-bursttrig ARG
-ppsin ARG
-ppsout ARG
-ppshi ARG
-wdwthresh ARG

Shared vs Allocated Bandwidth

All bandwidth that is not guaranteed is considered by the ET/BWMGR to be "shared" bandwidth. What this means is that bandwidth that is not used by one rule or policy may be used by any other rule on a demand basis. Guaranteed bandwidth must be allocated for a specific policy and will not be available to other policies at any time.

Allocated Bandwidth vs Minimum Bandwidth

"Allocating" bandwidth is a meaningless concept unless there is a finite pool from which to allocate from. If a rule with a minimum bandwidth allocation is a member of a Group, the bandwidth must be "allocated" from the Group's bandwidth pool, in order to guarantee that other members can't use it. A rule that is not a member of a group will have its bandwidth allocated from the pool of bandwidth available to the Interface. The Group or Interface is said to be the rule's "Container", because it defines the amount of bandwidth available to all of the "members" within its scope. If there is no bandwidth allotment for the rule's Container, then the bandwidth can not be "Guaranteed", because there is no reference to the amount of bandwidth available. In this case, the "guaranteed" bandwidth is simply a minimum bandwidth setting.

Auto-Shaping

The ET/BWMGR includes an "auto-shaping" feature designed to automatically manage network congestion without any "rules" whatsoever. Auto-shaping can be used alone or in co-ordination with any other settings.

To enable auto-shaping, simply set -autothresh on an interface (typically, the outside interface):

# bwmgr em0 -ifac -autothresh_in 5800000 -autothresh_out 5800000

Typically you would set this to just below your upstream capacity. So if you have 6Mb/s of upstream bandwidth, set the threshold to 5800000. Whenever your network traffic is over the threshold, the auto-shaping mechanism will "kick in" and begin slowing all of the traffic on the network in a very "fair" manner.

Note that auto-shaping is a pacing mechanism only, and will not apply any hard limit or delay any traffic other than by natural means (such as the TCP window). So your bandwidth will likely exceed the threshold, but your congestion (ie backup traffic, queue depths, etc) will be much less.

Setting Hard Limits - Allocating Shared Bandwidth with a Rule

The simplest type of bandwidth management rule is the hard limit. Hard limits set a ceiling on the amount of data that a host or type of traffic can use, and will regulate it at the ceiling when operating over the limit. This effectively simulates any data rate you choose regardless of the medium. Limits can be set for incoming or outgoing traffic, or for incoming and outgoing combined.

# bwmgr em0 -x 500 -addr 100.1.1.1 -bwin 56000 -bwout 128000

This command sets a bandwidth limit of 56000bps for incoming traffic and 128000bps for outgoing traffic. "Incoming" traffic is defined as by viewing direction as relative to the "Outside" bridge interface. Typically, all rules are added to the Outside interface, which is also set as the Default interface, so when you log in to the GUI, you will see the rules on the correct interface.

To set a combined setting, use -bwboth as follows:

# bwmgr em0 -x 500 -addr 100.1.1.1 -bwboth 64000

This setting will allow the total of incoming + outgoing traffic to be no more than 64000bps.

Minimum Bandwidth Allocation

"Allocating" bandwidth can be achieved by setting a bandwidth minimum, which defines the minimum data rate that a host or traffic type can use before any action will be taken on the data stream. When the rules container (Group rule or Interface) has a bandwidth setting, this setting will represent "guaranteed" bandwidth. Without an interface or Group bandwidth setting, this value represents bandwidth that will be considered high priority.

Suppose you have VoIP running on udp port 15100. This setting will set the minimum bandwidth to 64kbs for the port, giving it the highest priority until the minimum is exceeded.

# bwmgr em0 -ipprot udp -port 15100 -bwmin 64000

To see this in context, these two rules use a -bwmin value to allocate a bandwidth guarantee of 64kb/s out of the 1.54Mb/s allocated to the interface.

# bwmgr em0 -ifac -bwin 1544000 -bwout 1544000
# bwmgr em0 -ipprot udp -port 15100 -bwmin 64000

You can combine all of the bandwidth settings in a single rule. Note that -bwmin and -bwburst (discussed below) inherit the "combined" attribute if -bwboth is specified. So, for example, if -bwboth is specified, then -bwmin and -bwburst apply to the combined (incoming + outgoing) total, if -bwboth is not set, then itt is applied for each direction.

Burst Settings on the ET/BWMGR

The ET/BWMGR bursting feature allows for 2 levels of limits for each defined rule - the "normal" limit (-bwin / -bwout), and a higher "burst" limit. In order to define a burst bandwidth allocation for the rule, you must define two additional parameters, the burst limit and the burst trigger.

The burst limit is defined by the -bwburst parameter for a rule. The "trigger" is set by the -bursttrig parameter. The trigger must exist before the burst rule is created.

Defining a Burst Trigger

Any rule with a Burst Threshold (-burstthresh_in / -burstthresh_out) can be used as a burst trigger. A "burst threshold" is a bandwidth level that is used as a criteria for determining whether any burst rule(s) that use this rule as a trigger is allowed to burst, or if the normal limit is enforced.

# bwmgr em0 -x 500 -name alltraffic -global -stats -burstthresh_in 10000000 -burstthresh_out 10000000

This creates a global rule that matches all traffic, sets the burst threshold to 10Mb/s, but performs no limiting. It's an efficient way to store statistics and set up a burst trigger on the interface, without setting up an interface rule, while also configuring the necessary trigger. Any rule that specifies alltraffic as the burst trigger, will allow the higher "burst" limit when traffic crossing the em0 interface is under 10Mb/s, and their "normal" (-bwin / -bwout) settings when em0 is passing more than 10Mb/s.

# bwmgr em0 -x 3000 -name fools -group -burstthresh_in 800000 -burstthresh_out 8000000

This rule defines a group with a burst threshhold of 800000bps. So any rule that defines "fools" as their burst trigger would burst when the traffic level of Group "fools" was under 800000bps.

# bwmgr em0 -x 3200 -addr 211.14.1.1 -bwin 64000 -bwout 64000 -bwburst_in 256000 -bwburst_out 256000 -bursttrig fools

The above rule specifies that traffic to and from address 211.14.1.1 will be controlled at 64000bps if the usage level of the group "fools" is over 800000bps, and that 211.14.1.1 will get 256000bps when "fools" is below 800000bps. In this example, the rule 3200 is not a member of the "fools" group. There is no requirement that a burst rule that uses a group rule as a trigger actually be in that group, although it might often make sense for that to be the case.

Setting the Maximum Burst Duration

The -burstmax parameter allows you to set a maximum length of time in seconds that a rule can burst. This is designed to prevent long downloads or uploads. With a burstmax setting of 10, for example, the rule will allow "bursting" (which is activity over the standard bandwidth setting) for 10 seconds, and then will not allow bursting again until the rule has activity under its normal bandwidth allotment. So with a continuous download, the rule will be limited to its regular bandwidth setting (non-burst) until the transfer is completed.

Changing The Burst Reset (unburst) Period

Rules using the -burstmax setting need a criteria for determining when the traffic has subsided. The default setting is 10 seconds. What this means is that once a rule has bursted for its max burst period, the traffic must slow below its non-burst limit for at least unburst seconds before its allowed to burst again.

# bwmgr unburst 20

Sets the Burst Reset to 20 seconds.

Defining a Burst Limit for a Bandwidth Rule

Any rule can be given a -bwburst setting which defines the maximum amount of bandwith the rule can use when the burst trigger is below its threshold. When traffic levels on the burst trigger are above the threshold, bursting is disabled. When traffic levels are below the threshold, bursting is enabled. Note that starting in v3.21, the "default" burst is wire speed (ie no limit). In previous versions if you didnt specify a -bwburst setting for a rule, the default was to limit to the normal bandwidth setting all the time. Now you must explicitly set the burst limit, using either -bwburst, or -bwburst_in and -bwburst_out.

# bwmgr em0 -ifac -burstthresh 1000000

# bwmgr em0 -x 800 -addr 100.1.1.1 -bwin 56000 -bwout 56000 -bwburst_in 512000 -bwburst_out 512000 -bursttrig em0

# bwmgr em0 -x 900 -addr 100.1.1.2 -bwin 128000 -bwout 128000

# bwmgr em0 -x 1000 -addr 100.1.1.3 -bwin 128000 -bwout 128000 -burstmax 20 -bursttrig em0

The above settings would set the burst threshold to 1000000bps on interface em0. The second setting (-x 800) would set traffic to and from address 100.1.1.1 to 56000bps when traffic on the interface is above 1Mb/s, and to 512000bps when traffic is below 1Mb/s. The third rule (-x 900) sets a hard limit on address 100.1.1.2 to 128000bps (no -bwburst setting). The third rule (-x 1000) allows bursting to wire speed (ie no bandwidth limit), with a below-threshold limit set to 128000bps, and a maximum burst duration of 20 seconds.

Note that bursting is effectively disabled for a rule if no burst trigger is defined.

Setting Asymmetrical Burst Rates

New in v5.0 is Asymmetrical Bursting. You can now define separate incoming and outgoing burst values, for both triggers and rules. Suppose you wanted to set a global trigger for your network with bursting allowed when the incoming bandwidth was 800K and outgoing at 250K.

# bwmgr em0 -x 100 -global -name GlobalTrigger -burstthresh_in 800000 -burstthresh_out 250000

# bwmgr em0 -x 1000 -name AcmeWidgetCorp -bwin 512000 -bwout 512000 -bwburst_in 600000 -bwburst_out 600000 -bursttrig GlobalTrigger

Setting a Burst Period

The "burst period" is the window within which the BurstManager will gather bandwidth information about a trigger and enable or disable bursting. Traffic usage is continuously averaged over the period. Longer periods may increase the amount of time that rules may burst, as the longer into the period high activity begins, the longer it will take the average to reach the threshold. Note that values under 10 seconds are not recommended.

Interface Level Bandwidth Control

You may want to limit a complete interface to a specific amount of bandwidth, regardless of the traffic type. However, before doing so, see the section immediately below on Global Limits. You can set a bandwidth limit for the entire interface by specifying the -ifac option. For example:

# bwmgr em0 -ifac -bwin 800000 -bwout 800000

The above command will set an overall bandwidth limit of 800000 bit/s in and out for the interface. This limit is absolute (except for pass-thru rules), so no more than 800000 would be allowed to pass regardless of other rules present. This implies that:

# bwmgr em0 -ifac -bwboth 800000
# bwmgr em0 -addr 10.1.1.1 -bwin 1000000 -bwout 1000000

with the above rules, host 10.1.1.1 would not be able to get more than 800000 even though its individual limit is higher.

Interface bandwidth limits are useful for defining how much actual bandwidth there is available at your network edge. For the purposed of priorities and bandwidth allocation, you may set the outside interface (ie the adapter connected to your edge router's side of the network) to the bandwidth you have (such at 45Mb/s if you have a T3), or the maximum bandwidth you want to use. With an interface bandwidth limit, minimum settings become guaranteed bandwidth on the interface. Without an interface limit, you are only guaranteeing access to the local wire, which may not be what you want to do.

Note that if you have no rules on an interface, you could alternatively use:

# bwmgr em0 -bwboth 800000

as a single rule which limits all the traffic on the interface without setting an interface limit.

Using Global Limits INSTEAD of Interface Limits

Interface limits have an additional overhead, and sometimes its just better to set a Global limit. It performs the same function, and it's a little bit cleaner internally. Simply set a global rule that matches everything as your trigger.

# bwmgr em0 -x 100 -global -name GlobalTrigger -burstthresh_in 800000 -burstthresh_out 250000 -bwin 1000000 -bwout 1000000

Set a global limit and also define a trigger for the entire interface. In this example, we have 1Mb/s limit, and bursting is disabled when overall traffic reached 800Kb/s

TCP Rate Limiting

TCP rate limiting is a powerful bandwidth management technique that reduces the amount of traffic in your network and potentially slows each TCP session by reducing the TCP receive window. While some products on the market use rate limiting as their primary bandwidth management mechanism, there are undesireable results when windows are dropped too low, as overhead on your network in terms of the number of packets and bandwidth usage can significantly increase. Because of this, the ET/BWMGR uses rate limiting as one of several mechanisms to manage your network without having an adverse effect on overall traffic volumes.

Disabling Window Shaping

Window shaping can slow a connection too much if the delay for the connection is high, causing a stop and wait situation. Typically, ACKs are sent well before a full window can be transmitted, so traffic simply slows down. When shortening the window causes a big delay in receiving the ACK, the per-connection throughput can be slower that intended. In this case, you can disable shaping for a rule, which can alleviate the problem.

# bwmgr em0 -name SlowUser -bwin 512000 -bwout 512000 -no_shaping

Per Session Bandwidth Management - Rate Limiting

TCP rate limiting effectively is a "per session" method of controlling traffic. Each TCP session within a container is automatically controlled. When you set a bandwidth limit, the default behavior of the software is to use an internal "adaptive window". The adaptive window algorithm attempts to reduce the window based on the traffic flows, setting it just where the traffic flows at the rate specified without excessive backup. The adaptive window will never be set lower than 1 full packet (1500 bytes), so it will not increase the overhead on your network as some rate limiting products do.

You can effectively "shape" traffic without limiting it by creating a container with a bandwidth setting that is just below your available bandwidth. So for example, if you have 3Mb/s of bandwidth, you could rate shape all traffic with :

# bwmgr em0 -ipprot tcp -bwlink GroupA -bwin 3000000 -bwout 3000000

While this may seem trivial, it in fact is quite powerful. The rule will match all tcp traffic and limit it to 3Mb/s. But the effect of the adaptive window management is to slow each session equally to create overall fairness within the usage of the bandwidth limit. It will also maintain minimum flows, reduce the likelihood that a drop will occur, and create a scenario with minimum delay for tcp traffic.

Of course window shaping only works on TCP traffic so you'll need other rules to control other traffic. But this illustrates how you can substantially improve your network flows with a minimum of controls.

Packets Per Second Bandwidth Rules

Packet per second rules can also be used to control users. PPS rules create a loosely-defined bandwidth limit which adjusts for the possibility that many small packets can use up resources even at low bandwidth levels. Give that a maximum packet is (usually) 1500 bytes and a minimum is usually 60 bytes, you can use a pps rule to give a wide range of capability to an end user or application without giving them unlimited bandwidth. So the following:

# bwmgr fxp2 -x 4000 -name AcmeCorp -addr 192.168.17.1 -ppsin 300 -ppsout 300

would limit the throughout of the specified address to 300 packets/second. This translates to a bandwidth limit of between 144000bps and 2.4Mb/s depending on the average packet size of their transactions. Note that there is no burst setting for packets per second rules.

Combining Global Rules for Bandwidth Management

Using global rules, you can set multiple control criteria for traffic that matches multiple rules.

# bwmgr fxp2 -x 1000 -addr 10.1.1.0 -addrmsk 255.255.255.0 -bwin 1000000 -bwout 1000000 -global

# bwmgr fxp2 -x 1200 -addr 10.1.1.17 -bwprofile default

The above allows you to set a limit for a class C without having to create a group. Since rule 1000 is a global rule, it will be enforced in addition to rule 1200 for traffic matching 10.1.1.17

Combined BW and PPS rules

Rules can have both bandwidth and PPS settings. This allows you to create a single rule that can control by both bandwidth volume and activity level.

# bwmgr em1 -x 500 -addr 10.1.1.1 -ppsin 100 -ppsout 100 -bwin 512000 -bwout 512000

The above rule would limit address 10.1.1.1 to the EITHER 100pps or 512000bps. If all packets are 1500 bytes, a setting of 100pps could allow as much as 1.2Mb/s of bandwidth, which may be more than you want. So adding the bandwidth setting will ALSO cap the rule at 512000. This rule allows both criteria to be estabished in a single rule. With 60 byte packets 100pps allows for only 48Kb/s.

Using Bandwidth Profiles

Bandwidth profiles are templates that allow you to apply common settings to any number of rules.

What is a Profile?

Each "profile" has a unique identifier, along with a series of options that will be applied to any rules that use the profile.

The profile depicted above defines a simple profile which has incoming and outgoing bandwidth limits of 512,000 bps.

The advantage of a profile is that you can change many rules at once by changing the profile. If you decide to change the default bandwidth to 600,000bps, you can just change the profile, and it will be applied immediately to all rules which use the profile.

Defining Bandwidth Profiles

Profiles are defined from within the HTML GUI, and stored in a database.

Applying Bandwidth Profiles

To apply a profile to a rule, use the -bwprofile rule option.

bwmgr em0 -x 1000 -addr 192.168.17.11 -bwprofile default

The above would apply the settings of the current "default" profile to the rule, as if you had entered them individually. Note: In the CLI, do NOT specify any bandwidth settings in the same command with -bwprofile or you may get the wrong results. In the HTML interface, other settings are ignored when a profile is specified.

The Effect of Using Bandwidth Profiles

At first glance, bandwidth profiles might just appear to be a shortcut for entering rules. But they are much more than that. Profiles are a vehicle for managing large rule sets, and also for implementing large-scale SLAs (service level agreements) or bandwidth tiers. Firstly, profiles are not only applied when you create the rule. If you change a profile, ALL of the rules which reference the modified profile will be changed accordingly. So if you have hundreds of customers/users who are using the "default" profile, you can modify global behavior of all of them that reference a specific profile in a single operation.

Profile Transitions

Previous to version 5.0, the ET/BWMGR used profile transitions to change profiles depending on the time of day. In v5, a more flexible mechanism is used, where multiple time changes can be applied by using Time Profiles. Time Profiles are described in a separate document.

Implementing Tiered SLAs with ET/BWMGR profiles

A simple example of a tiered SLA (service level agreement) would involve defining profiles for various service levels, and then applying them to customers/users who subscribe to that level of service.

Using the above simple settings, you have 4 distinct levels of service that simplify the adding new customers.

# bwmgr em0 -name cust_a -addr 192.168.17.1 -bwprofile default
# bwmgr em0 -name cust_b -addr 192.168.17.2 -bwprofile default
# bwmgr em0 -name cust_c -addr 192.168.17.3 -bwprofile silver
# bwmgr em0 -name cust_d -addr 192.168.17.123 -bwprofile gold

Reverse Rules

"Reverse rules" are basically dynamic "Groups" that are created automatically by selection of a "trigger" rule. Not to be confused with burst triggers; the trigger here is a rule which causes another rule to be created. The trigger can be any criteria, but is most usually a single packet which signals that a stream which needs to be controlled is being created. For example, suppose you wanted to limit "each host that accesses Server A" to 256kbs. You could use a reverse rule which triggers on a tcp connect to Server A which would create a limit on that host (and optionally, the port). Once this trigger is detected, you then know that a transfer is occuring on a specific address/port combination and a dynamic control could be put in place automatically.

Another example is a WWW server, where you may want to limit hosts accessing the site on a per session basis; that is to limit individual sessions to an address independently, either with a ceiling, a floor or on a shared, equal basis. This allows you so set up a rule which will limit EACH USER accessing a site to either a specific bandwidth or a percentage of the bandwidth dependent on how many users are currently accessing the site. Actually, reverse rules are better than limiting each session. Other products limit each TCP session to a host, but a host many open many simultaneous sessions (such as browsers typically do for graphics and text), so you cannot precisely limit a "session" because you don't know how many sessions exist. Many FTP clients will open multiple sessions to increase throughput, which defeats the "per session" limiting mechanisms. By limiting the source IP address, you are limiting the remote user no matter how many sessions are open.

Suppose you had a site with a video feed, and you wish to limit each user to 80K to avoid one user hogging all of your backbone bandwidth. You're not concerned with an upper limit for the entire site, you just don't want any single user to get more than 80K. You could use a reverse limit to achieve this very easily.

# bwmgr em0 -name VideoPlanet -daddr 211.14.1.7 -r -bwin 80000 -bwout 80000

When a "reverse rule" is accessed (specified with the -r switch as shown above), the bandwidth management software will attempt to "spawn" a new rule in an available rule slot directly before the reverse rule. For example, if the name address rule index is 5000, the first dynamically learned rule would be inserted at 4999, then 4998 and so forth. If a slot is occupied by another, non-associated rule, then the new rule can not be created. This implies that you must leave rule space before the reverse rule in order for the rule to be created. If no slots are available, the rule index plus 1 (NOT the next rule, it MUST be 1 index higher than the target rule) will be enforced. For example:

bwmgr -x 4800 -addr 10.11.1.1
bwmgr -x 5000 -name reverse_test -daddr 211.14.1.7 -r -bwboth 56000
bwmgr -x 5001 -daddr 211.14.1.7 -bwboth 56000

The above allows for 199 dynamic entries between 5000 and 4800. Once all of the slots are filled, no more new rules will be created. With the above settings, if more than 199 users were connected to the site at one time, all additional users (for which new rules could not be created), would be limited to a total of 56000 (as rule 5001 would then be enforced). This implies that the 200th user is actually the sum of all of the users over 199. Note that rule 5001 above is also a regular rule, so make certain to give it match criteria, typically the same as the target reverse rule without the reverse specifier.

Note that reverse rules must be named. Reverse rules create pseudo-groups, so the link to the head entry needs to have a name associated to create the proper linkage.

By default, reverse rules only limit the source address of the host accessing the trigger. So with the above rule, if host 192.168.0.1 accessed the server at 211.14.1.7, the following dynamic rule would be created:

bwmgr -x 4999 -addr 192.168.0.1 -t -bwboth 56000

If you don't want the entire IP limited, you could specify -usesport or -usedport indicate that the source or destination port should be used. -usesport indicated that the source port should be used, while -usdport specifies the destaination port.

# bwmgr -x 5000 -name reverse_test -daddr 211.14.1.7 -usersport -r -bwboth 56000

The above rule tells the ET/BWMGR to also use the source port when creating the dynamic rule. So now if 192.168.0.1 accesses the server, the following might be created:

# bwmgr -x 4999 -addr 192.168.0.1 -port 41003 -t -bwboth 56000

The negative of using the port also is that typically, a browser will open up multiple sessions when accessing a web site, so for each access you may see 3 or more rules created. This increases system overhead (as the dynamic rules have to be created and deleted, and you will have more overally rules in the ruleset), and further limits the number of "actual" users you can limit. If you have 1000 slots available, for example, perhaps only 300 actual users will be limited.

To control which address is used, there are several controls available. As noted before, the default is to use the source IP address.

-usesaddr Use the Source IP Address
-usedaddr Use the Destination Address
-usesport Use the Source Port
-usedport Use the Destination Port
-useprot Use the Protocol
-r Reverse IP Address Rule
-rm Reverse MAC Rule

The -r or -rm controls whether the MAC or IP is used, and the -use?addr controls whether to use the source or destination IP address, and the -use?port control whether or not the port should be included. -useprot is useful for creating reverse rules for pseudo-protocols such as p2p.

#bwmgr em0 -x 3000 -ipprot p2p -bwin 64000 -bwout 64000 -usesaddr -useprot

Be careful when using MAC rules that you understand what your traffic looks like, and that you don't block or limit your gateway if thats not your intention.

A consideration with reverse rules is what happens when a dynamic rule cannot be created. The most common reason being that there are no more slots available, or -maxlinks has been reached. When a dynamic rule cannot be created, the packet not acted upon.

Reverse Rule Modifiers

There are several new settings associated primarily with reverse limiting. The first allows finer control of how many users can access a site. This is independent of the discussion above. If you wanted to limit a site to 50 accesses, you could use the following:

bwmgr -x 4800 -addr 10.11.1.1
bwmgr -x 5000 -name reverse_test -addr 211.14.1.7 -r -bwboth 56000 -maxlinks 50

So even though there are 199 slots available before rule 5000, only 50 dynamic rules will be allowed to be created.

A second concept is the duration in which a dynamically created rule will remain active. You don't want rules to continue to be in the table long after a user has gone away. The default is 5 minutes of inactivity before the dynamically created rule will be deleted. Using the -ruletmo switch, you can specify a timeout for each rule. For example

bwmgr em0 -name reverse_test -addr 207.11.14.1 -bwboth 56000 -r -ruletmo 30

The above specifies a timeout of 30 seconds, which may be more appropriate is using ports with a web server (since each page access generally uses a new port).

The normal "timeout" for a rule is an idle timeout. This means that when no matches occur for x seconds, the rule will be removed, where x is the timeout. Of course in a situation where there is continuous traffic, the rule may never timeout. If that is a problem, you can use -fixedtmo instead of -ruletmo. The -fixedtmo setting will cause the rule to be removed x seconds after it is created. For single trigger events (such as access to a port that is later redirected), this may not work properly, as the "event" may not occur again while the session may continue.

Creating Dynamic Groups with Reverse Rules

You can use the dynamic nature of reverse rules to create "dynamic groups". Once you understand what the default behavior of a Reverse Rule is, you will have the context to understand the flexibility that comes from changing those defaults.

Every reverse rule must have a name, since the rule acts as a group. By default, rules spawned by a reverse rule are added to the reverse rule's group, with automatically generated names.

If a reverse rule is a member of a group, then any dynamic rules created from the reverse trigger will also be a member of that group.

For example, suppose you had a class C and you wanted to limit each address to a specific setting. You'd have to create a group, and create 254 rules in the group, which can be tedious. Instead, you could do this:

# bwmgr em1 -x 3000 -name MyGroup -bwin 1000000 -bwout 1000000 -group
# bwmgr em1 -x 3001 -name MyGroup_rev -addr 11.1.1.0 -addrmsk 255.255.255.0 -r -bwin 64000 -bwout 64000 -bwlink MyGroup

The above defines a group at index 3000 that is limited to 1Mb/s. Rule 3001 defines a reverse rule, which is a member of the group MyGroup. Because the reverse rule is a member of the group MyGroup, the dynamic rules will be created before index 3000 (NOT the index of the reverse rule in this case), and they will inherit the group membership of the trigger rule, which means that they will be added to the group MyGroup, with the bandwidth settings from MyGroup. This simple construct allows up to 254 dynamic rules (each possible address in the 11.1.1.0 network) with a single rule.

Host Name Reverse Rules

As described earlier, the -nameaddr option can be used to Match virtual hosts even though many hosts may use the same IP address. There is a particular problem with virtual hosts which is that only the initial HTTP header contains the virtual host information. Subsequent packets look like regular http traffic going to an IP address shared with many hosts. In order to simplify the management of virtual host traffic, -nameaddr rules are simply specified as reverse triggers for dynamic reverse rules. The -nameaddr rule is used as a trigger to create dynamic rules to hosts accessing the virtual host.

# bwmgr em0 -name thishost -nameaddr www.thishost.com -r -bwboth 128000

The above specifies that each IP address accessing host www.thishost.com will be limited to a 128000bps. The -nameaddr tag will be triggered when an http request with the proper host header info is detected, at which point a temporary, dynamic rule will be spawned limiting the requester, which will inherit the properties of the trigger rule. The -ruletmo and -maxlinks switches may also be used to control name address rules.

Optionally, Groups or the -b option may be used in conjunction with -nameaddr to change the behavior of the creation of the reverse rules. With the -b option set, the bandwidth value of the trigger rule would be a ceiling reference, and the dynamic rules will be balanced in the same way as standard reverse balanced rules.

# bwmgr em0 -name thishost -nameaddr www.thishost.com -r -bwboth 128000

will cause all of the hosts accessing www.thishost.com to be limited to 128000bbs each, while

# bwmgr em0 -name thishost -nameaddr www.thishost.com -r -b -bwmin 32000 -bwboth 128000

will result in each host accessing www.thishost.com to have an equal share of 128000bps with no less than 32000bps.

Reverse Firewall Rules

The ET/BWMGR allows you to define rules which can be set up to signal that some remote host should be blocked from access to your network. Port scanners are an example, as are address probers (applications that attempt to discover hosts and services on your network). By using what we call "reverse firewall rules", you can set up triggers which cause hosts to be blocked for some predetermined length of time when they access a port or address or a specific service on a specific host.

Reverse firewall rules could also be called "reverse discard" rules, because thats what they do. You set up a trigger rule, and when the rule Matches, a dynamic firewall rule is set up that blocks the source address from any further access to your network. For example, recall a virus that caused hosts to sent a specific URL that could infect specific types of servers. The virus caused servers to scan addresses and send a "default.ida" URL, hoping to find hosts to infect. You could easily thwart this with the following:

# bwmgr em0 -x 1000 -url default.ida -fw -r -priority discard

The above rule would look for http traffic with a URL of "default.ida". When a Match was encountered, a reverse rule would be created (starting at index 999 and moving down) with the source address of the frame (ie the "sender" of the specified URL) and a priority of FW-Deny (or discard). Any subsequent traffic from that host would be blocked.

To stop an address scanner, select an address on your network that isnt in use (or is only used internally). The theory is that anyone accessing such an address is bad.

# bwmgr em0 -x 1000 -addr 209.14.1.1 -r -fw -ruletmo 300 -l -priority discard

this rule would block anyone who attempted to access 209.14.1.1 for 5 minutes. Note that -l is set, so that each access will also be logged.

To thwart a port scanner, you could select a taboo port or range of ports that shouldn't be accessed by outside hosts. pop3 is a popular port scan port:

# bwmgr em0 -x 2000 -ipproto tcp -port pop3 -r -fw -priority discard

This rule would cause anyone accessing a pop3 port to be blocked. When using such rule, remember to insert rules before the triggers that specifically allow hosts that are permitted to access the addresses or port, otherwise you will block them also.

Temporary Rules

You can add a temporary rule using the -t option (for adding a temp rule) and either the -fixedtmo or -ruletmo option.

Using these options together, you create a rule that will be deleted after some specified number of seconds.

# bwmgr em0 -x 50000 -t -fixedtmo 300 -addr 10.1.1.1 -bwin 128000 -bwout 128000

The above rule will limit address 10.1.1.1 to 128000 in and out for 5 minutes. After 5 minutes the rule will be deleted. If you wanted to make it an idle timeout (ie the rule would be deleted after 5 minutes of no activity), use -ruletmo instead of -fixedtmo.

Group Bandwidth Management

"Groups" allow you to create rules which act on a number of rules with some common parameters. Groups (and also interfaces) can also be referred to generically as "containers". Groups are created by adding a rule with the option -group, which indicates that the rule is a head entry for a group. The -name option is required to define a group. Group containers exist only to hold other objects (including rules and other groups - nested groups are allowed), and as such they should not have any matching criteria. Group containers can have specific bandwidth settings, or profiles attached.

# bwmgr em0 -name renegades -group

To add members to the group renegades, use the -bwlink option:

# bwmgr em0 -addr 10.1.1.1 -bwlink renegades

The above rule creates a new rule and links it to the group renegades. Of course, a group with no properties, as in the example above, isn't very interesting or useful. By adding properties to the rule, you can create powerful super rules that can increase your flexibility to manage bandwidth.

Setting a Bandwidth Ceiling for a Group

The simplest form of a group rule is the bandwidth limit for a group. This allows you to set a common bandwidth limit that applies to all of the members of the group as if it were a single rule. For example:

# bwmgr em0 -name Group0 -bwboth 256000 -group
# bwmgr em0 -addr 10.1.1.0 -addrmsk 255.255.255.0 -bwlink Group0
# bwmgr em0 -addr 12.1.7.14 -bwlink Group0
# bwmgr em0 -addr 12.1.7.15 -bwlink Group0

The above ruleset will set a bandwidth limit of 256000 bps which will be shared by the 10.1.1 network, plus 2 hosts on the 12.1.7 network. Grouping allows you to aggregate any number of disjoint hosts, network or traffic types and bundle them as if they were one rule. One benefit of this is that they can each have their own statistics, if required. You can also apply other bandwidth properties which will apply to groups as a whole such as -bwmin, -bwin, -bwout and -bwburst_in/out.

Setting Bandwidth Limits Within Groups

You can give rules within a group its own properties which will work in concert with the group properties. For example, in the above example, suppose you didn't want any member of the group to use more than 128K of the bandwidth at any time (the default would allow it to use up to the full 256K if none of the other members were active. You could do the following:

# bwmgr em0 -name Group0 -bwboth 256000 -group
# bwmgr em0 -addr 10.1.1.0 -addrmsk 255.255.255.0 -bwboth 128000 -bwlink Group0
# bwmgr em0 -addr 12.1.7.14 -bwboth 128000 -bwlink Group0
# bwmgr em0 -addr 12.1.7.15 -bwboth 128000 -bwlink Group0

The above would allow up to 128000bps for each member, but limit the group to 256000bps as a whole.

Balanced Group Bandwidth

You may want to achieve "fairness" in a group, so that each member will get an equal chunk of bandwidth depending on how many members are active at any time. By using the -b option, you can "balance" the bandwidth across active members in a group. So, for example, if you had a group with 30 members and a bandwidth allotment of 512000, if 2 of them were active they would get 256000 each and if 10 were active they would get 51200 each. You can additionally use the -bwmin to assure the the allocation per user was at least a certain amount (because if all 30 were on each member would have an allocation so low that none of them would be functional). Using the example above again, you could simply set the -b option in the group rule to achieve balancing:

# bwmgr em0 -name Group0 -bwboth 256000 -b
# bwmgr em0 -addr 10.1.1.0 -addrmsk 255.255.255.0 -bwlink Group0
# bwmgr em0 -addr 12.1.7.14 -bwlink Group0
# bwmgr em0 -addr 12.1.7.15 -bwlink Group0

Simply by adding the -b option, you transform the group into a balanced bandwidth group. When you view the rules, the current setting (based on how many of the rules are currently in use) will be shown. Note that it may not be safe to set the -b option to an existing group that has active members, as some required settings in group members may not be initialized properly.

Nested Groups (Groups within Groups)

You can define a group within a group, that is a group tag which is a member of another group. This allows you to use a group as if it were a single rule within a group, or allow a group bandwidth setting to be dependent on the activity or settings within another group. You can even have balanced groups withing balanced groups, allowing for full dynamic bandwidth settings on a demand basis.

# bwmgr em1 -x 1000 -name BigGroup -bwin 1000000 -bwout 1000000 -group -b
# bwmgr em1 -x 1010 -addr 10.1.1.1 -bwlink BigGroup
# bwmgr em1 -x 1500 -name LittleGroup -bwlink BigGroup -b
# bwmgr em1 -x 1510 -addr 11.1.1.1 -bwlink LittleGroup
# bwmgr em1 -x 1510 -addr 15.1.1.3 -bwlink LittleGroup

The above creates a balanced group with a bandwidth limit of 1Mb/s in and 1Mb/s out. There are 2 members in the group, IP address 10.1.1.1 and LittleGroup, which has 2 of its own members. The bandwidth of BigGroup will be distributed equally between IP address 10.1.1.1 and LittleGroup as needed, as if LittleGroup was a single rule. Within LittleGroup, the bandwidth will be further distributed to its members.

Excluding Traffic (Pass-thru Rules)

You may wish to completely exclude certain traffic from group or interface statistics and limits, such as the traffic from certain servers. You can exclude traffic by setting a rule with a priority of "pass-thru" which allows traffic to pass and not be counted towards limits.

# bwmgr em0 -ifac -bwin 1000000 -bwout 1000000
# bwmgr em0 -x 1 -saddr 10.1.1.0 -saddrmsk 255.255.255.0 -daddr 45.1.1.1 -priority pass-thru
# bwmgr em0 -x 2 -saddr 45.1.1.1 -daddr 10.1.1.0 -daddrmsk 255.255.255.0 -priority pass-thru

The above will allow only 1Mb/s of traffic through em0, but will not count traffic between host 45.1.1.1 and the 10.1.1.0 class C network. The pass-thru designation excluded the traffic from being counted towards limits in interfaces or groups in which the rule is a member.

Prioritization

The fastest way to move data is to transmit it as soon as you can. But since only one frame can be transmitted at a time onto a medium, data becomes "queued" up and important data may have to wait until less important data is finished. By "prioritizing" the data, you can reorder the data so that important data is sent first. While this may all seem to be what you want, there are caveats as well. The big issue with prioritizing data is that there is a time penalty to pay for every frame that is prioritized, because you have to hold some amount of data so that you can reorder it according the prescribed priorities. When traffic is not so heavy (that is, when you would be perfectly happy with the throughput if no prioritization is done), then you are introducing unnecessary latency into ALL of your traffic. This is something that you want to avoid.

ET/BWMGR Prioritization

The ET/BWMGR uses an internal algorithm to determine when queued traffic is to be forwarded. Priorities have 2 functions; first, higher priority traffic is sent first where there is traffic with multiple priorities sitting in a delay queue. So when both high and low priority traffic is delayed, high priority traffic will be processed first. Lower priority traffic is also more likely to be dropped. Priority 1 traffic is very likely to be dropped if it is delayed. Priority 10 traffic will probably not be dropped.

The effect of this is that if traffic isn't being delayed, then priorities have no meaning. Non-delayed traffic is simply sent when it is received.

How to Set Global Data Priorities

The ET/BWMGR has 10 levels of priority (1-10), with 1 being the lowest priority and 10 being the hightest. When data is set to be prioritized, the priority subsystem tries to find a Match in the ruleset. If a Match occurs, then the priority of the data is set to the priority indicated in the Match.

The rule for setting a priority is simple. The first non-normal priority that matches the traffic is the priority of the traffic, even if a later, more specific rule also has a priority setting.

# bwmgr em0 -global -x 100 -ipprot icmp -priority 1
# bwmgr em0 -global -x 110 -ipprot icmp -addr 10.1.1.1 -priority 6
# bwmgr em0 -x 120 -addr 10.1.1.1 -bwprofile default

An ICMP packet from 10.1.1.1 would be set to a priority of 1 in the above ruleset, even though it appears that rule 100 is "more" of a match. A match is a match. Since rule 120 doesn't have any priority set, the first global rule is used. If you wanted rule 110 to hit, you should give it a lower index then rule 100. Alternatively:

# bwmgr em0 -x 100 -ipprot icmp -priority 1 -global
# bwmgr em0 -x 120 -addr 10.1.1.1 -bwprofile default -priority 8

for the above, ICMP traffic for 10.1.1.1 would be set to a priority of 1 as per rule 100, and set to 8 for all other traffic. Additionally:

# bwmgr em0 -x 100 -addr 10.1.1.1 -bwprofile default -priority 8
# bwmgr em0 -x 110 -ipprot icmp -priority 1 -global

The above ruleset would set an ICMP packet for 10.1.1.1 to have a priority of 8, while other ICMP traffic (for rules with indexes greater than 110) would be set to 1.

Setting Priorities for Bandwidth Rules

All bandwidth rules include are assigned a priority, with the default being NORMAL, or 5. You can optionally set a different priority with the -priority command line option. If a data frame Matches a bandwidth rule, the priority of the data will be temporarily set to that value. If the priority threshold of the interface or the bandwidth limit of the rule has been exceeded, the data will be checked against the Global Priority Ruleset for a Match. If a Match occurs, then the priority of the data will be set to the priortity indication in the Global Priority rule.

Statistics

Showing "Hits" and "Drops"

Any time that a rule Matches or "hits", a counter is incremented. To view a specific rule on the console, use the "show" command:

# bwmgr em0 show 200

200 IO Rule200 10.1.1.1 / 32 14,175/35

where the 200 is the index or rule number and the numbers at the end represent the hits/drops number of times a packet that matched this criteria was Matched, and also the number of times that this rule resulted in a dropped data frame. You can also use get rule, to show all info about the rule.

Saving Stats for a Rule

Saving statistics is generally a function that is integrated with the Graphical Interface, but an overview of how it works will help you to debug problems and also do things manually if you have the need to do so.

There are two command line options which cause statistics for a particular rule to be stored with the bandwidth manager subsystem. The first option, -stats, is a flag that tells the bandwidth manager to store stats for this rule.

# bwmgr em0 -x 7000 -addr 10.1.1.1 -stats

Stats are best viewed using the GUI.

About the bwmgrd daemon

bwmgrd is a php program that runs every 5 minutes, gathering and storing stats. It uses the "stats" table in the database to get a list of stats-enabled rules. For each entry in the stats table, it updates the associated RRD file.

RRD files are stored in /usr/local/etc/bwmgr/graphs. If the rule name is AcmeCorp, the file name will be AcmeCorp.rrd. RRD files can also be used by other graph utilities (such as Cacti) that can parse RRD files.

Global Stats Rules

Note that when a normal rule "hits", no more rules are checked. This is done for efficiency and also as a matter of basic policy. Unfortunately, for statistical purposes it may hinder your ability to gather statistics because traffic types overlap.

For example, if you wanted to gather stats for the total www traffic on your network and also the www traffic to a particular host, to following would fail:

# bwmgr em0 -x 100 -name A -addr 207.11.14.1 -prot tcp -port www -stats
# bwmgr em0 -x 200 -name B -prot tcp -prot www -stats

The reason it would fail to give the results you want is because any traffic for host 207.11.14.1 would not be included in the numbers gathered by rule 200 because once rule 100 hits, no other rules could hit with a higher index.

To override this, you can use a global rule with a stats setting.

# bwmgr em0 -x 100 -name tcp_all -prot tcp -stats -global
# bwmgr em0 -x 200 -name www_all -prot tcp -port www -stats -global
# bwmgr em0 -x 300 -name AWC -addr 207.11.14.1 -stats -global
# bwmgr em0 -x 5000 -addr 207.11.14.1 -bwboth 250000

The above allows you to limit all of the traffic to host 207.11.14.1 to 250000bps and also gather statistics on specific traffic types, even if the types overlap with other rules. The -stats switch would allow you to store and graph each type above.

Note that stats rules still must have a lower index than any non-global rules. In the above example, any global rule with an index higher than 5000 would not be counted for host 207.11.14.1. As a simple policy, you should segregate your global and bandwidth rules by setting an index below which all rules are non-global to avoid conflicts.

Changing the Statistical Period

You can change the statistical period with the stats_period option.

To set the period to 1 minute, you would use the following:

# bwmgr set stats_period 60

Note that the statistical period is the same as the burst period, so if you are using bursting its likely that its more important to use a period to conform to your bursting requirements than statistical viewing.

Advanced Topics

Setting the GUI password

In the event that you forget the password or have to rebuild your database, you can easily set a new password for the ET/BWMGR GUI from the command line.

# bwmgr guipassword PASSWORD

Changing the Webmin Password

Webmin is the system administration GUI that runs on port 10000. If you lose access or can't remember the password, you can change it from the CLI.

# cd /usr/local/webmin
# perl changepass.pl /etc/webmin admin password

This will change the "admin" user's password to password. If you are trying to change the password for a different ET/ADMIN user, simply replace "admin" with the correct username.

Controlling File Sharing Programs - Gnutella, Kazaa, Audiogalaxy, Morpheus and iMesh

A special filter is available for detecting these file sharing protocols. While most file sharing protocols do act on specific ports, the way that they work requires some special intelligence to detect, which we have built-in the the ET/BWMGR software. The ET/BWMGR software can detect when these protocols are being run on non-standard ports, allowing you to control the traffic without having to manually sniff out the connections.

Controlling p2p Protocols - The p2p protocol Group

An internal special protocol, p2p, is defined which will match many different types of p2p protocol. You can enhance these settings by defining newly discovered agent or urls in the /etc/bmgr-protocols file.

To control all known P2P traffic with a single rule:

# bwmgr em1 -ipprot p2p -bwin 128000 -bwout 128000

The above rule is all thats required to control all of the p2p protocols, no matter how many streams there are. You can also use the p2p protocol definition along with other settings, just like any other protocol:

# bwmgr em1 -name groupA -group -bwin 256000 -bwout 256000
# bwmgr em1 -addr 10.1.1.1 -ipprot p2p -bwin 64000 -bwout 64000 -bwlink groupA
# bwmgr em1 -addr 10.1.1.1 -bwlink groupA

The above settings would limit 10.1.1.1 to 256000bps and limit the p2p bandwidth to 64000bps out of the 256000.

Gnutella, Kazaa and DC

Special protocols for Gnutella, Kazaa and DirectConnect (DC++) are available in case you want to see counters or stats for these, or in the oddball case where you want to have different settings for one or the other. The protocol "p2p" is a "group", so that it matches gnutella, kazaa, DC++ and all other supported p2p protocols. Sometimes there may be some gnutella or kazaa connections that are classified as P2P or incorrectly (for example if you run kazaa on the gnutella port it will be shown as gnutella), so p2p is safer if controlling p2p is an important function.

General Maintenance

Deleting a Bandwidth Rule

To delete a bandwidth rule, use the delrule keyword with the interface and index or name of the entry you want to delete.

To delete an entry displayed with index 200:

# bwmgr em0 delrule 200

If the entry is named, then you can use the name:

#bwmgr em0 delete cust0

To delete a firewall entry, use delfw instead of delrule. To delete a global priority rule, use delpri.

Saving The Rules

You'll want your current rules to be restored when you reboot your system. If you want your rules to be saved automatically, there is a setting called Auto Rebuild in the Settings tab in the ET/BWMGR GUI that must be enabled by entering a 1 as the value. Save the settings to make this change.

If Auto Rebuild is on, then any changes made to the ruleset are saved the next time that bwmgrd is run, which is every 5 minutes by default.

Rules are saved in the shell script /etc/rc.bwmgr, which is run at boot time.

If you have Auto Rebuild turned off, then you will have to manually save your rules. The rebuild will output the current ruleset.

To save the ruleset in a file, simply redirect the output of rebuild to a file. In this, we use the default of /etc/rc.bwmgr, so that the rules will be restored at next boot.

# bwmgr rebuild >/etc/rc.bwmgr

You can manually check the file /etc/rc.bwmgr to make sure your rules are being saved. After entering a large number of rules it's a good idea to check them manually before you reboot or have a crash.

Other ways to rebuild:

# bwmgr rebuild userules > /etc/rc.bwmgr
# bwmgr rebuild db

userules will create a startup file from the rules in the system (in case the DB is wrong). The DB rules can get messed up with some operations. rebuild db will rebuild the DB from the rules in the system.

Other Topics

Performance Tips

Specifications, as with any networking product, are variable based on the actual usage of the machine. When we specify that a particular appliance can handle a given amount of traffic, for example, 1Gb/s, this means that this is the maximum it can reasonably handle with minimal settings. An example of minimal settings isn't so much the number of rules, but which features are enabled. The more features in use, and the more complex the typical rule, the more CPU power is needed, which can diminish peak capacity.

This section exists to document the settings that require more processing power than others, so that you can tune a system that is overloaded, or adjust the "capacities" according to your needs.

Rule Overhead

Certain types of rules add more overhead than others, and the existence of even a single rule with some features will create system overhead, because additional filters will be required for every frame.

Firewall Rules

Firewall rules are not indexed; they must be scanned sequentially. So having a lot of firewall rules on a loaded system is not a good idea. If you have a lot more CPU power than you need in your system, you can use the flexibility of the firewall features without having to worry. But if you have a loaded system with high traffic levels or a lot of rules, you should consider a separate box as your main firewall, and just use the integrated firewall to control access to the system itself. Also, If you have firewall rules for full IP addresses, its more efficient to place them in the bandwidth rules section so they will be indexed. Note that only the first rule can be a FW-deny in the bandwidth section.

MAC address Rules:

If you have even 1 rule on an interface that has a MAC address as a matching criteria, a significant increase in resources will be required as the MAC addresses of every single frame that passes through the interface will have to be examined. By default, no MAC checking is done (as a performance enhancement). If you need to use MAC matching there is no problem in doing so, but if you just have 1 or 2 rules and don't really need them you can reduce overhead by eliminating them.

Multiple Interface Usage:

If there are no settings on an interface, frames will not be decoded and/or examined. If you have 1 setting on an interface, then every frame will have to be classified. If possible put all of your rules on one interface. Also be very careful not to affect the same data on 2 different interfaces. The timings required for proper delay and control of bandwidth does not account for more than 1 action on a frame, so if you have limits on 2 different interfaces for the same data frame the results will not be what you expect and will likely cause unworkable delays.

The recommended usage for a 2-port bridge is to have the Outside interface also be your Default interface (which is set in the "Settings" GUI tab,) and to put all of your rules on this single interface. In a typical set-up this allows for bandwidth management and fire-walling of both internal hosts and the BWMGR itself, as traffic must first come in through the bridge before it reaches the inside switch.

Groups

Group rules require some extra processing, although they are far from the biggest offenders. Avoiding nested groups can help diminish the impact on performance.

URL Rules

URL rules are a powerful feature, however the use of them necessitate the "sniffing" of every http frame to decode the URL. String processing requires overhead, so if you have no URL rules you can release some CPU if you are overloading.

Protocol Sniffing

"Special" protocols, like p2p, require the contents of all TCP and/or UDP traffic to be "sniffed" (or looked at) to determine what specific protocols are running, and also require the tracking of the streams after detection. While this is very powerful, it does require much more CPU than raw decoding of packet contents (such as addresses and ports) without tracking. If you are limiting by IP address only and don't care about protocol decoding, you can turn off "sniffing", which will reduce the overhead significantly.

# bwmgr set sniffing off

Large Bandwidth Settings

To allow "full-wire" access, use no bandwidth limit rather than a high one (like 10Mb/s). When a bandwidth rule has a bandwidth limit, there are numerous comparisions and monitoring functions that must be done, which causes overhead. If the rule is to allow a host or traffic type full bandwidth access to a wire, use no bandwidth limit as in the following example:

# bwmgr em0 -x 3000 -addr 207.11.11.1
# bwmgr em0 -x 100000 -addr 207.11.11.0 -addrmsk 255.255.255.0 -bwboth 128000

The above rules limit the 207.11.11 network to 128000bps except for 207.11.11.1, which is granted full access to the wire.

Rule Placement

If possible, place higher traffic rules before lower traffic ones (unless precedence matters of course). This is particularly true of "global" rules, that is rules that do not use IP or MAC addresses as match criteria, and also firewall rules. The sooner a "hit" occurs, the less matching has to be performed.

Customer Graph Access

Any rule with stats enabled can be configured for third-party access to the usage graph, by associating a customer with a graph.

1) Create a customer in the customers tab, and give them a login and password.
2) Associate the customer record with a Rule name.
3) Have the customer access graph.php. When they log in using the login and password, they'll get the graph for the rule specified.

Setting up Mail

The ET/BWMGR uses ssmtp as its outgoing-only mailer; please refer to the FreeBSD docs for configuring SSMTP. The config file is

/usr/local/etc/ssmtp/ssmtp.conf

The mailer is specified in /etc/mail/mailer.conf; you're free to use sendmail if you know how to configure it, but we don't support it's configuration.

Next Steps

For a more concise reference of the ET/BWMGR CLI commands, see the ET/BWMGR v5.0 CLI Manual. Also visit the Documentation section of our web site to see our other manuals.