Re: Rate limiting with "tc"?

Ravenpi wrote:

Hi, all. My company has a reasonably large pipe to the Internet;
however, certain large transfers (e.g., our hourly database
replication with our offsite server) spike the usage. I'd like to
limit our traffic, both inbound and outbound, such that no single user
can use more than 50% of our bandwidth. I've poked around, and it
seems that the best way to accomplish this is by way of the "tc"
command... unfortunately, its documentation is, ummmm... "in depth."
While I'm willing to wade through it, and play with trial-and-error,
this is one of those times where I just want that one magic command,
and then I'll move on. So any help would be greatly appreciated.

First:
Outbound policing is possible, but inbound is not.

Now onto tc:
tc works together with something to classify packets. There's a number of
tools to do so, some with more features, others simpler to maintain. I for
myself use iptables for this task. To do so, you need the MARK target
available for iptables, because it's a mark applied to each packet used to
classify and thus assign a packet to a specific rate-limited queue.
Also I'm using the HTB (Hierarchical Token Bucket) to classify packets.
That's based on historic reasons: I've did it with htb years ago and never
had any reason or will to change it.

Now onto the usage:
1. You need a qdisc for your nic.
tc qdisc add dev <NIC> root handle <x>:0 htb default <n>
NIC is the network device to work on, x is the master number of your chains,
n is the default chain if packets can not be assigned by other means.

2. Now you need a master chain:
tc class add dev <NIC> parent <x>:0 classid <x>:1 htb \
rate <NORM>kbit ceil <MAX>kbit
NIC and x are always the same as used for 'tc qdisc' above, NORM and MAX are
the normal and maximum rates (in kbit/s) available for this nic on the
outbound direction. both might be the same, but NORM should not be greater
than MAX. This means that normally not more than NORM kbit/s should be
sent, but if traffic exceeds this rate, there's more bandwith available.

3.a You need a default chain:
tc class add dev <NIC> parent <x>:1 classid <x>:<n> htb \
rate <(NORM-T)>kbit ceil <MAX>kbit prio 2
(NORM-T) is the normal traffic bandwith without traffic handled by specific
chains. By specifying 'ceil <MAX>kbit' the hard limit is anyway the maximum
bandwith you allow. Thus, if default traffic exceeds the normal rate and no
other traffic is using bandwith, default traffic can use up to MAX kbit/s.
There's no need to explicitly put all packets into this chain, they are put
into this chain by default due to 'default <n>' from step 1.

3.b Now you need chains with limited bandwith and priorities:
tc class add dev <NIC> parent <x>:1 classid <x>:<y> htb \
rate 10kbit ceil <MAX>kbit prio 0
This is a chain for very low traffic but with maximum priority. I use it for
ACK packets speeding up my downloads even if there's lot of other upstream
traffic. But it's also limited in bandwith. y is a unique number to name
the chain.

3.c To use it you need an iptables rule:
iptables -t mangle -A OUTPUT -p tcp \
-m length --length :64 -j MARK --set-mark <z>
This needs the length match for iptables. You can set the mark (z) to
whatever you like; I prefer the number of the chain to make it easier to
see which iptables rule(s) belong to which chain.

3.d Now you need to tell the system to put packets with a specific mark to a
chain:
tc filter add dev <NIC> parent <x>:0 prio 0 protocol ip \
handle <z> fw flowid <x>:<y>

That's it. Now packets up to the size of 64byte are prioritized first but
may normally not use more than 10kbit/s of bandwith.
Repeat steps 3.b-d for all types of traffic you may need to prioritize.

HTH,
Felix
.

Relevant Pages

  • Re: Help with Iptables on with RH linux
    ... iptables -P OUTPUT DROP ... # PREROUTING chain rules ... INPUT only when packets have a destination IP of your firewall. ...
    (RedHat)
  • Need Help Figuring Out a DMZ Setup
    ... i have just built a new firewall box running redhat 8 & iptables. ... i can access my web site on the server in the dmz. ... packets that came from that lan (like checking mail, ... # Create chain for bad tcp packets ...
    (comp.os.linux.security)
  • DNAT packets not getting to FORWARD chain
    ... Anyway, its a simple setup (the smoothwalls, fyi are a red hat flavour ... kernel 2.4.26, iptables v1.2.8) ... The chain policy count is zero, as are the counts for all rules in the ... packets supposedly been readdressed to 192.168.1.40 (an address I can ping ...
    (comp.os.linux.networking)
  • Re: Help with Iptables on with RH linux
    ... iptables -P OUTPUT DROP ... INPUT only when packets have a destination IP of your firewall. ... the FORWARD chain contains rules that affect packets passing through ... Yes I flushed the rules before calling the script... ...
    (RedHat)
  • Desperate need of Help for FTP
    ... Configuration options. ... echo 1.5 IPTables Configuration. ... # Create chain for bad tcp packets ...
    (comp.os.linux.security)