Blog

Debian advanced router for ISP – firewall, traffic shaping, smp_affinity, taskset, sysctl and more …

Advanced routing
Firewall services
Traffic shaping
smp_affinity – hardware allocation of processor cores
taskset – software allocation of processor cores
tcp and conntrack tunning in Linux kernel

DNS servers /etc/resolf.conf

nameserver 208.67.220.220
nameserver 208.67.222.222

DNS services

apt-get install dnsmasq

Quagga routing daemon

apt-get install quagga

Quagga /etc/quagga/zebra.conf

!
! Zebra configuration saved from vty
!   2011/02/18 09:27:57
!
hostname Router
password zebra
enable password zebra
log syslog informational
!
interface eth0
 ip address 172.16.100.1/30
 ipv6 nd suppress-ra
!
interface eth0.100
 ip address 93.155.131.1/28
 ip address 93.155.130.17/28
 ip address 93.155.130.65/26
 ip address 93.155.130.129/25
 ip address 93.155.131.33/27
 ip address 93.155.131.65/26
 ip address 93.155.131.129/25
 ip address 93.155.162.1/24
 ip address 93.155.169.1/24
 ipv6 nd suppress-ra
!
interface eth0.523
 ip address 93.155.131.17/29
 ip address 194.141.68.2/27
 ip address 194.141.68.3/27
 ip address 194.141.68.4/27
 ip address 194.141.68.5/27
 ip address 194.141.68.6/27
 ip address 194.141.68.7/27
 ip address 194.141.68.8/27
 ip address 194.141.68.9/27
 ip address 194.141.68.10/27
 ipv6 nd suppress-ra
!
interface eth1
 ip address 172.16.101.1/30
 ipv6 nd suppress-ra
!
interface eth1.149
 ip address 212.70.158.90/30
 ipv6 nd suppress-ra
!
interface eth2
 ip address 93.155.130.1/28
 ipv6 nd suppress-ra
!
interface eth3
 ip address 192.168.201.1/24
 ipv6 nd suppress-ra
!
interface gre0
 ipv6 nd suppress-ra
!
interface lo
!
interface tun1
 ip address 93.155.130.33/30
 ipv6 nd suppress-ra
!
interface tun2
 ip address 93.155.130.37/30
 ipv6 nd suppress-ra
!
ip route 0.0.0.0/0 194.141.68.1
ip route 10.0.0.0/8 93.155.131.11
ip route 10.121.0.0/16 93.155.131.11
ip route 10.122.0.0/16 93.155.131.11
ip route 10.123.0.0/16 93.155.131.11
ip route 10.124.0.0/16 93.155.131.11
ip route 10.125.0.0/16 93.155.131.11
ip route 10.126.0.0/16 93.155.131.11
ip route 10.127.0.0/16 93.155.131.11
ip route 10.128.0.0/16 93.155.131.11
ip route 10.129.0.0/16 93.155.131.11
ip route 194.141.67.0/24 194.141.68.1
ip route 194.141.68.0/24 194.141.68.1
ip route 194.141.69.0/24 194.141.68.1
ip route 195.138.132.0/24 93.155.131.18
ip route 195.138.153.0/24 93.155.131.18
ip route 212.233.128.0/24 93.155.131.18
ip route 212.233.129.0/24 93.155.131.18
ip route 212.233.130.0/24 93.155.131.18
ip route 212.233.131.0/24 93.155.131.18
ip route 212.233.132.0/24 93.155.131.18
ip route 212.233.133.0/24 93.155.131.18
ip route 212.233.134.0/24 93.155.131.18
ip route 212.233.135.0/24 93.155.131.18
ip route 212.233.136.0/24 93.155.131.18
ip route 212.233.137.0/24 93.155.131.18
ip route 212.233.138.0/24 93.155.131.18
ip route 212.233.139.0/24 93.155.131.18
ip route 212.233.140.0/24 93.155.131.18
ip route 212.233.141.0/24 93.155.131.18
ip route 212.233.142.0/24 93.155.131.18
ip route 212.233.143.0/24 93.155.131.18
ip route 212.233.144.0/24 93.155.131.18
ip route 212.233.145.0/24 93.155.131.18
ip route 212.233.146.0/24 93.155.131.18
ip route 212.233.147.0/24 93.155.131.18
ip route 212.233.148.0/24 93.155.131.18
ip route 212.233.149.0/24 93.155.131.18
ip route 212.233.150.0/24 93.155.131.18
ip route 212.233.151.0/24 93.155.131.18
ip route 212.233.152.0/24 93.155.131.18
ip route 212.233.153.0/24 93.155.131.18
ip route 212.233.154.0/24 93.155.131.18
ip route 212.233.155.0/24 93.155.131.18
ip route 212.233.156.0/24 93.155.131.18
ip route 212.233.157.0/24 93.155.131.18
ip route 212.233.158.0/24 93.155.131.18
ip route 212.233.159.0/24 93.155.131.18
ip route 212.233.160.0/24 93.155.131.18
ip route 212.233.161.0/24 93.155.131.18
ip route 212.233.162.0/24 93.155.131.18
ip route 212.233.163.0/24 93.155.131.18
ip route 212.233.164.0/24 93.155.131.18
ip route 212.233.165.0/24 93.155.131.18
ip route 212.233.166.0/24 93.155.131.18
ip route 212.233.167.0/24 93.155.131.18
ip route 212.233.168.0/24 93.155.131.18
ip route 212.233.169.0/24 93.155.131.18
ip route 212.233.170.0/24 93.155.131.18
ip route 212.233.171.0/24 93.155.131.18
ip route 212.233.172.0/24 93.155.131.18
ip route 212.233.173.0/24 93.155.131.18
ip route 212.233.174.0/24 93.155.131.18
ip route 212.233.175.0/24 93.155.131.18
ip route 212.233.176.0/24 93.155.131.18
ip route 212.233.177.0/24 93.155.131.18
ip route 212.233.178.0/24 93.155.131.18
ip route 212.233.179.0/24 93.155.131.18
ip route 212.233.180.0/24 93.155.131.18
ip route 212.233.181.0/24 93.155.131.18
ip route 212.233.182.0/24 93.155.131.18
ip route 212.233.183.0/24 93.155.131.18
ip route 212.233.184.0/24 93.155.131.18
ip route 212.233.185.0/24 93.155.131.18
ip route 212.233.186.0/24 93.155.131.18
ip route 212.233.187.0/24 93.155.131.18
ip route 212.233.188.0/24 93.155.131.18
ip route 212.233.189.0/24 93.155.131.18
ip route 212.233.190.0/24 93.155.131.18
ip route 212.233.191.0/24 93.155.131.18
ip route 212.233.192.0/24 93.155.131.18
ip route 212.233.193.0/24 93.155.131.18
ip route 212.233.194.0/24 93.155.131.18
ip route 212.233.195.0/24 93.155.131.18
ip route 212.233.196.0/24 93.155.131.18
ip route 212.233.197.0/24 93.155.131.18
ip route 212.233.198.0/24 93.155.131.18
ip route 212.233.199.0/24 93.155.131.18
ip route 212.233.200.0/24 93.155.131.18
ip route 212.233.201.0/24 93.155.131.18
ip route 212.233.202.0/24 93.155.131.18
ip route 212.233.203.0/24 93.155.131.18
ip route 212.233.204.0/24 93.155.131.18
ip route 212.233.205.0/24 93.155.131.18
ip route 212.233.206.0/24 93.155.131.18
ip route 212.233.207.0/24 93.155.131.18
ip route 212.233.208.0/24 93.155.131.18
ip route 212.233.209.0/24 93.155.131.18
ip route 212.233.210.0/24 93.155.131.18
ip route 212.233.211.0/24 93.155.131.18
ip route 212.233.212.0/24 93.155.131.18
ip route 212.233.213.0/24 93.155.131.18
ip route 212.233.214.0/24 93.155.131.18
ip route 212.233.215.0/24 93.155.131.18
ip route 212.233.216.0/24 93.155.131.18
ip route 212.233.217.0/24 93.155.131.18
ip route 212.233.218.0/24 93.155.131.18
ip route 212.233.219.0/24 93.155.131.18
ip route 212.233.220.0/24 93.155.131.18
ip route 212.233.221.0/24 93.155.131.18
ip route 212.233.222.0/24 93.155.131.18
ip route 212.233.223.0/24 93.155.131.18
ip route 212.233.224.0/24 93.155.131.18
ip route 212.233.225.0/24 93.155.131.18
ip route 212.233.226.0/24 93.155.131.18
ip route 212.233.227.0/24 93.155.131.18
ip route 212.233.228.0/24 93.155.131.18
ip route 212.233.229.0/24 93.155.131.18
ip route 212.233.230.0/24 93.155.131.18
ip route 212.233.231.0/24 93.155.131.18
ip route 212.233.232.0/24 93.155.131.18
ip route 212.233.233.0/24 93.155.131.18
ip route 212.233.234.0/24 93.155.131.18
ip route 212.233.235.0/24 93.155.131.18
ip route 212.233.236.0/24 93.155.131.18
ip route 212.233.237.0/24 93.155.131.18
ip route 212.233.238.0/24 93.155.131.18
ip route 212.233.239.0/24 93.155.131.18
ip route 212.233.240.0/24 93.155.131.18
ip route 212.233.241.0/24 93.155.131.18
ip route 212.233.242.0/24 93.155.131.18
ip route 212.233.243.0/24 93.155.131.18
ip route 212.233.244.0/24 93.155.131.18
ip route 212.233.245.0/24 93.155.131.18
ip route 212.233.246.0/24 93.155.131.18
ip route 212.233.247.0/24 93.155.131.18
ip route 212.233.248.0/24 93.155.131.18
ip route 212.233.249.0/24 93.155.131.18
ip route 212.233.250.0/24 93.155.131.18
ip route 212.233.251.0/24 93.155.131.18
ip route 212.233.252.0/24 93.155.131.18
ip route 212.233.253.0/24 93.155.131.18
ip route 212.233.254.0/24 93.155.131.18
ip route 212.233.255.0/24 93.155.131.18
!
ip forwarding
!
!
line vty
!

Quagga /etc/quagga/bgpd.conf

!
! Zebra configuration saved from vty
!   2011/01/03 17:21:21
!
hostname bgpd
password zebra
log stdout
!
router bgp 47453
 bgp router-id 93.155.130.1
 bgp log-neighbor-changes
 network 93.155.130.0/24
 network 93.155.131.0/24
 network 93.155.162.0/24
 network 93.155.169.0/24
 neighbor 212.70.158.89 remote-as 12615
 neighbor 212.70.158.89 description BGP-GCN
 neighbor 212.70.158.89 next-hop-self
 neighbor 212.70.158.89 soft-reconfiguration inbound
 neighbor 212.70.158.89 prefix-list gcn out
!
ip prefix-list gcn seq 10 permit 93.155.130.0/24
ip prefix-list gcn seq 20 permit 93.155.131.0/24
ip prefix-list gcn seq 30 permit 93.155.162.0/24
ip prefix-list gcn seq 40 permit 93.155.169.0/24
!
line vty
!

Netscript is script that start/restart services SNAT, DNAT, TRAFFIC SHAPER, NOPAY WEB SERVICES and small statistics for clients /etc/init.d/netscript

#!/bin/bash
 
#
# NetScript v1.0 - 2004-2011 by Samuil Arsov
# Project homepage: http://itservice-bg.net
# E-mail:           support@itservice-bg.net
#
# NetScript is script that start/restart services SNAT, DNAT, TRAFFIC SHAPER, NOPAY WEB SERVICES
# and small statistics for clients get data from /etc/ipclient syntax lanip:wanip:speed
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License
#
 
#!/bin/bash
 
echo "NetScript"
echo "check duplicate LAN IP"
DUPLICATE=$(cat /etc/ipclient | cut -d":" -f1 | cut -d"#" -f2 | sort | uniq -d)
for DUPLICATE in ${DUPLICATE}; do
    if [ "$DUPLICATE" = "$DUPLICATE" ]
    then
        echo "DUPLICATE LAN IP: ${DUPLICATE} SCRIPT WILL NOT RUN"
exit 1
    else
        continue
    fi
done
 
echo "check duplicate WAN IP"
DUPLICATE2=$(cat /etc/ipclient | cut -d":" -f2 | cut -d"#" -f2 | sort | uniq -d)
for DUPLICATE2 in ${DUPLICATE2}; do
    if [ "$DUPLICATE2" = "$DUPLICATE2" ]
    then
        echo "DUPLICATE WAN IP: ${DUPLICATE2} SCRIPT WILL NOT RUN"
exit 1
    else
        continue
    fi
done
 
echo "flush iptables rules"
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -F
iptables -F INPUT
iptables -F OUTPUT
iptables -F FORWARD
iptables -F -t mangle
iptables -F -t nat
iptables -X
iptables -Z
 
iptables -t nat -A POSTROUTING -o eth0.523 -s 10.125.3.2 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0.523 -s 10.124.3.2 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0.523 -s 10.129.3.7 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0.523 -s 10.121.0.0/16 -j SNAT --to 194.141.68.2
iptables -t nat -A POSTROUTING -o eth0.523 -s 10.122.0.0/16 -j SNAT --to 194.141.68.3
iptables -t nat -A POSTROUTING -o eth0.523 -s 10.123.0.0/16 -j SNAT --to 194.141.68.4
iptables -t nat -A POSTROUTING -o eth0.523 -s 10.124.0.0/16 -j SNAT --to 194.141.68.5
iptables -t nat -A POSTROUTING -o eth0.523 -s 10.125.0.0/16 -j SNAT --to 194.141.68.6
iptables -t nat -A POSTROUTING -o eth0.523 -s 10.126.0.0/16 -j SNAT --to 194.141.68.7
iptables -t nat -A POSTROUTING -o eth0.523 -s 10.127.0.0/16 -j SNAT --to 194.141.68.8
iptables -t nat -A POSTROUTING -o eth0.523 -s 10.128.0.0/16 -j SNAT --to 194.141.68.9
iptables -t nat -A POSTROUTING -o eth0.523 -s 10.129.0.0/16 -j SNAT --to 194.141.68.10
 
echo "start SNAT iptables rules"
ADDRESS=$(grep -v \# /etc/ipclient | cut -d":" -f1);
for LANIP in $ADDRESS; do
WANIP=$(grep -v \# /etc/ipclient | grep -w -n ${LANIP} | cut -d":" -f3)
iptables -t nat -A POSTROUTING -s ${LANIP} -j SNAT --to ${WANIP}
done;
 
echo "start DNAT iptables rules"
ADDRESS=$(grep -v \# /etc/ipclient | cut -d":" -f1);
for LANIP in $ADDRESS; do
WANIP=$(grep -v \# /etc/ipclient | grep -w -n ${LANIP} | cut -d":" -f3)
iptables -t nat -A PREROUTING -d ${WANIP} -j DNAT --to ${LANIP}
done;
 
#echo "mark traffic to MU"
#ADDRESS=$(grep -v \# /etc/ipclient | grep mu | cut -d":" -f1);
#for FWMARK in $ADDRESS; do
#iptables -A PREROUTING -t mangle -s ${FWMARK} -j MARK --set-mark 101
#done;
#FWMARK101=`ip rule | grep fwmark`
#if [ "$FWMARK101" = "" ]; then
#echo "fwmark rule 101 for MU is empty"
#else
#ip rule del fwmark 101
#fi
#echo "fwmark rule 101 for MU adding"
#ip rule add fwmark 101 table T1
 
echo "start REDIRECT no pay clients"
ADDRESS=$(cat /etc/ipclient | grep '#' | cut -d":" -f1 | cut -d"#" -f2);
for NOPAY in $ADDRESS; do
iptables -t nat -A PREROUTING -s ${NOPAY} -p tcp --dport 80 -j DNAT --to 93.155.131.1:80
done;
 
echo "start INPUT iptables rules"
iptables -A INPUT -s 93.155.130.0/24 -p tcp -m multiport --ports 22 -j ACCEPT
iptables -A INPUT -s 93.155.131.0/24 -p tcp -m multiport --ports 22 -j ACCEPT
iptables -A INPUT -s 93.155.162.0/24 -p tcp -m multiport --ports 22 -j ACCEPT
iptables -A INPUT -s 93.155.169.0/24 -p tcp -m multiport --ports 22 -j ACCEPT
iptables -A INPUT -s 212.233.128.0/17 -p tcp -m multiport --ports 22 -j ACCEPT
iptables -A INPUT -s 10.0.0.0/8 -p tcp -m multiport --ports 22 -j ACCEPT
iptables -A INPUT -s 213.226.53.0/24 -p tcp -m multiport --ports 22 -j ACCEPT
iptables -A INPUT -s 213.226.63.0/24 -p tcp -m multiport --ports 22 -j ACCEPT
iptables -A INPUT -p tcp -m multiport --dport 22 -j DROP
 
echo "start FORWARD iptables rules"
iptables -A FORWARD -p tcp -m multiport --dport 137,138,139,445 -j DROP
iptables -A FORWARD -p udp -m multiport --dport 137,138,139,445 -j DROP
 
echo "start TRAFFIC SHAPER global"
DEV1=eth0.100
DEV2=eth1.149
SPEEDIN=30Mbit
SPEEDOUT=5Mbit
SPEEDIN2=45Mbit
SPEEDOUT2=10Mbit
 
QDISCDEV1=$(tc qdisc show dev $DEV1 | grep qdisc)
if [ "$QDISCDEV1" = "" ]; then
	tc qdisc add dev $DEV1 root handle 1: htb default 1
	echo "adding qdisc" $DEV1
else
	tc qdisc del dev $DEV1 root
	tc qdisc add dev $DEV1 root handle 1: htb default 1
	tc class add dev $DEV1 parent 1: classid 1:1 htb rate 1000Mbit
	echo "restart qdisc" $DEV1
fi
 
QDISCDEV2=$(tc qdisc show dev $DEV2 | grep qdisc)
if [ "$QDISCDEV2" = "" ]; then
	tc qdisc add dev $DEV2 root handle 1: htb default 1
	echo "adding qdisc" $DEV2
else
	tc qdisc del dev $DEV2 root
	tc qdisc add dev $DEV2 root handle 1: htb default 1
	tc class add dev $DEV2 parent 1: classid 1:1 htb rate 1000Mbit
	echo "restart qdisc" $DEV2
fi
 
echo "start TRAFFIC SHAPER rules speed1"
ADDRESS=$(grep -v \# /etc/ipclient | grep speed1 | cut -d":" -f1)
for LANIP in $ADDRESS; do
MARK=$(cat /etc/ipclient | grep -w -n $LANIP | cut -d":" -f1)
MARKUP=$(($MARK + 1000))
MARKUP1=$(($MARK + 2000))
iptables -t mangle -A FORWARD -d $LANIP -j CLASSIFY --set-class 1:$MARKUP
tc class add dev $DEV1 parent 1:1 classid 1:$MARKUP htb rate $SPEEDIN prio 7 quantum 1500
tc qdisc add dev $DEV1 parent 1:$MARKUP handle $MARKUP: sfq perturb 10
iptables -t mangle -A FORWARD -s $LANIP -j CLASSIFY --set-class 1:$MARKUP1
tc class add dev $DEV2 parent 1:1 classid 1:$MARKUP1 htb rate $SPEEDOUT prio 7 quantum 1500
tc qdisc add dev $DEV2 parent 1:$MARKUP1 handle $MARKUP1: sfq perturb 10
done;
 
echo "start TRAFFIC SHAPER rules speed2"
ADDRESS=$(grep -v \# /etc/ipclient | grep speed2 | cut -d":" -f1)
for LANIP in $ADDRESS; do
MARK=$(cat /etc/ipclient | grep -w -n $LANIP | cut -d":" -f1)
MARKUP=$(($MARK + 1000))
MARKUP1=$(($MARK + 2000))
iptables -t mangle -A FORWARD -d $LANIP -j CLASSIFY --set-class 1:$MARKUP
tc class add dev $DEV1 parent 1:1 classid 1:$MARKUP htb rate $SPEEDIN2 quantum 1500
tc qdisc add dev $DEV1 parent 1:$MARKUP handle $MARKUP: sfq perturb 10
iptables -t mangle -A FORWARD -s $LANIP -j CLASSIFY --set-class 1:$MARKUP1
tc class add dev $DEV2 parent 1:1 classid 1:$MARKUP1 htb rate $SPEEDOUT2 quantum 1500
tc qdisc add dev $DEV2 parent 1:$MARKUP1 handle $MARKUP1: sfq perturb 10
done;
 
QDISCTUN1=$(tc qdisc show dev tun1 | grep qdisc)
if [ "$QDISCTUN1" = "" ]; then
echo "adding qdisc tun1"
else
tc qdisc del dev tun1 root
tc qdisc del dev tun1 ingress
fi
 
QDISCTUN2=$(tc qdisc show dev tun2 | grep qdisc)
if [ "$QDISCTUN2" = "" ]; then
echo "adding qdisc tun2"
else
tc qdisc del dev tun2 root
tc qdisc del dev tun2 ingress
fi
 
echo "start TRAFFIC SHAPER rules tunnels"
TUNSPEEDIN=45Mbit
TUNSPEEDDOWN=10Mbit
TUNBURST=1Mbit
TUN="tun1 tun2";
for TUN in $TUN; do
tc qdisc add dev $TUN root handle 1: htb default 1
tc class add dev $TUN parent 1: classid 1:1 htb rate $TUNSPEEDIN
tc qdisc add dev $TUN handle ffff: ingress
tc filter add dev $TUN parent ffff: protocol all prio 20 u32 match ip src 0.0.0.0/0 police rate $TUNSPEEDDOWN burst $TUNBURST drop flowid :1
iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o $TUN -j TCPMSS --clamp-mss-to-pmtu
done;
 
echo "Statistic"
CLIENTS=$(grep -v \# /etc/ipclient | grep speed1 | wc -l)
echo clients speed1: $CLIENTS
CLIENTS2=$(grep -v \# /etc/ipclient | grep speed2 | wc -l)
echo clients speed2: $CLIENTS2
#CLIENTSMU=$(grep -v \# /etc/ipclient | grep mu | wc -l)
#echo Clients_MU: $CLIENTSMU
CLIENTSTOTAL=$(grep -v \# /etc/ipclient | wc -l)
echo clients total: $CLIENTSTOTAL
CLIENTSTNOPAY=$(cat /etc/ipclient | grep '#' | cut -d":" -f1 | cut -d"#" -f2 | sort | wc -l)
echo clients no pay: $CLIENTSTNOPAY

ipclient is a file which are IP addresses of clients and that information takes netscript /etc/ipclient
syntax LAN-IP:WAN-IP:TRAFFIC-SPEED

10.125.3.2:93.155.130.40:speed1
10.127.10.2:93.155.130.41:speed1
10.122.2.9:93.155.130.42:speed1
10.123.4.6:93.155.130.43:speed1
10.127.1.5:93.155.130.44:speed1
10.126.9.13:93.155.130.45:speed1
10.125.21.5:93.155.130.46:speed1
10.126.20.13:93.155.130.47:speed1
10.127.9.3:93.155.130.48:speed1
10.126.21.3:93.155.130.49:speed1
10.125.15.6:93.155.130.50:speed1
10.126.1.6:93.155.130.60:speed1
#10.126.17.2:93.155.130.70:speed1
10.125.22.4:93.155.130.62:speed1
10.126.6.2:93.155.130.63:speed1
10.121.9.2:93.155.130.64:speed1
10.121.1.4:93.155.130.65:speed2
10.121.24.10:93.155.130.66:speed1
10.121.1.9:93.155.130.67:speed2
10.122.9.3:93.155.130.68:speed2
10.129.3.7:93.155.130.69:speed1
10.123.11.2:93.155.130.61:speed2
10.123.6.2:93.155.130.71:speed2
10.122.14.4:93.155.130.72:speed1
10.122.2.2:93.155.130.73:speed1
10.125.15.2:93.155.130.74:speed2
10.125.10.5:93.155.130.75:speed2
10.126.18.2:93.155.130.76:speed2
10.127.9.7:93.155.130.77:speed1
10.121.3.6:93.155.130.78:speed1
10.125.13.2:93.155.130.79:speed2
10.128.1.9:93.155.130.80:speed1
10.126.10.2:93.155.130.81:speed1
10.129.1.8:93.155.130.82:speed1
#10.125.12.6:93.155.130.83:speed1
10.121.8.6:93.155.130.84:speed1
10.125.15.4:93.155.130.85:speed1
#10.126.11.5:93.155.130.86:speed1
10.122.8.8:93.155.130.87:speed1
10.125.22.3:93.155.130.88:speed1
#10.125.7.3:93.155.130.89:speed1
10.125.7.2:93.155.130.90:speed1
#10.121.9.3:93.155.130.91:speed1
10.122.3.3:93.155.130.92:speed1
10.125.18.6:93.155.130.93:speed1
10.129.11.6:93.155.130.94:speed1
#10.121.24.11:93.155.130.95:speed1
10.122.17.8:93.155.130.96:speed1
#10.126.1.7:93.155.130.97:speed1
10.125.13.6:93.155.130.98:speed1
10.129.2.9:93.155.130.99:speed1
10.123.8.12:93.155.130.100:speed1
10.126.29.6:93.155.130.101:speed1
10.121.1.6:93.155.130.102:speed1
10.126.1.2:93.155.130.103:speed1
10.126.30.2:93.155.130.104:speed1
#10.125.1.6:93.155.130.105:speed1
10.126.14.16:93.155.130.106:speed1
10.129.1.9:93.155.130.107:speed1
10.126.3.10:93.155.130.108:speed1
10.123.12.8:93.155.130.109:speed1
10.126.15.5:93.155.130.110:speed1
#10.121.17.5:93.155.130.112:speed1
10.125.20.2:93.155.130.113:speed1
10.125.12.4:93.155.130.114:speed1
10.125.19.7:93.155.130.115:speed1
10.126.25.10:93.155.130.116:speed1
10.126.10.9:93.155.130.117:speed1
10.129.6.3:93.155.130.118:speed1
10.121.1.8:93.155.130.119:speed1
10.127.10.6:93.155.130.120:speed1
10.125.22.10:93.155.130.121:speed1
10.129.5.6:93.155.130.122:speed1
#10.125.18.9:93.155.130.123:speed1
10.121.8.2:93.155.130.124:speed1
#10.123.13.4:93.155.130.125:speed1
10.123.16.4:93.155.130.126:speed1
10.121.28.8:93.155.130.127:speed1
10.126.28.7:93.155.130.128:speed1
10.128.6.3:93.155.130.129:speed1
10.128.2.6:93.155.130.130:speed1
10.129.10.6:93.155.130.131:speed1
10.129.13.3:93.155.130.132:speed1
10.121.9.4:93.155.130.133:speed1
10.121.18.5:93.155.130.134:speed1
10.121.2.4:93.155.130.135:speed1
10.121.1.5:93.155.130.136:speed1
10.121.4.2:93.155.130.137:speed1
10.121.11.2:93.155.130.138:speed1
#10.123.21.13:93.155.130.139:speed1
10.121.4.3:93.155.130.140:speed1
10.121.4.4:93.155.130.141:speed1
10.121.18.6:93.155.130.142:speed1
10.121.24.5:93.155.130.143:speed1
10.128.1.4:93.155.130.144:speed1
10.121.28.3:93.155.130.145:speed1
10.121.17.2:93.155.130.146:speed1
10.121.12.2:93.155.130.147:speed1
10.128.2.4:93.155.130.148:speed1
10.123.11.9:93.155.130.149:speed1
10.121.4.6:93.155.130.150:speed1
10.121.4.7:93.155.130.151:speed1
10.121.6.3:93.155.130.152:speed1
10.121.24.6:93.155.130.153:speed1
10.129.7.4:93.155.130.154:speed1
10.121.28.5:93.155.130.155:speed1
10.127.6.6:93.155.130.156:speed1
10.121.1.11:93.155.130.157:speed1
10.123.9.8:93.155.130.158:speed1
10.121.24.7:93.155.130.159:speed1
10.121.10.3:93.155.130.160:speed1
10.121.28.6:93.155.130.161:speed1
10.121.4.12:93.155.130.162:speed1
10.121.3.5:93.155.130.163:speed1
#10.121.24.9:93.155.130.164:speed1
#10.121.26.4:93.155.130.165:speed1
10.121.26.5:93.155.130.166:speed1
10.121.7.3:93.155.130.167:speed1
#10.121.11.3:93.155.130.168:speed1
10.121.2.5:93.155.130.169:speed1
10.129.2.4:93.155.130.170:speed1
10.124.7.7:93.155.130.171:speed1
10.126.20.6:93.155.130.172:speed1
10.122.17.2:93.155.130.173:speed1
10.121.28.7:93.155.130.174:speed1
10.121.14.3:93.155.130.175:speed1
10.121.10.4:93.155.130.176:speed1
10.121.18.7:93.155.130.177:speed1
10.121.2.6:93.155.130.178:speed1
10.121.5.5:93.155.130.179:speed1
10.123.5.9:93.155.130.180:speed1
10.122.5.4:93.155.130.181:speed1
#10.121.26.7:93.155.130.182:speed1
10.121.2.7:93.155.130.183:speed1
10.121.1.14:93.155.130.184:speed1
10.121.4.13:93.155.130.185:speed1
#10.121.17.4:93.155.130.186:speed1
10.121.6.6:93.155.130.187:speed1
10.121.8.3:93.155.130.188:speed1
#10.121.8.4:93.155.130.189:speed1
10.123.9.10:93.155.130.190:speed1
10.121.14.6:93.155.130.191:speed1
10.121.7.2:93.155.130.192:speed1
#10.126.20.15:93.155.130.193:speed1
10.126.4.6:93.155.130.194:speed1
10.121.10.5:93.155.130.195:speed1
10.121.8.5:93.155.130.196:speed1
10.127.2.4:93.155.130.197:speed1
10.121.5.2:93.155.130.198:speed1
10.121.3.10:93.155.130.199:speed1
10.127.17.3:93.155.130.200:speed1
10.121.6.2:93.155.130.201:speed1
10.121.3.3:93.155.130.202:speed1
10.121.25.6:93.155.130.203:speed1
#10.121.18.8:93.155.130.204:speed1
10.121.2.9:93.155.130.205:speed1
#10.121.4.10:93.155.130.206:speed1
10.127.3.6:93.155.130.207:speed1
10.121.6.8:93.155.130.208:speed1
10.128.8.4:93.155.130.209:speed1
10.122.9.2:93.155.130.210:speed1
10.122.5.2:93.155.130.211:speed1
#10.123.14.5:93.155.130.212:speed1
10.127.9.6:93.155.130.213:speed1
10.122.8.10:93.155.130.214:speed1
10.122.7.13:93.155.130.215:speed1
10.122.8.2:93.155.130.216:speed1
10.122.13.2:93.155.130.217:speed1
10.122.6.2:93.155.130.218:speed1
10.122.17.4:93.155.130.219:speed1
10.122.5.5:93.155.130.220:speed1
10.126.20.3:93.155.130.221:speed1
#10.122.14.2:93.155.130.222:speed1
10.122.9.4:93.155.130.223:speed1
#10.122.14.3:93.155.130.224:speed1
#10.129.3.9:93.155.130.225:speed1
10.122.8.3:93.155.130.226:speed1
10.123.14.9:93.155.130.227:speed1
10.122.17.6:93.155.130.228:speed1
10.122.13.3:93.155.130.229:speed1
10.122.6.3:93.155.130.230:speed1
#10.122.13.4:93.155.130.231:speed1
10.122.17.3:93.155.130.232:speed1
10.122.13.5:93.155.130.233:speed1
#10.122.8.4:93.155.130.234:speed1
10.122.5.8:93.155.130.235:speed1
#10.122.8.5:93.155.130.236:speed1
10.122.7.3:93.155.130.237:speed1
10.122.14.5:93.155.130.238:speed1
10.122.8.6:93.155.130.239:speed1
10.123.5.13:93.155.130.240:speed1
10.122.13.6:93.155.130.241:speed1
10.122.9.5:93.155.130.242:speed1
10.122.2.4:93.155.130.243:speed1
10.122.6.4:93.155.130.244:speed1
10.122.2.5:93.155.130.245:speed1
10.122.2.6:93.155.130.246:speed1
#10.122.14.7:93.155.130.247:speed1
10.122.15.3:93.155.130.248:speed1
10.123.16.6:93.155.130.249:speed1
10.121.2.3:93.155.130.250:speed1
10.122.14.8:93.155.130.251:speed1
10.122.5.10:93.155.130.252:speed1
10.126.20.9:93.155.130.253:speed1
10.122.2.8:93.155.130.254:speed1
#:93.155.131.1:employed
#:93.155.131.2:employed
#:93.155.131.3:employed
#:93.155.131.4:employed
#:93.155.131.5:employed
#:93.155.131.6:employed
#:93.155.131.7:employed
#:93.155.131.8:employed
#:93.155.131.9:employed
#:93.155.131.10:employed
#:93.155.131.11:employed
#:93.155.131.12:employed
#:93.155.131.13:employed
#:93.155.131.14:employed
#:93.155.131.15:employed
#:93.155.131.16:employed
#:93.155.131.17:employed
#:93.155.131.18:employed
#:93.155.131.19:employed
#:93.155.131.20:employed
#:93.155.131.21:employed
#:93.155.131.22:employed
#:93.155.131.23:employed
10.128.5.8:93.155.131.24:speed1
10.121.18.3:93.155.131.25:speed1
10.121.18.4:93.155.131.26:speed1
#10.121.2.2:93.155.131.27:speed1
#10.121.24.3:93.155.131.28:speed1
10.124.3.6:93.155.131.29:speed1
10.122.6.5:93.155.131.30:speed1
10.122.9.6:93.155.131.31:speed1
10.122.7.5:93.155.131.32:speed1
#10.124.3.4:93.155.131.33:speed1
#10.122.13.8:93.155.131.34:speed1
10.122.2.7:93.155.131.35:speed1
10.122.5.3:93.155.131.36:speed1
10.122.7.6:93.155.131.37:speed1
#10.122.17.5:93.155.131.38:speed1
#10.122.6.6:93.155.131.39:speed1
10.122.13.7:93.155.131.40:speed1
10.122.8.7:93.155.131.41:speed1
#10.121.25.5:93.155.131.42:speed1
10.122.5.6:93.155.131.43:speed1
10.123.9.2:93.155.131.44:speed1
10.123.5.2:93.155.131.45:speed1
10.123.3.2:93.155.131.46:speed1
10.123.8.2:93.155.131.47:speed1
10.123.8.3:93.155.131.48:speed1
#10.123.3.3:93.155.131.49:speed1
10.123.11.4:93.155.131.50:speed1
10.123.9.3:93.155.131.51:speed1
10.123.11.5:93.155.131.52:speed1
10.123.8.4:93.155.131.53:speed1
10.123.8.5:93.155.131.54:speed1
10.123.6.3:93.155.131.55:speed1
10.123.11.6:93.155.131.56:speed1
10.123.21.2:93.155.131.57:speed1
#10.123.21.3:93.155.131.58:speed1
10.123.7.2:93.155.131.59:speed1
10.123.8.6:93.155.131.60:speed1
10.126.5.4:93.155.131.61:speed1
#10.123.7.5:93.155.131.62:speed1
10.123.8.7:93.155.131.63:speed1
10.123.14.2:93.155.131.64:speed1
10.123.12.5:93.155.131.65:speed1
10.123.13.2:93.155.131.66:speed1
10.123.6.4:93.155.131.67:speed1
10.123.12.6:93.155.131.68:speed1
10.123.2.4:93.155.131.69:speed1
10.123.2.5:93.155.131.70:speed1
10.123.8.8:93.155.131.71:speed1
10.123.9.5:93.155.131.72:speed1
#10.121.14.4:93.155.131.73:speed1
10.123.3.6:93.155.131.74:speed1
10.125.2.6:93.155.131.75:speed1
10.121.19.3:93.155.131.76:speed1
10.123.1.2:93.155.131.77:speed1
10.123.5.6:93.155.131.78:speed1
10.122.6.7:93.155.131.79:speed1
10.129.11.3:93.155.131.80:speed1
10.123.6.5:93.155.131.81:speed1
10.123.4.3:93.155.131.82:speed1
10.121.28.2:93.155.131.83:speed1
#10.123.1.3:93.155.131.84:speed1
10.123.1.4:93.155.131.85:speed1
10.123.6.6:93.155.131.86:speed1
10.125.17.2:93.155.131.87:speed1
#10.127.7.4:93.155.131.88:speed1
10.123.4.5:93.155.131.89:speed1
#10.125.15.7:93.155.131.90:speed1
10.125.4.2:93.155.131.91:speed1
10.123.7.3:93.155.131.92:speed1
10.123.7.6:93.155.131.93:speed1
10.123.21.7:93.155.131.94:speed1
#10.126.14.17:93.155.131.95:speed1
10.123.3.7:93.155.131.96:speed1
#10.123.8.11:93.155.131.97:speed1
10.123.14.6:93.155.131.98:speed1
#10.123.1.5:93.155.131.99:speed1
10.123.9.7:93.155.131.100:speed1
10.123.16.3:93.155.131.101:speed1
10.123.13.3:93.155.131.102:speed1
#10.123.12.9:93.155.131.103:speed1
10.123.12.10:93.155.131.104:speed1
#10.123.5.8:93.155.131.105:speed1
10.123.11.7:93.155.131.105:speed1
#10.123.12.4:93.155.131.106:speed1
10.123.6.7:93.155.131.107:speed1
10.123.16.5:93.155.131.108:speed1
10.123.11.8:93.155.131.109:speed1
10.124.2.2:93.155.131.110:speed1
#10.124.9.3:93.155.131.111:speed1
10.124.7.2:93.155.131.112:speed1
10.124.4.2:93.155.131.113:speed1
10.124.17.2:93.155.131.114:speed1
#10.124.7.3:93.155.131.115:speed1
10.124.9.2:93.155.131.116:speed1
10.124.5.16:93.155.131.117:speed1
#10.124.5.2:93.155.131.118:speed1
#10.124.3.3:93.155.131.119:speed1
10.124.4.3:93.155.131.120:speed1
10.124.5.3:93.155.131.121:speed1
10.123.6.10:93.155.131.122:speed1
10.123.7.8:93.155.131.123:speed1
10.124.5.4:93.155.131.124:speed1
10.123.14.8:93.155.131.125:speed1
10.124.2.5:93.155.131.126:speed1
10.124.5.6:93.155.131.127:speed1
#10.124.6.3:93.155.131.128:speed1
10.124.5.7:93.155.131.129:speed1
10.124.8.2:93.155.131.130:speed1
10.124.12.2:93.155.131.131:speed1
10.124.1.2:93.155.131.132:speed1
10.124.7.4:93.155.131.133:speed1
10.124.5.8:93.155.131.134:speed1
10.124.8.3:93.155.131.135:speed1
#10.124.7.5:93.155.131.136:speed1
10.124.14.3:93.155.131.137:speed1
10.121.11.4:93.155.131.138:speed1
10.124.5.9:93.155.131.139:speed1
10.121.14.5:93.155.131.140:speed1
#10.124.5.10:93.155.131.141:speed1
10.124.1.3:93.155.131.142:speed1
10.124.5.11:93.155.131.143:speed1
10.124.5.12:93.155.131.144:speed1
10.124.5.13:93.155.131.145:speed1
#10.124.8.4:93.155.131.146:speed1
#10.124.1.4:93.155.131.147:speed1
10.124.5.14:93.155.131.148:speed1
10.126.11.15:93.155.131.149:speed1
#10.124.17.4:93.155.131.150:speed1
#10.124.3.5:93.155.131.151:speed1
10.126.25.12:93.155.131.152:speed1
#10.124.7.6:93.155.131.153:speed1
10.121.13.2:93.155.131.154:speed1
10.124.6.2:93.155.131.155:speed1
10.124.2.6:93.155.131.156:speed1
#10.129.13.5:93.155.131.157:speed1
10.125.12.2:93.155.131.158:speed1
10.125.2.2:93.155.131.159:speed1
10.125.22.2:93.155.131.160:speed1
#10.125.10.2:93.155.131.161:speed1
10.125.13.4:93.155.131.162:speed1
10.125.5.2:93.155.131.163:speed1
10.125.11.2:93.155.131.164:speed1
10.125.18.3:93.155.131.165:speed1
10.125.5.5:93.155.131.166:speed1
10.125.2.3:93.155.131.167:speed1
10.125.11.3:93.155.131.168:speed1
10.125.17.3:93.155.131.169:speed1
#10.125.17.4:93.155.131.170:speed1
10.125.3.3:93.155.131.171:speed1
10.123.14.7:93.155.131.172:speed1
10.125.21.3:93.155.131.173:speed1
10.125.5.6:93.155.131.174:speed1
10.125.14.4:93.155.131.175:speed1
10.125.12.5:93.155.131.176:speed1
10.126.26.8:93.155.131.177:speed1
10.125.10.4:93.155.131.178:speed1
10.125.1.2:93.155.131.179:speed1
10.123.6.8:93.155.131.180:speed1
10.125.4.4:93.155.131.181:speed1
10.125.3.5:93.155.131.182:speed1
10.125.15.3:93.155.131.183:speed1
10.125.3.6:93.155.131.184:speed1
10.125.14.5:93.155.131.185:speed1
10.125.1.4:93.155.131.186:speed1
10.125.17.5:93.155.131.187:speed1
10.125.14.6:93.155.131.188:speed1
#10.125.20.3:93.155.131.189:speed1
10.125.22.6:93.155.131.190:speed1
10.126.14.15:93.155.131.191:speed1
10.124.6.4:93.155.131.192:speed1
10.125.16.4:93.155.131.193:speed1
10.123.21.15:93.155.131.194:speed1
10.125.11.5:93.155.131.195:speed1
10.125.18.5:93.155.131.196:speed1
10.125.16.5:93.155.131.197:speed1
10.122.8.9:93.155.131.198:speed1
10.126.4.5:93.155.131.199:speed1
10.125.19.4:93.155.131.200:speed1
10.125.8.2:93.155.131.201:speed1
10.125.4.5:93.155.131.202:speed1
10.125.5.7:93.155.131.203:speed1
#10.125.5.11:93.155.131.204:speed1
#10.125.19.5:93.155.131.205:speed1
#:93.155.131.206:speed2
10.122.13.9:93.155.131.207:speed1
10.127.1.11:93.155.131.208:speed1
10.125.17.8:93.155.131.209:speed1
#10.125.11.6:93.155.131.210:speed1
10.125.14.7:93.155.131.211:speed1
10.125.21.6:93.155.131.212:speed1
10.125.4.6:93.155.131.213:speed1
10.125.5.9:93.155.131.214:speed1
10.125.19.6:93.155.131.215:speed1
10.125.4.7:93.155.131.216:speed1
10.125.17.7:93.155.131.217:speed1
10.125.14.3:93.155.131.218:speed1
10.125.14.2:93.155.131.219:speed1
10.125.5.10:93.155.131.220:speed1
10.126.16.8:93.155.131.221:speed1
10.125.4.8:93.155.131.222:speed1
10.123.8.10:93.155.131.223:speed1
10.127.2.16:93.155.131.224:speed1
10.121.3.7:93.155.131.225:speed1
10.121.25.3:93.155.131.226:speed1
10.121.7.4:93.155.131.227:speed1
10.121.1.2:93.155.131.228:speed1
10.121.26.2:93.155.131.229:speed1
10.121.18.2:93.155.131.230:speed1
10.129.2.7:93.155.131.231:speed1
10.123.12.7:93.155.131.232:speed1
#10.125.10.7:93.155.131.233:speed1
10.125.11.7:93.155.131.234:speed1
10.125.1.7:93.155.131.235:speed1
10.126.13.2:93.155.131.236:speed1
10.128.8.6:93.155.131.237:speed1
10.126.22.2:93.155.131.238:speed1
#10.126.20.2:93.155.131.239:speed1
10.126.9.15:93.155.131.240:speed1
10.128.4.8:93.155.131.241:speed1
10.126.14.2:93.155.131.242:speed1
10.126.14.3:93.155.131.243:speed1
10.126.28.2:93.155.131.244:speed1
10.125.8.4:93.155.131.245:speed1
10.126.16.9:93.155.131.246:speed1
10.126.9.12:93.155.131.247:speed1
10.126.28.4:93.155.131.248:speed1
10.121.1.16:93.155.131.249:speed1
10.126.25.11:93.155.131.250:speed1
10.126.3.2:93.155.131.251:speed1
10.126.25.3:93.155.131.252:speed1
#10.126.5.3:93.155.131.253:speed1
10.126.7.2:93.155.131.254:speed1
10.126.19.2:93.155.162.1:speed1
10.123.8.13:93.155.162.2:speed1
10.126.8.4:93.155.162.3:speed1
10.126.2.4:93.155.162.4:speed1
10.126.13.4:93.155.162.5:speed1
10.126.11.3:93.155.162.6:speed1
10.126.2.5:93.155.162.7:speed1
10.126.13.5:93.155.162.8:speed1
10.126.15.2:93.155.162.9:speed1
10.126.14.4:93.155.162.10:speed1
10.123.6.9:93.155.162.11:speed1
10.126.10.4:93.155.162.12:speed1
10.126.13.6:93.155.162.13:speed1
10.126.1.3:93.155.162.14:speed1
10.126.13.7:93.155.162.15:speed1
10.126.9.4:93.155.162.16:speed1
10.126.12.2:93.155.162.17:speed1
10.126.14.5:93.155.162.18:speed1
10.126.22.6:93.155.162.19:speed1
#10.129.5.8:93.155.162.20:speed1
10.126.4.4:93.155.162.21:speed1
10.126.14.7:93.155.162.22:speed1
10.126.28.5:93.155.162.23:speed1
10.126.12.3:93.155.162.24:speed1
10.126.20.5:93.155.162.25:speed1
10.126.5.5:93.155.162.26:speed1
#10.126.18.3:93.155.162.27:speed1
#10.126.7.3:93.155.162.28:speed1
10.126.21.4:93.155.162.29:speed1
10.126.3.3:93.155.162.30:speed1
#10.126.3.4:93.155.162.31:speed1
10.123.5.10:93.155.162.32:speed1
10.123.3.8:93.155.162.33:speed1
10.126.25.4:93.155.162.34:speed1
10.126.3.5:93.155.162.35:speed1
10.126.26.2:93.155.162.36:speed1
10.126.7.4:93.155.162.37:speed1
#10.126.28.6:93.155.162.38:speed1
10.126.3.6:93.155.162.39:speed1
10.126.29.4:93.155.162.40:speed1
10.126.10.5:93.155.162.41:speed1
10.126.12.5:93.155.162.42:speed1
10.126.14.8:93.155.162.43:speed1
10.126.3.7:93.155.162.44:speed1
10.123.5.14:93.155.162.45:speed1
10.126.25.5:93.155.162.46:speed1
#10.126.6.4:93.155.162.47:speed1
10.126.19.3:93.155.162.48:speed1
#10.126.1.4:93.155.162.49:speed1
10.126.23.2:93.155.162.50:speed1
10.126.19.5:93.155.162.51:speed1
10.126.10.6:93.155.162.52:speed1
10.126.14.9:93.155.162.53:speed1
10.126.9.6:93.155.162.54:speed1
10.126.25.6:93.155.162.55:speed1
10.121.25.4:93.155.162.56:speed1
#10.126.28.8:93.155.162.57:speed1
#10.126.10.7:93.155.162.58:speed1
10.126.15.6:93.155.162.59:speed1
10.126.3.8:93.155.162.60:speed1
10.126.18.4:93.155.162.61:speed1
10.126.30.4:93.155.162.62:speed1
10.126.2.6:93.155.162.63:speed1
10.126.2.7:93.155.162.64:speed1
10.126.25.7:93.155.162.65:speed1
10.126.9.7:93.155.162.66:speed1
10.126.28.3:93.155.162.67:speed1
10.126.14.11:93.155.162.68:speed1
10.126.1.5:93.155.162.67:speed1
10.126.22.7:93.155.162.68:speed1
#10.126.9.8:93.155.162.69:speed1
10.126.16.6:93.155.162.70:speed1
#10.126.16.7:93.155.162.71:speed1
10.126.3.9:93.155.162.72:speed1
10.128.1.8:93.155.162.73:speed1
10.126.14.12:93.155.162.74:speed1
10.126.20.14:93.155.162.75:speed1
10.123.16.8:93.155.162.76:speed1
#10.126.22.8:93.155.162.77:speed1
10.126.9.9:93.155.162.78:speed1
10.126.15.7:93.155.162.79:speed1
10.126.14.13:93.155.162.80:speed1
10.126.26.6:93.155.162.81:speed1
#10.125.12.7:93.155.162.82:speed1
10.126.9.10:93.155.162.83:speed1
10.126.3.11:93.155.162.84:speed1
10.126.15.8:93.155.162.85:speed1
#10.126.19.4:93.155.162.86:speed1
#10.126.20.10:93.155.162.87:speed1
10.126.27.4:93.155.162.88:speed1
10.121.6.5:93.155.162.89:speed1
#10.126.9.11:93.155.162.90:speed1
#10.126.9.3:93.155.162.91:speed1
10.126.9.5:93.155.162.92:speed1
#10.126.15.4:93.155.162.93:speed1
10.126.26.3:93.155.162.94:speed1
10.126.26.7:93.155.162.95:speed1
10.126.10.3:93.155.162.96:speed1
10.126.2.8:93.155.162.97:speed1
10.126.7.5:93.155.162.98:speed1
10.123.5.11:93.155.162.99:speed1
10.126.21.5:93.155.162.100:speed1
10.126.26.5:93.155.162.101:speed1
10.126.10.8:93.155.162.102:speed1
10.126.4.2:93.155.162.103:speed1
#10.126.29.5:93.155.162.104:speed1
10.126.7.6:93.155.162.105:speed1
#10.126.22.3:93.155.162.106:speed1
10.126.2.3:93.155.162.107:speed1
10.126.25.9:93.155.162.108:speed1
#10.126.14.14:93.155.162.109:speed1
10.128.5.9:93.155.162.110:speed1
10.126.27.5:93.155.162.111:speed1
#10.127.3.2:93.155.162.112:speed1
10.127.2.2:93.155.162.113:speed1
10.126.13.10:93.155.162.114:speed1
10.127.17.2:93.155.162.115:speed1
#10.127.3.3:93.155.162.116:speed1
10.123.21.8:93.155.162.117:speed1
10.123.13.6:93.155.162.118:speed1
10.127.5.2:93.155.162.119:speed1
10.127.12.2:93.155.162.120:speed1
10.127.1.4:93.155.162.121:speed1
10.123.9.9:93.155.162.122:speed1
10.127.3.5:93.155.162.123:speed1
10.125.3.10:93.155.162.124:speed1
10.121.3.9:93.155.162.125:speed1
10.127.7.2:93.155.162.126:speed1
10.127.5.3:93.155.162.127:speed1
10.123.21.9:93.155.162.128:speed1
#10.127.2.7:93.155.162.129:speed1
10.127.3.7:93.155.162.130:speed1
10.127.9.4:93.155.162.131:speed1
10.127.5.4:93.155.162.132:speed1
10.127.5.5:93.155.162.133:speed1
#10.129.10.9:93.155.162.134:speed1
10.127.2.9:93.155.162.135:speed1
10.127.1.6:93.155.162.136:speed1
10.126.14.10:93.155.162.137:speed1
10.129.8.5:93.155.162.138:speed1
#10.127.8.4:93.155.162.139:speed1
10.127.9.5:93.155.162.140:speed1
10.127.2.11:93.155.162.141:speed1
10.127.10.5:93.155.162.142:speed1
10.127.6.2:93.155.162.143:speed1
#10.127.5.6:93.155.162.144:speed1
10.127.8.5:93.155.162.145:speed1
10.127.5.7:93.155.162.146:speed1
10.127.2.12:93.155.162.147:speed1
10.127.6.3:93.155.162.148:speed1
10.127.4.4:93.155.162.149:speed1
10.127.3.11:93.155.162.150:speed1
10.127.6.5:93.155.162.151:speed1
10.127.9.8:93.155.162.152:speed1
10.127.5.11:93.155.162.153:speed1
#10.127.12.3:93.155.162.154:speed1
#10.127.2.15:93.155.162.155:speed1
10.123.5.12:93.155.162.156:speed1
10.127.12.5:93.155.162.157:speed1
10.121.26.6:93.155.162.158:speed1
10.127.12.6:93.155.162.159:speed1
10.127.3.10:93.155.162.160:speed1
#10.127.7.5:93.155.162.161:speed1
10.127.7.6:93.155.162.162:speed1
#10.127.3.12:93.155.162.163:speed1
10.127.1.9:93.155.162.164:speed1
10.127.9.9:93.155.162.165:speed1
10.127.9.10:93.155.162.166:speed1
10.127.10.4:93.155.162.167:speed1
#10.127.9.11:93.155.162.168:speed1
10.127.12.4:93.155.162.169:speed1
#10.127.2.13:93.155.162.170:speed1
10.127.8.6:93.155.162.171:speed1
#10.127.1.10:93.155.162.172:speed1
10.127.12.7:93.155.162.173:speed1
10.127.5.9:93.155.162.174:speed1
#10.127.9.12:93.155.162.175:speed1
10.127.7.3:93.155.162.176:speed1
#10.126.15.10:93.155.162.177:speed1
10.128.4.2:93.155.162.178:speed1
10.128.1.3:93.155.162.179:speed1
10.128.5.2:93.155.162.180:speed1
10.128.8.2:93.155.162.181:speed1
10.128.4.3:93.155.162.182:speed1
10.128.6.2:93.155.162.183:speed1
10.128.5.3:93.155.162.184:speed1
10.128.5.4:93.155.162.185:speed1
10.128.3.2:93.155.162.186:speed1
10.128.1.5:93.155.162.187:speed1
10.126.12.4:93.155.162.188:speed2
10.128.6.4:93.155.162.189:speed1
10.128.3.4:93.155.162.190:speed1
#10.128.4.4:93.155.162.191:speed1
10.128.6.5:93.155.162.192:speed1
10.128.8.3:93.155.162.193:speed1
10.121.1.15:93.155.162.194:speed1
10.128.4.6:93.155.162.195:speed1
10.128.6.6:93.155.162.196:speed1
10.128.5.5:93.155.162.197:speed1
#10.128.4.7:93.155.162.198:speed1
10.128.5.6:93.155.162.199:speed1
10.128.1.6:93.155.162.200:speed1
10.128.8.5:93.155.162.201:speed1
10.128.1.7:93.155.162.202:speed1
10.124.2.3:93.155.162.203:speed2
10.123.21.10:93.155.162.204:speed1
#10.128.4.9:93.155.162.205:speed1
#10.128.3.3:93.155.162.206:speed1
10.125.11.9:93.155.162.207:speed1
10.128.4.10:93.155.162.208:speed1
10.128.2.5:93.155.162.209:speed1
10.128.6.7:93.155.162.210:speed1
#10.128.3.5:93.155.162.211:speed1
10.124.13.2:93.155.162.212:speed1
10.129.3.2:93.155.162.213:speed1
10.129.10.2:93.155.162.214:speed1
10.121.1.18:93.155.162.215:speed1
10.129.3.4:93.155.162.216:speed1
10.129.13.2:93.155.162.217:speed1
10.129.1.2:93.155.162.218:speed1
10.129.10.4:93.155.162.219:speed1
10.129.3.6:93.155.162.220:speed1
#10.129.5.3:93.155.162.221:speed1
10.121.26.3:93.155.162.222:speed2
10.125.4.3:93.155.162.223:speed2
#10.129.2.3:93.155.162.224:speed1
10.129.11.4:93.155.162.225:speed1
10.129.11.5:93.155.162.226:speed1
#10.129.10.7:93.155.162.227:speed1
10.123.9.11:93.155.162.228:speed1
10.129.10.8:93.155.162.229:speed1
10.129.7.2:93.155.162.230:speed1
#10.129.5.5:93.155.162.231:speed1
10.129.6.5:93.155.162.232:speed1
#10.129.6.6:93.155.162.233:speed1
10.121.11.5:93.155.162.234:speed1
#10.129.6.8:93.155.162.235:speed1
10.129.13.4:93.155.162.236:speed1
10.129.1.4:93.155.162.237:speed1
10.122.5.14:93.155.162.238:speed1
10.127.3.13:93.155.162.239:speed1
10.129.11.8:93.155.162.240:speed1
10.127.6.7:93.155.162.241:speed1
10.123.16.7:93.155.162.242:speed1
10.121.1.17:93.155.162.243:speed1
#10.129.10.10:93.155.162.244:speed1
10.129.1.6:93.155.162.245:speed1
10.129.1.7:93.155.162.246:speed1
10.129.5.7:93.155.162.247:speed1
10.129.2.8:93.155.162.248:speed1
10.129.3.3:93.155.162.249:speed1
10.126.13.3:93.155.130.250:speed1
#10.126.11.4:93.155.130.251:speed1
10.126.14.6:93.155.130.252:speed1
10.128.2.3:93.155.130.253:speed1
10.121.3.2:93.155.162.254:speed1
10.127.5.10:93.155.169.1:speed1
#10.127.3.9:93.155.169.2:speed1
10.128.4.11:93.155.169.3:speed1
10.125.18.7:93.155.169.4:speed1
10.123.7.7:93.155.169.5:speed1
10.125.20.5:93.155.169.6:speed1
10.122.15.4:93.155.169.7:speed1
#10.124.4.6:93.155.169.8:speed1
10.126.25.8:93.155.169.9:speed1
10.126.22.9:93.155.169.10:speed1
10.126.14.18:93.155.169.11:speed1
10.126.21.6:93.155.169.12:speed1
10.125.8.3:93.155.169.13:speed1
10.126.16.3:93.155.169.14:speed1
10.127.1.3:93.155.169.15:speed1
10.124.7.8:93.155.169.16:speed1
#10.126.29.7:93.155.169.17:speed1
10.122.17.9:93.155.169.18:speed1
10.122.17.10:93.155.169.19:speed1
10.126.15.9:93.155.169.20:speed1
10.122.2.3:93.155.169.21:speed1
10.121.14.8:93.155.169.22:speed1
10.125.14.8:93.155.169.23:speed1
10.121.19.4:93.155.169.24:speed1
10.125.13.7:93.155.169.25:speed1
10.123.21.11:93.155.169.26:speed1
10.123.9.12:93.155.169.27:speed1
10.123.21.12:93.155.169.28:speed1
10.129.12.2:93.155.169.29:speed1
10.123.1.6:93.155.169.30:speed1
10.124.5.5:93.155.169.31:speed1
10.123.1.7:93.155.169.32:speed1
#10.122.14.10:93.155.169.33:speed1
10.126.22.10:93.155.169.34:speed1
10.125.21.7:93.155.169.35:speed1
10.123.2.7:93.155.169.36:speed1
10.123.21.14:93.155.169.37:speed1
10.123.16.9:93.155.169.38:speed1
10.128.5.10:93.155.169.39:speed1
10.123.11.10:93.155.169.40:speed1
10.121.25.7:93.155.169.41:speed1
10.123.8.14:93.155.169.42:speed1
10.121.4.5:93.155.169.43:speed1
10.121.11.14:93.155.169.44:speed1
10.123.13.5:93.155.169.45:speed1
10.121.5.6:93.155.169.46:speed1
10.123.11.11:93.155.169.47:speed1
10.129.1.11:93.155.169.48:speed1
10.127.2.17:93.155.169.49:speed1
10.121.19.2:93.155.169.50:speed1
10.125.10.3:93.155.169.51:speed1
10.121.3.12:93.155.169.52:speed1
10.121.24.14:93.155.169.53:speed1
10.121.4.15:93.155.169.54:speed1
10.124.5.17:93.155.169.55:speed1
10.121.19.5:93.155.169.56:speed1
10.126.8.5:93.155.169.57:speed1
10.121.25.8:93.155.169.58:speed1
10.122.13.10:93.155.169.59:speed1
10.123.5.15:93.155.169.60:speed1
10.123.16.10:93.155.169.61:speed1
10.123.1.8:93.155.169.62:speed1
10.122.2.10:93.155.169.63:speed1
10.123.17.2:93.155.169.64:speed1
10.123.16.11:93.155.169.65:speed1
10.129.1.12:93.155.169.66:speed1
10.125.18.8:93.155.169.67:speed1
10.123.14.10:93.155.169.68:speed1
10.123.11.12:93.155.169.69:speed1
10.125.5.12:93.155.169.70:speed1
10.126.9.14:93.155.169.71:speed1
10.123.16.12:93.155.169.72:speed1
10.123.6.11:93.155.169.73:speed1
10.127.1.12:93.155.169.74:speed1
10.123.3.9:93.155.169.75:speed1
10.123.16.13:93.155.169.76:speed1
10.123.16.14:93.155.169.77:speed1
10.122.3.4:93.155.169.78:speed1
10.122.9.7:93.155.169.79:speed1
10.128.1.10:93.155.169.80:speed1
10.124.9.4:93.155.169.81:speed1
10.126.8.3:93.155.169.82:speed1
10.127.3.8:93.155.169.83:speed1
10.123.5.16:93.155.169.84:speed1

Look interrupts the network cards … cat /proc/interrupts … !!!

            CPU0       CPU1       CPU2       CPU3
   0:       3164       4127       3125       4223   IO-APIC-edge      timer
   1:          0          0          0          2   IO-APIC-edge      i8042
   6:          1          1          1          0   IO-APIC-edge      floppy
   7:          0          0          0          0   IO-APIC-edge      parport0
   8:          0          0          1          0   IO-APIC-edge      rtc0
   9:          0          0          0          0   IO-APIC-fasteoi   acpi
  14:     773596     591837     592793    1796410   IO-APIC-edge      ata_piix
  15:          0          0          0          0   IO-APIC-edge      ata_piix
  16:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb2
  17:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4
  18:          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb5, ata_piix, uhci_hcd:usb8
  22:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb7
  23:          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb3, uhci_hcd:usb6
  76: 1363854121 2065544501  962817137  943646438   PCI-MSI-edge      eth0
  77: 1251160104 1304431999 1732246900 2496445892   PCI-MSI-edge      eth1
  78:   75138017  111299136  158498751   75067291   PCI-MSI-edge      eth2
  79:     402005     390169     397697     835988   PCI-MSI-edge      eth3
 NMI:          0          0          0          0   Non-maskable interrupts
 LOC: 1790256477 3371188297  236746236 4015059766   Local timer interrupts
 SPU:          0          0          0          0   Spurious interrupts
 PMI:          0          0          0          0   Performance monitoring interrupts
 PND:          0          0          0          0   Performance pending work
 RES:    1118337     901333    1145440     833137   Rescheduling interrupts
 CAL:     862183       5641     441760       5730   Function call interrupts
 TLB:    1443919    2399344    1436705    2329519   TLB shootdowns
 TRM:          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0   Threshold APIC interrupts
 MCE:          0          0          0          0   Machine check exceptions
 MCP:      14594      14594      14594      14594   Machine check polls
 ERR:          0
 MIS:          0

NOTE: When traffic increases lan cards to be distributed to individual processor cores to shorten response time.

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
CPU(s):                4
Thread(s) per core:    1
Core(s) per socket:    4
CPU socket(s):         1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 15
Stepping:              11
CPU MHz:               2394.473
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K

As seen in this case we have 4 processor cores in the example in the file /etc/rc.local will associate eth0 to 1-2 cores and eth1 to 3-4 cores. In general, the syntax is this:

echo 1 > /proc/irq/76/smp_affinity # eth0 core 1
echo 2 > /proc/irq/76/smp_affinity # eth0 core 2
echo 3 > /proc/irq/76/smp_affinity # eth0 core 1,2
echo 4 > /proc/irq/76/smp_affinity # eth0 core 3
echo 8 > /proc/irq/76/smp_affinity # eth0 core 4
echo c > /proc/irq/76/smp_affinity # eth0 core 3,4
echo f > /proc/irq/76/smp_affinity # eth0 all cores

Tuning the system at boot. smp_affinity allocation of processor cores on network cards. taskset software allocation of processor cores. modprobe nf * load modules of the firewall. TCP and CONNTRACK tunning in Linux kernel /proc/sys/net and all this in the file / etc/rc.local

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
 
vconfig add eth0 100
vconfig add eth0 523
vconfig add eth1 149
ip tunnel add tun1 mode gre remote 10.129.11.2 local 93.155.131.1 ttl 255
ip tunnel add tun2 mode gre remote 10.129.5.2 local 93.155.131.1 ttl 255
 
echo 3 > /proc/irq/76/smp_affinity #eth0 core 1,2
echo c > /proc/irq/77/smp_affinity #eth1 core 3,4
echo 4 > /proc/irq/78/smp_affinity #eth2 core 2
echo 8 > /proc/irq/79/smp_affinity #eth3 core 4
echo 8 > /proc/irq/14/smp_affinity #hdd core 4
taskset -cp 1,3 820350 #apache
taskset -cp 1,3 1135433 #zebra
taskset -cp 1,3 1135437 #bgpd
taskset -cp 1,3 1323014 #squid
taskset -cp 1,3 1476645 #dnsmasq
taskset -cp 1,3 1666 #snmpd
taskset -cp 1,3 1636 #ntp
taskset -cp 1,3 3835254 #cron
 
modprobe nf_conntrack
modprobe nf_conntrack_ftp
modprobe nf_conntrack_tftp
modprobe nf_nat
modprobe nf_nat_ftp
modprobe nf_nat_tftp
modprobe nf_nat_proto_gre
modprobe nf_nat_sip
modprobe nf_nat_h323
modprobe nf_nat_pptp
 
echo 1 > /proc/sys/net/ipv4/ip_forward
echo 50000000 > /proc/sys/net/nf_conntrack_max
echo 4096 > /proc/sys/net/netfilter/nf_conntrack_expect_max
echo 1 > /proc/sys/net/ipv4/conf/default/rp_filter
echo 1 > /proc/sys/net/ipv4/conf/default/arp_filter
echo 1 > /proc/sys/net/ipv4/conf/all/promote_secondaries
echo 0 > /proc/sys/net/ipv4/conf/default/accept_source_route
echo 0 > /proc/sys/kernel/sysrq
echo 1 > /proc/sys/kernel/core_uses_pid
echo 4194303 > /proc/sys/kernel/pid_max
echo 20000000 > /proc/sys/net/core/rmem_max
echo 20000000 > /proc/sys/net/core/wmem_max
echo 4096 87380 20000000 > /proc/sys/net/ipv4/tcp_rmem
echo 4096 87380 20000000 > /proc/sys/net/ipv4/tcp_wmem
echo 1 > /proc/sys/net/ipv4/tcp_no_metrics_save
echo 1 > /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal
echo 0 > /proc/sys/net/ipv4/tcp_syncookies
 
echo 5 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_syn_sent
echo 5 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_syn_sent2
echo 5 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_syn_recv
echo 86400 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
echo 10 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_fin_wait
echo 10 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close_wait
echo 10 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close
echo 10 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_last_ack
echo 10 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_time_wait
echo 10 > /proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout
echo 180 > /proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout_stream
echo 10 > /proc/sys/net/ipv4/netfilter/ip_conntrack_icmp_timeout
echo 600 > /proc/sys/net/ipv4/netfilter/ip_conntrack_generic_timeout
 
#ip route add 127.0.0.0/8 dev lo table T1
#ip route add 194.141.68.0/27 dev eth0.523 src 194.141.68.2 table T1
#ip route add 93.155.131.16/29 dev eth0.523 src 93.155.131.17 table T1
#ip route add 93.155.131.0/28 dev eth0.100 src 93.155.131.1 table T1
#ip route add 93.155.130.0/28 dev eth2 src 93.155.130.1 table T1
#ip route add default via 194.141.68.1 table T1
#ip route add 212.233.128.0/17 via 93.155.131.18 src 93.155.131.17 dev eth0.523 table T1
#ip route add 194.141.67.0/24 via 194.141.68.1 src 194.141.68.2 dev eth0.523 table T1
#ip route add 194.141.68.0/24 via 194.141.68.1 src 194.141.68.2 dev eth0.523 table T1
#ip route add 194.141.69.0/24 via 194.141.68.1 src 194.141.68.2 dev eth0.523 table T1
 
/etc/init.d/netscript
 
exit 0

NOTE: settings in /proc/sys/net are essential to enable the Linux kernel to pass big traffic.

linux router

Tags: , , , , , , , ,

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

This Post Has 9 Comments

  1. minuvach January 11, 2011

    Само не разбрах защо на всеки адрес трябва да има сорс и дестинеишън нат ?

  2. Самуил Арсов January 12, 2011

    Защото такава е топологията на мрежата, като не искаш нат просто коментирай реда с # и готово …

  3. NetPro January 16, 2011

    Асоциирането на ланкартите към процесора никога не ми се е налагало да го правя но на машина с толкова много правила явно е неизбежно. Няма ли начин да се съкратят правилата и да се махне тоя NAT като гледам на всяко реално ип има и публично. Въпреки всичко инфото е ценно макар и да няма много обяснения кое какво е но е насока 🙂

  4. lover January 16, 2011

    От това което разбирам когато ресурсите на машината започнат да свършват трябва да се разпределят правилно хардуерно в smp_affinity и софтуерно с taskset както и тези настройки в /proc които половината въобще не са ми ясни. Това добре, разбирам го но този вариант на всякакъв хардуер ли става или на определен защото нищо няма споменато на каква машина е ?

  5. NetPro January 26, 2011

    На всякаква, аз вече тествах на един стар батал с две ядра а след това и на един DELL с 8 ядра обаче за ядра 5,6,7,8 трябва да се добавят 0 … например 5 ядро е 10 а 8-мо ядро е 80 … тази функционалност е много яка но аз нямам толкова натоварен сървър за да я видя в действие.

  6. n3t@dministr@tor January 30, 2012

    Интереснооо,
    но на какъв принцип разделяш български от международен трафик?
    Не видях да използваш маркер, за да ги разграничиш.

  7. Самуил Арсов February 1, 2012

    Не го разделям, има достатъчно и не е нужно. Но по принцип е възможно да се направи с тази конфигурация.

  8. Rahan January 18, 2013

    Параметрите в /proc колко много главоболия ми спестиха … 🙂

  9. Ero June 23, 2015

    Hi.thanks for this tutorial.but what did you mean by “NOPAY WEB SERVICES” ?