L4(传输层)IP透明反向代理的实现(传递客户端真实IP)

这种需求，一般来说，会在应用层加个标记标明客户端的IP，例如说HTTP，就是添加个请求头的事情。但并不是所有服务器程序、协议你都能这样下手。所以能不能在不对协议和服务器程序本身做任何改动的情况下，传递客户端的IP呢？

最直接的方法应该就是，把L3的源IP改掉，没对协议进行任何改动，对上层完全透明。先来看看改了L3的source IP会出现什么问题：

上图中的A是个L4负载均衡，Router会把来自Internet的客户端请求转发给A，A再根据策略转发到B、C或D。

假设这时A选择了B，A把请求转发给B时，把L3的源IP改成了客户端的IP，那B收到数据包后，根据B系统上的路由表，回复的包会直接经switch交给router送出Internet，也就是说响应的内容不会经过A送回给客户端。听起来好像没什么问题，还省得A耗费资源转发响应内容，不过TCP需要三次握手，客户端跟A握手成功后，A还需要跟B握手：A这时以客户端的IP向B发出SYN，B收到的SYN后，响应一个目的地IP是客户端的SYN ACK，这个SYN ACK是直接由router送出Internet的，根本不会送到A，所以A跟B是无法成功握手的，那客户端的请求也自然无法送达B。

UDP虽然不用握手，但A转发到B用的UDP源端口不一定跟客户端到A的一样，所以客户端不一定能正常接收到B响应的UDP包。要解决这个问题，就要做到请求从哪转发过来的，响应就得送回哪去，既然是从A来的，就送回给A。

A虽然改了送往B的包的源IP，但即使B把响应送回A了，默认情况下，A收到数据包的目的地IP不是系统已绑定的IP时，也就是非送往本地的数据包，kernel只会选择转发或者丢弃，这种情况下，A跟B也是无法握手成功的，所以，这里还要解决的就是，让A对来自B的非本地数据包，当作本地数据包处理。

到这里，实现这个L4 IP透明代理要解决的问题就很清晰了：

反向代理服务器修改L3源IP
被代理服务器把响应转发回给反向代理服务器
反向代理服务器处理来自被代理服务器的非本地数据包

1. 修改L3源IP

Linux的kernel，有一个IP_TRANSPARENT的选项：

       IP_TRANSPARENT (since Linux 2.6.24)
              Setting this boolean option enables transparent proxying
              on this socket.  This socket option allows the calling
              application to bind to a nonlocal IP address and operate
              both as a client and a server with the foreign address as
              the local endpoint.  NOTE: this requires that routing be
              set up in a way that packets going to the foreign address
              are routed through the TProxy box (i.e., the system
              hosting the application that employs the IP_TRANSPARENT
              socket option).  Enabling this socket option requires
              superuser privileges (the CAP_NET_ADMIN capability).

              TProxy redirection with the iptables TPROXY target also
              requires that this option be set on the redirected socket.

IP_TRANSPARENT (since Linux 2.6.24)

Setting this boolean option enables transparent proxying

on this socket. This socket option allows the calling

application to bind to a nonlocal IP address and operate

both as a client and a server with the foreign address as

the local endpoint. NOTE: this requires that routing be

set up in a way that packets going to the foreign address

are routed through the TProxy box (i.e., the system

hosting the application that employs the IP_TRANSPARENT

socket option). Enabling this socket option requires

superuser privileges (the CAP_NET_ADMIN capability).

TProxy redirection with the iptables TPROXY target also

requires that this option be set on the redirected socket.

简单来说，这个选项允许应用程序的socket，绑定一个操作系统没有绑定的IP，并且以那个IP对外通讯。

所以，A的反向代理程序，通过这个选项让socket绑定客户端的IP，就能以客户端的IP与B进行通讯：

#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <memory.h>

static const int one = 1;

int main(int argc, char *argv[]) {
    int socket_fd = socket(AF_INET, SOCK_STREAM, 0);
    setsockopt(socket_fd, SOL_IP, IP_TRANSPARENT, &one, sizeof(one));

    // 以IP 192.168.2.2作为本地IP绑定socket，端口自动分配
    struct sockaddr_in bind_addr;
    bzero(&bind_addr, sizeof(bind_addr));
    bind_addr.sin_family = AF_INET;
    bind_addr.sin_port = htons(0);
    inet_pton(AF_INET, "192.168.2.2", &bind_addr.sin_addr);
    bind(socket_fd, (void *) &bind_addr, sizeof(bind_addr));

    // 连接到192.168.1.20:80
    struct sockaddr_in server_addr;
    bzero(&server_addr, sizeof(server_addr));
    server_addr.sin_family = AF_INET;
    server_addr.sin_port = htons(80);
    inet_pton(AF_INET, "192.168.1.20", &server_addr.sin_addr);
    if (connect(socket_fd, (void *) &server_addr, sizeof(server_addr)) < 0) {
        close(socket_fd);
        perror("connect()");
        return 1;
    }
    printf("connection established\n");
    close(socket_fd);

    return 0;
}

#include <sys/socket.h>

#include <netinet/in.h>

#include <arpa/inet.h>

#include <memory.h>

static const int one = 1;

int main(int argc, char *argv[]) {

int socket_fd = socket(AF_INET, SOCK_STREAM, 0);

setsockopt(socket_fd, SOL_IP, IP_TRANSPARENT, &one, sizeof(one));

// 以IP 192.168.2.2作为本地IP绑定socket，端口自动分配

struct sockaddr_in bind_addr;

bzero(&bind_addr, sizeof(bind_addr));

bind_addr.sin_family = AF_INET;

bind_addr.sin_port = htons(0);

inet_pton(AF_INET, "192.168.2.2", &bind_addr.sin_addr);

bind(socket_fd, (void *) &bind_addr, sizeof(bind_addr));

// 连接到192.168.1.20:80

struct sockaddr_in server_addr;

bzero(&server_addr, sizeof(server_addr));

server_addr.sin_family = AF_INET;

server_addr.sin_port = htons(80);

inet_pton(AF_INET, "192.168.1.20", &server_addr.sin_addr);

if (connect(socket_fd, (void *) &server_addr, sizeof(server_addr)) < 0) {

close(socket_fd);

perror("connect()");

return 1;

}

printf("connection established\n");

close(socket_fd);

return 0;

}

在A上编译运行上面的源码，抓包可以看到A在以192.168.2.2作为源IP向192.168.1.20:80发送SYN，ss或者netstat也可以看到程序的local IP是192.168.2.2，而A系统本身是没有绑定192.168.2.2的：

目前来说A还是无法与B成功握手的，抓包也只会抓到A一直在向B发送SYN，晚一点还会输出”connect(): Connection timed out”。

2. 让B把响应送回给A

在B上抓包，可以看到B会收到来自2.2的SYN，也会响应SYN ACK，但是会收到2.2的RST：

如何让B辨别来自A的包，并把响应送回A呢？

2.1 选择一：根据B的源IP和端口做策略路由

给B分配多个IP，例如默认用1.19出网，1.20专门给A访问用，可以用这种方法。如果只有一个IP，不建议用这种方法，因为这样B会把所有从被代理的端口发出的数据包都交给A，如果请求不是转发自A的，就无法正常处理。

# 以192.168.1.20:80发出的数据包，使用路由表60的规则
root@B:~# ip rule add from 192.168.1.20 sport 80 lookup 60
# 默认转发给192.168.1.6，也就是A
root@B:~# ip route add default via 192.168.1.6 table 60

# 以192.168.1.20:80发出的数据包，使用路由表60的规则

root@B:~# ip rule add from 192.168.1.20 sport 80 lookup 60

# 默认转发给192.168.1.6，也就是A

root@B:~# ip route add default via 192.168.1.6 table 60

因为A向B发出的请求的目的IP是1.20，所以B发出响应的源IP一定是1.20。而不是对A的响应，B默认使用1.19，不会发送给A。

2.2 选择二：通过mark做策略路由

例如可以通过mac地址识别来自A的包，并且做个标记：

# 查看A的mac地址
root@B:~# cat /proc/net/arp | grep "192.168.1.6"
192.168.1.6      0x1         0x2         52:54:00:91:33:33     *        ens192

# 把来自52:54:00:91:33:33的地址标记为60
root@B:~# iptables -t mangle -A INPUT -m mac --mac-source 52:54:00:91:33:33 -j MARK --set-mark 60
root@B:~# iptables -t mangle -A INPUT -j CONNMARK --save-mark
# 重新标记响应数据包
root@B:~# iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark

# 根据mark做策略路由
root@B:~# ip rule add fwmark 60 lookup 60
root@B:~# ip route add default via 192.168.1.6 table 60

# 查看A的mac地址

root@B:~# cat /proc/net/arp | grep "192.168.1.6"

192.168.1.6 0x1 0x2 52:54:00:91:33:33 * ens192

# 把来自52:54:00:91:33:33的地址标记为60

root@B:~# iptables -t mangle -A INPUT -m mac --mac-source 52:54:00:91:33:33 -j MARK --set-mark 60

root@B:~# iptables -t mangle -A INPUT -j CONNMARK --save-mark

# 重新标记响应数据包

root@B:~# iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark

# 根据mark做策略路由

root@B:~# ip rule add fwmark 60 lookup 60

root@B:~# ip route add default via 192.168.1.6 table 60

3. 让A处理非本地数据包

第二步的策略路由做好后，A上抓包，可以看到B的SYN ACK送回来了，但是A没有响应ACK：

因为A没有绑定192.168.2.2这个IP，前面提到过kernel是不会对非本地的数据包做出响应的。

哪些目的地IP作为要本地IP处理，是可以在路由表上指定的，可以把0.0.0.0/0，也就是任何IP都视为本地IP处理，当然不能加在默认路由表上，这样就无法对外通讯了，需要做策略路由。

# 匹配来自B，且源端口是80的数据包
root@A:~# ip rule add from 192.168.1.20 sport 80 lookup 60
root@A:~# ip route add local 0.0.0.0/0 dev lo table 60

# 匹配来自B，且源端口是80的数据包

root@A:~# ip rule add from 192.168.1.20 sport 80 lookup 60

root@A:~# ip route add local 0.0.0.0/0 dev lo table 60

上面有一个路由规则：“local 0.0.0.0/0”，意思是任何IP都作为本地IP处理，注意规则是加在表60的，符合条件的数据包，才会应用这个表。

第三步的规则做好后，再次运行ip_transparent，就可以看到立刻输出“connection established”了：

4. IP透明反向代理

上面的ip_transparent只是测试A和B之间的握手，并不能作为客户端与B之间通讯的代理，这里用Go实现了一个比较简单的反向代理：

package main

import (
	"fmt"
	"io"
	"log"
	"net"
	"os"
	"strconv"
	"syscall"
)

const network = "tcp"
const bufferSize = 4096

var destinationIP *[4]byte
var destinationPort int

func main() {
	parseDestination()
	listen, err := net.Listen(network, os.Args[1])
	if err != nil {
		log.Println(err)
		os.Exit(1)
	}
	log.Printf("waiting for incomming connection...")
	defer listen.Close()

	for {
		conn, err := listen.Accept()
		if err != nil {
			log.Println(err)
			continue
		}
		log.Printf("client %s accepted", conn.RemoteAddr().String())
		go clientHandler(conn, log.New(os.Stdout, "[" + conn.RemoteAddr().String() + "]", log.Ldate | log.Ltime))
	}
}

func clientHandler(client net.Conn, logger *log.Logger) {
	defer client.Close()
	server, err := createTransparentSocket(client, os.Args[2])
	if err != nil {
		logger.Println(err)
		return
	}
	defer server.Close()

	streamCopy := func(dst io.Writer, src io.Reader, direction string, logger *log.Logger) {
		n, _ := read2Write(dst, src)
		logger.Printf("%s %d bytes copied", direction, n)
		client.Close()
		server.Close()
	}

	go streamCopy(client, server, "to client", logger)
	streamCopy(server, client, "from client", logger)
}

func createTransparentSocket(client net.Conn, destination string) (net.Conn, error) {
	var socketFD int
	var file *os.File
	var server net.Conn
	var errorPrefix string
	var clientIP *net.TCPAddr
	var clientIPBytes [4]byte
	var err error
	var ok bool

	// Create transparent socket
	if socketFD, err = syscall.Socket(syscall.AF_INET, syscall.SOCK_STREAM, syscall.IPPROTO_TCP); err != nil {
		errorPrefix = "syscall.socket()"
		goto ReturnError
	}
	if err = syscall.SetsockoptInt(socketFD, syscall.SOL_IP, syscall.IP_TRANSPARENT, 1); err != nil {
		errorPrefix = "syscall.SetsockoptInt()"
		goto CloseSocket
	}

	if clientIP, ok = client.RemoteAddr().(*net.TCPAddr); ok == false {
		err = fmt.Errorf("unsupport client type")
		errorPrefix = "client.RemoteAddr().(*net.IPAddr)"
		goto CloseSocket
	}
	copy(clientIPBytes[:], clientIP.IP.To4())

	// Use client's IP address as socket local address
	if err = syscall.Bind(socketFD, &syscall.SockaddrInet4{
		Port: 0, // Port auto assignment
		Addr: clientIPBytes,
	}); err != nil {
		errorPrefix = "syscall.Bind()"
		goto CloseSocket
	}

	// Connect to server
	if err = syscall.Connect(socketFD, &syscall.SockaddrInet4{
		Port: destinationPort,
		Addr: *destinationIP,
	}); err != nil && err.Error() != "operation now in progress" {
		errorPrefix = "syscall.Connect()"
		goto CloseSocket
	}

	file = os.NewFile(uintptr(socketFD), destination)
	server, err = net.FileConn(file)
	if err == nil {
		syscall.Close(socketFD) // net.FileConn() has already duplicated this file descriptor
		return server, nil
	}
	errorPrefix = "net.FileConn()"
CloseSocket:
	syscall.Close(socketFD)
ReturnError:
	return nil, fmt.Errorf("%s: %s", errorPrefix, err)
}

func parseDestination() {
	var err error
	var host, port string
	if host, port, err = net.SplitHostPort(os.Args[2]); err != nil {
		println(err)
		os.Exit(1)
	}
	ip := net.ParseIP(host).To4()
	destinationIP = &[4]byte{ip[0], ip[1], ip[2], ip[3]}
	destinationPort, err = strconv.Atoi(port)
	if err != nil {
		println(err)
		os.Exit(1)
	}
}

/**
from: https://github.com/xtaci/kcptun/blob/master/generic/copy.go
*/
func read2Write(dst io.Writer, src io.Reader) (written int64, err error) {
	// If the reader has a WriteTo method, use it to do the copy.
	// Avoids an allocation and a copy.
	if wt, ok := src.(io.WriterTo); ok {
		return wt.WriteTo(dst)
	}
	// Similarly, if the writer has a ReadFrom method, use it to do the copy.
	if rt, ok := dst.(io.ReaderFrom); ok {
		return rt.ReadFrom(src)
	}

	// fallback to standard io.CopyBuffer
	buf := make([]byte, bufferSize)
	return io.CopyBuffer(dst, src, buf)
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

package main

import (

"fmt"

"io"

"log"

"net"

"os"

"strconv"

"syscall"

)

const network = "tcp"

const bufferSize = 4096

var destinationIP *[4]byte

var destinationPort int

func main() {

parseDestination()

listen, err := net.Listen(network, os.Args[1])

if err != nil {

log.Println(err)

os.Exit(1)

}

log.Printf("waiting for incomming connection...")

defer listen.Close()

for {

conn, err := listen.Accept()

if err != nil {

log.Println(err)

continue

}

log.Printf("client %s accepted", conn.RemoteAddr().String())

go clientHandler(conn, log.New(os.Stdout, "[" + conn.RemoteAddr().String() + "]", log.Ldate | log.Ltime))

}

func clientHandler(client net.Conn, logger *log.Logger) {

defer client.Close()

server, err := createTransparentSocket(client, os.Args[2])

if err != nil {

logger.Println(err)

return

}

defer server.Close()

streamCopy := func(dst io.Writer, src io.Reader, direction string, logger *log.Logger) {

n, _ := read2Write(dst, src)

logger.Printf("%s %d bytes copied", direction, n)

client.Close()

server.Close()

}

go streamCopy(client, server, "to client", logger)

streamCopy(server, client, "from client", logger)

}

func createTransparentSocket(client net.Conn, destination string) (net.Conn, error) {

var socketFD int

var file *os.File

var server net.Conn

var errorPrefix string

var clientIP *net.TCPAddr

var clientIPBytes [4]byte

var err error

var ok bool

// Create transparent socket

if socketFD, err = syscall.Socket(syscall.AF_INET, syscall.SOCK_STREAM, syscall.IPPROTO_TCP); err != nil {

errorPrefix = "syscall.socket()"

goto ReturnError

}

if err = syscall.SetsockoptInt(socketFD, syscall.SOL_IP, syscall.IP_TRANSPARENT, 1); err != nil {

errorPrefix = "syscall.SetsockoptInt()"

goto CloseSocket

}

if clientIP, ok = client.RemoteAddr().(*net.TCPAddr); ok == false {

err = fmt.Errorf("unsupport client type")

errorPrefix = "client.RemoteAddr().(*net.IPAddr)"

goto CloseSocket

}

copy(clientIPBytes[:], clientIP.IP.To4())

// Use client's IP address as socket local address

if err = syscall.Bind(socketFD, &syscall.SockaddrInet4{

Port: 0, // Port auto assignment

Addr: clientIPBytes,

}); err != nil {

errorPrefix = "syscall.Bind()"

goto CloseSocket

}

// Connect to server

if err = syscall.Connect(socketFD, &syscall.SockaddrInet4{

Port: destinationPort,

Addr: *destinationIP,

}); err != nil && err.Error() != "operation now in progress" {

errorPrefix = "syscall.Connect()"

goto CloseSocket

}

file = os.NewFile(uintptr(socketFD), destination)

server, err = net.FileConn(file)

if err == nil {

syscall.Close(socketFD) // net.FileConn() has already duplicated this file descriptor

return server, nil

}

errorPrefix = "net.FileConn()"

CloseSocket:

syscall.Close(socketFD)

ReturnError:

return nil, fmt.Errorf("%s: %s", errorPrefix, err)

}

func parseDestination() {

var err error

var host, port string

if host, port, err = net.SplitHostPort(os.Args[2]); err != nil {

println(err)

os.Exit(1)

}

ip := net.ParseIP(host).To4()

destinationIP = &[4]byte{ip[0], ip[1], ip[2], ip[3]}

destinationPort, err = strconv.Atoi(port)

if err != nil {

println(err)

os.Exit(1)

}

/**

from: https://github.com/xtaci/kcptun/blob/master/generic/copy.go

func read2Write(dst io.Writer, src io.Reader) (written int64, err error) {

// If the reader has a WriteTo method, use it to do the copy.

// Avoids an allocation and a copy.

if wt, ok := src.(io.WriterTo); ok {

return wt.WriteTo(dst)

}

// Similarly, if the writer has a ReadFrom method, use it to do the copy.

if rt, ok := dst.(io.ReaderFrom); ok {

return rt.ReadFrom(src)

}

// fallback to standard io.CopyBuffer

buf := make([]byte, bufferSize)

return io.CopyBuffer(dst, src, buf)

}

上面核心的部分是函数func createTransparentSocket(client net.Conn, destination string) (net.Conn, error)，简单来说，就是获取客户端的IP，然后通过bind()绑定为socket的本地IP，再向B发出构建连接的请求。

通过第一个运行参数指定本地监听的IP和端口，第二个运行参数指定被代理服务器的IP和端口：

root@A:~# ./go_ip_transparent_proxy 0.0.0.0:80 192.168.1.20:80
2021/02/10 02:08:53 waiting for incomming connection...

1 2	root@A:~# ./go_ip_transparent_proxy 0.0.0.0:80 192.168.1.20:80 2021/02/10 02:08:53 waiting for incomming connection...

从client向A发出请求：

B上查看Nginx的访问日志，可以看到记录的IP是client的：

A上的反向代理程序的输出：

这个程序只实现了IPv4 TCP的IP透明反向代理，UDP以及IPv6的实现也同理。

除了反向代理，其实正向代理也是可以通过IP_TRANSPARENT实现，配合iptables的TPROXY模块即可发挥作用，本文的目的只是反向代理，正向代理具体细节就不在本文介绍了。

Fix Ubuntu policy-based route with fwmark not work

近期发现Ubuntu基于ip rule add fwmark … lookup …所做的策略路由无法正常使用，而Debian正常。抓包的特征就是，出网数据包能应用到策略路由，也能抓到回应的数据包，但回应的数据包被内核丢弃，不会送达user space。

而ip rule add from … lookup …则正常。

查找资料得知是rp_filter的影响（见此：https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt），继而发现Ubuntu的/etc/sysctl.d是有以下这个文件的：

~# cat /etc/sysctl.d/10-network-security.conf
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks.
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1

# Turn on SYN-flood protections.  Starting with 2.6.26, there is no loss
# of TCP functionality/features under normal conditions.  When flood
# protections kick in under high unanswered-SYN load, the system
# should remain more stable, with a trade off of some loss of TCP
# functionality/features (e.g. TCP Window scaling).
net.ipv4.tcp_syncookies=1

~# cat /etc/sysctl.d/10-network-security.conf

# Turn on Source Address Verification in all interfaces to

# prevent some spoofing attacks.

net.ipv4.conf.default.rp_filter=1

net.ipv4.conf.all.rp_filter=1

# Turn on SYN-flood protections. Starting with 2.6.26, there is no loss

# of TCP functionality/features under normal conditions. When flood

# protections kick in under high unanswered-SYN load, the system

# should remain more stable, with a trade off of some loss of TCP

# functionality/features (e.g. TCP Window scaling).

net.ipv4.tcp_syncookies=1

编辑/etc/sysctl.d/10-network-security.conf，把net.ipv4.conf.default.rp_filter以及net.ipv4.conf.all.rp_filter改成2，然后应用：

~# sysctl --system
...
* Applying /etc/sysctl.d/10-network-security.conf ...
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
net.ipv4.tcp_syncookies = 1
...

~# sysctl --system

...

* Applying /etc/sysctl.d/10-network-security.conf ...

net.ipv4.conf.default.rp_filter = 2

net.ipv4.conf.all.rp_filter = 2

net.ipv4.tcp_syncookies = 1

...

QoS —— 交互式应用最低延迟，HTTP(S)、IMAP等常用服务优先，迅雷、P2P受限

前言

使用迅雷全速下载时，其余网络应用的体验几乎约等于断网状态。

全速下载时，迅雷并没有占用太多上行带宽，于路由器上观察上行速率，同样正常：

可见全速下载时出现的现象，问题在于下行。

迅雷全网搜索资源，这意味着会有许多来自不同地方的计算机，以高速率往我的计算机发来大量数据包。ISP处的队列早已被这些P2P数据包塞满，而SSH、网页等服务的数据包速率不高，根本不够P2P数据包竞争，这些数据包到达ISP处时，只能由于队列已满而被丢弃。

其实这个问题，并没有什么好的解决办法，因为ISP处的队列我们没有权限控制。

我们所能做的，只是控制P2P的速率，避免因ISP处队列满而使得数据包被丢弃。

不过，上行队列我们拥有完全控制权，倒可以自由规划。

蓝图

首先，我的主要目的是，当私有网络内有人使用迅雷等进行全速下载时，这个下载不会影响交互式应用、网页流量等，不受影响终端的包括下载者自己，其次，也要保证下载流的下行速率。

这是一项很有挑战性的任务。

下行流量大致可以分为三类：

最低延迟
优先传送
受限

这里没有普通流量这一类，因为普通流量在P2P软件，尤其是迅雷面前尤其不好区分，我尝试过Deep Packet Inspection，迅雷的流量绝大部分被分类到了Other里面。因此，与其花心思、花系统资源去区分这些“普通流量”，还不如直接使用白名单机制，把最低延迟、优先传送的区分出来。

上行流量：

最低延迟
优先传送
普通
受限

最低延迟

最低延迟的数据包，可能会以IP Header中的TOS来标记，但抓包分析发现，基本上没有使用此字段的数据包，甚至是SSH（但内网的SSH有把TOS设为0x10，表示最低延迟，可能是公网有路由清空了该字段）。所以说，只有KCPTUN之类的会“滥用”该字段，使用该字段区分最低延迟数据包意义不大。

TCP三次握手比较关键的是SYN、SYN & ACK，带有SYN和SYN & ACK标记的我们可以归类为最低延迟，此外ACK的小包（小于64字节），我们也可以归类为最低延迟。

SSH，DNS，远程桌面，QQ以及微信等交互式服务，归类为最低延迟。

优先传送

以下协议归类为优先传送：

HTTP
HTTPS
SMTP
IMAP
GIT

这里有个问题，如果使用迅雷进行HTTP下载，迅雷也会全网搜索HTTP资源，这里正常的HTTP流量与迅雷的HTTP流量不是很好区分。

一个最大的特征是，迅雷可能会与同一个IP建立多个TCP连接，据此可把迅雷下载用的TCP连接归类到受限。

对于单连接的，这里没有好的办法，有以下几个原因：

根据速率区分：前面已提到迅雷会全网搜索资源，意味着迅雷会构建很多TCP连接，单个TCP连接的速率不会很高。阈值低了，影响正常网页流量，高了，无效。

根据传输流量区分：首先HTTP 1.1支持Keep Alive，这意味着对于正常网页的TCP连接来说，流量达到阈值也是有可能的。

上述二者结合：如果我一个网站同时打开多几个页面，岂不是一样被归类为受限？况且现在一个页面数MB，也很正常吧？

根据内网IP构建的TCP连接数区分：我的目的是不要让迅雷影响正常的应用，如果让某个TCP连接数多的IP归类到受限，岂不是不能实现我这个目的？

普通

本分类仅用于上行，未被归类的均归于普通类。

受限

对于下行，未被归类的，均为受限。

P2P的上行发送的都是大于1350字节的UDP数据包，把这些数据包归类为受限即可。

实现

我使用的是中国电信的带宽

下行：100 Mbps，上行：20 Mbps。

互联网接口：eth0，私有网络接口：switch0。

首先设置以下数个变量，方便后面的命令使用：

internetDev="eth0" # 互联网接口
privateNetDev="switch0" # 私有网络接口

uploadRate="20mbit" # 互联网上行带宽
uploadP2PRate="1mbit" # P2P上行带宽
uploadEmerRate=${uploadRate} # 最低延迟上行带宽
uploadPrioRate=${uploadRate} # 优先传送上行带宽
uploadNormalRate="10mbit" # 普通流量上行带宽

downloadRate="100mbit" # 互联网下行带宽
downloadP2PRate="80mbit" # P2P下行带宽
downloadEmerRate=${downloadRate} # 最低延迟传送下行带宽
downloadPrioRate=${downloadRate} # 优先传送下行带宽

internetDev="eth0" # 互联网接口

privateNetDev="switch0" # 私有网络接口

uploadRate="20mbit" # 互联网上行带宽

uploadP2PRate="1mbit" # P2P上行带宽

uploadEmerRate=${uploadRate} # 最低延迟上行带宽

uploadPrioRate=${uploadRate} # 优先传送上行带宽

uploadNormalRate="10mbit" # 普通流量上行带宽

downloadRate="100mbit" # 互联网下行带宽

downloadP2PRate="80mbit" # P2P下行带宽

downloadEmerRate=${downloadRate} # 最低延迟传送下行带宽

downloadPrioRate=${downloadRate} # 优先传送下行带宽

这里我主要的目的在于不要让P2P把ISP处的队列占满，其次对于正常的流量，并不会占用太多带宽，因此正常流量不作速率限制，仅分配不同的优先级，而P2P则限制速率，设置最低的优先级。

区分P2P上行

把大于1350字节的UDP数据包，都标记为100。

# For QoS, MARK 100
iptables -t mangle -N MARK_100
iptables -t mangle -A MARK_100 -p udp -m length ! --length 0:1350 -j MARK --set-mark 100
iptables -t mangle -I PREROUTING -i ${privateNetDev} -j MARK_100

# For QoS, MARK 100

iptables -t mangle -N MARK_100

iptables -t mangle -A MARK_100 -p udp -m length ! --length 0:1350 -j MARK --set-mark 100

iptables -t mangle -I PREROUTING -i ${privateNetDev} -j MARK_100

区分迅雷TCP流

P2P的不需要区分，因为下行使用的是白名单机制。

主要问题在于迅雷的HTTP下载，实难区分，这里用了一个不太尽人意的方法，把迅雷的TCP流标记为200：

# For QoS, MARK 200
iptables -t mangle -N PRIVATE_NET_MARK_200
iptables -t mangle -A PRIVATE_NET_MARK_200 -p tcp -j CONNMARK --restore-mark
iptables -t mangle -A PRIVATE_NET_MARK_200 -m mark --mark 200 -j RETURN
iptables -t mangle -A PRIVATE_NET_MARK_200 -m connlimit --connlimit-above 4 --connlimit-mask 24 --connlimit-saddr -m hashlimit --hashlimit-above 512kb/second --hashlimit-burst 5mb --hashlimit-mode srcip,dstip --hashlimit-dstmask 24 --hashlimit-srcmask 32 --hashlimit-name rateLimit -j MARK --set-mark 200
iptables -t mangle -A PRIVATE_NET_MARK_200 -j CONNMARK --save-mark
iptables -t mangle -I POSTROUTING -o ${privateNetDev} -j PRIVATE_NET_MARK_200

# For QoS, MARK 200

iptables -t mangle -N PRIVATE_NET_MARK_200

iptables -t mangle -A PRIVATE_NET_MARK_200 -p tcp -j CONNMARK --restore-mark

iptables -t mangle -A PRIVATE_NET_MARK_200 -m mark --mark 200 -j RETURN

iptables -t mangle -A PRIVATE_NET_MARK_200 -m connlimit --connlimit-above 4 --connlimit-mask 24 --connlimit-saddr -m hashlimit --hashlimit-above 512kb/second --hashlimit-burst 5mb --hashlimit-mode srcip,dstip --hashlimit-dstmask 24 --hashlimit-srcmask 32 --hashlimit-name rateLimit -j MARK --set-mark 200

iptables -t mangle -A PRIVATE_NET_MARK_200 -j CONNMARK --save-mark

iptables -t mangle -I POSTROUTING -o ${privateNetDev} -j PRIVATE_NET_MARK_200

首先，不能在互联网接口上对TCP流进行归类，因为在互联网接口上，所有TCP流的目的地都是我们互联网接口的IP，这导致的问题就是无法独立统计单个内网IP对单个远程IP的链接数。想一想，如果网内有多个人同时访问某个网站，是不是有可能被归类为受限？

因此这里在私有网络的接口上进行归类。

归类用到了两个模块，分别是connlimit，hashlimit。

connlimit模块仅能统计单端的链接数，hashlimit模块能双端统计，但是只能统计速率，因此配合使用，以接近限制端对端链接数的效果。

这里connlimit用到的参数，表示远程一个/24与本地构建的TCP流超过四的情况；hashlimit的参数表示远程一个/24与本地一个IP突发5MB后，数据传输速率>512kb/s的情况。

这里还用到了connmark模块，这个模块用于记录TCP流的标记，下次遇到相同的TCP流时，取上次的标记即可，无需再使用connlimit与hashlimit模块进行判断，因为这两个模块资源消耗不低。

构建上行规则

这里使用了HTB对速率进行限制，并在每个类下面安置了一个SFQ队列，以公平分配一个队列中各流的带宽：

tc qdisc del dev ${internetDev} root # 删除现有的规则
tc qdisc add dev ${internetDev} root handle 1: htb default 50 # 默认归类到类50

tc class add dev ${internetDev} parent 1: classid 1:1 htb rate ${uploadRate}
tc class add dev ${internetDev} parent 1:1 classid 1:100 htb rate ${uploadP2PRate} ceil ${uploadP2PRate} prio 7 # P2P 优先级7
tc class add dev ${internetDev} parent 1:1 classid 1:10 htb rate ${uploadEmerRate} prio 0 # 最低延迟 优先级0
tc class add dev ${internetDev} parent 1:1 classid 1:20 htb rate ${uploadPrioRate} prio 2 # 优先传送 优先级2
tc class add dev ${internetDev} parent 1:1 classid 1:50 htb rate ${uploadNormalRate} prio 5 # 普通 优先级5

# 随机公平队列
tc qdisc add dev ${internetDev} parent 1:100 handle 100: sfq perturb 10
tc qdisc add dev ${internetDev} parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev ${internetDev} parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev ${internetDev} parent 1:50 handle 50: sfq perturb 10

tc qdisc del dev ${internetDev} root # 删除现有的规则

tc qdisc add dev ${internetDev} root handle 1: htb default 50 # 默认归类到类50

tc class add dev ${internetDev} parent 1: classid 1:1 htb rate ${uploadRate}

tc class add dev ${internetDev} parent 1:1 classid 1:100 htb rate ${uploadP2PRate} ceil ${uploadP2PRate} prio 7 # P2P 优先级7

tc class add dev ${internetDev} parent 1:1 classid 1:10 htb rate ${uploadEmerRate} prio 0 # 最低延迟优先级0

tc class add dev ${internetDev} parent 1:1 classid 1:20 htb rate ${uploadPrioRate} prio 2 # 优先传送优先级2

tc class add dev ${internetDev} parent 1:1 classid 1:50 htb rate ${uploadNormalRate} prio 5 # 普通优先级5

# 随机公平队列

tc qdisc add dev ${internetDev} parent 1:100 handle 100: sfq perturb 10

tc qdisc add dev ${internetDev} parent 1:10 handle 10: sfq perturb 10

tc qdisc add dev ${internetDev} parent 1:20 handle 20: sfq perturb 10

tc qdisc add dev ${internetDev} parent 1:50 handle 50: sfq perturb 10

接下来就把数据包归类，我选择使用u32选择器。

u32选择器强大，同时也难用。

u8, u16都会转换为u32，uX跟着一个值与掩码，at后面是偏移量，偏移从IP Header开始。

掩码将会与数据包偏移位置处进行位于，再与给定的值进行比较。

更多u32选择器的细节，见u32选择器的手册页：Universal 32bit classifier in tc(8) LinuxUniversal 32bit classifier in tc(8)

分类规则：

# Match TCP SYN
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 1 u32 \
match u8 0x10 0xff at 33 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match TCP ACK Small Packet (< 64 Bytes)
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 1 u32 \
match u8 0x10 0xff at 33 \
match u8 0x06 0xff at 9 \
match u16 0x0000 0xffc0 at 2 \
flowid 1:10

# Match TCP DESTINATION PORT 22
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0016 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match TCP DESTINATION PORT 23
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0017 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match TCP DESTINATION PORT 25
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0019 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP DESTINATION PORT 53
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0035 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match UDP DESTINATION PORT 53
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0035 0xffff at 22 \
match u8 0x11 0xff at 9 \
flowid 1:10

# Match TCP DESTINATION PORT 80
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0050 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP DESTINATION PORT 443
tc filter add dev ${internetDev} protocol ip parent 1: pref 10 u32 \
match u16 0x01bb 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match UDP DESTINATION PORT 443 (QUIC Protocol)
tc filter add dev ${internetDev} protocol ip parent 1: pref 10 u32 \
match u16 0x01bb 0xffff at 22 \
match u8 0x11 0xff at 9 \
flowid 1:20

# Match TCP DESTINATION PORT 465
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x01d1 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP DESTINATION PORT 993
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x03e1 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP DESTINATION PORT 3306
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0cea 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP DESTINATION PORT 3389
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0d3d 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match UDP DESTINATION PORT 3389
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0d3d 0xffff at 22 \
match u8 0x11 0xff at 9 \
flowid 1:10

# Match UDP DESTINATION PORT 8000 (QQ Port)
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x1f40 0xffff at 22 \
match u8 0x11 0xff at 9 \
flowid 1:10

# Match TCP DESTINATION PORT 8080 (WeChat Port)
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x1f90 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match TCP DESTINATION PORT 9418 (GIT)
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x24ca 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Limit UDP Huge Packet Rate
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 20 u32 \
match mark 0x0064 0x00ff \
flowid 1:100

100

101

102

103

104

105

106

107

108

109

110

111

112

113

# Match TCP SYN

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 1 u32 \

match u8 0x10 0xff at 33 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match TCP ACK Small Packet (< 64 Bytes)

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 1 u32 \

match u8 0x10 0xff at 33 \

match u8 0x06 0xff at 9 \

match u16 0x0000 0xffc0 at 2 \

flowid 1:10

# Match TCP DESTINATION PORT 22

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0016 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match TCP DESTINATION PORT 23

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0017 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match TCP DESTINATION PORT 25

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0019 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP DESTINATION PORT 53

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0035 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match UDP DESTINATION PORT 53

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0035 0xffff at 22 \

match u8 0x11 0xff at 9 \

flowid 1:10

# Match TCP DESTINATION PORT 80

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0050 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP DESTINATION PORT 443

tc filter add dev ${internetDev} protocol ip parent 1: pref 10 u32 \

match u16 0x01bb 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match UDP DESTINATION PORT 443 (QUIC Protocol)

tc filter add dev ${internetDev} protocol ip parent 1: pref 10 u32 \

match u16 0x01bb 0xffff at 22 \

match u8 0x11 0xff at 9 \

flowid 1:20

# Match TCP DESTINATION PORT 465

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x01d1 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP DESTINATION PORT 993

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x03e1 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP DESTINATION PORT 3306

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0cea 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP DESTINATION PORT 3389

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0d3d 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match UDP DESTINATION PORT 3389

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0d3d 0xffff at 22 \

match u8 0x11 0xff at 9 \

flowid 1:10

# Match UDP DESTINATION PORT 8000 (QQ Port)

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x1f40 0xffff at 22 \

match u8 0x11 0xff at 9 \

flowid 1:10

# Match TCP DESTINATION PORT 8080 (WeChat Port)

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x1f90 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match TCP DESTINATION PORT 9418 (GIT)

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x24ca 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Limit UDP Huge Packet Rate

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 20 u32 \

match mark 0x0064 0x00ff \

flowid 1:100

构建下行规则

队列的构建与上面同理，只是少了一个普通流量的类：

tc qdisc del dev ${privateNetDev} root # 删除现有的队列
tc qdisc add dev ${privateNetDev} root handle 1: htb default 100 # 默认归类到100

tc class add dev ${privateNetDev} parent 1: classid 1:1 htb rate ${downloadRate}
tc class add dev ${privateNetDev} parent 1:1 classid 1:100 htb rate ${downloadP2PRate} ceil ${downloadP2PRate} prio 7 # P2P
tc class add dev ${privateNetDev} parent 1:1 classid 1:10 htb rate ${downloadEmerRate} prio 0 # 最低延迟
tc class add dev ${privateNetDev} parent 1:1 classid 1:20 htb rate ${downloadPrioRate} prio 2 # 优先传送

# 随机公平队列
tc qdisc add dev ${privateNetDev} parent 1:100 handle 100: sfq perturb 10
tc qdisc add dev ${privateNetDev} parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev ${privateNetDev} parent 1:20 handle 20: sfq perturb 10

tc qdisc del dev ${privateNetDev} root # 删除现有的队列

tc qdisc add dev ${privateNetDev} root handle 1: htb default 100 # 默认归类到100

tc class add dev ${privateNetDev} parent 1: classid 1:1 htb rate ${downloadRate}

tc class add dev ${privateNetDev} parent 1:1 classid 1:100 htb rate ${downloadP2PRate} ceil ${downloadP2PRate} prio 7 # P2P

tc class add dev ${privateNetDev} parent 1:1 classid 1:10 htb rate ${downloadEmerRate} prio 0 # 最低延迟

tc class add dev ${privateNetDev} parent 1:1 classid 1:20 htb rate ${downloadPrioRate} prio 2 # 优先传送

# 随机公平队列

tc qdisc add dev ${privateNetDev} parent 1:100 handle 100: sfq perturb 10

tc qdisc add dev ${privateNetDev} parent 1:10 handle 10: sfq perturb 10

tc qdisc add dev ${privateNetDev} parent 1:20 handle 20: sfq perturb 10

归类：

# Match TCP SYN ACK
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 1 u32 \
match u8 0x12 0xff at 33 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match TCP ACK Small Packet (< 64 Bytes)
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 1 u32 \
match u8 0x10 0xff at 33 \
match u8 0x06 0xff at 9 \
match u16 0x0000 0xffc0 at 2 \
flowid 1:10

# Match multip thread download
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 5 u32 \
match mark 0x00c8 0x00ff \
flowid 1:100

# Match TCP SOURCE PORT 22
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0016 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match TCP SOURCE PORT 23
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0017 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match TCP SOURCE PORT 25
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0019 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP SOURCE PORT 53
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0035 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match UDP SOURCE PORT 53
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0035 0xffff at 20 \
match u8 0x11 0xff at 9 \
flowid 1:10

# Match TCP SOURCE PORT 80
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0050 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP SOURCE PORT 443
tc filter add dev ${privateNetDev} protocol ip parent 1: pref 10 u32 \
match u16 0x01bb 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match UDP SOURCE PORT 443 (QUIC Protocol)
tc filter add dev ${privateNetDev} protocol ip parent 1: pref 10 u32 \
match u16 0x01bb 0xffff at 20 \
match u8 0x11 0xff at 9 \
flowid 1:20

# Match TCP SOURCE PORT 465
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x01d1 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP SOURCE PORT 993
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x03e1 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP SOURCE PORT 3306
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0cea 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP SOURCE PORT 3389
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0d3d 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match UDP SOURCE PORT 3389
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0d3d 0xffff at 20 \
match u8 0x11 0xff at 9 \
flowid 1:10

# Match UDP SOURCE PORT 8000 (QQ Port)
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x1f40 0xffff at 20 \
match u8 0x11 0xff at 9 \
flowid 1:10

# Match TCP SOURCE PORT 8080 (WeChat Port)
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x1f90 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match TCP SOURCE PORT 9418 (GIT)
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x24ca 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:20

100

101

102

103

104

105

106

107

108

109

110

111

112

113

# Match TCP SYN ACK

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 1 u32 \

match u8 0x12 0xff at 33 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match TCP ACK Small Packet (< 64 Bytes)

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 1 u32 \

match u8 0x10 0xff at 33 \

match u8 0x06 0xff at 9 \

match u16 0x0000 0xffc0 at 2 \

flowid 1:10

# Match multip thread download

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 5 u32 \

match mark 0x00c8 0x00ff \

flowid 1:100

# Match TCP SOURCE PORT 22

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0016 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match TCP SOURCE PORT 23

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0017 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match TCP SOURCE PORT 25

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0019 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP SOURCE PORT 53

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0035 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match UDP SOURCE PORT 53

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0035 0xffff at 20 \

match u8 0x11 0xff at 9 \

flowid 1:10

# Match TCP SOURCE PORT 80

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0050 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP SOURCE PORT 443

tc filter add dev ${privateNetDev} protocol ip parent 1: pref 10 u32 \

match u16 0x01bb 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match UDP SOURCE PORT 443 (QUIC Protocol)

tc filter add dev ${privateNetDev} protocol ip parent 1: pref 10 u32 \

match u16 0x01bb 0xffff at 20 \

match u8 0x11 0xff at 9 \

flowid 1:20

# Match TCP SOURCE PORT 465

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x01d1 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP SOURCE PORT 993

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x03e1 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP SOURCE PORT 3306

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0cea 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP SOURCE PORT 3389

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0d3d 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match UDP SOURCE PORT 3389

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0d3d 0xffff at 20 \

match u8 0x11 0xff at 9 \

flowid 1:10

# Match UDP SOURCE PORT 8000 (QQ Port)

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x1f40 0xffff at 20 \

match u8 0x11 0xff at 9 \

flowid 1:10

# Match TCP SOURCE PORT 8080 (WeChat Port)

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x1f90 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match TCP SOURCE PORT 9418 (GIT)

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x24ca 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:20

完整的Script

#!/bin/bash
internetDev="eth0" # 互联网接口
privateNetDev="switch0" # 私有网络接口

uploadRate="20mbit" # 互联网上行带宽
uploadP2PRate="1mbit" # P2P上行带宽
uploadEmerRate=${uploadRate} # 最低延迟上行带宽
uploadPrioRate=${uploadRate} # 优先传送上行带宽
uploadNormalRate="10mbit" # 普通流量上行带宽

downloadRate="100mbit" # 互联网下行带宽
downloadP2PRate="80mbit" # P2P下行带宽
downloadEmerRate=${downloadRate} # 最低延迟传送下行带宽
downloadPrioRate=${downloadRate} # 优先传送下行带宽

# For QoS, MARK 100
iptables -t mangle -F MARK_100 > /dev/null 2>&1
iptables -t mangle -N MARK_100 > /dev/null 2>&1
iptables -t mangle -A MARK_100 -p udp -m length ! --length 0:1350 -j MARK --set-mark 100
while iptables -t mangle -D PREROUTING -i ${privateNetDev} -j MARK_100 >/dev/null 2>&1; do continue; done
iptables -t mangle -I PREROUTING -i ${privateNetDev} -j MARK_100

tc qdisc del dev ${internetDev} root >/dev/null 2>&1
tc qdisc add dev ${internetDev} root handle 1: htb default 50

tc class add dev ${internetDev} parent 1: classid 1:1 htb rate ${uploadRate}
tc class add dev ${internetDev} parent 1:1 classid 1:100 htb rate ${uploadP2PRate} ceil ${uploadP2PRate} prio 7
tc class add dev ${internetDev} parent 1:1 classid 1:10 htb rate ${uploadEmerRate} prio 0
tc class add dev ${internetDev} parent 1:1 classid 1:20 htb rate ${uploadPrioRate} prio 2
tc class add dev ${internetDev} parent 1:1 classid 1:50 htb rate ${uploadNormalRate} prio 5

tc qdisc add dev ${internetDev} parent 1:100 handle 100: sfq perturb 10
tc qdisc add dev ${internetDev} parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev ${internetDev} parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev ${internetDev} parent 1:50 handle 50: sfq perturb 10

# Match TCP SYN
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 1 u32 \
match u8 0x10 0xff at 33 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match TCP ACK Small Packet (< 64 Bytes)
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 1 u32 \
match u8 0x10 0xff at 33 \
match u8 0x06 0xff at 9 \
match u16 0x0000 0xffc0 at 2 \
flowid 1:10

# Match TCP DESTINATION PORT 22
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0016 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match TCP DESTINATION PORT 23
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0017 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match TCP DESTINATION PORT 25
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0019 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP DESTINATION PORT 53
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0035 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match UDP DESTINATION PORT 53
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0035 0xffff at 22 \
match u8 0x11 0xff at 9 \
flowid 1:10

# Match TCP DESTINATION PORT 80
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0050 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP DESTINATION PORT 443
tc filter add dev ${internetDev} protocol ip parent 1: pref 10 u32 \
match u16 0x01bb 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match UDP DESTINATION PORT 443 (QUIC Protocol)
tc filter add dev ${internetDev} protocol ip parent 1: pref 10 u32 \
match u16 0x01bb 0xffff at 22 \
match u8 0x11 0xff at 9 \
flowid 1:20

# Match TCP DESTINATION PORT 465
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x01d1 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP DESTINATION PORT 993
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x03e1 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP DESTINATION PORT 3306
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0cea 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP DESTINATION PORT 3389
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0d3d 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match UDP DESTINATION PORT 3389
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0d3d 0xffff at 22 \
match u8 0x11 0xff at 9 \
flowid 1:10

# Match UDP DESTINATION PORT 8000 (QQ Port)
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x1f40 0xffff at 22 \
match u8 0x11 0xff at 9 \
flowid 1:10

# Match TCP DESTINATION PORT 8080 (WeChat Port)
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x1f90 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match TCP DESTINATION PORT 9418 (GIT)
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x24ca 0xffff at 22 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Limit UDP Huge Packet Rate
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 20 u32 \
match mark 0x0064 0x00ff \
flowid 1:100



# For QoS, MARK 200
iptables -t mangle -F PRIVATE_NET_MARK_200 >/dev/null 2>&1
iptables -t mangle -N PRIVATE_NET_MARK_200 >/dev/null 2>&1
iptables -t mangle -A PRIVATE_NET_MARK_200 -p tcp -j CONNMARK --restore-mark
iptables -t mangle -A PRIVATE_NET_MARK_200 -m mark --mark 200 -j RETURN
iptables -t mangle -A PRIVATE_NET_MARK_200 -m connlimit --connlimit-above 4 --connlimit-mask 24 --connlimit-saddr -m hashlimit --hashlimit-above 512kb/second --hashlimit-burst 5mb --hashlimit-mode srcip,dstip --hashlimit-dstmask 24 --hashlimit-srcmask 32 --hashlimit-name packetRateLimit -j MARK --set-mark 200
iptables -t mangle -A PRIVATE_NET_MARK_200 -j CONNMARK --save-mark
while iptables -t mangle -D POSTROUTING -o ${privateNetDev} -j PRIVATE_NET_MARK_200 >/dev/null 2>&1; do continue; done
iptables -t mangle -I POSTROUTING -o ${privateNetDev} -j PRIVATE_NET_MARK_200

tc qdisc del dev ${privateNetDev} root >/dev/null 2>&1
tc qdisc add dev ${privateNetDev} root handle 1: htb default 100

tc class add dev ${privateNetDev} parent 1: classid 1:1 htb rate ${downloadRate}
tc class add dev ${privateNetDev} parent 1:1 classid 1:100 htb rate ${downloadP2PRate} ceil ${downloadP2PRate} prio 7
tc class add dev ${privateNetDev} parent 1:1 classid 1:10 htb rate ${downloadEmerRate} prio 0
tc class add dev ${privateNetDev} parent 1:1 classid 1:20 htb rate ${downloadPrioRate} prio 2

tc qdisc add dev ${privateNetDev} parent 1:100 handle 100: sfq perturb 10
tc qdisc add dev ${privateNetDev} parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev ${privateNetDev} parent 1:20 handle 20: sfq perturb 10


# Match TCP SYN ACK
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 1 u32 \
match u8 0x12 0xff at 33 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match TCP ACK Small Packet (< 64 Bytes)
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 1 u32 \
match u8 0x10 0xff at 33 \
match u8 0x06 0xff at 9 \
match u16 0x0000 0xffc0 at 2 \
flowid 1:10

# Match multip thread download
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 5 u32 \
match mark 0x00c8 0x00ff \
flowid 1:100

# Match TCP SOURCE PORT 22
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0016 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match TCP SOURCE PORT 23
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0017 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match TCP SOURCE PORT 25
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0019 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP SOURCE PORT 53
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0035 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match UDP SOURCE PORT 53
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0035 0xffff at 20 \
match u8 0x11 0xff at 9 \
flowid 1:10

# Match TCP SOURCE PORT 80
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0050 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP SOURCE PORT 443
tc filter add dev ${privateNetDev} protocol ip parent 1: pref 10 u32 \
match u16 0x01bb 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match UDP SOURCE PORT 443 (QUIC Protocol)
tc filter add dev ${privateNetDev} protocol ip parent 1: pref 10 u32 \
match u16 0x01bb 0xffff at 20 \
match u8 0x11 0xff at 9 \
flowid 1:20

# Match TCP SOURCE PORT 465
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x01d1 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP SOURCE PORT 993
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x03e1 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP SOURCE PORT 3306
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0cea 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:20

# Match TCP SOURCE PORT 3389
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0d3d 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match UDP SOURCE PORT 3389
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x0d3d 0xffff at 20 \
match u8 0x11 0xff at 9 \
flowid 1:10

# Match UDP SOURCE PORT 8000 (QQ Port)
tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x1f40 0xffff at 20 \
match u8 0x11 0xff at 9 \
flowid 1:10

# Match TCP SOURCE PORT 8080 (WeChat Port)
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x1f90 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:10

# Match TCP SOURCE PORT 9418 (GIT)
tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \
match u16 0x24ca 0xffff at 20 \
match u8 0x06 0xff at 9 \
flowid 1:20

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

#!/bin/bash

internetDev="eth0" # 互联网接口

privateNetDev="switch0" # 私有网络接口

uploadRate="20mbit" # 互联网上行带宽

uploadP2PRate="1mbit" # P2P上行带宽

uploadEmerRate=${uploadRate} # 最低延迟上行带宽

uploadPrioRate=${uploadRate} # 优先传送上行带宽

uploadNormalRate="10mbit" # 普通流量上行带宽

downloadRate="100mbit" # 互联网下行带宽

downloadP2PRate="80mbit" # P2P下行带宽

downloadEmerRate=${downloadRate} # 最低延迟传送下行带宽

downloadPrioRate=${downloadRate} # 优先传送下行带宽

# For QoS, MARK 100

iptables -t mangle -F MARK_100 > /dev/null 2>&1

iptables -t mangle -N MARK_100 > /dev/null 2>&1

iptables -t mangle -A MARK_100 -p udp -m length ! --length 0:1350 -j MARK --set-mark 100

while iptables -t mangle -D PREROUTING -i ${privateNetDev} -j MARK_100 >/dev/null 2>&1; do continue; done

iptables -t mangle -I PREROUTING -i ${privateNetDev} -j MARK_100

tc qdisc del dev ${internetDev} root >/dev/null 2>&1

tc qdisc add dev ${internetDev} root handle 1: htb default 50

tc class add dev ${internetDev} parent 1: classid 1:1 htb rate ${uploadRate}

tc class add dev ${internetDev} parent 1:1 classid 1:100 htb rate ${uploadP2PRate} ceil ${uploadP2PRate} prio 7

tc class add dev ${internetDev} parent 1:1 classid 1:10 htb rate ${uploadEmerRate} prio 0

tc class add dev ${internetDev} parent 1:1 classid 1:20 htb rate ${uploadPrioRate} prio 2

tc class add dev ${internetDev} parent 1:1 classid 1:50 htb rate ${uploadNormalRate} prio 5

tc qdisc add dev ${internetDev} parent 1:100 handle 100: sfq perturb 10

tc qdisc add dev ${internetDev} parent 1:10 handle 10: sfq perturb 10

tc qdisc add dev ${internetDev} parent 1:20 handle 20: sfq perturb 10

tc qdisc add dev ${internetDev} parent 1:50 handle 50: sfq perturb 10

# Match TCP SYN

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 1 u32 \

match u8 0x10 0xff at 33 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match TCP ACK Small Packet (< 64 Bytes)

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 1 u32 \

match u8 0x10 0xff at 33 \

match u8 0x06 0xff at 9 \

match u16 0x0000 0xffc0 at 2 \

flowid 1:10

# Match TCP DESTINATION PORT 22

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0016 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match TCP DESTINATION PORT 23

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0017 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match TCP DESTINATION PORT 25

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0019 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP DESTINATION PORT 53

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0035 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match UDP DESTINATION PORT 53

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0035 0xffff at 22 \

match u8 0x11 0xff at 9 \

flowid 1:10

# Match TCP DESTINATION PORT 80

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0050 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP DESTINATION PORT 443

tc filter add dev ${internetDev} protocol ip parent 1: pref 10 u32 \

match u16 0x01bb 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match UDP DESTINATION PORT 443 (QUIC Protocol)

tc filter add dev ${internetDev} protocol ip parent 1: pref 10 u32 \

match u16 0x01bb 0xffff at 22 \

match u8 0x11 0xff at 9 \

flowid 1:20

# Match TCP DESTINATION PORT 465

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x01d1 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP DESTINATION PORT 993

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x03e1 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP DESTINATION PORT 3306

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0cea 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP DESTINATION PORT 3389

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0d3d 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match UDP DESTINATION PORT 3389

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0d3d 0xffff at 22 \

match u8 0x11 0xff at 9 \

flowid 1:10

# Match UDP DESTINATION PORT 8000 (QQ Port)

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x1f40 0xffff at 22 \

match u8 0x11 0xff at 9 \

flowid 1:10

# Match TCP DESTINATION PORT 8080 (WeChat Port)

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x1f90 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match TCP DESTINATION PORT 9418 (GIT)

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x24ca 0xffff at 22 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Limit UDP Huge Packet Rate

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 20 u32 \

match mark 0x0064 0x00ff \

flowid 1:100

# For QoS, MARK 200

iptables -t mangle -F PRIVATE_NET_MARK_200 >/dev/null 2>&1

iptables -t mangle -N PRIVATE_NET_MARK_200 >/dev/null 2>&1

iptables -t mangle -A PRIVATE_NET_MARK_200 -p tcp -j CONNMARK --restore-mark

iptables -t mangle -A PRIVATE_NET_MARK_200 -m mark --mark 200 -j RETURN

iptables -t mangle -A PRIVATE_NET_MARK_200 -j CONNMARK --save-mark

while iptables -t mangle -D POSTROUTING -o ${privateNetDev} -j PRIVATE_NET_MARK_200 >/dev/null 2>&1; do continue; done

iptables -t mangle -I POSTROUTING -o ${privateNetDev} -j PRIVATE_NET_MARK_200

tc qdisc del dev ${privateNetDev} root >/dev/null 2>&1

tc qdisc add dev ${privateNetDev} root handle 1: htb default 100

tc class add dev ${privateNetDev} parent 1: classid 1:1 htb rate ${downloadRate}

tc class add dev ${privateNetDev} parent 1:1 classid 1:100 htb rate ${downloadP2PRate} ceil ${downloadP2PRate} prio 7

tc class add dev ${privateNetDev} parent 1:1 classid 1:10 htb rate ${downloadEmerRate} prio 0

tc class add dev ${privateNetDev} parent 1:1 classid 1:20 htb rate ${downloadPrioRate} prio 2

tc qdisc add dev ${privateNetDev} parent 1:100 handle 100: sfq perturb 10

tc qdisc add dev ${privateNetDev} parent 1:10 handle 10: sfq perturb 10

tc qdisc add dev ${privateNetDev} parent 1:20 handle 20: sfq perturb 10

# Match TCP SYN ACK

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 1 u32 \

match u8 0x12 0xff at 33 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match TCP ACK Small Packet (< 64 Bytes)

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 1 u32 \

match u8 0x10 0xff at 33 \

match u8 0x06 0xff at 9 \

match u16 0x0000 0xffc0 at 2 \

flowid 1:10

# Match multip thread download

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 5 u32 \

match mark 0x00c8 0x00ff \

flowid 1:100

# Match TCP SOURCE PORT 22

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0016 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match TCP SOURCE PORT 23

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0017 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match TCP SOURCE PORT 25

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0019 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP SOURCE PORT 53

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0035 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match UDP SOURCE PORT 53

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0035 0xffff at 20 \

match u8 0x11 0xff at 9 \

flowid 1:10

# Match TCP SOURCE PORT 80

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0050 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP SOURCE PORT 443

tc filter add dev ${privateNetDev} protocol ip parent 1: pref 10 u32 \

match u16 0x01bb 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match UDP SOURCE PORT 443 (QUIC Protocol)

tc filter add dev ${privateNetDev} protocol ip parent 1: pref 10 u32 \

match u16 0x01bb 0xffff at 20 \

match u8 0x11 0xff at 9 \

flowid 1:20

# Match TCP SOURCE PORT 465

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x01d1 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP SOURCE PORT 993

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x03e1 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP SOURCE PORT 3306

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0cea 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:20

# Match TCP SOURCE PORT 3389

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0d3d 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match UDP SOURCE PORT 3389

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x0d3d 0xffff at 20 \

match u8 0x11 0xff at 9 \

flowid 1:10

# Match UDP SOURCE PORT 8000 (QQ Port)

tc filter add dev ${internetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x1f40 0xffff at 20 \

match u8 0x11 0xff at 9 \

flowid 1:10

# Match TCP SOURCE PORT 8080 (WeChat Port)

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x1f90 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:10

# Match TCP SOURCE PORT 9418 (GIT)

tc filter add dev ${privateNetDev} protocol ip parent 1:0 pref 10 u32 \

match u16 0x24ca 0xffff at 20 \

match u8 0x06 0xff at 9 \

flowid 1:20

总结

对P2P的归类效果还是非常好的：

1:1是总的下行，1:100是P2P的类别的下行。

P2P上行也很精确地限制在了1 Mbps：

要对迅雷的全网HTTP资源多线程下载完全正确归类可能做不到，但IP对IP流量大，高数据包速率的都能受限，正常的网页流量，交互式应用受到的影响不会太大。

下行速率的控制会不太准确，毕竟我们无法控制远程计算机给我们发包的速率，需要根据实际情况适当减少rate的值。

对于延迟，就比较难保证了，这依赖于ISP的队列规则，即使我们尽量做到不让ISP处丢包，但大量的数据包排队还是会导致其余应用的延迟增加。我这里把P2P的下行速率限制为80 Mbps，P2P全速下载时，延迟也只是增加了5 – 10 ms，可以接受，ISP处的队列比较合理。

使用策略路由实现涉及getsockopt(…, SO_ORIGINAL_DST, …)程序的远程调用（Redsocks, ss-redir之类的远程调用）

Redsocks/ss-redir的实现

DNAT与REDIRECT模块在iptables target extension之列。

这两个模块都能根据参数，修改数据包的header中的目的IP和端口。

既然修改，那就不是附加，毕竟网络层传输层header的内容怎么可能随随便便增删呢，这意味着，数据包的真实目的地会被完完全全地抹掉！

Redsocks/ss-redir收到这样的数据包后，不做特殊处理，何以知道这数据包原来想发给谁呢？

这些程序之所以能正常工作，得益于NAT的透明性，被NAT处理的数据包，在内核中均有所记录，NAT后的正常通讯由NAT发起者维持，Redsocks/ss-redir这类程序便是通过内核中的记录得知真实目的地的。

这个原地址的获取可以通过系统调用实现：

       #include <sys/types.h>
       #include <sys/socket.h>

       int getsockopt(int sockfd, int level, int optname,
                      void *optval, socklen_t *optlen);

#include <sys/types.h>

#include <sys/socket.h>

int getsockopt(int sockfd, int level, int optname,

void *optval, socklen_t *optlen);

optname需要传值SO_ORIGINAL_DST。

Redsocks的调用见base.c的223行：

/* redsocks - transparent TCP-to-proxy redirector
 * Copyright (C) 2007-2011 Leonid Evdokimov <leon@darkk.net.ru>
 *
 * Licensed under the Apache License, Version 2.0 (the "License"); you may not
 * use this file except in compliance with the License.  You may obtain a copy
 * of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
 * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the
 * License for the specific language governing permissions and limitations
 * under the License.
 */

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <arpa/inet.h>
#include <sys/stat.h>
#include <sys/resource.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>
#include <pwd.h>
#include <grp.h>
#include <stdlib.h>
#include "config.h"
#if defined USE_IPTABLES
# include <limits.h>
# include <linux/netfilter_ipv4.h>
#endif
#if defined USE_PF
# include <net/if.h>
# include <net/pfvar.h>
# include <sys/ioctl.h>
# include <errno.h>
#endif
#include "log.h"
#include "main.h"
#include "parser.h"
#include "redsocks.h"

typedef struct redirector_subsys_t {
	int (*init)();
	void (*fini)();
	int (*getdestaddr)(int fd, const struct sockaddr_in *client, const struct sockaddr_in *bindaddr, struct sockaddr_in *destaddr);
	const char *name;
	// some subsystems may store data here:
	int private;
} redirector_subsys;

typedef struct base_instance_t {
	int configured;
	char *chroot;
	char *user;
	char *group;
	char *redirector_name;
	redirector_subsys *redirector;
	char *log_name;
	bool log_debug;
	bool log_info;
	bool daemon;
#if defined(TCP_KEEPIDLE) && defined(TCP_KEEPCNT) && defined(TCP_KEEPINTVL)
	uint16_t tcp_keepalive_time;
	uint16_t tcp_keepalive_probes;
	uint16_t tcp_keepalive_intvl;
#endif
	uint32_t rlimit_nofile;
	uint32_t redsocks_conn_max;
	uint32_t connpres_idle_timeout;
	uint32_t max_accept_backoff_ms;
} base_instance;

static base_instance instance;

#if defined __FreeBSD__ || defined USE_PF
static int redir_open_private(const char *fname, int flags)
{
	int fd = open(fname, flags);
	if (fd < 0) {
		log_errno(LOG_ERR, "open(%s)", fname);
		return -1;
	}
	instance.redirector->private = fd;
	return 0;
}

static void redir_close_private()
{
	close(instance.redirector->private);
	instance.redirector->private = -1;
}
#endif

#ifdef __FreeBSD__
static int redir_init_ipf()
{
#ifdef IPNAT_NAME
	const char *fname = IPNAT_NAME;
#else
	const char *fname = IPL_NAME;
#endif
	return redir_open_private(fname, O_RDONLY);
}

static int getdestaddr_ipf(int fd, const struct sockaddr_in *client, const struct sockaddr_in *bindaddr, struct sockaddr_in *destaddr)
{
	int natfd = instance.redirector->private;
	struct natlookup natLookup;
	int x;
#if defined(IPFILTER_VERSION) && (IPFILTER_VERSION >= 4000027)
	struct ipfobj obj;
#else
	static int siocgnatl_cmd = SIOCGNATL & 0xff;
#endif

#if defined(IPFILTER_VERSION) && (IPFILTER_VERSION >= 4000027)
	obj.ipfo_rev = IPFILTER_VERSION;
	obj.ipfo_size = sizeof(natLookup);
	obj.ipfo_ptr = &natLookup;
	obj.ipfo_type = IPFOBJ_NATLOOKUP;
	obj.ipfo_offset = 0;
#endif

	natLookup.nl_inport = bindaddr->sin_port;
	natLookup.nl_outport = client->sin_port;
	natLookup.nl_inip = bindaddr->sin_addr;
	natLookup.nl_outip = client->sin_addr;
	natLookup.nl_flags = IPN_TCP;
#if defined(IPFILTER_VERSION) && (IPFILTER_VERSION >= 4000027)
	x = ioctl(natfd, SIOCGNATL, &obj);
#else
	/*
	 * IP-Filter changed the type for SIOCGNATL between
	 * 3.3 and 3.4.  It also changed the cmd value for
	 * SIOCGNATL, so at least we can detect it.  We could
	 * put something in configure and use ifdefs here, but
	 * this seems simpler.
	 */
	if (63 == siocgnatl_cmd) {
		struct natlookup *nlp = &natLookup;
		x = ioctl(natfd, SIOCGNATL, &nlp);
	} else {
		x = ioctl(natfd, SIOCGNATL, &natLookup);
	}
#endif
	if (x < 0) {
		if (errno != ESRCH)
			log_errno(LOG_WARNING, "ioctl(SIOCGNATL)\n");
		return -1;
	} else {
		destaddr->sin_family = AF_INET;
		destaddr->sin_port = natLookup.nl_realport;
		destaddr->sin_addr = natLookup.nl_realip;
		return 0;
	}
}
#endif

#ifdef USE_PF
static int redir_init_pf()
{
	return redir_open_private("/dev/pf", O_RDWR);
}

static int getdestaddr_pf(
		int fd, const struct sockaddr_in *client, const struct sockaddr_in *bindaddr,
		struct sockaddr_in *destaddr)
{
	int pffd = instance.redirector->private;
	struct pfioc_natlook nl;
	int saved_errno;
	char clientaddr_str[INET6_ADDRSTRLEN], bindaddr_str[INET6_ADDRSTRLEN];

	memset(&nl, 0, sizeof(struct pfioc_natlook));
	nl.saddr.v4.s_addr = client->sin_addr.s_addr;
	nl.sport = client->sin_port;
	nl.daddr.v4.s_addr = bindaddr->sin_addr.s_addr;
	nl.dport = bindaddr->sin_port;
	nl.af = AF_INET;
	nl.proto = IPPROTO_TCP;
	nl.direction = PF_OUT;

	if (ioctl(pffd, DIOCNATLOOK, &nl) != 0) {
		if (errno == ENOENT) {
			nl.direction = PF_IN; // required to redirect local packets
			if (ioctl(pffd, DIOCNATLOOK, &nl) != 0) {
				goto fail;
			}
		}
		else {
			goto fail;
		}
	}
	destaddr->sin_family = AF_INET;
	destaddr->sin_port = nl.rdport;
	destaddr->sin_addr = nl.rdaddr.v4;
	return 0;

fail:
	saved_errno = errno;
	if (!inet_ntop(client->sin_family, &client->sin_addr, clientaddr_str, sizeof(clientaddr_str)))
		strncpy(clientaddr_str, "???", sizeof(clientaddr_str));
	if (!inet_ntop(bindaddr->sin_family, &bindaddr->sin_addr, bindaddr_str, sizeof(bindaddr_str)))
		strncpy(bindaddr_str, "???", sizeof(bindaddr_str));

	errno = saved_errno;
	log_errno(LOG_WARNING, "ioctl(DIOCNATLOOK {src=%s:%d, dst=%s:%d})",
			  clientaddr_str, ntohs(nl.sport), bindaddr_str, ntohs(nl.dport));
	return -1;
}
#endif

#ifdef USE_IPTABLES
static int getdestaddr_iptables(int fd, const struct sockaddr_in *client, const struct sockaddr_in *bindaddr, struct sockaddr_in *destaddr)
{
	socklen_t socklen = sizeof(*destaddr);
	int error;

	error = getsockopt(fd, SOL_IP, SO_ORIGINAL_DST, destaddr, &socklen);
	if (error) {
		log_errno(LOG_WARNING, "getsockopt");
		return -1;
	}
	return 0;
}
#endif

static int getdestaddr_generic(int fd, const struct sockaddr_in *client, const struct sockaddr_in *bindaddr, struct sockaddr_in *destaddr)
{
	socklen_t socklen = sizeof(*destaddr);
	int error;

	error = getsockname(fd, (struct sockaddr*)destaddr, &socklen);
	if (error) {
		log_errno(LOG_WARNING, "getsockopt");
		return -1;
	}
	return 0;
}

int getdestaddr(int fd, const struct sockaddr_in *client, const struct sockaddr_in *bindaddr, struct sockaddr_in *destaddr)
{
	return instance.redirector->getdestaddr(fd, client, bindaddr, destaddr);
}

int apply_tcp_keepalive(int fd)
{
	struct { int level, option, value; } opt[] = {
		{ SOL_SOCKET, SO_KEEPALIVE, 1 },
		{ IPPROTO_TCP, TCP_KEEPIDLE, instance.tcp_keepalive_time },
		{ IPPROTO_TCP, TCP_KEEPCNT, instance.tcp_keepalive_probes },
		{ IPPROTO_TCP, TCP_KEEPINTVL, instance.tcp_keepalive_intvl },
	};
	for (int i = 0; i < SIZEOF_ARRAY(opt); ++i) {
		if (opt[i].value) {
			int error = setsockopt(fd, opt[i].level, opt[i].option, &opt[i].value, sizeof(opt[i].value));
			if (error) {
				log_errno(LOG_WARNING, "setsockopt(%d, %d, %d, &%d, %zu)", fd, opt[i].level, opt[i].option, opt[i].value, sizeof(opt[i].value));
				return -1;
			}
		}
	}
	return 0;
}

uint32_t max_accept_backoff_ms()
{
	return instance.max_accept_backoff_ms;
}

uint32_t redsocks_conn_max()
{
	return instance.redsocks_conn_max;
}

uint32_t connpres_idle_timeout()
{
	return instance.connpres_idle_timeout;
}

static redirector_subsys redirector_subsystems[] =
{
#ifdef __FreeBSD__
	{ .name = "ipf", .init = redir_init_ipf, .fini = redir_close_private, .getdestaddr = getdestaddr_ipf },
#endif
#ifdef USE_PF
	{ .name = "pf",  .init = redir_init_pf,  .fini = redir_close_private, .getdestaddr = getdestaddr_pf },
#endif
#ifdef USE_IPTABLES
	{ .name = "iptables", .getdestaddr = getdestaddr_iptables },
#endif
	{ .name = "generic",  .getdestaddr = getdestaddr_generic  },
};

/***********************************************************************
 * `base` config parsing
 */
static parser_entry base_entries[] =
{
	{ .key = "chroot",     .type = pt_pchar,   .addr = &instance.chroot },
	{ .key = "user",       .type = pt_pchar,   .addr = &instance.user },
	{ .key = "group",      .type = pt_pchar,   .addr = &instance.group },
	{ .key = "redirector", .type = pt_pchar,   .addr = &instance.redirector_name },
	{ .key = "log",        .type = pt_pchar,   .addr = &instance.log_name },
	{ .key = "log_debug",  .type = pt_bool,    .addr = &instance.log_debug },
	{ .key = "log_info",   .type = pt_bool,    .addr = &instance.log_info },
	{ .key = "daemon",     .type = pt_bool,    .addr = &instance.daemon },
#if defined(TCP_KEEPIDLE) && defined(TCP_KEEPCNT) && defined(TCP_KEEPINTVL)
	{ .key = "tcp_keepalive_time",   .type = pt_uint16, .addr = &instance.tcp_keepalive_time },
	{ .key = "tcp_keepalive_probes", .type = pt_uint16, .addr = &instance.tcp_keepalive_probes },
	{ .key = "tcp_keepalive_intvl",  .type = pt_uint16, .addr = &instance.tcp_keepalive_intvl },
#endif
	{ .key = "rlimit_nofile", .type = pt_uint32, .addr = &instance.rlimit_nofile },
	{ .key = "redsocks_conn_max", .type = pt_uint32, .addr = &instance.redsocks_conn_max },
	{ .key = "connpres_idle_timeout", .type = pt_uint32, .addr = &instance.connpres_idle_timeout },
	{ .key = "max_accept_backoff", .type = pt_uint32, .addr = &instance.max_accept_backoff_ms },
	{ }
};

static int base_onenter(parser_section *section)
{
	if (instance.configured) {
		parser_error(section->context, "only one instance of base is valid");
		return -1;
	}
	memset(&instance, 0, sizeof(instance));
	instance.configured = 1;
	instance.max_accept_backoff_ms = 60000;
	instance.connpres_idle_timeout = 7440;
	return 0;
}

static int base_onexit(parser_section *section)
{
	if (!instance.max_accept_backoff_ms) {
		parser_error(section->context, "`max_accept_backoff` must be positive, 0 ms is too low");
		return -1;
	}

	if (instance.redirector_name) {
		redirector_subsys *ss;
		FOREACH(ss, redirector_subsystems) {
			if (!strcmp(ss->name, instance.redirector_name)) {
				instance.redirector = ss;
				instance.redirector->private = -1;
				break;
			}
		}
		if (!instance.redirector) {
			parser_error(section->context, "invalid `redirector` set <%s>", instance.redirector_name);
			return -1;
		}
	}
	else {
		parser_error(section->context, "no `redirector` set");
		return -1;
	}

	return 0;
}

static parser_section base_conf_section =
{
	.name    = "base",
	.entries = base_entries,
	.onenter = base_onenter,
	.onexit  = base_onexit
};

/***********************************************************************
 * `base` initialization
 */
static int base_fini();

static int base_init()
{
	uid_t uid = -1;
	gid_t gid = -1;
	int devnull = -1;

	if (!instance.configured) {
		log_error(LOG_ERR, "there is no configured instance of `base`, check config file");
		return -1;
	}

	if (instance.redirector->init && instance.redirector->init() < 0)
		return -1;

	if (instance.user) {
		struct passwd *pw = getpwnam(instance.user);
		if (pw == NULL) {
			log_errno(LOG_ERR, "getpwnam(%s)", instance.user);
			goto fail;
		}
		uid = pw->pw_uid;
	}

	if (instance.group) {
		struct group *gr = getgrnam(instance.group);
		if (gr == NULL) {
			log_errno(LOG_ERR, "getgrnam(%s)", instance.group);
			goto fail;
		}
		gid = gr->gr_gid;
	}

	if (log_preopen(
			instance.log_name ? instance.log_name : instance.daemon ? "syslog:daemon" : "stderr",
			instance.log_debug,
			instance.log_info
	) < 0 ) {
		goto fail;
	}

	if (instance.rlimit_nofile) {
		struct rlimit rlmt;
		rlmt.rlim_cur = instance.rlimit_nofile;
		rlmt.rlim_max = instance.rlimit_nofile;
		if (setrlimit(RLIMIT_NOFILE, &rlmt) != 0) {
			log_errno(LOG_ERR, "setrlimit(RLIMIT_NOFILE, %u)", instance.rlimit_nofile);
			goto fail;
		}
	} else {
		struct rlimit rlmt;
		if (getrlimit(RLIMIT_NOFILE, &rlmt) != 0) {
			log_errno(LOG_ERR, "getrlimit(RLIMIT_NOFILE)");
			goto fail;
		}
		instance.rlimit_nofile = rlmt.rlim_cur;
	}

	if (!instance.redsocks_conn_max) {
		instance.redsocks_conn_max = (instance.rlimit_nofile - instance.rlimit_nofile / 4)
			/ (redsocks_has_splice_instance() ? 6 : 2);
	}

	if (instance.daemon) {
		devnull = open("/dev/null", O_RDWR);
		if (devnull == -1) {
			log_errno(LOG_ERR, "open(\"/dev/null\", O_RDWR");
			goto fail;
		}
	}

	if (instance.chroot) {
		if (chroot(instance.chroot) < 0) {
			log_errno(LOG_ERR, "chroot(%s)", instance.chroot);
			goto fail;
		}
	}

	if (instance.daemon || instance.chroot) {
		if (chdir("/") < 0) {
			log_errno(LOG_ERR, "chdir(\"/\")");
			goto fail;
		}
	}

	if (instance.group) {
		if (setgid(gid) < 0) {
			log_errno(LOG_ERR, "setgid(%i)", gid);
			goto fail;
		}
	}

	if (instance.user) {
		if (setuid(uid) < 0) {
			log_errno(LOG_ERR, "setuid(%i)", uid);
			goto fail;
		}
	}

	if (instance.daemon) {
		switch (fork()) {
		case -1: // error
			log_errno(LOG_ERR, "fork()");
			goto fail;
		case 0:  // child
			break;
		default: // parent, pid is returned
			exit(EXIT_SUCCESS);
		}
	}

	log_open(); // child has nothing to do with TTY

	if (instance.daemon) {
		if (setsid() < 0) {
			log_errno(LOG_ERR, "setsid()");
			goto fail;
		}

		int fds[] = { STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO };
		int *pfd;
		FOREACH(pfd, fds)
			if (dup2(devnull, *pfd) < 0) {
				log_errno(LOG_ERR, "dup2(devnull, %i)", *pfd);
				goto fail;
			}

		close(devnull);
	}
	return 0;
fail:
	if (devnull != -1)
		close(devnull);

	base_fini();

	return -1;
}

static int base_fini()
{
	if (instance.redirector->fini)
		instance.redirector->fini();

	free(instance.chroot);
	free(instance.user);
	free(instance.group);
	free(instance.redirector_name);
	free(instance.log_name);

	memset(&instance, 0, sizeof(instance));

	return 0;
}

app_subsys base_subsys =
{
	.init = base_init,
	.fini = base_fini,
	.conf_section = &base_conf_section,
};

/* vim:set tabstop=4 softtabstop=4 shiftwidth=4: */
/* vim:set foldmethod=marker foldlevel=32 foldmarker={,}: */

217

218

219

220

221

222

223

224

225

226

227

228

229

230

#ifdef USE_IPTABLES

static int getdestaddr_iptables(int fd, const struct sockaddr_in *client, const struct sockaddr_in *bindaddr, struct sockaddr_in *destaddr)

{

socklen_t socklen = sizeof(*destaddr);

int error;

error = getsockopt(fd, SOL_IP, SO_ORIGINAL_DST, destaddr, &socklen);

if (error) {

log_errno(LOG_WARNING, "getsockopt");

return -1;

}

return 0;

}

#endif

问题所在

假定有以下的网络架构：

正常情况下，NAT与getsockopt(…, SO_ORIGINAL_DST, …)调用程序都运行在Router上，工作正常。

现在把getsockopt(…, SO_ORIGINAL_DST, …)调用程序迁移到Server 1上，很容易想到的，就是把原来的NAT规则改成DNAT的，并指定目的地IP与端口：

iptables -t nat ... -p tcp ... -j DNAT --to 192.168.1.2:GETSOCKOPT_PROGRAM_LISTEN_PORT

1	iptables -t nat ... -p tcp ... -j DNAT --to 192.168.1.2:GETSOCKOPT_PROGRAM_LISTEN_PORT

NAT的记录由内核管理，这隐含的意思是，只有在发起NAT处理的内核上才能获取到NAT前的信息。

若调用远程的getsockopt(…, SO_ORIGINAL_DST, …)调用程序，即发起NAT的与运行getsockopt(…, SO_ORIGINAL_DST, …)调用程序的不是同一个内核，很明显，这些程序对getsockopt()的调用不可能实现他们的目的。

问题已很明显，这类程序无法通过getsockopt()调用取得真实目的地。

实现getsockopt()的远程调用成本有点高，不太现实。

不过，我们把方式稍作改变，让“远程”变为“本地”。

实现

假定有以下的网络架构：

Server 1上运行着getsockopt(…, SO_ORIGINAL_DST, …)调用程序。

既然处理NAT要与调用getsockopt()的使用同一个内核，那么把需要NAT的那部分也交给Server 1处理好了。

要实现这种需求，可以利用策略路由，正常的数据包按正常处理，原本需要NAT到Server 1的数据包，使用路由规则转发到Server 1。这样就不需要网内其余联网设备作任何改变，只需要对Router与Server 1进行操作。

准备

Router 1上首先把针对getsockopt(…, SO_ORIGINAL_DST, …)调用程序的NAT规则删掉。

路由表

修改/etc/iproute2/rt_tables增加一个路由表，如12行，增加一个ID为250，名为server1的路由表：

#
# reserved values
#
255	local
254	main
253	default
0	unspec
#
# local
#
#1	inr.ruhep
250	server1

12	250 server1

往server1路由表中增加一条默认路由规则，把所有数据包都转发到Server 1：

ip route add default via 192.168.1.2 table server1

1	ip route add default via 192.168.1.2 table server1

数据包分类

接下来需要把数据包分类，即决定哪些数据包应使用路由表server1。

我这里选择iptables的MARK模块，把所有需要给getsockopt(…, SO_ORIGINAL_DST, …)调用程序处理的数据包，都加一个固定标记，然后让路由让有该特定标记的数据包都根据路由表server1处理。

iptables MARK模块的–set-mark操作仅能在mangle表完成。

假设MARK为150（把…改成你需要的参数）：

iptables -t mangle ...  -j MARK --set-mark 150

1	iptables -t mangle ... -j MARK --set-mark 150

这里需要注意，Server 1访问国际互联网，也要通过Router，如果getsockopt(…, SO_ORIGINAL_DST, …)调用程序访问互联网会匹配到上面的规则，必须排除，否则就有环路了，最简单的，就是直接排除掉Server 1的MAC地址：

iptables -t mangle ... -m mac --mac-source SERVER1_MAC -j RETURN

1	iptables -t mangle ... -m mac --mac-source SERVER1_MAC -j RETURN

这条排除Server 1的规则需要插入在MARK之前。

最后修改路由策略，让带150 MARK的数据包都使用路由表server1：

ip rule add fwmark 150 table server1

1	ip rule add fwmark 150 table server1

Server 1的处理

数据包已在Router处分类，并不经网络层和传输层Header的修改转发到Server 1处。

Server 1要区分这些数据包是要交给getsockopt(…, SO_ORIGINAL_DST, …)调用程序，还是交给自己的程序处理，很简单，看看数据包目的IP是不是自己就行了，要交给getsockopt(…, SO_ORIGINAL_DST, …)调用程序的数据包，目的IP肯定不是Server 1的IP：

iptables -t nat ... ! -d 192.168.1.2 -p tcp -j REDIRECT --to GETSOCKOPT_PROGRAM_LISTEN_PORT

1	iptables -t nat ... ! -d 192.168.1.2 -p tcp -j REDIRECT --to GETSOCKOPT_PROGRAM_LISTEN_PORT

总结

这个方法仅适用于Router与Server 1处于同一局域网的情况，因为路由转发仅能在链路层上进行，若需要跨网域调用，可能要考虑使用隧道封装，但这样开销有点大。

Router一般都在芯片上实现了NAT，NAT效率高，Server 1虽然没芯片级的NAT，但我相信，你处理被NAT的数据包的程序，开销比NAT大得多。

解决安装OpenVZ vzctl 4.7后iptables nat表提示”can’t initialize iptables table `nat’: Table does not exist (do you need to insm Perhaps iptables or your kernel needs to be upgraded.”

几天前给一个服务器安装OpenVZ后，使用iptables添加nat规则时，出现错误提示：

iptables v1.4.7: can't initialize iptables table `nat': Table does not exist (do
Perhaps iptables or your kernel needs to be upgraded.

1 2	iptables v1.4.7: can't initialize iptables table `nat': Table does not exist (do Perhaps iptables or your kernel needs to be upgraded.

执行lsmod看了下：

[root@localhost ~]# lsmod | grep nat
nf_nat_ftp              3523  0 
nf_conntrack_ftp       12929  1 nf_nat_ftp
iptable_nat             6302  0 
nf_nat                 23213  4 vzrst,ipt_REDIRECT,nf_nat_ftp,iptable_nat
nf_conntrack_ipv4       9946  4 iptable_nat,nf_nat
nf_conntrack           80281  8 vzrst,vzcpt,nf_nat_ftp,nf_conntrack_ftp,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
ip_tables              18119  3 iptable_mangle,iptable_filter,iptable_nat

[root@localhost ~]# lsmod | grep nat

nf_nat_ftp 3523 0

nf_conntrack_ftp 12929 1 nf_nat_ftp

iptable_nat 6302 0

nf_nat 23213 4 vzrst,ipt_REDIRECT,nf_nat_ftp,iptable_nat

nf_conntrack_ipv4 9946 4 iptable_nat,nf_nat

nf_conntrack 80281 8 vzrst,vzcpt,nf_nat_ftp,nf_conntrack_ftp,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4

ip_tables 18119 3 iptable_mangle,iptable_filter,iptable_nat

发现相关的NAT模块都已经加载，应该与内核无关。

更换内核回2.6.32-431.11.2.el6，问题还是如此，无奈之下重装系统。

安装系统后，我手动一个个下载OpenVZ的RPM包，一个个安装，然后重启后nat表是否可以使用，最后发现安装vzctl 4.7并且重启后，iptables的nat表就出问题了。

看了下官网，原来在OpenVZ在2014年04月15号更新了，难怪其他服务器没遇到这问题。

到OpenVZd Bugzilla瞧了瞧，是nf_conntrack模块的问题，官方的技术人员已经给出了解决办法了：

修改/etc/modprobe.d/openvz.conf，把

options nf_conntrack ip_conntrack_disable_ve0=1

1	options nf_conntrack ip_conntrack_disable_ve0=1

修改为：

options nf_conntrack ip_conntrack_disable_ve0=0

1	options nf_conntrack ip_conntrack_disable_ve0=0

尝试用rmmod卸载nf_conntrack，再用modporbe重新加载nf_conntrack，但是执行rmmod nf_conntrack有提示某些程序在使用nf_conntrack模块，所以只能reboot了。

系统启动后，已可以正常加载出nat表：

[root@localhost ~]# iptables -L -n -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

[root@localhost ~]# iptables -L -n -t nat

Chain PREROUTING (policy ACCEPT)

target prot opt source destination

Chain POSTROUTING (policy ACCEPT)

target prot opt source destination

Chain OUTPUT (policy ACCEPT)

target prot opt source destination

OpenVZ此次更新的改动貌似有点大，4.6.1和之前的版本都是用simfs的，4.7开始已默认使用ploop。