-
Notifications
You must be signed in to change notification settings - Fork 0
/
libautocork.txt
138 lines (108 loc) · 5.72 KB
/
libautocork.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
Build Instructions:
[acme@doppio ~]$ git-clone git://git.kernel.org/pub/scm/linux/kernel/git/acme/libautocork
[acme@doppio ~]$ cd libautocork/
[acme@doppio libautocork]$ make
gcc -O2 -g -Wall -fPIC -c -o libautocork.o libautocork.c
gcc -O2 -g -Wall -fPIC -shared -o libautocork.so libautocork.o -ldl
gcc -O2 -g -Wall -fPIC tcp_nodelay_client.c -o tcp_nodelay_client -lrt
gcc -O2 -g -Wall -fPIC tcp_nodelay_server.c -o tcp_nodelay_server
[acme@doppio libautocork]$
To use it:
LD_PRELOAD=./libautocork.o your_application
Using it with the test application in this tarball:
1. start the server:
[acme@doppio libautocork]$ ./tcp_nodelay_server 5001 10000
server: waiting for connection
2. start the client without using libautocork and using nodelay:
[acme@doppio libautocork]$ ./tcp_nodelay_client --verbose --no_delay localhost
10000 packets (15 buffers) sent in 1776.608398 ms: 168.861069 bytes/ms using TCP_NODELAY
3. Now lets try using libautocork with the same parameters used, i.e.
reproducing an application that uses TCP_NODELAY and will have it
turned into TCP_CORK by libautocork:
[acme@doppio libautocork]$ LD_PRELOAD=./libautocork.so ./tcp_nodelay_client --verbose --no_delay localhost
10000 packets (15 buffers) sent in 481.941132 ms: 622.482666 bytes/ms using TCP_NODELAY
It is important to use './' or the full path, as the linux library
loader will try to find it on the LD_LIBRARY_PATH list of paths or in
what is in /etc/ld.so.conf, and as the current directory ('.') normally
isn't you could get something confusing as:
[acme@doppio libautocork]$ LD_PRELOAD=libautocork.so ./tcp_nodelay_client --verbose --no_delay localhost
ERROR: ld.so: object 'libautocork.so' from LD_PRELOAD cannot be preloaded: ignored.
10000 packets (15 buffers) sent in 1780.601074 ms: 168.482437 bytes/ms using TCP_NODELAY
[acme@doppio libautocork]$
So using libautocork on a unmodified binary that uses TCP_NODELAY and
sends 15 buffers and then wait for one buffer we get througput from
168.861069 bytes/ms up to 622.482666 bytes/ms for 10000 packets.
The test case can be used to do lots of other tests with TCP_CORK,
TCP_NODELAY, sending packets built entirely in userspace (1 buffer), a
header + a payload (2 buffers), etc:
[acme@doppio libautocork]$ ./tcp_nodelay_client --help
Usage: tcp_nodelay_client [OPTION...] [SERVER]
-c, --cork use TCP_CORK
-H, --header_plus_payload send logical packets header + payload
-n, --no_delay use TCP_NODELAY
-N, --nr_logical_packets=NR send NR logical packets [DEFAULT=10000]
-p, --port=PORT connect to PORT [DEFAULT=5001]
-s, --single_request send logical packets as a single request
-v, --verbose be verbose
-?, --help Give this help list
--usage Give a short usage message
Mandatory or optional arguments to long options are also mandatory or
optional for any corresponding short options.
[acme@doppio libautocork]$
The server also helps understanding what really happens in terms of
number of times the read syscall returns with partial buffers, for
instance, for the above sessions:
[acme@doppio libautocork]$ ./tcp_nodelay_server 5001 10000
server: waiting for connection
server: accept from localhost.localdomain
server: received 119281 buffers
server: waiting for connection
server: accept from localhost.localdomain
server: received 10000 buffers
server: waiting for connection
server: accept from localhost.localdomain
server: received 121347 buffers
server: waiting for connection
The second one was the one with libautocork, so instead of receiving 120
thousand buffers (the number of real network packets is about the same)
it received 10000 buffers (that is exactly the number of networking
packets sans connection establishment and teardown) , the number of
logical packets sent.
To make sure that libautocork is being used you can take a look at the
process shared memory library map:
[acme@doppio libautocork]$ /sbin/pidof tcp_nodelay_server
19091
[acme@doppio libautocork]$ grep libautocork.so /proc/19091/smaps
2aaaaaaad000-2aaaaaaaf000 r-xp 00000000 fd:00 197010 /home/acme/libautocork/libautocork.so
2aaaaaaaf000-2aaaaacaf000 ---p 00002000 fd:00 197010 /home/acme/libautocork/libautocork.so
2aaaaacaf000-2aaaaacb0000 rw-p 00002000 fd:00 197010 /home/acme/libautocork/libautocork.so
[acme@doppio libautocork]$
You can also use an environment variable:
[acme@doppio libautocork]$ export AUTOCORK_DEBUG=1
[acme@doppio libautocork]$ LD_PRELOAD=./libautocork.so ./tcp_nodelay_client --verbose --no_delay localhost
libautocork: turning TCP_CORK ON fd 3
10000 packets (15 buffers) sent in 491.302338 ms: 610.621948 bytes/ms using TCP_NODELAY
[acme@doppio libautocork]$
Use a value of 2 to get more verbose output, such as when libautocork
pushes pending frames, aka autocorks:
[acme@doppio libautocork]$ export AUTOCORK_DEBUG=2
[acme@doppio libautocork]$ LD_PRELOAD=./libautocork.so
./tcp_nodelay_client --verbose --no_delay localhost 2>&1 | head -10
libautocork: turning TCP_CORK ON fd 3
libautocork: autocorking fd 3 on read
libautocork: autocorking fd 3 on read
libautocork: autocorking fd 3 on read
libautocork: autocorking fd 3 on read
libautocork: autocorking fd 3 on read
libautocork: autocorking fd 3 on read
libautocork: autocorking fd 3 on read
libautocork: autocorking fd 3 on read
libautocork: autocorking fd 3 on read
[acme@doppio libautocork]$
The autocorking is made on one of the following standard C library calls:
read, readv, recv, recvmsg, recvfrom, select, pselect, poll and ppoll
But only when data was sent on an file descriptor where
setsockopt(TCP_NODELAY) was done.
TODO: work on the cheapest possible timer to use to avoid having the socket
autocorked when the application doesn't use one of the library calls that
pushes the data.