-
Notifications
You must be signed in to change notification settings - Fork 103
Handling clients
Tempesta FW prints access log to the kernel log or directly to the Clickhouse database. The access log is switched off by default, but you can switch it on with the global configuration option
It is recommended to use mmap
logging mode for better performance and analytic
abilities.
access_log dmesg;
or
access_log mmap mmap_host=localhost mmap_log=/var/log/tempesta_access.log;
You can fetch the access log records with:
# dmesg |grep 'tempesta fw'
[367562.584711] [tempesta fw] 192.168.100.1 "tempesta-tech.com" "GET / HTTP/1.1" 200 25207 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0" "ja5t=66cbda9cafc40009" "ja5h=b1c0008c0280"
[367562.731121] [tempesta fw] 192.168.100.1 "tempesta-tech.com" "GET /img/forkme.png HTTP/1.1" 200 6040 "https://tempesta-tech.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0" "ja5t=66cbda9cafc40009" "ja5h=b1c0008c0280"
[367562.786937] [tempesta fw] 192.168.100.1 "tempesta-tech.com" "GET /img/features.png HTTP/1.1" 200 3547 "https://tempesta-tech.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0" "ja5t=66cbda9cafc40009" "ja5h=b1c0008c0280"
[367562.787396] [tempesta fw] 192.168.100.1 "tempesta-tech.com" "GET /img/accel.png HTTP/1.1" 200 2505 "https://tempesta-tech.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0" "ja5t=66cbda9cafc40009" "ja5h=b1c0008c0280"
[367562.790298] [tempesta fw] 192.168.100.1 "tempesta-tech.com" "GET /css/main.min.css HTTP/1.1" 200 8531 "https://tempesta-tech.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0" "ja5t=66cbda9cafc40009" "ja5h=b1c0008c0280"
[367562.798147] [tempesta fw] 192.168.100.1 "tempesta-tech.com" "GET /img/security.png HTTP/1.1" 200 2023 "https://tempesta-tech.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0" "ja5t=66cbda9cafc40009" "ja5h=b1c0008c0280"
[367562.798543] [tempesta fw] 192.168.100.1 "tempesta-tech.com" "GET /img/header_s1.png HTTP/1.1" 200 31288 "https://tempesta-tech.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0" "ja5t=66cbda9cafc40009" "ja5h=b1c0008c0280"
[367562.817580] [tempesta fw] 192.168.100.1 "tempesta-tech.com" "GET /js/prism.min.js HTTP/1.1" 200 21634 "https://tempesta-tech.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0" "ja5t=66cbda9cafc40009" "ja5h=b1c0008c0280"
[367562.823057] [tempesta fw] 192.168.100.1 "tempesta-tech.com" "GET /img/lb.png HTTP/1.1" 200 1776 "https://tempesta-tech.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0" "ja5t=66cbda9cafc40009" "ja5h=b1c0008c0280"
[367562.828901] [tempesta fw] 192.168.100.1 "tempesta-tech.com" "GET /js/main.min.js HTTP/1.1" 200 4354 "https://tempesta-tech.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0" "ja5t=66cbda9cafc40009" "ja5h=b1c0008c0280"
[367562.832737] [tempesta fw] 192.168.100.1 "tempesta-tech.com" "GET /css/prism.min.css HTTP/1.1" 200 2437 "https://tempesta-tech.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0" "ja5t=66cbda9cafc40009" "ja5h=b1c0008c0280"
[367563.056342] [tempesta fw] 192.168.100.1 "tempesta-tech.com" "GET /img/github.svg HTTP/1.1" 200 2865 "https://tempesta-tech.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0" "ja5t=66cbda9cafc40009" "ja5h=b1c0008c0280"
[367563.058078] [tempesta fw] 192.168.100.1 "tempesta-tech.com" "GET /img/support.png HTTP/1.1" 200 4462 "https://tempesta-tech.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0" "ja5t=66cbda9cafc40009" "ja5h=b1c0008c0280"
[367563.061045] [tempesta fw] 192.168.100.1 "tempesta-tech.com" "GET /img/db.png HTTP/1.1" 200 3962 "https://tempesta-tech.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0" "ja5t=66cbda9cafc40009" "ja5h=b1c0008c0280"
The log format is:
- local time
- "tempesta fw" banner to fetch the records
- remote client IP
- vhost name
- request line
- response status
- response bytes sent
- Referer request header value
- User-Agent request header value
- ja5t fingerprint
- ja5h fingerprint
Some of the headers values can be arbitrarily long if you don't use
special limits,
so Tempesta FW may truncate the strings for the access log. If the string is
truncated, then it has ...
suffix.
In this mode Tempesta FW writes the access log events to per-CPU buffers, mapped to the
user space, where they are read and sent to the Clickhouse database by the tfw_logger
daemon (also running per-CPU logs transmission threads).
Thanks to the per-CPU zero-copy design, this logging mode provides much better performance
and scaling to large number of CPUs in comparison with the dmesg
mode. Also Clickhouse
provides advanced analytic queries over the stored data, facilitating web performance
analytics and security incidents (e.g. application DDoS attacks) response.
Enable mmap log in the configuration file and specify Clickhouse server host and log path:
access_log mmap mmap_host=localhost mmap_log=/var/log/tfw_logger.log
Run Clickhouse server and start Tempesta FW. acces_log
table will be automatically
created and started to fill with access log data.
The following fields are currently included in the logs sent to ClickHouse:
-
timestamp
- the date and time of the request -
address
- the client's IPv6 address -
method
- the HTTP method used in the request -
version
- the HTTP protocol version -
status
- the HTTP response status code -
response_content_length
- the size of the response content in bytes -
response_time
- the time taken to process the request (in milliseconds) -
vhost
- the name of the virtual host serving the request -
uri
- the requested URI -
referer
- the referrer URL -
user_agent
- the client's user agent string -
dropped_events
- amount of dropped events
Field method
is a numerical value:
0: COPY
1: DELETE
2: GET
3: HEAD
4: LOCK
5: MKCOL
6: MOVE
7: OPTIONS
8: PATCH
9: POST
10: PROPFIND
11: PROPPATCH
12: PUT
13: TRACE
14: UNLOCK
15: PURGE
16: UNKNOWN
17: INCOMPLETE
Field version
is also a numerical value:
0: INVALID
1: HTTP 0.9
2: HTTP 1.0
3: HTTP 1.1
4: HTTP 2
Tempesta listens to incoming connections on specified address and port. The syntax is as follows:
listen <PORT> | <IPADDR>[:PORT] [proto=http|https|h2|h2,https];
IPADDR
may be either IPv4 or IPv6 address. Host names are not allowed.
IPv6 address must be enclosed in square brackets (e.g. "[::0]" but not "::0").
If only PORT
is specified, then address 0.0.0.0 (but not [::1]) is used.
If only IPADDR
is specified, then default HTTP port 80 is used.
Tempesta opens one socket for each listen
directive. Multiple listen
directives may be defined to listen on multiple addresses/ports.
If listen
directive is not defined in the configuration file,
then by default Tempesta listens on IPv4 address 0.0.0.0 and port 80,
which is an equivalent to listen 80
directive.
Below are examples of listen
directive:
listen 80;
listen 443 proto=h2;
listen [::0]:80;
listen 127.0.0.1:8001;
listen [::1]:8001;
It is allowed to specify the type of listening socket via the proto
attribute.
At the moment following values for proto
attribute are supported: http
(means
HTTP/1.1 over TCP), https
(means HTTP/1.1 over TLS), h2
(means HTTP/2
over TLS) and h2,https
(means that both HTTP/1.1 over TLS and HTTP/2 over TLS are
possible on the port at the same time).
If proto
option is absent, then proto=http
is supposed by default.
Tempesta may use a single TCP connection to send and receive multiple HTTP requests/responses. The syntax is as follows:
keepalive_timeout <TIMEOUT>;
TIMEOUT
is a timeout in seconds during which a keep-alive client connection
will stay open in Tempesta. The zero value disables keep-alive client
connections. Default value is 75.
Below are examples of keepalive_timeout
directive:
keepalive_timeout 75;
Tempesta FW supports proxying HTTP/1 websocket clients. For it there is separate option:
client_ws_timeout <TIMEOUT>;
TIMEOUT
is a timeout in seconds between consequtive messages on websocket level. If none messages arrives in it
then Tempesta FW closes websocket connection. Default value is 3600 seconds. This option overrides keepalive_timeout
.
Tempesta FW supports Websockets over HTTP/1.1 (ws://
) and HTTPS (wss://
).
We do not support Websockets over HTTP/2 as specified in RFC 8441
because:
- for HTTP/2 Server-Sent Events are mostly used;
- the major client libraries (e.g. SockJS and Socket.io) do not support RFC 8441
- Node.js WS also doesn't implement websockets over HTTP/2, but Node.js do implement necessary core logic to support RFC 8441
- Besides official supporting of RFC 8441 for Chrome
and Firefox it's quite hard in practice to make
browsers to do HTTP/2 websocket request from HTTP/2 loaded page and regardless announcing
SETTINGS_MAX_CONCURRENT_STREAMS
option
However, we have implemented Websockets over HTTP/2 and the implementation draft is available in the in pull request - if you do need Websockets over HTTP/2, then feel free to reopen the pull request to create new issue for the feature.
Basically, you don't need any specific configuration to make Websockets work through Tempesta FW. You can find several use cases in the functional test.
WebSockets use separate TCP backend connection (i.e. a connection isn't used to send websocket frames from other client and/or HTTP messages from the same or other clients). The same backend connection, used for previous client requests, is upgraded to websocket, meaning that a new TCP connection is established (provisioned) with the backend to satisfy upcoming HTTP client requests. You can find more technical detail in the implementation issue.
A configuration example to process Websockets over TLS on a designated server:
listen 192.168.100.4:443 proto=https;
block_action attack reply;
block_action error reply;
srv_group default {
server 192.168.100.4:8000;
}
srv_group websocket {
# Dummy connection: new connections will be established.
server 127.0.0.1:8080 conns_n=1;
}
vhost default {
resp_hdr_set Content-Security-Policy "upgrade-insecure-requests";
resp_hdr_set Strict-Transport-Security "max-age=31536000; includeSubDomains";
proxy_pass default;
}
vhost websocket {
proxy_pass websocket;
}
tls_match_any_server_name;
tls_certificate /root/tempesta/etc/tfw-root.crt;
tls_certificate_key /root/tempesta/etc/tfw-root.key;
cache 1;
cache_fulfill * *;
access_log on;
http_chain {
uri == "/ws" -> websocket;
-> default;
}
Tempesta FW allows to configure the error
response behavior: send HTTP error message to a client or just silently block the message
(requests and responses).
This can be configured for malformed and malicious message differently - via
directive block_action
:
block_action <MSG_TYPE> <ACTION> [OPTIONS];
MSG_TYPE
: type of incoming message (e.g. malicious).
ACTION
: operation which Tempesta FW must perform with specified message type.
OPTIONS
: currently only nolog option is supported.
The following MSG_TYPE
keywords are supported:
- error - Means that action must be applied only to malformed messages.
- attack - Action must be applied only to malicious (attack) messages.
The following ACTION
keywords are supported:
-
drop - Tempesta FW must block message silently (response won't be generated)
and reset (with TCP
RST
) the client connection. This action is typically used when administrator want to save resources in case of attack or a lot of errors which can be treated as attack. So Tempesta FW immediately close the connection with TCP RST both for HTTP1 and for HTTP2 connections. - reply - Response with appropriate error status will be sent to client. If attack is detected we immediately close socket after sending TCP FIN. This may leads to response loosing if especially it contains a body because of two problems. (Kernel sends TCP RST if data is received on the DEAD sock, so if the client send any data for us after socket closing, kernel sends TCP RST to such client. Another problem is that we need to process WINDOW_UPDATE frames to send error responses with a body, but we can't do it on the DEAD sock). If error is detected we send TCP FIN to the client after sending error response, but leave socket alive. Connection will be closed after TCP FIN from the client. In this case we continue to process WINDOW_UPDATE frames, while connection (HTTP2) is alive, but drop all other incoming requests.
The nolog option stated that information about error/attack situation must not be logged (by default logging is enabled).
Default setting are as follows:
block_action error reply;
block_action attack drop;
NOTE While it makes sense to drop an attack as early as possible and with
as little spent resources as possible, especially for DDoS, many browsers are
retrying failed or timed out requests. In practice, drop
action, even for
attack message types, may make a regular browser to retry an erroneously blocked
request, while leads to rate limiting on Tempesta FW side with following banning
of the client. Thus, we recommend to use reply
action in regular installations.
Administrator can specify files with page bodies for error responses generated by Tempesta FW. The syntax in Tempesta configuration file is as follows:
response_body <status_code> <file>;
-
status_code HTTP status code for which page body must be applied.
Currently only three-digit status codes are supported (e.g.
404
,500
etc.). Also the special formDIG*
(e.g4*
) is supported to cover the group of codes in one directive. - file Path to file with page body.
Below are examples:
response_body 403 /path/to/403.html;
response_body 5* /path/to/5xx.html;
These directive can be specified repeatedly. In case of the same status codes
(or code groups) are repeated, the latter directive overwrites the former one.
There is one exception in this rule: directive with simple status code has
always higher priority than directive with group of codes (4*
, 5*
); thus,
directive with simple status code cannot be overwritten by directive with group
of codes. Pay attention, that if we deal with attack, we can't be sure that
error response with body will be delivered to the client (We can't process
WINDOW_UPDATE frames for HTTP2 connection after socket closing, so we can't
send error response with a large body, moreover if kernel receive any data on
the DEAD sock, it send TCP RST for such client) .
By default error responses are sent without bodies.
White list marks are intended to designate HTTP requests from particular sources as allowable to bypass Frang and Sticky cookies modules. The syntax is as follows:
whitelist_mark <marks>
-
<marks>
is a list of space separated marks (integers in range from 0 to 4294967295), which can be set for particular types of network packets via nftables, iptables or bpfilter interfaces to designate such packets as allowable to bypass Frang filtering and Sticky cookies procedures of Tempesta FW.
By default white list marks are disabled.
Example:
whitelist_mark 1 7531 777 11235;
The current accounting data for HTTP limits is stored with the client descriptor. This data is kept for some time after closing all connections for a given client in order to take better account of security restrictions. When the connection is established, the client is determined by the address of the peer from which it is received. When processing requests, the client can be precised using the User-Agent and X-Forwarded-For headers.
client_lru_size <SIZE>
: Specifies the maximum number of client connections that Tempesta FW can
retain in database using a bounded Least Recently Used (LRU) list.
Each time a client connects, it is added to this list. If the list reaches
the specified limit, the oldest client connection is removed from both the
list and the database. When the client removed from database but still has
connection, it will be freed when connection closed. This limit helps manage
database memory usage and remove clients that was disconnected, but still
present in database.
Defaults: 8000
.
WARNING: It affects accuracy of Frang, don't set low values. The lower the value, the more clients will be not accounted by Frang.
During Tempesta work client accounting data is stored in a database that is configured by the following options:
сlient_db <PATH_TO_DB>
: Path to a client database file used as a storage for clients info.
Defaults: /opt/tempesta/db/client.tdb
The PATH must be absolute and the directory must exist. The database file must end with .tdb
.
client_tbl_size <SIZE>
: Size of the Tempesta DB file in bytes used as client accounting data storage.
Defaults: 16M
.
Can use suffixes K, M, G. The size must be multiple of 2MB (Tempesta DB extent size).
Clients can be permanently blocked according to security limits. The blocked client IP addresses are stored in the filter Tempesta DB table.
There are two configuration options for the table:
filter_db <PATH_TO_DB>
: Path to the database file.
Defaults: /opt/tempesta/db/filter.tdb
The PATH must be absolute and the directory must exist. The database file must end with .tdb
.
filter_tbl_size <SIZE>
: Size of the filter Tempesta DB.
Defaults: 16M
.
Can use suffixes K, M, G. The size must be multiple of 2MB (Tempesta DB extent size).
Maximum size of header list that the sender is prepared to accept, in bytes. The value is based on the uncompressed size of header fields, including the length of the name and value in bytes plus an overhead of 32 bytes for each header field (according RFC 9113 6.5.2.). Also this limit is acceptable for HTTP1.
http_max_header_list_size <SIZE>
: Total size of all headers.
Defaults: 16384
.
Administrator can specify maximal count of concurrently active streams for a single connection:
max_concurrent_streams <NUM>
Defaults: 100
.
- Home
- Requirements
- Installation
-
Configuration
- Migration from Nginx
- On-the-fly reconfiguration
- Handling clients
- Backend servers
- Load Balancing
- Caching Responses
- Non-Idempotent Requests
- Modify HTTP Messages
- Virtual hosts and locations
- HTTP Session Management
- HTTP Tables
- HTTP(S) Security
- Header Via
- Health monitor
- TLS
- Virtual host confusion
- Traffic Filtering by Fingerprints
- Run & Stop
- Application Performance Monitoring
- Use cases
- Performance
- Contributing