/share/man/man4/siftr.4

  1.\"
  2.\" Copyright (c) 2010 The FreeBSD Foundation
  3.\" All rights reserved.
  4.\"
  5.\" Portions of this software were developed at the Centre for Advanced
  6.\" Internet Architectures, Swinburne University of Technology, Melbourne,
  7.\" Australia by Lawrence Stewart under sponsorship from the FreeBSD
  8.\" Foundation.
  9.\"
 10.\" Redistribution and use in source and binary forms, with or without
 11.\" modification, are permitted provided that the following conditions
 12.\" are met:
 13.\" 1. Redistributions of source code must retain the above copyright
 14.\"    notice, this list of conditions, and the following disclaimer,
 15.\"    without modification, immediately at the beginning of the file.
 16.\" 2. The name of the author may not be used to endorse or promote products
 17.\"    derived from this software without specific prior written permission.
 18.\"
 19.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
 20.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 21.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
 22.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR
 23.\" ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 24.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
 25.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
 26.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
 27.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
 28.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
 29.\" SUCH DAMAGE.
 30.\"
 31.\" $FreeBSD$
 32.\"
 33.Dd November 12, 2010
 34.Dt SIFTR 4
 35.Os
 36.Sh NAME
 37.Nm SIFTR
 38.Nd Statistical Information For TCP Research
 39.Sh SYNOPSIS
 40To load
 41the driver
 42as a module at run-time, run the following command as root:
 43.Bd -literal -offset indent
 44kldload siftr
 45.Ed
 46.Pp
 47Alternatively, to load
 48the driver
 49as a module at boot time, add the following line into the
 50.Xr loader.conf 5
 51file:
 52.Bd -literal -offset indent
 53siftr_load="YES"
 54.Ed
 55.Sh DESCRIPTION
 56The
 57.Nm
 58.Po
 59.Em S Ns tatistical
 60.Em I Ns nformation
 61.Em F Ns or
 62.Em T Ns CP
 63.Em R Ns esearch
 64.Pc
 65kernel module logs a range of statistics on active TCP connections to
 66a log file.
 67It provides the ability to make highly granular measurements of TCP connection
 68state, aimed at system administrators, developers and researchers.
 69.Ss Compile-time Configuration
 70The default operation of
 71.Nm
 72is to capture IPv4 TCP/IP packets.
 73.Nm
 74can be configured to support IPv4 and IPv6 by uncommenting:
 75.Bd -literal -offset indent
 76CFLAGS+=-DSIFTR_IPV6
 77.Ed
 78.Pp
 79in
 80.Aq sys/modules/siftr/Makefile
 81and recompiling.
 82.Pp
 83In the IPv4-only (default) mode, standard dotted decimal notation (e.g.
 84"136.186.229.95") is used to format IPv4 addresses for logging.
 85In IPv6 mode, standard dotted decimal notation is used to format IPv4 addresses,
 86and standard colon-separated hex notation (see RFC 4291) is used to format IPv6
 87addresses for logging. Note that SIFTR uses uncompressed notation to format IPv6
 88addresses.
 89For example, the address "fe80::20f:feff:fea2:531b" would be logged as
 90"fe80:0:0:0:20f:feff:fea2:531b".
 91.Ss Run-time Configuration
 92.Nm
 93utilises the
 94.Xr sysctl 8
 95interface to export its configuration variables to user-space.
 96The following variables are available:
 97.Bl -tag -offset indent -width Va
 98.It Va net.inet.siftr.enabled
 99controls whether the module performs its
100measurements or not.
101By default, the value is set to 0, which means the module
102will not be taking any measurements.
103Having the module loaded with
104.Va net.inet.siftr.enabled
105set to 0 will have no impact on the performance of the network stack, as the
106packet filtering hooks are only inserted when
107.Va net.inet.siftr.enabled
108is set to 1.
109.El
110.Bl -tag -offset indent -width Va
111.It Va net.inet.siftr.ppl
112controls how many inbound/outbound packets for a given TCP connection will cause
113a log message to be generated for the connection.
114By default, the value is set to 1, which means the module will log a message for
115every packet of every TCP connection.
116The value can be set to any integer in the range [1,2^32], and can be changed at
117any time, even while the module is enabled.
118.El
119.Bl -tag -offset indent -width Va
120.It Va net.inet.siftr.logfile
121controls the path to the file that the module writes its log messages to.
122By default, the file /var/log/siftr.log is used.
123The path can be changed at any time, even while the module is enabled.
124.El
125.Bl -tag -offset indent -width Va
126.It Va net.inet.siftr.genhashes
127controls whether a hash is generated for each TCP packet seen by
128.Nm .
129By default, the value is set to 0, which means no hashes are generated.
130The hashes are useful to correlate which TCP packet triggered the generation of
131a particular log message, but calculating them adds additional computational
132overhead into the fast path.
133.El
134.Ss Log Format
135A typical
136.Nm
137log file will contain 3 different types of log message.
138All messages are written in plain ASCII text.
139.Pp
140Note: The
141.Qq \e
142present in the example log messages in this section indicates a
143line continuation and is not part of the actual log message.
144.Pp
145The first type of log message is written to the file when the module is
146enabled and starts collecting data from the running kernel. The text below
147shows an example module enable log. The fields are tab delimited key-value
148pairs which describe some basic information about the system.
149.Bd -literal -offset indent
150enable_time_secs=1238556193    enable_time_usecs=462104 \\
151siftrver=1.2.2    hz=1000    tcp_rtt_scale=32 \\
152sysname=FreeBSD    sysver=604000    ipmode=4
153.Ed
154.Pp
155Field descriptions are as follows:
156.Bl -tag -offset indent -width Va
157.It Va enable_time_secs
158time at which the module was enabled, in seconds since the UNIX epoch.
159.El
160.Bl -tag -offset indent -width Va
161.It Va enable_time_usecs
162time at which the module was enabled, in microseconds since enable_time_secs.
163.El
164.Bl -tag -offset indent -width Va
165.It Va siftrver
166version of
167.Nm .
168.El
169.Bl -tag -offset indent -width Va
170.It Va hz
171tick rate of the kernel in ticks per second.
172.El
173.Bl -tag -offset indent -width Va
174.It Va tcp_rtt_scale
175smoothed RTT estimate scaling factor.
176.El
177.Bl -tag -offset indent -width Va
178.It Va sysname
179operating system name.
180.El
181.Bl -tag -offset indent -width Va
182.It Va sysver
183operating system version.
184.El
185.Bl -tag -offset indent -width Va
186.It Va ipmode
187IP mode as defined at compile time.
188An ipmode of "4" means IPv6 is not supported and IP addresses are logged in
189regular dotted quad format.
190An ipmode of "6" means IPv6 is supported, and IP addresses are logged in dotted
191quad or hex format, as described in the
192.Qq Compile-time Configuration
193subsection.
194.El
195.Pp
196The second type of log message is written to the file when a data log message
197is generated.
198The text below shows an example data log triggered by an IPv4
199TCP/IP packet.
200The data is CSV formatted.
201.Bd -literal -offset indent
202o,0xbec491a5,1238556193.463551,172.16.7.28,22,172.16.2.5,55931, \\
2031073725440,172312,6144,66560,66608,8,1,4,1448,936,1,996,255, \\
20433304,208,66608,0,208,0
205.Ed
206.Pp
207Field descriptions are as follows:
208.Bl -tag -offset indent -width Va
209.It Va 1
210Direction of packet that triggered the log message.
211Either
212.Qq i
213for in, or
214.Qq o
215for out.
216.El
217.Bl -tag -offset indent -width Va
218.It Va 2
219Hash of the packet that triggered the log message.
220.El
221.Bl -tag -offset indent -width Va
222.It Va 3
223Time at which the packet that triggered the log message was processed by
224the
225.Xr pfil 9
226hook function, in seconds and microseconds since the UNIX epoch.
227.El
228.Bl -tag -offset indent -width Va
229.It Va 4
230The IPv4 or IPv6 address of the local host, in dotted quad (IPv4 packet)
231or colon-separated hex (IPv6 packet) notation.
232.El
233.Bl -tag -offset indent -width Va
234.It Va 5
235The TCP port that the local host is communicating via.
236.El
237.Bl -tag -offset indent -width Va
238.It Va 6
239The IPv4 or IPv6 address of the foreign host, in dotted quad (IPv4 packet)
240or colon-separated hex (IPv6 packet) notation.
241.El
242.Bl -tag -offset indent -width Va
243.It Va 7
244The TCP port that the foreign host is communicating via.
245.El
246.Bl -tag -offset indent -width Va
247.It Va 8
248The slow start threshold for the flow, in bytes.
249.El
250.Bl -tag -offset indent -width Va
251.It Va 9
252The current congestion window for the flow, in bytes.
253.El
254.Bl -tag -offset indent -width Va
255.It Va 10
256The current bandwidth-controlled window for the flow, in bytes.
257.El
258.Bl -tag -offset indent -width Va
259.It Va 11
260The current sending window for the flow, in bytes.
261The post scaled value is reported, except during the initial handshake (first
262few packets), during which time the unscaled value is reported.
263.El
264.Bl -tag -offset indent -width Va
265.It Va 12
266The current receive window for the flow, in bytes.
267The post scaled value is always reported.
268.El
269.Bl -tag -offset indent -width Va
270.It Va 13
271The current window scaling factor for the sending window.
272.El
273.Bl -tag -offset indent -width Va
274.It Va 14
275The current window scaling factor for the receiving window.
276.El
277.Bl -tag -offset indent -width Va
278.It Va 15
279The current state of the TCP finite state machine, as defined
280in
281.Aq Pa netinet/tcp_fsm.h .
282.El
283.Bl -tag -offset indent -width Va
284.It Va 16
285The maximum segment size for the flow, in bytes.
286.El
287.Bl -tag -offset indent -width Va
288.It Va 17
289The current smoothed RTT estimate for the flow, in units of TCP_RTT_SCALE * HZ,
290where TCP_RTT_SCALE is a define found in tcp_var.h, and HZ is the kernel's tick
291timer.
292Divide by TCP_RTT_SCALE * HZ to get the RTT in secs. TCP_RTT_SCALE and HZ are
293reported in the enable log message.
294.El
295.Bl -tag -offset indent -width Va
296.It Va 18
297SACK enabled indicator. 1 if SACK enabled, 0 otherwise.
298.El
299.Bl -tag -offset indent -width Va
300.It Va 19
301The current state of the TCP flags for the flow.
302See
303.Aq Pa netinet/tcp_var.h
304for information about the various flags.
305.El
306.Bl -tag -offset indent -width Va
307.It Va 20
308The current retransmission timeout length for the flow, in units of HZ, where HZ
309is the kernel's tick timer.
310Divide by HZ to get the timeout length in seconds. HZ is reported in the
311enable log message.
312.El
313.Bl -tag -offset indent -width Va
314.It Va 21
315The current size of the socket send buffer in bytes.
316.El
317.Bl -tag -offset indent -width Va
318.It Va 22
319The current number of bytes in the socket send buffer.
320.El
321.Bl -tag -offset indent -width Va
322.It Va 23
323The current size of the socket receive buffer in bytes.
324.El
325.Bl -tag -offset indent -width Va
326.It Va 24
327The current number of bytes in the socket receive buffer.
328.El
329.Bl -tag -offset indent -width Va
330.It Va 25
331The current number of unacknowledged bytes in-flight.
332Bytes acknowledged via SACK are not excluded from this count.
333.El
334.Bl -tag -offset indent -width Va
335.It Va 26
336The current number of segments in the reassembly queue.
337.El
338.Pp
339The third type of log message is written to the file when the module is disabled
340and ceases collecting data from the running kernel.
341The text below shows an example module disable log.
342The fields are tab delimited key-value pairs which provide statistics about
343operations since the module was most recently enabled.
344.Bd -literal -offset indent
345disable_time_secs=1238556197    disable_time_usecs=933607 \\
346num_inbound_tcp_pkts=356    num_outbound_tcp_pkts=627 \\
347total_tcp_pkts=983    num_inbound_skipped_pkts_malloc=0 \\
348num_outbound_skipped_pkts_malloc=0    num_inbound_skipped_pkts_mtx=0 \\
349num_outbound_skipped_pkts_mtx=0    num_inbound_skipped_pkts_tcb=0 \\
350num_outbound_skipped_pkts_tcb=0    num_inbound_skipped_pkts_icb=0 \\
351num_outbound_skipped_pkts_icb=0    total_skipped_tcp_pkts=0 \\
352flow_list=172.16.7.28;22-172.16.2.5;55931,
353.Ed
354.Pp
355Field descriptions are as follows:
356.Bl -tag -offset indent -width Va
357.It Va disable_time_secs
358Time at which the module was disabled, in seconds since the UNIX epoch.
359.El
360.Bl -tag -offset indent -width Va
361.It Va disable_time_usecs
362Time at which the module was disabled, in microseconds since disable_time_secs.
363.El
364.Bl -tag -offset indent -width Va
365.It Va num_inbound_tcp_pkts
366Number of TCP packets that traversed up the network stack.
367This only includes inbound TCP packets during the periods when
368.Nm
369was enabled.
370.El
371.Bl -tag -offset indent -width Va
372.It Va num_outbound_tcp_pkts
373Number of TCP packets that traversed down the network stack.
374This only includes outbound TCP packets during the periods when
375.Nm
376was enabled.
377.El
378.Bl -tag -offset indent -width Va
379.It Va total_tcp_pkts
380The summation of num_inbound_tcp_pkts and num_outbound_tcp_pkts.
381.El
382.Bl -tag -offset indent -width Va
383.It Va num_inbound_skipped_pkts_malloc
384Number of inbound packets that were not processed because of failed malloc() calls.
385.El
386.Bl -tag -offset indent -width Va
387.It Va num_outbound_skipped_pkts_malloc
388Number of outbound packets that were not processed because of failed malloc() calls.
389.El
390.Bl -tag -offset indent -width Va
391.It Va num_inbound_skipped_pkts_mtx
392Number of inbound packets that were not processed because of failure to add the
393packet to the packet processing queue.
394.El
395.Bl -tag -offset indent -width Va
396.It Va num_outbound_skipped_pkts_mtx
397Number of outbound packets that were not processed because of failure to add the
398packet to the packet processing queue.
399.El
400.Bl -tag -offset indent -width Va
401.It Va num_inbound_skipped_pkts_tcb
402Number of inbound packets that were not processed because of failure to find the
403TCP control block associated with the packet.
404.El
405.Bl -tag -offset indent -width Va
406.It Va num_outbound_skipped_pkts_tcb
407Number of outbound packets that were not processed because of failure to find
408the TCP control block associated with the packet.
409.El
410.Bl -tag -offset indent -width Va
411.It Va num_inbound_skipped_pkts_icb
412Number of inbound packets that were not processed because of failure to find the
413IP control block associated with the packet.
414.El
415.Bl -tag -offset indent -width Va
416.It Va num_outbound_skipped_pkts_icb
417Number of outbound packets that were not processed because of failure to find
418the IP control block associated with the packet.
419.El
420.Bl -tag -offset indent -width Va
421.It Va total_skipped_tcp_pkts
422The summation of all skipped packet counters.
423.El
424.Bl -tag -offset indent -width Va
425.It Va flow_list
426A CSV list of TCP flows that triggered data log messages to be generated since
427the module was loaded.
428Each flow entry in the CSV list is
429formatted as
430.Qq local_ip;local_port-foreign_ip;foreign_port .
431If there are no entries in the list (i.e., no data log messages were generated),
432the value will be blank.
433If there is at least one entry in the list, a trailing comma will always be
434present.
435.El
436.Pp
437The total number of data log messages found in the log file for a module
438enable/disable cycle should equate to total_tcp_pkts - total_skipped_tcp_pkts.
439.Sh IMPLEMENTATION NOTES
440.Nm
441hooks into the network stack using the
442.Xr pfil 9
443interface.
444In its current incarnation, it hooks into the AF_INET/AF_INET6 (IPv4/IPv6)
445.Xr pfil 9
446filtering points, which means it sees packets at the IP layer of the network
447stack.
448This means that TCP packets inbound to the stack are intercepted before
449they have been processed by the TCP layer.
450Packets outbound from the stack are intercepted after they have been processed
451by the TCP layer.
452.Pp
453The diagram below illustrates how
454.Nm
455inserts itself into the stack.
456.Bd -literal -offset indent
457----------------------------------
458           Upper Layers
459----------------------------------
460    ^                       |
461    |                       |
462    |                       |
463    |                       v
464 TCP in                  TCP out
465----------------------------------
466    ^                      |
467    |________     _________|
468            |     |
469            |     v
470           ---------
471           | SIFTR |
472           ---------
473            ^     |
474    ________|     |__________
475    |                       |
476    |                       v
477IPv{4/6} in            IPv{4/6} out
478----------------------------------
479    ^                       |
480    |                       |
481    |                       v
482Layer 2 in             Layer 2 out
483----------------------------------
484          Physical Layer
485----------------------------------
486.Ed
487.Pp
488.Nm
489uses the
490.Xr alq 9
491interface to manage writing data to disk.
492.Pp
493At first glance, you might mistakenly think that
494.Nm
495extracts information from
496individual TCP packets.
497This is not the case.
498.Nm
499uses TCP packet events (inbound and outbound) for each TCP flow originating from
500the system to trigger a dump of the state of the TCP control block for that
501flow.
502With the PPL set to 1, we are in effect sampling each TCP flow's control block
503state as frequently as flow packets enter/leave the system.
504For example, setting PPL to 2 halves the sampling rate i.e., every second flow
505packet (inbound OR outbound) causes a dump of the control block state.
506.Pp
507The distinction between interrogating individual packets versus interrogating the
508control block is important, because
509.Nm
510does not remove the need for packet capturing tools like
511.Xr tcpdump 1 .
512.Nm
513allows you to correlate and observe the cause-and-affect relationship between
514what you see on the wire (captured using a tool like
515.Xr tcpdump 1 Ns )
516and changes in the TCP control block corresponding to the flow of interest.
517It is therefore useful to use
518.Nm
519and a tool like
520.Xr tcpdump 1
521to gather the necessary data to piece together the complete picture.
522Use of either tool on its own will not be able to provide all of the necessary
523data.
524.Pp
525As a result of needing to interrogate the TCP control block, certain packets
526during the lifecycle of a connection are unable to trigger a
527.Nm
528log message.
529The initial handshake takes place without the existence of a control block and
530the final ACK is exchanged when the connection is in the TIMEWAIT state.
531.Pp
532.Nm
533was designed to minimise the delay introduced to packets traversing the network
534stack.
535This design called for a highly optimised and minimal hook function that
536extracted the minimal details necessary whilst holding the packet up, and
537passing these details to another thread for actual processing and logging.
538.Pp
539This multithreaded design does introduce some contention issues when accessing
540the data structure shared between the threads of operation.
541When the hook function tries to place details in the structure, it must first
542acquire an exclusive lock.
543Likewise, when the processing thread tries to read details from the structure,
544it must also acquire an exclusive lock to do so.
545If one thread holds the lock, the other must wait before it can obtain it.
546This does introduce some additional bounded delay into the kernel's packet
547processing code path.
548.Pp
549In some cases (e.g., low memory, connection termination), TCP packets that enter
550the
551.Nm
552.Xr pfil 9
553hook function will not trigger a log message to be generated.
554.Nm
555refers to this outcome as a
556.Qq skipped packet .
557Note that
558.Nm
559always ensures that packets are allowed to continue through the stack, even if
560they could not successfully trigger a data log message.
561.Nm
562will therefore not introduce any packet loss for TCP/IP packets traversing the
563network stack.
564.Ss Important Behaviours
565The behaviour of a log file path change whilst the module is enabled is as
566follows:
567.Bl -enum
568.It
569Attempt to open the new file path for writing.
570If this fails, the path change will fail and the existing path will continue to
571be used.
572.It
573Assuming the new path is valid and opened successfully:
574.Bl -dash
575.It
576Flush all pending log messages to the old file path.
577.It
578Close the old file path.
579.It
580Switch the active log file pointer to point at the new file path.
581.It
582Commence logging to the new file.
583.El
584.El
585.Pp
586During the time between the flush of pending log messages to the old file and
587commencing logging to the new file, new log messages will still be generated and
588buffered.
589As soon as the new file path is ready for writing, the accumulated log messages
590will be written out to the file.
591.Sh EXAMPLES
592To enable the module's operations, run the following command as root:
593sysctl net.inet.siftr.enabled=1
594.Pp
595To change the granularity of log messages such that 1 log message is
596generated for every 10 TCP packets per connection, run the following
597command as root:
598sysctl net.inet.siftr.ppl=10
599.Pp
600To change the log file location to /tmp/siftr.log, run the following
601command as root:
602sysctl net.inet.siftr.logfile=/tmp/siftr.log
603.Sh SEE ALSO
604.Xr tcpdump 1 ,
605.Xr tcp 4 ,
606.Xr sysctl 8 ,
607.Xr alq 9 ,
608.Xr pfil 9
609.Sh ACKNOWLEDGEMENTS
610Development of this software was made possible in part by grants from the
611Cisco University Research Program Fund at Community Foundation Silicon Valley,
612and the FreeBSD Foundation.
613.Sh HISTORY
614.Nm
615first appeared in
616.Fx 7.4
617and
618.Fx 8.2 .
619.Pp
620.Nm
621was first released in 2007 by Lawrence Stewart and James Healy whilst working on
622the NewTCP research project at Swinburne University of Technology's Centre for
623Advanced Internet Architectures, Melbourne, Australia, which was made possible
624in part by a grant from the Cisco University Research Program Fund at Community
625Foundation Silicon Valley.
626More details are available at:
627.Pp
628http://caia.swin.edu.au/urp/newtcp/
629.Pp
630Work on
631.Nm
632v1.2.x was sponsored by the FreeBSD Foundation as part of
633the
634.Qq Enhancing the FreeBSD TCP Implementation
635project 2008-2009.
636More details are available at:
637.Pp
638http://www.freebsdfoundation.org/
639.Pp
640http://caia.swin.edu.au/freebsd/etcp09/
641.Sh AUTHORS
642.An -nosplit
643.Nm
644was written by
645.An Lawrence Stewart Aq lstewart@FreeBSD.org
646and
647.An James Healy Aq jimmy@deefa.com .
648.Pp
649This manual page was written by
650.An Lawrence Stewart Aq lstewart@FreeBSD.org .
651.Sh BUGS
652Current known limitations and any relevant workarounds are outlined below:
653.Bl -dash
654.It
655The internal queue used to pass information between the threads of operation is
656currently unbounded.
657This allows
658.Nm
659to cope with bursty network traffic, but sustained high packet-per-second
660traffic can cause exhaustion of kernel memory if the processing thread cannot
661keep up with the packet rate.
662.It
663If using
664.Nm
665on a machine that is also running other modules utilising the
666.Xr pfil 9
667framework e.g.
668.Xr dummynet 4 ,
669.Xr ipfw 8 ,
670.Xr pf 4 Ns ,
671the order in which you load the modules is important.
672You should kldload the other modules first, as this will ensure TCP packets
673undergo any necessary manipulations before
674.Nm
675.Qq sees
676and processes them.
677.It
678There is a known, harmless lock order reversal warning between the
679.Xr pfil 9
680mutex and tcbinfo TCP lock reported by
681.Xr witness 4
682when
683.Nm
684is enabled in a kernel compiled with
685.Xr witness 4
686support.
687.It
688There is no way to filter which TCP flows you wish to capture data for.
689Post processing is required to separate out data belonging to particular flows
690of interest.
691.It
692The module does not detect deletion of the log file path.
693New log messages will simply be lost if the log file being used by
694.Nm
695is deleted whilst the module is set to use the file.
696Switching to a new log file using the
697.Em net.inet.siftr.logfile
698variable will create the new file and allow log messages to begin being written
699to disk again.
700The new log file path must differ from the path to the deleted file.
701.It
702The hash table used within the code is sized to hold 65536 flows.  This is not a
703hard limit, because chaining is used to handle collisions within the hash table
704structure.
705However, we suspect (based on analogies with other hash table performance data)
706that the hash table look up performance (and therefore the module's packet
707processing performance) will degrade in an exponential manner as the number of
708unique flows handled in a module enable/disable cycle approaches and surpasses
70965536.
710.It
711There is no garbage collection performed on the flow hash table.
712The only way currently to flush it is to disable
713.Nm .
714.It
715The PPL variable applies to packets that make it into the processing thread,
716not total packets received in the hook function.
717Packets are skipped before the PPL variable is applied, which means there may be
718a slight discrepancy in the triggering of log messages.
719For example, if PPL was set to 10, and the 8th packet since the last log message
720is skipped, the 11th packet will actually trigger the log message to be
721generated.
722This is discussed in greater depth in CAIA technical report 070824A.
723.It
724At the time of writing, there was no simple way to hook into the TCP layer
725to intercept packets.
726.Nm Ap s
727use of IP layer hook points means all IP
728traffic will be processed by the
729.Nm
730.Xr pfil 9
731hook function, which introduces minor, but nonetheless unnecessary packet delay
732and processing overhead on the system for non-TCP packets as well.
733Hooking in at the IP layer is also not ideal from the data gathering point of
734view.
735Packets traversing up the stack will be intercepted and cause a log message
736generation BEFORE they have been processed by the TCP layer, which means we
737cannot observe the cause-and-affect relationship between inbound events and the
738corresponding TCP control block as precisely as could be.
739Ideally,
740.Nm
741should intercept packets after they have been processed by the TCP layer i.e.
742intercept packets coming up the stack after they have been processed by
743tcp_input(), and intercept packets coming down the stack after they have been
744processed by tcp_output().
745The current code still gives satisfactory granularity though, as inbound events
746tend to trigger outbound events, allowing the cause-and-effect to be observed
747indirectly by capturing the state on outbound events as well.
748.It
749The
750.Qq inflight bytes
751value logged by
752.Nm
753does not take into account bytes that have been
754.No SACK Ap ed
755by the receiving host.
756.It
757Packet hash generation does not currently work for IPv6 based TCP packets.
758.It
759Compressed notation is not used for IPv6 address representation.
760This consumes more bytes than is necessary in log output.
761.El