/share/man/man7/tuning.7

   1.\" Copyright (C) 2001 Matthew Dillon. All rights reserved.
   2.\"
   3.\" Redistribution and use in source and binary forms, with or without
   4.\" modification, are permitted provided that the following conditions
   5.\" are met:
   6.\" 1. Redistributions of source code must retain the above copyright
   7.\"    notice, this list of conditions and the following disclaimer.
   8.\" 2. Redistributions in binary form must reproduce the above copyright
   9.\"    notice, this list of conditions and the following disclaimer in the
  10.\"    documentation and/or other materials provided with the distribution.
  11.\"
  12.\" THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
  13.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
  14.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
  15.\" ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
  16.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
  17.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
  18.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
  19.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
  20.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
  21.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  22.\" SUCH DAMAGE.
  23.\"
  24.\" $FreeBSD$
  25.\"
  26.Dd May 11, 2012
  27.Dt TUNING 7
  28.Os
  29.Sh NAME
  30.Nm tuning
  31.Nd performance tuning under FreeBSD
  32.Sh SYSTEM SETUP - DISKLABEL, NEWFS, TUNEFS, SWAP
  33When using
  34.Xr bsdlabel 8
  35or
  36.Xr sysinstall 8
  37to lay out your file systems on a hard disk it is important to remember
  38that hard drives can transfer data much more quickly from outer tracks
  39than they can from inner tracks.
  40To take advantage of this you should
  41try to pack your smaller file systems and swap closer to the outer tracks,
  42follow with the larger file systems, and end with the largest file systems.
  43It is also important to size system standard file systems such that you
  44will not be forced to resize them later as you scale the machine up.
  45I usually create, in order, a 128M root, 1G swap, 128M
  46.Pa /var ,
  47128M
  48.Pa /var/tmp ,
  493G
  50.Pa /usr ,
  51and use any remaining space for
  52.Pa /home .
  53.Pp
  54You should typically size your swap space to approximately 2x main memory
  55for systems with less than 2GB of RAM, or approximately 1x main memory
  56if you have more.
  57If you do not have a lot of RAM, though, you will generally want a lot
  58more swap.
  59It is not recommended that you configure any less than
  60256M of swap on a system and you should keep in mind future memory
  61expansion when sizing the swap partition.
  62The kernel's VM paging algorithms are tuned to perform best when there is
  63at least 2x swap versus main memory.
  64Configuring too little swap can lead
  65to inefficiencies in the VM page scanning code as well as create issues
  66later on if you add more memory to your machine.
  67Finally, on larger systems
  68with multiple SCSI disks (or multiple IDE disks operating on different
  69controllers), we strongly recommend that you configure swap on each drive.
  70The swap partitions on the drives should be approximately the same size.
  71The kernel can handle arbitrary sizes but
  72internal data structures scale to 4 times the largest swap partition.
  73Keeping
  74the swap partitions near the same size will allow the kernel to optimally
  75stripe swap space across the N disks.
  76Do not worry about overdoing it a
  77little, swap space is the saving grace of
  78.Ux
  79and even if you do not normally use much swap, it can give you more time to
  80recover from a runaway program before being forced to reboot.
  81.Pp
  82How you size your
  83.Pa /var
  84partition depends heavily on what you intend to use the machine for.
  85This
  86partition is primarily used to hold mailboxes, the print spool, and log
  87files.
  88Some people even make
  89.Pa /var/log
  90its own partition (but except for extreme cases it is not worth the waste
  91of a partition ID).
  92If your machine is intended to act as a mail
  93or print server,
  94or you are running a heavily visited web server, you should consider
  95creating a much larger partition \(en perhaps a gig or more.
  96It is very easy
  97to underestimate log file storage requirements.
  98.Pp
  99Sizing
 100.Pa /var/tmp
 101depends on the kind of temporary file usage you think you will need.
 102128M is
 103the minimum we recommend.
 104Also note that sysinstall will create a
 105.Pa /tmp
 106directory.
 107Dedicating a partition for temporary file storage is important for
 108two reasons: first, it reduces the possibility of file system corruption
 109in a crash, and second it reduces the chance of a runaway process that
 110fills up
 111.Oo Pa /var Oc Ns Pa /tmp
 112from blowing up more critical subsystems (mail,
 113logging, etc).
 114Filling up
 115.Oo Pa /var Oc Ns Pa /tmp
 116is a very common problem to have.
 117.Pp
 118In the old days there were differences between
 119.Pa /tmp
 120and
 121.Pa /var/tmp ,
 122but the introduction of
 123.Pa /var
 124(and
 125.Pa /var/tmp )
 126led to massive confusion
 127by program writers so today programs haphazardly use one or the
 128other and thus no real distinction can be made between the two.
 129So it makes sense to have just one temporary directory and
 130softlink to it from the other
 131.Pa tmp
 132directory locations.
 133However you handle
 134.Pa /tmp ,
 135the one thing you do not want to do is leave it sitting
 136on the root partition where it might cause root to fill up or possibly
 137corrupt root in a crash/reboot situation.
 138.Pp
 139The
 140.Pa /usr
 141partition holds the bulk of the files required to support the system and
 142a subdirectory within it called
 143.Pa /usr/local
 144holds the bulk of the files installed from the
 145.Xr ports 7
 146hierarchy.
 147If you do not use ports all that much and do not intend to keep
 148system source
 149.Pq Pa /usr/src
 150on the machine, you can get away with
 151a 1 GB
 152.Pa /usr
 153partition.
 154However, if you install a lot of ports
 155(especially window managers and Linux-emulated binaries), we recommend
 156at least a 2 GB
 157.Pa /usr
 158and if you also intend to keep system source
 159on the machine, we recommend a 3 GB
 160.Pa /usr .
 161Do not underestimate the
 162amount of space you will need in this partition, it can creep up and
 163surprise you!
 164.Pp
 165The
 166.Pa /home
 167partition is typically used to hold user-specific data.
 168I usually size it to the remainder of the disk.
 169.Pp
 170Why partition at all?
 171Why not create one big
 172.Pa /
 173partition and be done with it?
 174Then I do not have to worry about undersizing things!
 175Well, there are several reasons this is not a good idea.
 176First,
 177each partition has different operational characteristics and separating them
 178allows the file system to tune itself to those characteristics.
 179For example,
 180the root and
 181.Pa /usr
 182partitions are read-mostly, with very little writing, while
 183a lot of reading and writing could occur in
 184.Pa /var
 185and
 186.Pa /var/tmp .
 187By properly
 188partitioning your system fragmentation introduced in the smaller more
 189heavily write-loaded partitions will not bleed over into the mostly-read
 190partitions.
 191Additionally, keeping the write-loaded partitions closer to
 192the edge of the disk (i.e., before the really big partitions instead of after
 193in the partition table) will increase I/O performance in the partitions
 194where you need it the most.
 195Now it is true that you might also need I/O
 196performance in the larger partitions, but they are so large that shifting
 197them more towards the edge of the disk will not lead to a significant
 198performance improvement whereas moving
 199.Pa /var
 200to the edge can have a huge impact.
 201Finally, there are safety concerns.
 202Having a small neat root partition that
 203is essentially read-only gives it a greater chance of surviving a bad crash
 204intact.
 205.Pp
 206Properly partitioning your system also allows you to tune
 207.Xr newfs 8 ,
 208and
 209.Xr tunefs 8
 210parameters.
 211Tuning
 212.Xr newfs 8
 213requires more experience but can lead to significant improvements in
 214performance.
 215There are three parameters that are relatively safe to tune:
 216.Em blocksize , bytes/i-node ,
 217and
 218.Em cylinders/group .
 219.Pp
 220.Fx
 221performs best when using 16K or 32K file system block sizes.
 222The default file system block size is 32K,
 223which provides best performance for most applications,
 224with the exception of those that perform random access on large files
 225(such as database server software).
 226Such applications tend to perform better with a smaller block size,
 227although modern disk characteristics are such that the performance
 228gain from using a smaller block size may not be worth consideration.
 229Using a block size larger than 32K
 230can cause fragmentation of the buffer cache and
 231lead to lower performance.
 232.Pp
 233The defaults may be unsuitable
 234for a file system that requires a very large number of i-nodes
 235or is intended to hold a large number of very small files.
 236Such a file system should be created with an 4K, 8K, or 16K block size.
 237This also requires you to specify a smaller
 238fragment size.
 239We recommend always using a fragment size that is 1/8
 240the block size (less testing has been done on other fragment size factors).
 241The
 242.Xr newfs 8
 243options for this would be
 244.Dq Li "newfs -f 1024 -b 8192 ..." .
 245.Pp
 246If a large partition is intended to be used to hold fewer, larger files, such
 247as database files, you can increase the
 248.Em bytes/i-node
 249ratio which reduces the number of i-nodes (maximum number of files and
 250directories that can be created) for that partition.
 251Decreasing the number
 252of i-nodes in a file system can greatly reduce
 253.Xr fsck 8
 254recovery times after a crash.
 255Do not use this option
 256unless you are actually storing large files on the partition, because if you
 257overcompensate you can wind up with a file system that has lots of free
 258space remaining but cannot accommodate any more files.
 259Using 65536, 131072, or 262144 bytes/i-node is recommended.
 260You can go higher but
 261it will have only incremental effects on
 262.Xr fsck 8
 263recovery times.
 264For example,
 265.Dq Li "newfs -i 65536 ..." .
 266.Pp
 267.Xr tunefs 8
 268may be used to further tune a file system.
 269This command can be run in
 270single-user mode without having to reformat the file system.
 271However, this is possibly the most abused program in the system.
 272Many people attempt to
 273increase available file system space by setting the min-free percentage to 0.
 274This can lead to severe file system fragmentation and we do not recommend
 275that you do this.
 276Really the only
 277.Xr tunefs 8
 278option worthwhile here is turning on
 279.Em softupdates
 280with
 281.Dq Li "tunefs -n enable /filesystem" .
 282(Note: in
 283.Fx 4.5
 284and later, softupdates can be turned on using the
 285.Fl U
 286option to
 287.Xr newfs 8 ,
 288and
 289.Xr sysinstall 8
 290will typically enable softupdates automatically for non-root file systems).
 291Softupdates drastically improves meta-data performance, mainly file
 292creation and deletion.
 293We recommend enabling softupdates on most file systems; however, there
 294are two limitations to softupdates that you should be aware of when
 295determining whether to use it on a file system.
 296First, softupdates guarantees file system consistency in the
 297case of a crash but could very easily be several seconds (even a minute!\&)
 298behind on pending write to the physical disk.
 299If you crash you may lose more work
 300than otherwise.
 301Secondly, softupdates delays the freeing of file system
 302blocks.
 303If you have a file system (such as the root file system) which is
 304close to full, doing a major update of it, e.g.\&
 305.Dq Li "make installworld" ,
 306can run it out of space and cause the update to fail.
 307For this reason, softupdates will not be enabled on the root file system
 308during a typical install.
 309There is no loss of performance since the root
 310file system is rarely written to.
 311.Pp
 312A number of run-time
 313.Xr mount 8
 314options exist that can help you tune the system.
 315The most obvious and most dangerous one is
 316.Cm async .
 317Only use this option in conjunction with
 318.Xr gjournal 8 ,
 319as it is far too dangerous on a normal file system.
 320A less dangerous and more
 321useful
 322.Xr mount 8
 323option is called
 324.Cm noatime .
 325.Ux
 326file systems normally update the last-accessed time of a file or
 327directory whenever it is accessed.
 328This operation is handled in
 329.Fx
 330with a delayed write and normally does not create a burden on the system.
 331However, if your system is accessing a huge number of files on a continuing
 332basis the buffer cache can wind up getting polluted with atime updates,
 333creating a burden on the system.
 334For example, if you are running a heavily
 335loaded web site, or a news server with lots of readers, you might want to
 336consider turning off atime updates on your larger partitions with this
 337.Xr mount 8
 338option.
 339However, you should not gratuitously turn off atime
 340updates everywhere.
 341For example, the
 342.Pa /var
 343file system customarily
 344holds mailboxes, and atime (in combination with mtime) is used to
 345determine whether a mailbox has new mail.
 346You might as well leave
 347atime turned on for mostly read-only partitions such as
 348.Pa /
 349and
 350.Pa /usr
 351as well.
 352This is especially useful for
 353.Pa /
 354since some system utilities
 355use the atime field for reporting.
 356.Sh STRIPING DISKS
 357In larger systems you can stripe partitions from several drives together
 358to create a much larger overall partition.
 359Striping can also improve
 360the performance of a file system by splitting I/O operations across two
 361or more disks.
 362The
 363.Xr gstripe 8 ,
 364.Xr gvinum 8 ,
 365and
 366.Xr ccdconfig 8
 367utilities may be used to create simple striped file systems.
 368Generally
 369speaking, striping smaller partitions such as the root and
 370.Pa /var/tmp ,
 371or essentially read-only partitions such as
 372.Pa /usr
 373is a complete waste of time.
 374You should only stripe partitions that require serious I/O performance,
 375typically
 376.Pa /var , /home ,
 377or custom partitions used to hold databases and web pages.
 378Choosing the proper stripe size is also
 379important.
 380File systems tend to store meta-data on power-of-2 boundaries
 381and you usually want to reduce seeking rather than increase seeking.
 382This
 383means you want to use a large off-center stripe size such as 1152 sectors
 384so sequential I/O does not seek both disks and so meta-data is distributed
 385across both disks rather than concentrated on a single disk.
 386If
 387you really need to get sophisticated, we recommend using a real hardware
 388RAID controller from the list of
 389.Fx
 390supported controllers.
 391.Sh SYSCTL TUNING
 392.Xr sysctl 8
 393variables permit system behavior to be monitored and controlled at
 394run-time.
 395Some sysctls simply report on the behavior of the system; others allow
 396the system behavior to be modified;
 397some may be set at boot time using
 398.Xr rc.conf 5 ,
 399but most will be set via
 400.Xr sysctl.conf 5 .
 401There are several hundred sysctls in the system, including many that appear
 402to be candidates for tuning but actually are not.
 403In this document we will only cover the ones that have the greatest effect
 404on the system.
 405.Pp
 406The
 407.Va vm.overcommit
 408sysctl defines the overcommit behaviour of the vm subsystem.
 409The virtual memory system always does accounting of the swap space
 410reservation, both total for system and per-user.
 411Corresponding values
 412are available through sysctl
 413.Va vm.swap_total ,
 414that gives the total bytes available for swapping, and
 415.Va vm.swap_reserved ,
 416that gives number of bytes that may be needed to back all currently
 417allocated anonymous memory.
 418.Pp
 419Setting bit 0 of the
 420.Va vm.overcommit
 421sysctl causes the virtual memory system to return failure
 422to the process when allocation of memory causes
 423.Va vm.swap_reserved
 424to exceed
 425.Va vm.swap_total .
 426Bit 1 of the sysctl enforces
 427.Dv RLIMIT_SWAP
 428limit
 429(see
 430.Xr getrlimit 2 ) .
 431Root is exempt from this limit.
 432Bit 2 allows to count most of the physical
 433memory as allocatable, except wired and free reserved pages
 434(accounted by
 435.Va vm.stats.vm.v_free_target
 436and
 437.Va vm.stats.vm.v_wire_count
 438sysctls, respectively).
 439.Pp
 440The
 441.Va kern.ipc.maxpipekva
 442loader tunable is used to set a hard limit on the
 443amount of kernel address space allocated to mapping of pipe buffers.
 444Use of the mapping allows the kernel to eliminate a copy of the
 445data from writer address space into the kernel, directly copying
 446the content of mapped buffer to the reader.
 447Increasing this value to a higher setting, such as `25165824' might
 448improve performance on systems where space for mapping pipe buffers
 449is quickly exhausted.
 450This exhaustion is not fatal; however, and it will only cause pipes
 451to fall back to using double-copy.
 452.Pp
 453The
 454.Va kern.ipc.shm_use_phys
 455sysctl defaults to 0 (off) and may be set to 0 (off) or 1 (on).
 456Setting
 457this parameter to 1 will cause all System V shared memory segments to be
 458mapped to unpageable physical RAM.
 459This feature only has an effect if you
 460are either (A) mapping small amounts of shared memory across many (hundreds)
 461of processes, or (B) mapping large amounts of shared memory across any
 462number of processes.
 463This feature allows the kernel to remove a great deal
 464of internal memory management page-tracking overhead at the cost of wiring
 465the shared memory into core, making it unswappable.
 466.Pp
 467The
 468.Va vfs.vmiodirenable
 469sysctl defaults to 1 (on).
 470This parameter controls how directories are cached
 471by the system.
 472Most directories are small and use but a single fragment
 473(typically 2K) in the file system and even less (typically 512 bytes) in
 474the buffer cache.
 475However, when operating in the default mode the buffer
 476cache will only cache a fixed number of directories even if you have a huge
 477amount of memory.
 478Turning on this sysctl allows the buffer cache to use
 479the VM Page Cache to cache the directories.
 480The advantage is that all of
 481memory is now available for caching directories.
 482The disadvantage is that
 483the minimum in-core memory used to cache a directory is the physical page
 484size (typically 4K) rather than 512 bytes.
 485We recommend turning this option off in memory-constrained environments;
 486however, when on, it will substantially improve the performance of services
 487that manipulate a large number of files.
 488Such services can include web caches, large mail systems, and news systems.
 489Turning on this option will generally not reduce performance even with the
 490wasted memory but you should experiment to find out.
 491.Pp
 492The
 493.Va vfs.write_behind
 494sysctl defaults to 1 (on).
 495This tells the file system to issue media
 496writes as full clusters are collected, which typically occurs when writing
 497large sequential files.
 498The idea is to avoid saturating the buffer
 499cache with dirty buffers when it would not benefit I/O performance.
 500However,
 501this may stall processes and under certain circumstances you may wish to turn
 502it off.
 503.Pp
 504The
 505.Va vfs.hirunningspace
 506sysctl determines how much outstanding write I/O may be queued to
 507disk controllers system-wide at any given time.
 508It is used by the UFS file system.
 509The default is self-tuned and
 510usually sufficient but on machines with advanced controllers and lots
 511of disks this may be tuned up to match what the controllers buffer.
 512Configuring this setting to match tagged queuing capabilities of
 513controllers or drives with average IO size used in production works
 514best (for example: 16 MiB will use 128 tags with IO requests of 128 KiB).
 515Note that setting too high a value
 516(exceeding the buffer cache's write threshold) can lead to extremely
 517bad clustering performance.
 518Do not set this value arbitrarily high!
 519Higher write queueing values may also add latency to reads occurring at
 520the same time.
 521.Pp
 522The
 523.Va vfs.read_max
 524sysctl governs VFS read-ahead and is expressed as the number of blocks
 525to pre-read if the heuristics algorithm decides that the reads are
 526issued sequentially.
 527It is used by the UFS, ext2fs and msdosfs file systems.
 528With the default UFS block size of 32 KiB, a setting of 64 will allow
 529speculatively reading up to 2 MiB.
 530This setting may be increased to get around disk I/O latencies, especially
 531where these latencies are large such as in virtual machine emulated
 532environments.
 533It may be tuned down in specific cases where the I/O load is such that
 534read-ahead adversely affects performance or where system memory is really
 535low.
 536.Pp
 537The
 538.Va vfs.ncsizefactor
 539sysctl defines how large VFS namecache may grow.
 540The number of currently allocated entries in namecache is provided by
 541.Va debug.numcache
 542sysctl and the condition
 543debug.numcache < kern.maxvnodes * vfs.ncsizefactor
 544is adhered to.
 545.Pp
 546The
 547.Va vfs.ncnegfactor
 548sysctl defines how many negative entries VFS namecache is allowed to create.
 549The number of currently allocated negative entries is provided by
 550.Va debug.numneg
 551sysctl and the condition
 552vfs.ncnegfactor * debug.numneg < debug.numcache
 553is adhered to.
 554.Pp
 555There are various other buffer-cache and VM page cache related sysctls.
 556We do not recommend modifying these values.
 557As of
 558.Fx 4.3 ,
 559the VM system does an extremely good job tuning itself.
 560.Pp
 561The
 562.Va net.inet.tcp.sendspace
 563and
 564.Va net.inet.tcp.recvspace
 565sysctls are of particular interest if you are running network intensive
 566applications.
 567They control the amount of send and receive buffer space
 568allowed for any given TCP connection.
 569The default sending buffer is 32K; the default receiving buffer
 570is 64K.
 571You can often
 572improve bandwidth utilization by increasing the default at the cost of
 573eating up more kernel memory for each connection.
 574We do not recommend
 575increasing the defaults if you are serving hundreds or thousands of
 576simultaneous connections because it is possible to quickly run the system
 577out of memory due to stalled connections building up.
 578But if you need
 579high bandwidth over a fewer number of connections, especially if you have
 580gigabit Ethernet, increasing these defaults can make a huge difference.
 581You can adjust the buffer size for incoming and outgoing data separately.
 582For example, if your machine is primarily doing web serving you may want
 583to decrease the recvspace in order to be able to increase the
 584sendspace without eating too much kernel memory.
 585Note that the routing table (see
 586.Xr route 8 )
 587can be used to introduce route-specific send and receive buffer size
 588defaults.
 589.Pp
 590As an additional management tool you can use pipes in your
 591firewall rules (see
 592.Xr ipfw 8 )
 593to limit the bandwidth going to or from particular IP blocks or ports.
 594For example, if you have a T1 you might want to limit your web traffic
 595to 70% of the T1's bandwidth in order to leave the remainder available
 596for mail and interactive use.
 597Normally a heavily loaded web server
 598will not introduce significant latencies into other services even if
 599the network link is maxed out, but enforcing a limit can smooth things
 600out and lead to longer term stability.
 601Many people also enforce artificial
 602bandwidth limitations in order to ensure that they are not charged for
 603using too much bandwidth.
 604.Pp
 605Setting the send or receive TCP buffer to values larger than 65535 will result
 606in a marginal performance improvement unless both hosts support the window
 607scaling extension of the TCP protocol, which is controlled by the
 608.Va net.inet.tcp.rfc1323
 609sysctl.
 610These extensions should be enabled and the TCP buffer size should be set
 611to a value larger than 65536 in order to obtain good performance from
 612certain types of network links; specifically, gigabit WAN links and
 613high-latency satellite links.
 614RFC1323 support is enabled by default.
 615.Pp
 616The
 617.Va net.inet.tcp.always_keepalive
 618sysctl determines whether or not the TCP implementation should attempt
 619to detect dead TCP connections by intermittently delivering
 620.Dq keepalives
 621on the connection.
 622By default, this is enabled for all applications; by setting this
 623sysctl to 0, only applications that specifically request keepalives
 624will use them.
 625In most environments, TCP keepalives will improve the management of
 626system state by expiring dead TCP connections, particularly for
 627systems serving dialup users who may not always terminate individual
 628TCP connections before disconnecting from the network.
 629However, in some environments, temporary network outages may be
 630incorrectly identified as dead sessions, resulting in unexpectedly
 631terminated TCP connections.
 632In such environments, setting the sysctl to 0 may reduce the occurrence of
 633TCP session disconnections.
 634.Pp
 635The
 636.Va net.inet.tcp.delayed_ack
 637TCP feature is largely misunderstood.
 638Historically speaking, this feature
 639was designed to allow the acknowledgement to transmitted data to be returned
 640along with the response.
 641For example, when you type over a remote shell,
 642the acknowledgement to the character you send can be returned along with the
 643data representing the echo of the character.
 644With delayed acks turned off,
 645the acknowledgement may be sent in its own packet, before the remote service
 646has a chance to echo the data it just received.
 647This same concept also
 648applies to any interactive protocol (e.g.\& SMTP, WWW, POP3), and can cut the
 649number of tiny packets flowing across the network in half.
 650The
 651.Fx
 652delayed ACK implementation also follows the TCP protocol rule that
 653at least every other packet be acknowledged even if the standard 100ms
 654timeout has not yet passed.
 655Normally the worst a delayed ACK can do is
 656slightly delay the teardown of a connection, or slightly delay the ramp-up
 657of a slow-start TCP connection.
 658While we are not sure we believe that
 659the several FAQs related to packages such as SAMBA and SQUID which advise
 660turning off delayed acks may be referring to the slow-start issue.
 661In
 662.Fx ,
 663it would be more beneficial to increase the slow-start flightsize via
 664the
 665.Va net.inet.tcp.slowstart_flightsize
 666sysctl rather than disable delayed acks.
 667.Pp
 668The
 669.Va net.inet.tcp.inflight.enable
 670sysctl turns on bandwidth delay product limiting for all TCP connections.
 671The system will attempt to calculate the bandwidth delay product for each
 672connection and limit the amount of data queued to the network to just the
 673amount required to maintain optimum throughput.
 674This feature is useful
 675if you are serving data over modems, GigE, or high speed WAN links (or
 676any other link with a high bandwidth*delay product), especially if you are
 677also using window scaling or have configured a large send window.
 678If you enable this option, you should also be sure to set
 679.Va net.inet.tcp.inflight.debug
 680to 0 (disable debugging), and for production use setting
 681.Va net.inet.tcp.inflight.min
 682to at least 6144 may be beneficial.
 683Note however, that setting high
 684minimums may effectively disable bandwidth limiting depending on the link.
 685The limiting feature reduces the amount of data built up in intermediate
 686router and switch packet queues as well as reduces the amount of data built
 687up in the local host's interface queue.
 688With fewer packets queued up,
 689interactive connections, especially over slow modems, will also be able
 690to operate with lower round trip times.
 691However, note that this feature
 692only affects data transmission (uploading / server-side).
 693It does not
 694affect data reception (downloading).
 695.Pp
 696Adjusting
 697.Va net.inet.tcp.inflight.stab
 698is not recommended.
 699This parameter defaults to 20, representing 2 maximal packets added
 700to the bandwidth delay product window calculation.
 701The additional
 702window is required to stabilize the algorithm and improve responsiveness
 703to changing conditions, but it can also result in higher ping times
 704over slow links (though still much lower than you would get without
 705the inflight algorithm).
 706In such cases you may
 707wish to try reducing this parameter to 15, 10, or 5, and you may also
 708have to reduce
 709.Va net.inet.tcp.inflight.min
 710(for example, to 3500) to get the desired effect.
 711Reducing these parameters
 712should be done as a last resort only.
 713.Pp
 714The
 715.Va net.inet.ip.portrange.*
 716sysctls control the port number ranges automatically bound to TCP and UDP
 717sockets.
 718There are three ranges: a low range, a default range, and a
 719high range, selectable via the
 720.Dv IP_PORTRANGE
 721.Xr setsockopt 2
 722call.
 723Most
 724network programs use the default range which is controlled by
 725.Va net.inet.ip.portrange.first
 726and
 727.Va net.inet.ip.portrange.last ,
 728which default to 49152 and 65535, respectively.
 729Bound port ranges are
 730used for outgoing connections, and it is possible to run the system out
 731of ports under certain circumstances.
 732This most commonly occurs when you are
 733running a heavily loaded web proxy.
 734The port range is not an issue
 735when running a server which handles mainly incoming connections, such as a
 736normal web server, or has a limited number of outgoing connections, such
 737as a mail relay.
 738For situations where you may run out of ports,
 739we recommend decreasing
 740.Va net.inet.ip.portrange.first
 741modestly.
 742A range of 10000 to 30000 ports may be reasonable.
 743You should also consider firewall effects when changing the port range.
 744Some firewalls
 745may block large ranges of ports (usually low-numbered ports) and expect systems
 746to use higher ranges of ports for outgoing connections.
 747By default
 748.Va net.inet.ip.portrange.last
 749is set at the maximum allowable port number.
 750.Pp
 751The
 752.Va kern.ipc.somaxconn
 753sysctl limits the size of the listen queue for accepting new TCP connections.
 754The default value of 128 is typically too low for robust handling of new
 755connections in a heavily loaded web server environment.
 756For such environments,
 757we recommend increasing this value to 1024 or higher.
 758The service daemon
 759may itself limit the listen queue size (e.g.\&
 760.Xr sendmail 8 ,
 761apache) but will
 762often have a directive in its configuration file to adjust the queue size up.
 763Larger listen queues also do a better job of fending off denial of service
 764attacks.
 765.Pp
 766The
 767.Va kern.maxfiles
 768sysctl determines how many open files the system supports.
 769The default is
 770typically a few thousand but you may need to bump this up to ten or twenty
 771thousand if you are running databases or large descriptor-heavy daemons.
 772The read-only
 773.Va kern.openfiles
 774sysctl may be interrogated to determine the current number of open files
 775on the system.
 776.Pp
 777The
 778.Va vm.swap_idle_enabled
 779sysctl is useful in large multi-user systems where you have lots of users
 780entering and leaving the system and lots of idle processes.
 781Such systems
 782tend to generate a great deal of continuous pressure on free memory reserves.
 783Turning this feature on and adjusting the swapout hysteresis (in idle
 784seconds) via
 785.Va vm.swap_idle_threshold1
 786and
 787.Va vm.swap_idle_threshold2
 788allows you to depress the priority of pages associated with idle processes
 789more quickly then the normal pageout algorithm.
 790This gives a helping hand
 791to the pageout daemon.
 792Do not turn this option on unless you need it,
 793because the tradeoff you are making is to essentially pre-page memory sooner
 794rather than later, eating more swap and disk bandwidth.
 795In a small system
 796this option will have a detrimental effect but in a large system that is
 797already doing moderate paging this option allows the VM system to stage
 798whole processes into and out of memory more easily.
 799.Sh LOADER TUNABLES
 800Some aspects of the system behavior may not be tunable at runtime because
 801memory allocations they perform must occur early in the boot process.
 802To change loader tunables, you must set their values in
 803.Xr loader.conf 5
 804and reboot the system.
 805.Pp
 806.Va kern.maxusers
 807controls the scaling of a number of static system tables, including defaults
 808for the maximum number of open files, sizing of network memory resources, etc.
 809As of
 810.Fx 4.5 ,
 811.Va kern.maxusers
 812is automatically sized at boot based on the amount of memory available in
 813the system, and may be determined at run-time by inspecting the value of the
 814read-only
 815.Va kern.maxusers
 816sysctl.
 817Some sites will require larger or smaller values of
 818.Va kern.maxusers
 819and may set it as a loader tunable; values of 64, 128, and 256 are not
 820uncommon.
 821We do not recommend going above 256 unless you need a huge number
 822of file descriptors; many of the tunable values set to their defaults by
 823.Va kern.maxusers
 824may be individually overridden at boot-time or run-time as described
 825elsewhere in this document.
 826Systems older than
 827.Fx 4.4
 828must set this value via the kernel
 829.Xr config 8
 830option
 831.Cd maxusers
 832instead.
 833.Pp
 834The
 835.Va kern.dfldsiz
 836and
 837.Va kern.dflssiz
 838tunables set the default soft limits for process data and stack size
 839respectively.
 840Processes may increase these up to the hard limits by calling
 841.Xr setrlimit 2 .
 842The
 843.Va kern.maxdsiz ,
 844.Va kern.maxssiz ,
 845and
 846.Va kern.maxtsiz
 847tunables set the hard limits for process data, stack, and text size
 848respectively; processes may not exceed these limits.
 849The
 850.Va kern.sgrowsiz
 851tunable controls how much the stack segment will grow when a process
 852needs to allocate more stack.
 853.Pp
 854.Va kern.ipc.nmbclusters
 855may be adjusted to increase the number of network mbufs the system is
 856willing to allocate.
 857Each cluster represents approximately 2K of memory,
 858so a value of 1024 represents 2M of kernel memory reserved for network
 859buffers.
 860You can do a simple calculation to figure out how many you need.
 861If you have a web server which maxes out at 1000 simultaneous connections,
 862and each connection eats a 16K receive and 16K send buffer, you need
 863approximately 32MB worth of network buffers to deal with it.
 864A good rule of
 865thumb is to multiply by 2, so 32MBx2 = 64MB/2K = 32768.
 866So for this case
 867you would want to set
 868.Va kern.ipc.nmbclusters
 869to 32768.
 870We recommend values between
 8711024 and 4096 for machines with moderates amount of memory, and between 4096
 872and 32768 for machines with greater amounts of memory.
 873Under no circumstances
 874should you specify an arbitrarily high value for this parameter, it could
 875lead to a boot-time crash.
 876The
 877.Fl m
 878option to
 879.Xr netstat 1
 880may be used to observe network cluster use.
 881Older versions of
 882.Fx
 883do not have this tunable and require that the
 884kernel
 885.Xr config 8
 886option
 887.Dv NMBCLUSTERS
 888be set instead.
 889.Pp
 890More and more programs are using the
 891.Xr sendfile 2
 892system call to transmit files over the network.
 893The
 894.Va kern.ipc.nsfbufs
 895sysctl controls the number of file system buffers
 896.Xr sendfile 2
 897is allowed to use to perform its work.
 898This parameter nominally scales
 899with
 900.Va kern.maxusers
 901so you should not need to modify this parameter except under extreme
 902circumstances.
 903See the
 904.Sx TUNING
 905section in the
 906.Xr sendfile 2
 907manual page for details.
 908.Sh KERNEL CONFIG TUNING
 909There are a number of kernel options that you may have to fiddle with in
 910a large-scale system.
 911In order to change these options you need to be
 912able to compile a new kernel from source.
 913The
 914.Xr config 8
 915manual page and the handbook are good starting points for learning how to
 916do this.
 917Generally the first thing you do when creating your own custom
 918kernel is to strip out all the drivers and services you do not use.
 919Removing things like
 920.Dv INET6
 921and drivers you do not have will reduce the size of your kernel, sometimes
 922by a megabyte or more, leaving more memory available for applications.
 923.Pp
 924.Dv SCSI_DELAY
 925may be used to reduce system boot times.
 926The defaults are fairly high and
 927can be responsible for 5+ seconds of delay in the boot process.
 928Reducing
 929.Dv SCSI_DELAY
 930to something below 5 seconds could work (especially with modern drives).
 931.Pp
 932There are a number of
 933.Dv *_CPU
 934options that can be commented out.
 935If you only want the kernel to run
 936on a Pentium class CPU, you can easily remove
 937.Dv I486_CPU ,
 938but only remove
 939.Dv I586_CPU
 940if you are sure your CPU is being recognized as a Pentium II or better.
 941Some clones may be recognized as a Pentium or even a 486 and not be able
 942to boot without those options.
 943If it works, great!
 944The operating system
 945will be able to better use higher-end CPU features for MMU, task switching,
 946timebase, and even device operations.
 947Additionally, higher-end CPUs support
 9484MB MMU pages, which the kernel uses to map the kernel itself into memory,
 949increasing its efficiency under heavy syscall loads.
 950.Sh IDE WRITE CACHING
 951.Fx 4.3
 952flirted with turning off IDE write caching.
 953This reduced write bandwidth
 954to IDE disks but was considered necessary due to serious data consistency
 955issues introduced by hard drive vendors.
 956Basically the problem is that
 957IDE drives lie about when a write completes.
 958With IDE write caching turned
 959on, IDE hard drives will not only write data to disk out of order, they
 960will sometimes delay some of the blocks indefinitely under heavy disk
 961load.
 962A crash or power failure can result in serious file system
 963corruption.
 964So our default was changed to be safe.
 965Unfortunately, the
 966result was such a huge loss in performance that we caved in and changed the
 967default back to on after the release.
 968You should check the default on
 969your system by observing the
 970.Va hw.ata.wc
 971sysctl variable.
 972If IDE write caching is turned off, you can turn it back
 973on by setting the
 974.Va hw.ata.wc
 975loader tunable to 1.
 976More information on tuning the ATA driver system may be found in the
 977.Xr ata 4
 978manual page.
 979If you need performance, go with SCSI.
 980.Sh CPU, MEMORY, DISK, NETWORK
 981The type of tuning you do depends heavily on where your system begins to
 982bottleneck as load increases.
 983If your system runs out of CPU (idle times
 984are perpetually 0%) then you need to consider upgrading the CPU or moving to
 985an SMP motherboard (multiple CPU's), or perhaps you need to revisit the
 986programs that are causing the load and try to optimize them.
 987If your system
 988is paging to swap a lot you need to consider adding more memory.
 989If your
 990system is saturating the disk you typically see high CPU idle times and
 991total disk saturation.
 992.Xr systat 1
 993can be used to monitor this.
 994There are many solutions to saturated disks:
 995increasing memory for caching, mirroring disks, distributing operations across
 996several machines, and so forth.
 997If disk performance is an issue and you
 998are using IDE drives, switching to SCSI can help a great deal.
 999While modern
1000IDE drives compare with SCSI in raw sequential bandwidth, the moment you
1001start seeking around the disk SCSI drives usually win.
1002.Pp
1003Finally, you might run out of network suds.
1004The first line of defense for
1005improving network performance is to make sure you are using switches instead
1006of hubs, especially these days where switches are almost as cheap.
1007Hubs
1008have severe problems under heavy loads due to collision back-off and one bad
1009host can severely degrade the entire LAN.
1010Second, optimize the network path
1011as much as possible.
1012For example, in
1013.Xr firewall 7
1014we describe a firewall protecting internal hosts with a topology where
1015the externally visible hosts are not routed through it.
1016Use 100BaseT rather
1017than 10BaseT, or use 1000BaseT rather than 100BaseT, depending on your needs.
1018Most bottlenecks occur at the WAN link (e.g.\&
1019modem, T1, DSL, whatever).
1020If expanding the link is not an option it may be possible to use the
1021.Xr dummynet 4
1022feature to implement peak shaving or other forms of traffic shaping to
1023prevent the overloaded service (such as web services) from affecting other
1024services (such as email), or vice versa.
1025In home installations this could
1026be used to give interactive traffic (your browser,
1027.Xr ssh 1
1028logins) priority
1029over services you export from your box (web services, email).
1030.Sh SEE ALSO
1031.Xr netstat 1 ,
1032.Xr systat 1 ,
1033.Xr sendfile 2 ,
1034.Xr ata 4 ,
1035.Xr dummynet 4 ,
1036.Xr login.conf 5 ,
1037.Xr rc.conf 5 ,
1038.Xr sysctl.conf 5 ,
1039.Xr firewall 7 ,
1040.Xr eventtimers 7 ,
1041.Xr hier 7 ,
1042.Xr ports 7 ,
1043.Xr boot 8 ,
1044.Xr bsdlabel 8 ,
1045.Xr ccdconfig 8 ,
1046.Xr config 8 ,
1047.Xr fsck 8 ,
1048.Xr gjournal 8 ,
1049.Xr gstripe 8 ,
1050.Xr gvinum 8 ,
1051.Xr ifconfig 8 ,
1052.Xr ipfw 8 ,
1053.Xr loader 8 ,
1054.Xr mount 8 ,
1055.Xr newfs 8 ,
1056.Xr route 8 ,
1057.Xr sysctl 8 ,
1058.Xr sysinstall 8 ,
1059.Xr tunefs 8
1060.Sh HISTORY
1061The
1062.Nm
1063manual page was originally written by
1064.An Matthew Dillon
1065and first appeared
1066in
1067.Fx 4.3 ,
1068May 2001.