Unknown | 1068 lines | 1068 code | 0 blank | 0 comment | 0 complexity | f01bf4b94ee88f69ec9317bd678c42d2 MD5 | raw file
1.\" Copyright (C) 2001 Matthew Dillon. All rights reserved. 2.\" 3.\" Redistribution and use in source and binary forms, with or without 4.\" modification, are permitted provided that the following conditions 5.\" are met: 6.\" 1. Redistributions of source code must retain the above copyright 7.\" notice, this list of conditions and the following disclaimer. 8.\" 2. Redistributions in binary form must reproduce the above copyright 9.\" notice, this list of conditions and the following disclaimer in the 10.\" documentation and/or other materials provided with the distribution. 11.\" 12.\" THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND 13.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 14.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 15.\" ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE 16.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 17.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 18.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 19.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 20.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY 21.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF 22.\" SUCH DAMAGE. 23.\" 24.\" $FreeBSD$ 25.\" 26.Dd May 11, 2012 27.Dt TUNING 7 28.Os 29.Sh NAME 30.Nm tuning 31.Nd performance tuning under FreeBSD 32.Sh SYSTEM SETUP - DISKLABEL, NEWFS, TUNEFS, SWAP 33When using 34.Xr bsdlabel 8 35or 36.Xr sysinstall 8 37to lay out your file systems on a hard disk it is important to remember 38that hard drives can transfer data much more quickly from outer tracks 39than they can from inner tracks. 40To take advantage of this you should 41try to pack your smaller file systems and swap closer to the outer tracks, 42follow with the larger file systems, and end with the largest file systems. 43It is also important to size system standard file systems such that you 44will not be forced to resize them later as you scale the machine up. 45I usually create, in order, a 128M root, 1G swap, 128M 46.Pa /var , 47128M 48.Pa /var/tmp , 493G 50.Pa /usr , 51and use any remaining space for 52.Pa /home . 53.Pp 54You should typically size your swap space to approximately 2x main memory 55for systems with less than 2GB of RAM, or approximately 1x main memory 56if you have more. 57If you do not have a lot of RAM, though, you will generally want a lot 58more swap. 59It is not recommended that you configure any less than 60256M of swap on a system and you should keep in mind future memory 61expansion when sizing the swap partition. 62The kernel's VM paging algorithms are tuned to perform best when there is 63at least 2x swap versus main memory. 64Configuring too little swap can lead 65to inefficiencies in the VM page scanning code as well as create issues 66later on if you add more memory to your machine. 67Finally, on larger systems 68with multiple SCSI disks (or multiple IDE disks operating on different 69controllers), we strongly recommend that you configure swap on each drive. 70The swap partitions on the drives should be approximately the same size. 71The kernel can handle arbitrary sizes but 72internal data structures scale to 4 times the largest swap partition. 73Keeping 74the swap partitions near the same size will allow the kernel to optimally 75stripe swap space across the N disks. 76Do not worry about overdoing it a 77little, swap space is the saving grace of 78.Ux 79and even if you do not normally use much swap, it can give you more time to 80recover from a runaway program before being forced to reboot. 81.Pp 82How you size your 83.Pa /var 84partition depends heavily on what you intend to use the machine for. 85This 86partition is primarily used to hold mailboxes, the print spool, and log 87files. 88Some people even make 89.Pa /var/log 90its own partition (but except for extreme cases it is not worth the waste 91of a partition ID). 92If your machine is intended to act as a mail 93or print server, 94or you are running a heavily visited web server, you should consider 95creating a much larger partition \(en perhaps a gig or more. 96It is very easy 97to underestimate log file storage requirements. 98.Pp 99Sizing 100.Pa /var/tmp 101depends on the kind of temporary file usage you think you will need. 102128M is 103the minimum we recommend. 104Also note that sysinstall will create a 105.Pa /tmp 106directory. 107Dedicating a partition for temporary file storage is important for 108two reasons: first, it reduces the possibility of file system corruption 109in a crash, and second it reduces the chance of a runaway process that 110fills up 111.Oo Pa /var Oc Ns Pa /tmp 112from blowing up more critical subsystems (mail, 113logging, etc). 114Filling up 115.Oo Pa /var Oc Ns Pa /tmp 116is a very common problem to have. 117.Pp 118In the old days there were differences between 119.Pa /tmp 120and 121.Pa /var/tmp , 122but the introduction of 123.Pa /var 124(and 125.Pa /var/tmp ) 126led to massive confusion 127by program writers so today programs haphazardly use one or the 128other and thus no real distinction can be made between the two. 129So it makes sense to have just one temporary directory and 130softlink to it from the other 131.Pa tmp 132directory locations. 133However you handle 134.Pa /tmp , 135the one thing you do not want to do is leave it sitting 136on the root partition where it might cause root to fill up or possibly 137corrupt root in a crash/reboot situation. 138.Pp 139The 140.Pa /usr 141partition holds the bulk of the files required to support the system and 142a subdirectory within it called 143.Pa /usr/local 144holds the bulk of the files installed from the 145.Xr ports 7 146hierarchy. 147If you do not use ports all that much and do not intend to keep 148system source 149.Pq Pa /usr/src 150on the machine, you can get away with 151a 1 GB 152.Pa /usr 153partition. 154However, if you install a lot of ports 155(especially window managers and Linux-emulated binaries), we recommend 156at least a 2 GB 157.Pa /usr 158and if you also intend to keep system source 159on the machine, we recommend a 3 GB 160.Pa /usr . 161Do not underestimate the 162amount of space you will need in this partition, it can creep up and 163surprise you! 164.Pp 165The 166.Pa /home 167partition is typically used to hold user-specific data. 168I usually size it to the remainder of the disk. 169.Pp 170Why partition at all? 171Why not create one big 172.Pa / 173partition and be done with it? 174Then I do not have to worry about undersizing things! 175Well, there are several reasons this is not a good idea. 176First, 177each partition has different operational characteristics and separating them 178allows the file system to tune itself to those characteristics. 179For example, 180the root and 181.Pa /usr 182partitions are read-mostly, with very little writing, while 183a lot of reading and writing could occur in 184.Pa /var 185and 186.Pa /var/tmp . 187By properly 188partitioning your system fragmentation introduced in the smaller more 189heavily write-loaded partitions will not bleed over into the mostly-read 190partitions. 191Additionally, keeping the write-loaded partitions closer to 192the edge of the disk (i.e., before the really big partitions instead of after 193in the partition table) will increase I/O performance in the partitions 194where you need it the most. 195Now it is true that you might also need I/O 196performance in the larger partitions, but they are so large that shifting 197them more towards the edge of the disk will not lead to a significant 198performance improvement whereas moving 199.Pa /var 200to the edge can have a huge impact. 201Finally, there are safety concerns. 202Having a small neat root partition that 203is essentially read-only gives it a greater chance of surviving a bad crash 204intact. 205.Pp 206Properly partitioning your system also allows you to tune 207.Xr newfs 8 , 208and 209.Xr tunefs 8 210parameters. 211Tuning 212.Xr newfs 8 213requires more experience but can lead to significant improvements in 214performance. 215There are three parameters that are relatively safe to tune: 216.Em blocksize , bytes/i-node , 217and 218.Em cylinders/group . 219.Pp 220.Fx 221performs best when using 16K or 32K file system block sizes. 222The default file system block size is 32K, 223which provides best performance for most applications, 224with the exception of those that perform random access on large files 225(such as database server software). 226Such applications tend to perform better with a smaller block size, 227although modern disk characteristics are such that the performance 228gain from using a smaller block size may not be worth consideration. 229Using a block size larger than 32K 230can cause fragmentation of the buffer cache and 231lead to lower performance. 232.Pp 233The defaults may be unsuitable 234for a file system that requires a very large number of i-nodes 235or is intended to hold a large number of very small files. 236Such a file system should be created with an 4K, 8K, or 16K block size. 237This also requires you to specify a smaller 238fragment size. 239We recommend always using a fragment size that is 1/8 240the block size (less testing has been done on other fragment size factors). 241The 242.Xr newfs 8 243options for this would be 244.Dq Li "newfs -f 1024 -b 8192 ..." . 245.Pp 246If a large partition is intended to be used to hold fewer, larger files, such 247as database files, you can increase the 248.Em bytes/i-node 249ratio which reduces the number of i-nodes (maximum number of files and 250directories that can be created) for that partition. 251Decreasing the number 252of i-nodes in a file system can greatly reduce 253.Xr fsck 8 254recovery times after a crash. 255Do not use this option 256unless you are actually storing large files on the partition, because if you 257overcompensate you can wind up with a file system that has lots of free 258space remaining but cannot accommodate any more files. 259Using 65536, 131072, or 262144 bytes/i-node is recommended. 260You can go higher but 261it will have only incremental effects on 262.Xr fsck 8 263recovery times. 264For example, 265.Dq Li "newfs -i 65536 ..." . 266.Pp 267.Xr tunefs 8 268may be used to further tune a file system. 269This command can be run in 270single-user mode without having to reformat the file system. 271However, this is possibly the most abused program in the system. 272Many people attempt to 273increase available file system space by setting the min-free percentage to 0. 274This can lead to severe file system fragmentation and we do not recommend 275that you do this. 276Really the only 277.Xr tunefs 8 278option worthwhile here is turning on 279.Em softupdates 280with 281.Dq Li "tunefs -n enable /filesystem" . 282(Note: in 283.Fx 4.5 284and later, softupdates can be turned on using the 285.Fl U 286option to 287.Xr newfs 8 , 288and 289.Xr sysinstall 8 290will typically enable softupdates automatically for non-root file systems). 291Softupdates drastically improves meta-data performance, mainly file 292creation and deletion. 293We recommend enabling softupdates on most file systems; however, there 294are two limitations to softupdates that you should be aware of when 295determining whether to use it on a file system. 296First, softupdates guarantees file system consistency in the 297case of a crash but could very easily be several seconds (even a minute!\&) 298behind on pending write to the physical disk. 299If you crash you may lose more work 300than otherwise. 301Secondly, softupdates delays the freeing of file system 302blocks. 303If you have a file system (such as the root file system) which is 304close to full, doing a major update of it, e.g.\& 305.Dq Li "make installworld" , 306can run it out of space and cause the update to fail. 307For this reason, softupdates will not be enabled on the root file system 308during a typical install. 309There is no loss of performance since the root 310file system is rarely written to. 311.Pp 312A number of run-time 313.Xr mount 8 314options exist that can help you tune the system. 315The most obvious and most dangerous one is 316.Cm async . 317Only use this option in conjunction with 318.Xr gjournal 8 , 319as it is far too dangerous on a normal file system. 320A less dangerous and more 321useful 322.Xr mount 8 323option is called 324.Cm noatime . 325.Ux 326file systems normally update the last-accessed time of a file or 327directory whenever it is accessed. 328This operation is handled in 329.Fx 330with a delayed write and normally does not create a burden on the system. 331However, if your system is accessing a huge number of files on a continuing 332basis the buffer cache can wind up getting polluted with atime updates, 333creating a burden on the system. 334For example, if you are running a heavily 335loaded web site, or a news server with lots of readers, you might want to 336consider turning off atime updates on your larger partitions with this 337.Xr mount 8 338option. 339However, you should not gratuitously turn off atime 340updates everywhere. 341For example, the 342.Pa /var 343file system customarily 344holds mailboxes, and atime (in combination with mtime) is used to 345determine whether a mailbox has new mail. 346You might as well leave 347atime turned on for mostly read-only partitions such as 348.Pa / 349and 350.Pa /usr 351as well. 352This is especially useful for 353.Pa / 354since some system utilities 355use the atime field for reporting. 356.Sh STRIPING DISKS 357In larger systems you can stripe partitions from several drives together 358to create a much larger overall partition. 359Striping can also improve 360the performance of a file system by splitting I/O operations across two 361or more disks. 362The 363.Xr gstripe 8 , 364.Xr gvinum 8 , 365and 366.Xr ccdconfig 8 367utilities may be used to create simple striped file systems. 368Generally 369speaking, striping smaller partitions such as the root and 370.Pa /var/tmp , 371or essentially read-only partitions such as 372.Pa /usr 373is a complete waste of time. 374You should only stripe partitions that require serious I/O performance, 375typically 376.Pa /var , /home , 377or custom partitions used to hold databases and web pages. 378Choosing the proper stripe size is also 379important. 380File systems tend to store meta-data on power-of-2 boundaries 381and you usually want to reduce seeking rather than increase seeking. 382This 383means you want to use a large off-center stripe size such as 1152 sectors 384so sequential I/O does not seek both disks and so meta-data is distributed 385across both disks rather than concentrated on a single disk. 386If 387you really need to get sophisticated, we recommend using a real hardware 388RAID controller from the list of 389.Fx 390supported controllers. 391.Sh SYSCTL TUNING 392.Xr sysctl 8 393variables permit system behavior to be monitored and controlled at 394run-time. 395Some sysctls simply report on the behavior of the system; others allow 396the system behavior to be modified; 397some may be set at boot time using 398.Xr rc.conf 5 , 399but most will be set via 400.Xr sysctl.conf 5 . 401There are several hundred sysctls in the system, including many that appear 402to be candidates for tuning but actually are not. 403In this document we will only cover the ones that have the greatest effect 404on the system. 405.Pp 406The 407.Va vm.overcommit 408sysctl defines the overcommit behaviour of the vm subsystem. 409The virtual memory system always does accounting of the swap space 410reservation, both total for system and per-user. 411Corresponding values 412are available through sysctl 413.Va vm.swap_total , 414that gives the total bytes available for swapping, and 415.Va vm.swap_reserved , 416that gives number of bytes that may be needed to back all currently 417allocated anonymous memory. 418.Pp 419Setting bit 0 of the 420.Va vm.overcommit 421sysctl causes the virtual memory system to return failure 422to the process when allocation of memory causes 423.Va vm.swap_reserved 424to exceed 425.Va vm.swap_total . 426Bit 1 of the sysctl enforces 427.Dv RLIMIT_SWAP 428limit 429(see 430.Xr getrlimit 2 ) . 431Root is exempt from this limit. 432Bit 2 allows to count most of the physical 433memory as allocatable, except wired and free reserved pages 434(accounted by 435.Va vm.stats.vm.v_free_target 436and 437.Va vm.stats.vm.v_wire_count 438sysctls, respectively). 439.Pp 440The 441.Va kern.ipc.maxpipekva 442loader tunable is used to set a hard limit on the 443amount of kernel address space allocated to mapping of pipe buffers. 444Use of the mapping allows the kernel to eliminate a copy of the 445data from writer address space into the kernel, directly copying 446the content of mapped buffer to the reader. 447Increasing this value to a higher setting, such as `25165824' might 448improve performance on systems where space for mapping pipe buffers 449is quickly exhausted. 450This exhaustion is not fatal; however, and it will only cause pipes 451to fall back to using double-copy. 452.Pp 453The 454.Va kern.ipc.shm_use_phys 455sysctl defaults to 0 (off) and may be set to 0 (off) or 1 (on). 456Setting 457this parameter to 1 will cause all System V shared memory segments to be 458mapped to unpageable physical RAM. 459This feature only has an effect if you 460are either (A) mapping small amounts of shared memory across many (hundreds) 461of processes, or (B) mapping large amounts of shared memory across any 462number of processes. 463This feature allows the kernel to remove a great deal 464of internal memory management page-tracking overhead at the cost of wiring 465the shared memory into core, making it unswappable. 466.Pp 467The 468.Va vfs.vmiodirenable 469sysctl defaults to 1 (on). 470This parameter controls how directories are cached 471by the system. 472Most directories are small and use but a single fragment 473(typically 2K) in the file system and even less (typically 512 bytes) in 474the buffer cache. 475However, when operating in the default mode the buffer 476cache will only cache a fixed number of directories even if you have a huge 477amount of memory. 478Turning on this sysctl allows the buffer cache to use 479the VM Page Cache to cache the directories. 480The advantage is that all of 481memory is now available for caching directories. 482The disadvantage is that 483the minimum in-core memory used to cache a directory is the physical page 484size (typically 4K) rather than 512 bytes. 485We recommend turning this option off in memory-constrained environments; 486however, when on, it will substantially improve the performance of services 487that manipulate a large number of files. 488Such services can include web caches, large mail systems, and news systems. 489Turning on this option will generally not reduce performance even with the 490wasted memory but you should experiment to find out. 491.Pp 492The 493.Va vfs.write_behind 494sysctl defaults to 1 (on). 495This tells the file system to issue media 496writes as full clusters are collected, which typically occurs when writing 497large sequential files. 498The idea is to avoid saturating the buffer 499cache with dirty buffers when it would not benefit I/O performance. 500However, 501this may stall processes and under certain circumstances you may wish to turn 502it off. 503.Pp 504The 505.Va vfs.hirunningspace 506sysctl determines how much outstanding write I/O may be queued to 507disk controllers system-wide at any given time. 508It is used by the UFS file system. 509The default is self-tuned and 510usually sufficient but on machines with advanced controllers and lots 511of disks this may be tuned up to match what the controllers buffer. 512Configuring this setting to match tagged queuing capabilities of 513controllers or drives with average IO size used in production works 514best (for example: 16 MiB will use 128 tags with IO requests of 128 KiB). 515Note that setting too high a value 516(exceeding the buffer cache's write threshold) can lead to extremely 517bad clustering performance. 518Do not set this value arbitrarily high! 519Higher write queueing values may also add latency to reads occurring at 520the same time. 521.Pp 522The 523.Va vfs.read_max 524sysctl governs VFS read-ahead and is expressed as the number of blocks 525to pre-read if the heuristics algorithm decides that the reads are 526issued sequentially. 527It is used by the UFS, ext2fs and msdosfs file systems. 528With the default UFS block size of 32 KiB, a setting of 64 will allow 529speculatively reading up to 2 MiB. 530This setting may be increased to get around disk I/O latencies, especially 531where these latencies are large such as in virtual machine emulated 532environments. 533It may be tuned down in specific cases where the I/O load is such that 534read-ahead adversely affects performance or where system memory is really 535low. 536.Pp 537The 538.Va vfs.ncsizefactor 539sysctl defines how large VFS namecache may grow. 540The number of currently allocated entries in namecache is provided by 541.Va debug.numcache 542sysctl and the condition 543debug.numcache < kern.maxvnodes * vfs.ncsizefactor 544is adhered to. 545.Pp 546The 547.Va vfs.ncnegfactor 548sysctl defines how many negative entries VFS namecache is allowed to create. 549The number of currently allocated negative entries is provided by 550.Va debug.numneg 551sysctl and the condition 552vfs.ncnegfactor * debug.numneg < debug.numcache 553is adhered to. 554.Pp 555There are various other buffer-cache and VM page cache related sysctls. 556We do not recommend modifying these values. 557As of 558.Fx 4.3 , 559the VM system does an extremely good job tuning itself. 560.Pp 561The 562.Va net.inet.tcp.sendspace 563and 564.Va net.inet.tcp.recvspace 565sysctls are of particular interest if you are running network intensive 566applications. 567They control the amount of send and receive buffer space 568allowed for any given TCP connection. 569The default sending buffer is 32K; the default receiving buffer 570is 64K. 571You can often 572improve bandwidth utilization by increasing the default at the cost of 573eating up more kernel memory for each connection. 574We do not recommend 575increasing the defaults if you are serving hundreds or thousands of 576simultaneous connections because it is possible to quickly run the system 577out of memory due to stalled connections building up. 578But if you need 579high bandwidth over a fewer number of connections, especially if you have 580gigabit Ethernet, increasing these defaults can make a huge difference. 581You can adjust the buffer size for incoming and outgoing data separately. 582For example, if your machine is primarily doing web serving you may want 583to decrease the recvspace in order to be able to increase the 584sendspace without eating too much kernel memory. 585Note that the routing table (see 586.Xr route 8 ) 587can be used to introduce route-specific send and receive buffer size 588defaults. 589.Pp 590As an additional management tool you can use pipes in your 591firewall rules (see 592.Xr ipfw 8 ) 593to limit the bandwidth going to or from particular IP blocks or ports. 594For example, if you have a T1 you might want to limit your web traffic 595to 70% of the T1's bandwidth in order to leave the remainder available 596for mail and interactive use. 597Normally a heavily loaded web server 598will not introduce significant latencies into other services even if 599the network link is maxed out, but enforcing a limit can smooth things 600out and lead to longer term stability. 601Many people also enforce artificial 602bandwidth limitations in order to ensure that they are not charged for 603using too much bandwidth. 604.Pp 605Setting the send or receive TCP buffer to values larger than 65535 will result 606in a marginal performance improvement unless both hosts support the window 607scaling extension of the TCP protocol, which is controlled by the 608.Va net.inet.tcp.rfc1323 609sysctl. 610These extensions should be enabled and the TCP buffer size should be set 611to a value larger than 65536 in order to obtain good performance from 612certain types of network links; specifically, gigabit WAN links and 613high-latency satellite links. 614RFC1323 support is enabled by default. 615.Pp 616The 617.Va net.inet.tcp.always_keepalive 618sysctl determines whether or not the TCP implementation should attempt 619to detect dead TCP connections by intermittently delivering 620.Dq keepalives 621on the connection. 622By default, this is enabled for all applications; by setting this 623sysctl to 0, only applications that specifically request keepalives 624will use them. 625In most environments, TCP keepalives will improve the management of 626system state by expiring dead TCP connections, particularly for 627systems serving dialup users who may not always terminate individual 628TCP connections before disconnecting from the network. 629However, in some environments, temporary network outages may be 630incorrectly identified as dead sessions, resulting in unexpectedly 631terminated TCP connections. 632In such environments, setting the sysctl to 0 may reduce the occurrence of 633TCP session disconnections. 634.Pp 635The 636.Va net.inet.tcp.delayed_ack 637TCP feature is largely misunderstood. 638Historically speaking, this feature 639was designed to allow the acknowledgement to transmitted data to be returned 640along with the response. 641For example, when you type over a remote shell, 642the acknowledgement to the character you send can be returned along with the 643data representing the echo of the character. 644With delayed acks turned off, 645the acknowledgement may be sent in its own packet, before the remote service 646has a chance to echo the data it just received. 647This same concept also 648applies to any interactive protocol (e.g.\& SMTP, WWW, POP3), and can cut the 649number of tiny packets flowing across the network in half. 650The 651.Fx 652delayed ACK implementation also follows the TCP protocol rule that 653at least every other packet be acknowledged even if the standard 100ms 654timeout has not yet passed. 655Normally the worst a delayed ACK can do is 656slightly delay the teardown of a connection, or slightly delay the ramp-up 657of a slow-start TCP connection. 658While we are not sure we believe that 659the several FAQs related to packages such as SAMBA and SQUID which advise 660turning off delayed acks may be referring to the slow-start issue. 661In 662.Fx , 663it would be more beneficial to increase the slow-start flightsize via 664the 665.Va net.inet.tcp.slowstart_flightsize 666sysctl rather than disable delayed acks. 667.Pp 668The 669.Va net.inet.tcp.inflight.enable 670sysctl turns on bandwidth delay product limiting for all TCP connections. 671The system will attempt to calculate the bandwidth delay product for each 672connection and limit the amount of data queued to the network to just the 673amount required to maintain optimum throughput. 674This feature is useful 675if you are serving data over modems, GigE, or high speed WAN links (or 676any other link with a high bandwidth*delay product), especially if you are 677also using window scaling or have configured a large send window. 678If you enable this option, you should also be sure to set 679.Va net.inet.tcp.inflight.debug 680to 0 (disable debugging), and for production use setting 681.Va net.inet.tcp.inflight.min 682to at least 6144 may be beneficial. 683Note however, that setting high 684minimums may effectively disable bandwidth limiting depending on the link. 685The limiting feature reduces the amount of data built up in intermediate 686router and switch packet queues as well as reduces the amount of data built 687up in the local host's interface queue. 688With fewer packets queued up, 689interactive connections, especially over slow modems, will also be able 690to operate with lower round trip times. 691However, note that this feature 692only affects data transmission (uploading / server-side). 693It does not 694affect data reception (downloading). 695.Pp 696Adjusting 697.Va net.inet.tcp.inflight.stab 698is not recommended. 699This parameter defaults to 20, representing 2 maximal packets added 700to the bandwidth delay product window calculation. 701The additional 702window is required to stabilize the algorithm and improve responsiveness 703to changing conditions, but it can also result in higher ping times 704over slow links (though still much lower than you would get without 705the inflight algorithm). 706In such cases you may 707wish to try reducing this parameter to 15, 10, or 5, and you may also 708have to reduce 709.Va net.inet.tcp.inflight.min 710(for example, to 3500) to get the desired effect. 711Reducing these parameters 712should be done as a last resort only. 713.Pp 714The 715.Va net.inet.ip.portrange.* 716sysctls control the port number ranges automatically bound to TCP and UDP 717sockets. 718There are three ranges: a low range, a default range, and a 719high range, selectable via the 720.Dv IP_PORTRANGE 721.Xr setsockopt 2 722call. 723Most 724network programs use the default range which is controlled by 725.Va net.inet.ip.portrange.first 726and 727.Va net.inet.ip.portrange.last , 728which default to 49152 and 65535, respectively. 729Bound port ranges are 730used for outgoing connections, and it is possible to run the system out 731of ports under certain circumstances. 732This most commonly occurs when you are 733running a heavily loaded web proxy. 734The port range is not an issue 735when running a server which handles mainly incoming connections, such as a 736normal web server, or has a limited number of outgoing connections, such 737as a mail relay. 738For situations where you may run out of ports, 739we recommend decreasing 740.Va net.inet.ip.portrange.first 741modestly. 742A range of 10000 to 30000 ports may be reasonable. 743You should also consider firewall effects when changing the port range. 744Some firewalls 745may block large ranges of ports (usually low-numbered ports) and expect systems 746to use higher ranges of ports for outgoing connections. 747By default 748.Va net.inet.ip.portrange.last 749is set at the maximum allowable port number. 750.Pp 751The 752.Va kern.ipc.somaxconn 753sysctl limits the size of the listen queue for accepting new TCP connections. 754The default value of 128 is typically too low for robust handling of new 755connections in a heavily loaded web server environment. 756For such environments, 757we recommend increasing this value to 1024 or higher. 758The service daemon 759may itself limit the listen queue size (e.g.\& 760.Xr sendmail 8 , 761apache) but will 762often have a directive in its configuration file to adjust the queue size up. 763Larger listen queues also do a better job of fending off denial of service 764attacks. 765.Pp 766The 767.Va kern.maxfiles 768sysctl determines how many open files the system supports. 769The default is 770typically a few thousand but you may need to bump this up to ten or twenty 771thousand if you are running databases or large descriptor-heavy daemons. 772The read-only 773.Va kern.openfiles 774sysctl may be interrogated to determine the current number of open files 775on the system. 776.Pp 777The 778.Va vm.swap_idle_enabled 779sysctl is useful in large multi-user systems where you have lots of users 780entering and leaving the system and lots of idle processes. 781Such systems 782tend to generate a great deal of continuous pressure on free memory reserves. 783Turning this feature on and adjusting the swapout hysteresis (in idle 784seconds) via 785.Va vm.swap_idle_threshold1 786and 787.Va vm.swap_idle_threshold2 788allows you to depress the priority of pages associated with idle processes 789more quickly then the normal pageout algorithm. 790This gives a helping hand 791to the pageout daemon. 792Do not turn this option on unless you need it, 793because the tradeoff you are making is to essentially pre-page memory sooner 794rather than later, eating more swap and disk bandwidth. 795In a small system 796this option will have a detrimental effect but in a large system that is 797already doing moderate paging this option allows the VM system to stage 798whole processes into and out of memory more easily. 799.Sh LOADER TUNABLES 800Some aspects of the system behavior may not be tunable at runtime because 801memory allocations they perform must occur early in the boot process. 802To change loader tunables, you must set their values in 803.Xr loader.conf 5 804and reboot the system. 805.Pp 806.Va kern.maxusers 807controls the scaling of a number of static system tables, including defaults 808for the maximum number of open files, sizing of network memory resources, etc. 809As of 810.Fx 4.5 , 811.Va kern.maxusers 812is automatically sized at boot based on the amount of memory available in 813the system, and may be determined at run-time by inspecting the value of the 814read-only 815.Va kern.maxusers 816sysctl. 817Some sites will require larger or smaller values of 818.Va kern.maxusers 819and may set it as a loader tunable; values of 64, 128, and 256 are not 820uncommon. 821We do not recommend going above 256 unless you need a huge number 822of file descriptors; many of the tunable values set to their defaults by 823.Va kern.maxusers 824may be individually overridden at boot-time or run-time as described 825elsewhere in this document. 826Systems older than 827.Fx 4.4 828must set this value via the kernel 829.Xr config 8 830option 831.Cd maxusers 832instead. 833.Pp 834The 835.Va kern.dfldsiz 836and 837.Va kern.dflssiz 838tunables set the default soft limits for process data and stack size 839respectively. 840Processes may increase these up to the hard limits by calling 841.Xr setrlimit 2 . 842The 843.Va kern.maxdsiz , 844.Va kern.maxssiz , 845and 846.Va kern.maxtsiz 847tunables set the hard limits for process data, stack, and text size 848respectively; processes may not exceed these limits. 849The 850.Va kern.sgrowsiz 851tunable controls how much the stack segment will grow when a process 852needs to allocate more stack. 853.Pp 854.Va kern.ipc.nmbclusters 855may be adjusted to increase the number of network mbufs the system is 856willing to allocate. 857Each cluster represents approximately 2K of memory, 858so a value of 1024 represents 2M of kernel memory reserved for network 859buffers. 860You can do a simple calculation to figure out how many you need. 861If you have a web server which maxes out at 1000 simultaneous connections, 862and each connection eats a 16K receive and 16K send buffer, you need 863approximately 32MB worth of network buffers to deal with it. 864A good rule of 865thumb is to multiply by 2, so 32MBx2 = 64MB/2K = 32768. 866So for this case 867you would want to set 868.Va kern.ipc.nmbclusters 869to 32768. 870We recommend values between 8711024 and 4096 for machines with moderates amount of memory, and between 4096 872and 32768 for machines with greater amounts of memory. 873Under no circumstances 874should you specify an arbitrarily high value for this parameter, it could 875lead to a boot-time crash. 876The 877.Fl m 878option to 879.Xr netstat 1 880may be used to observe network cluster use. 881Older versions of 882.Fx 883do not have this tunable and require that the 884kernel 885.Xr config 8 886option 887.Dv NMBCLUSTERS 888be set instead. 889.Pp 890More and more programs are using the 891.Xr sendfile 2 892system call to transmit files over the network. 893The 894.Va kern.ipc.nsfbufs 895sysctl controls the number of file system buffers 896.Xr sendfile 2 897is allowed to use to perform its work. 898This parameter nominally scales 899with 900.Va kern.maxusers 901so you should not need to modify this parameter except under extreme 902circumstances. 903See the 904.Sx TUNING 905section in the 906.Xr sendfile 2 907manual page for details. 908.Sh KERNEL CONFIG TUNING 909There are a number of kernel options that you may have to fiddle with in 910a large-scale system. 911In order to change these options you need to be 912able to compile a new kernel from source. 913The 914.Xr config 8 915manual page and the handbook are good starting points for learning how to 916do this. 917Generally the first thing you do when creating your own custom 918kernel is to strip out all the drivers and services you do not use. 919Removing things like 920.Dv INET6 921and drivers you do not have will reduce the size of your kernel, sometimes 922by a megabyte or more, leaving more memory available for applications. 923.Pp 924.Dv SCSI_DELAY 925may be used to reduce system boot times. 926The defaults are fairly high and 927can be responsible for 5+ seconds of delay in the boot process. 928Reducing 929.Dv SCSI_DELAY 930to something below 5 seconds could work (especially with modern drives). 931.Pp 932There are a number of 933.Dv *_CPU 934options that can be commented out. 935If you only want the kernel to run 936on a Pentium class CPU, you can easily remove 937.Dv I486_CPU , 938but only remove 939.Dv I586_CPU 940if you are sure your CPU is being recognized as a Pentium II or better. 941Some clones may be recognized as a Pentium or even a 486 and not be able 942to boot without those options. 943If it works, great! 944The operating system 945will be able to better use higher-end CPU features for MMU, task switching, 946timebase, and even device operations. 947Additionally, higher-end CPUs support 9484MB MMU pages, which the kernel uses to map the kernel itself into memory, 949increasing its efficiency under heavy syscall loads. 950.Sh IDE WRITE CACHING 951.Fx 4.3 952flirted with turning off IDE write caching. 953This reduced write bandwidth 954to IDE disks but was considered necessary due to serious data consistency 955issues introduced by hard drive vendors. 956Basically the problem is that 957IDE drives lie about when a write completes. 958With IDE write caching turned 959on, IDE hard drives will not only write data to disk out of order, they 960will sometimes delay some of the blocks indefinitely under heavy disk 961load. 962A crash or power failure can result in serious file system 963corruption. 964So our default was changed to be safe. 965Unfortunately, the 966result was such a huge loss in performance that we caved in and changed the 967default back to on after the release. 968You should check the default on 969your system by observing the 970.Va hw.ata.wc 971sysctl variable. 972If IDE write caching is turned off, you can turn it back 973on by setting the 974.Va hw.ata.wc 975loader tunable to 1. 976More information on tuning the ATA driver system may be found in the 977.Xr ata 4 978manual page. 979If you need performance, go with SCSI. 980.Sh CPU, MEMORY, DISK, NETWORK 981The type of tuning you do depends heavily on where your system begins to 982bottleneck as load increases. 983If your system runs out of CPU (idle times 984are perpetually 0%) then you need to consider upgrading the CPU or moving to 985an SMP motherboard (multiple CPU's), or perhaps you need to revisit the 986programs that are causing the load and try to optimize them. 987If your system 988is paging to swap a lot you need to consider adding more memory. 989If your 990system is saturating the disk you typically see high CPU idle times and 991total disk saturation. 992.Xr systat 1 993can be used to monitor this. 994There are many solutions to saturated disks: 995increasing memory for caching, mirroring disks, distributing operations across 996several machines, and so forth. 997If disk performance is an issue and you 998are using IDE drives, switching to SCSI can help a great deal. 999While modern 1000IDE drives compare with SCSI in raw sequential bandwidth, the moment you 1001start seeking around the disk SCSI drives usually win. 1002.Pp 1003Finally, you might run out of network suds. 1004The first line of defense for 1005improving network performance is to make sure you are using switches instead 1006of hubs, especially these days where switches are almost as cheap. 1007Hubs 1008have severe problems under heavy loads due to collision back-off and one bad 1009host can severely degrade the entire LAN. 1010Second, optimize the network path 1011as much as possible. 1012For example, in 1013.Xr firewall 7 1014we describe a firewall protecting internal hosts with a topology where 1015the externally visible hosts are not routed through it. 1016Use 100BaseT rather 1017than 10BaseT, or use 1000BaseT rather than 100BaseT, depending on your needs. 1018Most bottlenecks occur at the WAN link (e.g.\& 1019modem, T1, DSL, whatever). 1020If expanding the link is not an option it may be possible to use the 1021.Xr dummynet 4 1022feature to implement peak shaving or other forms of traffic shaping to 1023prevent the overloaded service (such as web services) from affecting other 1024services (such as email), or vice versa. 1025In home installations this could 1026be used to give interactive traffic (your browser, 1027.Xr ssh 1 1028logins) priority 1029over services you export from your box (web services, email). 1030.Sh SEE ALSO 1031.Xr netstat 1 , 1032.Xr systat 1 , 1033.Xr sendfile 2 , 1034.Xr ata 4 , 1035.Xr dummynet 4 , 1036.Xr login.conf 5 , 1037.Xr rc.conf 5 , 1038.Xr sysctl.conf 5 , 1039.Xr firewall 7 , 1040.Xr eventtimers 7 , 1041.Xr hier 7 , 1042.Xr ports 7 , 1043.Xr boot 8 , 1044.Xr bsdlabel 8 , 1045.Xr ccdconfig 8 , 1046.Xr config 8 , 1047.Xr fsck 8 , 1048.Xr gjournal 8 , 1049.Xr gstripe 8 , 1050.Xr gvinum 8 , 1051.Xr ifconfig 8 , 1052.Xr ipfw 8 , 1053.Xr loader 8 , 1054.Xr mount 8 , 1055.Xr newfs 8 , 1056.Xr route 8 , 1057.Xr sysctl 8 , 1058.Xr sysinstall 8 , 1059.Xr tunefs 8 1060.Sh HISTORY 1061The 1062.Nm 1063manual page was originally written by 1064.An Matthew Dillon 1065and first appeared 1066in 1067.Fx 4.3 , 1068May 2001.