#### /Doc/library/heapq.rst

ReStructuredText | 241 lines | 176 code | 65 blank | 0 comment | 0 complexity | ce5647b6966b26e3b8d521d95ffa379e MD5 | raw file

1:mod:`heapq` --- Heap queue algorithm 2===================================== 3 4.. module:: heapq 5 :synopsis: Heap queue algorithm (a.k.a. priority queue). 6.. moduleauthor:: Kevin O'Connor 7.. sectionauthor:: Guido van Rossum <guido@python.org> 8.. sectionauthor:: Franรงois Pinard 9 10.. versionadded:: 2.3 11 12This module provides an implementation of the heap queue algorithm, also known 13as the priority queue algorithm. 14 15Heaps are arrays for which ``heap[k] <= heap[2*k+1]`` and ``heap[k] <= 16heap[2*k+2]`` for all *k*, counting elements from zero. For the sake of 17comparison, non-existing elements are considered to be infinite. The 18interesting property of a heap is that ``heap[0]`` is always its smallest 19element. 20 21The API below differs from textbook heap algorithms in two aspects: (a) We use 22zero-based indexing. This makes the relationship between the index for a node 23and the indexes for its children slightly less obvious, but is more suitable 24since Python uses zero-based indexing. (b) Our pop method returns the smallest 25item, not the largest (called a "min heap" in textbooks; a "max heap" is more 26common in texts because of its suitability for in-place sorting). 27 28These two make it possible to view the heap as a regular Python list without 29surprises: ``heap[0]`` is the smallest item, and ``heap.sort()`` maintains the 30heap invariant! 31 32To create a heap, use a list initialized to ``[]``, or you can transform a 33populated list into a heap via function :func:`heapify`. 34 35The following functions are provided: 36 37 38.. function:: heappush(heap, item) 39 40 Push the value *item* onto the *heap*, maintaining the heap invariant. 41 42 43.. function:: heappop(heap) 44 45 Pop and return the smallest item from the *heap*, maintaining the heap 46 invariant. If the heap is empty, :exc:`IndexError` is raised. 47 48.. function:: heappushpop(heap, item) 49 50 Push *item* on the heap, then pop and return the smallest item from the 51 *heap*. The combined action runs more efficiently than :func:`heappush` 52 followed by a separate call to :func:`heappop`. 53 54 .. versionadded:: 2.6 55 56.. function:: heapify(x) 57 58 Transform list *x* into a heap, in-place, in linear time. 59 60 61.. function:: heapreplace(heap, item) 62 63 Pop and return the smallest item from the *heap*, and also push the new *item*. 64 The heap size doesn't change. If the heap is empty, :exc:`IndexError` is raised. 65 This is more efficient than :func:`heappop` followed by :func:`heappush`, and 66 can be more appropriate when using a fixed-size heap. Note that the value 67 returned may be larger than *item*! That constrains reasonable uses of this 68 routine unless written as part of a conditional replacement:: 69 70 if item > heap[0]: 71 item = heapreplace(heap, item) 72 73Example of use: 74 75 >>> from heapq import heappush, heappop 76 >>> heap = [] 77 >>> data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0] 78 >>> for item in data: 79 ... heappush(heap, item) 80 ... 81 >>> ordered = [] 82 >>> while heap: 83 ... ordered.append(heappop(heap)) 84 ... 85 >>> print ordered 86 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] 87 >>> data.sort() 88 >>> print data == ordered 89 True 90 91Using a heap to insert items at the correct place in a priority queue: 92 93 >>> heap = [] 94 >>> data = [(1, 'J'), (4, 'N'), (3, 'H'), (2, 'O')] 95 >>> for item in data: 96 ... heappush(heap, item) 97 ... 98 >>> while heap: 99 ... print heappop(heap)[1] 100 J 101 O 102 H 103 N 104 105 106The module also offers three general purpose functions based on heaps. 107 108 109.. function:: merge(*iterables) 110 111 Merge multiple sorted inputs into a single sorted output (for example, merge 112 timestamped entries from multiple log files). Returns an :term:`iterator` 113 over the sorted values. 114 115 Similar to ``sorted(itertools.chain(*iterables))`` but returns an iterable, does 116 not pull the data into memory all at once, and assumes that each of the input 117 streams is already sorted (smallest to largest). 118 119 .. versionadded:: 2.6 120 121 122.. function:: nlargest(n, iterable[, key]) 123 124 Return a list with the *n* largest elements from the dataset defined by 125 *iterable*. *key*, if provided, specifies a function of one argument that is 126 used to extract a comparison key from each element in the iterable: 127 ``key=str.lower`` Equivalent to: ``sorted(iterable, key=key, 128 reverse=True)[:n]`` 129 130 .. versionadded:: 2.4 131 132 .. versionchanged:: 2.5 133 Added the optional *key* argument. 134 135 136.. function:: nsmallest(n, iterable[, key]) 137 138 Return a list with the *n* smallest elements from the dataset defined by 139 *iterable*. *key*, if provided, specifies a function of one argument that is 140 used to extract a comparison key from each element in the iterable: 141 ``key=str.lower`` Equivalent to: ``sorted(iterable, key=key)[:n]`` 142 143 .. versionadded:: 2.4 144 145 .. versionchanged:: 2.5 146 Added the optional *key* argument. 147 148The latter two functions perform best for smaller values of *n*. For larger 149values, it is more efficient to use the :func:`sorted` function. Also, when 150``n==1``, it is more efficient to use the builtin :func:`min` and :func:`max` 151functions. 152 153 154Theory 155------ 156 157(This explanation is due to Franรงois Pinard. The Python code for this module 158was contributed by Kevin O'Connor.) 159 160Heaps are arrays for which ``a[k] <= a[2*k+1]`` and ``a[k] <= a[2*k+2]`` for all 161*k*, counting elements from 0. For the sake of comparison, non-existing 162elements are considered to be infinite. The interesting property of a heap is 163that ``a[0]`` is always its smallest element. 164 165The strange invariant above is meant to be an efficient memory representation 166for a tournament. The numbers below are *k*, not ``a[k]``:: 167 168 0 169 170 1 2 171 172 3 4 5 6 173 174 7 8 9 10 11 12 13 14 175 176 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 177 178In the tree above, each cell *k* is topping ``2*k+1`` and ``2*k+2``. In an usual 179binary tournament we see in sports, each cell is the winner over the two cells 180it tops, and we can trace the winner down the tree to see all opponents s/he 181had. However, in many computer applications of such tournaments, we do not need 182to trace the history of a winner. To be more memory efficient, when a winner is 183promoted, we try to replace it by something else at a lower level, and the rule 184becomes that a cell and the two cells it tops contain three different items, but 185the top cell "wins" over the two topped cells. 186 187If this heap invariant is protected at all time, index 0 is clearly the overall 188winner. The simplest algorithmic way to remove it and find the "next" winner is 189to move some loser (let's say cell 30 in the diagram above) into the 0 position, 190and then percolate this new 0 down the tree, exchanging values, until the 191invariant is re-established. This is clearly logarithmic on the total number of 192items in the tree. By iterating over all items, you get an O(n log n) sort. 193 194A nice feature of this sort is that you can efficiently insert new items while 195the sort is going on, provided that the inserted items are not "better" than the 196last 0'th element you extracted. This is especially useful in simulation 197contexts, where the tree holds all incoming events, and the "win" condition 198means the smallest scheduled time. When an event schedule other events for 199execution, they are scheduled into the future, so they can easily go into the 200heap. So, a heap is a good structure for implementing schedulers (this is what 201I used for my MIDI sequencer :-). 202 203Various structures for implementing schedulers have been extensively studied, 204and heaps are good for this, as they are reasonably speedy, the speed is almost 205constant, and the worst case is not much different than the average case. 206However, there are other representations which are more efficient overall, yet 207the worst cases might be terrible. 208 209Heaps are also very useful in big disk sorts. You most probably all know that a 210big sort implies producing "runs" (which are pre-sorted sequences, which size is 211usually related to the amount of CPU memory), followed by a merging passes for 212these runs, which merging is often very cleverly organised [#]_. It is very 213important that the initial sort produces the longest runs possible. Tournaments 214are a good way to that. If, using all the memory available to hold a 215tournament, you replace and percolate items that happen to fit the current run, 216you'll produce runs which are twice the size of the memory for random input, and 217much better for input fuzzily ordered. 218 219Moreover, if you output the 0'th item on disk and get an input which may not fit 220in the current tournament (because the value "wins" over the last output value), 221it cannot fit in the heap, so the size of the heap decreases. The freed memory 222could be cleverly reused immediately for progressively building a second heap, 223which grows at exactly the same rate the first heap is melting. When the first 224heap completely vanishes, you switch heaps and start a new run. Clever and 225quite effective! 226 227In a word, heaps are useful memory structures to know. I use them in a few 228applications, and I think it is good to keep a 'heap' module around. :-) 229 230.. rubric:: Footnotes 231 232.. [#] The disk balancing algorithms which are current, nowadays, are more annoying 233 than clever, and this is a consequence of the seeking capabilities of the disks. 234 On devices which cannot seek, like big tape drives, the story was quite 235 different, and one had to be very clever to ensure (far in advance) that each 236 tape movement will be the most effective possible (that is, will best 237 participate at "progressing" the merge). Some tapes were even able to read 238 backwards, and this was also used to avoid the rewinding time. Believe me, real 239 good tape sorts were quite spectacular to watch! From all times, sorting has 240 always been a Great Art! :-) 241