/Doc/library/heapq.rst

http://unladen-swallow.googlecode.com/ · ReStructuredText · 241 lines · 176 code · 65 blank · 0 comment · 0 complexity · ce5647b6966b26e3b8d521d95ffa379e MD5 · raw file

  1. :mod:`heapq` --- Heap queue algorithm
  2. =====================================
  3. .. module:: heapq
  4. :synopsis: Heap queue algorithm (a.k.a. priority queue).
  5. .. moduleauthor:: Kevin O'Connor
  6. .. sectionauthor:: Guido van Rossum <guido@python.org>
  7. .. sectionauthor:: Franรงois Pinard
  8. .. versionadded:: 2.3
  9. This module provides an implementation of the heap queue algorithm, also known
  10. as the priority queue algorithm.
  11. Heaps are arrays for which ``heap[k] <= heap[2*k+1]`` and ``heap[k] <=
  12. heap[2*k+2]`` for all *k*, counting elements from zero. For the sake of
  13. comparison, non-existing elements are considered to be infinite. The
  14. interesting property of a heap is that ``heap[0]`` is always its smallest
  15. element.
  16. The API below differs from textbook heap algorithms in two aspects: (a) We use
  17. zero-based indexing. This makes the relationship between the index for a node
  18. and the indexes for its children slightly less obvious, but is more suitable
  19. since Python uses zero-based indexing. (b) Our pop method returns the smallest
  20. item, not the largest (called a "min heap" in textbooks; a "max heap" is more
  21. common in texts because of its suitability for in-place sorting).
  22. These two make it possible to view the heap as a regular Python list without
  23. surprises: ``heap[0]`` is the smallest item, and ``heap.sort()`` maintains the
  24. heap invariant!
  25. To create a heap, use a list initialized to ``[]``, or you can transform a
  26. populated list into a heap via function :func:`heapify`.
  27. The following functions are provided:
  28. .. function:: heappush(heap, item)
  29. Push the value *item* onto the *heap*, maintaining the heap invariant.
  30. .. function:: heappop(heap)
  31. Pop and return the smallest item from the *heap*, maintaining the heap
  32. invariant. If the heap is empty, :exc:`IndexError` is raised.
  33. .. function:: heappushpop(heap, item)
  34. Push *item* on the heap, then pop and return the smallest item from the
  35. *heap*. The combined action runs more efficiently than :func:`heappush`
  36. followed by a separate call to :func:`heappop`.
  37. .. versionadded:: 2.6
  38. .. function:: heapify(x)
  39. Transform list *x* into a heap, in-place, in linear time.
  40. .. function:: heapreplace(heap, item)
  41. Pop and return the smallest item from the *heap*, and also push the new *item*.
  42. The heap size doesn't change. If the heap is empty, :exc:`IndexError` is raised.
  43. This is more efficient than :func:`heappop` followed by :func:`heappush`, and
  44. can be more appropriate when using a fixed-size heap. Note that the value
  45. returned may be larger than *item*! That constrains reasonable uses of this
  46. routine unless written as part of a conditional replacement::
  47. if item > heap[0]:
  48. item = heapreplace(heap, item)
  49. Example of use:
  50. >>> from heapq import heappush, heappop
  51. >>> heap = []
  52. >>> data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0]
  53. >>> for item in data:
  54. ... heappush(heap, item)
  55. ...
  56. >>> ordered = []
  57. >>> while heap:
  58. ... ordered.append(heappop(heap))
  59. ...
  60. >>> print ordered
  61. [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
  62. >>> data.sort()
  63. >>> print data == ordered
  64. True
  65. Using a heap to insert items at the correct place in a priority queue:
  66. >>> heap = []
  67. >>> data = [(1, 'J'), (4, 'N'), (3, 'H'), (2, 'O')]
  68. >>> for item in data:
  69. ... heappush(heap, item)
  70. ...
  71. >>> while heap:
  72. ... print heappop(heap)[1]
  73. J
  74. O
  75. H
  76. N
  77. The module also offers three general purpose functions based on heaps.
  78. .. function:: merge(*iterables)
  79. Merge multiple sorted inputs into a single sorted output (for example, merge
  80. timestamped entries from multiple log files). Returns an :term:`iterator`
  81. over the sorted values.
  82. Similar to ``sorted(itertools.chain(*iterables))`` but returns an iterable, does
  83. not pull the data into memory all at once, and assumes that each of the input
  84. streams is already sorted (smallest to largest).
  85. .. versionadded:: 2.6
  86. .. function:: nlargest(n, iterable[, key])
  87. Return a list with the *n* largest elements from the dataset defined by
  88. *iterable*. *key*, if provided, specifies a function of one argument that is
  89. used to extract a comparison key from each element in the iterable:
  90. ``key=str.lower`` Equivalent to: ``sorted(iterable, key=key,
  91. reverse=True)[:n]``
  92. .. versionadded:: 2.4
  93. .. versionchanged:: 2.5
  94. Added the optional *key* argument.
  95. .. function:: nsmallest(n, iterable[, key])
  96. Return a list with the *n* smallest elements from the dataset defined by
  97. *iterable*. *key*, if provided, specifies a function of one argument that is
  98. used to extract a comparison key from each element in the iterable:
  99. ``key=str.lower`` Equivalent to: ``sorted(iterable, key=key)[:n]``
  100. .. versionadded:: 2.4
  101. .. versionchanged:: 2.5
  102. Added the optional *key* argument.
  103. The latter two functions perform best for smaller values of *n*. For larger
  104. values, it is more efficient to use the :func:`sorted` function. Also, when
  105. ``n==1``, it is more efficient to use the builtin :func:`min` and :func:`max`
  106. functions.
  107. Theory
  108. ------
  109. (This explanation is due to Franรงois Pinard. The Python code for this module
  110. was contributed by Kevin O'Connor.)
  111. Heaps are arrays for which ``a[k] <= a[2*k+1]`` and ``a[k] <= a[2*k+2]`` for all
  112. *k*, counting elements from 0. For the sake of comparison, non-existing
  113. elements are considered to be infinite. The interesting property of a heap is
  114. that ``a[0]`` is always its smallest element.
  115. The strange invariant above is meant to be an efficient memory representation
  116. for a tournament. The numbers below are *k*, not ``a[k]``::
  117. 0
  118. 1 2
  119. 3 4 5 6
  120. 7 8 9 10 11 12 13 14
  121. 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
  122. In the tree above, each cell *k* is topping ``2*k+1`` and ``2*k+2``. In an usual
  123. binary tournament we see in sports, each cell is the winner over the two cells
  124. it tops, and we can trace the winner down the tree to see all opponents s/he
  125. had. However, in many computer applications of such tournaments, we do not need
  126. to trace the history of a winner. To be more memory efficient, when a winner is
  127. promoted, we try to replace it by something else at a lower level, and the rule
  128. becomes that a cell and the two cells it tops contain three different items, but
  129. the top cell "wins" over the two topped cells.
  130. If this heap invariant is protected at all time, index 0 is clearly the overall
  131. winner. The simplest algorithmic way to remove it and find the "next" winner is
  132. to move some loser (let's say cell 30 in the diagram above) into the 0 position,
  133. and then percolate this new 0 down the tree, exchanging values, until the
  134. invariant is re-established. This is clearly logarithmic on the total number of
  135. items in the tree. By iterating over all items, you get an O(n log n) sort.
  136. A nice feature of this sort is that you can efficiently insert new items while
  137. the sort is going on, provided that the inserted items are not "better" than the
  138. last 0'th element you extracted. This is especially useful in simulation
  139. contexts, where the tree holds all incoming events, and the "win" condition
  140. means the smallest scheduled time. When an event schedule other events for
  141. execution, they are scheduled into the future, so they can easily go into the
  142. heap. So, a heap is a good structure for implementing schedulers (this is what
  143. I used for my MIDI sequencer :-).
  144. Various structures for implementing schedulers have been extensively studied,
  145. and heaps are good for this, as they are reasonably speedy, the speed is almost
  146. constant, and the worst case is not much different than the average case.
  147. However, there are other representations which are more efficient overall, yet
  148. the worst cases might be terrible.
  149. Heaps are also very useful in big disk sorts. You most probably all know that a
  150. big sort implies producing "runs" (which are pre-sorted sequences, which size is
  151. usually related to the amount of CPU memory), followed by a merging passes for
  152. these runs, which merging is often very cleverly organised [#]_. It is very
  153. important that the initial sort produces the longest runs possible. Tournaments
  154. are a good way to that. If, using all the memory available to hold a
  155. tournament, you replace and percolate items that happen to fit the current run,
  156. you'll produce runs which are twice the size of the memory for random input, and
  157. much better for input fuzzily ordered.
  158. Moreover, if you output the 0'th item on disk and get an input which may not fit
  159. in the current tournament (because the value "wins" over the last output value),
  160. it cannot fit in the heap, so the size of the heap decreases. The freed memory
  161. could be cleverly reused immediately for progressively building a second heap,
  162. which grows at exactly the same rate the first heap is melting. When the first
  163. heap completely vanishes, you switch heaps and start a new run. Clever and
  164. quite effective!
  165. In a word, heaps are useful memory structures to know. I use them in a few
  166. applications, and I think it is good to keep a 'heap' module around. :-)
  167. .. rubric:: Footnotes
  168. .. [#] The disk balancing algorithms which are current, nowadays, are more annoying
  169. than clever, and this is a consequence of the seeking capabilities of the disks.
  170. On devices which cannot seek, like big tape drives, the story was quite
  171. different, and one had to be very clever to ensure (far in advance) that each
  172. tape movement will be the most effective possible (that is, will best
  173. participate at "progressing" the merge). Some tapes were even able to read
  174. backwards, and this was also used to avoid the rewinding time. Believe me, real
  175. good tape sorts were quite spectacular to watch! From all times, sorting has
  176. always been a Great Art! :-)