PageRenderTime 64ms CodeModel.GetById 18ms app.highlight 37ms RepoModel.GetById 1ms app.codeStats 1ms

Python | 515 lines | 509 code | 0 blank | 6 comment | 0 complexity | a3166ae32e7134f513c7025107e81a44 MD5 | raw file
  2Performance Data Helper (PDH) Query Classes
  4Wrapper classes for end-users and high-level access to the PDH query
  5mechanisms.  PDH is a win32-specific mechanism for accessing the
  6performance data made available by the system.  The Python for Windows
  7PDH module does not implement the "Registry" interface, implementing
  8the more straightforward Query-based mechanism.
 10The basic idea of a PDH Query is an object which can query the system
 11about the status of any number of "counters."  The counters are paths
 12to a particular piece of performance data.  For instance, the path 
 13'\\Memory\\Available Bytes' describes just about exactly what it says
 14it does, the amount of free memory on the default computer expressed 
 15in Bytes.  These paths can be considerably more complex than this, 
 16but part of the point of this wrapper module is to hide that
 17complexity from the end-user/programmer.
 19EXAMPLE: A more complex Path
 20	'\\\\RAISTLIN\\PhysicalDisk(_Total)\\Avg. Disk Bytes/Read'
 21	Raistlin --> Computer Name
 22	PhysicalDisk --> Object Name
 23	_Total --> The particular Instance (in this case, all instances, i.e. all drives)
 24	Avg. Disk Bytes/Read --> The piece of data being monitored.
 26EXAMPLE: Collecting Data with a Query
 27	As an example, the following code implements a logger which allows the
 28	user to choose what counters they would like to log, and logs those
 29	counters for 30 seconds, at two-second intervals.
 31	query = Query()
 32	query.addcounterbybrowsing()
 33	query.collectdatafor(30,2)
 35	The data is now stored in a list of lists as:
 36	query.curresults
 38	The counters(paths) which were used to collect the data are:
 39	query.curpaths
 41	You can use the win32pdh.ParseCounterPath(path) utility function
 42	to turn the paths into more easily read values for your task, or
 43	write the data to a file, or do whatever you want with it.
 46	query.collectdatawhile(period) # start a logging thread for collecting data
 47	query.collectdatawhile_stop() # signal the logging thread to stop logging
 48	query.collectdata() # run the query only once
 49	query.addperfcounter(object, counter, machine=None) # add a standard performance counter
 50	query.addinstcounter(object, counter,machine=None,objtype = 'Process',volatile=1,format = win32pdh.PDH_FMT_LONG) # add a possibly volatile counter
 52### Known bugs and limitations ###
 53Due to a problem with threading under the PythonWin interpreter, there
 54will be no data logged if the PythonWin window is not the foreground
 55application.  Workaround: scripts using threading should be run in the
 56python.exe interpreter.
 58The volatile-counter handlers are possibly buggy, they haven't been
 59tested to any extent.  The wrapper Query makes it safe to pass invalid
 60paths (a -1 will be returned, or the Query will be totally ignored,
 61depending on the missing element), so you should be able to work around
 62the error by including all possible paths and filtering out the -1's.
 64There is no way I know of to stop a thread which is currently sleeping,
 65so you have to wait until the thread in collectdatawhile is activated
 66again.  This might become a problem in situations where the collection
 67period is multiple minutes (or hours, or whatever).
 69Should make the win32pdh.ParseCounter function available to the Query
 70classes as a method or something similar, so that it can be accessed
 71by programmes that have just picked up an instance from somewhere.
 73Should explicitly mention where QueryErrors can be raised, and create a
 74full test set to see if there are any uncaught win32api.error's still
 75hanging around.
 77When using the python.exe interpreter, the addcounterbybrowsing-
 78generated browser window is often hidden behind other windows.  No known
 79workaround other than Alt-tabing to reach the browser window.
 81### Other References ###
 82The win32pdhutil module (which should be in the %pythonroot%/win32/lib 
 83directory) provides quick-and-dirty utilities for one-off access to
 84variables from the PDH.  Almost everything in that module can be done
 85with a Query object, but it provides task-oriented functions for a
 86number of common one-off tasks.
 88If you can access the MS Developers Network Library, you can find
 89information about the PDH API as MS describes it.  For a background article,
 93The reference guide for the PDH API was last spotted at:
 97In general the Python version of the API is just a wrapper around the
 98Query-based version of this API (as far as I can see), so you can learn what
 99you need to from there.  From what I understand, the MSDN Online 
100resources are available for the price of signing up for them.  I can't
101guarantee how long that's supposed to last. (Or anything for that
105The eventual plan is for my (Mike Fletcher's) Starship account to include
106a section on NT Administration, and the Query is the first project
107in this plan.  There should be an article describing the creation of
108a simple logger there, but the example above is 90% of the work of
109that project, so don't sweat it if you don't find anything there.
110(currently the account hasn't been set up).
113If you need to contact me immediately, (why I can't imagine), you can
114email me at, or just post your question to the
115Python newsgroup with a catchy subject line.
118### Other Stuff ###
119The Query classes are by Mike Fletcher, with the working code
120being corruptions of Mark Hammonds win32pdhutil module.
122Use at your own risk, no warranties, no guarantees, no assurances,
123if you use it, you accept the risk of using it, etceteras.
126# Feb 12, 98 - MH added "rawaddcounter" so caller can get exception details.
128import win32pdh, win32api,time, thread,copy
130class BaseQuery:
131	'''
132	Provides wrapped access to the Performance Data Helper query
133	objects, generally you should use the child class Query
134	unless you have need of doing weird things :)
136	This class supports two major working paradigms.  In the first,
137	you open the query, and run it as many times as you need, closing
138	the query when you're done with it.  This is suitable for static
139	queries (ones where processes being monitored don't disappear).
141	In the second, you allow the query to be opened each time and
142	closed afterward.  This causes the base query object to be
143	destroyed after each call.  Suitable for dynamic queries (ones
144	which watch processes which might be closed while watching.)
145	'''
146	def __init__(self,paths=None):
147		'''
148		The PDH Query object is initialised with a single, optional
149		list argument, that must be properly formatted PDH Counter
150		paths.  Generally this list will only be provided by the class
151		when it is being unpickled (removed from storage).  Normal
152		use is to call the class with no arguments and use the various
153		addcounter functions (particularly, for end user's, the use of
154		addcounterbybrowsing is the most common approach)  You might
155		want to provide the list directly if you want to hard-code the
156		elements with which your query deals (and thereby avoid the
157		overhead of unpickling the class).
158		'''
159		self.counters = []
160		if paths:
161			self.paths = paths
162		else:
163			self.paths = []
164		self._base = None
165 = 0
166		self.curpaths = []
167	def addcounterbybrowsing(self, flags = win32pdh.PERF_DETAIL_WIZARD, windowtitle="Python Browser"):
168		'''
169		Adds possibly multiple paths to the paths attribute of the query,
170		does this by calling the standard counter browsing dialogue.  Within
171		this dialogue, find the counter you want to log, and click: Add,
172		repeat for every path you want to log, then click on close.  The
173		paths are appended to the non-volatile paths list for this class,
174		subclasses may create a function which parses the paths and decides
175		(via heuristics) whether to add the path to the volatile or non-volatile
176		path list.
177		e.g.:
178			query.addcounter()
179		'''
180		win32pdh.BrowseCounters(None,0, self.paths.append, flags, windowtitle)
181	def rawaddcounter(self,object, counter, instance = None, inum=-1, machine=None):
182		'''
183		Adds a single counter path, without catching any exceptions.
185		See addcounter for details.
186		'''
187		path = win32pdh.MakeCounterPath( (machine,object,instance, None, inum,counter) )
188		self.paths.append(path)
190	def addcounter(self,object, counter, instance = None, inum=-1, machine=None):
191		'''
192		Adds a single counter path to the paths attribute.  Normally
193		this will be called by a child class' speciality functions,
194		rather than being called directly by the user. (Though it isn't
195		hard to call manually, since almost everything is given a default)
196		This method is only functional when the query is closed (or hasn't
197		yet been opened).  This is to prevent conflict in multi-threaded
198		query applications).
199		e.g.:
200			query.addcounter('Memory','Available Bytes')
201		'''
202		if not
203			try:
204				self.rawaddcounter(object, counter, instance, inum, machine)
205				return 0
206			except win32api.error:
207				return -1
208		else:
209			return -1
211	def open(self):
212		'''
213		Build the base query object for this wrapper,
214		then add all of the counters required for the query.
215		Raise a QueryError if we can't complete the functions.
216		If we are already open, then do nothing.
217		'''
218		if not # to prevent having multiple open queries
219			# curpaths are made accessible here because of the possibility of volatile paths
220			# which may be dynamically altered by subclasses.
221			self.curpaths = copy.copy(self.paths)
222			try:
223				base = win32pdh.OpenQuery()
224				for path in self.paths:
225					try:
226						self.counters.append(win32pdh.AddCounter(base, path))
227					except win32api.error: # we passed a bad path
228						self.counters.append(0)
229						pass
230				self._base = base
231 = 1
232				return 0 # open succeeded
233			except: # if we encounter any errors, kill the Query
234				try:
235					self.killbase(base)
236				except NameError: # failed in creating query
237					pass
238 = 0
239				self.curpaths = []
240				raise QueryError(self)
241		return 1 # already open
243	def killbase(self,base=None):
244		'''
245		### This is not a public method
246		Mission critical function to kill the win32pdh objects held
247		by this object.  User's should generally use the close method
248		instead of this method, in case a sub-class has overridden
249		close to provide some special functionality.
250		'''
251		# Kill Pythonic references to the objects in this object's namespace
252		self._base = None
253		counters = self.counters
254		self.counters = []
255		# we don't kill the curpaths for convenience, this allows the
256		# user to close a query and still access the last paths
257 = 0
258		# Now call the delete functions on all of the objects
259		try:
260			map(win32pdh.RemoveCounter,counters)
261		except:
262			pass
263		try:
264			win32pdh.CloseQuery(base)
265		except:
266			pass
267		del(counters)
268		del(base)
269	def close(self):
270		'''
271		Makes certain that the underlying query object has been closed,
272		and that all counters have been removed from it.  This is
273		important for reference counting.
274		You should only need to call close if you have previously called
275		open.  The collectdata methods all can handle opening and
276		closing the query.  Calling close multiple times is acceptable.
277		'''
278		try:
279			self.killbase(self._base)
280		except AttributeError:
281			self.killbase()
282	__del__ = close
283	def collectdata(self,format = win32pdh.PDH_FMT_LONG):
284		'''
285		Returns the formatted current values for the Query
286		'''
287		if self._base: # we are currently open, don't change this
288			return self.collectdataslave(format)
289		else: # need to open and then close the _base, should be used by one-offs and elements tracking application instances
290 # will raise QueryError if couldn't open the query
291			temp = self.collectdataslave(format)
292			self.close() # will always close
293			return temp
294	def collectdataslave(self,format = win32pdh.PDH_FMT_LONG):
295		'''
296		### Not a public method
297		Called only when the Query is known to be open, runs over
298		the whole set of counters, appending results to the temp,
299		returns the values as a list.
300		'''
301		try:
302			win32pdh.CollectQueryData(self._base)
303			temp = []
304			for counter in self.counters:
305				ok = 0
306				try:
307					if counter:
308						temp.append(win32pdh.GetFormattedCounterValue(counter, format)[1])
309						ok = 1
310				except win32api.error:
311					pass
312				if not ok:
313					temp.append(-1) # a better way to signal failure???
314			return temp
315		except win32api.error: # will happen if, for instance, no counters are part of the query and we attempt to collect data for it.
316			return [-1] * len(self.counters)
317	# pickle functions
318	def __getinitargs__(self):
319		'''
320		### Not a public method
321		'''
322		return (self.paths,)
324class Query(BaseQuery):
325	'''
326	Performance Data Helper(PDH) Query object:
328	Provides a wrapper around the native PDH query object which
329	allows for query reuse, query storage, and general maintenance
330	functions (adding counter paths in various ways being the most
331	obvious ones).
332	'''
333	def __init__(self,*args,**namedargs):
334		'''
335		The PDH Query object is initialised with a single, optional
336		list argument, that must be properly formatted PDH Counter
337		paths.  Generally this list will only be provided by the class
338		when it is being unpickled (removed from storage).  Normal
339		use is to call the class with no arguments and use the various
340		addcounter functions (particularly, for end user's, the use of
341		addcounterbybrowsing is the most common approach)  You might
342		want to provide the list directly if you want to hard-code the
343		elements with which your query deals (and thereby avoid the
344		overhead of unpickling the class).
345		'''
346		self.volatilecounters = []
347		BaseQuery.__init__(*(self,)+args, **namedargs)
348	def addperfcounter(self, object, counter, machine=None):
349		'''
350		A "Performance Counter" is a stable, known, common counter,
351		such as Memory, or Processor.  The use of addperfcounter by 
352		end-users is deprecated, since the use of 
353		addcounterbybrowsing is considerably more flexible and general.
354		It is provided here to allow the easy development of scripts
355		which need to access variables so common we know them by name
356		(such as Memory|Available Bytes), and to provide symmetry with
357		the add inst counter method.
358		usage:
359			query.addperfcounter('Memory', 'Available Bytes')
360		It is just as easy to access addcounter directly, the following
361		has an identicle effect.
362			query.addcounter('Memory', 'Available Bytes')
363		'''
364		BaseQuery.addcounter(self, object=object, counter=counter, machine=machine)
365	def addinstcounter(self, object, counter,machine=None,objtype = 'Process',volatile=1,format = win32pdh.PDH_FMT_LONG):
366		'''
367		The purpose of using an instcounter is to track particular
368		instances of a counter object (e.g. a single processor, a single
369		running copy of a process).  For instance, to track all python.exe
370		instances, you would need merely to ask:
371			query.addinstcounter('python','Virtual Bytes')
372		You can find the names of the objects and their available counters 
373		by doing an addcounterbybrowsing() call on a query object (or by
374		looking in performance monitor's add dialog.)
376		Beyond merely rearranging the call arguments to make more sense,
377		if the volatile flag is true, the instcounters also recalculate
378		the paths of the available instances on every call to open the
379		query.
380		'''
381		if volatile:
382			self.volatilecounters.append((object,counter,machine,objtype,format))
383		else:
384			self.paths[len(self.paths):] = self.getinstpaths(object,counter,machine,objtype,format)
386	def getinstpaths(self,object,counter,machine=None,objtype='Process',format = win32pdh.PDH_FMT_LONG):
387		'''
388		### Not an end-user function
389		Calculate the paths for an instance object. Should alter
390		to allow processing for lists of object-counter pairs.
391		'''
392		items, instances = win32pdh.EnumObjectItems(None,None,objtype, -1)
393		# find out how many instances of this element we have...
394		instances.sort()
395		try:
396			cur = instances.index(object)
397		except ValueError:
398			return [] # no instances of this object
399		temp = [object]
400		try:
401			while instances[cur+1] == object:
402				temp.append(object)
403				cur = cur+1
404		except IndexError: # if we went over the end
405			pass
406		paths = []
407		for ind in range(len(temp)):
408			# can this raise an error?
409			paths.append(win32pdh.MakeCounterPath( (machine,'Process',object,None,ind,counter) ) )
410		return paths # should also return the number of elements for naming purposes
412	def open(self,*args,**namedargs):
413		'''
414		Explicitly open a query:
415		When you are needing to make multiple calls to the same query,
416		it is most efficient to open the query, run all of the calls,
417		then close the query, instead of having the collectdata method
418		automatically open and close the query each time it runs.
419		There are currently no arguments to open.
420		'''
421		# do all the normal opening stuff, self._base is now the query object
422*(self,)+args, **namedargs)
423		# should rewrite getinstpaths to take a single tuple
424		paths = []
425		for tup in self.volatilecounters:
426			paths[len(paths):] = self.getinstpaths(*tup)
427		for path in paths:
428			try:
429				self.counters.append(win32pdh.AddCounter(self._base, path))
430				self.curpaths.append(path) # if we fail on the line above, this path won't be in the table or the counters
431			except win32api.error:
432				pass # again, what to do with a malformed path???
433	def collectdatafor(self, totalperiod, period=1):
434		'''
435		Non-threaded collection of performance data:
436		This method allows you to specify the total period for which you would
437		like to run the Query, and the time interval between individual
438		runs.  The collected data is stored in query.curresults at the
439		_end_ of the run.  The pathnames for the query are stored in
440		query.curpaths.
441		e.g.:
442			query.collectdatafor(30,2)
443		Will collect data for 30seconds at 2 second intervals
444		'''
445		tempresults = []
446		try:
448			for ind in xrange(totalperiod/period):
449				tempresults.append(self.collectdata())
450				time.sleep(period)
451			self.curresults = tempresults
452		finally:
453			self.close()
454	def collectdatawhile(self, period=1):
455		'''
456		Threaded collection of performance data:
457		This method sets up a simple semaphor system for signalling 
458		when you would like to start and stop a threaded data collection
459		method.  The collection runs every period seconds until the
460		semaphor attribute is set to a non-true value (which normally
461		should be done by calling query.collectdatawhile_stop() .)
462		e.g.:
463			query.collectdatawhile(2)
464			# starts the query running, returns control to the caller immediately
465			# is collecting data every two seconds.
466			# do whatever you want to do while the thread runs, then call:
467			query.collectdatawhile_stop()
468			# when you want to deal with the data.  It is generally a good idea
469			# to sleep for period seconds yourself, since the query will not copy
470			# the required data until the next iteration:
471			time.sleep(2)
472			# now you can access the data from the attributes of the query
473			query.curresults
474			query.curpaths
475		'''
476		self.collectdatawhile_active = 1
477		thread.start_new_thread(self.collectdatawhile_slave,(period,))
478	def collectdatawhile_stop(self):
479		'''
480		Signals the collectdatawhile slave thread to stop collecting data
481		on the next logging iteration.
482		'''
483		self.collectdatawhile_active = 0
484	def collectdatawhile_slave(self, period):
485		'''
486		### Not a public function
487		Does the threaded work of collecting the data and storing it
488		in an attribute of the class.
489		'''
490		tempresults = []
491		try:
492 # also sets active, so can't be changed.
493			while self.collectdatawhile_active:
494				tempresults.append(self.collectdata())
495				time.sleep(period)
496			self.curresults = tempresults
497		finally:
498			self.close()
500	# pickle functions
501	def __getinitargs__(self):
502		return (self.paths,)
503	def __getstate__(self):
504		return self.volatilecounters
505	def __setstate__(self, volatilecounters):
506		self.volatilecounters = volatilecounters
509class QueryError:
510	def __init__(self, query):
511		self.query = query
512	def __repr__(self):
513		return '<Query Error in %s>'%repr(self.query)
514	__str__ = __repr__