/Client/src/bkr/client/wizard.py

Large files files are truncated, but you can click here to view the full file

# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.

DESCRIPTION = """Beaker Wizard is a tool which can transform that
"create all the necessary files with correct names, values, and paths"
boring phase of every test creation into one-line joy. For power
users there is a lot of inspiration in the man page. For quick start
just ``cd`` to your test package directory and simply run
``beaker-wizard``.
"""

__doc__ = """
beaker-wizard: Tool to ease the creation of a new Beaker task
=============================================================

.. program:: beaker-wizard

Synopsis
--------

| :program:`beaker-wizard` [*options*] <testname> <bug>

The *testname* argument should be specified as::

    [[[NAMESPACE/]PACKAGE/]TYPE/][PATH/]NAME

which can be shortened as you need::

    TESTNAME
    TYPE/TESTNAME
    TYPE/PATH/TESTNAME
    PACKAGE/TYPE/NAME
    PACKAGE/TYPE/PATH/NAME
    NAMESPACE/PACKAGE/TYPE/NAME
    NAMESPACE/PACKAGE/TYPE/PATH/NAME

| :program:`beaker-wizard` Makefile

This form will run the Wizard in the Makefile edit mode which allows you to
quickly and simply update metadata of an already existing test while trying to
keep the rest of the Makefile untouched.

Description
-----------

%(DESCRIPTION)s

The beaker-wizard was designed to be flexible: it is intended not only for
beginning Beaker users who will welcome questions with hints but also for
experienced test writers who can make use of the extensive command-line
options to push their new-test-creating productivity to the limits.

For basic usage help, see Options_ below or run ``beaker-wizard -h``.
For advanced features and expert usage examples, read on.

Highlights
~~~~~~~~~~

* provide reasonable defaults wherever possible
* flexible confirmation (``--every``, ``--common``, ``--yes``)
* predefined skeletons (beaker, beakerlib, simple, multihost, empty)
* saved user preferences (defaults, user skeletons, licenses)
* Bugzilla integration (fetch bug info, reproducers, suggest name, description)
* Makefile edit mode (quick adding of bugs, limiting archs or releases...)
* automated adding created files to the git repository

Skeletons
~~~~~~~~~

Another interesting feature is that you can save your own skeletons into
the preferences file, so that you can automatically populate the new
test scripts with your favourite structure.

All of the test related metadata gathered by the Wizard can be expanded
inside the skeletons using XML tags. For example: use ``<package/>`` for
expanding into the test package name or ``<test/>`` for the full test name.

The following metadata variables are available:

* test namespace package type path testname description
* bugs reproducers requires architectures releases version time
* priority license confidential destructive
* skeleton author email

Options
-------

-h, --help        show this help message and exit
-V, --version     display version info and quit

Basic metadata:
  -d DESCRIPTION  short description
  -a ARCHS        architectures [All]
  -r RELEASES     releases [All]
  -o PACKAGES     run for packages [wizard]
  -q PACKAGES     required packages [wizard]
  -t TIME         test time [5m]

Extra metadata:
  -z VERSION      test version [1.0]
  -p PRIORITY     priority [Normal]
  -l LICENSE      license [GPLv2+]
  -i INTERNAL     confidential [No]
  -u UGLY         destructive [No]

Author info:
  -n NAME         your name [Petr Splichal]
  -m MAIL         your email address [psplicha@redhat.com]

Test creation specifics:
  -s SKELETON     skeleton to use [beakerlib]
  -j PREFIX       join the bug prefix to the testname [Yes]
  -f, --force     force without review and overwrite existing files
  -w, --write     write preferences to ~/.beaker_client/wizard
  -b, --bugzilla  contact bugzilla to get bug details
  -g, --git       add created files to the git repository

Confirmation and verbosity:
  -v, --verbose   display detailed info about every action
  -e, --every     prompt for each and every available option
  -c, --common    confirm only commonly used options [Default]
  -y, --yes       yes, I'm sure, no questions, just do it!

Examples
--------

Some brief examples::

    beaker-wizard overload-performance 379791
        regression test with specified bug and name
        -> /CoreOS/perl/Regression/bz379791-overload-performance

    beaker-wizard buffer-overflow 2008-1071 -a i386
        security test with specified CVE and name, i386 arch only
        -> /CoreOS/perl/Security/CVE-2008-1071-buffer-overflow

    beaker-wizard Sanity/options -y -a?
        sanity test with given name, ask just for architecture
        -> /CoreOS/perl/Sanity/options

    beaker-wizard Sanity/server/smoke
        add an optional path under test type directory
        -> /CoreOS/perl/Sanity/server/smoke

    beaker-wizard -by 1234
        contact bugzilla for details, no questions, just review
        -> /CoreOS/installer/Regression/bz1234-Swap-partition-Installer

    beaker-wizard -byf 2007-0455
        security test, no questions, no review, overwrite existing files
        -> /CoreOS/gd/Security/CVE-2007-0455-gd-buffer-overrun

All of the previous examples assume you're in the package tests
directory (e.g. ``cd git/tests/perl``). All the necessary directories and
files are created under this location.

Bugzilla integration
~~~~~~~~~~~~~~~~~~~~

The following example creates a regression test for bug #227655.
Option ``-b`` is used to contact Bugzilla to automatically fetch bug
details and ``-y`` to skip unnecessary questions.

::

    # beaker-wizard -by 227655
    Contacting bugzilla...
    Fetching details for bz227655
    Examining attachments for possible reproducers
    Adding test.pl (simple test using Net::Config)
    Adding libnet.cfg (libnet.cfg test config file)

    Ready to create the test, please review
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    /CoreOS/perl/Regression/bz227655-libnet-cfg-in-wrong-directory

                 Namespace : CoreOS
                   Package : perl
                 Test type : Regression
             Relative path : None
                 Test name : bz227655-libnet-cfg-in-wrong-directory
               Description : Test for bz227655 (libnet.cfg in wrong directory)

        Bug or CVE numbers : bz227655
      Reproducers to fetch : test.pl, libnet.cfg
         Required packages : None
             Architectures : All
                  Releases : All
                   Version : 1.0
                      Time : 5m

                  Priority : Normal
                   License : GPLv2+
              Confidential : No
               Destructive : No

                  Skeleton : beakerlib
                    Author : Petr Splichal
                     Email : psplicha@redhat.com

    [Everything OK?]
    Directory Regression/bz227655-libnet-cfg-in-wrong-directory created
    File Regression/bz227655-libnet-cfg-in-wrong-directory/PURPOSE written
    File Regression/bz227655-libnet-cfg-in-wrong-directory/runtest.sh written
    File Regression/bz227655-libnet-cfg-in-wrong-directory/Makefile written
    Attachment test.pl downloaded
    Attachment libnet.cfg downloaded

Command line
~~~~~~~~~~~~

The extensive command line syntax can come in handy for example
when creating a bunch of sanity tests for a component. Let's
create a test skeleton for each of wget's feature areas::

    # cd git/tests/wget
    # for test in download recursion rules authentication; do
    >   beaker-wizard -yf $test -t 10m -q httpd,vsftpd \\
    >       -d "Sanity test for $test options"
    > done

    ...

    /CoreOS/wget/Sanity/authentication

                 Namespace : CoreOS
                   Package : wget
                 Test type : Sanity
             Relative path : None
                 Test name : authentication
               Description : Sanity test for authentication options

        Bug or CVE numbers : None
      Reproducers to fetch : None
         Required packages : httpd, vsftpd
             Architectures : All
                  Releases : All
                   Version : 1.0
                      Time : 10m

                  Priority : Normal
                   License : GPLv2+
              Confidential : No
               Destructive : No

                  Skeleton : beakerlib
                    Author : Petr Splichal
                     Email : psplicha@redhat.com

    Directory Sanity/authentication created
    File Sanity/authentication/PURPOSE written
    File Sanity/authentication/runtest.sh written
    File Sanity/authentication/Makefile written

    # tree
    .
    `-- Sanity
        |-- authentication
        |   |-- Makefile
        |   |-- PURPOSE
        |   `-- runtest.sh
        |-- download
        |   |-- Makefile
        |   |-- PURPOSE
        |   `-- runtest.sh
        |-- recursion
        |   |-- Makefile
        |   |-- PURPOSE
        |   `-- runtest.sh
        `-- rules
            |-- Makefile
            |-- PURPOSE
            `-- runtest.sh

Notes
-----

If you provide an option with a "?" you will be given a list of
available options and a prompt to type your choice in.

For working Bugzilla integration you need ``python-bugzilla`` package installed on your system.
If you are trying to access a bug with restricted access, log
in to Bugzilla first with the following command::

    bugzilla login

You will be asked for email and password and after successfully logging in a
``~/.bugzillacookies`` file will be created which then will be used
in all subsequent Bugzilla queries. Logout can be performed with
``rm ~/.bugzillacookies`` ;-)

Files
-----

All commonly used preferences can be saved into ``~/.beaker_client/wizard``.
Use "write" command to save current settings when reviewing gathered
test data or edit the file with you favourite editor.

All options in the config file are self-explanatory. For confirm level choose
one of: nothing, common or everything.

Library tasks
-------------

The "library" skeleton can be used to create a "library task". It allows you to bundle
together common functionality which may be required across multiple
tasks. To learn more, see `the BeakerLib documentation for library
tasks <https://fedorahosted.org/beakerlib/wiki/libraries>`__.

Bugs
----

If you encounter an issue or have an idea for enhancement, please `file a new bug`_.
See also `open bugs`_.

.. _file a new bug: https://bugzilla.redhat.com/enter_bug.cgi?product=Beaker&component=command+line&short_desc=beaker-wizard:+&status_whiteboard=BeakerWizard&assigned_to=psplicha@redhat.com
.. _open bugs: https://bugzilla.redhat.com/buglist.cgi?product=Beaker&bug_status=__open__&short_desc=beaker-wizard&short_desc_type=allwordssubstr

See also
--------

* `Beaker documentation <http://beaker-project.org/help.html>`_
* `BeakerLib <https://fedorahosted.org/beakerlib>`_
""" % globals()

from optparse import OptionParser, OptionGroup, IndentedHelpFormatter, SUPPRESS_HELP
from xml.dom.minidom import parse, parseString
from datetime import date
from time import sleep
import subprocess
import textwrap
import pwd
import sys
import re
import os

# Version
WizardVersion = "2.3.0"

# Regular expressions
RegExpPackage    = re.compile("^(?![._+-])[.a-zA-Z0-9_+-]+(?<![._-])$")
RegExpRhtsRequires = re.compile("^(?![._+-])[.a-zA-Z0-9_+-/()]+(?<![._-])$")
RegExpPath       = re.compile("^(?![/-])[a-zA-Z0-9/_-]+(?<![/-])$")
RegExpTestName   = re.compile("^(?!-)[a-zA-Z0-9-_]+(?<!-)$")
RegExpBug        = re.compile("^\d+$")
RegExpBugLong    = re.compile("^bz\d+$")
RegExpBugPrefix  = re.compile("^bz")
RegExpCVE        = re.compile("^\d{4}-\d{4}$")
RegExpCVELong    = re.compile("^CVE-\d{4}-\d{4}$")
RegExpCVEPrefix  = re.compile("^CVE-")
RegExpAuthor     = re.compile("^[a-zA-Z]+\.?( [a-zA-Z]+\.?){1,2}$")
RegExpEmail      = re.compile("^[a-z._-]+@[a-z.-]+$")
RegExpYes        = re.compile("Everything OK|y|ye|jo|ju|ja|ano|da", re.I)
RegExpReproducer = re.compile("repr|test|expl|poc|demo", re.I)
RegExpScript     = re.compile("\.(sh|py|pl)$")
RegExpMetadata   = re.compile("(\$\(METADATA\):\s+Makefile.*)$", re.S)
RegExpTest       = re.compile("TEST=(\S+)", re.S)
RegExpVersion    = re.compile("TESTVERSION=([\d.]+)", re.S)

# Suggested test types (these used to be enforced)
SuggestedTestTypes = """Regression Performance Stress Certification
            Security Durations Interoperability Standardscompliance
            Customeracceptance Releasecriterium Crasher Tier1 Tier2
            Alpha KernelTier1 KernelTier2 Multihost MultihostDriver
            Install FedoraTier1 FedoraTier2 KernelRTTier1
            KernelReporting Sanity Library""".split()

# Guesses
GuessAuthorLogin = pwd.getpwuid(os.getuid())[0]
GuessAuthorDomain = re.sub("^.*\.([^.]+\.[^.]+)$", "\\1", os.uname()[1])
GuessAuthorEmail = "%s@%s" % (GuessAuthorLogin, GuessAuthorDomain)
GuessAuthorName = pwd.getpwuid(os.getuid())[4]

# Make sure guesses are valid values
if not RegExpEmail.match(GuessAuthorEmail):
    GuessAuthorEmail = "your@email.com"
if not RegExpAuthor.match(GuessAuthorName):
    GuessAuthorName = "Your Name"

# Commands
GitCommand="git add".split()

# Constants
MaxLengthSuggestedDesc = 50
MaxLengthTestName = 50
ReviewWidth = 22
MakefileLineWidth = 17
VimDictionary = "# vim: dict+=/usr/share/beakerlib/dictionary.vim cpt=.,w,b,u,t,i,k"
BugzillaUrl = 'https://bugzilla.redhat.com/show_bug.cgi?id='
BugzillaXmlrpc = 'https://bugzilla.redhat.com/xmlrpc.cgi'
PreferencesDir = os.getenv('HOME') + "/.beaker_client"
PreferencesFile = PreferencesDir + "/wizard"
PreferencesTemplate = """<?xml version="1.0" ?>

<wizard>
    <author>
        <name>%s</name>
        <email>%s</email>
        <confirm>common</confirm>
        <skeleton>beakerlib</skeleton>
    </author>
    <test>
        <time>5m</time>
        <type>Sanity</type>
        <prefix>Yes</prefix>
        <namespace>CoreOS</namespace>
        <priority>Normal</priority>
        <license>GPLv2+</license>
        <confidential>No</confidential>
        <destructive>No</destructive>
    </test>
    <licenses>
        <license name="GPLvX">
            This is GPLvX license text.
        </license>
        <license name="GPLvY">
            This is GPLvY license text.
        </license>
        <license name="GPLvZ">
            This is GPLvZ license text.
        </license>
    </licenses>
    <skeletons>
        <skeleton name="skel1" requires="gdb" rhtsrequires="library(perl/lib1) library(scl/lib2)">
            This is skeleton 1 example.
        </skeleton>
        <skeleton name="skel2">
            This is skeleton 2 example.
        </skeleton>
        <skeleton name="skel3">
            This is skeleton 3 example.
        </skeleton>
    </skeletons>
</wizard>
""" % (GuessAuthorName, GuessAuthorEmail)


def wrapText(text):
    """ Wrapt text to fit default width """
    text = re.compile("\s+").sub(" ", text)
    return "\n".join(textwrap.wrap(text))

def dedentText(text, count = 12):
    """ Remove leading spaces from the beginning of lines """
    return re.compile("\n" + " " * count).sub("\n", text)

def indentText(text, count = 12):
    """ Insert leading spaces to the beginning of lines """
    return re.compile("\n").sub("\n" + " " * count, text)

def shortenText(text, max = 50):
    """ Shorten long texts into something more usable """
    # if shorter, nothing to do
    if not text or len(text) <= max:
        return text
    # cut the text
    text = text[0:max+1]
    # remove last non complete word
    text = re.sub(" [^ ]*$", "", text)
    return text

def shellEscaped(text):
    """
    Returns the text escaped for inclusion inside a shell double-quoted string.
    """
    return text.replace('\\', '\\\\')\
               .replace('"', r'\"')\
               .replace('$', r'\$')\
               .replace('`', r'\`')\
               .replace('!', r'\!')

def unique(seq):
    """ Remove duplicates from the supplied sequence """
    dictionary = {}
    for i in seq:
        dictionary[i] = 1
    return dictionary.keys()

def hr(width = 70):
    """ Return simple ascii horizontal rule """
    if width < 2: return ""
    return "# " + (width - 2) * "~"

def comment(text, width = 70, comment = "#", top = True, bottom = True, padding = 3):
    """ Create nicely formated comment """
    result = ""
    # top hrule & padding
    if width and top: result += hr(width) + "\n"
    result += int(padding/3) * (comment + "\n")
    # prepend lines with the comment char and padding
    result += re.compile("^(?!#)", re.M).sub(comment + padding * " ", text)
    # bottom padding & hrule
    result += int(padding/3) * ("\n" + comment)
    if width and bottom: result += "\n" + hr(width)
    # remove any trailing spaces
    result = re.compile("\s+$", re.M).sub("", result)
    return result

def dashifyText(text, allowExtraChars = ""):
    """ Replace all special chars with dashes, and perhaps shorten """
    if not text: return text
    # remove the rubbish from the start & end
    text = re.sub("^[^a-zA-Z0-9]*", "", text)
    text = re.sub("[^a-zA-Z0-9]*$", "", text)
    # replace all special chars with dashes
    text = re.sub("[^a-zA-Z0-9%s]+" % allowExtraChars, "-", text)
    return text

def createNode(node, text):
    """ Create a child text node """
    # find document root
    root = node
    while root.nodeType != root.DOCUMENT_NODE:
        root = root.parentNode
    # append child text node
    node.appendChild(root.createTextNode(text))
    return node

def getNode(node):
    """ Return node value """
    try: value = node.firstChild.nodeValue
    except: return None
    else: return value

def setNode(node, value):
    """ Set node value (create a child if necessary) """
    try: node.firstChild.nodeValue = value
    except: createNode(node, value)
    return value

def findNode(parent, tag, name = None):
    """ Find a child node with specified tag (and name) """
    try:
        for child in parent.getElementsByTagName(tag):
            if name is None or child.getAttribute("name") == name:
                return child
    except:
        return None

def findNodeNames(node, tag):
    """ Return list of all name values of specified tags """
    list = []
    for child in node.getElementsByTagName(tag):
        if child.hasAttribute("name"):
            list.append(child.getAttribute("name"))
    return list

def parentDir():
    """ Get parent directory name for package name suggestion """
    dir = re.split("/", os.getcwd())[-1]
    if dir == "": return "kernel"
    # remove the -tests suffix if present
    # (useful if writing tests in the package/package-tests directory)
    dir = re.sub("-tests$", "", dir)
    return dir

def addToGit(path):
    """ Add a file or a directory to Git """

    try:
        process = subprocess.Popen(GitCommand + [path],
                stdout=subprocess.PIPE, stderr=subprocess.PIPE,)
        out, err = process.communicate()
        if process.wait():
            print "Sorry, failed to add %s to git :-(" % path
            print out, err
            sys.exit(1)
    except OSError:
        print("Unable to run %s, is %s installed?"
                % (" ".join(GitCommand), GitCommand[0]))
        sys.exit(1)

def removeEmbargo(summary):
    return summary.replace('EMBARGOED ', '')


class Preferences:
    """ Test's author preferences """

    def __init__(self, load_user_prefs=True):
        """ Set (in future get) user preferences / defaults """
        self.template = parseString(PreferencesTemplate)
        self.firstRun = False
        if load_user_prefs:
            self.load()
        else:
            self.xml = self.template
            self.parse()


    # XXX (ncoghlan): all of these exec invocations should be replaced with
    # appropriate usage of setattr and getattr. However, beaker-wizard needs
    # decent test coverage before embarking on that kind of refactoring...
    def parse(self):
        """ Parse values from the xml file """
        # parse list nodes
        for node in "author test licenses skeletons".split():
            exec("self.%s = findNode(self.xml, '%s')" % (node, node))

        # parse single value nodes for author
        for node in "name email confirm skeleton".split():
            exec("self.%s = findNode(self.author, '%s')" % (node, node))
            # if the node cannot be found get the default from template
            if not eval("self." + node):
                print "Could not find <%s> in preferences, using default" % node
                exec("self.%s = findNode(self.template, '%s').cloneNode(True)"
                        % (node, node))
                exec("self.author.appendChild(self.%s)" % node)

        # parse single value nodes for test
        for node in "type namespace time priority confidential destructive " \
                "prefix license".split():
            exec("self.%s = findNode(self.test, '%s')" % (node, node))
            # if the node cannot be found get the default from template
            if not eval("self." + node):
                print "Could not find <%s> in preferences, using default" % node
                exec("self.%s = findNode(self.template, '%s').cloneNode(True)" % (node, node))
                exec("self.test.appendChild(self.%s)" % node)

    def load(self):
        """ Load user preferences (or set to defaults) """
        preferences_file = os.environ.get("BEAKER_WIZARD_CONF", PreferencesFile)
        try:
            self.xml = parse(preferences_file)
        except:
            if os.path.exists(preferences_file):
                print "I'm sorry, the preferences file seems broken.\n" \
                        "Did you do something ugly to %s?" % preferences_file
                sleep(3)
            else:
                self.firstRun = True
            self.xml = self.template
            self.parse()
        else:
            try:
                self.parse()
            except:
                print "Failed to parse %s, falling to defaults." % preferences_file
                sleep(3)
                self.xml = self.template
                self.parse()

    def update(self, author, email, confirm, type, namespace, \
            time, priority, confidential, destructive, prefix, license, skeleton):
        """ Update preferences with current settings """
        setNode(self.name, author)
        setNode(self.email, email)
        setNode(self.confirm, confirm)
        setNode(self.type, type)
        setNode(self.namespace, namespace)
        setNode(self.time, time)
        setNode(self.priority, priority)
        setNode(self.confidential, confidential)
        setNode(self.destructive, destructive)
        setNode(self.prefix, prefix)
        setNode(self.license, license)
        setNode(self.skeleton, skeleton)

    def save(self):
        """ Save user preferences """
        # try to create directory
        try:
            os.makedirs(PreferencesDir)
        except OSError, e:
            if e.errno == 17:
                pass
            else:
                print "Cannot create preferences directory %s :-(" % PreferencesDir
                return

        # try to write the file
        try:
            file = open(PreferencesFile, "w")
        except:
            print "Cannot write to %s" % PreferencesFile
        else:
            file.write((self.xml.toxml() + "\n").encode("utf-8"))
            file.close()
            print "Preferences saved to %s" % PreferencesFile
        sleep(1)

    def getAuthor(self): return getNode(self.name)
    def getEmail(self): return getNode(self.email)
    def getConfirm(self): return getNode(self.confirm)
    def getType(self): return getNode(self.type)
    def getPackage(self): return parentDir()
    def getNamespace(self): return getNode(self.namespace)
    def getTime(self): return getNode(self.time)
    def getPriority(self): return getNode(self.priority)
    def getConfidential(self): return getNode(self.confidential)
    def getDestructive(self): return getNode(self.destructive)
    def getPrefix(self): return getNode(self.prefix)
    def getVersion(self): return "1.0"
    def getLicense(self): return getNode(self.license)
    def getSkeleton(self): return getNode(self.skeleton)

    def getLicenseContent(self, license):
        content = findNode(self.licenses, "license", license)
        if content:
            return re.sub("\n\s+$", "", content.firstChild.nodeValue)
        else:
            return None


class Help:
    """ Help texts """

    def __init__(self, options = None):
        if options:
            # display expert usage page only
            if options.expert():
                print self.expert();
                sys.exit(0)
            # show version info
            elif options.ver():
                print self.version();
                sys.exit(0)

    def usage(self):
        return "beaker-wizard [options] [TESTNAME] [BUG/CVE...] or beaker-wizard Makefile"

    def version(self):
        return "beaker-wizard %s" % WizardVersion

    def description(self):
        return DESCRIPTION

    def expert(self):
        os.execv('/usr/bin/man', ['man', 'beaker-wizard'])
        sys.exit(1)


class Makefile:
    """
    Parse values from an existing Makefile to set the initial values
    Used in the Makefile edit mode.
    """

    def __init__(self, options):
        # try to read the original Makefile
        self.path = options.arg[0]
        try:
            # open and read the whole content into self.text
            print "Reading the Makefile..."
            file = open(self.path)
            self.text = "".join(file.readlines())
            file.close()

            # substitute the old style $TEST sub-variables if present
            for var in "TOPLEVEL_NAMESPACE PACKAGE_NAME RELATIVE_PATH".split():
                m = re.search("%s=(\S+)" % var, self.text)
                if m: self.text = re.sub("\$\(%s\)" % var, m.group(1), self.text)

            # locate the metadata section
            print "Inspecting the metadata section..."
            m = RegExpMetadata.search(self.text)
            self.metadata = m.group(1)

            # parse the $TEST and $TESTVERSION
            print "Checking for the full test name and version..."
            m = RegExpTest.search(self.text)
            options.arg = [m.group(1)]
            m = RegExpVersion.search(self.text)
            options.opt.version = m.group(1)
        except:
            print "Failed to parse the original Makefile"
            sys.exit(6)

        # disable test name prefixing and set confirm to nothing
        options.opt.prefix = "No"
        options.opt.confirm = "nothing"

        # initialize non-existent options.opt.* vars
        options.opt.bug = options.opt.owner = options.opt.runfor = None
        # uknown will be used to store unrecognized metadata fields
        self.unknown = ""
        # map long fields to short versions
        map = {
            "description" : "desc",
            "architectures" : "archs",
            "testtime" : "time"
        }

        # parse info from metadata line by line
        print "Parsing the individual metadata..."
        for line in self.metadata.split("\n"):
            m = re.search("echo\s+[\"'](\w+):\s*(.*)[\"']", line)
            # skip non-@echo lines
            if not m: continue
            # read the key & value pair
            try: key = map[m.group(1).lower()]
            except: key = m.group(1).lower()
            # get the value, unescape escaped double quotes
            value = re.sub("\\\\\"", "\"", m.group(2))
            # skip fields known to contain variables
            if key in ("name", "testversion", "path"): continue
            # save known fields into options
            for data in "owner desc type archs releases time priority license " \
                    "confidential destructive bug requires runfor".split():
                if data == key:
                    # if multiple choice, extend the array
                    if key in "archs bug releases requires runfor".split():
                        try: exec("options.opt.%s.append(value)" % key)
                        except: exec("options.opt.%s = [value]" % key)
                    # otherwise just set the value
                    else:
                        exec("options.opt.%s = value" % key)
                    break
            # save unrecognized fields to be able to restore them back
            else:
                self.unknown += "\n" + line

        # parse name & email
        m = re.search("(.*)\s+<(.*)>", options.opt.owner)
        if m:
            options.opt.author = m.group(1)
            options.opt.email = m.group(2)

        # add bug list to arg
        if options.opt.bug:
            options.arg.extend(options.opt.bug)

        # success
        print "Makefile successfully parsed."

    def save(self, test, version, content):
        # possibly update the $TEST and $TESTVERSION
        self.text = RegExpTest.sub("TEST=" + test, self.text)
        self.text = RegExpVersion.sub("TESTVERSION=" + version, self.text)

        # substitute the new metadata
        m = RegExpMetadata.search(content)
        self.text = RegExpMetadata.sub(m.group(1), self.text)

        # add unknown metadata fields we were not able to parse at init
        self.text = re.sub("\n\n\trhts-lint",
                self.unknown + "\n\n\trhts-lint", self.text)

        # let's write it
        try:
            file = open(self.path, "w")
            file.write(self.text.encode("utf-8"))
            file.close()
        except:
            print "Cannot write to %s" % self.path
            sys.exit(3)
        else:
            print "Makefile successfully written"


class Options:
    """
    Class maintaining user preferences and options provided on command line

    self.opt  ... options parsed from command line
    self.pref ... user preferences / defaults
    """

    def __init__(self, argv=None, load_user_prefs=True):
        if argv is None:
            argv = sys.argv
        self.pref = Preferences(load_user_prefs)
        formatter = IndentedHelpFormatter(max_help_position=40)
        #formatter._long_opt_fmt = "%s"

        # parse options
        parser = OptionParser(Help().usage(), formatter=formatter)
        parser.set_description(Help().description())

        # examples and help
        parser.add_option("-x", "--expert",
            dest="expert",
            action="store_true",
            help=SUPPRESS_HELP)
        parser.add_option("-V", "--version",
            dest="ver",
            action="store_true",
            help="display version info and quit")

        # author
        groupAuthor = OptionGroup(parser, "Author info")
        groupAuthor.add_option("-n",
            dest="author",
            metavar="NAME",
            help="your name [%s]" % self.pref.getAuthor())
        groupAuthor.add_option("-m",
            dest="email",
            metavar="MAIL",
            help="your email address [%s]" %  self.pref.getEmail())

        # create
        groupCreate = OptionGroup(parser, "Test creation specifics")
        groupCreate.add_option("-s",
            dest="skeleton",
            help="skeleton to use [%s]" % self.pref.getSkeleton())
        groupCreate.add_option("-j",
            dest="prefix",
            metavar="PREFIX",
            help="join the bug prefix to the testname [%s]"
                    % self.pref.getPrefix())
        groupCreate.add_option("-f", "--force",
            dest="force",
            action="store_true",
            help="force without review and overwrite existing files")
        groupCreate.add_option("-w", "--write",
            dest="write",
            action="store_true",
            help="write preferences to ~/.beaker_client/wizard")
        groupCreate.add_option("-b", "--bugzilla",
            dest="bugzilla",
            action="store_true",
            help="contact bugzilla to get bug details")
        groupCreate.add_option("-g", "--git",
            dest="git",
            action="store_true",
            help="add created files to the git repository")

        # setup default to correctly display in help
        defaultEverything = defaultCommon = defaultNothing = ""
        if self.pref.getConfirm() == "everything":
            defaultEverything = " [Default]"
        elif self.pref.getConfirm() == "common":
            defaultCommon = " [Default]"
        elif self.pref.getConfirm() == "nothing":
            defaultNothing = " [Default]"

        # confirm
        groupConfirm = OptionGroup(parser, "Confirmation and verbosity")
        groupConfirm.add_option("-v", "--verbose",
            dest="verbose",
            action="store_true",
            help="display detailed info about every action")
        groupConfirm.add_option("-e", "--every",
            dest="confirm",
            action="store_const",
            const="everything",
            help="prompt for each and every available option" + defaultEverything)
        groupConfirm.add_option("-c", "--common",
            dest="confirm",
            action="store_const",
            const="common",
            help="confirm only commonly used options" + defaultCommon)
        groupConfirm.add_option("-y", "--yes",
            dest="confirm",
            action="store_const",
            const="nothing",
            help="yes, I'm sure, no questions, just do it!" + defaultNothing)

        # test metadata
        groupMeta = OptionGroup(parser, "Basic metadata")
        groupMeta.add_option("-d",
            dest="desc",
            metavar="DESCRIPTION",
            help="short description")
        groupMeta.add_option("-a",
            dest="archs",
            action="append",
            help="architectures [All]")
        groupMeta.add_option("-r",
            dest="releases",
            action="append",
            help="releases [All]")
        groupMeta.add_option("-o",
            dest="runfor",
            action="append",
            metavar="PACKAGES",
            help="run for packages [%s]" % self.pref.getPackage())
        groupMeta.add_option("-q",
            dest="requires",
            action="append",
            metavar="PACKAGES",
            help="required packages [%s]" % self.pref.getPackage())
        groupMeta.add_option("-Q",
            dest="rhtsrequires",
            action="append",
            metavar="TEST",
            help="required RHTS tests or libraries")
        groupMeta.add_option("-t",
            dest="time",
            help="test time [%s]" % self.pref.getTime())

        # test metadata
        groupExtra = OptionGroup(parser, "Extra metadata")
        groupExtra.add_option("-z",
            dest="version",
            help="test version [%s]" % self.pref.getVersion())
        groupExtra.add_option("-p",
            dest="priority",
            help="priority [%s]" % self.pref.getPriority())
        groupExtra.add_option("-l",
            dest="license",
            help="license [%s]" % self.pref.getLicense())
        groupExtra.add_option("-i",
            dest="confidential",
            metavar="INTERNAL",
            help="confidential [%s]" % self.pref.getConfidential())
        groupExtra.add_option("-u",
            dest="destructive",
            metavar="UGLY",
            help="destructive [%s]" % self.pref.getDestructive())

        # put it together
        parser.add_option_group(groupMeta)
        parser.add_option_group(groupExtra)
        parser.add_option_group(groupAuthor)
        parser.add_option_group(groupCreate)
        parser.add_option_group(groupConfirm)

        # convert all args to unicode
        uniarg = []
        for arg in argv[1:]:
             uniarg.append(unicode(arg, "utf-8"))

        # and parse it!
        (self.opt, self.arg) = parser.parse_args(uniarg)

        # parse namespace/package/type/path/test
        self.opt.namespace = None
        self.opt.package = None
        self.opt.type = None
        self.opt.path = None
        self.opt.name = None
        self.opt.bugs = []
        self.makefile = False

        if self.arg:
            # if we're run in the Makefile-edit mode, parse it to get the values
            if re.match(".*Makefile$", self.arg[0]):
                self.makefile = Makefile(self)

            # the first arg looks like bug/CVE -> we take all args as bugs/CVE's
            if RegExpBug.match(self.arg[0]) or RegExpBugLong.match(self.arg[0]) or \
                    RegExpCVE.match(self.arg[0]) or RegExpCVELong.match(self.arg[0]):
                self.opt.bugs = self.arg[:]
            # otherwise we expect bug/CVE as second and following
            else:
                self.opt.bugs = self.arg[1:]
                # parsing namespace/package/type/path/testname
                self.testinfo = self.arg[0]
                path_components = os.path.normpath(self.testinfo.rstrip('/')).split('/')
                if len(path_components) >= 1:
                    self.opt.name = path_components.pop(-1)
                if len(path_components) >= 3 and re.match(Namespace().match() + '$', path_components[0]):
                    self.opt.namespace = path_components.pop(0)
                    self.opt.package = path_components.pop(0)
                    self.opt.type = path_components.pop(0)
                elif len(path_components) >= 2 and path_components[1] in SuggestedTestTypes:
                    self.opt.package = path_components.pop(0)
                    self.opt.type = path_components.pop(0)
                elif len(path_components) >= 1:
                    self.opt.type = path_components.pop(0)
                if path_components:
                    self.opt.path = '/'.join(path_components)

        # try to connect to bugzilla
        self.bugzilla = None
        if self.opt.bugzilla:
            try:
                from bugzilla import Bugzilla
            except:
                print "Sorry, the bugzilla interface is not available right now, try:\n" \
                        "    yum install python-bugzilla\n" \
                        "Use 'bugzilla login' command if you wish to access restricted bugs."
                sys.exit(8)
            else:
                try:
                    print "Contacting bugzilla..."
                    self.bugzilla = Bugzilla(url=BugzillaXmlrpc)
                except:
                    print "Cannot connect to bugzilla, check your net connection."
                    sys.exit(9)

    # command-line-only option interface
    def expert(self):   return self.opt.expert
    def ver(self):      return self.opt.ver
    def force(self):    return self.opt.force
    def write(self):    return self.opt.write
    def verbose(self):  return self.pref.firstRun or self.opt.verbose
    def confirm(self):  return self.opt.confirm or self.pref.getConfirm()

    # return both specified and default values for the rest of options
    def author(self):       return [ self.opt.author,       self.pref.getAuthor() ]
    def email(self):        return [ self.opt.email,        self.pref.getEmail() ]
    def skeleton(self):     return [ self.opt.skeleton,     self.pref.getSkeleton() ]
    def archs(self):        return [ self.opt.archs,        [] ]
    def releases(self):     return [ self.opt.releases,     ['-RHEL4', '-RHELClient5', '-RHELServer5'] ]
    def runfor(self):       return [ self.opt.runfor,       [self.pref.getPackage()] ]
    def requires(self):     return [ self.opt.requires,     [self.pref.getPackage()] ]
    def rhtsrequires(self): return [ self.opt.rhtsrequires, [] ]
    def time(self):         return [ self.opt.time,         self.pref.getTime() ]
    def priority(self):     return [ self.opt.priority,     self.pref.getPriority() ]
    def confidential(self): return [ self.opt.confidential, self.pref.getConfidential() ]
    def destructive(self):  return [ self.opt.destructive,  self.pref.getDestructive() ]
    def prefix(self):       return [ self.opt.prefix,       self.pref.getPrefix() ]
    def license(self):      return [ self.opt.license,      self.pref.getLicense() ]
    def version(self):      return [ self.opt.version,      self.pref.getVersion() ]
    def desc(self):         return [ self.opt.desc,         "What the test does" ]
    def description(self):  return [ self.opt.description,  "" ]
    def namespace(self):    return [ self.opt.namespace,    self.pref.getNamespace() ]
    def package(self):      return [ self.opt.package,      self.pref.getPackage() ]
    def type(self):         return [ self.opt.type,         self.pref.getType() ]
    def path(self):         return [ self.opt.path,         "" ]
    def name(self):         return [ self.opt.name,         "a-few-descriptive-words" ]
    def bugs(self):         return [ self.opt.bugs,         [] ]



class Inquisitor:
    """
    Father of all Inquisitors

    Well he is not quite real Inquisitor, as he is very
    friendly and accepts any answer you give him.
    """
    def __init__(self, options = None, suggest = None):
        # set options & initialize
        self.options = options
        self.suggest = suggest
        self.common = True
        self.error = 0
        self.init()
        if not self.options: return

        # finally ask for confirmation or valid value
        if self.confirm or not self.valid():
            self.ask()

    def init(self):
        """ Initialize basic stuff """
        self.name = "Answer"
        self.question = "What is the answer to life, the universe and everything"
        self.description = None
        self.default()

    def default(self, optpref=None):
        """ Initialize default option data """
        # nothing to do when options not supplied
        if not optpref: return

        # initialize opt (from command line) & pref (from user preferences)
        (self.opt, self.pref) = optpref

        # set confirm flag
        self.confirm = self.common and self.options.confirm() != "nothing" \
                or not self.common and self.options.confirm() == "everything"

        # now set the data!
        # commandline option overrides both preferences & suggestion
        if self.opt:
            self.data = self.opt
            self.confirm = False
        # use suggestion if available (disabled in makefile edit mode)
        elif self.suggest and not self.options.makefile:
            self.data = self.suggest
        # otherwise use the default from user preferences
        else:
            self.data = self.pref
            # reset the user preference if it's not a valid value
            # (to prevent suggestions like: x is not valid what about x?)
            if not self.valid():
                self.pref = "something else"

    def defaultify(self):
        """ Set data to default/preferred value """
        self.data = self.pref

    def normalize(self):
        """ Remove trailing and double spaces """
        if not self.data: return
        self.data = re.sub("^\s*", "", self.data)
        self.data = re.sub("\s*$", "", self.data)
        self.data = re.sub("\s+", " ", self.data)

    def read(self):
        """ Read an answer from user """
        try:
            answer = unicode(sys.stdin.readline().strip(), "utf-8")
        except KeyboardInterrupt:
            print "\nOk, finishing for now. See you later ;-)"
            sys.exit(4)
        # if just enter pressed, we leave self.data as it is (confirmed)
        if answer != "":
            # append the data if the answer starts with a "+"
            m = re.search("^\+\s*(.*)", answer)
            if m and type(self.data) is list:
                self.data.append(m.group(1))
            else:
                self.data = answer
        self.normalize()

    def heading(self):
        """ Display nice heading with question """
        print "\n" + self.question + "\n" + 77 * "~";

    def value(self):
        """ Return current value """
        return self.data

    def show(self, data = None):
        """ Return current value nicely formatted (redefined in children)"""
        if not data: data = self.data
        if data == "": return "None"
        return data

    def singleName(self):
        """ Return the name in lowercase singular (for error reporting) """
        return re.sub("s$", "", self.name.lower())

    def matchName(self, text):
        """ Return true if the text matches inquisitor's name """
        # remove any special characters from the search string
        text = re.sub("[^\w\s]", "", text)
        return re.search(text, self.name, re.I)

    def describe(self):
        if self.description is not None:
            print wrapText(self.description)

    def format(self, data = None):
        """ Display in a nicely indented style """
        print self.name.rjust(ReviewWidth), ":", (data or self.show())

    def formatMakefileLine(self, name = None, value = None):
        """ Format testinfo line for Makefile inclusion """
        if not (self.value() or value): return ""
        return '\n            	@echo "%s%s" >> $(METADATA)' % (
                ((name or self.name) + ":").ljust(MakefileLineWidth),
                shellEscaped(value or self.value()))

    def valid(self):
        """ Return true when provided value is a valid answer """
        return self.data not in ["?", ""]

    def suggestion(self):
        """ Provide user with a suggestion or detailed description """
        # if current data is valid, offer is as a suggestion
        if self.valid():
            if self.options.verbose(): self.describe()
            return "%s?" % self.show()
        # otherwise suggest the default value
        else:
            bad = self.data
            self.defaultify()

            # regular suggestion (no question mark for help)
            if bad is None or "".join(bad) != "?":
                self.error += 1
                if self.error > 1 or self.options.verbose(): self.describe()
                return "%s is not a valid %s, what about %s?" \
                    % (self.show(bad), self.singleName(), self.show(self.pref))
            # we got question mark ---> display description to help
            else:
                self.describe()
                return "%s?" % self.show()

    def ask(self, force = False, suggest = None):
        """ Ask for valid value """
        if force: self.confirm = True
        if suggest: self.data = suggest
        self.heading()
        # keep asking until we get sane answer
        while self.confirm or not self.valid():
            sys.stdout.write("[%s] " % self.suggestion().encode("utf-8"))
            self.read()
            self.confirm = False

    def edit(self, suggest = None):
        """ Edit = force to ask again
        returns true if changes were made """
        before = self.data
        self.ask(force = True, suggest = suggest)
        return self.data != before


class SingleChoice(Inquisitor):
    """ This Inquisitor accepts just one value from the given list """

    def init(self):
        self.name = "SingleChoice"
        self.question = "Give a valid answer from the list"
        self.description = "Supply a single value from the list above."
        self.list = ["list", "of", "valid", "values"]
        self.default()

    def propose(self):
        """ Try to find nearest match in the list"""
        if self.data == "?": return
        for item in self.list:
            if re.search(self.data, item, re.I):
                self.pref = item
                return

    def valid(self):
        if self.data in self.list:
            return True
        else:
            self.propose()
            return False

    def heading(self):
        Inquisitor.heading(self)
        if self.list: print wrapText("Possible values: " + ", ".join(self.list))


class YesNo(SingleChoice):
    """ Inquisitor expecting only two obvious answers """

    def init(self):
        self.name = "Yes or No"
        self.question = "Are you sure?"
        self.description = "All you need to say is simply 'Yes,' or 'No'; \
                anything beyond this comes from the evil one."
        self.list = ["Yes", "No"]
        self.default()

    def normalize(self):
        """ Recognize yes/no abbreviations """
        if not self.data: return
        self.data = re.sub("^y.*$", "Yes", self.data, re.I)
        self.data = re.sub("^n.*$", "No", self.data, re.I)

    def formatMakefileLine(self, name = None, value = None):
        """ Format testinfo line for Makefile inclusion """
        # testinfo requires lowercase yes/no
        return Inquisitor.formatMakefileLine(self,
                name = name, value = self.data.lower())

    def valid(self):
        self.normalize()
        return SingleChoice.valid(self)


class MultipleChoice(SingleChoice):
    """ This Inquisitor accepts more values but only from the given list """

    def init(self):
        self.name = "MultipleChoice"
        self.question = "Give one or more values from the list"
        self.description = "You can supply more values separated with space or comma\n"\
            "but they all must be from the list above."
        self.list = ["list", "of", "valid", "values"]
        self.emptyListMeaning = "None"
        self.sort = True
        self.default()

    def default(self, optpref):
        # initialize opt & pref
        (self.opt, self.pref) = optpref

        # set confirm flag
        self.confirm = self.common and self.options.confirm() != "nothing" \
                or not self.common and self.options.confirm() == "everything"

        # first initialize data as an empty list
        self.data = []

        # append possible suggestion to the data (disabled in makefile edit mode)
        if self.suggest and …