DiveIntoPython(十五)

DiveIntoPython(十五)

英文书地址：
http://diveintopython.org/toc/index.html

Chapter 16.Functional Progamming

16.1.Diving in
The following is a complete Python program that acts as a cheap and simple regression testing framework. It takes unit tests that you've written for individual modules, collects them all into one big test suite, and runs them all at once.

example 16.1.regression.py
import sys, os, re, unittest

def regressionTest():
    path = os.path.abspath(os.path.dirname(sys.argv[0]))  
    files = os.listdir(path)                              
    test = re.compile("test\.py$", re.IGNORECASE)         
    files = filter(test.search, files)                    
    filenameToModuleName = lambda f: os.path.splitext(f)[0]
    moduleNames = map(filenameToModuleName, files)        
    modules = map(__import__, moduleNames)                
    load = unittest.defaultTestLoader.loadTestsFromModule 
    return unittest.TestSuite(map(load, modules))         

if __name__ == "__main__":                  
    unittest.main(defaultTest="regressionTest")

example 16.2.Sample output of regression.py
E:\book\opensource\python\diveintopython-5.4\py>python regression.py -v
info should fail with no object ... ok
info should return known result for apihelper ... ok
info should honor collapse argument ... ok
info should honor spacing argument ... ok

...snip...

fromRoman(toRoman(n))==n for all n ... ok
toRoman should fail with non-integer input ... ok
toRoman should fail with negative input ... ok
toRoman should fail with large input ... ok
toRoman should fail with 0 input ... ok
soundex should give known result with known input ... ok

----------------------------------------------------------------------
Ran 81 tests in 0.593s

OK

16.2.Finding the path
When running Python scripts from the command line, it is sometimes useful to know where the currently running script is located on disk.
The key to it is sys.argv. As you saw in Chapter 9, XML Processing, this is a list that holds the list of command-line arguments. However, it also holds the name of the running script, exactly as it was called from the command line, and this is enough information to determine its location.

example 16.3.fullpath.py
import sys, os

print 'sys.argv[0] =', sys.argv[0]
pathname = os.path.dirname(sys.argv[0])
print 'path =', pathname
print 'full path =', os.path.abspath(pathname)

Regardless of how you run a script, sys.argv[0] will always contain the name of the script, exactly as it appears on the command line. This may or may not include any path information, as you'll see shortly.

os.path.dirname takes a filename as a string and returns the directory path portion. If the given filename does not include any path information, os.path.dirname returns an empty string.

os.path.abspath is the key here. It takes a pathname, which can be partial or even blank, and returns a fully qualified pathname.

example 16.4.Further explanation of os.path.abspath
>>> import os
>>> os.getcwd()
'E:\\book\\opensource\\python\\diveintopython-5.4\\py'
>>> os.path.abspath('')
'E:\\book\\opensource\\python\\diveintopython-5.4\\py'
>>> os.path.abspath('.ssh')
'E:\\book\\opensource\\python\\diveintopython-5.4\\py\\.ssh'
>>> os.path.abspath('E:\\book\\opensource\\python\\diveintopython-5.4\\py\\.ssh')
'E:\\book\\opensource\\python\\diveintopython-5.4\\py\\.ssh'
>>> os.path.abspath('.ssh/../foo/')
'E:\\book\\opensource\\python\\diveintopython-5.4\\py\\foo'

os.getcwd() returns the current working directory.

Calling os.path.abspath with an empty string returns the current working directory, same as os.getcwd().

It normalizes the path by making it as simple as possible. If you just want to normalize a pathname like this without turning it into a full pathname, use os.path.normpath instead.

example 16.5.Sample output from fullpath.py
E:\book\opensource\python\diveintopython-5.4\py>python fullpath.py
sys.argv[0] = fullpath.py
path =
full path = E:\book\opensource\python\diveintopython-5.4\py

E:\>python E:\book\opensource\python\diveintopython-5.4\py\fullpath.py
sys.argv[0] = E:\book\opensource\python\diveintopython-5.4\py\fullpath.py
path = E:\book\opensource\python\diveintopython-5.4\py
full path = E:\book\opensource\python\diveintopython-5.4\py

If the script is run from the current directory without giving any path, os.path.dirname will simply return an empty string. Given an empty string, os.path.abspath returns the current directory, which is what you want, since the script was run from the current directory.

sys.argv[0] includes the full path of the script. You can then use the os.path.dirname function to strip off the script name and return the full directory name, and os.path.abspath simply returns what you give it.

example 16.6.Running scripts in the current directory
import sys, os, re, unittest

def regressionTest():
    path = os.getcwd()      
    sys.path.append(path)   
    files = os.listdir(path)

Instead of setting path to the directory where the currently running script is located, you set it to the current working directory instead. This will be whatever directory you were in before you ran the script, which is not necessarily the same as the directory the script is in.

Append this directory to the Python library search path, so that when you dynamically import the unit test modules later, Python can find them. You didn't need to do this when path was the directory of the currently running script, because Python always looks in that directory.

The rest of the function is the same.

This technique will allow you to re-use this regression.py script on multiple projects. Just put the script in a common directory, then change to the project's directory before running it. All of that project's unit tests will be found and tested, instead of the unit tests in the common directory where regression.py is located.

16.3.Filtering lists revisited
Python has a built-in filter function which takes two arguments, a function and a list, and returns a list.

The function passed as the first argument to filter must itself take one argument, and the list that filter returns will contain all the elements from the list passed to filter for which the function passed to filter returns true.

example 16.7.Introducing filter
>>> def odd(n):
... return n % 2
...
>>> li = [1,2,3,5,9,10,256,-3]
>>> filter(odd,li)
[1, 3, 5, 9, -3]
>>> [e for e in li if odd(e)]
[1, 3, 5, 9, -3]
>>> filteredList = []
>>> for n in li:
... if odd(n):
... filteredList.append(n)
...
>>> filteredList
[1, 3, 5, 9, -3]

odd uses the built-in mod function “%” to return True if n is odd and False if n is even.

filter takes two arguments, a function (odd) and a list (li). It loops through the list and calls odd with each element. If odd returns a true value (remember, any non-zero value is true in Python), then the element is included in the returned list, otherwise it is filtered out. The result is a list of only the odd numbers from the original list, in the same order as they appeared in the original.

example 16.8.filter in regression.py
files = os.listdir(path)                               
test = re.compile("test\.py$", re.IGNORECASE)          
files = filter(test.search, files) 

Either way, files will end up with the names of the files in the same directory as this script you're running.

If the regular expression matches, the method will return a Match object, which Python considers to be true, so the element will be included in the list returned by filter. If the regular expression does not match, the search method will return None, which Python considers to be false, so the element will not be included.

example 16.9.Filtering using list comprehensions instead
files = os.listdir(path)                              
test = re.compile("test\.py$", re.IGNORECASE)         
files = [f for f in files if test.search(f)]

>>> files = os.listdir("")
>>> test = re.compile("test\.py$", re.IGNORECASE)
>>> files = [f for f in files if test.search(f)]
>>> files
['apihelpertest.py', 'kgptest.py', 'odbchelpertest.py', 'pluraltest.py', 'romantest.py', 'soundextest.py']

16.4.Mapping lists revisited
You're already familiar with using list comprehensions to map one list into another. There is another way to accomplish the same thing, using the built-in map function. It works much the same way as the filter function.

example 16.10.Introducing map
>>> def double(n):
... return n*2
...
>>> li = [1,2,3,5,9,10,256,-3]
>>> map(double,li)
[2, 4, 6, 10, 18, 20, 512, -6]
>>> [double(n) for n in li]
[2, 4, 6, 10, 18, 20, 512, -6]
>>> newList = []
>>> for n in li:
... newList.append(double(n))
...
>>> newList
[2, 4, 6, 10, 18, 20, 512, -6]

map takes a function and a list[8] and returns a new list by calling the function with each element of the list in order. In this case, the function simply multiplies each element by 2.

example 16.11.map with lists of mixed datatypes
>>> li = [5,'a',(2,'b')]
>>> map(double,li)
[10, 'aa', (2, 'b', 2, 'b')]

As a side note, I'd like to point out that map works just as well with lists of mixed datatypes, as long as the function you're using correctly handles each type.

example 16.12.map in regression.py
filenameToModuleName = lambda f: os.path.splitext(f)[0]
moduleNames = map(filenameToModuleName, files)

And as you saw in Example 6.17, “Splitting Pathnames”, os.path.splitext takes a filename and returns a tuple (name, extension). So filenameToModuleName is a function which will take a filename and strip off the file extension, and return just the name.

Calling map takes each filename listed in files, passes it to the function filenameToModuleName, and returns a list of the return values of each of those function calls. In other words, you strip the file extension off of each filename, and store the list of all those stripped filenames in moduleNames.

16.5.Data-centric programming

16.6.Dynamically importing modules

example 16.13.Importing multiple modules at once
import sys,os,re,unittest

This imports four modules at once: sys (for system functions and access to the command line parameters), os (for operating system functions like directory listings), re (for regular expressions), and unittest (for unit testing).

example 16.14.Importing modules dynamically
>>> sys = __import__('sys')
>>> os = __import__('os')
>>> re = __import__('re')
>>> unittest = __import__('unittest')
>>> sys
<module 'sys' (built-in)>
>>> os
<module 'os' from 'C:\Python26\lib\os.pyc'>

The built-in __import__ function accomplishes the same goal as using the import statement, but it's an actual function, and it takes a string as an argument.

example 16.15.Importing a list of modules dynamically
>>> moduleNames
['sys', 'os', 're', 'unittest']
>>> modules = map(__import__,moduleNames)
>>> modules
[<module 'sys' (built-in)>, <module 'os' from 'C:\Python26\lib\os.pyc'>, <module 're' from 'C:\Python26\lib\re.pyc'>, <module 'unittest' from 'C:\Python26\lib\unittest.pyc'>]
>>> modules[0].version
'2.6.4 (r264:75706, Jan 22 2010, 16:41:54) [MSC v.1500 32 bit (Intel)]'
>>> import sys
>>> sys.version
'2.6.4 (r264:75706, Jan 22 2010, 16:41:54) [MSC v.1500 32 bit (Intel)]'

Surprise, you wanted to import them, and you did, by mapping the __import__ function onto the list. Remember, this takes each element of the list (moduleNames) and calls the function (__import__) over and over, once with each element of the list, builds a list of the return values, and returns the result.

16.7.Putting it all together

example 16.16.The regressionTest function
def regressionTest():
    path = os.path.abspath(os.path.dirname(sys.argv[0]))  
    files = os.listdir(path)                              
    test = re.compile("test\.py$", re.IGNORECASE)         
    files = filter(test.search, files)                    
    filenameToModuleName = lambda f: os.path.splitext(f)[0]
    moduleNames = map(filenameToModuleName, files)        
    modules = map(__import__, moduleNames)                
load = unittest.defaultTestLoader.loadTestsFromModule 
return unittest.TestSuite(map(load, modules)) 

example 16.7.Step 1:Get all the files
>>> import sys,os,re,unittest
>>> path = r'E:\book\opensource\python\diveintopython-5.4\py'
>>> files = os.listdir(path)
>>> files
['apihelper.py', 'apihelper.pyc', 'apihelpertest.py', 'apihelpertest.pyc', 'argecho.py', 'autosize.py', 'BaseHTMLProcessor.py', 'BaseHTMLProcessor.pyc', 'builddialectexamples.py', 'colorize.py', 'dialect.py', 'fibonacci.py', 'fileinfo.py', 'fileinfo.pyc', 'fileinfo_fromdict.py', 'fullpath.py', 'kgp', 'kgptest.py', 'kgptest.pyc', 'LICENSE.txt', 'makerealworddoc.py', 'odbchelper.py', 'odbchelper.pyc', 'odbchelpertest.py', 'odbchelpertest.pyc', 'openanything.py', 'openanything.pyc', 'parsephone.py', 'piglatin.py', 'plural', 'plural-rules.en', 'plural.py', 'plural.pyc', 'pluraltest.py', 'pluraltest.pyc', 'pyfontify.py', 'regression.py', 'roman', 'roman.py', 'roman.pyc', 'romantest.py', 'romantest.pyc', 'search.py', 'soundex', 'soundex.py', 'soundex.pyc', 'soundextest.py', 'soundextest.pyc', 'statsout.py', 'statsout.pyc', 'temp.py', 'unicode2koi8r.py', 'urllister.py', 'urllister.pyc']

files is a list of all the files and directories in the script's directory.

example 16.18.Step 2:Filter to find the files you care about
>>> test = re.compile("test\.py$",re.IGNORECASE)
>>> files = filter(test.search,files)
>>> files
['apihelpertest.py', 'kgptest.py', 'odbchelpertest.py', 'pluraltest.py', 'romantest.py', 'soundextest.py']

This regular expression will match any string that ends with test.py. Note that you need to escape the period, since a period in a regular expression usually means “match any single character”, but you actually want to match a literal period instead.

example 16.19.Step 3:Map filenames to module names
>>> filenameToModuleName = lambda f:os.path.splitext(f)[0]
>>> filenameToModuleName('romantest.py')
'romantest'
>>> moduleNames = map(filenameToModuleName,files)
>>> moduleNames
['apihelpertest', 'kgptest', 'odbchelpertest', 'pluraltest', 'romantest', 'soundextest']

example 16.20.Step 4:Mapping module names to modules
>>> sys.path.append(r"E:\book\opensource\python\diveintopython-5.4\py")
>>> sys.path.append(r"E:\book\opensource\python\diveintopython-5.4\py\kgp")
>>> moduleNames
['apihelpertest', 'kgptest', 'odbchelpertest', 'pluraltest', 'romantest', 'soundextest']
>>> modules = map(__import__,moduleNames)
>>> modules
[<module 'apihelpertest' from 'E:\book\opensource\python\diveintopython-5.4\py\apihelpertest.pyc'>, <module 'kgptest' from 'E:\book\opensource\python\diveintopython-5.4\py\kgptest.pyc'>, <module 'odbchelpertest' from 'E:\book\opensource\python\diveintopython-5.4\py\odbchelpertest.pyc'>, <module 'pluraltest' from 'E:\book\opensource\python\diveintopython-5.4\py\pluraltest.pyc'>, <module 'romantest' from 'E:\book\opensource\python\diveintopython-5.4\py\romantest.pyc'>, <module 'soundextest' from 'E:\book\opensource\python\diveintopython-5.4\py\soundextest.pyc'>]
>>> modules[-1]
<module 'soundextest' from 'E:\book\opensource\python\diveintopython-5.4\py\soundextest.pyc'>

example 16.21.Step 5:Loading the modules into a test suite
>>> load = unittest.defaultTestLoader.loadTestsFromModule
>>> map(load,modules)
[<unittest.TestSuite tests=[<unittest.TestSuite tests=[<apihelpertest.BadInput testMethod=testNoObject>]>, <unittest.TestSuite tests=[<apihelpertest.KnownValues testMethod=testApiHelper>]>, <unittest.TestSuite tests=[<apihelpertest.ParamChecks testMethod=testCollapse>, <apihelpertest.ParamChecks testMethod=testSpacing>]>, <unittest.TestSuite tests=[]>]>, <unittest.TestSuite tests=[<unittest.TestSuite
...snip...
>>> unittest.TestSuite(map(load,modules))

That's what the loadTestsFromModule method does: it introspects into each module and returns a unittest.TestSuite object for each module. Each TestSuite object actually contains a list of TestSuite objects, one for each TestCase class in your module, and each of those TestSuite objects contains a list of tests, one for each test method in your module.

Finally, you wrap the list of TestSuite objects into one big test suite. The unittest module has no problem traversing this tree of nested test suites within test suites; eventually it gets down to an individual test method and executes it, verifies that it passes or fails, and moves on to the next one.

example 16.22.Step 6:Telling unittest to use your test suite
if __name__ == "__main__":                  
    unittest.main(defaultTest="regressionTest")