Document Preview

Extracts from the notes are below, to see the PDF you'll receive please use the links above

---

Advanced Python Programming
David M
...
uchicago
...
uchicago
...

Operating system interfaces
Programming with Threads
Network programming
Database interfaces
Restricted execution
Extensions in C
...

Goal is to highlight many of Python’s capabilities
...
uchicago
...

Python programmers who want to know more
...


Disclaimer
This tutorial is aimed at an advanced audience
I assume prior knowledge of topics in Operating Systems and Networks
...


My Background
I was drawn to Python as a C programmer
...

Wrote the "Python Essential Reference" in 1999 (New Riders Publishing)
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 3
July 17, 2000, beazley@cs
...
edu

A Very Brief Tour of Python

O’Reilly OSCON 2000, Advanced Python Programming, Slide 4
July 17, 2000, beazley@cs
...
edu

Starting and Stopping Python
Unix
unix % python
Python 1
...
2 (#1, Sep 19 1999, 16:29:25) [GCC 2
...
2
...

An interpreter window will appear and you will see the prompt
...

Type Control-D or Control-Z at the interactive prompt
...
uchicago
...
py
print "Hello World"

Running a file
unix % python hello
...
uchicago
...
5
(a+b)/2
...

Variables are just names for an object
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 7
July 17, 2000, beazley@cs
...
edu

Conditionals
if-else
# Compute maximum (z) of a and b
if a < b:
z = b
else:
z = a

The pass statement
if a < b:
pass
else:
z = a

# Do nothing

Notes:
Indentation used to denote bodies
...

There is no ’?:’ operator
...
uchicago
...


Boolean expressions: and, or, not
if b >= a
print
if not (b
print

and b <= c:
"b is between a and c"
< a or b > c):
"b is still between a and c"

O’Reilly OSCON 2000, Advanced Python Programming, Slide 9
July 17, 2000, beazley@cs
...
edu

Basic Types (Numbers and Strings)
Numbers
a
b
c
d

=
=
=
=

3
4
...
’" # A mix of both
d = ’’’A triple quoted string
can span multiple lines
like this’’’
e = """Also works for double quotes"""

O’Reilly OSCON 2000, Advanced Python Programming, Slide 10
July 17, 2000, beazley@cs
...
edu

Basic Types (Lists)
Lists of Arbitrary Objects
a
b
c
d
e

=
=
=
=
=

[2,
[2,
[]
[2,
a +

3, 4]
7, 3
...
uchicago
...
x = 3
# Slices
...
z = 4

Comments
Tuples are like lists, but size is fixed at time of creation
...
uchicago
...
has_key("directory"):
d = c[’directory’]
else:
d = None

# Get an element
# Set an element
# Check for presence of an member

d = c
...
uchicago
...
uchicago
...
uchicago
...
balance = initial
def deposit(self, amt):
self
...
balance + amt
def withdraw(self,amt):
self
...
balance - amt
def getbalance(self):
return self
...
00)
a
...
23)
a
...
withdraw(50)
print a
...
uchicago
...
Sorry
...
uchicago
...
write("Hello World")
data = g
...
readline()
lines = g
...
write("2 times %d = %d\n" % (i, 2*i))

O’Reilly OSCON 2000, Advanced Python Programming, Slide 18
July 17, 2000, beazley@cs
...
edu

Modules
Large programs can be broken into modules
# numbers
...
divide(42,5)
n = numbers
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 19
July 17, 2000, beazley@cs
...
edu

Python Library
Python is packaged with a large library of standard modules
String processing
Operating system interfaces
Networking
Threads
GUI
Database
Language services
Security
...


All of these are accessed using ’import’
import string

...
split(x)

O’Reilly OSCON 2000, Advanced Python Programming, Slide 20
July 17, 2000, beazley@cs
...
edu

Quick Summary
This is not an introductory tutorial
Consult online docs or Learning Python for a gentle introduction
...

Generally speaking, most programmers don’t have trouble picking up Python
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 21
July 17, 2000, beazley@cs
...
edu

String Processing

O’Reilly OSCON 2000, Advanced Python Programming, Slide 22
July 17, 2000, beazley@cs
...
edu

The string module
Various string processing functions
string
...
atoi(s)
string
...
count(s,pattern)
string
...
split(s, sep)
string
...
replace(s,old,new)

#
#
#
#
#
#
#
#

Convert to float
Convert to integer
Convert to long
Count occurrences of pattern in s
Find pattern in s
String a string
Join a list of string
Replace occurrences of old with new

Examples
s
a
b
c

=
=
=
=

"Hello World"
string
...
replace(s,"Hello","Goodbye")
string
...
uchicago
...

Generally contain a mix of text and special characters
foo
...

Details follow
...
uchicago
...

^
$
*
+
?
*?
+?
{m,n}
{m,n}?
[
...
]
A | B
(
...
uchicago
...

Literal backslash

Raw strings
Because of backslashes and special characters, raw strings are used
...
(\d*)’

# Matches numbers like 3
...
uchicago
...

Compiled into a regular expression "object"
...


Example
import re
pat = r’(\d+)\
...
compile(pat)
m = r
...

else:
# Nope
...


# My pattern
# Compile it
# See if string s matches

O’Reilly OSCON 2000, Advanced Python Programming, Slide 27
July 17, 2000, beazley@cs
...
edu

The re Module (cont)
Regular Expression Objects
Objects created by re
...
search(s [,pos [,endpos]])
r
...
split(s)
r
...
sub(repl,s)

#
#
#
#
#

Search for a match
Check string for match
Split on a regex match
Find all matches
Replace all matches with repl

When a match is found a ’MatchObject’ object is returned
...

Also contains group information
...

The match method looks for a match starting with the first character
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 28
July 17, 2000, beazley@cs
...
edu

The re Module (cont)
Match Objects
Contain information about the match itself
But it is based on a notion of "groups"

Grouping Rules
(\d+)\
...
group(n)
m
...
end(n)

# Return text matched for group n
# Return starting index for group n
# Return ending index for group n

O’Reilly OSCON 2000, Advanced Python Programming, Slide 29
July 17, 2000, beazley@cs
...
edu

The re Module (cont)
Matching Example
import re
r = re
...
(\d*)’)
m = r
...
37")
a = m
...
37’
b = m
...
group(2)
# Returns ’37’
print m
...
python
...
[\w-]+)*((/[\w-~]*)?))’
r = re
...
sub(’\\1’,s)
# Replace in string

Where to go from here?
Mastering Regular Expressions, by Jeffrey Friedl
Online docs
Experiment

O’Reilly OSCON 2000, Advanced Python Programming, Slide 30
July 17, 2000, beazley@cs
...
edu

Working with Files

O’Reilly OSCON 2000, Advanced Python Programming, Slide 31
July 17, 2000, beazley@cs
...
edu

File Objects
open(filename [,mode])
Opens a file and returns a file object
By default, opens a file for reading
...

This is required for portability to Windows
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 32
July 17, 2000, beazley@cs
...
edu

File Objects
File Methods
The following methods can be applied to an open file f
f
...
readline([n])
f
...
write(s)
f
...
close()
f
...
seek(offset [,where])

f
...
flush()
f
...
fileno()

#
#
#
#
#
#
#
#
#
#
#
#
#
#
#

Read at most n bytes
Read a line of input with max length of n
Read all input and return a list of lines
Write string s
Write a list of strings
Close a file
Return current file pointer
Seek to a new position
where = 0: Relative to start
where = 1: Relative to current
where = 2: Relative to end
Return 1 if interactive terminal
Flush output
Truncate file to at most size bytes
Return integer file descriptor

O’Reilly OSCON 2000, Advanced Python Programming, Slide 33
July 17, 2000, beazley@cs
...
edu

File Objects
File Attributes
The following attributes provide additional file information
f
...
mode
f
...
softspace

#
#
#
#
#
#

Set to 1 if file object has been closed
I/O mode of the file
Name of file if created using open()
...


Other notes
File operations on lines are aware of local conventions (\n\r vs
...

String data read and written to files may contain embedded nulls and other binary content
...
uchicago
...
stdin - Standard input
sys
...
stderr - Standard error

These are used by several built-in functions
print outputs to sys
...
stdin
s = raw_input("type a command : ")
print "You typed ", s

Error messages and the interactive prompts go to sys
...
stdout = open("output","w")

O’Reilly OSCON 2000, Advanced Python Programming, Slide 35
July 17, 2000, beazley@cs
...
edu

File and Path Manipulation
os
...
)

#
#
#
#
#
#
#
#
#
#
#

Returns the absolute pathname of a path
Returns filename component of path
Returns directory component of path
Normalize capitalization of a name
Normalize a pathname
Split path into (directory, file)
Split path into (drive, pathname)
Split path into (filename, suffix)
Expands ~user components
Expands environment vars ’$name’ or ’${name}’
Join pathname components

Examples
abspath("
...
/bin/python")
split("/usr/bin/python")
splitext("index
...
html")

O’Reilly OSCON 2000, Advanced Python Programming, Slide 36
July 17, 2000, beazley@cs
...
edu

File Tests
os
...


Notes:
samefile() and sameopenfile() useful if file referenced by symbolic links or aliases
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 37
July 17, 2000, beazley@cs
...
edu

Globbing
glob module
Returns filenames in a directory that match a pattern
import glob
a = glob
...
html")
b = glob
...
gif")

Pattern matching is performed using rules of Unix shell
...


fnmatch module
Matches filenames according to rules of Unix shell
import fnmatch
if fnmatch(filename,"*
...


Case-sensitivity depends on the operating system
...
uchicago
...
open(file [,flags [,mode]])
Opens a file and returns an integer file descriptor
flags is the bitwise-or of the following
O_RDONLY
O_WRONLY
O_RDWR
O_APPEND
O_CREAT
O_NONBLOCK
O_TRUNC
O_TEXT
O_BINARY

Open file for reading
Open file for writing
Open file for read/write
Append to the end of the file
Create file if it doesn’t exit
Don’t block on open,read, or write
...
open("foo", O_WRONLY | O_CREAT, 0644)

O’Reilly OSCON 2000, Advanced Python Programming, Slide 39
July 17, 2000, beazley@cs
...
edu

Low-Level I/O operations
The os module contains a variety of low-level I/O functions
os
...
dup(fd)
os
...
fdopen(fd [,mode [,bufsize]])
os
...
fstatvfs(fd)
os
...
lseek(fd,pos,how)

#
#
#
#
#
#
#
#
#
#
#

Close a file
Duplicate file descriptor fd
Duplicate oldfd to newfd
Create a file object from an fd
Return file status for fd
Return file system info for fd
Truncate file to given size
Seek to new position
how = 0: beginning of file
how = 1: current position
how = 2: end of file

os
...
write(fd,str)

# Read at most n bytes
# Write data in str

Notes
The os
...
fileno() methods convert between file objects and file descriptors
...
uchicago
...
access(path,accessmode)
os
...
chown(path,uid,gid)
os
...
listdir(path)
os
...
remove(path)
os
...
rmdir(path)
os
...
statvfs(path)
os
...
unlink(path)
os
...
path module for some of these operations
...
Consult a reference
...
uchicago
...
flock(f
...
LOCK_EX)

tempfile
Creates temporary files

gzip
Creates file objects with compression/decompression
Compatible with the GNU gzip program
...
open("foo","wb")
f
...
close()

O’Reilly OSCON 2000, Advanced Python Programming, Slide 42
July 17, 2000, beazley@cs
...
edu

Strings and Files
The StringIO and cStringIO modules
Provide a file-like object that reads/writes from a string buffer
Example:
import StringIO
f = StringIO
...
write("Hello World\n")

...
getvalue()

# Get saved string value

Notes
StringIO objects support most of the normal file operations
cStringIO is implemented in C and is significantly faster
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 43
July 17, 2000, beazley@cs
...
edu

Object Serialization and Persistence

O’Reilly OSCON 2000, Advanced Python Programming, Slide 44
July 17, 2000, beazley@cs
...
edu

Object Serialization
Motivation
Sometimes you need to save an object to disk and restore it later
...


Problem
Manual implementation requires a lot of work
...

Must write code to marshal objects to and from the encoding
...

Python provides several modules to do all of this for you

O’Reilly OSCON 2000, Advanced Python Programming, Slide 45
July 17, 2000, beazley@cs
...
edu

The pickle and cPickle Module
The pickle and cPickle modules serialize objects to and from files
To serialize, you ’pickle’ an object
import pickle
p = pickle
...
dump(obj)

# file is an open file object
# Dump object

To unserialize, you ’unpickle’ an object
p = pickle
...
load()

# file is an open file
# Load object

Notes
Most built-in types can be pickled except for files, sockets, execution frames, etc
...

Any file-like object that provides write(),read(), and readline() methods can be used as a file
...

cPickle is like pickle, but written in C and is substantially faster
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 46
July 17, 2000, beazley@cs
...
edu

The marshal Module
The marshal module can also be used for serialization
To serialize
import marshal
marshal
...
load(file)

Notes
marshal is similiar to pickle, but is intended only for simple objects
Can’t handle recursion or class instances
...

Data is stored in a binary architecture independent format

O’Reilly OSCON 2000, Advanced Python Programming, Slide 47
July 17, 2000, beazley@cs
...
edu

The shelve Module
The shelve module provides a persistent dictionary
Idea: works like a dictionary, but data is stored on disk
import shelve
d = shelve
...
has_key(key)
d
...
close()

#
#
#
#
#
#

Store an object
Retrieve an object
Delete an object
Test for existence of key
Return a list of all keys
Close the shelf

Comments
Keys must be strings
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 48
July 17, 2000, beazley@cs
...
edu

DBM-Style Databases
Python provides a number of DBM-style database interfaces
Key-based databases that store arbitrary strings
...
open("database","r")
d["foo"] = "bar"
# Store a value
s = d["spam"]
# Retrieve a value
del d["name"]
# Delete a value
d
...

Don’t use these if you should really be using a relational database (e
...
, MySQL)
...
uchicago
...
uchicago
...

The interface is based on POSIX
...


Let’s take a tour
...

This is mostly a survey of what Python provides
...
uchicago
...
environ - A dictionary containing current environment variables
user = os
...
environ[’PATH’] = "/bin:/usr/bin"

Current directory and umask
os
...
getcwd()
os
...
Returns previous umask

User and group identification
os
...
geteuid()
os
...
getuid()
os
...
setuid(uid)

#
#
#
#
#
#

Get
Get
Get
Get
Set
Set

effective group id
effective user id
group id
user id
group id
user id

O’Reilly OSCON 2000, Advanced Python Programming, Slide 52
July 17, 2000, beazley@cs
...
edu

Process Creation and Destruction
fork-exec-wait
os
...
execv(path,args)
os
...
execvp(path, args)
os
...
wait([pid)]
os
...
system(command)
os
...

# Execute a process
# Execute process, use default path
#
#
#
#

Wait for child process
Wait for change in state of child
Execute a system command
Exit immediately with status n
...
fork()
# Create child
if pid == 0:
# Child process
os
...
wait()
# Wait for child

O’Reilly OSCON 2000, Advanced Python Programming, Slide 53
July 17, 2000, beazley@cs
...
edu

Pipes
os
...
read()
f
...


The popen2 module
Spawns processes and provides hooks to stdin, stdout, and stderr
popen2(cmd)
popen3(cmd)

# Run cmd and return (stdout, stdin)
# Run cmd and return (stdout, stdin, stderr)

Example
(o,i) = popen2
...
write(data)
# Write to child’s input
i
...
read()
# Get child’s output
o
...
uchicago
...
getoutput("ls -l")

Also includes a quoting function
arg = mkarg(str)

# Turns str into a argument suitable
# for use in the shell (to prevent mischief)

Comments
Really this is just a wrapper over the popen2 module
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 55
July 17, 2000, beazley@cs
...
edu

Error Handling
System-related errors are typically translated into the following
OSError - General operating system error
IOError - I/O related system error

Cause of the error is contained in errno attribute of exception
Can use the errno module for symbolic error names

Example:
import os, errno

...
execlp("foo")
except OSError,e:
if e
...
ENOENT:
print "Program not found
...
errno == errno
...
"
else:
# Some other kind of error

O’Reilly OSCON 2000, Advanced Python Programming, Slide 56
July 17, 2000, beazley@cs
...
edu

Signal Handling
Signals
Usually correspond to external events and arrive asynchronously
...


The signal module
Provides functions for writing Unix-style signal handlers in Python
...
signal(signalnum, handler)
signal
...
pause()
signal
...
uchicago
...
0
ticks = 0
def alarm_handler(signo,frame):
global ticks
print "Alarm ", ticks
ticks = ticks + 1
signal
...
signal(signal
...
alarm(interval)
# Spin forever--should see handler being called every second
while 1:
pass

O’Reilly OSCON 2000, Advanced Python Programming, Slide 58
July 17, 2000, beazley@cs
...
edu

Signal Handling (Cont)
Ignoring signals
signal
...
SIG_IGN)

Default behavior
signal
...
SIG_DFL)

Comments
Signal handlers remain installed until explicitly reset
...

Signals are only handled between atomic instructions of the interpreter
...

Certain signals can’t be handled from Python (SIGSEGV for instance)
...

Mixing signals and threads is extremely problematic
...

Signal handling on Windows and Macintosh is of limited functionality
...
uchicago
...
clock()
time
...
localtime(secs)
time
...
asctime(tuple)
time
...
mktime(tuple)
time
...

Convert time to GMT (returns a tuple)
Creates a string representing the time
Create a string representing local time
Convert time tuple to seconds
Go to sleep for awhile

Example
import time
t = time
...
localtime(t)
# Produces a string like ’Mon Jul 12 14:45:23 1999’
print time
...
uchicago
...
getpwuid(uid)
pwd
...
getpwall()

# Returns passwd entry for uid
# Returns passwd entry for login
# Get all entries

x = pwd
...
Beazley’, ’/home/beazley’,
#
’/usr/bin/csh’)

The grp module
Provides access to Unix group database
grp
...
getgrnam(gname)
grp
...
uchicago
...

Used to encrypt passwords

locale
Support for the POSIX locale functions
...


termios
Low-level terminal I/O handling
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 62
July 17, 2000, beazley@cs
...
edu

Windows and Macintosh
Comment
Most of Python’s OS interfaces are Unix-centric
...

With a number of omissions (especially in process and user management)
...

Functions to read and write characters
...

But not a substitute for PythonWin
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 63
July 17, 2000, beazley@cs
...
edu

Threads

O’Reilly OSCON 2000, Advanced Python Programming, Slide 64
July 17, 2000, beazley@cs
...
edu

Thread Basics
Background
A running program is called a "process"
Each process has memory, list of open files, stack, program counter, etc
...


Process creation with fork(),system(), popen(), etc
...

Child process runs independently of the parent
...

There is minimal sharing of information between parent and child
...


Threads
A thread is kind of like a process (it’s a sequence of control-flow)
...

A single process may have multiple threads of execution
...

Think about a browser (loading pages, animations, etc
...
uchicago
...

This can be done by the user process (user-level threads)
...


Resource Sharing
Since threads share memory and other resources, must be very careful
...


Synchronization
Threads often need to coordinate actions
...
)

O’Reilly OSCON 2000, Advanced Python Programming, Slide 66
July 17, 2000, beazley@cs
...
edu

Python Threads
Python supports threads on the following platforms
Solaris
Windows
Systems that support the POSIX threads library (pthreads)

Thread scheduling
Tightly controlled by a global interpreter lock and scheduler
...

Thread switching only occurs between the execution of individual byte-codes
...

However, most I/O operations do not block
...

Effectiveness may be limited on multiple CPUs (due to interpreter lock)
...

Not all extension modules are thread-safe
...
uchicago
...

Simple mutex locks
...
start_new_thread(func,[args [,kwargs]])
Executes a function in a new thread
...
sleep(delay)
print time
...
time())
# Start the thread
thread
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 68
July 17, 2000, beazley@cs
...
edu

The thread module (cont)
Thread termination
Thread silently exits when the function returns
...
exit() or sys
...

Uncaught exception causes thread termination (and prints error message)
...


Simple locks
allocate_lock()
...

import thread
lk = thread
...
acquire()
# Acquire the lock
critical section
lk
...

Threads block indefinitely until lock becomes available
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 69
July 17, 2000, beazley@cs
...
edu

The thread module (cont)
The main thread
When Python starts, it runs as a single thread of execution
...
"
On its own, it’s no big deal
...


Termination of the main thread
If the main thread exits and other threads are active, the behavior is system dependent
...

Cleanup actions of the main thread may be limited as well
...

Otherwise you will get an error (in the signal module)
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 70
July 17, 2000, beazley@cs
...
edu

The threading module
The threading module is a high-level threads module
Implements threads as classes (similar to Java)
Provides an assortment of synchronization and locking primitives
...


Creating a new thread (as a class)
Idea: Inherit from the "Thread" class and provide a few methods
import threading, time
class PrintTime(threading
...
Thread
...
interval = interval
def run(self):
while 1:
time
...
interval)
print time
...
time())
t = PrintTime(5)
t
...


# Create a thread object
# Start it

O’Reilly OSCON 2000, Advanced Python Programming, Slide 71
July 17, 2000, beazley@cs
...
edu

The threading module (cont)
The Thread class
When defining threads as classes all you need to supply is the following:
A constructor that calls threading
...
__init__(self)
A run() method that performs the actual work of the thread
...
join([timeout])
t
...
setName(name)
t
...
isDaemon()
t
...

Return daemonic flag
Set daemonic flag

Daemon threads
Normally, interpreter exits only when all threads have terminated
...

Interpreter really only exits when all non-daemonic threads exit
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 72
July 17, 2000, beazley@cs
...
edu

The threading module (cont)
The threading module provides the following synchronization primitives
Mutual exclusion locks
Reentrant locks
Conditional variables
Semaphores
Events

Why would you need these?
Threads are updating shared data structures
Threads need to coordinate their actions in some manner (events)
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 73
July 17, 2000, beazley@cs
...
edu

Lock Objects
The Lock object
Provides a simple mutual exclusion lock
import threading
data = [ ]
lck = threading
...
acquire()
data
...
release()
def get_obj():
lck
...
pop()
lck
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 74
July 17, 2000, beazley@cs
...
edu

RLock Objects
The RLock object
A mutual-exclusion lock that allows repeated acquisition by the same thread
Allows nested acquire(), release() operations in the thread that owns the lock
...

import threading
data = [ ]
lck = threading
...
acquire()
data
...

put_obj(otherobj)

...
release()

# Some data
# Create a lock

# Some kind of recursion

def get_obj():
lck
...
pop()
lck
...
uchicago
...

Synchronization primitive typically used when a thread is interested in an event or state change
...

# Create data queue and a condition variable
data = []
cv = threading
...
acquire()
# Acquire the lock
while not len(data):
cv
...
pop()
cv
...
acquire()
# Acquire the lock
data
...
notify()
# Notify a consumer
cv
...
uchicago
...

Each acquire() method decrements the counter
...

If the counter reaches zero, future acquire() methods block
...
Semaphore(5)
def fetch_file(host,filename):
sem
...

blah

...
release()

# No more than 5 threads allowed
# Decrements count or blocks if zero

# Increment count

O’Reilly OSCON 2000, Advanced Python Programming, Slide 77
July 17, 2000, beazley@cs
...
edu

Event Objects
Events
A communication primitive for coordinating threads
...

# Create an event object
e = Event()
# Signal the event
def signal_event():
e
...
wait()
# Clear event
def clear_event():
e
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 78
July 17, 2000, beazley@cs
...
edu

Locks and Blocking
By default, all locking primitives block until lock is acquired
In general, this is uninterruptible
...
acquire(0):
# lock couldn’t be acquired!

This works for Lock, RLock, and Semaphore objects

Timeouts
Condition variables and events provide a timeout option
cv = Condition()

...
wait(60
...
Up to caller to detect errors
...
uchicago
...
qsize()
q
...
full()
q
...
get()

#
#
#
#
#
#

Create a queue
Return current size
Test if empty
Test if full
Put an item on the queue
Get item from queue

Notes:
The Queue object also supports non-blocking put/get
...
put_nowait(item)
q
...
Full or Queue
...

Return values for qsize(), empty(), and full() are approximate
...
uchicago
...

You don’t get the degree of parallelism you might expect
...

The culprit: Not releasing global lock before starting a long-running function
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 81
July 17, 2000, beazley@cs
...
edu

Network Programming

O’Reilly OSCON 2000, Advanced Python Programming, Slide 82
July 17, 2000, beazley@cs
...
edu

Network Overview
Python provides a wide assortment of network support
Low-level programming with sockets (if you want to create a protocol)
...
)
Web programming (CGI scripting and HTTP servers)
Data encoding

I can only cover some of this
Programming with sockets
HTTP and Web related modules
...
Richard Stevens
...
uchicago
...

UDP - An unreliable packet-oriented protocol (datagrams)
...


Both protocols are supported using "sockets"
A socket is a file-like object
...

But it also includes functions to accept and establish connections
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 84
July 17, 2000, beazley@cs
...
edu

Network Basics: Ports
Ports
In order to receive a connection, a socket must be bound to a port (by the server)
...

Used to identify a particular network service (or listener)
...


Socket programming in a nutshell
Server creates a socket, binds it to some well-known port number, and starts listening
...

Server-client exchange some data
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 85
July 17, 2000, beazley@cs
...
edu

Socket Programming Example
The socket module
Provides access to low-level network programming functions
...
bind(("",8888))
s
...
accept()
# Wait for a connection
print "Got a connection from ", addr
client
...
ctime(time
...
close()

Notes:
Socket first opened by server is not the same one used to exchange data
...

listen() specifies max number of pending connections
...
uchicago
...
connect(("makemepoor
...
recv(1024)
s
...

Aside from connection process, it’s relatively straightforward
...

And are there ever a LOT of details
...
uchicago
...
)
A direct translation of the BSD socket interface
...
gethostbyname(hostname)
socket
...
ntohl(x)
socket
...
htonl(x)
socket
...

Host order may be little-endian or big-endian (depends on the machine)
...
uchicago
...

family is usually set to AF_INET
type is one of:
SOCK_STREAM
SOCK_DGRAM
SOCK_RAW

Stream socket (TCP)
Datagram socket (UDP)
Raw socket

proto is usually only used with raw sockets
IPPROTO_ICMP
IPPROTO_IP
IPPROTO_RAW
IPPROTO_TCP
IPPROTO_UDP

Comments
Currently no support for IPv6 (although its on the way)
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 89
July 17, 2000, beazley@cs
...
edu

The socket Module (cont)
socket methods
s
...
bind(address)
s
...
connect(address)
s
...
getpeername()
s
...
getsockopt(
...
listen(backlog)
s
...
recv(bufsize)
s
...
send(string)
s
...
setblocking(flag)
s
...
)
s
...

You’ll definitely want a good reference at your side
...
uchicago
...
)
Provides a series of handler classes that specify additional server behavior
...


Example
# Simple time server
import SocketServer
import time
# This class actually implements the server functionality
class TimeHandler(SocketServer
...
request
...
ctime(time
...
TCPServer(("",8888),TimeHandler)
server
...

Ex: ForkingTCPServer, ThreadingTCPServer, StreamRequestHandler, etc
...
uchicago
...

Require a good understand of the underlying protocol
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 92
July 17, 2000, beazley@cs
...
edu

The httplib Module
Implements the HTTP 1
...


HTTP in two bullets
Client (e
...
, a browser) sends a request to the server
GET /index
...
0
Connection: Keep-Alive
Host: www
...
org
User-Agent: Mozilla/4
...
6 sun4u)
[blank line]

Server responds with something like this:
HTTP/1
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 93
July 17, 2000, beazley@cs
...
edu

The httplib Module (cont)
Making an HTTP connection
import httplib
h = httplib
...
python
...
putrequest(’GET’,’/index
...
putheader(’User-Agent’,’Lame Tutorial Code’)
h
...
endheaders()
errcode,errmsg, headers = h
...
getfile()
# Get file object for reading data
data = f
...
close()

Comments
Some understanding of HTTP is required
...
0 is currently supported
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 94
July 17, 2000, beazley@cs
...
edu

The urllib Module
A high-level interface to HTTP and FTP
Provides a file-like object that can be used to connect to remote servers
import urllib
f = urllib
...
python
...
html")
data = f
...
close()

Utility functions
urllib
...
quote_plus(str)
urllib
...
unquote_plus(str)
urllib
...
quote("beazley@cs")
# Produces "beazley%40cs"
urllib
...
uchicago
...
urlparse("http://www
...
org/index
...
python
...
html’,’’,’’,’’)

urlunparse(tuple) - Turns tuple of components back into a URL string
url = urlparse
...
python
...
html’,
’bar=spam’,’’))
# Produces "http://www
...
org/foo
...
urljoin("http://www
...
org/index
...
html")
# Produces "http://www
...
org/help
...
uchicago
...

Typical uses: forms processing, dynamic content generation

How it works
You write some sort of form in your HTML document

print "

Hello World

Problem
To do anything useful, have to receive and decode "query string" from server
Which is tedious

The cgi module
Provides a variety of functions for writing CGI programs
...

Decoding query strings
Getting header information
...
uchicago
...
FieldStorage()
name = form["name"]
...
value

# Read query string
# Get ’name’ field from form
# Get ’email’ field from form

print "Content-type: text/html"
print
print "

Hello %s
...

Plus a number of security implications
...
)

O’Reilly OSCON 2000, Advanced Python Programming, Slide 99
July 17, 2000, beazley@cs
...
edu

Miscellaneous Network Topics
Modules not discussed
select - Access to the select() system call
...

asyncore - A framework for writing highly threaded servers based on asynchronous I/O
...


A few related extensions
Fnorb - A CORBA implementation for Python
...


A small plug
Python is a great language for experimenting with network programming
...

Programs are relatively simple
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 100
July 17, 2000, beazley@cs
...
edu

Data Encoding

O’Reilly OSCON 2000, Advanced Python Programming, Slide 101
July 17, 2000, beazley@cs
...
edu

Data Encoding
Problem
Data is managed in a variety of formats
...
uchicago
...
encode(inputfile,outputfile)
es = base64
...
decode(inputfile, outputfile)
s = base64
...
uchicago
...
encode(inputfile,outputfile)

Decoding
import uu
uu
...
uchicago
...
encode(inputfile,outputfile)

Decoding
import quopri
quopri
...
uchicago
...

Commonly used to encode binary files on the Macintosh
...
binhex(inputfile,outputfile)

Decoding
import binhex
binhex
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 106
July 17, 2000, beazley@cs
...
edu

RFC822 Headers
The rfc822 module
Used to parse RFC-822 encoded headers
...

Return-Path:
Date: Sat, 17 Jul 1999 10:18:21 -500 (CDT)
Reply-To: beazley@cs
...
edu
Context-Type: text/plain; charset=US-ASCII
From: David Beazley
To: guido@cnri
...
va
...


General idea:
Headers are parsed into a special Message object
...
uchicago
...
Message(f)
# Extract some fields
m_from = m["From"]
m_to = m
...
keys()
# Return a list of all header names
m
...
items()
# Return list of (header,value) pairs
m
...
getallmatchingheaders(name)
# Return list of all matching headers
m
...
getaddrlist(name) # Get a list of email addresses
m
...
uchicago
...


Packing data with pack(fmt, v1, v2,
...
pack("hhii", 34, 73, 162773, 2222)
s = struct
...
uchicago
...

Returns a tuple of unpacked values
t = struct
...
unpack("hhii",s)

Data alignment and bit ordering
First character of format string can specify encoding rules
’@’
’=’
’<’
’>’
’!’

Native byte order
Native byte order
Little endian
Big endian
Network order

Native size and alignment
Standard size and alignment
Standard size and alignment
Standard size and alignment
Standard size and alignment

Native alignment uses the size and alignment rules of the C compiler
...
uchicago
...

Commonly used in Remote Procedure Call (RPC)

MIME
The MimeWriter, multifile, mimetypes, and mimetools modules
Decoding and encoding of MIME encoded mail messages
...


htmllib
Parsing of HTML documents

sgmllib and xmllib
Parsing of SGML and XML documents
Caveat: deprecated
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 111
July 17, 2000, beazley@cs
...
edu

Restricted Execution

O’Reilly OSCON 2000, Advanced Python Programming, Slide 112
July 17, 2000, beazley@cs
...
edu

Restricted Execution
Problem
Sometimes want to run code in a restricted environment
CGI scripts
Agents
Applets

Python solution
rexec module - Restricted code execution
Bastion - Restricted access to objects

O’Reilly OSCON 2000, Advanced Python Programming, Slide 113
July 17, 2000, beazley@cs
...
edu

The rexec Module
Provides a restricted environment for code execution
Defines a class RExec that provides a controlled execution environment
Class attributes:
RExec
...
ok_builtin_modules
RExec
...
ok_posix_names
RExec
...
r_eval(code)
r
...
r_execfile(filename)

# Evaluate code in restricted mode
# Execute code in restricted mode
# Execute file in restricted more

A few methods which may be redefined
r
...
r_open(filename,mode)

# Called whenever code imports
# Called whenever code opens a file

O’Reilly OSCON 2000, Advanced Python Programming, Slide 114
July 17, 2000, beazley@cs
...
edu

The rexec Module (cont)
Example
# Create a little restricted environment
import rexec
class AppletExec(rexec
...

raise SystemError, "Go away"
r = AppletExec()
r
...

Restricted programs can’t access the __dict__ attribute of classes and instances
...


O’Reilly OSCON 2000, Advanced Python Programming, Slide 115
July 17, 2000, beazley@cs
...
edu

The Bastion Module
Problem
Sometimes a restricted program needs to access an object created in unrestricted mode

Solution
A Bastion
Basically just a "wrapper" that’s placed around the object
...


Example
import Bastion, StringIO
s = StringIO("")
# Create a file like object
sbast = Bastion
...
readline()
# Okay
sbast
...
Attribute error
...
uchicago
...
uchicago
...

You need serious performance

Extension Building
Python interpreter can be extended with functions written C
This is how many of the built-in modules work
...
uchicago
...
uchicago
...
h"
extern int gcd(int, int);
/* Wrapper for gcd */
static PyObject *
py_gcd(PyObject *self, PyObject *args) {
int x,y,g;
/* Get arguments */
if (!PyArg_ParseTuple(args,"ii",&x,&y)) {
return NULL;
}
/* Call the C function */
g = gcd(x,y);
/* Return result */
return Py_BuildValue("i",g);
}

O’Reilly OSCON 2000, Advanced Python Programming, Slide 120
July 17, 2000, beazley@cs
...
edu

Example (cont)
Step two: package into a module
/* Module ’spam’
#include "Python
...
blah
...
uchicago
...
c spammodule
...
pre
...

% cp /usr/local/lib/python1
...
pre
...


Type the following
% make -f Makefile
...
in boot
% make

This will (hopefully) create a shared object file with the module

O’Reilly OSCON 2000, Advanced Python Programming, Slide 122
July 17, 2000, beazley@cs
...
edu

Example (cont)
Step four: Use your module
linux % python
Python 1
...
2 (#1, Jul 11, 1999 13:56:44) [C] on linux
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> import spam
>>> spam
...
gcd(71,89)
1
>>>

It’s almost too easy
...
uchicago
...

Differences between platforms extremely problematic
...

Complex C++ libraries can be an even greater challenge
...
python
...
sourceforge
...
sourceforge
...
uchicago
...
uchicago
...

However, there are well over 150 standard modules in the standard library
...


Experiment!
Python is a great language for experimentation
...

This is a great way to learn about the various modules

For more information:
Python Essential Reference (shameless plug)
Online documentation (www
...
org)

Acknowledgments
Guido van Rossum
David Ascher
Paul Dubois

O’Reilly OSCON 2000, Advanced Python Programming, Slide 126
July 17, 2000, beazley@cs
...
edu



---

Title: Advance Python Programming
Description: This is the best notes for Python programming. You can learn ny your own at home.

Buy These NotesPreview