Python Pooled threading 101.

Hey’a guys – Today I’m going to show you how to write a single pooled multi-threaded program in python.

I’ve been doing pooled python for a while but I wasn’t sure if there was any good libraries out there for it, so I jumped on google and looked at everything I could find.

I ended up finding a good module called “threadpool”

ThreadPool allows for an easy enough implementation of multithreading whilst incorporating high-level interfaces that provide easy control.

So lets get started:

firstly, using easy_install-2.6 lets get thread pool:

# easy_install-2.6 threadpool

Now lets implement it in our test.py:

# vim test.py

Paste in the following code:

import threadpool
import time

def long_process(time_length):
    print( 'yo process with arguments %d starting' % time_length)
    time.sleep(time_length) 
    print( 'Ended process with argument %d' % time_length)

def process_tasks():
    pool = threadpool.ThreadPool(4)

    argument_list = [1, 6, 2, 4, 5, 6, 7, 8]

    #let threadpool format your requests into a list
    requests = threadpool.makeRequests(long_process, argument_list)

    #insert the requests into the threadpool
    for req in requests:
        pool.putRequest(req) 

    #wait for them to finish (or you could go and do something else)
    pool.wait()

if __name__ == '__main__': 
    process_tasks()

Now save that file and lets run it:

#python2.6 test.py

You should see a nice output:

Starting process with argument 1
Starting process with argument 6
Starting process with argument 2
Starting process with argument 4
Ended process with argument 1
Starting process with argument 5
Ended process with argument 2
Starting process with argument 6
Ended process with argument 4
Starting process with argument 7
Ended process with argument 6
Starting process with argument 8
Ended process with argument 5
Ended process with argument 6
Ended process with argument 7
Ended process with argument 8

as you can see the pooled threads startup nicely.

In my next post I will be breaking down the thread pool.

Cheers,
Karl.