The last few years a lot more people have been focusing on async IO in Python. This is also referred to as “non-blocking IO”. Gevent is one project that makes it easy to do this sorta stuff. How it works is simple, you spawn up “greenlets” which are sorta like threads but very lightweight. You are spawning these in a single Python process, and single thread (which because of the GIL really can only have one synchronous flow of execution anyhow). So this normal Python process will start processing your greenlet(s) and when one hits a point where IO is performed, say it goes out and talks to Redis, execution yeilds back while IO is being performed and another greenlet is picked and then put into the running state. If you had a bunch of greenlets all waiting for IO then they could come back and start processing as soon as one got it’s output. The important thing to grasp here is only one process, and single thread of execution is happening, but if one of the greenlets hits IO rather than block we can start executing some other code, and process the piece of code waiting for IO to return when it is ready.
If you’re looking to try Greenlets and gevent and you are on a mac you should be able to follow these directions.
First install libevent via Mac Ports. If you do not already have Mac Ports then first install it. I hear you can also use homebrew but I’ve been a ports user and it’s more like the apt-get I’m used to.
sudo port install libevent
Now you should be ready to install gevent (which’ll install greenlets).
First set this flag so that gevent can build:
export CFLAGS=-I/opt/local/include
That’ll point the C compiler to the ports folder where the event.h (and other) files are.
I created a virtualenv for this project, therefore to install gevent I simply did a:
pip install gevent
After which I had gevent and greenlets ready to go! I advise you go try this example to get a very simple start using greenlets and gevent.
