Eventlet

From Second Life Wiki
Revision as of 10:52, 25 August 2007 by Which Linden (talk | contribs) (→‎Limitations: rephrase)
Jump to navigation Jump to search

Eventlet

Eventlet is a networking library written in Python. It achieves high scalability by using non-blocking io while at the same time retaining high programmer usability by using coroutines to make the non-blocking io operations appear blocking at the source code level.

Releases

Name Date Zip SVN
beta-1 Aug 24, 2007 [1] [2]

Requirements

Eventlet runs on Python version 2.3 or greater, with the following dependenceis:

  • greenlet
  • (if running python versions < 2.4) collections.py from the 2.4 distribution or later

Eventlet at Linden

eventlet is the networking library used to implement the backbone architecture. Backbone is primarily an http server, and as such uses the eventlet.httpd module. eventlet itself is responsible only for handling http protocol-level semantics; "web framework" concepts such as URL traversal and html rendering are implemented in a separate python package, mulib.

Files

Files with a corresponding _test.py unit test are called out in bold in these lists.

API

  • api.py : Exposes all of the most important apis from eventlet, such as apis for managing threads of control (api.spawn) and apis for managing network sockets. Contains 9 apis intended for external use.
  • backdoor.py : Implements a Python REPL over a raw socket, suitable for use by telnetting in to the port and screwing around with the live state of the application.
  • channel.py : Channel is the main scheduling primitive. It is a Queue pattern with send and recieve methods. It is used for synchronizing across coroutines.
  • coros.py : Coroutine communication tools, such as events (like channels, but only for a single use), and a coroutine pool.
  • httpc.py : Nonblocking http client, and a client factory that can make content-swizzling clients.
  • httpd.py : HTTP server written by Donovan. Should probably be merged with wsgi
  • httpdate.py : Very small utility module for formatting http dates. This module should grow http date parsing code and code for dealing with conversion to and from utc as well, but currently does not.
  • jsonhttp.py : Http client that transparently converts python data structures to and from json.
  • pools.py : Implements pooled resources. Coroutines that try to get items from a depleted pool block until another coroutine puts a resource back in the pool.
  • processes.py : Implements popen-like process objects for communicating with external programs.
  • wsgi.py : An implementation of a wsgi (web services gateway interface) (http) server. Not used by linden; should be merged with httpd.


Core implementation details

  • greenlib.py : Private implementation module which adds the concept of a "tracked" greenlet: A greenlet that is notified as it is being scheduled or unscheduled. Used by backdoor to swizzle stdin and stdout during context switches so that input and output always goes to the right place.
  • runloop.py : Generic runloop which when combined with the appropriate "hub" for the current architecture forms the main loop of eventlet. Manages lists of observers for the runloop states: ['entry', 'before_timers', 'before_waiting', 'after_waiting', 'exit'] along with managing all timers.
  • timer.py : Timers track code which is scheduled to be run after some timeout. Usually this module is not used directly, but is used indirectly through the api.sleep call
  • util.py : A bunch of utilities for operating on socket objects. Could probably be factored better.
  • wrappedfd.py : A socket object which cooperates with the main loop to yield control of the current coroutine each time a socket read or write operation occurs.


"EventHub" implementations

  • kqueuehub.py : An event hub which uses kqueue. Used on OS X and FreeBSD.
  • pollhub.py : An event hub which uses the poll() call.
  • selecthub.py : An event hub which uses the select() call.


Misc

  • logutil.py : Mish-mash of utilities for simplifying the use of the python standard "logging" module and for logging using syslog.
  • tests.py : A place to keep test utilities. Lightly used, but it will become more useful as test coverage grows.
  • tls.py : Thread-local storage implementation for Python 2.3.


Supporting modules

Limitations

  • Sorely lacking in documentation
    • Please help by adding to the wiki documentation, and submitting patches with more and better docstrings for the source.
  • Not enough test coverage -- the goal is 100%, but we are not there yet.
  • Eventlet does not currently run on stackless using tasklets, though it is a goal to do so in the future.
    • If you're familiar with the stackless architecture, take a look at stacklesssupport.py and see if you can wrangle it to get one of the example programs running on top of stackless.
  • The SSL client does not properly connect to the SSL server, though both client and server interoperate with other SSL implementations (e.g. curl and apache).
    • We could use a) a unit test that reproduces the bug (rather than the hand-testing we have been doing) and b) a fix for said bug.
  • Not tested on Windows
    • There are probably some simple Unix dependencies we introduced by accident. If you're running Eventlet on Windows and run into errors, let us know.

Eventlet History

Eventlet began life as Donovan Preston was talking to Bob Ippolito about coroutine-based non-blocking networking frameworks in Python. Most non-blocking frameworks require you to run the "main loop" in order to perform all network operations, but Donovan wondered if a library written using a trampolining style could get away with transparently running the main loop any time i/o was required, stopping the main loop once no more i/o was scheduled. Bob spent a few days during PyCon 2005 writing a proof-of-concept. He named it eventlet, after the coroutine implementation it used, greenlet. Donovan began using eventlet as a light-weight network library for his spare-time project Pavel, and also began writing some unittests.

When Donovan started at Linden Lab in May of 2006, he added eventlet as an svn external in the indra/lib/python directory, to be a dependency of the yet-to-be-named backbone project (at the time, it was named restserv). However, including eventlet as an svn external meant that any time the externally hosted project had hosting issues, Linden developers were not able to perform svn updates. Thus, the eventlet source was imported into the linden source tree at the same location, and became a fork.

Bob Ippolito has ceased working on eventlet and has stated his desire for Linden to take it's fork forward to the open source world as "the" eventlet.

Retrieved from "https://wiki.secondlife.com/w/index.php?title=Eventlet&oldid=29270"