Reflection and introspection: an analysis of µWeb (part 2)

Request handler selected only by route match

While route matching itself works fine for many frameworks, the direct coupling of a route to a handling method leaves very little flexibility in the system. There is no built-in functionality to discriminate and delegate requests based on

• request method (GET, POST, PUT etc);
• match-values from the route regex;
• content type of the request body;
• accept headers (for content negotiation);
• whether or not the request is an XMLHttpRequest.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

class Application(uweb.PageMaker):
    def article(self, article_id_from_route):
        if self.req.method == 'POST':
            return self._article_update(article_id_from_route)
        elif self.req.method == 'GET':
            return self._article_view(article_id_from_route)
        # return "404 Not Found" or "405 Method Not Allowed"
        # Alternatively treat all non-POST as if it were GET

    def _article_update(self, article_id):
        # check if user is logged in and may edit
        # check form; update article or return warnings

    def _article_view(self, article_id):
        # return article

1
2
3
4
5
6
7

@view_config(route_name='article', request_method='GET')
def view_article(request):
    # return article

@view_config(route_name='article', request_method='POST', permission='edit')
def update_article(request):
    # check form, update article or return warnings

No way to assign a renderer to a request handler

The design for µWeb is such that request handler methods must return the string which contains the full response, or return a Response object. There is no framework-provided means of attaching a template renderer (or any other renderer) to a handler. This means that request methods must load and parse their templates explicitly and return the result, instead of returning a dictionary of keys to be used by the renderer mechanism in creating the response.

Separating the rendering of the template from the creation of the template input also enforces a clean separation between presentation and preparation. It keeps the code clean and allows for easier testing of what goes into the template, because it only requires validating a dictionary instead of parsing the constructed template.

@route('/hello')
@route('/hello/<name>')
@view('hello_template')
def hello(name='World'):
    return dict(name=name)

@view_config(route_name='search', renderer='full-page-result.mak')
@view_config(route_name='search', renderer='partial-items.mak', xhr=True)
def view_search(request):
    return perform_search(request.params.get('q'))

class PageMaker(uweb.PageMaker):
    def view_search(self):
        # There's combined 'params' attribute, so read the query param
        results = perform_search(self.req.get.get('q'))
        if self.req.headers.get('X-Requested-With') == 'XMLHttpRequest':
            return self.parser.Parse('partial-result.html', **results)
        return self.parser.Parse('full-page-results.html', **results)

Static content handler limitations

µWeb comes included with a handler for static content, which does a fairly good job of fulfilling the usual needs of static content serving. However, the way it’s implemented has some downsides, because of the initial design assumption that every project needs one and only one static content handler.

The static content directory defaults to static, relative to the module that contains the project’s PageMaker. If this needs to be changed, the class variable PUBLIC_DIR should be set to the desired directory.

While no second static directory can be served, multiple static paths are possible. The following route definitions lead to three subdirectories of the static directory:

ROUTES = [
    ('/(robots.txt)', 'Static'),
    ('/(images/.*)', 'Static'),
    ('/(javascript/.*)', 'Static'),
    ('/(stylesheets/.*)', 'Static'),
]

These route definitions will serve files from the following directory tree:

static/
    robots.txt
    images/
        ceilingcat.jpg
        longcat.jpg
    javascript/
        jquery-1.11.1.min.js
        application-0.2-min.js
    stylesheets/
        normalize.min.css
        application-0.1.min.css

What the sole static content handler will not allow you to do is serve content from two different root directories. If you have a situation like that, you’ll have to write your own static content handler (or adapt the one included in the source).

All handlers are forced into one class

As we established previously, µWeb’s request handlers are weighed down by code preparing template variables because of limitations in the template parser. A design decision in the presenter aspect of µWeb forces all request handlers to be part of the same class. These two limitations combined mean that non-trivial projects quickly reach a point where the PageMaker class exceeds a thousand lines of code. This doesn’t cause problems in and of itself, but it makes it more difficult to create a good mental map of the class.

“Can’t you separate things?” Well, yes you can. You can create a series of separate classes, grouped by their function within the project, and store each in their separate module. You have your main class PageMaker inherit from all of those classes and the µWeb main PageMaker and you’re on your way to success.

The downside of this is that many (eventually) sibling methods will be defined in another class, and that care needs to be taken to not accidentally define two methods of the same name. Accessing methods defined in another class (or in the framework’s provided PageMaker) will cause warnings in analytical tools and context-aware code completion will fail to work nicely.

Bad transactional support

If there is any one reason to not use the provided database model module, it’s because of its transaction management. There is transaction support, and transactions are used, but the scope of them is just plain wrong.

For a web application, the scope of the transaction is usually the handling of the request. During the lifetime of this request, the application will read from one or more tables, update some rows, and insert across multiple tables when adding an object. If any of the operations fail, we typically want none of them persisted because it leaves the system in a bad state. Pseudocode for this interaction might look a bit like this:

with transaction_factory as session:
    # Transaction begins
    bob = session.query(User, {'name': 'bob'})
    session.insert(Charge, {'user': bob, 'amount': 29.50})
    session.insert(Charge, {'user': bob, 'amount': -10})
    session.update(Balance, {'user': bob, 'amount': 19.50})
    # Transaction commits

In this scenario, should the balance updating fail because of a key constraint, or in the case that something else in the handling of the request goes wrong, the whole of the transaction will be left uncommitted. Either all actions succeed, or nothing will have happened.

In the µWeb ORM, the transactional scoping is effectively the following:

with transaction_factory as session:
    bob = session.query(User, {'name': 'bob'})
with transaction_factory as session:
    session.insert(Charge, {'user': bob, 'amount': 29.50})
with transaction_factory as session:
    session.insert(Charge, {'user': bob, 'amount': -10})
with transaction_factory as session:
    session.update(Balance, {'user': bob, 'amount': 19.50})

What happens is that each change to the system is made permanent, regardless of any errors that might happen later on. This means that if for some reason the balance update fails (or any other error happens), the two charges are still stored, leaving the database in an inconsistent state.

This means that every request handler that changes or adds data in two separate actions is a potential point of data corruption.

Relationship loading replaces the foreign key value

One of the things that makes the µWeb ORM easy to get going with is the automatic loading of (assumed) relationships. That is, for a set of tables like the following:

-- TABLE `message`
+----+--------+--------------------------------------------------+
| ID | author | message                                          |
+----+--------+--------------------------------------------------+
|  1 |      1 | First message!                                   |
|  2 |      2 | Robert'); DROP TABLE Students;--                 |
|  3 |      1 | You didn't think it would be this easy, did you? |
+----+--------+--------------------------------------------------+

-- TABLE `author`
+----+-------+-------------------+
| ID | name  | emailAddress      |
+----+-------+-------------------+
|  1 | John  | john@example.com  |
|  2 | Bobby | bobby@tables.com  |
+----+-------+-------------------+

And a model definition like this:

from uweb import model
class Author(model.Record):
  """Abstraction class for author records."""

class Message(model.Record):
  """Abstraction class for messages records."""

Accessing the 'author' key on a loaded message object will automatically query the database for the relevant author object and replace the numeric value with it, before returning the author object. This is great if you need to know something about the author, like their name or email address. But if you only needed the key value, it gets in the way and it costs a query.

Lack of output visibility

Because the standalone server redirects its output to two log files, there is no easily digested output on the console from where the application is run. To get the desired output you’ll have to find the output files for the daemon and tail [5] them. The daemon outputs are stored in one of two locations (in order of preference):

• /var/log/underdark/{package}/
• ~/.underdark/{package}/logs/

Another pain due to this forking nature is that the output of the startup progress is reported to the redirected output file. This means that on the terminal you started it from, there is zero feedback on whether the project started successfully or not. Nor does it tell you the port the project is served on.

Lack of plaintext logging

The stdout and stderr log files by default do not contain the output of the logging module. µWeb redirects these to its own SQLite database (stored in the same location), which is not simply viewable by tailing. An application to browse and filter these databases comes bundled with µWeb, providing much-needed access to the logs. Running uweb start logviewer starts a daemon that listens on http://localhost:8001/, which serves the log viewer.

The lack of plaintext logging means that the developer has to actively refresh the page of the log viewer (there is no automatic updating system for it). It also means that quick debugging with print statements is less effective because the log database and stdout file need to be correlated. And while a log statement is not that much more to write, it does take the speed out of debugging, making the lack of an interactive debugger that much more apparent.

Lack of automatic reload

The µWeb standalone server lacks an automatic reloading mechanism. This means that whenever code has changed, the server needs to be manually restarted. Most modern frameworks come with a command line option that allows for automatic reloading.

Template files are automatically reloaded when they have been changed, though this is a feature of the template system, not the standalone server.

Daemonization makes management difficult

The PID of the server process is not communicated, nor is its location. The storage location based on the package name and the router name, and cannot be defined by the user. The storage location is /var/lock/underdark/{package}/{router}.pid. The indirect way in which the uweb script starts a web project makes it impossible to track with a system like Upstart, and probably other similar task managers. See the Upstart appendix for a solution on how to manage µWeb projects with it.

Appendix A: Making standalone play nice with Upstart

When you’re trying to make µWeb’s standalone server play nice with Ubuntu’s Upstart, you’re going to run into some problems. Upstart supports managing (double-forking) daemons, but starting a project with the uweb script triggers 4 forks: [6]

• 1: Python interpreter for the uweb script (coming from the shell)
• 2: A subprocess call to load the project’s router module and start it
• 3 & 4: Double fork to daemonize the standalone process

Upstart expects [7] only up to two forks to happen, so it won’t track the resulting process. This means that starting a project this way will cause it to start (successfully), while Upstart believes it failed to start. This makes stopping or restarting it using Upstart impossible.

To make the standalone server work well with Upstart, the server starting usually performed by the uweb script must be placed in hte job configuration script. Assuming we want to start the µWeb logviewer from a virtualenv installed in /usr/local/newweb/env, the script looks like this:

description "uweb-logviewer"

manual
respawn
console log
env PYTHON="/usr/local/newweb/env/bin/python"
env ROUTER="uweb.logviewer.router.logging"
expect daemon

exec $PYTHON -m $ROUTER start