Writing a comment format plugin system in Python

When I first started writing gibe, I wasn't too concerned about much beyond the adventure of writing my own software.  In fact, I'm still not particularly concerned about more than that.  But my initial three-second decision and implementation of TinyMCE for comments has caused at least as much trouble as I expected it to, and I've now crossed off the "make it so that different comment formats are supported" item in my checklist.

Gibe was always intended to have a plugin architecture.  Plugins would add new URLs to the routes router, allowing me to add RSS, tags pages, and so forth, all without changing any core code.  Plugins would register themselves as using pkg_resources.  It would be a sunny, but cool, day at the beach with the dogs.  And so forth.

But until that wonderful day (I'm hoping this weekend), I'll have to settle with the comment format problem.

Initially, the comment form was generated using TurboGears's wonderful widgets system (and will soon use its heir, ToscaWidgets):

class CommentFormFields(widgets.WidgetsList):
    name = widgets.TextField(validator=validators.NotEmpty,
        label = "Your name", attrs=dict(size=60))
    email = widgets.TextField(
        validator=validators.All(validators.Email, validators.NotEmpty),
        label = "Email", attrs=dict(size=60))
    website = widgets.TextField(validator=validators.URL,
        label = "Site", attrs=dict(size=60))
    comment = TinyMCE(validator=validators.NotEmpty, label = "Comment",
        mce_options = dict(
            mode = "exact",
            theme = "advanced",
            plugins = "fullscreen",
            relative_urls = False,
            theme_advanced_buttons2_add = "fullscreen",
            extended_valid_elements = "a[href|target|name]",
            paste_auto_cleanup_on_paste = True,
            paste_convert_headers_to_strong = True,
            paste_strip_class_attributes = "all",
            theme_advanced_buttons3 = "",
            remove_linebreaks = False,
            browsers = "msie,gecko",
        ),
        rows=15,
        cols=60,
    )
    postid = widgets.HiddenField()

comment_form = widgets.TableForm(fields=CommentFormFields(), submit_text="Post")

The comments were displayed directly from the model from the templates, assuming HTML output:

        <div class="blogcomment" py:for="comment in post.comments" py:if="comment.approved">
        <div id="au${comment.comment_id}">
        <p><span class="blog_comment_post_time" 
          py:content="comment.posted_time.strftime(str('%B %d, %y %X'))">September 20, 2006 22:00</span>
        |
          <span class="reference" py:content="tg.ET(comment.getReference())"><a href="http://www.greenman.co.za/b2evolution/blogs/">Ian</a></span></p>
        <p py:content="HTML(comment.content)">Any idea what the other three were using?</p>
        </div>
        </div>

Next step - add a getContentHtml method to the model (could've been a property, I suppose. Can always change later...), and use that from the template. First, just the plain method for the current case:

    def getContentHtml(self):
        return self.content

Then, add a content_format column to my Comment model (I'm currently using ActiveMapper on SQLAlchemy, but I'll be moving to the Elixir declarative layer over SQLAlchemy soon), defaulting to html if no content_format is provided, so that a default conversion to HTML can be done for it:

class Comment(ActiveMapper):
    class mapping:
        comment_id = column(Integer, primary_key=True)
        post_id = column(Integer, foreign_key=ForeignKey('post.post_id'))
        blog_id = column(Integer, foreign_key=ForeignKey('blog.blog_id'))
        author_id = column(Integer, foreign_key=ForeignKey('user.user_id'))
        author_name = column(Unicode(255))
        author_url = column(String(255))
        author_email = column(String(255))
        content = column(Unicode())
        posted_time = column(DateTime)
        approved = column(Boolean, default=False)
        content_format = column(Unicode(25), default = "html")

I decided to use the dispatch module (sorry, that's the best link I could find) to decide how to convert from the comment's content in its format to HTML - it allowed me to easily provide alternate implementations in the same place before going fully into the plugin space:

    @dispatch.generic()
    def getContentHtml(self):
        pass

    @getContentHtml.when('self.content_format == "html"')
    def getContentHtmlfromHtml(self):
        return self.content

Everything still works, but there's no way to change the comment form field, and no way to specify the comment format and do anything necessary before writing it to the database. I now added an entry point, a means by which modules can advertise themselves (or their classes or functions) as available for a particular topic. In this case, a comment format "topic", which is basically a registry of comment formats and how to deal with them. First, edit setup.py to add an entry point for the current TinyMCE-based system:

    entry_points = """
        [gibe.comment_formats]
        tinymce = gibe.tinymcesupport
        postmarkup = gibe.postmarkupsupport
    """,

I'm probably committing a major faux pas, but I read the entry points into a simple format registry dictionary:

format_registry = {}
for comment_format_mod in pkg_resources.iter_entry_points("gibe.comment_formats"):
    mod = comment_format_mod.load()
    format_registry[comment_format_mod.name] = mod

Now the comment form needs to be updated so that it can use whichever format it is configured to use in the configuration file. The comment form fields need to be modified by the comment format plugin, and a content_format field needs to be added to the comment form too (and should be validated to be a format we can handle). This turns out to be pretty easy:

def fields():
    formats = []

    formats_to_check = [
        cherrypy.config.get('gibe.comment_format.preferred'),
        cherrypy.config.get('gibe.comment_format.fallback', None),
        ['postmarkup', 'tinymce'],
    ]

    for fs in formats_to_check:
        if isinstance(fs, (str, unicode)):
            fs = [fs]
        for f in fs:
            if f not in formats:
                formats.append(f)

    for format in formats:
        if format in format_registry:
            class CommentFormFields(widgets.WidgetsList):
                postid = widgets.HiddenField()

                content_format = widgets.HiddenField(default=format,
                    validator=validators.OneOf(format_registry.keys()))

                name = widgets.TextField(validator=validators.NotEmpty,
                    label = "Your name", attrs=dict(size=60))

                email = widgets.TextField(label = "Email",
                    validator=validators.All(validators.Email, validators.NotEmpty),
                    attrs=dict(size=60))

                website = widgets.TextField(validator=validators.URL,
                    label = "Site", attrs=dict(size=60))

            comment_form_fields = CommentFormFields()

            format_registry[format].addCommentFields(comment_form_fields)
            return comment_form_fields

comment_form = widgets.TableForm(fields=fields(), submit_text="Post")

On the TinyMCE format plugin side, the addCommentFields function adds the TinyMCE form field to the widgets list:

from tinymce import TinyMCE

from turbogears import widgets, validators
def addCommentFields(wl):
    class CommentFormFieldsExtra(widgets.WidgetsList):
        comment = TinyMCE(validator=validators.NotEmpty, label = "Comment",
            mce_options = dict(
                mode = "exact",
                theme = "advanced",
                plugins = "fullscreen",
                relative_urls = False,
                theme_advanced_buttons2_add = "fullscreen",
                extended_valid_elements = "a[href|target|name]",
                paste_auto_cleanup_on_paste = True,
                paste_convert_headers_to_strong = True,
                paste_strip_class_attributes = "all",
                theme_advanced_buttons3 = "",
                remove_linebreaks = False,
                browsers = "msie,gecko",
            ),
            rows=15,
            cols=60,
        )
    wl.extend(CommentFormFieldsExtra())

Almost done with the core code - the add comment method needs to allow the comment format plugin to convert, sanitise, or reject the incoming comment. In the original, the Genshi HTMLSanitizer filter is used to sanitise the HTML. I decided that the plugin modules would have a class named Commenting available for doing conversion and rejection and possibly for post-save actions. (This also allows for modules to convert from a format to HTML, and then change the content_format variable to html, and not have to write their getContentHtml implementation.)

    @error_handler(post)
    @validate(form=comment_form)
    def add_comment(self, blog, **kw):
        post = Post.get_by(post_id = kw['postid'])
        if not post.accept_comments:
            flash("Post does not allow comments")
            raise routes.redirect_to('posts', post = post)

        content_format = kw['content_format']
        commenting = format_registry[content_format].Commenting()

        commenting.convert(kw)

        ckw = {
            'post_id': kw['postid'],
            'blog_id': blog.blog_id,
            'author_name': kw['name'],
            'author_url': kw['website'],
            'author_email': kw['email'],
            'content': kw['comment'],
            'approved': True,
            'posted_time': datetime.now(),
            'content_format': kw['content_format'],
        }

        c = Comment(**ckw)

        commenting.post_save(kw)

        self._check_for_spam(blog, c, kw)

        raise routes.redirect_to('posts', post = post)

Implementing this again is quite trivial:

import commenting
from gibe.util import sanitise
class Commenting(commenting.Commenting):
    def convert(self, kw):
        kw['comment'] = sanitise(kw['comment']).decode('utf-8')

And that's about it for the core code, and recreating the TinyMCE support. Of course, the whole point was to offer other comment formats, and so I implemented the newly created postmarkup module which provides BBCode-like support for user-supplied data in a controlled fashion.

First, the postmarkup object needs to be created:

pm = postmarkup.PostMarkup().default_tags()
pm.add_tag('link', postmarkup.LinkTag, 'link')
pm.add_tag('quote', postmarkup.QuoteTag)
pm.add_tag('code', CodeTag)

Then, a function to convert from postmarkup to HTML:

from gibe.model import Comment
@Comment.getContentHtml.when('self.content_format == "postmarkup"')
def getContentHtmlPostMarkup(self):
    return pm.render_to_html(self.content.encode('utf-8')).decode('utf-8')

An addCommentFields function:

def addCommentFields(wl):
    class CommentFormFieldsExtra(widgets.WidgetsList):
        comment = widgets.TextArea(label = "Your comment", rows=15, cols=45,
            validator=validators.NotEmpty,
        )
        explanation = PostMarkupExplanation()
    wl.extend(CommentFormFieldsExtra())

No conversion necessary to write to the database, so just an empty Commenting class:

class Commenting(commenting.Commenting):
    def convert(self, kw):
        pass
    def post_save(self, kw):
        pass

For bonus points, I added an explanation to the add comment form so that people know how to format their comments:

from turbogears import widgets, validators
class PostMarkupExplanation(widgets.FormField):
    template = """
    <div xmlns:py="http://purl.org/kid/ns#"
        class="${field_class}"
        id="${field_id}"
    >
        <p>The text area above accepts Post Markup, a BBCode work-alike.</p>

        <pre>
[b]foo[/b]: <strong>foo</strong>
[i]foo[/i]: <em>foo</em>
[link]http://nxsy.org/[/link]: <a href="http://nxsy.org/">http://nxsy.org/</a> [nxsy.org]
[link http://nxsy.org/]Neil[/link]: <a href="http://nxsy.org/">Neil</a> [nxsy.org]
        </pre>

        <p>You can also use:</p>
        <pre>
[code python]
import foo
[/code]
        </pre>
    </div>
    """

Add it to the entry_points in setup.py, and we're done.