Building an aggregator - indexing content (with PyLucene, ultimately)

09 May

Tags: , , , , , , , , ,

Another important aspect of building an aggregator, besides tags/keywords, is being able to find posts based on particular words in the text.  In the "assigned keywords" vs. "words in the page" battle, "words in the page" still has the lead.

This is typically done using a "full text index" of the content - of a post in our case.  There are two main ways to do this - either in your standard database (MySQL's MATCH ... AGAINST construct, or tsearch2 for PostgreSQL), or in an external indexing system just for this sort of indexing and searching, using something like Lucene or Xapian.

There are some interesting trade-offs here...

Hi, I'm Neil Blakey-Milner, a technology generalist most interested by open source, security, and building scalable systems (especially if I get to use the Python programming language).

I "work" at the incredibly fun web-sites-for-everyone startup SynthaSite based in beautiful Cape Town, South Africa.

What I do

  • StarCamp

    I was the main instigator behind this 2-day unconference in December 2007 in Cape Town. There'll be another one in Cape Town around July.

  • GeekDinner

    I help out in making these geek social events happen every two months in Cape Town since March 2007.

  • SynthaSite

    I make sure that the systems run smoothly at this Cape Town-born Web 2.0 startup.

  • Presentations

    I speak publically on a number of topics - generally related to technical aspects of open source software or on software development.

  • Code

    I've written a few software projects of my own over the years (although mostly I'm the only user of them).

  • Photos

    I'm far from an artist, but I enjoy capturing my experiences on camera.

Recent Entries