Today was not a nice day. In between servers throwing wobblers, the internet breaking (I sank teh Intarweb!) and the general hubbub of office life, my mail archiving program decided it was going to creak under pressure and stop our mailserver from coping with a copious mail influx.
I’m actually rather proud of my little archive system. It uses a PHP frontend with AJAX goodness to search any email that has come in or out of the building and then forward said email to whomever we please or download the attachment(s). The search through about 5GB of emails (that doesn’t include the attachments!) takes about 5 seconds at most.
The data is all stored in MySQL and is pumped in using a python extension to our mailserver software, vPop3.
The problem that caused a few emails to get stuck in a big old queue was that the python extension has to connect to the mysql database from scratch every time vpop processes an email. Under heavy load this can take up to 5 seconds per email and when vpop is accepting 4 or 5 emails a second there is soon a huge buildup.
The solution is simple: Make the extension that is run by vpop place the email to be archived in a spool folder of it’s own and let vpop do it’s thing asap. That way I can have a seperate process run once a minute (or run as a service if I decide to get really smart) to parse and archive the spooled mail to the database.
I’m right in the middle of splitting the current incarnation into two components and having one process run all the time. I’m also planning to release it to the community at large. Even though the user base for vpop is relatively small, I still think it will be of help to smaller businesses.
anxious | 28-Sep-05 at 10:05 pm | Permalink
I don’t know what you’re talking about, but it’s good to see you back in Blogland, Spleeman ;)
Now, what’s all this pink in aid of?
Splee | 28-Sep-05 at 11:17 pm | Permalink
Hey Little Miss Anxious, hows things?
The pink is in aid of a whim. I was looking for a colour scheme and pink was the scheme I chose. :)
drac | 29-Sep-05 at 2:48 pm | Permalink
That description of the system - especially the archive browsing stuff - sounds extremely neat. I can think of places besides small businesses that could benefit. Nice work!
Splee | 29-Sep-05 at 8:44 pm | Permalink
Thanks drac.
That idea isn’t bad actually. I could develop different backends and the front end wouldn’t need to change; an email is an email after all…
As it is this will only work with vPop3. I don’t see myself doing a lot of work for other mailservers, although doing a free one for exchange would be fun, just to piss of the big boys. :D