There are four main scripts that I’m using to make things nice. My Where’s Seth script is back, but now instead of just pulling from my Google Calendar, it also works off Jabber away messages that my Trillian client sends. I need to make it more robust, so we’ll see what other ways I can pull information, such as Twitter feeds. For my linkblog, I rolled a little bit of jQuery to fetch JSON data from Ma.gnolia (which is like del.icio.us but nicer), so I don’t have to maintain links in two separate places.

On the WordPress plugins side, I’ve added OpenID and Gravatar support. WP-OpenID allows commenters to use their OpenID, both to verify their identity and to obviate having to type all that commenter info every time. Gravatar was acquired by WordPress, which makes me finally comfortable enough to use it on a blog. Before, it was just too unstable– the service was unavailable all the time and I didn’t feel like it was “official” enough to make it widespread and worth using. With official adoption by WordPress, it should become more of a de facto standard, one that I feel comfortable supporting.

Javascript is fun :O

It was one thing after another. “I’ll do a new blog when ZetaBoards is done.” ZB got done (or at least I was done with ZB). “I’ll do a new blog when I have a lull in coursework.” Physical Chemistry was a beast, and I started my research this semester. “I’ll do a new blog when winter break comes.” I start studying for the MCAT. But at long last, I gave in to Nicola’s pleadings and set aside time to hash out a new blog… because if I didn’t get it done soon it was never going to happen!

This is the first time I’ve had someone wonderful like Nic to work with on a personal design. She made theming ZetaBoards so much easier; I can code, but I have little color sense and less graphical ability. But wow, was this blog design so much easier with her. I just needed to lay out a general idea of what I wanted (sometimes with MS Paint illustrations!) and she would go to work. As I coded the HTML and CSS, she would supply me with graphics to drop in place. And this is the result– a blog that has more eye candy than my first blog (css-only, circa 1998), second blog, (maybe a background… remember the black-on-scanlines?), or even third blog (modified Kubrick). It’s truly a treat.

One of my New Year’s resolutions is to follow Sam in blogging every day. So let’s see if I can’t bring this site back from the dead ^_^ It isn’t fully functional yet, as there are no comments or pages, and some things are broken. But I can’t stand not blogging anymore, so let the posts begin.

I had been lazy for a couple weeks about sitting down and inputting my new classes. So today, I fired up Google Calendar and input my schedule items just for fun. It features XML and iCal export, so I thought I could hack up an equivalent version of the Where’s Seth? script using that.

I first tried just pulling the XML data and parsing that with my script. However, it wasn’t RSS, but Atom. I didn’t feel like rewriting everything to handle Atom XML, so I then investigated the iCal format. Googling “PHP iCal parser” got nothing decent to turn up (I refuse to register at crappy sites to download questionable libraries), but searching for “parse iCal” showed a JavaScript solution. That still wasn’t ideal, but I read the comments of that blog entry to find that there was a WordPress iCal parser! It was easily to grab and install, but it only showed events that had already passed. So I hacked it up a little bit to accept feeds from multiple iCals and to only show an event if one was currently in progress; it only took about an hour from start to finish. And now I have a slick Google Calendar that I’ll be a lot more likely to keep updated than I did the silly local iCal

This, my friends, is Stalking Done Right™.

So while on the topic of spam, Scotty asked me today about fighting e-mail spam more effectively. Even though I’ve used SpamAssassin for a couple years, I would still get a few hundred spam e-mails a week that made it through the filter. Thunderbird would collect about 90% of those, which meant that I didn’t have to look at a lot of spam… but still far more than I would have liked. Plus, any time I looked at my mail from a webmail interface, I had to sort through a few dozen spams.

I read up a couple months ago on SpamAssassin tweaking. Let me summarize some steps I took that caused the number of spam e-mails I get a week to fall to under 10.

1) Training Camp

The quick, two-line explanation of how SpamAssassin works: it looks for various signs of a spam e-mail (the word “Viagra”, e-mail sent to a large number of recipients, spoofed headers, etc.) and for each one it finds, it adds a varying number of “points” to the e-mail based on how spam-like the behavior is. If an e-mail gets too many points, it’s spam! Most SA installations learn by default what “strong spam” and “strong ham” are (those with very high positive and negative scores) and thus are able to adapt to your own personal spam preferences. However, it’s that remaining 20% of e-mails, the “weak spam”, with which you need to lend SpamAssassin a hand.

The sa-learn command allows you to show SpamAssassin additional examples of spam it didn’t catch the first time around, thus improving its Bayesian filtering. I use this small shell script to train it every so often:
#!/bin/sh
echo "Learning from Ham..."
sa-learn --ham --progress --mbox ~/mail/seth/inbox
echo "Learning from Spam..."
sa-learn --spam --progress --mbox ~/mail/seth/Junk

You need to train equal amounts of ham and spam for optimum results, according to the manual. So I train SA for ham on my inbox, and spam on Thunderbird’s Junk folder.

2) You make the rules

Most e-mail programs let you view the full e-mail headers, where you’ll find a line like this:
X-Spam-Status: No, score=4.9 required=5.0 tests=ADDRESS_IN_SUBJECT,BAYES_50, NO_REAL_NAME autolearn=no version=3.1.7

The three tests in the header are the ones this particular message failed. It included an e-mail address in the subject line, the Bayes Filter gave it a 50% probability of being spam, and the e-mail address it was sent from did not include a name to go with the address. This gave it a score of 4.9. To be marked as spam, it needs a score of 5.0. In this case, the message was not spam, so SpamAssassin was correct.

The scoring system on a default SpamAssassin installation is very different. It’s a lot harder for e-mails to be marked as spam by default, because SpamAssassin wants to play it safe and make sure it doesn’t trash any of your real e-mails. This is a good goal, but if you’ve been training it for a few weeks then it’s much safer to bump the scores for some of the bad tests higher.

Don’t change these scores until you’ve looked at a good sample of your own mail to see how SpamAssassin is scoring it. If you subscribe to some newsletters you like, make sure they’re not being marked as spammy, or you’ll need to take some additional steps to protect them. If the spam you receive isn’t failing any SpamAssassin tests to begin with, it won’t matter how high you raise their scores.

To change the scores of spam tests, edit the file ~/.spamassassin/user_prefs (~/ is your home directory). I looked at some of the spam that was getting through and saw three tests most often:

• FORGED_RCVD_HELO
• HTML_SHORT_LINK_IMG_1
• BAYES_90

(A giant list of all SpamAssassin’s tests is on the website.) These 3 tests would be a good place to start fighting spam, by raising their point value. I added these lines to the bottom of my user_prefs file:
score BAYES_90 5.0
score FORGED_RCVD_HELO 2.0
score HTML_SHORT_LINK_IMG_1 3.5
Since I’ve been training SpamAssassin, if Bayes says there’s a 90% chance of the message being spam, that’s good enough for me. I’ll give it the whole 5 points it needs to be marked as spam. Forged HELOs are still common from places like my university’s mailing system, so I can’t afford to give it a very high score. And messages that are nothing but a linked image are pretty shady, so they get a high score.

3) Quality Assurance

For the first few weeks, take some time to review the spam folder that SpamAssassin dumps all your spam into, to make sure that one of your rules didn’t mistakenly cast its net too far. If you see some of your mail dropping into spam, just move it back to the inbox and train SpamAssassin again. You can also adjust your rules to make sure you don’t weight something too heavily. The spam folder gets pretty large, so you’ll want to clean it out every once in awhile. You can just delete it and SpamAssassin will recreate it the next time you get spammed. I just added a line to my training script above that cleans out /mail/spam while it trains on the other folders.

Following these steps will net you an easy 90% reduction in the amount of spam you get, and over time it will get even better, thanks to Bayesian filtering. Plus, not having to download the mail at all will make your mail reading in Thunderbird or other e-mail clients even faster and more enjoyable.

I installed WP-Hashcash as a helper to Akismet. So far, no more comment spam. I usually get 3,000+ a day, as Akismet shows…

Akismet has caught 268,126 spam for you since you first installed it.

I finished up a bunch of miscellaneous tweaks to the blog to tide me over until I can finally get a new theme up, with Nic‘s help. Poke me if commenting suddenly broke horrendously, but I did try to test it. I also finally got around to backing up all my websites. Since that includes all of ZetaBoards’ skin files and graphics, as well as a few thousand dollars worth of projects I host for various companies around town… yeah. I should have done that a long time ago. Hopefully they aren’t reading this

The first link in the new linkblog happens to be YouMail, which replaces your cellphone’s normal voicemail service and adds some spiffy features. You can record greetings for each different phone number that calls you (but of course record a default greeting for everyone else to hear), and you can listen to and delete / save your voicemail messages from your computer. Very cool, and free (it does use cellphone minutes to check your voicemail, but I have plenty of those). Check it out.

I went shopping for a refrigerator today for my dorm room, but came back empty-handed. Too bad. However, I am keeping up my inexorable pace of one blog post per day rather nicely, if I do say so.

Well, as with all leadingly rhetorical questions, the answer is of course yes. OpenID allows you to set up a single account with a single PIP, and then use authentication tokens to allow that site to distribute your information (partially or fully) to requesting sites. The coolest part is that you can set up your own site as an OpenID server or delegate, which means your login name can be yourdomain.com. If the PIP you are using changes its TOS or even goes under, you can switch PIPs with just a few lines of HTML and keep your same ID (assuming you hold on to your domain name).

OpenIDs are so ridiculously easy to set up that everyone should really get ahold of one so that adoption becomes more widespread. As more and more sites start to accept it as a form of login (and my site will do so soon), it will become easier and easier to use your existing credentials rather than create more. Here’s one of the easiest ways to create your very own OpenID.

Step 1: Sign up with an OpenID Personal Identity Provider
The beauty of OpenID is its decentralization. It doesn’t matter who stores your identity, and you can change hosts at any time if you find a better one. I found Verisign Labs PIP, which has all the features I need and is pretty clean and trimmed-down. I created an account seth, which made my PIP Identity URL seth.pip.verisignlabs.com.

Other PIPs if you prefer:

• LiveJournal accounts all include OpenID
• Vox
• MyOpenID has some cool features like multiple identity support

Step 2: Point your website to the PIP (optional)
You only get half the benefit of OpenID if you keep the PIP’s URL for yourself. Then if you want to change PIPs, you’ll have a new URL to log in with. For those without their own vanity site, this may be a better option, however.

Go to the homepage of your website and pop it into an editor. You’ll need to insert two lines into the <head> of the document.
<link rel="openid.server" href="http://pip.verisignlabs.com/server/" />
<link rel="openid.delegate" href="http://you.pip.verisignlabs.com/" />

These URLs delegate your authentication to another site. The openid.server tells sites the URL of your PIP. The openid.delegate tells sites the URL of your identity. (Make sure you replace “you” with your PIP username.) Since you control your website, you don’t need to worry about someone else impersonating your domain. And since your PIP is protected by a password, no one else can steal your identity by pointing their website to your PIP.

Step 3: Log in to something!
LiveJournal is a fun place to start, but there are plenty more, such as Zooomr and Ma.gnolia. We’re looking into using this system for ZetaBoards as well!

• 23 hours of class a week
• Hundreds of hours of RA responsibility
• 4 hours of tutoring a week
• Miscellaneous computer work (such as Pre-Med Club)
• And all the other normal college student responsibilities

It was a tough semester, to be sure. But with 5 out of 6 classes reporting, I was still able to keep all A’s… although one of them was very close (and it’s a good story too). Here’s how it came about.

One of my 6 classes was Pathogenics— the study of infectious diseases and the organisms that cause them. We studied pathogens such as Yersinia enterocolitica and Staphylococcus aureus, as well as nastier stuff like Neisseria gonorrhoeae and Clostridium difficile. I really enjoyed the material (it’s related to what I hope to work with, after all), and I didn’t find the class very difficult. However, for some reason I could just never get a handle on the tests the professor gave us. I would go in feeling as if I knew all the material, and come out with a mid-to-low B on each exam.

Going into the final, I had three Bs and a C on the 4 exams… not an A in the bunch. The C was dropped, but I still needed a 95% on the final to make an A. I doubted that could happen, since I hadn’t made an A even on the “easier” tests. But then, the professor told us that not only would the final not be cumulative, but that he would add 10% to each person’s final exam score. At that point, I needed only an 85%, which I thought was doable. I studied an obscene amount for the final, and went in feeling good about it. However, as I started answering questions, I left more and more blank. I ended up leaving 35% of the test blank on the first run-through (in which I answered only stuff I knew immediately). Not good. I went back and started working the problems that were left, and got many of them answered, but had to guess on a few. I walked out of the test unsure of how I had done. It was graded that night, and I got… 82%. This gave me an 88.13% for the class, and an 88.5% was needed for an A. Only 0.37% away! I e-mailed the professor out of desperation, just to see if there was anything that could be done.

It turned out that he remembered a day of class a couple months back during which I had noticed an error in his lecture notes and corrected it. He told me that I would be given 2 bonus points for catching the error, which pushed me up to an 88.6%! Enough to round to the A. More grace than I deserved, probably, but warmly welcomed.

Work on ZetaBoards continues mostly on-schedule. We’ve hit some frustrating setbacks, but all in all it’s shaping up very well. I think the beta testers have had a good time working with it and I’m definitely looking forward to the reactions of those who said it would never come. Plus, it’s going to remove so much workload from our moderating team, with the new moderation features that are in place. We’ll be able to concentrate more on building and less on maintaining.

This is one of the most open breaks I’ve had in awhile; not much is going on, and I don’t have any specific plans, other than ZetaBoards development. So I decided to start with something I’ve wanted to do for awhile: invest in the stock market! I’m opening an account with TD Ameritrade (although if someone has another suggestion I’d be happy to listen), but I’ll start small with only a few companies that I’ve personally researched and looked into (mostly ones from which I own products). I’ve taken quite a few hours to research things before even opening an account. It seems that expected return, as per the S&P 500, is about 11% yearly. Thus, if you can beat 11%, you are “beating the market”, and if you can’t beat 11%, you might as well have your money in a SPYDR (Ticker: SPY) or similar. I tend to have fun with games in which one tries to outsmart the {opponent, AI}, so the market isn’t much different. Since I’m not looking to invest in high-growth items that fluctuate wildly, as long as I don’t try to get out immediately I should be all right. The principle of buy-and-hold tends to work out in the long run.

In addition to all this, I’d sure like to get my site redesign in place. I’ve talked to Nicola about helping me with it, but I’m afraid it has to wait until after ZetaBoards, which might mean this summer. I have some new ideas to implement, including OpenID integration (another post soon about this), better scheduling and visualization of where I am throughout the day, and overall making the site more interactive.

This break will be fun

There are several good ways I could backup my e-mail. It'd be trivial to write scripts to tar up my mbox files every once in awhile and download them. Or I could rsync them to another account. However, I decided to go with a truly off-site backup, that was stored in a more usable format that I could search through immediately and restore piecewise if necessary. Aka GMail!

All I needed to do was setup a forwarder from my primary e-mail to my new backup GMail account. (I have a real GMail account for things that might spam me; this is a separate account.) Now I'm receiving mirrored copies of everything to that account. I don't need to do any spam prevention or anything, since I can search for whatever mail I need quickly and easily. And GMail has a much more lenient account dormancy policy— 9 months— than do most free mail providers.

All in all, I think it's going to be a fine solution… but let me know if I'm missing something