@rshahbaz I usually approve them. I do a quick glance at the URL, and see if the headline is at least in sync with the topic of the post. If it’s not, I’ll go check out the site. If it doesn’t jibe with mine, or if it’s a low-value site, I’ll delete it.

@rshahbaz Sorry I missed this post – not sure how I did. 

I would say that trackbacks can only hurt your site’s SEO in the same way that linking to bad sites would.  I typically don’t approve of any trackbacks unless they come from a legit blog and the post actually links back to mine within the content (ie. not a Zemanta or other type of link autogenerated at the bottom of the post to generate trackbacks).

Of course, after awhile, I was hit by so much spam in the pingback / trackback department that I turned mine off altogether.  You can still see links to your blog posts in the Incoming Links portion of your WordPress dashboard, so you won’t miss them.