I heard back from the instructor of the class that I TA, and he reminded me why the student got that particular grade, and told me to stick to my guns (especially because, if word got out that we were renegotiating grades...). He also suggested how to word such a response, which helped me to clear my head and get my confidence back.
(Yes, Student, I did acknowledge that you had perfect attendance and were willing to speak within your group, and those were reflected in your grade. However, attendance played only one small role [this is important, considering that his complaint was that someone with imperfect attendance got a higher grade than him], and I was looking for you to take more initiative in your group. Here are some specific requirements that you were aware of and ignored. Here are some occasions where you could have taken leadership within your group, or in the larger group, and did not. Your grade is actually a good one, and a few percentage points on participation will actually have no impact: stop worrying about this and start worrying about the upcoming midterm. And stop worrying about other people's grades, and I won't discuss them with you.) (Or something like that.)
It's still hard for me to be the "bad guy." I like people to like me, but now I'm trying to work on getting them to respect my authority.