For me, anime > manga pretty much always, so I may not be the best person to ask. I do think the music & the color add to stuff, as well as getting to see the physical comedy stuff animated. (I linkdrop this all over the place, so apologies if you are seeing this for the 23423423rd time, but the OP to the second season is awesome. I love how we see the scenes first as anyone else would, then as Natsume would--w/all the spirits added in.)

Over in natsumeyuujinchou there seems to be a general consensus that the manga is darker than the anime, which seems to hold w/an interview w/Midorikawa where she said she felt that Natsume's world was v. cold, & when the anime came out, she was surprised & moved to find Natsume in a warm caring world. So you might feel slightly differently about the anime anyway?