From what I've read, leaving the stones in stone fruits provides additional depth. I don't have any direct experience with it though. Tonsmiere talks about it a bit in American Sour Beers:
"Some brewers treasure the character that cherry pits add to beer, often described as woody or almond-like. To replace the pits removed from processed cherries, some brewers add a small amount of crushed sour cherry pits (sold as mahlab for Middle Eastern cooking). Dogfish Head Brewery added mahlab to Midas Touchstone (they suggest 0.2 oz./gal. [1.5g/L]), a sour cherry variation on Midas Touch. Be careful of this, however, as the TTB has set a limit of 1ppm of cyanide, a compound which cherry pits contain naturally."
I'd throw them in, even with a cherry pitting device, I imagine it would be a long and messy experience to pit thawed cherries. I have some myself from our tree that finally started producing enough that I plan to throw in pit and all.
Good luck!