I just picked up an XDS to replace my LC9. I couldn't be happier with the decision. Yes, it has a little more recoil, but not so much that I really notice. If you want a few more rounds, the XD sub compact .45 is just slightly larger. The Taurus PT145 or PT745 are also great shooters, but are not quite as high of quality as the Springfields (in my opinion). I will say the only gun that I have ever sold that I truly miss is the PT145.
I have been carrying the XDS for a couple weeks now and I seem to notice it less than I did with the LC9. It just seems to ride in my IWB holster better.
I was actually going to get the Kahr CW45, but found a deal on an XDS for less than the Kahr. I couldn't pass it up. They do have 7 round extended magazines for the XDS now so if you want the option for more rounds and a bigger grip, you could get one or two of those. Then you have the option of smaller or larger grip.