I came to some hard decisions when I bought my .380 TCP. It really depends on what you want it for. I love .45, but I've been giving the 9mm another chance, for similar reasons specified. I love recoil, so that's a moot point. For a home defense situation, where I probably have one or two intruders? My Sig P220 with 8 rounds of .45 is perfect. I'd rather justify one or two rounds of .45 that stopped or ended the intruder to a court than say four rounds of 9mm. I'd rather go 12 gauge, but that's less usefull in my narrow hallway. In that situation, size doesn't matter, and I want to stop someone fast. Now if you need to face a lot of people, like a gang? Give me maximum round count in something 9mm or larger. 17+ rounds of 9mm is hard to look down on.
And for CCW? My personal requirement for my .380 was (minimum) center mass hits at 40 feet, and head shots at 20 feet. If I couldn't do that, I wasn't going to get a .380. Six rounds to the face will do the job. And for something only slightly bigger? I like my PT111 G2. 12 rounds of 9mm is more comforting than 7 rounds of .45.... especially when I have no clue what I'll need it for. If you had to face an active shooter with a rifle or shotgun, you're undergunned no matter what. The trade-off of accuracy, round count, and stopping power is a tough one.
It depends on what you need, expect, and what your capabilities are. If you are confident in accuracy, smaller is ok. The British SAS never had a problem with the 9mm. They just train for double taps to the head.
If you are counting on only torso shots (like a 6" group or so) to stop someone... go as big as possible. A shotgun, preferably.