I have had the S4 since June, and recently impulse bought the Moto X when they had the $60 off coupon. I did however end up taking the Moto X back and stuck with my S4. The Moto X is a great phone, but for me it was too much of a downgrade to be worth it.
Like you I love the screen size of the S4 and the soft buttons on the Moto X took up even more real estate off the smaller 4.7 inch screen. The fact there was only 16GB of memory with no option for an SD card was another downside. The camera on the X was nothing compared to the S4, which was a surprising downside because cameras are not usually a factor for me. And I missed the HD screen res, I thought I wouldn't but I indeed did.
While the Moto X is a snappy and great feeling phone in the hand, it had to go back to Verizon. Sooner or later they will unlock this new bootloader and I will be able to flash Cyanogen Mod to get the stock Google experience again.
Plus with the Moto X rumored to drop to $99 within the next three months even paying $139 for it was too much for me, especially with no Moto Maker available for Verizon. My final thoughts about the Moto X are that it is a perfect device for someone that has been sitting on a year old or older phone and wants a cheaper upgrade to a fantastic device. Yet, those of us that already have the S4, HTC One, or another newer device, the switch backwards is a hard one to sell.