The SIM card has nothing to do with whether you are 3G or 4G. All it does is allow you to communicate on Verizon's network. What determines whether or not you are 3G or 4G is the type and quality of the signal your phone is receiving and transmitting.
Your office is probably on the outer edge of a 4G area or in a 4G extended area and the 3G to 4G hand off wasn't being handled very well. Also, Verizon has three levels of 4G. They have 4G LTE, 4G LTE Extended, and Extended 4G LTE.
* Access the 4G LTE network within the Coverage Area.
** Access the 4G LTE network within the Verizon Extended Coverage Area; certain conditions may cause your service to connect to 3G in this Area.
*** Access the 4G LTE network within the Extended Coverage Area; Some of the Coverage Areas include networks run by other carriers, the coverage depicted is based on their information and public sources, and we cannot ensure its accuracy.
Check Verizon's coverage map for your area and you will see what kind of area your office is in. Where I was at in Tucson that got the 4 down and 1 up was a 4G extended area, but (according to Verizon) 31 down and 15 up was in a 4G area.
4G LTE Network | Verizon Wireless