OK don't quote me on this, but this might be it. Thought that before but now that I know the specific triggers I can say what is what isn't much better. And it seems to work even in chrome now. SO here is the deal. Went into the input settings, and everywhere I could change a keyboard default language, I forced it to US english. So go to the AOSP, the omap4, the mapphone-switch and go and make sure that ONLY English US is chosen for keyboard layout. Then go into google voice typing, uncheck the auto language and choose us english. Now my chrome WASN'T updated what I tried this so hopefully installing the updates doesn't break it, or if it does hopefully I can just go put all these settings back and be good, but I was able to transition between physical and onscreen typing without it goofing my shift key up after doing this so that is good news to me!