Not really. All of the things that Touchwiz adds take up resources. For smart stay to work, a bit of processing and batter power have to be redirected to the front camera to watch for your eyes. That's just one example, but there are lots of things in Touchwiz that are constantly monitoring what you're doing so they can provide helpful shortcuts and things. I'm not saying it's bad. I'm just saying, when you've got a phone that's monitoring for certain input so that it can do something to help you and one that doesn't monitor like that, the one that doesn't monitor will theoretically have more resources available to do what you're actively trying to do.
It really has nothing to do with the launcher. As you said, that's easily replaced. I think if you were to root the Note 5 and completely take away those things that are using little bits of resources here and there, you'd find a much faster device.
This is a real world comparison, but if you really wanted to level the playing field and get to JUST which one has the better hardware, you'd want to root both and run the same ROM on each and load them up with all of the same apps.
Tests like this are all in good fun anyway.