It seems that I had "Send Off Messages" On for the group those buttons were in.
When unchecking it, it all worked.
Strange thing is, I've got a page in the Buttons tab with scenes and they're not in a group and apparently, they don't need to be.
But I guess that's because the first command is a scene command.
In my local buttons, the first command is a macro.