Every scene button has actions that control the Effect Gizmo, so effectively it is in the same group.
Here's an example. If you're on scene 3, loops 1, 2 and 4 are on. Now, you press scene 1. The scene 1 button says that loops 1, 2, and 3 are on, so it'll turn those loops on. The scene 3 button automatically turns off, which makes it turn off loops 1, 2 and 4. You'll end up with loop 3 on, but loops 1 and 2 turned off.
Likewise, if you did that with Send Off Messages turned off, you would start with loops 1, 2 and 4, then you press scene 1. Loops 1, 2 and 43will turn on (although loops 1 and 2 are already on). Nothing will tell loop 4 to turn off, so it remains on as well.
The answer is to have actions for each loop on each scene button. Scene 1 would look like this
Effect Gizmo:Loop 1 / CC Toggle #80 off:0 on:127
Effect Gizmo:Loop 2 / CC Toggle #81 off:0 on:127
Effect Gizmo:Loop 3 / CC Toggle #82 off:0 on:127
Effect Gizmo:Loop 4 / CC Toggle #83 off:0 on:0
Effect Gizmo:Loop 5 / CC Toggle #84 off:0 on:0
Scene 3 would look like this:
Effect Gizmo:Loop 1 / CC Toggle #80 off:0 on:127
Effect Gizmo:Loop 2 / CC Toggle #81 off:0 on:127
Effect Gizmo:Loop 3 / CC Toggle #82 off:0 on:0
Effect Gizmo:Loop 4 / CC Toggle #83 off:0 on:127
Effect Gizmo:Loop 5 / CC Toggle #84 off:0 on:0
I only showed 5 loops because that's all you're referenced in your configuration, but if you use more loops, you'll need to list them on every button.
Be sure to have Send Off Messages turned off for the group when you use this approach.