I second moving to an 18650. When I built my first saber, I used a x4 AAA pack with a 5V regulator to drive a single green Cree and a Hasbro Econoboard. That barely had enough power to drive the saber for 45 minutes before new batteries were needed, and the LED dimmed visibly after the first 15 minutes. Running two blues off 3 AAAs? I hope you have shares in Duracell, because you'll go through a lot of batteries, fast.
Your wiring diagram is solid, and upgrading to an 18650 shouldn't be that much of a hassle, especially if you're 3D printing your own chassis. Having it connected via a JST will work, but you'll have to wire in a kill switch to ensure that the battery doesn't drain while you're not using it.
If space really is at a premium, then there are some alternatives to 18650s: 14500s, 14650s, 16650s, etc. they're harder to find, and will have a lower capacity, and may not come with flying leads/JST connectors, but they're lithium-ion cells, so they all have the same voltage supply.