Hm, that's a pity. Apparantly you didn't leave enough thickness in the ipé itself to make it a stiff handle without the rosewood. Although the ipé does have additional thickness in the handle compared to the limbs, it's not enough. The clean ipé and rosewood also indicate a bad glue application or improper surface preparation. Both woods are notiously difficult to glue, so it makes sense.
I'd say you need to rough up the ipé surface with 60 grit sandpaper. Then wipe down with acetone to degrease. Instead of using one block of rosewood for the handle, glue down several thin laminations of about 1/8" thick. Also rough up the rosewood and degrease it as well. Then continue with the titebond 3 for the glue up. That is plenty strong enough.