Do you like her? Do you trust her? Is it possible for all of you to do things together once in a while? If so, and if you are ‘allowed’ to be poly, too, then go ahead and talk it over with both of them with an open mind.
Do you have children? With this guy? Would they be ok spending more time with these other kids? How old are all the kids? Teens tend to assume that whatever is going on is nefarious – even reassurances by all adults involved that everyone is being honest and giving permission isn’t enough to placate them. Little kids tend to like having more people to pay attention to them.
Do your bf and his ex like each other for more than sex? If they’re just wanting to scratch each other’s itches, I’d be very wary. Oh, and speaking of sex – Protection! If you can’t trust him to wear it with her, he’ll have to with you – you can’t trust her that far.
But you know what, the bottom line is, neither you nor your bf own each other, or can actually give permission. So don’t think of it that way. Instead:
Think about it like “sure honey, she’s a fine woman. I wouldn’t be comfortable if you started forgetting about things you and I had planned together, but if you want to visit her sometimes I’ll find something else fun for me to do while you’re gone.”
Or else like “Well, honey, I’m just not feeling comfortable enough with the strength of our relationship right now. Maybe if I get to know her better, and if you and I spend some more time exploring intimate adventures, maybe we can both join a forum or a Yahoo group about polyamory and see what kinds of things people with experience talk about. But I’d really not be happy at this time, maybe never but I’ll try to think about it, if you go over to her place and have sex.”
See, you and your bf need to be able to love each other enough to talk about it completely, with love and respect and with joy.