Should I always wear protection with someone who is friends with benefits?

My friend and I have been using protection over the summer, but I find it annoying to wear one when I've known him a while. I don't like with protection really. It is annoying for me. Anyway, I wouldn't use one by now if it was me and my girlfriend who were together this long. However, I don't know if I should wear one or not because our relationship isn't at all monogamous. We both have girlfriends. With my girlfriend we aren't at all monogamous and I see a lot of other girls besides her. I ume he does the same. I really don't know. So is there anyway around it or should I just stop using protection?