I told my boyfriend that his son doesnt need to come here every weekend is that wrong?please help 10pts?

So I live with my boyfriend he has three kids from previous relationship. I am 25 he is 38 I have 0 kids. So they all use to live with us I use to take care of the four year old now 5 for 5 months . Well now they are all scattered around the oldest son 14 lives with his grandmother because he was flunking out of school and running away, the five year old lives with his mom again and the 12 year old lives here with us. I am a college student and my boyfriend got laid off from his job 2 months ago he hasn't not looked for a job .He gets unemployment but it barley covers rent. I revive financial aid from school every few months so i give money out of that but i'm not getting that till end of next month. Anyway we went down to the welfare office to apply for food stamps for him and his son come to find out his baby mama had the son that his been living for over a year wit us is on her case and was trying or getting cash and food stamps for him and she not one time gave my b/f money for him or food stamps. I was so upset i told my b/f he better fix this and I guess she told him she will take him off but my b/f is still not getting food stamps for his son and its been over a month. And for sure shes getting money food stamps for the son that living wit her. Anyway she has been dropping him off every weekend lately and we dont have a car because we had to sell to pay rent. so we cant drop him off and sometimes she wont even come pick him back up till monday or my b/f has to find a way to pay someone hella gas to take him home. He invades his older brothers room and cries if he doesn't let him play the xbox. He wants my b/f full attention and he ignores me if my boyfriend is talking to me in the room for a second he comes and dist rubs. He whines for everything and gets bored easily and we cant take him any where cuz no car or money. I wanted to study for my finals this week and didn't want him over but his mom text talkin about shes going out town and dropping "the baby off" (the ****** kid is 5) and boyfriend just lets her . I'm tried off It him not doin what hes suppose to do, him letting his baby mama walk all over him, his son over every weekend. So i told him today that he needs to make some type of arrangement with her because this is getting out of hand and that we need our weekend to do something special too. And he flipped he said "my son and come here anytime he wants" and its not of your Business blah blah but then later said i am Right but then said he will not come next weekend and will suffer because of me. And he got mad cuz i said we cant afford to have him here every weekend because hes another mouth to feed, so we went back and fourth of himagreeingg with me and then saying I was *****. What do you think do i have good reasons of not having every weekend or am i wrong? or and btw his mother never packs him a bag of clothes when he spends the night becausee she wants my boyfriend to buy him more clothes