Am i wrong for not letting my son see his dad until he learns how to respect our relationship as parents?

my sons dad never wanted his son to begin with. my son is now 2 and nothings change. we been on and off for the past 4 yrs but now we are officially done. i dnt talk to him, bug, nor go by his house. i wanted thing to just be about our son. the only time we speak is every 3-6 weeks when he would out the blue decide he wants to see his son. he never keeps the set schedule we have and he comes and goes as he pleases. he disrespects me, has his girl, fam, and friends disprespect me for no reason and im tired of it. but he always plays the victim when its his fault he dosent see his son..everytime he ask i try my best to give him to him even though i dont like that i dnt hear from him for weeks at a time. all i want is for respect so we can raise our son. ive tolerated so much from him throughout the years and im fed up. it hurts me that were like this way. i want him in his life and i hate that hes this way but im tired of the coming and going as he pleases and the disrespect from him. its unessesary. all he cares about is his girlfriend. i pray about this every night. trying to find stregnth to do whats right but ive tried for years and nothin seems to work with him...its been a month since he seen his son cause the last time we talked all i did was ask for his help to watch him more cause my mom is sick. i work and going to school for nursing full time. he cussed me out for askin then had his girlfriend cuss me out. thats when i was officially done dealing with him until he changes his ways.. he still blaming me..to him everything is my fault even though he disrespects me. i dnt know what to do..this hurts me more than it will ever hurt him but it seems like he could care less cause he hasnt tried to reach out to me or his son. what should i do?