I'm not a grandparent, but we went through a similar situation with two sets of parents, with two kids.
You need to explain to them that you want to raise your own child and its important to you to spend time with them at this age.
Set up times they can see the child, not every weekend. Maybe a couple days a week and every other weekend. Plan days you can all get together, then take your child home.
It's not going to be easy, they will probably get mad, but that's the way it has to be.
Don't put the child in the middle, he did nothing wrong and like most kids, he will probably want to go to his grandparents house... gets spoiled and no rules.
Good luck to you.