Co-worker doesn’t get a hint

I like the occasional personal conversations at the office or venting, but there are some coworkers that don’t know when it’s time to get back to work! One coworker in particular comes by every day and makes comments that I never visit their desk, but I can’t seem to possibly walk over there if I think that a daily visit is excessive. I’m always the one to cut our conversations short and I feel bad saying something because I enjoy their friendship. Help!