It’s one thing to love your neighbor in the abstract. It’s quite another to love your literal neighbor.
As G. K. Chesterton explains:
We make our friends; we make our enemies; but God makes our next-door neighbor. … The duty towards humanity may often take the form of some choice which is personal or even pleasurable. That duty may be a hobby … We may be made as to be particularly fond of lunatics or specially interested in leprosy … But we have to love our neighbor because he is there — a much more alarming reason for a much more serious operation. He is the sample of humanity actually given us.
Quote found in From the Library of C. S. Lewis