I've had similar callouses on two fingers. I filed them down over a period of a couple of weeks and have had no problems since then.

I've also come to the conclusion that washing pots a couple of times a day does wonders for keeping fingertip callouses in a great condition.