How much was the delivery charge originally? This should be the amount he will not refund to you.
If it is significant, i.e you don't want to write it off to experience, after you recieve the refund, send him a letter (recorded delivery) demanding the missing money. If he refuses or, after a reasonable period, doesn't respond, go to the Money Claim On-Line (part of HM Courts Service) (https://www.moneyclaim.gov.uk/web/mcol/welcome and https://www.gov.uk/make-court-claim-for-money/overview) and issue a small claim against him for the missing amount and the MCOL fee.
I have used this service a few times and had no problems. Even had to get the bailiffs out to one guy, but always got my money back
Good luck
Dave