There are a couple of reasons. The first is HMRC could have made a mistake with your tax code in the last 4 years.
The other is you might not know there are tax allowances, (or were tax allowances as some of them have gone now) you were entitled to ask for so you paid less tax. If you didn’t know about these allowances, there’s no way you could ask HMRC to put them in your tax code, so they give you the wrong code. That meant you paid more tax than you needed to.
The good news is it’s not too late to get your money back because HMRC lets us go back up to 4 years to claim any allowances you missed out on, or to correct any mistakes.
The problem you’ve got is unless you know what to look for, you won’t spot any mistakes and unless you know what allowances you were entitled to, you can’t ask HMRC to let you have them! That means any money you overpaid is going to stay in HMRC’s bank account, instead of yours! HMRC also gives you a deadline to claim your money back. If you miss it, they’ll keep your money!