How can I defeat the Caretaker?

Make sure you have applied Relict Oil on your silver-sword. Also, equipping Ekimmara (Damage dealt to foes regenerates Vitality) or Ekhidna Decoctions (Performing actions that consume Stamina regenerates Vitality) can tremendously help you in this endeavor.
Next, use the Quen sign on Geralt as frequently as possible. This debilitates the Caretaker's life-steals when he hits you. If possible, learn the Fast Attack, 'Whirl' to deal more damage to the Caretaker in his moment of weakness.
Wait, weakness? Yes. This boss has one weakness, or opening so to speak, to deal extra damage.
When the Caretaker begins charging his petrification attack, (as seen when he charges his shovel into the ground and a circle of blue light emits around him) use your crossbow to stop him midway, thus causing him to momentarily be stunned. Use this opportunity to attack him with 'Whirl'. Normal attacks are fine too.
When he starts to summon his spirits, don't fret. Focus your offence on the spirits themselves. Attacking the Caretaker would only lead you being flung further away and gives him ample time to regenerate. Precise, the spirits disappears after one or two blows. One last tip of advice is to practice patience.
The more you rush, the more mistakes you would likely make. Go in steady. Keep your wits about you. Don't rush when attacking and don't get greedy too. Hope this helps.
P.S: It took me half an hour to learn and counter his techniques. If I can do it, so can you.
Godspeed.

Use relic oil on your sword and igni fire stream and he will keep doing the ground smash move, dodge out of the way then dodge back and attack him

After trying everything and dying continually, I hit on the solution, when he goes for the spirits attack him thus preventing him from healing. I should have thought of that right away.

Just run to the fountain and run around it to the opposite side continuously until he starts attacking. Wasn’t for him to do the dash and use Igni to do damage. You can keep doing that :p