Harrim's quest - siding with kobolds

In my first playthrough, during the Harrim's quest, I've killed both trolls and kobolds and after Harrim destroyed the anvil, he received a gem. Later on, a group of dwarves moves to the ruined fortress and Harrim gets two more quests that inform him about his destiny.

In the current one, as a chaotic character, I've decided to vassalize kobolds - Harrim still managed to break the anvil but there is no gem and no dwarves: does it mean that his personal quest won't progress and he is doomed to die in the end?

I'm playing on the patch 2.0x, so contrary to some comments that I've seen online, I can get back inside the fortress!

Update: The dwarves show up around the "The vanishing of Varnhold" quest, even after siding with kobolds. The gem (Hearth of the Anvil) apparently appears only if you pass the skill checks near the destroyed Torag statue and/or (?) the glowing dwarven rune (I've read conflicting information about it, I haven't confirmed it myself).