How to dig through "rough hewn diorite wall"

I've built an up/down stairs on layer 3, and now I want to dig to layer 4. However, layer 4 reports rough hewn diorite wall on the same squares that the up/down stairs occupy in layer 3.

I cannot build any stairs on it since it says 'blocked'. I have tried placing stairs on different areas in layer 3, but no one will dig to layer 4. How do I break through?