No one has built an earthbag fortress or fortified castle yet to my knowledge. I’m just throwing it out there as a possibility. Building something like this out of earthbags would be much faster than working with stone, and certainly less expensive.
If this seems like just a fanciful thought, go back through some previous blog posts for background information. Tamped earthbags are comparable to rammed earth, which can last thousands of years, as explained in my blog post Ancient Rammed Earth Structures.
One good option for extremely durable and/or fortified structures is building with geopolymer. Geopolymer is man-made stone consisting of low cost natural materials such as loose limestone, diatomaceous earth, fly ash or rice hull ash, and low cost binders. This is how many scientists now believe the pyramids in Egypt and many other ancient structures were built. You could use geopolymer in earthbags, cast in place with forms, or make geopolymer bricks or blocks. About 20 geopolymer products are already on the market if you prefer off-the-shelf solutions. All these options are discussed on Owen Geiger’s Geopolymer House Blog. If you’re short on time, here’s a brief Geopolymer Summary that provides a brief overview of some options.