The Jews accused Jesua of threatening to destroy the Temple.
You might be presenting the idea, that the Romans spent considerable effort in destroying the hewn rocks, or perhaps moving them elsewhere.
It is a dark foreshadowing, however, that the Jews had accused Jesua of threatening this type of annihilation.
'No stone left standing', as Jesua told the disciples.
Only the foundation stones were huge.
Remember there was a third temple.. Emperor Julian tried to build it in 363 AD..
Evidently it was destroyed by fire and earthquake.
Excerpt:
Till then the foundations and some ruins of the walls of the temple subsisted, as appears from St. Cyril: and Eusebius says, the inhabitants still carried away the stones for their private buildings. These ruins the Jews first demolished with their own hands, thus concurring to the accomplishment of our Saviour's prediction.
Then they began to dig the new foundation, in which work many thousands were employed. But what they had thrown up in the day was, by repeated earthquakes, the night following cast back again into the trench. "And when Alypius the next day earnestly pressed on the work, with the assistance of the governor of the province, there issued," says Ammianus, "'such horrible balls of fire out of the earth near the foundations,' which rendered the place, from time to time, inaccessible to the scorched and blasted workmen. And the victorious element continuing in this manner obstinately and resolutely bent as it were to drive them to a distance, Alypius thought proper to give over the enterprise."
This is also recorded by the Christian authors, who, besides the earthquake and fiery eruption, mention storms, tempests, and whirlwinds, lightning, crosses impressed on the bodies and garments of the assistants, and a flaming cross in the heavens, surrounded with a luminous circle. The order whereof seems to have been as follows.
This judgment of the Almighty was ushered in by storms and whirlwinds, by which prodigious heaps of lime and sand and other loose materials were carried away.
After these followed lightning, the usual consequence of collision of clouds in tempests. Its effects were, first the destroying the more solid materials, and melting down the iron instruments; and secondly, the impressing shining crosses on the bodies and garments of the assistants without distinction, in which there was something that in art and elegance exceeded all painting or embroidery; which when the infidels perceived, they endeavored, but in vain, to wash them out.
In the third place came the earthquake which cast out the stones of the old foundations, and shook the earth into the trench or cavity dug for the new; besides overthrowing the adjoining buildings and porticoes wherein were lodged great numbers of Jews designed for the work, who were all either crushed to death, or at least maimed or wounded.
The number of the killed or hurt was increased by the fiery eruption in the fourth place, attended both with storms and tempests above, and with an earthquake below.
From this eruption, many fled to a neighboring church for shelter, but could not obtain entrance; whether on account of its being closed by a secret invisible hand, as the fathers state the case, or at least by a special providence, through the entrance into the oratory being choked up by a freighted crowd, all pressing to be foremost.
Rebuilding the Jewish Temple, in 363 A.D.