From Shakespeare’s Richard III, Act III, Scene 1:
PRINCE EDWARD
I do not like the Tower, of any place.
Did Julius Caesar build that place, my lord?
BUCKINGHAM
He did, my gracious lord, begin that place;
Which, since, succeeding ages have re-edified.
PRINCE EDWARD
Is it upon record, or else reported
Successively from age to age, he built it?
BUCKINGHAM
Upon record, my gracious lord.
PRINCE EDWARD
But say, my lord, it were not register'd,
Methinks the truth should live from age to age,
As 'twere retail'd to all posterity,
Even to the general all-ending day.
It may be that “the truth should live from age to age” but it’s fairly clear that even upon record, should there have been one, it does not. The Normans, not the Romans, build the Tower (or so I think).