But if the Latin loanwords came into Old English via French IA people, then that means the Old English people got basically no loanwords at all from the local population (whether Brythonic or Romance). Having virtually no loanwords at all from the local population of Britain and only loanwords from other migrants (i.e. French IA) just seems implausible to me since interactions must have happened between the Old English and locals.
And considering that numerous English toponyms are derived from Latin/Latinized Brythonic toponyms (these Latin toponyms were from the Roman Britain period, so can't be from French IA), that means Latin words unrelated to French IA people were definitely borrowed in Old English. If toponyms could be taken from British Latin, then a lot of normal words (non-toponyms) could also be taken from them.
Still, there were hundreds of Latin loanwords in Old English. Even if some (or even most) were from French IA people, I doubt almost all of them were. Compare this to the loanwords from Brythonic which are basically non-existent.

