Ok so took the car to South Canterbury Toyota for the recall work to be done and had them check the transmission.
They confirmed with my regular mechanic what fluid he used and were able to confirm that it was the wrong fluid. They took the bottom plate off and showed me the insane amount of metal filings inside and on the magnet.
He replaced the fluid and flashed the TCU using the pak file from Subaru and it's still doing it, he says its slipping mostly in 1-3rd gear when under a fair amount off accel. He recommended driving it easy as he said that often the glazing on the clutch disks will get better with time once the right oil is in but he's 50/50 on whether the damage is already done. So, probably for the bin really.
Wish it were easy to do a manual conversion!