So wait, it will build boost if you freeboost it?
If so it's a problem with your boost control, but you want to be very very sure it's not in limp mode and restricting boost because of that.
Definitely sort that exhaust leak mate - all the exhaust gas that should be spooling your turbo is exiting out that leak before it makes it to the turbo so that will likely be your main issue (if not all of it).