Here is what HMRC says, there is no mentioning anything about different brokersā¦
"Shares of the same class in the same company are identical. Suppose youāve a holding of 12,000 Wilson and Strickland plc 25 pence ordinary shares acquired at different times for different prices. You then sell 2,000 shares. To calculate the gain you need to know how much cost can be attributed to the shares youāve sold. The capital gains rules for shares allow you to do this.
From 6 April 2008 all shares of the same class, in the same company, are together called a āSection 104 holdingā. You add together the costs of the shares in this holding: each share in the holding is treated as if acquired at the same average cost."