My use-case - which I’d welcome feedback on - is that I’ve built a home-brew system to gather fund prices and produce a daily email report of the top funds per (Morningstar) sector, the concept being that short-term investing in the funds which are consistently the best performing in well-performing sectors should be a good way to maximise returns.
I used to do this by spidering Trustnet, which was efficient and worked well - but then they moved to client-side JS rendering which would have meant migrating to use something such as Selenium to retrieve any data.
At this point I changed to using the ft.com Morningstar data - but many of these funds do not appear to be available from UK brokers. Given the lack of any available APIs to date I tried a highly rate-limited simple HEAD against HL’s public website… and got my IP banned for my efforts . YouInvest, on the other hand, has a much more sensible website that’s behind a CDN - but they’ve also raised questions with my ISP simply based on the volume of requests (although I’d argue surely not the impact - it’s at most a single HEAD per fund per day, and only for new SEDOL/ISINs which haven’t been previously encountered!). I had some success making the request via a public HTTP(s) proxy… but I should probably just migrate the whole setup to a cloud platform and get a Lambda/Function to populate the database.
@finki I don’t want to cause any harm to your service - what rate-limiting do you impose? I was planning simply to use the ‘ukSymbol’ call to confirm the existence of the given ISIN on freetrade/Trading 212 platforms… is this a reasonable plan? Is this the lightest-weight method of determining ISIN validity/availability?
I also just looked at the HL and YouInvest sites and I can see queryable APIs you can hit with Sedols and ISINs and reverse engineer and the youinvest site uses a Morningstar feed that you can also manipulate to query by ISIN… so you could probably revert to using Morningstar if you used that previously !? Or if you wanna go back to using trustNet then go through something like ScrapeStack to render JS and make it scrapable
Don’t get me wrong … use Finki. But your use case seems easier and better served elsewhere. That said I don’t truly understand exactly what you’re trying to do.
I kinda feel I can build something for you… so ping me more details if you’re so inclined and I’ll have a hack when I get time
I’m not a (frontend… or other!) developer so I could have missed something here… but HL has an external API? I actually contacted them and they explicitly denied that such a thing existed… could you point me at any details?
I had assumed that, should such a thing exist, it’d be predicated on a session-specific token or similar to prevent use by anything other than their own website (… without additional effort, at least)!
I’ll take a look… (although it’d probably get my IP blocked again, knowing my luck).
They seem quite vociferous in the belief that their (extensive…) T&Cs apply to any and every client connecting to their website
(In terms of XHR requests, there only seems to be a clickTrack response with any data)
I note that later in the buy/sell process, HL does begin to predicate requests on SEDOL codes, so I assume they’re chosen to obfuscate these on the majority of their public-facing pages as some means to maintain a competitive advantage…
I think I noticed the share search on HL was driven by Sedol or could be hacked by throwing a singular letter at it … maybe … I looked for 3 seconds. Looks like II use sedol ‘openly’ if you’re interested!
I imagine they limit sedol use due to horrible lse licensing requirements. Whereas ISIN is ‘generally’ less stringent