US Order status delay - 8 September 2020

We’re sorry if you experienced a delay with your order this afternoon. Your order executed at the time you placed it, but we had a technical issue showing the status of the order. This issue has now been resolved and we’re working to prevent this issue happening again in the future.

We’re sorry if this impacted you, please do let us know if you have any questions. We’ll also provide a more detailed update on this technical issue later this week.

8 Likes

Yes, I placed a buy order that became queued. I had nothing to indicate why this happened. I have now cancelled the queued order and re-placed it (this time it went through OK although I got fewer shares for my money). What happened there? Why didn’t I get any notification that there was an issue?

I’m very sorry to everybody impacted by today’s outage. As an investment platform, it’s fundamental that we can function seamlessly during these critical moments. Today we failed to do that. We also failed to update our customers quickly enough.

Today’s issue was caused by a bottleneck in our reception of trade status updates from our partner. A combination of large order volumes following a three day weekend along with today’s market conditions meant this bottleneck significantly impacted the product - preventing users from seeing successful orders for over an hour.

We have been aware of slight delays during peak time and have been working on multiple strategies to resolve the delays. Unfortunately, those changes were not due to be released until later this week.

I apologise to those affected and can assure you that as well as fixing the core issue seen today we will be improving our incident response and working to prevent similar issues from happening in the future.

21 Likes

Hi @tbutz. I’m sorry that you didn’t see a status update. The type of delay experienced today means the app isn’t updated with anything. As part of the postmortem we’re working on we’ll ensure all order delays are shown clearly in-app.

4 Likes

Hi Ian, thanks… can I respectfully ask for a more detailed description though please?.. This appears an entirely predictable and preventable event, and something that I’d hope was fully covered and signed-off via extensive Non Functional Testing prior to putting peoples money on the line.

With the cloud infrastructure that you operate on there is no excuse for these forms of bottlenecks in todays market infrastructure unless the partners you work with hold constraints on you that for some reason cannot be overcome (which I’d like to understand better)

To be clear, the state the app goes into is highly confusing. If an order goes into a PENDING state (as I was eventually told on support by Jordan) then the status is not QUEUED for execution - it is unknown awaiting a reply from the partner… The app says “We’re still trying to place your instant order” which is not correct as it could very well have been executed (at a price unknown) and you cannot do anything about it.

For me this affected both buy and sell orders. In the sell scenario the app continued to report I had shares to sell (which I didn’t) and the activity feed had no reference to any pending/queued action that needed to resolve before I could attempt to do anything about it (which in actual fact I couldn’t)

I’m fully aware running infrastructure like this can be complex (having built the London Stock Exchange’s Infolect platform and having been accountable for Market Data globally at UBS in my past), but this is what a lot of investor money has been promised to in order to ensure robust, predictable and scalable service. If user signup needs to be paused until elements of the service can be improved then that should be a consideration @adam before reputation and trust is lost.

Incidents will occur, of course, but I’d like to point to the level of detail mondo/monzo went into with some of their early outages that gave confidence things would improve (RESOLVED: Current account payments may fail - Major Outage (27/10/2017) - #95 by oliver - Help - Monzo Community)… a similar level of transparency would do wonders here - especially when, in my case, I have 100x more money invested via your platform than sat in one of the many options to choose for current accounts.

Thank you

20 Likes

Hi @Ian, I will double down on @thepoisson request for full transparency on the incident. Today is the second time that a transaction got in internal pending status, costing me £1,000, once again. I would also recommend not to downplay the incident - it’s not that we couldn’t see the successful orders, our money were trapped during a turmoil of the market. I believe quite a lot of people have a technical background, me included, and the excuse of a bottleneck is not enough (and not acceptable, as the nature of your business relies heavily on spikes). My biggest worry is that, the last time that happened, the excuse was the same. I can’t stress enough how much I advocate for you, invested in your crowdcube campaign, and tolerating the heavy fees for US stocks compared to other platforms. But, as @thepoisson said very nice, only extreme transparency might make us not lose the trust, even if we lost our money.

10 Likes

Thanks @thepoisson @AlexChristod. I fully appreciate the need for transparency here. We’re going share more details including but not limited to:

  • how the order status is handled (and why that left you in such confused state)
  • technical details on why the bottleneck occurred
  • what we’re doing to mitigate this now
  • how our testing and development process will prevent this happening in future
14 Likes

Hi @Gemhappe, @Ian,

Has this issue been affecting deposits? I made a transfer from my bank just after lunchtime but it hasn’t appeared in my FT account yet.

Hi @FebruaryJones - this issue was specifically around trading. I’ll let the team know about your deposit though.

Hi @Freetrade_Admin

Is the US Market down today. Or FT has stopped processing…

I think there is an issue at the moment that is being looked into.

1 Like

Yes, recvd notification an hour later from FT.

Hi there :wave:

Can you drop us a message on in-app chat and we’ll check this out for you? :blush:

Thanks!

1 Like

Something similar happened last Friday where I tried to sell half my Apple shares. App said to check the activity feed to see if the sale had gone through. There was nothing there so tried to sell again. Ended up selling all my shares as both orders ended up being silently executed.

Yes this happened to me last week as well. I was told it was the same thing as happened yesterday.

As discussed yesterday, I wanted to follow up with more detail on the outage we had this week. I’ll discuss how orders are handled, why there was a delay, and what we plan to do/have already done, to prevent these issues from happening again.

Summary

  • Cumulative US order history (tens of thousands per holding) prevented our US partner from updating the status on orders.
  • Our app failed to display information on any orders which were delayed in this fashion (pending).
  • We have moved to a new integration model which shortcuts receipt of order status updates.
  • We’ll be improving the app to show pending orders more clearly.
  • We’ll improve our load testing to identify non-linear scaling issues sooner.
  • We’ll improve how quickly we identify, and then communicate these issues to customers.

Orders

Placing orders either through the UK’s retail service providers or via our US clearing partner is handled asynchronously. This means that successfully sending the order to our trading partner and confirming its status are two separate processes. When you place a trade within the Freetrade app we attempt to complete both placing the order and updating its status.

Once an order has been successfully created it is put into a “Placed” status after which we keep checking with our partner for any updates. A successful placement and execution of an order will move its status to “Complete” and the app will tell you that your order has been completed. In the case of our US orders on Tuesday, we successfully placed the trades with our US partner but we were unable to receive any updates on their statuses.

If we’re unable to confirm an order’s status within 45 seconds we inform you at the end of the order flow and continue to try in the background. Right now this is a poor experience for two reasons: 1) The message shown is not clear enough and can easily be dismissed; 2) If you leave the order flow this status is not displayed anywhere else in the app - it appears as if the order never happened. We are going to improve how we handle these exceptions to ensure you always know the status of your order.

Partner integration

The Freetrade Invest platform is designed to support different protocols for different trading partners. Our US trading partner now supports both HTTP and Amazon SQS for order updates. We have been using their HTTP service to query updates on individual orders.

Historically our partner has worked in a fully disclosed manner, meaning end users are interacting directly with their platform. In our case, we’re working as an omnibus account, meaning Freetrade is seen as a single user. This means we can provide a better long term service across multiple markets as well as providing greater flexibility for our customers.

A side effect of working in this omnibus fashion, which was not properly accounted for, was how it would affect the processing of individual holdings. Rather than a number of users with hundreds or maybe thousands of orders, we’re seen as a single user with tens of thousands of orders for a single holding. Before the HTTP service is able to respond with an order status it has to process an increasingly large number of historic records for each holding. As these orders have accumulated and order volume has grown the processing of these holdings has increased leading to yesterday’s outage.

We became aware of the impact of these cumulative orders about two weeks ago. Since discovering the issue both Freetrade and our partner have been working to address it. We have been migrating to their new SQS service which doesn’t block on processing before sending out status updates. And our partner has been working to improve their HTTP service. Unfortunately, neither of these changes were ready in time for Tuesday, leading to the outage. However, I’m pleased to say that our migration to the SQS platform is complete and has now gone into production - mitigating this issue and improving US order response times.

Testing

Our Invest Platform undergoes a number of checks before launching any new features. We have a CI environment which runs a suite of unit, integration, driver and feature tests on a nightly basis. Once our automation test suites have passed we have a pre-production environment that goes through manual validation with our QA team. In addition to this our production changes are often gated by feature flags which allow us to deploy features in a controlled manner.

As well as this functional testing we also carry out load testing. And, although our trade performance indicated that we were able to handle the load seen this week, it did not account for the build-up in historic orders. We had failed to correctly project the load testing data that we had. Assuming the performance was simply a factor of the order flow.

This is a definite weakness in our testing armoury and more detailed analysis of the performance profile of our trade execution at scale would have captured this. To this end, our teams are working on a formal process to capture, analyse and project this usage more accurately in future.

Thank you again for your patience whilst we put this together and we appreciate the feedback!

29 Likes
  1. Sorry for my technical ignorance but will using Amazon SQS in addition to HTTP still result in FT’s being a seen as a single user from the partner’s perspective?

  2. Does using Amazon’s SQS increase the bandwidth (number of orders, ‘historic orders’ you are referring to, that can go through at a single time) and can be scaled to any number of users?

  3. Will their be any changes to the notification system within the app in case of problems? There seemed to have been a mixed bag between the community posts and in-app notifications for impacted clients.

Thanks @szb - that’s a great question. Once we receive an update from either service we can update the order status. Right now the SQS service wins out most of the time. As we don’t use anything from the HTTP service we will remove it once we’re confident the change is completely successful.

3 Likes

If I understood it right the orders were executed correctly and the problem was with updating the order status only. Thanks for the feedback @Ian

In a nutshell, the update of the order status should be resolved after migrating to Amazon’s SQS.

The issue still remains with notifying customers in case of any problem arising.

If the community supports this, is it possible to have a notification that it is shown for every user about any ongoing issue in the app to avoid any ambiguity.

Discovering any issue on the community forum or after having already executed an order via the live chat seems to be a poor experience, unless flawless execution and the app functioning can be guaranteed by FT.

1 Like

Exactly that. The switch to SQS should have fixed yesterdays issue.

I like your idea too. Like a list of known open issues? Let me talk to @Duncan and the rest of the product team to work out how best to do that.

5 Likes