In a very short time we released V1.7.5 -> 1.7.8 of our very popular YourSites component to manage multiple Joomla! and Wordpress sites.
Why did we release so many versions so quickly?
Version 1.7.5 came with some new features and significant performance gains within the list views when viewing hundreds of sites. Great! however, we started getting reports of complete lockups of the extensions view, sites view and even the backups view. This really concerned us, we had done our pre-release testing on a variety of configurations - some on local servers and some remote and had not encountered these issues. Strangely, after the reports started to come in, we found that the problems seem to be related to the speed of the internet connection - a user with full fibre broadband was seeing a problem and another on a 3G mobile phone connection to the same YourSites server was not seeing the issue.
Our initial thought was it must have been how we were rendering data for UiKit (the CSS and JS framework we use) to process so we made some CSS changes not to hide certain elements to make YourSites was functional again and released V1.7.6. This version resolved the issues on the sites we were able to test with at the time.
The problems continued however and so we were forced to release 1.7.7 which re-ordered the loading of the scripts sinc this seemed to resolve the issue on the sites we tested. However, fundamentally we were just reducing the problem and not solving the key issue. It was an extremely tricky problem to solve given how it locks up the browser until you stop it, didn't happen on every page load and seemed to be dependent on the web connection quality of the user. We were unable to run it through the JS debugger easily.
We removed all hidden siteInfo data from the site view and started loading it on demnd - which helped performance but didn't resolve the issue.
We then went back to first principles and isolated everything that had changed between V1.7.4 and 1.7.5 and we realised that UIkit was updated to version 3.1.6 as part of our build script, so we reverted UiKit back to 3.1.4 and instantly our problems were solved. We were then able to isolate the issue as something that changed in UIkit between version 3.1.5 and 3.1.6. We therefore packaged up V3.1.5 of UIkit as part of YourSites 1.7.8 for a few very patient users who had offered to test it.
How will you stop this from happening again?
Quite simply we have removed the auto-updating of UiKit and we are offering a few long term users free subscriptions for their efforts in testing pre-releases too. So even more testing :-), we already had automated tests and use YourSites to manage our own client sites and they help greatly. But, as these issues demonstrated, they just can't pick up all issues, especially ones like this!
We care for all our users and realise how important YourSites is to them and greatly appreciate all feedback and comments. It's you users who make YourSites what it has become today, we will continue to improve, be open and honest at all times.