- Not only do simulations, but also test in real life
- Improve not only the software but also the organizational processes
- Monitor every process where possible. There is never too much graph.
Inaccessible systems at peak times are a specter for any organization. Whether it's the webshop during the holidays, or the planning system in case of a legal change: you don't want this. How do you properly prepare the organization and systems for such a situation of stress? You know it's coming.
Every industry has its periodic moments when the servers run overtime to do all the math. It could be a peak in customer demand or a scheduled time when almost all records in the database need to be updated. “Black Friday in retail is well known, but the insurance industry has its own Black Friday in the first week of January,” says Joost van der Maarel. At Novulo, he is responsible for optimizing customer success with the platform. “When the inflation adjustment is incorporated into insurance premiums in early January, everyone should get a new policy sheet. There are insurers who are unable to do back office work for a few days. But you'll just want to overwrite your new car. If it takes longer than a weekend, that requires improvisation in the workplace.”
Van der Maarel has experience in preventing problems at peak times. “It's not like if you expect three times as much traffic, you can assume you'll be able to do with three times the normal server capacity.” Often, there are different processes that interact with each other. This causes more stress on the organization than expected. “There is a good chance that you will miss out on something.”
Van der Maarel gives a number of tips:
- Start stress-testing the software on a test environment
A test environment simply offers a safe environment to see where the weaknesses in the software or infrastructure are. There are many automatic stress tools available to increase the pressure on the system under different conditions. If historical data is available, such as log data from the last “Black Friday event”, it is possible to accurately simulate a real-life peak moment.
- Use as much monitoring of sub-processes as possible and analyze all graphs
The Novulo platform offers plenty of tooling to monitor system performance and discover where congestion occurs.
- Optimise code and database
The instructions that Novulo's Architect development platform generates all consist of plain C# code. The Windows or Azure environment where the code runs is optimized to run C# code. The code is already compiled during implementation, making it extremely fast while using the application. However, it happens that components in the catalog can be improved, for example because a process in one organization is slightly different than in another. Then the module code can be modified or the use of the database can be optimized.
- Use caching where it's obvious
One of the improvements that often yield results is the use of caching when the same calculations are repeated over and over again. An example is when a daily turnover has to be updated every ten minutes and therefore all amounts on the invoices are always added. They no longer change, so it is better to remove the total invoice amount from the cache. With automatic caching, it can even be checked if anything has changed so that a new calculation is needed.
- Look at scaling up cloud processes
Although cloud providers like to advertise that their services scale up automatically when the pressure on the system increases, actually setting up and making optimal use of them can be a time-consuming task. Finding out in time what options there are and what costs are associated with them is recommended.
- Perform realistic stress tests
No simulation can ever replace a real-life test. Only when the entire organization holds a real “fire drill” will all the weaknesses come to the surface. So create a realistic test under well-controlled conditions. Save up a few days' work and then try to make up for this backlog “with all hands on deck” in the shortest possible time. Do this at a time when run-out is possible.
- Improve organizational processes
Such a real life test not only highlights the software problems, but it also becomes clear, for example, where people get in each other's way at peak times. By creating more space in the flows with a different setup, the speed of the process improves.
- Increase the pressure until it really goes wrong
Look not just at the weakest link, but at the top five possible problems. If the weakest link has been fixed, one of the following items on the list is the problem. Tackle those right away.
- Use iterative optimization
Simulations and real-life stress tests reveal potential improvements. These can then be executed. However, this does not mean that the organization is completely ready for the next moment when it comes down to it. Improvements to one process may reveal or even cause weaknesses in another process. So ensure regular tests in the organization before the real stress period occurs again.
- Document improvements and communicate them to users
When changes are made to the systems and workflows on the floor, it is important that everyone affected is also aware of them. Good documentation also serves as a good basis for the next iteration of the stress test in the organization.
