Performance testing for continuous volume testing

• Client profile: largest European electronics retailer 
• Project: performance testing implementation 

• Capacity limitation: the client’s solution platform needed initial gauging in real-world scenario conditions. This would allow to better assess risks under a normal daily load or during peak sales periods. Based on the client’s existing measurements, the application was able to handle 400 transactions per second (TPS). As the company grew and diversified its needs, this number of transactions represented a potential risk.  
• Undefined performance test strategy: performance testing revolved around a rarely conclusive end-to-end (E2E) approach, preventing early issue detection, by overlooking components, features, and endpoint-level isolated assessment.  
• Limited test platform coverage: limited test scripts customisations at country or brand level. Limitations were also present in terms of request distribution, usage profile and load-sourcing patterns (mobile, web, app). 
• Deprecated test-data script sourcing that were put in place for previous testing sessions 

• Apache JMeter and HP LoadRunner to replicate user load on the SUT
• BlazeMeter and Octoperf 3rd party cloud provider
• Multi-zone load distribution using AWS, Scaleway, DigitalOcean
• Grafana , DynaTrace, Munin, AppDynamics, NewRelic, Kibana as monitoring tools (APM)
• Jenkins, Taurus for CI / CD integration of Performance Validation in the pipeline
• JAVA, C, Groovy, BeanShell as scripting languages
• Containerised load generating infrastructure deployed on client premise

One of the greatest achievements for this project was that we collaborated with our strategic partner’s developers to make the shift towards performance testing. We did it by changing the mindset and by looking at performance testing as the most important and final validation gate between the features and the end-user. This approach reshaped and amplified their overall inherent skills to a mass-user-centred perspective. Some of the benefits achieved through our work and close collaboration with the development teams were: 

• Capability Buffered Solution: during our collaboration we’ve helped visualise pain-points, component strain, scaling factors, and balancing strategy issues. After this endeavour, we were able to deliver to the company comprehensive reports of issues to be one-by-one addressed and fixed. Our contribution enabled the platform to go from 400 TPS (in 2014) to 9000 TPS (measured during 2018’s Black Friday preparation). 
• Full test coverage: test scenarios and reporting mechanisms were put in place. These assess 100% emulated user-behaviour, with integrated real user monitoring (RUM) and application performance monitoring (APM) solutions, allowing the company to act accordingly. 
• Extended customisation: multi-environment, multi-brand, multi-country with on-the-fly scaling to any new resource accommodated by the solution platform is now a reality. We also implemented a custom utility framework that aggregates test-data from multiple sources, making it seamlessly available for all test scenarios. 
• OpenSource: at the heart of our test scenario strategy lays the ubiquity of test-script deployment. That’s why we’re focusing on OpenSource-driven solutions. We delivered full compatibility with small-scale debugging approaches and large-scale live assessment via in-house or commercial SaaS solutions.