Client background & business context

  • Client profile: largest European electronics retailer 
  • Project: performance testing implementation 


Challenges

  • Capacity limitation: the client’s solution platform needed initial gauging in real-world scenario conditions. This would allow to better assess risks under a normal daily load or during peak sales periods. Based on the client’s existing measurements, the application was able to handle 400 transactions per second (TPS). As the company grew and diversified its needs, this number of transactions represented a potential risk.  
  • Undefined performance test strategy: performance testing revolved around a rarely conclusive end-to-end (E2E) approach, preventing early issue detection, by overlooking components, features, and endpoint-level isolated assessment.  
  • Limited test platform coverage: limited test scripts customisations at country or brand level. Limitations were also present in terms of request distribution, usage profile and load-sourcing patterns (mobile, web, app). 
  • Deprecated test-data script sourcing that were put in place for previous testing sessions 

Our approach

Setting up the performance testing strategy was our prio 0 aspect. While keeping close contact with the client’s development, infrastructure, and middle-ware team, we’ve started implementing and documenting a long-term, sustainable approach for performance testing and expanding coverage, in a prioritised manner. In doing so, we’ve provided our client with an early detection mechanism, accurate prediction strategies, invaluable load-model emulation, and scalable cross-brand and cross-country test coverage. All these were vital assets in ultimately fulfilling the end-customers’ desire for a customer-experience oriented shopping experience

Technologies

  • Apache JMeter and HP LoadRunner to replicate user load on the SUT
  • BlazeMeter and Octoperf 3rd party cloud provider
  • Multi-zone load distribution using AWS, Scaleway, DigitalOcean
  • Grafana , DynaTrace, Munin, AppDynamics, NewRelic, Kibana as monitoring tools (APM)
  • Jenkins, Taurus for CI / CD integration of Performance Validation in the pipeline
  • JAVA, C, Groovy, BeanShell as scripting languages
  • Containerised load generating infrastructure deployed on client premise

Results

One of the greatest achievements for this project was that we collaborated with our strategic partner’s developers to make the shift towards performance testing. We did it by changing the mindset and by looking at performance testing as the most important and final validation gate between the features and the end-user. This approach reshaped and amplified their overall inherent skills to a mass-user-centred perspective. Some of the benefits achieved through our work and close collaboration with the development teams were: 

  • Capability Buffered Solution: during our collaboration we’ve helped visualise pain-points, component strain, scaling factors, and balancing strategy issues. After this endeavour, we were able to deliver to the company comprehensive reports of issues to be one-by-one addressed and fixed. Our contribution enabled the platform to go from 400 TPS (in 2014) to 9000 TPS (measured during 2018’s Black Friday preparation). 
  • Full test coverage: test scenarios and reporting mechanisms were put in place. These assess 100% emulated user-behaviour, with integrated real user monitoring (RUM) and application performance monitoring (APM) solutions, allowing the company to act accordingly. 
  • Extended customisation: multi-environment, multi-brand, multi-country with on-the-fly scaling to any new resource accommodated by the solution platform is now a reality. We also implemented a custom utility framework that aggregates test-data from multiple sources, making it seamlessly available for all test scenarios. 
  • OpenSource: at the heart of our test scenario strategy lays the ubiquity of test-script deployment. That’s why we’re focusing on OpenSource-driven solutions. We delivered full compatibility with small-scale debugging approaches and large-scale live assessment via in-house or commercial SaaS solutions. 

Related case studies

Accounts Payable Process Automation

Cloud-based, automated invoice processing software solutio ...

Cloud-First Strategy: Microsoft Exchange to Office 365 Mig ...

On-premises Microsoft Exchange infrastructure migration to ...

E-commerce migration to SAP Commerce Cloud (SAP Hybris) fo ...

Due to the enhanced customer-oriented approach, the new SA ...

München

Mies-van-der-Rohe-Strasse 6, c/o Design Offices
80807 München, Germany
E-mail: hello (at) accesa.eu

Cluj-Napoca

Constanta 12, Platinia
400158 Cluj-Napoca, Romania
E-mail: hello (at) accesa.eu

Oradea

Cetatii Square 1, Oradea Plaza
410520 Oradea, Romania
E-mail: hello (at) accesa.eu