About the Organization
Our client is a Class I railroad and one of the nation’s premier transportation companies. It operates the most extensive intermodal network in the East, consisting of approximately 19,500 route miles in 22 states and the District of Columbia. It serves every major container port in the eastern United States and provides efficient connections to other rail carriers. It is a major transporter of industrial and agricultural products, construction materials, energy, automobiles, and automotive parts.
Challenge
The batch processing system for its integrated pricing model had a high failure rate and extremely long run times. Large batch processing would incorporate 600 lane opportunities with an average processing time of four (4) hours.
The organization had a legacy application designed to align opportunities to offered services. The system was accessed by many users on a daily basis to respond to RFQs, processing hundreds of origin/destination pairs in batches. The process was composed of web service calls, cross-domain data aggregation and calculations. The customer suspected a memory leak was causing the system to crash resulting in extremely long run times.
Observations
The recurring process failures had a significant impact on business performance — batches had to be ushered through the system and waiting for completion. If there was any failure during processing, the system would have to start the entire process over. As a result, users were unable to work on other tasks and they were experiencing poor turnaround times when responding with quotes.
Strategy
Prime 3 took a surgical and systematic approach to analyzing the systems and processing.
-
Profiled the system for run times and memory usage. We were able to quickly identify the most resource intensive steps in the process and conclude there was in fact no memory leak.
-
Identified the source of the client failures and made immediate improvements to reduce failures. The first change corrected the user interface to make it more resilient, allowing the batch to run to completion with any individual process failures marked as errors, rather than failing the entire batch.
-
Developed a full set of integration tests from the existing service. This is a safety net to ensure any software changes would not corrupt the system results.
-
By introducing a series of timers into the code, we were able to further benchmark the code from the initial memory profile. This also gave us our baseline run numbers that we could use for performance testing our enhancements.
-
Using the timer results, we addressed the longest running processes first. We used a series of techniques to standardize the data, cached data that was frequently requested, eliminated redundancies, and introduced multithreaded processing. We systematically worked down the timer results from least performant to most in order to address each step in the batch process.
Results
Prime 3 took the average performance processing time of 600 lane opportunities from 4 hours to 90 seconds. We made these changes over the course of three weeks. The return on the customer’s investment from these changes was achieved within a week. Importantly, the pricing analysts were freed to focus on improved customer service and capturing more business.
Our strategic approach allowed us to make relatively minor code changes that had a significant impact on the business process. The fix included installing a client calling web service for a large batch process and incorporating prior fixes that upgraded memory on the application server and client hardware.