Shandong Energy | High-availability transformation of the image reimbursement system for a 199,500-person group

Enterprise overview

Shandong Energy Group is a state-owned capital investment company in Shandong Province's energy industry and an important backbone enterprise in China's energy industry, undertaking the primary responsibility of safeguarding energy security and optimizing energy layout and energy structure. It has 199,500 employees and 608 affiliated enterprises, including 8 listed companies and 5 New Third Board-listed companies, distributed across 22 provinces (regions) in China and 17 countries (regions) overseas. In 2024, it achieved operating revenue of 856 billion yuan, ranking 1st among China's Top 50 Coal Enterprises, 23rd among China's Top 500 Enterprises, and 75th in the Fortune Global 500.

Business pain points

• Lack of high availability: applications, databases, and storage are centrally deployed in a single data center without clustered redundancy. A single-point service failure triggers a business disaster, such as a standalone database outage making the entire image reimbursement system inaccessible
• The system has zero disaster recovery capability, and a regional failure directly paralyzes business.
• Database access is not layered. Frequently accessed "reimbursement images from the past 3 months" compete with low-frequency "data from 5 years ago" for retrieval resources, and retrieval latency soars from 1 second to 10 seconds
• No degradation or fault tolerance: minor failures easily escalate into avalanches. For example, one network fluctuation caused a 40% upload failure rate, and repeated employee retries tripled server requests, further crushing the application

Solutions

• Implement high-availability and disaster recovery transformation for the imaging reimbursement system: clustered redundant deployment of applications and databases, eliminating single points of failure
• Establish cross-data-center disaster recovery capabilities, so regional failures no longer cause overall unavailability
• Store image data in tiers by access frequency, separating high-frequency recent images from low-frequency historical data, avoiding contention for retrieval resources.
• Complete degradation and fault-tolerance mechanisms, limit retry storms under abnormal scenarios such as network fluctuations, and prevent small failures from escalating into avalanches.

Results achieved

• The image reimbursement system has moved from single-room single-point deployment to clustered redundant deployment, so a single node failure no longer causes overall unavailability
• After image data is tiered by access frequency, retrieval latency for recent reimbursement images returns to normal levels
• After degradation and fault-tolerance mechanisms go live, anomalies such as network fluctuations no longer amplify into server overload due to retry storms