- City of San Diego, CA -

Mitigating a Potential Catastrophic Sewage Spill

Sewer Pump Station No. 2
Electrical + I&C; Power Reliability and Surge Protection
Author: Eric Lovering, PE, Vice President + Chief Engineer

page1-hero-1


Download the White Paper

Click here to download the white paper!

 

Download the White Paper

page1-1

 

page1-1-2

BACKGROUND

 
The City of San Diego’s (City) Public Utilities Department’s (PUD) Pump Station No. 2 (PS2) is the largest pump station in the City’s wastewater system and handles close to 80% of the sewage flow generated by 3.3 million residents who live in the greater metropolitan service area.

The station is situated along San Diego’s high-profile, picturesque, famed San Diego Bay and west of the San Diego International Airport. PS2 is capable of pumping up to 432 million gallons per day (mgd) of peak wet weather flow. This critical facility is central to the successful operation of the City’s entire wastewater system; therefore, the power reliability of PS2 is of utmost importance to the City.

As a large-scale water infrastructure project, PS2 requires a total of 18,000 horsepower (hp) to convey 432 mgd of sewage to the Point Loma Wastewater Treatment Plant. To put this into perspective, PS2 has enough pumping capacity to fill a sports stadium up to the roofline with sewage in a single day!

In 2020, PS2’s intake was an average daily flow of approximately 160 mgd. PS2 receives gravity flows from the 96-inch North Metro Interceptor, which serves the northern San Diego region, and the 108-inch South Metro Interceptor, which serves the southern communities of San Diego. Wastewater is collected in PS2’s wet well and is then pumped into two parallel 87-inch diameter force mains to the Point Loma Wastewater Treatment Plant. These two force mains were built in 1963 and were not designed to handle a hydraulic surge event.

 

 

page1-2

SITUATION

 
Preventing a Hydraulic Surge Event

Hydraulic surges (aka “water hammer”) can occur if there is a sudden and total loss of pumping power. This sudden stopping of all the pumps can create a damaging pressure wave. Condition assessments of the City’s force mains showed that these pipelines are susceptible to rupture from a hydraulic surge, which could lead to a catastrophic sewage spill into San Diego Bay. Not only would a sewage spill cause irreparable harm to the environment and the City’s public image, but the cost of a spill could be as high as $50 to $100 million, depending on the duration and severity of the event.

Challenges of a Legacy Power System

Given the catastrophic stakes of a potential surge event, for the past 30 years the City has mitigated the risk of hydraulic surge by implementing two independent sources of power so that no one failure could trigger a sudden and complete loss of pumping power.

page2-1-1

 

page2-2

PS2 is uniquely equipped with two existing reciprocating natural gas engine–driven pumps that are independent of the local utility electric power provider, Sempra Energy’s San Diego Gas & Electric (SDG&E). The existing onsite 2,400 hp direct-drive engine–driven pumps provide redundancy to the off-site utility power system to ensure that no single point of failure shall cause all the operating sewage pumps to stop and cause a surge event. The natural gas engine–driven pumps were installed in the early 1990’s and had, thus far, worked as intended to prevent hydraulic surge.

PS2’s electrical motors are fed from 3 electrical feeds from two SDG&E substations. If one substation were to fail, then only one substation would be left to serve PS2. The existing agreement between the City and SDG&E limited a maximum of two pumps to run per utility feed under normal flow conditions and three pumps per utility feed under emergency conditions, which is necessary during high flows in the rainy season. Under emergency conditions (with a substation out of service), only five pumps would be available: three electric powered pumps powered by a single substation plus the two natural gas engine-driven pumps.

In late 2010 the City had two issues at PS2 that needed addressing:

  1. The natural gas engine–driven pumps which were installed back in the early 1990’s, and had prevented any hydraulic surges from occurring, were reaching the end of their service life and would need to be replaced.

  2. The Environmental Protection Agency (EPA) guidelines recommend that critical wastewater lift stations be equipped with two separate and independent sources of power.

Replacing the natural gas engines would be a major capital investment, but even if the NG engines were replaced, PS2’s configuration would not meet EPA Reliability Guidelines.

The City recognized that the time was right to take a step back and reevaluate the entire collections infrastructure at PS2. So, in early 2011, the City’s Public Utilities Department (PUD) commissioned a study to analyze different alternative upgrade paths.

An Unplanned Stress Test: The 2011 Southwest Blackout

On September 8, 2011 at 3:48 PM the Southwest Region of the U.S., including 1.1 million residents of San Diego County experienced a system-wide blackout that would last approximately 25 hours until power was fully restored.

page3-1

Thankfully, PS2 did not spill sewage during that blackout event and the system did not experience a hydraulic surge since the 2 existing natural gas engine–driven pumps worked as designed and kept at least partial flows pumping during the power outage. The two pumps were able to maintain operation because natural gas services were not immediately disrupted from an instantaneous systemwide power outage. Residual pressure remains in gas pipelines which allowed the gas provider more time to bring backup generators online without disrupting service.

page3-2

 

In addition, many of PS2’s outlying upstream pump stations were not equipped with onsite emergency diesel generators during the 2011 blackout, which limited sewage flows conveyed to PS2.

The Southwest Blackout proved to be an unplanned stress test that revealed the strengths and weaknesses in the City’s wastewater emergency power plan/operations. It was also necessary to consider if the City had avoided a massive sanitary sewage overflow merely as a function of timing, weather, or luck.

As a result, all outlying pump stations that feed the interceptors that were influent to PS2 had onsite emergency diesel generators installed. This allows for the outlying pump stations to maintain pumping service in the event of another blackout, but would increase demand at PS2 during a blackout event. Temporary 2MW emergency diesel generators were also installed at PS2 in response but were only considered a stopgap measure until the permanent emergency diesel generators would be installed under this project.

The event opened the doors to a larger question for the City: how best should they address the potential risks of an unplanned power outage while complying with EPA reliability requirements and mitigating hydraulic surge?

page3-3

 

The Commission Results

The results of the 2011 commissioned study provided the City with two distinctly different paths for moving forward:

  1. Replacement of the Existing Force Mains: Since the existing force mains are susceptible to damage from hydraulic surge, replacing them with new pipelines would address the problem, but at a cost of $350 million. In addition, this approach would require major efforts for permitting and environmental mitigation, and the technical challenges of replacing these large diameter force mains include considerable design and construction risk. This option would also require installation of new diesel generators to meet EPA reliability standards.

  2. Construct New Onsite Power Generation: Constructing onsite power generation would provide additional power sources that are independent of the electric utility grid. This project could be constructed in a shorter timeframe and for a significantly lower cost of $60 million.

The City chose to proceed with option two.



"The key to the project success was LEE + RO’s interdisciplinary coordination and production capabilities."

 

CHALLENGE


In 2012 LEE + RO was retained as the prime consultant to perform a focused study to determine the different possible technical and energy options for an onsite power generation solution.


Starting First by Defining the Problem

The primary work for the LEE + RO team was to research and present potential solutions that would address the City’s six main goals:

  1. How to implement the EPA recommendations
  2. Determine the optimal surge protection solution
  3. Meet APCD regulations
  4. Provide power reliability
  5. Deliver operational flexibility
  6. Maintain cost effectiveness

The LEE + RO team considered many different alternatives and schemes for PS2’s onsite power generation. The research consisted of a thorough examination of potential energy sources and the appropriate mechanical equipment to convert them into pumping power. The energy sources considered included: Solar, Natural Gas, “Green” Gas / Biogas, and Diesel; and assessed the suitability of engine-driven pumps and engine-driven generators.

The recommendations of the 2013 Focused Study were to install natural gas engine–driven generators for prime power, with diesel emergency generators for backup power. While solar and biogas bring value, neither energy source is abundant enough; nor is there space to rely on either source for continuous operation.

The study showed that though numerous alternatives were considered, very few alternatives could meet and deliver the specific requirements of the project: this critical piece of infrastructure is located on a highly visible, environmentally sensitive, relatively small parcel of land and requires too much electrical power.


FUEL TYPES + GENERATION METHODS CONSIDERED

Technology Solutions Graphic PS2 v6

 


page6-1border

SOLUTION


For Prime Power Generation: Engine-Driven Generators

The greatest advantage to onsite natural gas engine–driven generators is that the onsite power can be applied to any of the eight pumps. This offers critical flexibility compared to the direct-drive engine–driven pumps, which can only supply direct mechanical power to the pump that is mechanically tied to it via transmission and driveshaft. The engine-driven generators can also run at 100% speed because they are locked to the generator, which must supply 60Hz rotational power. Running at only 100% speed allows for more efficient operation and cleaner emissions when compared to the direct-drive engine–driven pumps.

Even though this option requires larger engines to handle the inrush loads of the pump’s electric motors during pump starts, the excess power from the onsite power generation can be utilized for all of PS2’s ancillary loads, which saves the rate payers considerable money on operating costs. When looking at the system in its entirety, this alternative offered the lowest net present value (NPV).

This option also provided the best constructability option since the new onsite power generation facility can be built adjacent to the existing, operating pump station.

page6-2

 

Supplemental Backup Power: Emergency Diesel Generators

In addition to the two 3-MV natural gas engine generators, the project included two 4-MW diesel generators. At the time of design, these were the largest diesel generators commercially available on the market that could meet air permitting requirements. In addition, the site was fully built out and had no additional real estate to accommodate more than two additional emergency generators. The two 4-MW diesel generators provide EPA recommended power reliability for emergency utility power outages, while the two continuous duty 3-MW natural gas generators provide the cheap, clean, reliable power that is independent from the utility grid, as well as the electrical isolation to provide hydraulic surge protection.

It should be noted that the diesel emergency generators are not suitable for continuous duty to meet hydraulic surge protection requirements during non-emergency scenarios. The diesel engines have air permitting restrictions, and diesel fuel is a far more expensive form of fuel when compared to natural gas or the power from the utility grid.

RESULTS


Final Engineering Design Addresses the Two Critical Failure Scenarios

EPA guidelines do not specifically address a complicated sewage pump station scenario such as PS2’s. But, in the spirit of their guidelines, this project provides full backup power. The two failure scenarios considered to meet the EPA intent of redundant backup power and mitigate hydraulic surge at PS2 were as follows:

  1. Substation Utility Failure: Should an unplanned electric utility power provider failure occur during an extreme wet weather event, PS2 can operate four main sewage pumps with the two diesel generators, and up to three main sewage pumps with the two onsite natural gas generators. These two independent electric power sources (the onsite diesel-powered generators and the natural gas-powered generators) will be electrically isolated from one another during this emergency scenario to continue to meet the hydraulic surge protection requirements of the system even during an emergency event.

  2. Natural Gas Supply Failure: Should an unplanned natural gas utility power provider failure occur during an extreme wet weather event, the system will run off of the diesel generators and the electric utility grid. As in the first scenario, PS2 can operate four main sewage pumps with the two diesel generators. During normal operations, the electric utility provider only allows two pumps to run off of each utility feed. However, during an emergency, they will allow their substations to run overloaded and that will allow 3 pumps to run off of a single feeder. Thus, PS2 can operate up to seven main sewage pumps with all three of the electric utility feeds operational. There will still be two independent sources of electric power (the onsite diesel-powered generators and the offsite utility provider electric grid) that will be electrically isolated from one another to continue to meet the hydraulic surge protection requirements of the system, even during an emergency event.

page7

 

The New System Utilizes:
  • Two existing medium voltage switchgears that were modified to suit the project as well as a new switchgear to connect the four new sources of onsite power generation. The system has a total of three main switchgears located in two different buildings that are all interconnected, and this requires a unique engineering approach to provide safety and reliability. These challenges were overcome with physically hardwired safety interlocks as well computer-controlled logic via the distributive control system (DCS). In addition, the DCS system is physically separated from the internet with its own dedicated fiber optic network. This provides a controls system “air gap” to make sure that the system is secure and cannot be hacked by third parties.

  • PS2 was commissioned before modern variable frequency drives (VFDs) were commercially available. For three of the legacy pumps, the system uses liquid rheostat controllers for variable speed control. A liquid rheostat is an electromechanical device with hydraulic pumping and valving of conductive fluids with mechanical mechanisms to raise and lower rods in a fluid bath. This now-obsolete control system technology was installed in the 1960’s and is reminiscent of a Rube Goldberg machine. The technology is surprisingly robust and is still working effectively and reliably after 60 years of service.
  • The system also has three existing constant speed pumps controlled by reduced voltage auto transformers known as RVATs. These operate with the simple principal of a two-step starter. The first step is at lower power and the second step is at full power. This approach limits the starting inrush current required from the utility grid. The project team looked into replacing the existing RVATs, but decided it was not cost-effective to replace them at this time.

  • Motor inrush was a critical engineering consideration. Take a moment to recall Isaac Newton’s first law of motion, “an object at rest stays at rest and an object in motion stays in motion with the same speed and in the same direction unless acted upon by an unbalanced force.” To get an object moving, in this case a giant sewage pump attached to a 2,250 hp electric motor via 40 feet of mechanical shafting requires huge amounts of torque which is related to the required amount of inrush current or (thousands of) “starting volt-amps” otherwise known as SKVA.

  • The full load running current of a 2,250 hp electric motor is approximately 2,000 KVA, but the instantaneous inrush current when starting from rest can be upwards of three times that amount. The natural gas generators can produce the required power for steady state loads but do not have enough instantaneous capacity to handle such large starting KVAs, even though the utility power grid has more than enough capacity to start these motor loads. Thus, paralleling with the grid under low flow conditions is necessary for starting the liquid rheostats and RVATs to remain in compliance with the surge protection requirements of the system.

 

page8-1border

 

 

 

page8-2

Meeting Reliability and Surge Protection Requirements

To provide motor starting flexibility, we ended up replacing the two natural gas engine–driven pumps with new electric motors equipped with VFDs. The VFDs can be programmed to limit this instantaneous inrush current that occurs when starting up a pump. This means that we can be islanded from the utility, and still start the main sewage pumps.

PS2’s surge study analysis confirmed that a damaging hydraulic surge to the force mains can only occur when the system is operating more than three main sewage pumps. Thus, before we island our generators, they can be paralleled with the utility and start any of the three types of starters: liquid rheostats, RVATs, or the VFDs. This paralleling scheme also requires an interconnect agreement with the local utility provider, SDG&E.

The arrangement of the three switchgears and the bus ties, is known as a ring bus. This allows great flexibility in moving power around as well as opening tiebreakers to electrically isolate different sources of power. This ability to isolate different sources of power, makes this project unique in utilizing power reliability to control hydraulic surge protection.

Other major project elements at the site include 4,160VAC switchgear improvements, synchronizing relays, DCS and PLCs controls, over 800 hardwired inputs and outputs, highly efficient sewage shell and tube heat exchangers, two new 2,250 hp electric motors and associated VFDs, SDG&E switchyard improvements, electrical load banks, black start generators, fuel storage and containment, a new 8” natural gas service, and new ADA compliant office space and restrooms.

 

page8-3

Project Construction

One of the project elements for the power generation is a new 8,000 sq. ft., 45 ft high building to house the generators on the site. Located along iconic Pacific Coast Highway, the dynamic building will provide passerby’s a view of a brilliant white architectural finish of the upper building elements that mirror the depressions of the local vernal tide pools as well as colored concrete horizontal formliner impressions that replicates the color and coastal striations of the nearby beach cliffs.

The new facility will capture storm water runoff for collection at the site and pumped for treatment. Some of the improvements will include a maze of cooling distribution piping and systems will provide reliable engine cooling; 4160VAC electrical switchgear to safely transfer medium voltage power; accessible bridge cranes for maintenance of heavy project elements; an innovative energy-efficient ventilation and climate control system; as well as an engine exhaust and emissions systems that provide safety to the workers and neighbors. And just to put the air ventilation system scale into context, it is equivalent to one million people breathing out all at once.

The project’s construction effort is broken up into two main phases. The first phase is projected to complete commissioning in 2022 and the second phase should be sustainably complete in 2023.


ACKNOWLEDGEMENTS


Public works projects of this scale and complexity are a huge team effort. LEE + RO wants to thank and acknowledge other team members below that have made this project successful

Client: City of San Diego
Prime Consultant: LEE + RO
Contractors: Steve P. Rados, and Black & Veatch
Engineering Sub-Contractors: ABC Acoustics, Aguirre and Associates,
Allied Geotechnical Engineers, Bluescape Environmental, David Reed Landscape Architects,
Flow Science, Kocher Schirra Goharizi, and Swift Lee Office

Leadership_Eric

 

About the Author


Eric Lovering, PE

Vice President + Chief Engineer, LEE + RO
Licensed Civil & Electrical Engineer

As the Chief Engineer, Eric’s responsibilities cover a wide range of fields, from overseeing the entire engineering department to ensuring that all relevant standards are met throughout the completion of a project. He has successfully completed dozens of public works projects from engineering design through construction, start-up and commissioning. Eric is a pump station subject matter expert, which has led to guest speaking engagements for ASCE and CWEA. Eric has extensive training and hands-on knowledge in civil, hydraulics, mechanical and electrical, including instrumentation & controls (I&C), programmable logic controllers (PLCs) and SCADA systems.

He received his B.S. degree in aeronautical engineering from UC Davis. He holds California-registered licenses as both a professional civil engineer and a professional electrical engineer.


More Solutions

Download the White Paper