Mineral and white oils are often considered as heat transfer fluids for use in immersion cooling of...
Mineral and white oils are often considered for use as heat transfer fluids for immersion cooling of bit miners and electronics due to its low cost and dielectric properties. Unfortunately, most potential and current users of mineral oil are unaware of the corrosive effects of the many naturally occurring sulfur compounds that exist in mineral and white oils.
In this post, I want to highlight the corrosive sulfur problem and the impacts it can have on your servers and electrical equipment. Let’s start by clarifying that this is not a new problem; in my thirty career as a developer of dielectric fluids for use in the power and transformer industry I’ve wrestled with this in transformer cooling applications for decades (See Notes 1,2, 3 below). However, what I have noticed is that the frequency of sulfur corrosion and the sulfur levels in many mineral oils has been increasing over the past decade due to changes in the petroleum industries source of crude oils and in their bid to reduce the costs of processing.
Sulfur is a naturally occurring element found in almost all crude oils extracted from the ground. In the past, refiners would avoid using high sulfur crude as it caused corrosive damage in the refineries and required additional processing to meet product specifications. However due to changes in the market availability of “sweet crude” we are seeing a far greater variety of different sulfur compounds make it through the refining process unchanged with the result being an increasing sulfur content in most mineral and white oils (going forward I’ll refer to both of these as “mineral oil”).
Not all sulfur is inherently aggressive – but several types can later turn into one of the corrosive forms of the compound, the most corrosive being dibenzo-disulfides (DBDS).
Yes, you read that right – over time, noncorrosive types of sulfur can change in use into a corrosive type
. These corrosive sulfur species then attack electronics in one of two ways:
Attack Mechanism One: Direct Attack on Copper and Other Metals
Corrosive sulfur, as its name implies, directly attacks the copper and zinc materials in electronics. The corrosive sulfur erodes these metals from circuit boards and electronic components typically attacking areas where current is flowing, and the charged sulfur particles are attracted. This is often seen when mineral-oil based heat transfer fluids and lubricants are used with copper or silver bearings, or with exposed copper and zinc on circuit boards. See Figure 1, which illustrates a relatively early, but widespread sulfur-induced corrosion of a circuit board that has been immersed in mineral oil.
Figure 1, Direct Corrosive Attack
Figure 2 shows corrosion of a bearing race, where silver plating has been attacked and removed, causing failure of the bearings.
Bearing Raceway Corrosion by Sulfur
Attack Mechanism 2: Combination of sulfur with copper ions and precipitation of Copper Salts
As sulfur erodes the copper, zinc, and other metals on circuit boards, the sulfur ions combine in solution with copper ions forming copper salts such as Cu2S (Copper Sulfide), CuSO3 (Copper Sulfite) and CuSO4 (Copper Sulfate) in the oil. These metallic salts are not soluble in mineral oil and immediately begin to precipitate, or in laymen’s terms – to form crystals – onto “substrates”, as they’re called, which in this case means on circuit boards, insulation, or spacers – any structure where the salts can begin to crystallize and build up their lattice structures. Copper-sulfur salt crystals are highly conductive and build rapidly so they easily bridge connections the large windings in power transformers with explosive results. This phenomenon is only seen when mineral oil-based dielectric heat transfer fluids are used. Short circuiting by copper salts accumulations causes tens of millions of dollars in damage every year in the power transformer industry – this happens even easier on circuit boards due to their highly dense traces.
In their comprehensive study, “Case Study of Printed Circuit Board Corrosion and Countermeasures”, NTT Corporation describes these failures and their root causes in great detail. (6)
Figure 3 shows copper-sulfur salts that have precipitated onto and caused failure of a circuit board immersed in the coolant.(6).
Figure 3, Copper Salt corrosion causing circuit board failure (photo thanks to NTT Corporation)
Corrosive Sulfur Failures Are Often Mis-Attributed
Device failures associated with corrosive sulfur in mineral oils were relatively rare before the 1980’s. Crude oil sources and refining methods changed in the 1980s and 1990s, which caused corrosive sulfur to become far more common and the trend has increased ever since. One of the most difficult things about addressing corrosive sulfur issues is that these types of failures are often consider random device failures because the mechanism of corrosive sulfur attack is not well understood or well considered. As more applications for Single-phase, Liquid Immersion Cooling of electronics are evaluated, the incidence of mineral oil-based failures due to corrosive sulfur will undoubtedly rise and be recognized as a noticeable factor in data center and electronic equipment failure rates.
How did Sulfur Get Into the Mineral Oil?
Sulfur is a naturally occurring material in crude oil. In fact, there are over 30 sulfur compounds normally found in mineral crude oils. Older refining methods, such as acid-clay treatment or severe hydrogenation, removed these sulfur species, leaving a clean, non-corrosive petroleum base oil. Today’s refining methods, however, have as their goal the maximization of fuel output from a barrel of crude oil, and the naturally occurring sulfur compounds are not removed or changed. These sulfur species are then present in the lubricant base oils and cause problems with any active metals that are exposed to the mineral oil.(4)
Is This A Real Problem?
Corrosive Sulfur is not only real problem, it is very costly one. The cost of Sulfur Corrosion due to mineral oils can run into 10 millions of dollars each year and that does not include the maintenance, mitigation, damage and downtime created by these events. It is important to understand that this problem is not limited to lubrication of industrial equipment, in fact it’s responsible for millions of dollars of damage to electrical apparatus annually (5). Device failure as a result of sulfur attack is not uncommon. As Mr. Lewand puts it in his paper: “The extent of the corrosion damage caused by sulfur, if left unchecked, can be so severe as to cause failure of the apparatus”.
Are we seeing Sulfur Damage in the Data Center?
Data Centers and cryptocurrency mining using mineral oil as a dielectric are a relatively new application and therefore most users haven’t been in operation long enough to experience this failure mode. When I consult with mineral oil users that are experiencing what they perceive to be increasing number of random device failures I always start my investigation in two areas. One, I check for bad electrolytic caps on power supplies that have failed due to mineral oil absorption of the sealing plug. Two, I always require that we take a sample and run a sulfur level content test. It’s very common for us to find that their mineral or white oil dielectric have unexpectedly high level of sulfur and therefore also badly contaminated with Cu2S, CuS03, CuSO4. All are sure signs their system is experiencing a high degree of sulfur corrosion and its highly likely that they are experiencing random shorts. Any mineral oil with a greater than 3 ppb levels of sulfur is going to experience some level of corrosion, the damage and speed at which the problem propagates will depend on the operating temperatures, flow rates, device composition, and amounts of copper and zinc on your devices.
If I Already Have A Sulfur Problem, Can It Be Fixed?
The most troubling part of Corrosive Sulfur is that once you have the problem it’s impossible to fix the damage done to the electronics. The copper and zinc losses are at a microscopic level and typically very wide spread. In addition, because of the complexity of electronic circuits and their sensitivity to signal propagation these device failures are often cataloged incorrectly as component failures, when in fact they are a coolant failure. This often means that users continue to put new equipment into the mineral oil that already has elevated levels of Corrosive Sulfur and therefore not only accelerating the problem but increasing the level of damage to the new and old equipment alike.
The only way to fully stop the progression of the problem is to replace the mineral with a fully synthetic dielectric. However, this will not repair the damage already done to the electronics. If you are already using raw mineral oil as a dielectric coolant, then check your sulfur levels (again, < 3 ppb is considered satisfactory) and if necessary Engineered Fluids can provide you with a highly activated sulfur inhibitor additive that we developed for use in the power transformer market that can slow down the corrosive rate and buy you time to save your equipment.
What Can I Do To Avoid Sulfur Problems?
The simplest and most effective way to avoid issues with Corrosive Sulfur is to use a dielectric coolant that has no sulfur content. When I formulated our Engineered Fluids’ ElectroCool, AmpCool, and VoltCool single-phase dielectric coolants I specifically designed them to have a zero ppb sulfur content. My whole purpose in designing our fully synthetic, purpose-built dielectric coolants was to avoid all these common issues we had wrestled with mineral and while oils for so many years in the transformer industry.
Notes and Citations:
1. “Lubricant Induced Metal Corrosion”, L. Kogel https://www.linkedin.com/pulse/lubricant-induced-metal-corrosion-how-impedance-can-kogel/
2. “Bearing Damage: Corrosion”, A White Paper by Waukesha Bearing, Inc. https://www.waukbearing.com/en/technical-resources/bearing-damage-index/bearing-damage:-corrosion/
3. “Testing for Corrosive Sulfur Effects”, L. Lewand, https://www.netaworld.org/sites/default/files/public/neta-journals/NWsu07-Lewand.pdf
5. The Role of Corrosive Sulfur in Transformers and Transformer Oils;
Lance R. Lewand, Doble Engineering Company, USA, 2012
6. U.S. Bureau of Standards, “Sulfur in Petroleum Oils”, C. Waters, 1921
7. “Case Study of Printed Circuit Board Corrosion and Countermeasures”, https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201207fa4.html
