[RCaffin]

I am going to 'have two bob each way' here. (Australian slang from gambling on horse racing.)
I think GITRDUN has or had multiple problems:
1) Bad noise problems from outside (another, possibly distant, factory, especially at start-up time in the morning)
2) High leg delta transformer system of dubious parentage
3) Poor grounding (high resistance)
4) Possible lightning strikes somewhere else, maybe

One thing which was emphasised to me many years ago by a grid engineer is that the power grid is not a passive collection of zero-impedance wires linking you to the generators. It is an extremely complex electrical system (yes, with inductances and capacitances) which varies hugely over time as people connect and disconnect all sorts of equipment to it. There can be high impedance points in the network, and low impedance points. Some places will see huge noise spikes while other places see little. And this can change during the day.

Point 2 has been fixed. Point 3 can be alleviated with only a small effort - mainly extra ground stakes, but never entirely solved. It's worth doing anyhow. Point 1 cannot be stopped in itself, although a Ferro Resonant Transfomer will block the effects. Point 4 is also blocked by the FR transformer.

A thought, of no consequence really: if GITRDUN instals a large FRT on his factory, this will change the local grid so someone else may now have problems. Such is life.

Cheers

[wizard]

GITRDUN

Start with a complete go through with the Grounding & bonding of everything in the building,to correct what is wrong in this area, You need someone that knows what they are doing to put it right

This is the key and frankly why I questioned using the same electrician for the last upgrade to the plant. I also have to wonder what is up with the power company, they should have demanded a properly grounded system when the business changed over.

The other hard reality here is that this isn't likely to be solved cost effectively in this forum!

The fired phone is possibly the most interesting clue here. I'd personally take both apart to see what happened to them. If they have been fried due to high voltage then the power company has some sort of power regulation problem or there was a lightening strike someplace in the area. That could be caused by a company many miles from the plant in question.

[wizard]

You don't have Loki of the Asgard visiting your plant in the middle of the night do you?

Ok i have an update and another riddle.

The main shop which is now running the 208V WYE power had a weird occurance this morning. When we came in to work and powered up the machines 3 machines came on with alarms that wouldnt clear. Same old story. Also one employee noticed his radio wouldnt come on.

Did you have your voltage and transient recorder running? You really can't give up on collecting data with the assumption that the change to a Wye based system will help anything.

My maint man checked voltage on the 110V outlet and it checked fine, then checked all three legs of the 3-phase and checked ok from phase to phase and also phase to ground. After about 2 hours he noticed the power light on the radio was on so he turned up the volume and it was working fine. So we then turned the three machines on and two of them came on and run fine with no alarms, one had a breaker tripped in the back but nothing blown. The third machine has the spindle drive blown up.

Im not quite sure what to make of this but no doubt its an indicator as to where our true problem lies.

It would be nice to know what the line voltages where at the moment of power turn on.

The interesting part here is that you indicate that the machines where turned on, but what does that mean? Was a disconnect flip on or did somebody just push the " power on " buttons at the control panel. Most likely your machines are being left idle not really turned off in the sense of being isolated from AC power. However the damage to the radio and cell phone kinda indicates that external factors are at work here.

There is or are some exceptions here. For one does anything run automatically in your plant before you get there? I mean anything here from furnaces to air compressors to induction heaters to what have you. Right now we assume that the problem is external to the plant due to Loki crapping on you for some unknown reason. However jumping to conclusions have lead many astray in the past, so the question is what happens in your plant in the morning or for that matter through out the night.

In any event that transient record has to stay running until there is clear evidence that the problem has been solved.

[GITRDUN]

I should have elaborated a little, the machines are now being shut completely off at the end of the day. That is the control is turned off and then the breaker on the back of the machine is switched off. Power on is just the opposite.

The phone will power on but it gives an error message and doesnt work. The machine that failed had 2 blown drive modules in the spindle drive and had a bad chip on the board.

I will be recording the line voltages again now that i know the problem aparently isnt gone.

[GITRDUN]

I have the meter back on the lines and set to continuously record and overwrite when memory is full. If we get another weird occurance i can pull the meter and see if it shows anything.

I am also going to have my maintenance man run a ground wire from each machine up to the ground bar on our 480V bus bar. Thats gotta be the best ground connection we have in the plant. Might as well keep trying to eliminate possible problems.

We will see what happens from here.

[GITRDUN]

Took another hit this morning. Got 2 of the same machines that took a hit last week, one simply tripped its main breaker in the cabinet and the other blew the spindle drive. Checked the recording meters and showed nothing unusual at all. Both machines had the new ground wire hooked up. Same situation as always, both happened when they were powered up in the morning.

I will be connecting the meters to the machines that seem to be taking the hits right now and see if that shows anything. Would it be better to connect the meter on the load side of the machines main breaker or down the line somewhere say after the machines transformer?

[Al_The_Man]

Anywhere in the chain should catch it because it will ripple through.
Start at the transformer secondary, maybe?
Al.

[GITRDUN]

I would expect that when regulators, capacitors and things pop on a board it would send a fast surge through the machines electrics. Wouldnt it be difficult to tell the difference between that and what caused it?

[Al_The_Man]

But if it is a high voltage spike etc that would cause it then this should occur just prior to the damage.
When something faults it usually causes high current as a result of the high voltage being imposed on it.
High current caused by too high voltage.
Al.

[htrantx]

I would expect that when regulators, capacitors and things pop on a board it would send a fast surge through the machines electrics. Wouldnt it be difficult to tell the difference between that and what caused it?

The following paper described the failure mechanism of your problems, and suggested some solutions.

http://www.pq-ride.co.kr/dataroom/fi...nces%20IAS.pdf

Good luck;

htrantx

[htrantx]

another paper:

IEEE Xplore - Understanding the dominant field failure mechanism for DC power supplies

Abstract:
... This paper shows that current-inrush surges, and not voltage surges, are the leading cause of equipment failure. These current-inrush surges occur at the end of frequent voltage sag events, when the inrush current limiting circuits are bypassed and offer minimal protection. The paper presents definitive data on the statistics of power disturbances, detailed analysis of the failure mechanism for various types of input stages, and a methodology for predicting the susceptibility of equipment....

Causes of Vsag:
http://www.omniverter.com/support/wh...oltage_Sag.pdf

htrantx

[wizard]

I would expect that when regulators, capacitors and things pop on a board it would send a fast surge through the machines electrics. Wouldnt it be difficult to tell the difference between that and what caused it?

I have a big problem with the failures you describe and the stated fact that the line voltage recorder isn't seeing anything odd. Odd can be both high and low voltages. If the recorder isn't catching any events worth noting then one of two things are a reality. One the recorder is junk. Or two nothing is happening at the incoming lines that it can record. If nothing is happening you need to look internally to see what is happening in your plant at the time the machines are turned on. If the problem isn't external to the plant you need to know what is happening at the time of upstart in the entire plant.

By the way, the machine that trips the breaker may have totally unrelated issues including a weak breaker or something operating out of spec. You should note however where that breaker supplies power to.

[GITRDUN]

The recorders are picking up spikes and sags but at the time of the machines being powered up there was nothing unusual showing up. I have saved all of the recorded data i have and made a daily text log to point out any spikes or sags that were recorded but i have yet to see anything happening at the time of switch on.

That machine could easily have unrelated issues but the fact that it had an issue pop up at the same time as another machine twice in 7 days is a little suspicous IMO.

[Al_The_Man]

The recorders are picking up spikes and sags .

What are the numbers, anything you can post easily?
Al.

[GITRDUN]

Here are a couple events i have saved. The first one is a voltage sag that was recorded about 3:38PM yesterday just a few minutes after our shop was shut down and everybody was out the door heading home. The second is a 1500V spike that i caught a few weeks ago, it hit in the evening long after we were gone for the day. The spike was recorded when we had the 240V high leg delta power in the main building still.

[RCaffin]

The omnivertor and pq-ride papers posted by htranx are pretty scary. What's worse, they are very real. The grid is not a pretty place.

What is worse is that there does not seem to be any real effort to clean up the problem for industry. To quote: 'An EPRI study in 2005 (Ref 2) suggests that the cost to North American industry of production stoppages caused by voltage sags now exceeds U$250 billion per annum.'

The omnivertor paper suggested ferro-resonant transformers (FRT) and voltage sag corrector (VSC) gear. (They sell the latter.) It mentioned UPS systems as well, but I had one of those and it blew after a high voltage spike. The mess! Not suited to large plants.

I have not used voltage sag corrector gear, but I have used a number of FRTs and am still using one very happily. What neither paper mentions is that both systems tend to be a bit brutal to the grid. If you are going to use one for a significant amount of power, you need to cover the whole plant with it. Otherwise the unprotected gear in the plant can see even worse conditions (sometimes, it depends).

I think this is a case where the factory owner (hi GITRDUN) will have to 'make his own arrangements'. I suggest discussion with FRT and VSC vendors - several of them.

Cheers

[htrantx]

Here are a couple events i have saved. The first one is a voltage sag that was recorded about 3:38PM yesterday just a few minutes after our shop was shut down and everybody was out the door heading home. The second is a 1500V spike that i caught a few weeks ago, it hit in the evening long after we were gone for the day. The spike was recorded when we had the 240V high leg delta power in the main building still...

The recorders are picking up spikes and sags but at the time of the machines being powered up there was nothing unusual showing up...

The killer VSAGs are only few cycles at a time (0.01666 sec/cycle) as described in the papers. You need to set the sampling rate much faster than the nyquist rate to see them in the time domain. Use a scope or seek professional helps.

htrantx

[RCaffin]

The killer VSAGs are only few cycles at a time (0.01666 sec/cycle) as described in the papers. You need to set the sampling rate much faster than the nyquist rate to see them in the time domain. Use a scope or seek professional helps.

Shouldn't a proper mains data logger handle this, down to half a cycle?
Yes, glitch detection is a separate circuit.

Cheers

[GITRDUN]

The killer VSAGs are only few cycles at a time (0.01666 sec/cycle) as described in the papers. You need to set the sampling rate much faster than the nyquist rate to see them in the time domain. Use a scope or seek professional helps.

htrantx

My data logger does show the recorded transient waves seperately and as far as i can tell it does show it in cycles. I have an electric consultant coming in Wednesday to do a walk through and look at our problem. I will show him all the recorded data i have and see if he can make any sense of the whole mess.

[htrantx]

Most data loggers has a RMS input stage, and since the tail of the RMS filter is much longer than a cycle, you cannot see the true sagging magnitude of a glitch in milliseconds.

The problem is deep inside your machine at the board and component level, mainly the DC power supply and the so called “negative thermal coefficient (NTC) resistor”, which its sole purpose is for taming the inrush current.

If you take a look at the author list in the Brendre et al. paper, you’ll see it took 3 PhDs just for saying: “Hey, Vsag is the problem!” The paper published later than 2002, so equipments built prior might need to be updated.

As for why the hell your board did not blown up during operation, but only when started-up; one possible explanation is because your board was designed to do that! Since there was no Vsag specification, the calculations for Irush, Idc and temperature recovery requirements are completely FUBAR. Moreover; because there was no spec, no one was testing or even looking for the problem. You might want to contact the machine manufacturer and check for updated of the designs.

One thing that I would do is putting a fan directly on the supply board and turned it on before power up, just to see it would help. Other thing is checking for the daisy chain of power lines to the machines, and rewiring them to star wiring.

htrantx