[nmorong]

Hi,

I have a problem that's stumping me on my Kogna/Milltronics machine, I wanted to see if anyone thinks it could be a servo tuning issue. Until today, I had a very intermittent banging sound that only happened using the MPG to jog the X axis (maybe 3-4 times in 2 months). It had never happened when I was really paying attention and I had almost convinced myself it was one of the spring loaded way covers getting stuck and banging back into place. I've used the machine quite a bit with no problems, but only for a couple hours at a time until today (it was also the first day that my shop was kinda hot).

So today I run the machine for about 8 hours, and at the end of the day I hear the banging sound a couple times. I went back to the machine and my program was still running, but while I was watching the X axis visibly jerked, making the banging sound. Maybe 30 seconds later the X axis went into a violent oscillation, and I Estopped the machine. After the machine sat for about half an hour, it took me about 10 minutes of jogging the axis back and forth to reproduce the problem (the jerk/bang, it didn't oscillate again).


My thoughts are: I did the servo tuning with brand new linear rails and trucks, and fresh seals on the ball screw. The system may have had significantly more resistance to motion when I did the servo tuning that it has now when it's fully warmed up, and the X axis has the least mass to move of the 3 axes.

Is it possible or likely that too much gain could cause these symptoms? I revisited the step/response screen, but everything looked the same as when I initially did the servo tuning.

[RCaffin]

X axis servo controller may be dying in the heat.
The Y axis is fine?

Cheers
Roger

[TomKerekes]

Additional friction can make the Servo more stable and loss of friction can make the Servo less stable. You might check the tuning margins by temporarily increasing P and I gains by something like 20% to see if there is some margin of stability to allow for variations in the system.

You might also check the level of Output you are using vs the max Output limit to see if there is margin.

Normally to protect the system as well as to let you know if the axis is ever not following properly, the max following error is set to a small number. In this case if the system can't follow properly or goes unstable it should immediately fault and disable rather than oscillating violently or possibly catching up to the destination with a bang.

[nmorong]

I'm not sure it's the only problem, but I definitely need to revisit the servo tuning. What seemed nice and stable in a ~55 degree F shop in the winter, with new grease and brand new seals providing drag, seems borderline unstable once warmed up now that it's starting to wear in.

I finished my project this morning without issue when the shop was still cool. By late afternoon when things got hot (hotter than yesterday), both the x and y axes were vibrating/oscillating audibly during motion. Z was fine, but was tuned with much less P and I gain. I'll try your suggestion of increasing the gain to see if there is any margin the next time I get the machine hot, but I bet it's just going to go unstable right away based on what I observed today.

I set the max following error to the same value the old controller had, (0.100" or ~1000 counts if I remember correctly?), would you recommend I set it considerably smaller than that?

[TomKerekes]

Yes I think max following error could be set much smaller.

[nmorong]

OK, I'll try setting the following error to maybe .025 or .030. It is currently at .250, I had misremembered.

But first, the problem is not just servo tuning.

I increased the P and I gain for the X and Y axes, and it was not stable, so I backed off both P and I gain from where I had it by 20%. It sounds normal when running now, even when warmed up.

The X axis did its bang noise again today, which was slightly less violent now. But I didn't estop the machine this time, I rechecked the part zero and it had lost position by .100", or almost exactly 1/2 motor/ballscrew revolution. I'm going to wait to set the max following error lower to see if it's losing position by the same amount every time. It ended up going into oscillation again and I had to estop it.

So all this had me thinking I had a problem with the X axis... And then, while the machine was sitting idle, the Y axis did the same losing position/banging thing. So I have some unknown glitch that only show up after several hours of running, and is not specific to an axis.

[TomKerekes]

If the system is loosing position that somewhat narrows things down. Regardless of tuning or instability or pretty much anything the encoders should properly keep track of the position.

If loosing position then either the encoders are not always working correctly or there is something mechanical going on such as the encoder shaft slipping or lead screw shifting, assuming something isn't resetting the encoder position.

To determine if the encoder is working properly you might physically mark the encoder shaft angle at the initial part 0. Then after a shift check if the encoder shaft is at the same place physically when moving back to zero.

You should check the position in KMotion Axis Screen (or Machine Coordinates) to eliminate any issues with GCode offsets changing.

HTH

[nmorong]

I doubt this is a mechanical problem, as it is occurring on both X and Y axes, albeit only once on Y so far. The servos contain resolvers that output position to the servo drives, and I am using the simulated encoder signals output by the servo drives. If the resolver position physically moved motor commutation would be incorrect, and I suspect that the drive would fault quickly. I suspect that either the simulated encoder signal is to blame, or because it seems to lose position by 1/2 motor revolution possibly something causing the servo drives commutation of the motor to glitch and end up 180 degrees off. It's a 6 pole 3 phase brushless AC servomotor, not sure if this is possible.

Maybe if the resolver and encoder signals aren't too sensitive to probe I can figure out how to log them with an oscilloscope. I have one, I just don't know how to use it very well.

I ordered 3 10000 lines/rev encoders, I had planned to eventually use a higher resolution feedback system than the resolver-simulated encoder setup from the beginning, I just wanted to get things working first.

Can you think of anything that may be resetting the encoder position? And yes, machine coordinates had shifted, not just a gcode offset.

[TomKerekes]

They might be using a 6 pole resolver that would match the servo motor so the resolver would output 3 cycles for each revolution. The encoder simulator would be absolute over 1/3rd revolution but then would need to add in 1/3 rev when moving to the next cycle. Or something like that If the encoder simulator/interpolator somehow got confused from a glitch or something it might get off by some multiple of 1/3 rev.

How many simulated counts per rev are there?

What is the overall axis resolution in counts/inch?

Regarding resetting the encoder position: I was thinking you might be monitoring an external button to resets the axis position, that calls Zero(x); In such a case noise on the button signal could trigger an unexpected reset.

[nmorong]

The resolvers are single cycle/rev, or at least the one on the Y axis motor is. I stupidly took it apart when I first got the machine and had to learn how to use a DC power supply to lock the rotor and an oscilloscope and function generator to get it repositioned correctly so I'm pretty confident about that .

There are 512 simulated counts per rev (pre-quadrature), this is selectable from 128 to 1024 with DIP switches on a "personality board" that plugs into the servo drive. I left it set at 512 thinking that at 1024 it may dither back and forth too much. Maybe Milltronics had some weird proprietary "personality board"? The overall axis resolution is 10240 count/inch. Now that I think about it, I should pull out and clean the contacts of the personality boards. If the servo drive began outputting 256 or 1024 counts/rev, I think it would cause the symptoms I'm seeing. I know the usual suspects for losing position would be grounding issues/noise, but I don't lose any counts for hours and then it's suddenly off by ~1000 counts makes me think it's something else.

In any case, I'm going to install real encoders and stop using the simulated encoder signals and hope that the problem goes away.

The only C programs that reset axis positions are my homing program, and the tool change program resets the spindle axis position at each tool change.

[TomKerekes]

The resolvers are single cycle/rev, or at least the one on the Y axis motor is. I stupidly took it apart when I first got the machine and had to learn how to use a DC power supply to lock the rotor and an oscilloscope and function generator to get it repositioned correctly so I'm pretty confident about that.

Interesting I wonder how they do that

The overall axis resolution is 10240 count/inch.

So the 0.1 inch shift would be 1024 counts. Which it seems would be 1/2 of the 512 x 4 = 2048 counts.

In any case, I'm going to install real encoders and stop using the simulated encoder signals and hope that the problem goes away.

Sounds good. But it would have been nice to understand what was happening.

good luck!

[nmorong]

So I fabricated mounts for encoders on all 3 axes, installed and wired the encoders, modified the fiberglass covers to fit the encoders, etc.


The problem didn't go away, but presents differently. I set the max following error to 1000 counts (now .02 inches), and the resulting failure is much less violent.

However, it now happens much more frequently, and a couple of minutes is all it takes to reproduce it. I was able to see it happen on the step response screen several times. The X, Y, and Z axes have all done it now. It looks like for some reason, the Kogna is suddenly seeing a huge, instantaneous spike in position error, but it usually returns to the correct position and continues its movement without a corresponding actual physical motion. I suspect this has been happening all along, I just never noticed it unless it didn't return to the correct position until I set the max following error low enough that it disables the axis.

I've attached a couple pictures showing the instantaneous glitching.

As for what I've done to try to diagnose/fix it: The problem has occurred with 6 different feedback devices of 2 different kinds on 3 different axes... I think I can rule out the mechanical stuff and bad electrical connections between the feedback devices and the Kogna.

I dug into the wiring with the multimeter and I actually did find a ground loop, but eliminating it didn't help. I verified that the Kogna ground was connected to earth ground at only one point, and the problem still happened. Next, I tried leaving the Kogna ground floating, with no connection to earth ground. No change. I've tried running the X axis with absolutely everything else off in the hope that maybe some other component is making electrical noise causing the problem, no change.

I'm starting to suspect either a damaged or defective Kogna at this point, or something in the software causing it to momentarily lose axis position. Any help would be greatly appreciated.

[nmorong]

Also, I tried leaving the X axis holding a position for a long time (20 minutes or so) with a .0001 indicator on it. Then I moved the Y and Z axes around a bunch. The X axis did not lose or gain any counts on the axis screen, or physically lose position.

[TomKerekes]

That looks like a bug on our end. Thanks for the detailed information. Please give us some time to try to duplicate it here.

Could you post your Initialization C Program so we can try your exact configuration?

You only have a Kogna board correct?

What Version are you running?

[nmorong]

I have a Kogna and a Konnect. The Konnect is used for 24V logic only.

I'm running Kmotion 32 bit 5.3.2

I've attached my init file.

Please let me know if there's anything else I can provide to help.

[TomKerekes]

I think we were able to duplicate and fix this. It was a clock domain issue. The encoder counters run from a 66.66MHz clock (4X) and the logic reading the counters from a 16.66MHz clock. A PLL is supposed to make the clocks synchronous but it seems they were not synchronized well enough. So if the counter was changing while the counter was being read there could be an invalid reading. We changed the encoders to run from the negative clock edge of the 66.66MHz clock. This provides at least 7.5ns from when the counters change to when they are read.

We were able to duplicate the problem by generating a Step/Dir Quadrature signal on Axis 0 Differential Output 6 and wire to Differential Encoder input 14. Then command Axis 0 to Jog at 200001 counts/sec. The +1 causes the count to shift gradually relative to the sampling. A User program then looks for the glitch of a change > 100. A User program should never see any change > 30 at that count rate. This typically detected the glitch within a few seconds. After the fix we ran overnight without a single error.

#include "KMotionDef.h"

void main()
{
    double T0=Time_sec();
    double cur, last=ch1->Position;
    for (;;)
    {
        cur = ch1->Position;
        if (cur > last + 100 || cur < last - 100)
        {
            printf("cnt %f new %f last %f diff %f\n", Time_sec()-T0, cur, last, cur-last);
            break;
        }
        last=cur;
    }
}

Here is a Firmware Path for Version 5.3.3. Please copy the 3 files below to the DSP_KOGNA folder then Flash New Version and reboot Kogna. The Firmware Version should report as 5.3.3x.

DSPKOGNA.bin
DSPKOGNA.out
DSPKOGNA.sym

Please let us know if it resolves the issue.

[nmorong]

Thank you Tom!

I will try it and report back later this afternoon or evening when I can get back to the machine.

[nmorong]

I flashed the Kogna, rebooted it, and verified that it was reporting version 5.3.3x.

I added the glitch detection code to my init file, and used the step response screen to command moves at 200001 counts/sec. I am still seeing what looks like a smaller version of the glitch, but it occurs ~30% of the time now.

Note I changed the code to look for a change of >50 counts from >100 counts. I did this because I copied and pasted the code, missing that it contained a break and my for loop was no longer running. I was confused about why it wasn't printing to console when I could see spikes on the step response error plot. I made it more sensitive before I realized my stupidity.

I've attached screenshots of what I'm seeing. I don't know if this causing a more than momentary loss of actual position, I didn't get out the indicator to see if it was returning to the same position at the same count. The spikes I'm seeing now are much smaller, maybe ~45 or 50 counts ish. Before flashing the Kogna it looked like the spikes in position error might have been occurring in multiples of 256.

[TomKerekes]

It seems 7.5ns is still too short for the complex logic involving all the combined signals involved with the data bus.

Here is a new Version that first synchronizes to the 16.66MHz clock so provides a full 60ns of propagation time through the data bus logic.

Please download the files again. This Firmware Version should report as 5.3.3xx

Thanks for your patience.

[nmorong]

No worries, I appreciate the support.

The DSPKOGNA.bin link is giving me a 404 error though.