[nmorong]

I have 2 (probably?) unrelated problems that I'm hoping to eliminate my Kogna/Konnect and Kmotion as possible causes. Both problems are intermittent enough that I can't reproduce them when I'm trying to.

First, my spindle (servo spindle controlled by +/-10v DAC) is "glitching" similar to how my X,Y,Z axes were jumping in position prior to updating to Kogna firmware 5.3.3xx. I am now running Kmotion and Kogna firmware versions 5.3.8. I suspect a failing resolver in the motor but I thought I would ask here to confirm that the software fix applies to all axes, not just those part of the coordinated motion system.

Second, I have an output on the Konnect sticking in the ON position. The output is driving a solid state relay which runs my coolant pump. I can confirm on the Kmotion I/O screen that the appropriate bit is off, but the LED on the Konnect remains on, and the pump is still running. I can't turn the Konnect bit back ON in this condition.

Does the LED for a Konnect output simply monitor if current is flowing through the output? Or should the state of the LED depend on the state of the bit in Kmotion?

Thanks,
-Nate

[joeavaerage]

Hi,
cannot answer any questions about the Konga/Konnect system, but to test the spindle, just feed it with the output of a litte 9V battery....it should run at 90% of full speed, and should not glitch.
If the glitch you are talking about goes away then its related to the Knoga/Konnect, but if not then its in the servo and/or drive.

Next question is the output have a pull down resistor?. Surely would not hurt to put a 10k resistor between the output and ground, just to ensure that the output does in fact go low when the
output is inactive.

Craig

[nmorong]

If I feed the drive with a battery, and it DOES glitch, that would rule out the Kogna. But if it doesn't glitch I could easily stand there watching it for 4 hours and I won't know if it went away or not because it's a very intermittent problem.

The Konnect output in question behaves like a switch between 2 terminals, a (+) terminal connected to +24V, and a (-) terminal connected to the relay "coil". The other side of the relay coil is grounded. There is a flyback diode, but no resistor on the Konnect side of the relay coil. If I understand correctly, you are suggesting I put a 10k resistor on the Konnect (-) terminal. That makes sense, as I don't know the temperature/resistance behavior of the solid state relay "coil".

[joeavaerage]

Hi,
I thought you said that you had a solid state relay?

You are correct if its a regular relay then you should not need a pull-down resistor, but if its a solid state relay....who knows?......it may require some resistance to earth to pull it low.

Craig

[nmorong]

It is a solid state relay, I just keep calling its input a coil because I don't know what else to call it.

[nmorong]

I replaced the relay and added a resistor and the output problem hasn't reoccurred.

The spindle, however, seems like it might have the encoder glitch that had plagued my motion axes with earlier Kmotion versions. I still wonder if I managed to damage my Kogna somehow, because it seems like it happens more often when the shop gets hot.

I added a modified version of the glitch detection code from this thread to my for loop:

https://www.cnczone.com/forums/dynom...vo-tuning.html

--------------------------------------------------------------------------------

// Below is in main function in init program
    double TGlitch=Time_sec();						//Glitch detect time variable
    double curGlitchX, lastGlitchX=ch0->Position;	//Glitch detect X axis variables
    double curGlitchY, lastGlitchY=ch1->Position;   //Glitch detect Y axis variables
    double curGlitchZ, lastGlitchZ=ch2->Position;   //Glitch detect Z axis variables
    double curGlitchS, lastGlitchS=ch3->Position;   //Glitch detect spindle axis variables


// Below is in for loop in init program
        curGlitchX = ch0->Position;
        if (curGlitchX > lastGlitchX + 200 || curGlitchX < lastGlitchX - 200)
        {
			printf("X axis glitch detected\n");
            printf("cnt %f new %f last %f diff %f\n", Time_sec()-TGlitch, curGlitchX, lastGlitchX, curGlitchX-lastGlitchX);
        }
        lastGlitchX=curGlitchX;
//////////// Y axis
        curGlitchY = ch1->Position;
        if (curGlitchY > lastGlitchY + 200 || curGlitchY < lastGlitchY - 200)
        {
			printf("Y axis glitch detected\n");
            printf("cnt %f new %f last %f diff %f\n", Time_sec()-TGlitch, curGlitchY, lastGlitchY, curGlitchY-lastGlitchY);
        }
        lastGlitchY=curGlitchY;
//////////// Z axis
        curGlitchZ = ch2->Position;
        if (curGlitchZ > lastGlitchZ + 200 || curGlitchZ < lastGlitchZ - 200)
        {
			printf("Z axis glitch detected\n");
            printf("cnt %f new %f last %f diff %f\n", Time_sec()-TGlitch, curGlitchZ, lastGlitchZ, curGlitchZ-lastGlitchZ);
        }
        lastGlitchZ=curGlitchZ;
//////////// Spindle axis
        curGlitchS = ch3->Position;
        if (curGlitchS > lastGlitchS + 300 || curGlitchS < lastGlitchS - 300)
        {
			printf("Spindle axis glitch detected\n");
            printf("cnt %f new %f last %f diff %f\n", Time_sec()-TGlitch, curGlitchS, lastGlitchS, curGlitchS-lastGlitchS);
        }
        lastGlitchS=curGlitchS;

If I look for changes of less than 300 counts it triggers more often (at 100 counts it's continually triggered), but at 300 counts I actually hear the spindle stutter. The spindle axis encoder is a simulated encoder from the servo drive, and I can set the PPR by parameter. For now I'm going to set it to the maximum PPR the drive can output (I think I can increase it from 1024 PPR to 16384PPR) in the hopes of the physical change in velocity that results from whatever is going on will be much smaller.

Stuff showing in the console above the spindle jogging and glitch stuff is from my attempt at creating a synchronous tapping program.

[TomKerekes]

Hi nmorong,

Sorry somehow I wasn't following this Thread.

You might post all your code because it isn't clear how many WaitNextTimeSlice() there are in the loop. If 2 Threads are running (System + Spindle) then each time slice is 270us. With 3 Time slices that would be 810us per loop. Also maybe there is variation in the loop timing?

So Jogging at 204000 counts/sec the expected delta would be 165 counts.

You might try this code to display the last 3 deltas and the last 3 times between checks. Maybe that would give some clues on the nature of the error.

        double TGlitch=Time_sec();                        //Glitch detect time variable
        double lastT=Time_sec();                        //Glitch detect time variable
        double curGlitchS, lastGlitchS=ch3->Position;   //Glitch detect spindle axis variables
        double deltaS, last_deltaS=0, last_last_deltaS=0;
        double deltaT, last_deltaT=0, last_last_deltaT=0;
        
        
        TGlitch=Time_sec();
        deltaT=TGlitch-lastT;
        curGlitchS = ch3->Position;
        deltaS = curGlitchS - lastGlitchS;
        if (deltaS > 300 || deltaS < -300)
        {
            printf("Spindle axis glitch detected\n");
            printf("cnt %f new %f last %f diff %f\n", Time_sec()-TGlitch, curGlitchS, lastGlitchS, deltaS, last_deltaS, last_last_deltaS);
            printf("deltas %f %f %f \n", deltaT, last_deltaT, last_last_deltaT);
        }
        lastGlitchS=curGlitchS;
        last_last_deltaS = last_deltaS;
        last_deltaS = deltaS;
        last_last_deltaT = last_deltaT;
        last_deltaT = deltaT;
        lastT = TGlitch;

How long did you have to wait before detecting the error? 5773 seconds? 1.5 hrs?

[nmorong]

Hi Tom,

I always appreciate you stopping in and helping me with my latest problem.

It looks there are 3 time slices in the for loop, outside of if statements that will not usually be true. I can confirm that when I get back to the machine and look at the latest version again. I will try the code to see the times between checks, is there a way to read the DAC output and log that along with the times as well?

I've done some more experimenting today and it looks like changing the axis resolution does nothing noticeable to reduce the stutter. It's now at 8192PPR (pre-quadrature). Maybe I can rule the encoder glitch out entirely by setting the deadband larger than the possible instantaneous change in position? After all, it's the spindle and I can live with a large positional error. Or maybe I can temporarily disable the axis, and replace my normal m3 program with simply commanding the DAC to the appropriate voltage while running a test program (not cutting anything).

I did perhaps find one clue, I was watching the servo drive status in the Parker software and I could see the DC bus voltage increase from ~360V to 600V in the fraction of a second that the spindle stutters. So the drive is trying hard to stop the motor, and I don't know if this is because of the command signal or something else.

It took about 1.5 hours for the stutter to occur for the first time today. I would guess an average frequency might be once every 2-3 hours.

-Nate

[TomKerekes]

Hi Nate,

is there a way to read the DAC output and log that along with the times as well?

DAC_Buffer[ch] will have the current DAC setting as a value from 0 - 4095 where 2048 is 0V

Maybe I can rule the encoder glitch out entirely by setting the deadband larger than the possible instantaneous change in position?

I don't think that would work. Deadband is sort of like backlash. If the error is at the edge of the Deadband and the error glitches it will still be seen. You might reduce the gains so that the response is very slow.

If you post your code we can look it over.

[nmorong]

Hi Tom,

I've attached my init file. I changed the glitch detection for the spindle axis by a factor of 8, because I increased the axis resolution by a factor of 8. I'm using the standard C programs for spindle control using jogs. The acceleration in the spindle axis configuration is relatively low to avoid the servo drive DC bus voltage from getting too high when stopping the motor. The velocity is set lower than I'm actually using, apparently controlling the spindle with jogs doesn't respect the velocity limit? I didn't notice that this was the case until I was going through and changing everything related to the axis resolution. I can confirm that it has been spinning at the correct speed all along.

I'll try to incorporate the code you've shared to log time, deltaT, change in position, and DAC output in the hopes of capturing what is happening. Unfortunately, I don't know if I'll be able to work on it today. Also, I ordered stuff to make up a different Kogna-Drive cable, my spindle drive supports step-direction input and I realized I should be able to set it up to run as an open loop step-direction axis from the Kogna, with the servo loop closed at the drive instead. If you think this might be a superior way to control the drive I can try this first. I have a separate encoder on the spindle so I can still monitor what the spindle axis is doing.

-Nate

[nmorong]

Just kinda trying to organize my thoughts here.

If I assume all feedback mechanisms are working correctly: A change in axis position between samples greater than double the expected change for a given velocity implies a doubling in velocity. I can think of endless reasons for the motor to slow down, but if it speeds up it either the drive must be receiving a higher voltage to the analog input, or the drive is failing somehow.

The sampling time for the servo drive analog input is 125 microseconds. If the motor exerts its peak torque, I have calculated that it should be able to change velocity by ~.05rev/s in 125 microseconds. The problem occurs frequently at a motor speed of 3000RPM, or 50 rev/s. If the drive is increasing the motor speed due to an increase in the analog input, it would take ~.125 s to double. This is not realistic, even if the motor could maintain peak torque at that speed, the drive would fault (It should fault at a motor speed of over 3800RPM). A transient voltage spike from something to the analog input might cause a noticeable stutter, but basically no possible input to the drive could cause the motor to actually double in speed from 50 rev/s, even briefly. So I think IF I can detect a change in position of double the expected change in position while running the motor at 50 rev/s, I can rule out an actual, physical change of position that large.

That leaves the feedback. Because I have an encoder on the spindle, which is mechanically connected to the motor at a fixed ratio, I can also test for a change in position between the 2 encoders that does not conform to the known ratio. If the spindle drive axis glitches, and the position of the 2 encoders stays shifted relative to each other, I am looking at some feedback problem likely unrelated to the Kogna encoder counters.

So I guess what I need to do is A: Test to see if the spindle stutter is actually resulting in a position change of double the expected change. B: Test the 2 encoders relative to each other to see if one loses position relative to the other.

-Nate

[TomKerekes]

Hi Nate,

apparently controlling the spindle with jogs doesn't respect the velocity limit?

That is correct. Max Velocity applies to move trajectories, If commanded to Jog at some speed it will do so.

I agree systems with mass can't change velocity significantly in microseconds. It's most certainly a measurement issue of either the Spindle Drive or Kogna. I've seen some simulated encoders have quirks where they sometimes output a burst of counts to catch up. But I sort of think this is not likely for a Spindle Drive with a real encoder.

I'm thinking Kogna may still have an issue. Kogna's FPGA uses 10-bit counters for the encoders which are read every 90us. The deltas between readings are then accumulated by software as a 64-bit double. The readings from hardware are a 0 to 1023 value that wraps around. The deltas between readings can be up to +/-511 counts. A corrupt reading will usually introduce a glitch of up to 511 counts which will normally be corrected on the next reading without any overall loss of counts. In rare cases it may occur right on a counter wrap and end up off by an entire loss/gain of 1024 counts.

Please try to capture the glitch with the display of the previous 2 deltas.

[nmorong]

Hopefully I'll be able to do that tomorrow morning.

[TomKerekes]

No big rush from our end. But I would like to get to cause of this without doing a workaround.

[nmorong]

I've incorporated the code to test for the glitch and record the deltas. I tested it by zeroing the axis from the console while moving slowly, and it appears to work correctly.

Now I've been running for several hours, with no sign of the stutter or the glitch.

[nmorong]

I've got in about 6 hours of run time on the machine today, and nothing. No sign of the problem whatsoever. So I'm giving up for the day, and I will just keep the glitch detection code running and post back here the next time it does it. Intermittent problems are the worst.

[TomKerekes]

Hi Nate,

Yes. Its not cooperating.

[nmorong]

Well things have taken a new turn. For a few days it seemed like the problem had fixed itself, now my Kogna seems to be fried.

When physically testing with a multimeter my DACs all output between ~3-6 volts, regardless of the commanded DAC value. The analog status screen shows the commanded DAC voltage, but the physically measured voltage doesn't change. This behavior persists even with everything disconnected from the Kogna other than power and ethernet. Bit 279 is enabled to power the DACs. I removed the Kogna from the machine, powered it from USB, and tested it again on the bench with no change.

I inspected the Kogna for physical damage, and it looks like a chip on the board might show some heat damage. A little bit of searching turned up TL074CDR as the DAC output amplifier used on the Kanalog boards, and the suspect chip on the Kogna appears to also be a TL074 series chip, so I am guessing I've identified at least one of my problems...

I would like to figure out why this happened, I want to get a replacement Kogna but I don't want to fry $600 again. Maybe I can replace that chip and fix this one as a spare.

-Nate

[TomKerekes]

Hi Nate,

Sorry to hear that. Those amplifier chips are normally very robust and have continuous output short-circuit protection. Could an output have been shorted to a supply? JP8 does have +/-15V pins right next to the Outputs.

Note that USB power is normally insufficient to power the +/-15V circuitry.

Here is the amplifier circuit if you would like to test if the DACs still function. The DACs basically output a 0-2V signal before the amplifiers.

Attachment 514890

[nmorong]

I've ordered a replacement Kogna. Hopefully I don't fry this one. I appreciate the information on testing the DACs, I will test these and maybe I can fix this Kogna. I don't like not having a spare on hand anyway.

I looked for and couldn't find any continuity between the output pins and any other pins on the same connector. I can't rule out FOD, maybe a loose strand of wire fell where it shorted something. Or something internal to one of the servo drives.

I was just trying to power the board on the bench by USB to rule out something wrong with my 5V DC power supply. The actual DAC output may not have been as high, I just checked to confirm that it wasn't zero when it should have been and still didn't change with commanded DAC voltage.