[bobeson]

I suspect that somehow the latest Sprutcam post is generating code that triggers a Mach bug, because I now have a reproduceable test case that causes an axis offset error when a g0 move fails. I am able to induce the error reliably by running the code fragment attached below.

Here is the interesting part: the failure no longer appears if I change *any* of the code *following* the line where the error appears. In fact, if I so much as delete the comment "(1)" near the end of the file, then the error does not appear. Put the comment back, and the error appears. I was able to modify some of the parameter values a small amount without causing the error to go away, but not more than some undetermined threshold, something like 0.05" or so.

The failure occurs on the third line, "G0 Z-0.62". What happens during the failure mode is that the axis stops moving at approx. z=0, making a distinctly audible humming noise while stalled, then the DRO updates as though it completed the move to -0.625, then the lateral movements start. You can kill the execution as the lateral cut moves start on line 5. If the error occured, then the Z axis offset will be approximately 0.62" too high. If I change *anything* else about the program, including deleting later moves or even the comment, then the error does not occur.

Here is the failing code:

M3
G0 X-4.8179 Y-3.1978 Z1
G0 Z-0.62
G1 Z-0.625 F2.8
G3 X-4.8077 Y-3.1746 Z-0.625 I-0.02286 J0.02387
G1 X-4.8074 Y-3.1573
G2 X-4.8978 Y-3.0791 Z-0.625 I0.34972 J0.4959
G2 X-4.9531 Y-3.0124 Z-0.625 I0.44891 J0.42809
G1 X-5.1326 Y-2.7558
G2 X-5.1746 Y-2.6918 Z-0.625 I0.95593 J0.67395
G1 Y-2.8998
G3 X-4.8998 Y-3.1746 Z-0.625 I0.30019 J0.02541
G1 X-4.4827
G1 X-4.4824 Y-3.1642
G1 X-4.4778 Y-2.9463
G2 X-4.6868 Y-2.826 Z-0.625 I0.0327 J0.29853
G1 X-4.863 Y-2.5742
G2 X-5.0601 Y-1.5185 Z-0.625 I1.06505 J0.74512
G2 X-4.9157 Y-1.1653 Z-0.625 I1.26403 J-0.31068
G1 X-5.1746 Y-1.1383
G1 X-5.1798 Y-1.188
G0 Z0.2
(1)
m5
m30


For reference, here is the header from the actual sprutcam output, for initialization parameters that my machine is left in when I run the code above. This snippet ends where the failure occurs. I reformatted the code above to find the minimal set of code that still induces the error.

N10 (Postprocessor: )
N20 G90 G54 G64 G50 G17 G40 G80 G49
N30 G20 (Inch)
(Roughing waterline - Non-TTS end mill holder without spring washers)
N40 G54
N50 M998
N60 T134 G43 H134 M6
(1/2" 2-fl End Mill - non-TTS)
N70 S993 M3
N80 G0 G94
N90 X-4.8179 Y-3.1978 Z0.2
N100 G0 M8
N110 Z-0.6

Although the reproduceable bug I posted here is occuring on a Z axis move, I also experienced an unexpected X axis offset during execution of the remainder of the program this code was taken from, after I worked around this first repeatable occurance. So, I do not believe this bug is limited to Z axis control. I suspect that a number of recent threads have been started by people experiencing manifestations of this bug.

Note that I am using the latest released version of Sprutcam with the latest (recent) Tormach post, and the current release of the Tormach control software.

I would appreciate it if folks with access to a PCNC1100 would try the code snippet I posted and let me know what happens.

Set Z=0 at 1" above a surface, and if the tool ends up performing the lateral cut moves at 0.375 above the surface, the error did not occur. If the lateral moves start when the tool is still approx. 1" above the surface, then the error has occured. You should be able to hear a noise during the G0 Z-0.62 move when the error is occuring.

[rowbare]

From what you describe, your Z axis is stalling/losing steps at that point. Just for fun, replace the G0 with a G1 F20 and see what happens.

Have you modified any Mach motor parameters like speed or acceleration? What model is your machine? Series 1 (with or without VFD mod) or Series 11? I remember reading that some people with an upgraded Series 1 have to detune the motor parameters a bit since the latest control software is set for 90 IPM (not sure if the acceleration is different though) and not all Series 1 machines can handle that.

bob

[bobeson]

I have a series II, and I have not modified the stock parameters in any way. And like I mentioned, this appears to happen with both my Z axis *and* my X axis, although I have not so far isolated the specific code which induces the X-axis failure. I cannot make any further modifications to this code, as far as I can tell, without making the error go away. In other words, it's easy enough to work around, just massage the code, but the bug is bound to be encountered again, which in fact I have done already in the same program. I can't keep inserting workarounds like this until they're all gone, it's just not practical.

rowbare, are you able to run this code on your machine? Does it produce the failure mode? I'd be most appreciative if other owners could help out by trying the code I posted. It only takes a minute; just set the z height of whatever tool you have inserted at 1 inch above something recognizeable and see what happens. It'll only cut air. Try three or four restarts in a row, since sometimes it seems like the initial spindle position may also influence whether it fails or not, although I wasn't able to conclusively demonstrate this.

Thanks for any data points people can offer. I contacted Tormach about this yesterday, but haven't heard back yet. From what I gather, this might be related to a bug triggered by small radius arcs, which this code includes, and which the latest Sprutcam post was supposed to fix. I'm not clear on the details of that issue, however, I just gleaned a little information about it during my call to Tormach.

[mactec54]

bobeson

Try the N80 line as a G90 not G94 & the N110 line G43G0 Z-0.6 & remove the G0 Z-.62
It looks like your Post processor needs some work, you should program in absolute (Which is the G90) you may have to redo your program in the absolute to use the G90

[bobeson]

Hi Mactec,

Thanks for the suggestions, but as I mentioned, there are many ways to make the error not happen, but that's not what I'm looking for. I'm trying to identify the nature of the Mach bug that this code triggers, which is why I'm hoping to find out if other machines are experiencing this same bug with the code I posted.

The G90 mode is already set on line N20, and G94 just sets feed-per-minute mode (perhaps you are confusing G94 with G91?). Having it appear there does seems odd, but as you can see from the re-formatted code test case that I posted which actually exhibits the bug, I removed both the G90 and G94 codes entirely. The machine still has those modes set when I run the test case code.

Do you have a PCNC1100 that you could try running this code on? It would be most helpful to know if other people are experiencing this problem, or if it is somehow isolated to my machine. It should be a very quick experiment.

[mactec54]

bobeson

There is no problem with the Mach control your G code format is totally incorrect

The N20 line is not a call line, that is there to cancel any calls of those G codes used through your program, It's placed at the beginning of a program as a safety line of the G codes used in your program that are canceled at the start of your program

The G94 is mostly used for Turning not Milling & has no use in your program You are trying to do things that are not the norm for G code programing

[bobeson]

Mactec54,

I think you may be misinterpreting the situation here.
This is the code which reliably produces an error that I firmly believe is a Mach problem:

M3
G0 X-4.8179 Y-3.1978 Z1
G0 Z-0.62
G1 Z-0.625 F2.8
G3 X-4.8077 Y-3.1746 Z-0.625 I-0.02286 J0.02387
G1 X-4.8074 Y-3.1573
G2 X-4.8978 Y-3.0791 Z-0.625 I0.34972 J0.4959
G2 X-4.9531 Y-3.0124 Z-0.625 I0.44891 J0.42809
G1 X-5.1326 Y-2.7558
G2 X-5.1746 Y-2.6918 Z-0.625 I0.95593 J0.67395
G1 Y-2.8998
G3 X-4.8998 Y-3.1746 Z-0.625 I0.30019 J0.02541
G1 X-4.4827
G1 X-4.4824 Y-3.1642
G1 X-4.4778 Y-2.9463
G2 X-4.6868 Y-2.826 Z-0.625 I0.0327 J0.29853
G1 X-4.863 Y-2.5742
G2 X-5.0601 Y-1.5185 Z-0.625 I1.06505 J0.74512
G2 X-4.9157 Y-1.1653 Z-0.625 I1.26403 J-0.31068
G1 X-5.1746 Y-1.1383
G1 X-5.1798 Y-1.188
G0 Z0.2
(1)
m5
m30


That's it, no line N20 needed, no G94 codes at all. If I just boot up my machine and run the code I just listed in this message, I'll observe the error condition. I've done it dozens of times now. The code you are referencing, which I mentioned seperately but included as part of my initial posting to this thread, was a sample of the raw output from the Sprutcam post, which I included just as an FYI to show that I'm using inch mode, absolute references, etc. It's not really relevant to inducing this bug. That section is pure Sprutcam postprocessor output. The Gcode I write looks much different than that, believe me. I know why the N20 line is put there, and I don't need general guidance regarding gcode style in order to address this observed error condition.

For the sake of characterizing the nature of this Mach control bug, which is the point of this thread, I am soliciting help from other Tormach owners in running the code fragment I included in this message, and reporting if they observe the error or not. It would be of great assistance to know if anybody else observes these failures, or if they are for some reason limited to my machine. I have spent considerable time in carefully isolating and characterizing the code which produces this failure. I believe that the code I just posted provides strong evidence which points to some kind of Mach bug which I'd like to have fixed. If you doubt my claims, please try to explain how deleting the comment near the end of the code I posted can affect the execution of the third line of code, "G0 Z-0.62" ? Perhaps if you read my initial posting again, you will conclude that the observed nature of this error is indicative of a control system failure, not a gcode failure.

[wvines]

Bobeson,

I tried the failing code several times on my 1100 but could not duplicate the problem.

Walter

[neilw20]

It really sounds like the acceleration/feedrate is the problem causing the steppers to lose steps.
The clue: "is making a distinctly audible humming noise while stalled."
This can be caused many things. Here are some.

1. While moving above a critical speed the PC interrupts the smooth flow of pulses (pauses for some milliseconds).
Above a critical speed the stepper cannot instantaneously change speed. Inertia wins.
IDE drives (CDROM or HD) can steal cpu cycles with DMA interrupts.
If your BIOS allows disabling DMA access to the hard disk, this often will fix it. A hard disk in poor condition will can make the PC go off in the 'weeds' and Windows doesn't care. You just drop CPU cycles for the end user. Tuff!
SCSI drives do not create this problem.

2. Stiffness or restriction in the movement. This would not appear to be the case unless the X axis is also sticky.

3. Motor tuning is not optimal..

4. The power supply voltage to the drivers is lower than what it was for the original setup for motor tuning, or your stepper motors characteristics are different.

Initial tests to resolve this problem: Item 3.

Reduce the acceleration setting to half of what it is, and reduce the max speed rate to half or even much less.
Keep halving the rate(s) until the problem is not there.

Now keep increasing the max speed until you get the fault.
Reduce the setting that works reliably by maybe 20%.

Do the same now for the acceleration until it faults, then reduce by maybe 30%.
Do this for each axis.

Do the tests under program control as the rapid feeds are not the same process as rapid feed jogging.

For test purposes, do one axis at a time just using rapid to move backward and forward between 2 points, just marked with a pencil on the slides.

You can use the Mach3 feed control slider to increase or decrease to save a lot of changing of the tuning.

When you find the sweet spot, multiply the tuning value by the slider setting, then go back to 100% to fine tune further.

While you use IDE drives, there is always a possibility of interrupted pulse train(s) to the driver(s) above a critical speed.

On my machine, for critical work I halve the max speed, as insurance.
It clunks (which makes your heart miss a beat) but NEVER looses steps, after 1500 hours of the odd clunks, machining expensive material. It a 2GHz CPU, with IDE drive, and DMA can't be disabled. Thanks IBM !!

[bobeson]

wvines,

Thanks for the data, that is helpful to know.
Are you running the most recent control software release?

[bobeson]

neilw20,

Thanks for the suggestions. I'm confident that motor tuning and PC performance are not the causes of my failures, given that I can induce the failure or make it not happen simply by removing a comment in an apparently-unrelated section of code. No changes to feed rate, no apparent problems with axis moves over hundreds of operating hours so far. I'm running a stock machine here, nothing tweaked about it. For some reason, the control system is *reliably* giving out bad commands under certain conditions, which results in the motor "stalling" (I don't know what the actual failure mechanism is, only that the axis doesn't move and a distinct humming sound is produced).

[neilw20]

Try running DriverTest.exe (in Mach3 folder)
It could show up something. -- Like DMA interrupts stealing CPU cycles.
Does running the machine at reduced feed rate on the Mach3 feedrate slider eventually make it work?
When is the last time you defragged the HD?

[bobeson]

After both Walter (wvines) and Daniel from Tormach reported they were unable to reproduce the error I observed, I elected to re-install the system control software. I used a freshly-downloaded copy of the same identical system software upgrade as I'd previously installed, only this time I didn't include the Duality software. I copied over my tool3.dat and fixtures.dat file, loaded the test case, and did not observe the failure mode that I had previously been able to reliably induce. I tried numerous versions of my code that had previously failed, and was entirely unable to reproduce the error. So, apparently something unique about my control software installation was sensitive to the code in my test case, which is very odd indeed.

Based on my inability to reproduce the error, I guess I'm going to press ahead with production, and keep a close eye on the spindle position for a while. I do hear some kind of "clunk" sound while the G0 Z-0.62 move that previously failed passes through Z=0, which is where it used to fail with a sustained humming sound. On a couple of tests I measured ~0.003-0.005" of Z-axis offset that I suspected may have been from the clunk, but it was not readily reproduceable, and given the small magnitude of the offset I cannot rule out measurement error of one sort or another.

Anyhow, back to work for me now.

Thanks for your efforts to reproduce the error, Walter and Daniel!

[bobeson]

Sadly, I spoke too soon. As it turns out, the initial reproduceable error had gone away, but once I reset my part and started cutting again, half an hour later I hear that distinctive humming sound again and then my X axis has an offset error, in the same program location I had discovered it a few days ago. I tried again and the error occured again in the same portion of the code (I didn't isolate the exact move, but it was a rapid) and with the same resulting X axis offset (~0.75") . The error condition was marked by a distinct humming sound before the axis error then registered in the DRO.

Tired of debugging code, I decided just to push my way through, and I massaged the code enough so the error would disappear. Mostly I deleted extraneous G0 lines, added and deleted comments, and cleaned up a little bit of the messy postprocessor stuff, for the remainder of my part program. Then the X axis error went away, and I proceeded for another 2-3 minutes of cutting before another error happened, this time during a Z rapid up move. More code massaging, this time only with comments, and the error goes away when I restart. Now I decide to run the rest of the ~hour long program in air, without changing the (slow) feed rate, and observe several more error instances, all of them Z axis G0 moves, which were eliminated by just changing around the text of the gcode a little bit, all meaningless stuff like adding/deleting comments and line spacings. Once these were all removed, I proceed to run my code.

Keep in mind that this is taking place on a newly-installed copy of the control software, on a control PC purchased from and configured by Tormach, with no additional software installed, never put on a network, etc. I've kept it as pristine as possible, to avoid this kind of headache.

Then another old nemesis appears, the spindle speed error condition. This is something I've encountered on and off the entire time I've owned my mill (~3 years), where the spindle will suddenly drop in speed by a consistent ratio of the original speed. Tormach suggested checking all the parallel port connections, and it seems like opening the cabinet and reseating the cable there might help for a while, but the problem kept coming back nonetheless. I then adjusted all of my programed speeds by the error-mode speed ratio and pressed onward, hoping that the error-mode would quickly re-appear if it ever went away, which it seemed to do fortunately (or not). Eventually I got through all of the roughing, and now I'm about to re-install the TTS drawbar setup (I had to use R8 set-screw holders for roughing because of TTS collet pullout problems) and hope the finishing moves are able to complete.

I'm not having much fun here.

[lwoestman]

Have you checked the computer that is running Mach for ram errors? Your description reminds me of that cause. There is a self-contained ram test called "memtest86+" that you could run to rule out that cause.

[neilw20]

http://www.cnczone.com/forums/954056-post9.html

[bobeson]

Neilw20,

I do appreciate your efforts at helping me resolve this matter, and I hate to think you feel you're wasting your time. In this particular matter, the specific suggestions you've made are not indicated by the observations I'm making, IMHO. More to follow...

-Bob

[bobeson]

Yesterday, after finally completing the part I'd been struggling with so much, I needed to make a second copy, and elected to reprogram the CAM based on what I'd learned from the previous episode.

I made quite a number of small changes, perhaps most significantly was eliminating many vestigial entry and exit moves that I'd shortened up considerably in an effort to minimize non-cutting time. This eliminated a lot of small-radius arcs and short distance z-axis cut moves. I also changed my tool geometry - I went from a 2-flute to a 4-flute end mill for my rouging operationgs. I'd switched my previous code to use a HSS 2-flute (I'm profile cutting full-thickness 3/8" 1020CR) in an effort to minimize horsepower to prevent collet slip, but now that I've switched to a set-screw R8 holder for roughing the 4-flute is a much better choice, so I reprogrammed for that.

Then, and I suspect this may have been the most important part, I went through the post-processesor output with a fine-toothed comb, and rewrote all of the funky, extraneous G0 commands and feed commands, etc. that seem to be at the beginning and end of all the operations in the output of the current postprocessor. I put the code into as clean a format as possible, but without changing any of the actual toolpath code at all. I also went and tightened up all the parallel port cabling I could, for the umpteenth time, but this time I also really wiggled the IDC cable connecting to the control board, hoping to reseat any loose connectors in their piercing seats.

For the second part, with identical geometry and overall strategy but with the changes mentioned above, I was able to run it from start to finish, including four setups, without error. I tested it all the way through the roughing stages in air first, and did not observe any of the repeatable motor "stalling" error condition I'd observed with the previous part program. When I went to cut the part, the machine vibrations were much improved thanks to the switch to a four-flute rougher, and I did not experience any of the spindle speed change gremlins that had struck me during the first part.

So, now I'm having fun with my machine again, and I have two pretty parts to admire.

However, based on all of my experiences with this control system error, I am still confidently asserting that something about the current postprocessor output is triggering a Mach bug. It could be just my short entry moves or some other aspect of the programming of my first part program, but there is no question that my pcnc1100, with a clean system software install on a clean computer, was performing consistently flawed moves under certain apparently-permissible conditions, and that these errors went away when I simply "massaged" the gcode format a bit (comments, whitespace, little stuff). I would not be at all surprised to learn that other users have experienced or will experience this error condition at some point as well. Based on the repeatable nature of the error, and the absence of the error under other conditions, I do not believe that RAM or motor tuning or computer tuning are at issue here. I say this as a computer hardware design engineer with deep experience in debugging complex embedded systems such as this; I'm not just dismissing otherwise valuable suggestions as to those other causes. Given the amount of time I've spent already characterizing this bug, I'm inclined just to stick with my workarounds for now, which will be to be cautious with my entry and exit moves and rewrite all the SprutCAM postprocessor output, and air-cut everything the first time through, listening for the axis motor "stall" sound again before I run any new programs in stock.

As for the spindle-speed error, I am suspecting that Tormach's diagnosis of a parallel port connection issue is correct. I will try to order a new IDC cable for the control board. Unfortunately, the connector header on the control board has no retention mechanism, so the connector will always be prone to loosen again. The main header connection itself probably isn't the problem though; I suspect the actual insulation displacing cable piercing elements, which tend to loosen up or never be properly set to begin with. I notice that the control board has no surface-mount components on it, only through-hole, which tends to drive up the cost of a board these days. Presumably, this was chosen in an effort to improve reliability in the vibration-filled machine control cabinet. But then they chose a cheap IDC cable connector with no retention mechanism - those things are the worst! Sigh. Maybe I'll just solder the ribbon cable to the board. This would be a great application for simple flexible printed circuit, with soldered connectors replacing insulation-displacement connections.

In any case, thanks for all of the suggestions. I'll be most interested to hear if anybody else experiences unexplained axis offset errors such as I've characterized here. I do think a bug is lurking in Mach, and some "magic" code combinations are prone to encountering it.

[neilw20]

Hi Bob,

I have based my notes on the following, and past experience of similar problems. I have a hardware/software background in real time systems (DEC RSX11M 8 years)
Time
and attached document, from the Author of Mach3. Thanks Art!

Neil

[flyinchips]

No un-explained problems here! I just keep a close eye on the operator (me)! So far, 100% of the problems have been that operator.